## Motivation
Applying #165 recreated the Alliance SwissPass pull zone without its custom hostname because the hostname association was not represented in OpenTofu.
The recreated zone also received a new numeric ID, making the weekly purge configuration stale.
## Changes
- Manage `cran.allianceswisspass.devxy.io` as a pull-zone hostname with TLS and forced HTTPS.
- Resolve the Alliance pull-zone ID from its hostname before purging instead of persisting a replaceable numeric ID.
- Install `jq` in the purge step for the Bunny API lookup.
## Verification
- Targeted `prek` hooks pass.
- `tofu validate` passes.
- `crow lint .crow/` passes.
- `just edge-test` passes all 14 routing steps.
- `bash -n scripts/purge_cdn_zone.sh` passes.
## Deployment
Run `tofu apply` to restore the Alliance hostname on the recreated pull zone.
Reviewed-on: #166
## Motivation
`cran.rpkgs.com` and `cran.allianceswisspass.devxy.io` serve the same B2 repository through separate Bunny pull zones, but only the first zone was managed and purged after weekly reindexing.
This allowed the Alliance endpoint to retain stale repository metadata and left locked `renv` restores unable to retrieve versions whose binary archive object was absent.
## Changes
- Adopt the Alliance SwissPass pull zone `3265648` into OpenTofu and configure it with the shared B2 origin and middleware script.
- Purge both Bunny pull zones after the weekly rebuild reindex.
- Preserve the requested public hostname in middleware redirects.
- Redirect missing archived binaries to the corresponding CRAN source package, checking whether the version is archived or still current.
- Cover the existing archived-binary passthrough behavior in the edge routing matrix.
## Verification
- `prek run -a`
- `just edge-test`
- `crow lint .crow/`
- `tofu validate`
- `bash -n scripts/purge_cdn_zone.sh`
## Deployment
Run `tofu apply` to adopt pull zone `3265648`, publish the middleware release, and align both pull zones.
After the apply, rerun the Alliance SwissPass CI restore that requested `cli 3.6.5` and `AzureStor 3.7.1`.
Reviewed-on: #165
## Motivation
Weekly rebuild shards can all hit a transient CRAN DNS/index outage at once, and the dependent CDN purge always fails because it tries to clone over the checkout preserved from the re-index step.
## Changes
- Retry `uvr add` resolution up to four times with bounded backoff.
- Reuse the existing Crow workspace checkout in the CDN purge step.
- Remove the purge step's unused Git package and repository token.
## Validation
- `crow lint .crow/`
- `shellcheck local/uvr-install.sh scripts/purge_cdn_zone.sh`
- `git diff --check`
Reviewed-on: #164
## Problem
`weekly-rebuild-missing` runs one job per `<os>-<arch>` and walks that slot's list serially in a single `R -q -e` argument.
That was cheap while every source fallback was skipped as "already built".
Since bincraft #105/#106/#107 and #159 the gate works, and the lists are large: 8 917 source-served records on `amd64/alpine324`, 15 023 on `amd64/resolute`.
Pipeline 10910 (`weekly_rebuild_missing:alpine-324-amd64`) ran for two days, reached `[8692/23885] cholera`, and was killed there.
Two failures follow from that shape:
- **No parallelism.** The work is embarrassingly parallel across packages; one job does all of it.
- **No resumability and no clean stopping point.** The loop ends only by exhausting the list, so the only way to stop it is a kill. A restart re-walks from the first entry, paying a CRAN version resolution and an S3 `HEAD` per package before reaching new work. And a kill matches neither `success` nor `failure`, so the `Purge CDN cache` step never ran: the ~4 600 binaries 10910 did publish stayed hidden behind stale edge copies.
## What this changes
**Three shards per slot.** Each of the 18 `OS`/`ARCH` rows gains `SPLIT_INTO`/`SPLIT_INDEX`, mirroring `build-all-versions.yaml`. Cron and manual routing are unchanged: both filters already match on `${OS}-${ARCH}`, so they now match all three shards of a slot.
**`local/rebuild-missing.R`** replaces the ~1 500-character inline one-liner. The slice is interleaved rather than contiguous, because the list is alphabetical and cost clusters by name (`Rcpp*`, `Bioc*`, `rstan*`).
**Resume by re-deriving state from the bucket.** One `s3_dir_info()` listing gives ETags for the slot; a package is outstanding iff its object's ETag equals CRAN's published `MD5sum`, i.e. it is still byte-identical to CRAN's source. That is `check_s3_root_package()` evaluated in bulk. No progress file, no volume, no DB cursor, and correct when a sibling shard or a `process-updates` run completes something concurrently.
It reads ETags rather than the index's `Built` field the way `packages-to-build.R` does, because the index is no longer rewritten until the dependent pipeline runs and so cannot reflect the current run's progress.
Unknown always means "already a binary", never "rebuild it": a multipart ETag, an unreadable CRAN index or an empty listing can never mass-schedule work.
**A 20 h wall-clock budget** per shard. It exits 0, so the re-index and purge always fire and the remainder is picked up next run with no bookkeeping.
**`.crow/weekly-rebuild-reindex.yaml`** takes over re-indexing and the purge, with `depends_on: [weekly-rebuild-missing]` and `runs_on: [success, failure]`. Three shards writing one slot's `PACKAGES` concurrently would race: `update_PACKAGES()` lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work.
## Verification
`crow lint .crow/` passes on all 11 pipelines. `prek run` passes.
19 assertions in `local/tests/test-rebuild-missing.R`, 0 failures, covering the partition (disjoint, covering, deterministic, short lists, out-of-range index) and the outstanding filter (source ETag kept, binary ETag dropped, absent object kept, multipart and missing-from-CRAN treated as built).
One of those tests caught a real bug before it shipped: an empty ETag table indexed to zero length rather than to `NA`, which recycled the result away and reported "nothing to build" — the dangerous direction. Fixed with an explicit `lookup()`.
The filter run against the live `amd64/alpine324` index, using its `MD5sum` column as the ETag (established to match the objects):
```
index packages: 24343
outstanding (filter): 8950
no Built stamp: 8917
filter vs no-Built agreement: 8917 of 8917
outstanding but stamped Built: 33 (version drift vs CRAN)
shard sizes: 2984/2983/2983 (sum 8950, unique 8950)
```
It reproduces the source-served set exactly. The extra 33 are packages whose slot version differs from CRAN's current one, so no object exists at the CRAN version key: correctly outstanding.
## Notes for review
- The 20 h budget is a chosen default, exposed as `REBUILD_BUDGET_HOURS` in the pipeline.
- `depends_on` is file-level, not row-level, so on a full cron run no slot is re-indexed until the slowest of all 54 jobs finishes. The budget bounds that at roughly a day.
- An explicit cancel still skips the re-index. Recovery is to trigger `weekly-rebuild-reindex` on its own.
- The purge runs on every re-index row rather than one designated slot: a cron fires only its own slot's row, so gating on a named slot would leave every other slot unpurged.
- Out of scope: `build-all-versions` still cannot rebuild source fallbacks, because `local/build-all.R:113-122` drops every version with any `single_builds` row, which is precisely the source-fallback set.
Design: `specs/2026-08-12-shard-weekly-rebuild-design.md`
Reviewed-on: #163
## Problem
The rebuild now works — `AATtools 0.0.3` was detected as a source fallback, built, and published:
```
ℹ `upload_single_binary()`: Replacing the CRAN source published for AATtools 0.0.3 … with the binary.
✔ Successfully uploaded package AATtools with tag 0.0.3.
```
But clients still get the source, and will for about a year:
```
$ curl -sI .../src/contrib/AATtools_0.0.3.tar.gz
etag: "ea8127d953ca6a2f118ea49441772af6" # CRAN's source MD5
cdn-cache: HIT
cdn-cachedat: 08/09/2026 16:43:32 # predates the 18:12 upload
```
Two causes, both specific to a rebuild:
1. **The slot is never re-indexed.** `weekly-rebuild-missing` has no `upload_package_index` step, so the index keeps the old MD5 and — for anything that had been served from source — no `Built` stamp. This one self-heals at the next `process-updates` run.
2. **The tarball URL is never purged.** A normal update publishes new packages at *new* URLs, so `purge_cdn_cache.sh` only needs the five index files. A rebuild replaces an object *in place*, and the zone caches tarballs for `cache_expiration_time = 31919000` (~370 days). This does not self-heal.
Nothing about a stale package looks wrong from the outside, which is what makes it worth fixing rather than documenting.
## What this changes
**Re-index at the end of a rebuild**, flat and per-minor, mirroring the tail of `process-updates`. The codename is detected from the image's `/etc/os-release` (as `local/packages-to-build.R` already does) rather than adding `OS_ID` to all 18 matrix rows.
**Purge the zone afterwards**, via a new `scripts/purge_cdn_zone.sh`. One call to `POST /pullzone/{id}/purgeCache` covers every replaced object, and all three hostnames — `cran.devxy.io`, `cran.allianceswisspass.devxy.io`, `cran.rpkgs.com` — share pull zone `3857050`, confirmed from the `cdn-pullzone` response header.
Purging per URL was the alternative and is worse here: ~13.5k rate-limited calls per arch, where a single missed call leaves a package silently stale. The cost of the zone purge is a cold cache for everything else, which is why it stays out of the daily update path — `purge_cdn_cache.sh` is untouched.
The purge runs on failure too (`when: status: [success, failure]`): a rebuild that died part-way still replaced objects, and those are exactly the ones a stale edge keeps hiding.
## Verification
`crow lint .crow/` reports all ten configs valid; `bash -n` on the new script passes; prek hooks pass.
Not yet exercised against Bunny — it needs `BUNNYNET_API_KEY`, which is a CI secret. The failure mode is explicit rather than silent: any status other than 200/204 prints the response body and exits non-zero.
Reviewed-on: #160
Unblocks the ubuntu 26.04 builds without waiting for an image rebuild.
## What broke
```
i Installing R package dependencies
! Error in installing dependencies for package huge with tag 2.0.1:
uvr sync --install-system-deps --library /mnt/cache/R-pkgs-4.4 failed (exit 1)
! WARN Missing system dependencies for 1 package(s)
igraph needs: libglpk-dev
> Running: apt-get install -y libglpk-dev
E: Unable to locate package libglpk-dev
```
Neither the package name nor the sysreq mapping is wrong. `libglpk-dev` is `5.0-2build1` in `universe`, which `ubuntu:resolute` enables by default. What is missing is the apt index: uvr v0.4.5 runs `apt-get install` with no `apt-get update` before it, and every ubuntu build image ends its apt layers with `rm -rf /var/lib/apt/lists/*`, so the shipped image has no index at all.
Not specific to 26.04 — jammy and noble fail identically for any sysreq the image's preinstall line doesn't already cover. 26.04 surfaced it first, through `igraph` and everything depending on it (`huge` was the first to fail).
## Fix
`apt-get update` before the sync in `local/uvr-install.sh`.
That one site covers every pipeline: `build-all-versions`, `build-all-versions-install-deps`, `weekly-rebuild-missing`, `weekly-audit-missing`, `trial-build-registry`, `auto-apply-patches` and `weekly-patch-proposals` all reach it, either directly or through `install-bincraft.R`. The index it populates persists for the rest of the step, so the bincraft-driven `uvr sync` calls that follow are covered too.
Guarded on `apt-get` and non-fatal: `apk add` fetches its index implicitly and dnf refreshes expired metadata on its own, so alpine and redhat are unaffected and skip it.
## Why here as well as in the image
The real fix is upstream `e491b2e`, which refreshes the index before the auto-install. It landed one day after v0.4.5 was tagged, so no release carries it ([nbafrank/uvr#250](https://github.com/nbafrank/uvr/issues/250)); [build-env-images#23](https://codefloe.com/rpkgs/build-env-images/pulls/23) pins the ubuntu image to a commit that has it.
That only reaches CI after an image rebuild is triggered. This change takes effect on the next pipeline run. It also stays useful afterwards: a baked index goes stale within weeks, and a stale index turns the same install into a 404 on the `.deb`.
## Verified
In `ubuntu:resolute`, with the image state reproduced exactly (R installed, then `rm -rf /var/lib/apt/lists/*` to empty the index again):
| Check | Result |
|---|---|
| `apt-get install -y libglpk-dev`, empty index | `E: Unable to locate package libglpk-dev` — the reported failure, reproduced |
| Same container after `apt-get update` | `Candidate: 5.0-2build1`, `Components: main universe restricted multiverse` |
| `igraph` + `UVR_INSTALL_SYSREQS=1 uvr sync`, empty index, uvr with the refresh | `Setting up libglpk-dev:amd64 (5.0-2build1)`, `System dependencies installed.`, `Installed 11 package(s)` |
| `sh -n` and `shellcheck local/uvr-install.sh` | pass |
## Reverting
Delete the guarded block. It is marked TEMPORARY with the upstream reference, and can go once the images ship a uvr above v0.4.5.
Reviewed-on: #161
## Problem
This is the gap flagged in rpkgs/bincraft#106. `build_binary_package()` has a fast path that compares against `s3_package_cache` instead of querying S3 per package, and that cache is produced here:
```r
s3_pkgs <- s3fs::s3_dir_ls(".../latest/src/contrib", recurse = TRUE)
saveRDS(basename(s3_pkgs), "/mnt/cache/packages/s3_cache.rds")
```
A raw bucket listing cannot tell a binary from a package whose build failed and was published as its CRAN source — the two occupy the same key. So every source fallback reads as "already built" and is skipped for good. That is how `alpine324` accumulated ~13.5k of them.
The same listing feeds `s3_dt`, which is subtracted from the build list at line 194 (`pkgs <- pkgs_no_error[!s3_dt]`). That one matters more: it excludes the very packages that need building, before `build_binary_package()` is even called.
## What this changes
Drops from the listing every object the slot's own index reports as served from source. bincraft leaves the `Built` stamp off exactly those records (rpkgs/bincraft#105), so the index already carries the answer and no credentials, downloads or extra API calls are needed.
Both consumers are fixed: the saved cache and `s3_dt`.
The cache stays a plain filename vector, so the build container still needs no `s3fs`/reticulate — that was the point of saving it in the first place.
Two deliberately conservative edges:
- archived objects have no index record, so they are kept. Unknown means binary, never "rebuild it".
- if the index cannot be read, the full listing is kept and a warning is printed, so a CDN blip cannot mass-schedule a rebuild.
## Verification
The script parses, and the new block run against the live indices:
```
amd64/alpine324: index=24235 source-served=13542 e.g. AATtools_0.0.3.tar.gz, ABCDscores_7.0.0.tar.gz
amd64/noble: index=24681 source-served=0
```
`alpine324` is re-indexed by bincraft 5.1.1, so 13 542 objects drop out and those packages become buildable. `noble` has not been re-indexed yet, so every record still carries `Built`, nothing is dropped, and its behaviour is exactly what it is today — the safe failure mode this relies on.
The new log line makes it visible per run:
```
S3 cache: N objects, M served as CRAN source, K usable binaries
```
## Sequencing
Needs rpkgs/bincraft#106 (and a release) before a rebuild actually builds: this fixes the bulk build path's list, #106 fixes the per-package pre-build skip.
Reviewed-on: #159
## Problem
A manual `crow pipeline create` instantiates **every** file in `.crow/`, and a declared variable default is applied even when the run never passed that variable. A gate is therefore only a gate if its default matches nothing.
#155 fixed the three pipelines that had no manual gate at all. It missed that a *permissive default* leaves a pipeline just as exposed. Demonstrated the expensive way: creating a pipeline with only
```
--var weekly_audit_missing=alpine-324-amd64
```
also started `build-all-versions` — because its gate `target_arch` defaults to `amd64`, which matches its own amd64 matrix rows — and `process-updates` across every row, because that gate defaults to `all`. The run was killed before any `Upload package indexes` step produced output and both alpine324 indices were verified unchanged, but `build-all-versions` uploads binaries and rewrites indexes, so the next one might not be caught in time.
Before:
| pipeline | gate | default | fired on an unrelated manual run |
| --- | --- | --- | --- |
| `build-all-versions` | `target_arch` | `amd64` | amd64 rows — builds and uploads |
| `build-all-versions-install-deps` | `target_arch` | `amd64` | amd64 rows |
| `weekly-rebuild-missing` | `weekly_rebuild_missing` | `all` | every row |
| `weekly-audit-missing` | `weekly_audit_missing` | `all` | every row |
| `process-updates` | `process_cran_updates` | `all` | every row |
| `repair-built-stamp` | `repair_built_stamp` | `arm64` | arm64 rows |
`archive-missed-packages` was the one that behaved, because its gate variable is never declared and so matches nothing. That is the property this restores everywhere.
## What this changes
Each of the six gets a `none` option on its gate variable and defaults to it, so a manual run has to name its target explicitly. The reason is recorded next to the default, where someone would go to change it.
`none` is used rather than dropping the default so the expression always has a defined value to compare, instead of relying on undefined-variable semantics.
Cron triggers are untouched — they match on the `cron:` name, not the variable.
## Verification
`crow lint .crow/` reports all ten configs valid. Auditing every pipeline that accepts a manual event:
```
archive-missed-packages.yaml: gate=task default=<none>
auto-apply-patches.yaml: gate=auto_apply_patches default='false'
build-all-versions-install-deps.yaml gate=target_arch default=none
build-all-versions.yaml: gate=target_arch default=none
process-updates.yaml: gate=process_cran_updates default=none
repair-built-stamp.yaml: gate=repair_built_stamp default=none
trial-build-registry.yaml: gate=trial_build_registry default='false'
weekly-audit-missing.yaml: gate=weekly_audit_missing default=none
weekly-patch-proposals.yaml: gate=weekly_patch_proposals default='false'
weekly-rebuild-missing.yaml: gate=weekly_rebuild_missing default=none
```
Every gate now defaults to something that matches no matrix row.
Reviewed-on: #158
## Problem
The audit's definition of "missing" was `missing_dt <- cran_dt[!s3_dt]`, where `s3_dt` came from listing the bucket. But an object being present does not mean a binary was built: when a build fails, bincraft publishes the CRAN source tarball in its place so the package stays installable, and a listing cannot tell the two apart.
So every failed build looked done. On `amd64/alpine324` that is 13 547 of 24 134 comparable objects (56%) — all CRAN sources from the June 2026 bootstrap, none ever reported here, so `weekly-rebuild-missing` never retried any of them. `amd64/noble` sits at 4.6% for comparison.
## What this changes
Reads the slot's own `PACKAGES` index instead of listing the bucket, and counts only records carrying a `Built` stamp. bincraft writes that stamp only for what it actually built (rpkgs/bincraft#105), so the index answers the question a listing cannot.
Side effects of reading the index rather than the bucket:
- the audit reports how many records the index holds and how many are served as source, so the gap is visible in the log
- `s3fs` is no longer used, so the audit needs no B2 credentials
- an object present but absent from the index is now counted as missing, which is correct: clients only see the index
## Sequencing
This is inert until the matching bincraft change ships and a slot is re-indexed. A slot last indexed by an older bincraft carries `Built` on every record, including the fallbacks, so the audit reports exactly what it does today. Verified against the live indices:
```
amd64/alpine324: records=24152 built=24152 source-served=0
amd64/noble: records=24681 built=24681 source-served=0
```
After a re-index those `source-served` counts become the real ones, and the packages behind them start appearing in issue #63 and getting rebuilt.
Reviewed-on: #157
## Problem
`tofu plan` after importing the middleware script shows:
```
~ resource "bunnynet_pullzone" "cran_rpkgs_com" {
~ cache_vary_headers = [
- "User-Agent",
]
```
The zone carries `cache_vary_headers = ["User-Agent"]`, set before this configuration existed. `cdn.tf` never declared it, so the first apply of the managed middleware would remove it — as a side effect of an unrelated change, with no decision recorded anywhere.
## What this changes
Declares the attribute with its current value, so the pull zone is a no-op in that plan.
## Why keep it rather than let it go
On paper the router makes it redundant. The only UA-dependent responses it produces are redirects, and those carry `Cache-Control: no-store`; their targets are concrete per-slot, per-minor URLs whose content depends only on the path. Dropping the vary would also be a genuine win, since otherwise every distinct R version string (`R (4.5.3 x86_64-pc-linux-musl …)`) keys its own copy of every tarball.
It stays anyway, for now:
- it is the second line of defence against the one failure mode that would be quiet and confusing — an R 4.6 client served the 4.5 index
- Bunny honouring `no-store` on an edge-script response has been confirmed for today's redirects (`cdn-cache: BYPASS` on `max-age=0`), but not for the new script in production
- keeping it is the status quo, so it cannot regress anything
Removing it is worth doing on its own, once per-minor routing is confirmed live and the redirects can be observed bypassing cache — not as a side effect of enabling that routing.
## Verification
`tofu validate` passes. Re-planning after this merges should leave `bunnynet_pullzone.cran_rpkgs_com` unchanged, reducing the plan to the script `content` update and the new `UNION_SLOTS` variable.
Reviewed-on: #156
## Problem
A manual `crow pipeline create` instantiates **every** pipeline in `.crow/`, and each one decides for itself whether to run. Three had nothing to decide with — their only manual condition was a bare `event: manual`:
- `auto-apply-patches` — pushes to `auto/registry-patch-proposals` and opens/updates a PR
- `weekly-patch-proposals` — posts and edits two Forgejo issues
- `trial-build-registry` — starts a build per matrix row, on both arches
So they fired on *any* manual trigger in this repo, whatever it was for. That is how they came to run alongside a manual `process-updates` run for `alpine-324-arm64` (#10723), which is also why that pipeline is marked failure.
`repair-built-stamp.yaml` already documents the rule this breaks:
> The gate variable is `repair_built_stamp`, not `target_arch` … Every pipeline here gates on a variable named after itself for exactly that reason.
## What this changes
Each of the three gets a gate variable named after the pipeline, `evaluate`d on the manual event, defaulting to off:
```yaml
variables:
auto_apply_patches:
description: 'Run the auto-patch proposer. Also gates this pipeline.'
options: ['true', 'false']
default: 'false'
when:
- event: manual
evaluate: 'auto_apply_patches == "true"'
- event: cron
cron: auto-apply-patches
```
Cron triggers are untouched, so the scheduled runs behave exactly as before.
The run-manually comments in all three headers were also stale: they documented `--var task=<name>` with `woodpecker-cli`, and no pipeline evaluates a `task` variable. They now show the real invocation.
## Note on the sibling pipelines
The already-gated pipelines use `default: all` (e.g. `weekly_rebuild_missing`). If Crow applies a declared default to a variable that an API-created pipeline never passed, those would match on an unrelated manual run too — `weekly-rebuild-missing` would be an expensive way to find out. I could not settle that from #10723 because its step logs have since expired, so I left them alone rather than guess. The three fixed here default to `'false'`, which is safe under either semantics.
## Verification
`crow lint .crow/` passes. Auditing every pipeline that accepts a manual event now reports a gate on all ten:
```
build-all-versions-install-deps.yaml: gated
auto-apply-patches.yaml: gated
weekly-patch-proposals.yaml: gated
weekly-audit-missing.yaml: gated
repair-built-stamp.yaml: gated
archive-missed-packages.yaml: gated
weekly-rebuild-missing.yaml: gated
trial-build-registry.yaml: gated
build-all-versions.yaml: gated
process-updates.yaml: gated
```
Reviewed-on: #155
## Problem
`install.packages("curl")` fails in `reg.devxy.io/r/r-alpine:4.5-3.24` with "package 'curl' is not available for this version of R", on both arches.
`curl` is not missing from the repo: it is in `…/latest/src/contrib/4.5/` and `…/4.6/`, the per-minor slots that base R cannot address. The image's repo URL resolves to `…/latest/src/contrib`, whose index does not list it. On `amd64/alpine324` that is 2 886 packages invisible to `install.packages()` (23 on `amd64/noble`) — what issue #63 records as "missing binaries".
Two further findings while investigating:
- The middleware only ever rewrote the bare `cran.rpkgs.com/src/contrib/…` form, and that form was broken for every Linux client on a stock R user agent: `ALPINE_REGEX`/`UBUNTU_REGEX`/`RHEL_REGEX` need a Posit-style UA that carries the distro, so stock R fell through to `extractOs()` and got redirected to `/amd64/linux-musl/latest/…`, a slot that does not exist.
- `PACKAGES*` is served `cdn-cache: BYPASS` (bincraft uploads it `no-store`), so the middleware sees every index request and no purge is needed for routing changes to take effect.
## What this changes
**`edge/rpkgs-router.ts`** — the middleware, now a reviewed file in this repo rather than dashboard state. It routes `PACKAGES`, `PACKAGES.gz` and `PACKAGES.rds` into `…/src/contrib/<x.y>/` for slots listed in `UNION_SLOTS`, and nothing else.
Tarballs are deliberately left alone. R keeps the `contriburl` it *asked for*, not the one the redirect served it, so every tarball URL is resolved against the flat directory and the union index steers the per-minor ones with a `Path: <x.y>` field. Rewriting a tarball request here would send flat-slot packages into a directory that does not hold them.
Also in the script: the phantom `linux-gnu`/`linux-musl` fallback is gone (an unidentifiable distro goes to CRAN, as an unparseable UA already did), and every redirect carries `Cache-Control: no-store` since its target depends on the User-Agent. The macOS branches are unchanged.
**`cdn.tf`** — `bunnynet_compute_script.rpkgs_router` with `content = file("edge/rpkgs-router.ts")`, the `UNION_SLOTS` variable, and `middleware_script` pointing at the resource instead of the literal `29277`.
`UNION_SLOTS` is empty, so merging and applying this changes no client's behaviour. A slot is added only once bincraft has republished its per-minor index as a union (rpkgs/bincraft#97); routing to a raw per-minor index would hide every package it does not carry. Rolling back is a variable edit, not a deploy.
**`specs/`, `plans/`** — the design and the implementation plan, including the two approaches that were rejected (edge-side merge, moving the minor up the path) and why.
## Verification
`just edge-test` runs 13 routing cases against the SDK's local server, so what is tested is the artifact that gets deployed; pass-through cases proxy to the real origin. All pass.
End to end, with the middleware in front of a locally built union index for `amd64/alpine324` (31 507 records), inside the runtime image:
```
curl: 7.1.0 -> …/latest/src/contrib/4.5 -> curl_7.1.0.tar.gz 717 725 B
jsonlite: 2.0.0 -> …/latest/src/contrib -> jsonlite_2.0.0.tar.gz 1 055 849 B
```
`tofu validate` passes. `tofu plan` has not been run: no `BUNNYNET_API_KEY` available in this environment.
## Before applying
The script pre-dates this configuration, so it must be adopted, not created:
```sh
tofu import bunnynet_compute_script.rpkgs_router 29277
tofu plan
```
The plan should show an in-place `content` update and no replacement of the pull zone. Without the import, tofu creates a second script and repoints the zone at it. Note that `name = "rpkgs-router"` will rename the existing script on apply.
## Not fixed here
`install.packages("curl")` on `alpine324` will now *resolve*, then fail to build: that slot's tarballs are byte-identical CRAN **source** tarballs (no `Meta/`, no `Built:` in DESCRIPTION) which the index nevertheless stamps `Built: R 4.5.3; …-linux-musl`. Sampled: `amd64/alpine324` 3/12 binary, `arm64/alpine324` 13/30, `amd64/noble` 12/12, `amd64/alpine323` 17/20. That slot needs a rebuild, tracked separately.
Reviewed-on: #152
## Problem
`repair-built-stamp.yaml` gated on `target_arch`:
```yaml
when:
- event: manual
evaluate: 'target_arch == "${ARCH}"'
```
That is the same variable `build-all-versions.yaml` and `build-all-versions-install-deps.yaml` gate on. A manual run passing `target_arch=arm64` to reach the repair therefore matched all three, so triggering a dry-run repair also queued a full arm64 rebuild.
I hit this triggering the alpine 3.22 dry run (pipeline 10706), which I killed. `crow pipeline ps` renders empty states on this version, so I could not confirm from the CLI whether the rebuild workflows started before the kill or only sat queued behind the running cron jobs; no output was attributable to them.
## Change
Gate on `repair_built_stamp` instead. Every other pipeline here already gates on a variable named after itself (`process_cran_updates`, `weekly_audit_missing`, `weekly_rebuild_missing`), and `target_arch` was the odd one out being shared by two.
The header now records the collision and the exact invocation, so the next pipeline added here does not repeat it.
## Verification
`crow lint` passes; `prek` clean.
Grep of every trigger condition in `.crow/` confirms `repair_built_stamp` is unique and that no other pipeline gates on `OS`, `OS_VERSION`, `R_VERSION` or `dry_run` alone:
```
archive-missed-packages.yaml: task == "archive-missed-packages"
build-all-versions.yaml: target_arch == "${ARCH}"
build-all-versions-install-deps.yaml: target_arch == "${ARCH}"
weekly-audit-missing.yaml: weekly_audit_missing == ...
repair-built-stamp.yaml: repair_built_stamp == "${ARCH}"
process-updates.yaml: process_cran_updates == ...
weekly-rebuild-missing.yaml: weekly_rebuild_missing == ...
```
## Note for whoever runs it
`crow pipeline create` against this instance returned HTTP 504 while still creating the pipeline. Retrying on that error duplicates the run: I created four before noticing. Verify with `pipeline ls` rather than trusting the exit status.
Reviewed-on: #151
## Problem
`arm64/alpine321` and `arm64/alpine322` advertise a broken stamp:
```
arm64/alpine321 :: 22930 Built: R 4.4.0; NA; 2026-07-31 13:35:52 UTC; unix
arm64/alpine322 :: 24696 Built: R 4.5.0; NA; 2026-07-31 13:51:54 UTC; unix
```
The per-minor sub-slots (`contrib/4.4`, `4.5`, `4.6`) are affected too.
All 18 other slots are correct.
uvr matches the stamp's platform triple plus R minor to pick binary over source, so nothing matches `NA` and both slots silently serve as source-only, which is exactly the regression bincraft#85 added the stamp to prevent.
`install.packages()` is unaffected, since it reads `Built:` from each tarball's own `DESCRIPTION`.
The tarballs are fine (`arm64/alpine322/.../dress.graph_0.8.3.tar.gz` carries `aarch64-unknown-linux-musl`), and so is the R that built them (`r-4.5.0_1_aarch64.apk` ships `R_PLATFORM='aarch64-unknown-linux-musl'`).
Only the index is wrong.
bincraft#96 stops a stamp like this being written again, but it cannot repair what is already there.
## Why not just re-run the index update
`upload_package_index()` reuses the slot's remote `PACKAGES.db`, and cranlike's `update_db()` only reparses files whose md5 changed, so entries already in the database keep the stamp they were written with.
Dropping `PACKAGES.db` to force a full reparse does work, and it is what bincraft#85's rollout note suggested, but for an S3 repo cranlike reads each package's metadata from the CRAN *source* mirror on GitHub.
A 25k-entry slot is then 25k requests to raw.githubusercontent.com, with a real risk of being rate-limited part-way through and leaving the slot half-written.
Only the `Built` column is wrong, so this corrects it in place instead: patch the column in `PACKAGES.db`, put the database back, and let `upload_package_index()` re-emit `PACKAGES*` from it.
`update_db()` always rewrites the index files even when nothing was reparsed, so no tarball is re-read and nothing is fetched from GitHub.
## Change
- `local/repair-built-stamp.R` — repairs the generic slot plus every per-minor sub-slot. Dry-run by default; `--apply` writes. The replacement comes from `bincraft::built_stamp()` under the R running the script, so it is exactly what a healthy run would have written, and bincraft#96's guard makes a broken build image fail rather than write a second bad stamp.
- `.crow/repair-built-stamp.yaml` — manual pipeline, routed by `target_arch` to the matching agent group and platform image, with `dry_run` defaulting to `true`.
- `.crow/archive-missed-packages.yaml` — drop the two `alpine321` matrix entries. Alpine 3.21 is EOL: the website advertises only 3.23/3.24 and `process-updates.yaml` already dropped it, so that slot is retired rather than repaired.
## Verification
`crow lint .crow/` passes on all nine pipelines.
`air format` and `jarl check` are clean; the script parses, and the `/opt/R` minor-version derivation was checked against `4.4.3 / 4.5.3 / 4.6.0 / current` → `4.4 4.5 4.6`.
The repair itself is unrun by design — it needs B2 credentials and an arm64 agent.
## Rollout
1. Cut a bincraft release so `local/install-bincraft.R` picks up #96 (it resolves the latest `vX.Y.Z` tag, and #96 is only on `main`).
2. Run this pipeline with `target_arch=arm64`, `OS=alpine`, `OS_VERSION=3.22`, `R_VERSION=4.5.3`, `dry_run=true` and check the reported counts.
3. Re-run with `dry_run=false`.
4. Confirm: `curl -sS https://cran.devxy.io/arm64/alpine322/latest/src/contrib/PACKAGES | grep '^Built:' | sort | uniq -c`
`arm64/alpine321` is deliberately left alone.
Reviewed-on: #150
`bincraft` dropped pak in favour of uvr (5.0.x, "Dependencies and their system requirements are now installed with `uvr` instead of pak during `build_binary_package()`"), so the pipelines, helper scripts and images in this repo move with it.
## Approach
uvr is project-scoped in a way pak is not: `uvr add` refuses to run outside a project and always installs into `.uvr/library/`, and only `uvr sync` honours `--library`. So there is no one-line `pak::pak(...)` equivalent. `local/uvr-install.sh` encapsulates the dance — bootstrap a pinned uvr, mint a throwaway project under `TMPDIR`, `uvr add --no-install`, then `uvr sync --library <target>`. Keeping the project outside the checkout also keeps `uvr init`'s `.Rprofile` from hijacking `.libPaths()` for every other R call in the pipeline.
This matches what bincraft itself does (`uvr sync --install-system-deps --library <lib>`), and bincraft requires `uvr` on `PATH`, which the bootstrap provides: every pipeline that calls `bincraft::` runs `install-bincraft.R` (and therefore the bootstrap) first.
## Changes
| File | Change |
| --- | --- |
| `local/uvr-install.sh` | **New.** The single replacement for `pak::pak(...)`. Bootstraps uvr `v0.4.4`, resolves the R interpreter from `UVR_R_BIN`/`R_VERSION`/`PATH`, pins the manifest to that R's exact version, and syncs into `UVR_TARGET_LIB`/`R_LIBS_USER`. |
| `local/install-bincraft.R` | Installs `forgejo::codefloe.com/rpkgs/bincraft@<tag>` instead of a `git::` URL; keeps the `git ls-remote` tag resolution. Exports `UVR_R_BIN`/`UVR_TARGET_LIB` from `R.home()`/`.libPaths()[1]` so the per-R-minor passes target their own R and library. |
| `.crow/auto-apply-patches.yaml`, `.crow/weekly-patch-proposals.yaml`, `.crow/weekly-audit-missing.yaml`, `.crow/weekly-rebuild-missing.yaml`, `.crow/build-all-versions-install-deps.yaml` | `pak::pak(...)` → `UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh ...`. The explicit `UVR_R_BIN` matters in `build-all-versions-install-deps.yaml`, which has no `R_VERSION` in its step environment. |
| `.crow/build-all-versions.yaml`, `.crow/process-updates.yaml`, `.crow/weekly-rebuild-missing.yaml` | `R_PKG_CACHE_DIR` → `UVR_CACHE_DIR` + `UVR_PACKAGES_DIR` on the same `/mnt/cache` volume, preserving the amd64-off/arm64-on split. Drops the `rm -rf .../pkgcache/_metadata/...` cleanup. |
| `local/r-minor-helpers.R`, `local/build-all.R`, `local/tests/test-trim-pkgcache.R` | Removes `trim_pkgcache_metadata()`, its every-25-packages call and its tests. uvr's cache does not mint a fresh ~70 MB snapshot per `PACKAGES` change. |
| `.crow/build-all-versions-install-deps.yaml` | Drops `pak::sysreqs_db_update()`; uvr resolves sysreqs from its vendored `r-system-requirements` rules via `--install-system-deps`. |
| `docker/Containerfile-shiny-app` | Bootstraps uvr and drives both dependency installs through one uvr project with `UVR_LIBRARY` pointed at the image's R library. |
| `docker/build-one.Dockerfile` | Ships `uvr-install.sh` at `/work/local/` so `install-bincraft.R` finds it. |
| `docker/reprex/alpine.sh` | Replaces `pak::local_install_deps()` with DESCRIPTION parsing + `uvr add`. |
| `local/test-package-loading.R` | Installs via the helper instead of `pak::pkg_install()`. |
| `README.md` | Documents uvr for sysreq inference, archived-version installs and cache clearing. |
| `renovate.json` | Tracks the `UVR_PIN` in `uvr-install.sh` via `github-releases`. |
## Behaviour notes
- **`weekly-audit-missing` still takes bincraft from the default branch**, not the latest release tag, matching what the `git::` pak call did. Called out in a comment rather than silently changed.
- **The uvr pin is repo-wide.** bincraft resolves `uvr` from `PATH` and pins no version of its own, so `UVR_PIN` in `uvr-install.sh` governs the whole pipeline.
- **Persistent caches now also benefit bincraft**, which reads `UVR_CACHE_DIR`/`UVR_PACKAGES_DIR` from the inherited pipeline environment.
- **`uvr sync` will not prune the shared library.** Pruning is disabled whenever `--library` is passed (`do_prune = prune && library_override.is_none()`), so `/mnt/cache/R-pkgs` keeps bincraft and its dependencies. The wipe-on-ABI-mismatch path is *not* similarly guarded, which is why the helper pins the manifest to the active R's exact version.
- **`plans/` and `specs/` are untouched** — they are dated records of decisions made in June 2026 and describe bincraft's then-pak-based internals; rewriting them would misstate history.
## Verification
`shellcheck`, all pre-commit hooks (on this commit's file range) and the `local/tests/` suite (100 assertions) pass.
Not yet exercised in CI: the build-env images do not ship `uvr`, so the per-step `curl install.sh` bootstrap is untested against a real image. Worth a manual `build-all-versions-install-deps` run before merging.
Reviewed-on: #147
## Why
`RcppParallel` binaries built on ubuntu 22.04 still record `NEEDED libtbb.so.12` and cannot `dyn.load` without the distro TBB, although #145 landed the link-order patch and build-env-images dropped `TBB_INC`/`TBB_LIB` the same morning.
The link-order fix does not survive a build image that exports those variables, and the published images still do: `.crow/images.yaml` in build-env-images triggers on `cron`/`manual` only, so every image in the registry is still the 2026-07-21 build.
With `TBB_LIB` set, `configure.R` never reaches the bundled-oneTBB branch at all, `install.libs.R` fills `RcppParallel/lib` with absolute symlinks into the image's library dir, and the plain `-L` from #145 just points at the system TBB instead of the bundled one.
Nothing on the pipeline side can undo this: R reads `~/.Renviron` **after** the process environment, so neither a Crow `environment:` block nor the registry `env` field wins.
```
# the image, not the Containerfile
docker run --rm reg.devxy.io/rpkgs/build-env-ubuntu:jammy sh -lc 'grep TBB ~/.Renviron'
TBB_INC=/usr/include/oneapi
TBB_LIB=/usr/lib/x86_64-linux-gnu
```
Rather than couple correctness to image freshness for a third round-trip, the patch now forces the bundled TBB itself.
## Changes
- `local/patches/RcppParallel/bundled-tbb-link-order.patch` -> `local/patches/RcppParallel/force-bundled-tbb.patch`.
Keeps the `-Wl,-L` -> `-L` link-order hunk (the bundled branch has the same problem when a distro libtbb is installed) and adds an opt-out, `BINCRAFT_ALLOW_SYSTEM_TBB=TRUE`, for the ambient `TBB_ROOT`/`TBB_LIB`/`TBB_INC`.
- All four read sites have to move together; patching only `configure.R` builds the bundled TBB and then **segfaults R on the post-install load test**:
- `tools/config/configure.R`: branch selection.
- `src/install.libs.R`: `R CMD INSTALL` runs it outside the `tbb` rule in `src/Makevars` that passes the configured values, so at install time it still sees the image environment.
- `R/tbb.R` `tbbLibraryPath()`: run time. `.onLoad()` would `dyn.load` the system TBB into the global scope next to the bundled one that `RcppParallel.so` needs - two copies of the same symbols, hence the segfault.
- `R/tbb.R` `tbbCxxFlags()` / `tbbLdFlags()`: otherwise dependents such as `rstan` get the system TBB handed back to them.
- `local/patches/registry.json`: new patch path and a `reason` describing both failure modes.
## Verification
Built through `bincraft:::prepare_patched_repo()` on the **current, unfixed** images (all three still export `TBB_INC`/`TBB_LIB`), with bincraft v5.0.1, i.e. what CI deploys:
| image | `NEEDED` | rpath | `RcppParallel/lib` | load, all system `libtbb*` moved away |
| --- | --- | --- | --- | --- |
| `build-env-ubuntu:jammy` | `libtbb.so` | `$ORIGIN/../lib` | real `libtbb.so.2`, 5.0 MB | OK, 12 threads |
| `build-env-redhat:9` | `libtbb.so` | `$ORIGIN/../lib` | real `libtbb.so.2`, 5.1 MB | OK, 12 threads |
| `build-env-alpine:3.24` | `libtbb.so` | `$ORIGIN/../lib` | real `libtbb.so.2`, 4.7 MB | OK, 12 threads |
For contrast, the same jammy build without this change:
```
NEEDED libtbb.so.12
RUNPATH /usr/lib/x86_64-linux-gnu:$ORIGIN/../lib
lib/ libtbb.so.12.5 -> /usr/lib/x86_64-linux-gnu/libtbb.so.12.5 (dangling off the image)
load libtbb.so.12: cannot open shared object file: No such file or directory
```
With `TBB_LIB` still exported, `RcppParallel::tbbLibraryPath()` resolves to the package's own `lib`, and `CxxFlags()` emits the package's own `include` plus `-DTBB_INTERFACE_NEW` - not `/usr/include/oneapi`.
## Behaviour change
Published `RcppParallel` binaries carry their own oneTBB on every platform, whichever image version CI pulls. The patch content changed, so the patched-binary cache key changes with it and no stale entry is reused.
The build-env-images fix is still worth rolling out (those images also carry the broken el8 `uvr` and uvr 0.4.1), but RcppParallel no longer waits on it.
## Follow-up, not in this PR
The binaries already in B2 are the broken ones; they need a rebuild, and a Bunny `/purge` does not evict Perma-Cache.
Reviewed-on: #146
## Why
Every build logs:
```
! Patch for RcppParallel 6.2.0 did not apply cleanly; skipping patched build.
```
RcppParallel 6.x rewrote its build system. `src/Makevars.in` no longer contains the `USE_TBB=Linux` block that `system-tbb.patch` edited (it is now a short `@VAR@` template driven by `tools/config/configure.R`), so the patch can never apply again.
The workaround it implemented is obsolete too. 6.x bundles **oneTBB 2022** and builds it with cmake, which works on musl and with g++ 8-15, so the 5.x reason for linking a system TBB is gone. Linking one is now harmful: `install.libs.R` symlinks the system libraries into `RcppParallel/lib`, so the published binary depends on a TBB the consumer does not have — the same failure mode as `fs` and libuv.
What is still broken upstream is the **link order**. `configure.R` names the TBB directory with `-Wl,-L`, and gcc expands its own search dirs into `-L` options ahead of anything forwarded verbatim with `-Wl,`:
```
$ gcc -v -o t t.c -Wl,-L/usr/local/lib64 -ltbb
-L/usr/lib/gcc/x86_64-redhat-linux/8 -L/lib/../lib64 -L/usr/lib/../lib64 ... -L/usr/local/lib64
```
So on any build host with a distro TBB installed, `-ltbb` resolves to that library and `RcppParallel.so` records *its* SONAME (`libtbb.so.12`, or `libtbb.so.2` for the classic Intel TBB on el8/el9) instead of the bundled `libtbb.so`.
## Changes
- Replace `RcppParallel/system-tbb.patch` with `RcppParallel/bundled-tbb-link-order.patch`, which changes the three `-Wl,-L` occurrences in `tools/config/configure.R` to plain `-L`.
- Scope the RcppParallel entry to `>=6.0.0` and widen `platforms` to `*`.
- Drop the `rstan` entry: its `-DTBB_INTERFACE_NEW` is already emitted by `RcppParallel::CxxFlags()` once the bundled oneTBB is used, and its `-I/usr/local/include` was an el8/el9 path applied on every platform.
- `.pre-commit-config.yaml`: exclude `local/patches/*.patch` from `trailing-whitespace`, `end-of-file-fixer` and `editorconfig-checker`. They rewrite blank context lines (` ` -> ``) in every diff in the registry; `git apply` happens to tolerate it today, but a patch with meaningful trailing whitespace would be silently corrupted. The excludes are per-hook so `validate patch registry` still runs.
- README: the RcppParallel example described the 5.x problem.
## Verification
Built in the published images:
| image | NEEDED | RPATH | loads with system libtbb removed |
| --- | --- | --- | --- |
| `build-env-alpine:3.24` | `libtbb.so` | `$ORIGIN/../lib` | yes |
| `build-env-redhat:8` | `libtbb.so`, `libtbbmalloc.so` | `$ORIGIN/../lib` | yes |
| `build-env-ubuntu:noble` | `libtbb.so` | `$ORIGIN/../lib` | yes |
Without the patch the same builds record `libtbb.so.12` (alpine, ubuntu) or `libtbb.so.2` (el8, the classic 2018 TBB) and fall back to the system library at load time.
rstan 2.32.7 compiles and loads against the patched RcppParallel on ubuntu noble with no makevars override and with the system libtbb moved away. On el8 it also compiles; loading it there is blocked by an unrelated image bug (see below).
## Behaviour change
Requires the paired build-env-images PR (drops `TBB_INC`/`TBB_LIB`) and an image rebuild — with those env vars set, configure still takes the system-TBB branch. After that, RcppParallel binaries ship their own oneTBB and are self-contained.
## Also found, not fixed here
- The `uvr lock failed ... GLIBC_2.29 not found` errors in the same log are the `-gnu` uvr artifact on el8's glibc 2.28; fixed in build-env-images.
- `build-env-redhat:8` R 4.5.3/4.6.0 cannot load `stats` (`libRlapack.so: undefined symbol: dgemmtr_`): the el8 R RPM symlinks `libRblas.so` to openblas 0.3.15, which predates that symbol. R 4.4.3 and el9 are fine. Belongs in the R RPM build.
Reviewed-on: #145
Extends #137. el8/el9 were excluded because they ship only classic Intel TBB. build-env-images now vendors oneTBB 2021 into `/usr/local` on el8/el9, so every platform has oneTBB 2021+. Widen the scope `redhat-10` → `redhat` (family token covers el8/el9/el10) and fix the reason text (no longer 'left on bundled'). **Requires the el8/el9 images rebuilt with the vendored oneTBB (build-env-images PR).** Verified: patched RcppParallel builds + links the vendored oneTBB on both el8 (g++ 8.5) and el9 (g++ 11.5).
Reviewed-on: #139
## Problem
A fresh `weekly-audit-missing` run followed by `weekly-rebuild-missing` reported that almost every package "already exists in the remote bucket", even though the audit had just flagged them as missing.
Root cause: the audit's Forgejo-issue update matched the existing `### <arch>` subsection by its **bare** header (`### arm64`) while writing headers with a ` (N missing, M to rebuild)` suffix.
The equality test never matched, so every run **appended** a new block instead of replacing the old one.
Issue #63 had accumulated 38 arch subsections under `## alpine-323` where 2 are expected (75 total across the alpine platforms; body ~77k lines).
`fetch-rebuild-packages-from-issue.R` reads the **first** matching block, which was the oldest snapshot.
So the rebuild kept re-checking a months-old list (180 packages, mostly already built), while the genuinely-missing packages, ~1921 for alpine-323/arm64 in the freshest block, were never fed to the rebuild and the backlog grew silently.
## Changes
- **`local/weekly-missing-binaries-audit.R`**: match arch subsections by prefix (`^### <arch>( |$)`) and remove **all** blocks for that arch before writing one fresh block. Accumulation now self-heals on every run.
- **`local/dedupe-audit-issue.R`** (new): one-off cleanup that collapses each `## platform` section to the freshest block per arch across the three OS-family issues. Supports `DRY_RUN=1`.
- **`local/fetch-rebuild-packages-from-issue.R`**: prefer the audit's freshly-written RDS (overwritten each run, immune to issue-body drift), falling back to issue parsing when absent.
## Validation
Simulated the dedupe logic against the live #63 body: **75 → 8** arch subsections, body 77k → 28k lines, and the kept alpine-323/arm64 block correctly resolves to `GARCH.X (3.0)` (the stale first block held `2.0`).
## Follow-ups (not in this PR)
- `alpine-321` is audited but has no row in the rebuild matrix (7,230 missing, never rebuilt).
- An `alpine-324` section exists in the issue but is in neither matrix.
Reviewed-on: #138
## What
Replace RcppParallel's `disable-tbb` registry patch with `system-tbb`: build RcppParallel against the system oneTBB instead of stripping TBB entirely.
## Why
`disable-tbb` skipped the bundled Intel TBB build by forcing the TinyThread backend, which also removed RcppParallel's TBB linkage.
That broke every dependent that links TBB through `RcppParallelLibs()` -- `rstan` and the whole Stan cluster -- with `symbol not found: tbb::detail::r1::observe`.
This is the root cause behind the large "blocked on RcppParallel" / rstan clusters (issues #115, #120): the packages were not individually broken, they were all waiting on one TBB-linkage regression.
## Change
`system-tbb.patch` leaves `USE_TBB` unset (so the bundled build is still skipped on musl and g++ 15) but keeps the TBB backend and links the system oneTBB now shipped in the build-env images:
```
PKG_CXXFLAGS += -DRCPP_PARALLEL_USE_TBB=1 -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -DTBB_INTERFACE_NEW
PKG_LIBS += -ltbb -ltbbmalloc
```
Scoped to `alpine` + `ubuntu-2604`, the only platforms where the bundled build fails; redhat and older ubuntus keep the bundled TBB.
## Requires
The companion image change that exports `TBB_INC`/`TBB_LIB` so `RcppParallelLibs()` hands the system-TBB flags to dependents: build-env-images PR #12. Both must ship together.
## Verified
In `build-env-alpine:3.24` with the new images:
- RcppParallel builds against oneTBB 2022, no ABI errors, patch applies cleanly to the target clone.
- `RcppParallelLibs()` returns `-L/usr/lib -Wl,-rpath,/usr/lib -ltbb -ltbbmalloc`.
- rstan links (`-ltbb -ltbbmalloc`) and loads with no missing symbol -- `* DONE (rstan)`, trial build exit 0.
Reviewed-on: #135
`local/trial-build-patch.R` -- the single-package acceptance gate the proposer prints for humans to run (`PGPASS=... Rscript local/trial-build-patch.R <pkg>`) -- reported **`Trial build OK`** for rstan even though rstan.so failed to load (`symbol not found: tbb::detail::r1::observe`).
Same false-green as the registry gate before #130: `build_binary_package()` catches failures internally and returns `"error"` rather than throwing, so a `tryCatch` that only treats a thrown exception as failure passes a broken build. Now inspects the return value (mirrors trial-build-registry.R). Found while verifying the rstan/oneTBB unblock.
Reviewed-on: #134
Follow-up to #132. The no-op path already worked (the `else` branch runs and `exit 0`s), but using `git fetch` as the *existence check* printed `fatal: couldn't find remote ref auto/registry-patch-proposals` to stderr — which looks like a failure even though the step succeeds.
Probe with `git ls-remote --exit-code --heads origin ${patch_branch}` instead: verified against the live repo it exits 0 for an existing branch and 2 for a missing one, **silently** (no `fatal`). Only fetch + checkout when the branch actually exists.
Net: same behavior, clean log — a run with no pending auto-patch branch prints just `No ... branch; no pending auto-patch proposals to verify.` and exits 0.
Reviewed-on: #133
## Problem
The trial pipeline errored:
```
git clone -q --branch auto/registry-patch-proposals ...
fatal: Remote branch auto/registry-patch-proposals not found in upstream origin
```
I deleted the stale `auto/registry-patch-proposals` branch during cleanup, and — now that cascade detection (#128) makes the proposer correctly find **no candidates** — the proposer exits before ever recreating it. So the branch is legitimately absent, and the gate's hard `git clone --branch` fails.
## Fix
"Branch absent" = "no pending auto-patch proposals" = **nothing to verify**, which should be a clean no-op, not a failure. Clone `main`, then fetch + checkout the patch branch only if it exists; otherwise log and `exit 0`. A present branch is verified exactly as before (checkout its tip, diff registry vs `main`).
This also means the gate now runs `main`'s `trial-build-registry.R` (with the #130 fix) rather than a possibly-stale copy on the branch.
Reviewed-on: #132
## Why
With cascade detection (#128) live, the latest `auto-apply-patches` run did exactly the right thing — **proposed nothing** (`No auto-proposable candidates`) because every failure is a dependency cascade, and it surfaced the ~30 root-cause dependencies to fix.
But the "Blocked on a dependency" list printed **one line per fingerprint group**, so the same dependency repeated (rstan ×4, lpsymphony ×4, salso ×2, BH ×2, GO.db ×2, RcppCWB ×2, …), burying the priority.
## What
Aggregate blocked packages across all groups **by the dependency they wait on**:
- Expose `blocked_map` (package → dependency) from `build_triage_report()`.
- Add `blocked_by_dependency()` — dedupes dependents (a package in two groups counts once) and ranks dependencies by how many distinct dependents they block.
- Proposer and tracker (log + issue) now print one line per dependency, sorted by impact. Replaces the per-group `blocked_summary`.
## Result (same data, aggregated)
```
Blocked on a dependency (3 dependencies block 6 dependents; fix the dependency, not each dependent):
RcppParallel 3 dependent(s)
rstan 2 dependent(s)
sf 1 dependent(s)
```
So the real run becomes a crisp, ranked worklist: RcppParallel (894), sf (128), rstan (~96), Rfast (33), clarabel/DescTools (26), Rglpk (22), xgboost (18), …
## Verified
New test covers cross-group aggregation, dedup (a dependent in two groups counted once), the example cap, and ranking. 112 tests pass; hooks pass.
Reviewed-on: #131
## Critical: the gate was false-green
The latest run printed **`3/3 passed on alpine-324`** while all three builds actually **failed** (their `rstan` dependency won't compile). A false-green gate would let broken registry entries merge — worse than no gate.
Root cause: `bincraft::build_binary_package()` catches build failures internally and **returns `"error"`** for the failed tag rather than throwing. The gate's `tryCatch` only treated a *thrown* exception as failure, so every non-throwing failure looked like a pass.
Fix: inspect the return value. A tag passes only if the flattened result is non-empty and contains no `"error"` sentinel; a thrown error still counts as failure. Verified the verdict against `error`/`skipped`/`TRUE`/`list(success=TRUE)`/`NULL`/mixed inputs.
With this, the current rstan-blocked entries will correctly show **0/3 (red)** — which is the right answer until rstan builds.
Reviewed-on: #130
## Why (from the #127 trial-build gate)
The gate did its job: 0/3 passed, merge blocked. The log showed *why* -- BFpack, BayesERtools, GMLTM all fail while building their shared dependency **`rstan`**, not in their own code:
```
Failed to build source package rstan.
.../StanHeaders/include/stan/math/prim/core/init_threadpool_tbb.hpp:9:10:
fatal error: tbb/tbb_stddef.h: No such file or directory
```
So the per-package `-DTBB_INTERFACE_NEW` makevars entries the classifier proposed are useless for these packages -- they're blocked on `rstan` (which already has a registry entry). This is the **same dependency cascade** the RcppParallel `applies_to` guard catches, but `tbb-stddef-removed` is a generic signature with no such pin, so ~73 Stan packages kept getting proposed.
## What
Generalise cascade detection beyond the RcppParallel special case:
- `failing_dependency(error_text, package)` -- when the log names a **different** package as the one that failed to compile (`Failed to build source package X`, `compilation failed for package 'X'`, `dependency 'X' ... not available`), that package is the real cause.
- `build_triage_report()` now blocks any package whose **every** failing build is such a cascade: reported as `blocked_on` that dependency, never proposed a bogus per-package entry. A package that fails in its **own** compilation is still proposed.
- The `applies_to` (RcppParallel) and data-driven (rstan) cases are unified into one `blocked_packages` / `blocked_on` model; the report, proposer, and `blocked_summary` count the actually-blocked packages, and the blocked note shows even when a group also has genuine proposals.
## Effect
Next auto-apply run will stop proposing the rstan-blocked Stan packages (and any future dependency cascade) and surface them as "blocked on rstan" instead. Fixing `rstan` once clears the whole cluster.
## Verification
- New tests: `failing_dependency` (cascade vs own-compile vs none), and an end-to-end split where BFpack/GMLTM (blocked on rstan) are not proposed while an own-compile package still is.
- Full suite: 112 tests pass; all pre-commit hooks pass.
Refs #120, #127. (Separate follow-ups: fixing rstan's build itself, and quieting the gate's metadata-DB retry storm -- both root-caused to bincraft.)
Reviewed-on: #128
Companion to bincraft codefloe #65. Sets `BINCRAFT_VERBOSE_PATCH_BUILD=TRUE` in the trial-build gate so a failed isolated patched build prints the real compiler error instead of `System command 'R' failed`. Harmless on bincraft versions without the flag (unknown env var is ignored). Takes effect once bincraft #65 is released and the build-env images pick it up.
Reviewed-on: #129
## Problem
The first real `auto-apply-patches` run classified, limited to the top-10, and validated the candidate registry cleanly, then died at the push step:
```
Validating candidate registry:
Patch registry OK (13 entries).
sh: syntax error: unexpected "("
Error in git("commit", "-m", ...): git commit -m feat(patches): auto-propose 10 registry entries ... failed
```
Root cause: R's `system2()` with captured output (`stdout=TRUE`) runs the command through `/bin/sh`, and the arguments were passed **unquoted**. The commit message `feat(patches): ...` contains `()`, which the shell tried to interpret. The same class of bug affects the `^{commit}` and `ref:path` git refs in the trial-build gate.
## Fix
`shQuote()` every git argument:
- `propose-patches.R` -- the `git()` helper used by `--open-pr` (commit, push, checkout).
- `trial-build-registry.R` -- the base-ref reads (`rev-parse ... ^{commit}`, `ls-tree`, `show ref:path`).
## Verification
- Reproduced against a real git repo: the unquoted call fails with status 2 (the same `unexpected "("`); the `shQuote`d call commits successfully, and `rev-parse HEAD^{commit}` resolves.
- 105 tests pass; all pre-commit hooks pass.
Everything else in that run was correct: 842,659 failing builds classified, RcppParallel's 895 dependents correctly reported as blocked (not proposed), top-10 tbb-stddef candidates selected, 69 deferred, registry validated. Only the shell quoting was broken.
Reviewed-on: #126
Follow-up to #124 (merged): the auto-patch pipeline required a separate write-scoped `REPO_RW_TOKEN` to push the branch. Reuse the existing `FORGEJO_TOKEN` instead.
- `propose-patches.R --open-pr` now pushes `auto/registry-patch-proposals` over HTTPS with `FORGEJO_TOKEN` (the same token used for the PR API); the read-only `origin` clone URL can't push, so it builds an authenticated URL explicitly.
- Drop `REPO_RW_TOKEN` from `.crow/auto-apply-patches.yaml` and the docs.
No new secret needed: the pipeline's secrets are now `PGPASS`, `REPO_RO_TOKEN`, and `FORGEJO_TOKEN` (all existing). `FORGEJO_TOKEN` must have repository write scope for the push to succeed.
105 tests pass; all pre-commit hooks pass.
Reviewed-on: #125
Closes the classifier loop (issue #115, step 3): turns the auto-proposable candidates into an actual PR, gated by a real trial build **in our own build-env images**.
Chosen model (from the design discussion): **autonomous PR**, **PR-first with a CI trial-build gate**, **bounded top-N batch** per run.
## Creating the patch PR
- `propose-patches.R` gains:
- `--limit N` -- act on the top-N candidates by failure volume; the rest defer to the next run (logged).
- `--open-pr` -- write the entries onto the reused `auto/registry-patch-proposals` branch, push (with `REPO_RW_TOKEN`), and open/update **one** PR via the Forgejo API (so re-runs update the same PR instead of piling up).
- `.crow/auto-apply-patches.yaml` -- a single job that runs `--open-pr --limit` on a cron/manual trigger. Needs `FORGEJO_TOKEN` + a write-scoped `REPO_RW_TOKEN`.
Novel source diffs and unknown signatures are still never proposed; nothing merges.
## The merge gate (our build-env images)
- `.crow/trial-build-registry.yaml` -- matrixed over the real `OS/IMG` build-env matrix (alpine:3.24, redhat:8/9/10, ubuntu:jammy/noble/**resolute** for ubuntu-2604). Each platform runs `local/trial-build-registry.R`, which diffs the branch registry against `main` and trial-builds **only the entries the branch adds** that apply to that platform, inside `reg.devxy.io/rpkgs/build-env-*`. Green only if every new entry builds; a platform with no new entries is a fast no-op. Nothing is uploaded/archived/recorded.
- The base-registry read **fails loud** if it can't read `registry.json` at `main`, rather than silently treating the base as empty and trial-building the whole registry.
## Notes / follow-up
- The repo uses **no `pull_request` triggers**, so the gate runs manually or on a cron against the branch (`--var patch_branch=...`). Wiring it to fire automatically on the PR needs `event: pull_request` enabled on the Forgejo webhook -- a one-line addition once that's on.
- Two new crons to register in the crow UI: `auto-apply-patches` and `trial-build-registry`. New secret needed: `REPO_RW_TOKEN` (write scope) for the push.
## Verification
- New pure helpers `entry_applies_to_os()` / `new_registry_packages()` covered by tests (platform codename/family/wildcard matching; added-vs-unchanged entry detection; per-platform filtering).
- `--limit` smoke (stubbed DB): top-2 by volume proposed, 3 deferred, candidate registry validates.
- Full suite: 105 tests pass; all pre-commit hooks pass (air, prettier, markdownlint, yamllint, validate-patches).
Reviewed-on: #124
## Why
Issue #120 (the auto-proposed-patches issue) only lists **auto-proposable** fixes -- currently just the TBB signatures. So a reasonable read of it was "TBB is our only build failure", when in fact three whole categories are simply not shown there:
- **Unclassified failures** -- anything that doesn't match a seeded signature is routed to human triage and never appears (we've only seeded TBB and libuv signatures).
- **Dependency-blocked failures** -- the ~800 RcppParallel dependents (post #121) are still failing; they only show as a log line.
- Human-only signatures (libuv).
These blind spots are exactly where the *next* signatures should come from, so they deserve the same visibility as the proposals.
## What
Extend the feedback-loop tracker to surface the classifier's blind spots:
- **`unclassified_summary()`** -- groups every unknown-signature failure by normalised fingerprint, ranked by build count, capped with an explicit `dropped_groups` count (no silent truncation), each with example packages + platforms. These are the candidates for new `build_signatures()` rules.
- **`blocked_summary()`** -- lists each dependency (e.g. RcppParallel) and how many dependents wait on it.
- `proposal-tracking.R` prints both sections, and a new **`--open-issue`** mode posts/updates a *"Unclassified build failures (needs signatures) (#115)"* Forgejo issue.
- The weekly crow pipeline now runs the tracker with `--open-issue`, so it maintains a second tracking issue alongside the proposals one. Read-only on the DB; the only writes are the two issues.
## Verification
- New tests cover `unclassified_summary` (ranking + both caps) and `blocked_summary`.
- Tracker smoke with a stubbed DB (proposable + blocked + unclassified mix) prints the hit rate, `Blocked on a dependency: RcppParallel: 2 dependent(s)`, and `Unclassified failures ... [2 builds | 2 pkgs] ld: undefined reference ...`.
- Full suite: 95 tests pass; all pre-commit hooks pass (air, prettier, markdownlint, yamllint, validate-patches).
Reviewed-on: #122
## Problem
The scheduled `--open-issue` run (issue #120) posted **800+** proposed entries, all `rcppparallel-bundled-tbb` on `ubuntu-2604`, each pre-filling `RcppParallel/disable-tbb.patch` for an unrelated package (`ACEsimFit`, `AovBay`, `AdaptGauss`, ...).
Root cause: those packages fail on ubuntu-2604 only because their **RcppParallel dependency** does not build there. bincraft records RcppParallel's TBB error text against each dependent, so every one of them matches the `rcppparallel-bundled-tbb` signature. Proposing a RcppParallel source patch for `ACEsimFit` is not just noise -- it is **broken**: the diff targets RcppParallel's `Makevars.in`, so it cannot apply to a dependent's source.
## Fix
Add an optional `applies_to` field that pins a package-specific fix to the package it targets:
- A signature whose fix is a curated per-package source patch (`rcppparallel-bundled-tbb` -> `RcppParallel`) is only ever proposed **for that package**.
- Other packages matching the signature are downstream failures, reported as **"blocked on `<dependency>`"** and never proposed a bogus entry. Fix the dependency once and the whole cascade clears.
Generic levers (the `tbb-stddef-removed` makevars fix) are unaffected -- they carry no `applies_to`, so they still apply to any matching package.
## Changes
- gate proposal generation in `build_triage_report()` on `applies_to`; add a `blocked_on` field to group records
- surface blocked groups in both `failing-builds-report.R` and `propose-patches.R` instead of the misleading "already registered" note
- document `applies_to` in the signature-table header
## Verification
- New tests: RcppParallel dependents are blocked (no `proposed_entries`, `blocked_on = "RcppParallel"`), while RcppParallel *itself* is still proposed when it is the failing package.
- Reproduced the avalanche end-to-end with a stubbed DB: 5 dependents now report "Blocked on RcppParallel: 5 package(s) fail because RcppParallel does not build" and **zero** candidates are emitted; a genuine `tbb-stddef` proposal in the same run is unaffected.
- Full suite: 84 tests pass; all pre-commit hooks pass.
Refs #120.
Note: the underlying build problem (RcppParallel failing on ubuntu-2604 despite its registry entry) is real and separate -- this PR stops the classifier from spamming broken per-dependent proposals about it.
Reviewed-on: #121
## Problem
A live `propose-patches.R --open-issue` run aborted with:
```
Patch registry validation FAILED:
- ambiguous duplicate entries: hmmTMB|ubuntu-2604|*, imt|ubuntu-2604|*, refundBayes|ubuntu-2604|*
Error: Candidate registry failed validation; not writing or proposing.
```
Root cause: a package whose failing builds match **more than one** auto-proposable signature (e.g. some logs hit `tbb-stddef-removed`, others `rcppparallel-bundled-tbb`) landed in two separate signature groups, so the proposer emitted two candidate entries with the same `package|platforms|versions` key. The validator's ambiguous-duplicate check then failed the whole candidate set, aborting the run — so even the unambiguous candidates (e.g. `vacalibration`) never got proposed.
## Fix
Add a pure, tested `dedupe_candidates()` that:
- keeps a package's candidate only when it maps to a **single** signature (collapsing an accidental repeat under the same signature), and
- routes a package that maps to **multiple** signatures (conflicting fix tiers -- makevars vs source patch) to **human triage** instead of guessing between them.
The proposer prints the skipped ambiguous packages and proceeds with the clean candidates, so one ambiguous package no longer blocks the rest. This matches the issue #115 guardrail: don't auto-pick when the fix is ambiguous.
## Verification
- New unit tests in `test-proposal-tracking-lib.R` cover the single-signature, multi-signature (conflict), duplicate-under-one-signature, and empty cases.
- Reproduced the original failure end-to-end with a stubbed DB (`hmmTMB` ambiguous + `vacalibration` clean): `hmmTMB` is now listed under "Ambiguous -> human triage", `vacalibration` is proposed, and the candidate registry validates (`Patch registry OK`).
- Full suite: 78 tests pass; all pre-commit hooks pass.
Reviewed-on: #119
Adds the scheduled pipeline for the failure-triage proposer/tracker landed in #117 (issue #115, steps 3 + 4).
## `.crow/weekly-patch-proposals.yaml`
A single (non-matrix) job -- the classifier groups over all of `single_builds`, so unlike the per-platform `weekly-audit-missing` it runs once globally. Weekly it:
1. `propose-patches.R --open-issue` -- posts/updates a Forgejo tracking issue with the auto-proposable registry entries (human reviews, trial-builds, opens the PR; nothing merges).
2. `proposal-tracking.R` -- logs the feedback-loop metrics (signature hit rate, proposed-vs-merged, retirement candidates).
Low blast radius: clones read-only (`REPO_RO_TOKEN`); the only write is the tracking issue via `FORGEJO_TOKEN`. It builds nothing (no B2 keys, no bincraft).
Triggers: `cron: weekly-patch-proposals` (register the cron in the crow UI) or manual with `task=weekly-patch-proposals`, matching the `archive-missed-packages` convention. Documented under "Scheduled run" in `local/patches/README.md`.
`yamllint` and the other pre-commit hooks pass.
Reviewed-on: #118
Implements steps 3 + 4 of #115, building on the classifier merged in #116. Now that bincraft **v4.4.3** applies registry `patch`/`makevars`/`configure_args` to the *target* package build (previously deps-only), a trial patched build is a meaningful acceptance gate, so the "propose" half is viable.
## Step 3 — propose, do not apply
- **`local/propose-patches.R`** — for each classified, safe fix affecting a package with no current registry entry, emits a pre-filled `registry.json` entry and validates the candidate set against a *temporary* merged registry (the real one is never touched unless asked).
- default: print candidates + validation, **take no action**
- `--write`: append entries to `registry.json` + the proposals ledger (you commit + open the PR)
- `--open-issue`: post/update a Forgejo tracking issue (reuses the weekly-audit `httr2` + `FORGEJO_TOKEN` pattern)
- **`local/trial-build-patch.R`** — isolated bincraft build of one package with the registry applied (no upload/archive/metadata; `patchhash` keeps it out of the real cache). Exit 0/1, so it gates a CI step or manual pre-merge check.
The human gate stays: nothing merges. Acceptance = `validate-patches.R` passes (checked automatically) **and** the trial build succeeds. Novel source diffs and unknown signatures are never proposed (they carry `auto = FALSE`).
## Step 4 — feedback loop
- **`local/proposal-tracking.R`** (read-only) — signature hit rate (builds/pkgs/addressed/open per signature), proposed-vs-merged (a proposal counts merged once its package is in the registry), and retirement candidates (registry entries whose package no longer fails, i.e. likely fixed upstream).
- **`local/proposal-tracking-lib.R`** — the pure metric/ledger helpers.
## Supporting changes
- Refactored the classify helpers to expose a pure `build_triage_report()` + a list-returning entry builder; `failing-builds-report.R` now renders from the shared function (no behaviour change).
- `validate-patches.R` gains optional `PATCH_DIR`/`REGISTRY_FILE` overrides (backward-compatible) so a candidate registry can be validated in isolation.
- Documented the propose/trial-build/tracking workflow in `local/patches/README.md`.
## Verification
- 71 unit tests pass (incl. new `test-proposal-tracking-lib.R`) under the Dockerized R 4.5.3 build env.
- All pre-commit hooks pass (`air-format`, `validate-patches`, prettier, etc.).
- Smoke-tested all three entrypoints end-to-end with a stubbed DB: dry-run, `--write` (produces a registry that passes the canonical validator + a valid ledger, then reverted), and the tracker.
Closes#115
Reviewed-on: #117
Implements steps 1 + 2 of #115: turn recorded build failures into triaged patch suggestions instead of hand-scraping Crow logs.
## What this adds
A **read-only** reporting pipeline over the `single_builds` metadata table. It never writes to the DB or the registry.
- `local/failing-builds-classify.R` — pure, DB-free helpers:
- `normalise_error()` strips temp paths, version numbers, hex addresses, and the package name so the same root cause collapses to one fingerprint.
- `fingerprint_error()` extracts the salient error line and normalises it.
- `classify_error()` matches against a seed signature set; unmatched errors are never guessed at.
- `propose_registry_entry()` renders a schema-valid `registry.json` entry.
- `local/failing-builds-report.R` — entrypoint: queries `single_builds WHERE error_occurred = TRUE AND removed = FALSE`, groups by root cause (signature when classified, fingerprint otherwise), classifies each group, and prints a triaged report. Flags: `--platform`, `--arch`, `--min`, `--json`; `PLATFORM`/`ARCH` env fallbacks.
- `local/tests/test-failing-builds-classify.R` — unit tests for the helpers.
- `local/patches/README.md` — documents the workflow.
## Seed signatures
Each rule carries a fix tier, confidence, and an auto/human-only flag:
| Signature | Fix | Disposition |
| --- | --- | --- |
| `tbb/tbb_stddef.h: No such file` | makevars `-DTBB_INTERFACE_NEW` | auto-proposable |
| RcppParallel bundled TBB (musl / new g++) | curated `disable-tbb.patch` | auto-proposable |
| system `libuv.so` link leak | force vendored/static lib | **human triage** (novel source diff) |
| unmatched | none | **human triage** |
## Guardrails honored
- No autonomous novel source diffs: only known env/makevars levers and already-curated package patches are auto-proposable; anything needing a brand-new diff, and any unknown signature, is routed to human triage.
- No DB or registry writes; no change to the public `src/contrib` index.
- Reuses `single_builds.error_text`; no new failure-capture pipeline.
## Verification
- All helper unit tests pass under the Dockerized R 4.5.3 build env.
- Pre-commit hooks pass (`air-format`, `validate-patches`, prettier, etc.).
- Smoke-tested the full report path with a stubbed DB; generated proposals pass the real `local/validate-patches.R`.
Steps 3 (auto-open PRs) and 4 (feedback loop) are intentionally deferred, per the issue's suggestion to validate the signature set first.
Closes#115
Reviewed-on: #116
## Motivation
`rstan` fails to compile when it is pulled in as a transitive dependency (e.g. of `ctsem`), aborting the dependency install:
```
/mnt/cache/R-pkgs/StanHeaders/include/stan/math/prim/core/init_threadpool_tbb.hpp:9:10:
fatal error: tbb/tbb_stddef.h: No such file or directory
9 | #include <tbb/tbb_stddef.h>
```
StanHeaders' `init_threadpool_tbb.hpp` unconditionally includes the legacy `<tbb/tbb_stddef.h>` to detect the TBB version, but that header was removed in oneTBB 2021+, which is the TBB that StanHeaders now bundles.
## Fix
Add an `rstan` entry to `local/patches/registry.json` that pre-defines `TBB_INTERFACE_NEW` via `CPPFLAGS`.
The header's own guard skips the removed include when that macro is already defined and selects the modern `tbb/global_control.h` + `tbb/task_arena.h` path that the bundled oneTBB actually provides:
```cpp
#ifndef TBB_INTERFACE_NEW
#include <tbb/tbb_stddef.h> // removed in oneTBB 2021+
#if TBB_VERSION_MAJOR >= 2020
#define TBB_INTERFACE_NEW
#endif
#endif
#ifdef TBB_INTERFACE_NEW
#include <tbb/global_control.h> // taken when we pre-define the macro
#include <tbb/task_arena.h>
```
This is a makevars-tier override (no source diff), applied through bincraft's `R_MAKEVARS_USER`, so `CPPFLAGS` stays additive and does not clobber rstan's own `PKG_CPPFLAGS`.
`-I/usr/local/include` is kept because the override replaces the default `CPPFLAGS`.
## Files changed
- `local/patches/registry.json`: add the `rstan` entry (`versions: "*"`, `platforms: ["*"]`, `makevars.CPPFLAGS = "-DTBB_INTERFACE_NEW -I/usr/local/include"`).
## Behaviour change
When bincraft resolves `rstan` (direct or transitive) it builds a patched binary with `TBB_INTERFACE_NEW` defined, so `rstan` (and dependents such as `ctsem`) compile against the bundled oneTBB.
The scope is `["*"]` because the failure is bound to StanHeaders' bundled TBB version, not the OS or toolchain.
Reviewed-on: #114
## Problem
The arm64 `build-all` pipeline fills the macmini (gaia) host disk despite an 8h prune.
Root cause is not images or job volumes: it is the persistent dep-cache volume, specifically `pkgcache/R/pkgcache/_metadata`, which grew to ~165 GB.
`{pkgcache}` mints a new content hash for the "patched" binaries repo on every PACKAGES change, so each per-package build writes a fresh ~70 MB `pkgs-<hash>.rds` (+ `patched-<hash>/`) that is never evicted (2407 snapshots observed).
When the disk hits 100% OrbStack stops and the on-host prune can no longer connect to the daemon, so it never self-heals.
## Change (Workstream A of the disk-fill fix)
- Add `trim_pkgcache_metadata()` to `local/r-minor-helpers.R`: keeps the newest `keep` (default 20) `patched-*`/`pkgs-*.rds` entries under `_metadata`, deleting only entries older than `min_age_secs` (default 600s) so it never races the up-to-4 concurrent split-jobs sharing the volume.
Preserves `pkg/` downloads and the stable CRAN/BioC/INLA repo dirs.
No-op when `R_PKG_CACHE_DIR` is empty (amd64) or `_metadata` is absent (first run).
- Call it every 25 packages inside the build loop in `local/build-all.R`.
- Add a defensive start-of-run cleanup of `_metadata/patched-*` + `pkgs-*.rds` to the two workflows that mount the persistent volume (`build-all-versions.yaml`, `build-all-versions-install-deps.yaml`).
Only these paths are touched; `process-updates.yaml`/`weekly-rebuild-missing.yaml` (no persistent volume) are unchanged.
Follow-ups (separate workstreams): on-host self-healing prune watcher + OrbStack disk cap (ansible), and Prometheus/Grafana alerting (k8s-talos).
Upstream: bincraft patched-repo hash churn is the true source fix.
New unit tests (6) for the helper; full suite 24/24 green.
Reviewed-on: #110
## Motivation
Build jobs were cycling through hundreds of packages that were only ever printed as `Skipping … due to previous build error recorded in metadata DB`, wasting wall-clock on per-package preparation before dropping each one.
## Cause
The prefilter query in `local/build-all.R` selected only successfully-built versions (`error_occurred = FALSE`) into `built`, so line 112 removed only those from the chunk.
Every previously-errored version stayed in the work list and was walked one-by-one, each hitting the internal skip in `build_binary_package()`.
This also explains the misleading `Skipped 0 already-built package versions` line for alphabetical chunks whose leading packages only have error records.
## Changes
- `local/build-all.R`: drop the `AND error_occurred = FALSE` clause so `built` holds every version already attempted (built or errored) for this platform/arch; the existing filter then removes all of them up front.
- Rename the log line to `already-attempted` so the reported count reflects successes and errors.
- Update the surrounding comment to explain why errored versions are excluded.
## Behaviour change
Previously-errored versions are now dropped before the build loop instead of being iterated and individually skipped.
No package that would otherwise build is affected, `build_binary_package()` already skipped these internally.
Retrying errored versions is out of scope and would need a separate opt-in flag on both the prefilter and the in-loop skip.
Reviewed-on: #113
BuildKit's default GC caps the ephemeral cache tier — `RUN --mount=type=cache` mounts, local build context, git checkouts — at a hardcoded **512 MB** (shown as `488.3 MiB` in `buildx inspect`).
Across our 7-distro build matrix that fills instantly and forces re-downloads of system + R packages on every rebuild.
## Changes
- **`docker/buildkitd.toml`** (new) — GC config passed via `--config` when creating the `docker-container` builders `artemis` (amd64) and `gaia` (arm64):
- cache-mount tier: **512 MB → 8 GB**, retained 7 days
- total cache bounded at **40 GB** with **20 GB min-free**
- **`justfile`** — document the `--config docker/buildkitd.toml` flag on the builder-create commands so a recreate does not silently revert to the 512 MB default.
## Notes
- Limits are absolute (not `%`) because the two hosts differ ~6× in free space (Hetzner ~42 GiB free vs Mac mini ~279 GiB). The 20 GB min-free is the safety valve on the disk-tight Hetzner host; the old default wanted 64 GiB free, which does not exist there.
- Both live builders were already recreated with this config and verified running.
Reviewed-on: #112
## Problem
The `fs` 2.1.0 binary links **system libuv** (`readelf -d fs.so` shows `NEEDED libuv.so.1`).
fs's `configure` prefers system libuv whenever `pkg-config` resolves it, and our build images ship `libuv-devel` (installed as a pak build-time system requirement), so the resulting binary is dynamically linked against `libuv.so.1`.
That binary fails to load on any consumer machine without runtime libuv:
```text
unable to load shared object '.../fs/libs/fs.so':
libuv.so.1: cannot open shared object file: No such file or directory
```
`install.packages()`/renv do **not** install `SystemRequirements` (only `pak` does, and only inside the build container), so most consumers hit this.
Older fs 1.6.x always vendored libuv, so only the 2.x binaries regressed.
Reproduced in a clean `reg.devxy.io/r/r-alma:4.5-9`.
## Fix
Add `local/patches/fs/force-vendored-libuv.patch`, registered for all platforms.
It short-circuits `configure` to `cp -f src/Makevars.vendor src/Makevars; exit 0` before the pkg-config detection, forcing the bundled static libuv build (`tools/libuv-v1.52.0.tar.gz`, built via cmake).
An env/pkg-config override (`PKG_CONFIG_LIBDIR`) was tried first but the rebuilt binary still linked `libuv.so.1` (the registry `env` tier does not reach fs's configure step), so a source patch is used instead.
## Verification
Built end-to-end inside the real `build-env-redhat:9` image (system libuv present):
- patch fires (`Building static libuv (bincraft: forced vendored)`),
- cmake compiles the vendored libuv,
- resulting `fs.so` has **no `libuv.so.1`** in `NEEDED` (only libR, libstdc++, libm, libgcc_s, libc).
`Rscript local/validate-patches.R` passes (2 entries).
cmake confirmed present in the build-env images.
## Follow-up (not in this PR)
- Rebuild `fs 2.1.0` on every affected platform (rhel8/9/10, ubuntu jammy/noble, alpine 3.22/3.23; amd64 + arm64) and purge the CDN binary paths.
- CDN delivery gap: `purge_cdn_cache.sh` only purges `PACKAGES*`, never package binaries, so rebuilt binaries stay masked until their `.tar.gz` path is purged.
Reviewed-on: #111
## Summary
Stop hardcoding the bincraft version. Every `.crow` workflow and the build-one image pinned `@vX.Y.Z` (and a `packageVersion() != "X.Y.Z"` guard), so each bincraft release meant editing the version in ~8 places — and it was easy to miss one (the Dockerfile lagged at v4.2.1; v4.4.1 shipped without the empty-env fix because of exactly this churn).
## Change
New `local/install-bincraft.R` resolves the **latest release tag dynamically**:
- `git ls-remote --tags` on the public repo (no token),
- keep `vX.Y.Z` tags, pick the highest version (filtered/sorted in R for portability, not via git `--sort`/refspec which behaved inconsistently under `system2()`),
- `pak::pak("git::…@<latest>")` — idempotent on the git ref, so re-runs keep the package unless a newer tag exists.
All call sites now invoke the helper instead of a pinned version:
- `.crow/build-all-versions.yaml` (primary + per-minor pass)
- `.crow/build-all-versions-install-deps.yaml`
- `.crow/process-updates.yaml` (primary + per-minor pass)
- `.crow/weekly-rebuild-missing.yaml`
- `.crow/archive-missed-packages.yaml`
- `docker/build-one.Dockerfile` (ships the helper into the image; `ensure_bincraft` sources it)
## Effect
Tag a new bincraft release → the next CI run / `just rebuild` picks it up automatically. No more pin edits, and no more "forgot to bump the Dockerfile" drift.
## Verified
- Resolver returns the current latest tag (`v4.4.2`) via `git ls-remote` + R-side version sort.
- All five workflow YAMLs parse; helper R parses; air/editorconfig clean.
Note: this tracks the latest **tag**, so cutting a release is still the deliberate gate — CI won't pick up un-tagged main.
Reviewed-on: #107
## Summary
Fixes the RcppParallel patch, which was a **no-op** and left the build hanging.
The previous registry entry set `env: { RCPP_PARALLEL_USE_TBB: "0" }`. But `RCPP_PARALLEL_USE_TBB` is a **compile-time `-D` flag** in RcppParallel's Makevars — it is never read from the environment. So the override did nothing: `USE_TBB=Linux` (hardcoded from `uname`) still triggered the **bundled Intel TBB build**, which hangs/fails on musl (Alpine) and newer toolchains (g++ 15 on ubuntu-2604). The `Applying patch …` log only meant the env was set, not that it had any effect.
## Fix
Replace the env entry with a **source patch** (`local/patches/RcppParallel/disable-tbb.patch`) on `src/Makevars.in` that, on Linux:
- leaves `USE_TBB` unset → the whole bundled-TBB build/link path is skipped (no hang), and
- forces `PKG_CXXFLAGS += -DRCPP_PARALLEL_USE_TBB=0` → the sources compile the **TinyThread** backend (needed because `RcppParallel.h` otherwise auto-defaults TBB on for glibc Linux).
## Verification
In a Linux container, applying the patch and running `R CMD INSTALL RcppParallel`:
```
bundled_TBB_build=0 # bundled TBB build never runs
* DONE (RcppParallel) # installs via TinyThread
```
## Note
bincraft's `apply_source_patch` shells out to `patch`. If a build-env image lacks the `patch` tool (common on Alpine), the patch will report "did not apply cleanly" and fall back to an unpatched (hanging) build. If that happens, the follow-up is to switch bincraft's patch application to `git apply` (git is always present) — happy to do that if needed.
Reviewed-on: #106
## Summary
Silences the recurring `OpenTelemetry error: there is no package called 'otelsdk'` warnings during builds.
## Root cause
The `build-env-*` images configure an OTel exporter (traces/logs/metrics), but `otelsdk` (the R OTel SDK backend) isn't installed. pak's `otel` instrumentation therefore tries to load `otelsdk` on every run and logs the error, falling back to a no-op.
`OTEL_SDK_DISABLED=true` (already set in `build-one.Dockerfile`) does **not** help — the R `otel` package ignores it and gates purely on `OTEL_R_<SIGNAL>_EXPORTER` (then the standard `OTEL_<SIGNAL>_EXPORTER`). When that resolves to a real exporter (`otlp`/`http`/…) with no SDK present, you get the error.
## Fix
Set `OTEL_R_TRACES_EXPORTER`, `OTEL_R_LOGS_EXPORTER`, and `OTEL_R_METRICS_EXPORTER` to `none` so the R otel providers are clean no-ops:
- `docker/build-one.Dockerfile` — exported alongside the existing OTel var.
- `.crow/process-updates.yaml`, `weekly-rebuild-missing.yaml`, `build-all-versions.yaml`, `build-all-versions-install-deps.yaml` — added to each step's `environment` block.
Using the R-specific variables (not the standard `OTEL_*_EXPORTER`) keeps OTel intact for any non-R tooling in the images.
Reviewed-on: #105
## Summary
Fixes the **targeted rebuild** path (`just rebuild` → `docker/build-one.Dockerfile`) so it actually applies patches.
Two problems, both of which would make `build-one.R`'s `patches = "local/patches"` a silent no-op or a hard error:
1. **Stale bincraft pin.** The Dockerfile pinned bincraft `v4.2.1`, which predates the `patches` argument — so `build_binary_package(patches = ...)` would fail with `unused argument`. Bumped to **v4.4.0** to match the `.crow` workflows.
2. **Registry not shipped into the image.** Only `build-one.R` was copied in; `local/patches/registry.json` was absent, so `patches = "local/patches"` resolved to a nonexistent `/work/local/patches` and silently applied nothing. Added `COPY patches /work/local/patches` (build context is `local/`, CWD is `/work`).
## Why
This is the path used to verify the patching end-to-end, e.g.:
```bash
just rebuild alpine 3.23 amd64 rts2 1.0.3
```
rts2 depends on RcppParallel; with this fix the container installs bincraft v4.4.0, ships the registry, and the patched RcppParallel binary (`RCPP_PARALLEL_USE_TBB=0`) is served to pak during dependency install. Expect `Applying patch to RcppParallel …` in the log, RcppParallel installed as a binary (no recompile), then rts2 building and uploading.
Reviewed-on: #104
## Summary
Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`).
Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading.
## What's included
- `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs.
- `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace).
- `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes).
- `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`.
- `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`.
## ⚠️ Merge ordering (blocker)
This PR adds a `patches = ...` argument to `build_binary_package()` calls.
The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until:
1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and
2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`.
The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then.
Reviewed-on: #103
The function defaults to Hetzner S3 with HETZNER_S3_*_K3S env vars that
are not set in these workflows, so paws failed with "No compatible
credentials provided". Pass the B2 endpoint/region/bucket and
B2_S3_* secrets explicitly, matching the other S3 calls. Also use
set_codename(NULL) for the build-all-versions codename so it matches
the S3 repo path.
## Summary
Two changes:
1. **`OS`/`OS_VERSION` manual-run dropdowns** — give these form variables explicit `options:` lists (like `target_arch` and `R_VERSION`), so the manual-run form shows dropdowns instead of free-text, in both `build-all-versions.yaml` and `build-all-versions-install-deps.yaml`. Crow form variables are independent (no cascading), so the operator still has to pick a coherent `OS` + `OS_VERSION` combination (e.g. `redhat` + `9`, not `alpine` + `jammy`).
2. **Repairs `main`** — the crow fix from PR #96 (`bc2f6f1`) was lost when that PR was squashed (only the first commit was captured). As a result `main` currently carries the `OS: ${OS}` env vars that break Crow parsing (`unable to parse variable name`) and the unfixed `build-all.R`. This PR re-applies that fix: drop the env additions and derive `platform`/`arch` inside `build-all.R` from the container (bincraft codename → platform mapping + `Sys.info()` arch).
## Notes
- `OS_VERSION` options are quoted strings so tags like `8`/`9`/`10` aren't parsed as integers.
- Validated: both YAMLs parse, `build-all.R` parses.
Reviewed-on: #97
## Summary
Fixes the recurring `build-all-*` failure on agents where `build-all-versions-install-deps` did **not** run.
Both symptoms in the logs — `Parameter 3 does not have length 1` (repeated, in the metadata DB queries) and `argument is of length zero` (system-dependency install) — were the same bug: inside bincraft, `platform` was zero-length. `local/build-all.R` calls `build_binary_package()` without passing `platform`, so bincraft resolves it from the container codename. The primary build step never (re)installed/pinned bincraft and relied on whatever sat in the **per-agent** cache volume; `depends_on` only orders steps, it does not co-locate them on the same agent, so a job landing where install-deps never ran got a stale bincraft that left `platform` empty.
## Changes
- **`build-all-versions.yaml`**: pin bincraft `@v4.2.3` in the primary build step (mirroring the R-minor pass and `process-updates.yaml`), so every agent uses a known-good bincraft regardless of where install-deps ran.
- Align the R-minor pass `v4.2.2 → v4.2.3`.
- Export `OS`/`OS_VERSION`/`ARCH` as runtime env vars — previously only available for `${...}` interpolation, so `build-all.R`'s already-built dedup query matched platform `"-"` and skipped nothing.
- Fix the unarchive call: pass the codename via `bincraft::set_codename(NULL)` instead of the malformed `paste(OS, OS_VERSION)` (`"alpine 3.24"`), matching `archive-missed-packages.yaml`.
- **`build-all-versions-install-deps.yaml`**: pin install-deps to `@v4.2.3` (was installing HEAD), so the precomputed snapshot and per-agent library stay consistent pipeline-wide.
Reviewed-on: #96
## Summary
`build-all.R` reads three snapshot files from `/mnt/cache/packages/` that the `build-all-versions-install-deps` step precomputes: `pkgs_to_build.rds`, `r_minor_sensitive_pkgs.rds`, and `s3_cache.rds`.
That cache volume is **per-agent**, so a build job scheduled on a different (fresh) agent than the one that ran install-deps finds the snapshot absent and dies at `readRDS` (`cannot open compressed file '/mnt/cache/packages/pkgs_to_build.rds'`).
This adds a conditional guard at the top of `build-all.R`: when any of the three files is missing, it sources `local/packages-to-build.R` (which has all needed creds via `PGPASS` / `B2_S3_*` env, already present in the build step) and writes the derived `.rds` files — exactly mirroring the install-deps command.
- The first build job on a fresh agent repopulates the shared cache, so subsequent jobs on that agent reuse it.
- Concurrent jobs that also miss simply redo the work (accepted tradeoff vs. slow shared storage like NFS).
- Saves use a temp-file + atomic `file.rename`, so a concurrent reader never sees a half-written `.rds`.
Reviewed-on: #95
## Summary
- Switch crow workflow placement from a per-agent label (`agent: ${AGENT}`) to a group label (`group: rpkgs-${ARCH}`), so jobs target the `rpkgs-amd64` / `rpkgs-arm64` agent pools instead of a single named agent (`artemis`/`gaia`).
- Remove the now-unused `AGENT` matrix variable from all build/process/weekly workflows.
- Update header comments to reflect group-based placement.
Agents opt into a pool via `CROW_AGENT_LABELS="group=rpkgs-<arch>"`. The group name derives from each matrix row's `ARCH`, so adding or replacing agents no longer requires touching the workflow files.
Reviewed-on: #94
The matrix consolidation hardcoded `xvfb-run`, which is absent on the
rhel-10 image (it ships Xwayland's xwfb-run), causing
`xvfb-run: command not found`. Re-add the per-platform launcher detection
the old rhel-10 files used: prefer xwfb-run + weston, fall back to xvfb-run.
Unquoted `PROCESS_NEW: FALSE` is parsed as a YAML boolean and interpolated
into the R command as lowercase `false`, causing `process_new = false` and
`Error: object 'false' not found`. Quoting keeps the literal FALSE/TRUE that
R expects.
The matrix consolidation routed cron events through
`evaluate: 'CI_PIPELINE_CRON == "..."'`, but CI_PIPELINE_CRON is not
exposed to the when.evaluate constraint context, so no matrix row ever
matched and the scheduled process-updates / weekly runs stopped firing.
Switch back to the first-class `cron:` name filter (still parameterized
per matrix row) which the server matches directly against the fired cron.
## Summary
Consolidate the four `build-all-versions*` files into two, mirroring the matrix approach used for the process-updates / weekly families:
| Before | After |
|---|---|
| `build-all-versions-amd64.yaml` + `build-all-versions-arm64.yaml` | `build-all-versions.yaml` |
| `build-all-versions-install-deps-amd64.yaml` + `-arm64.yaml` | `build-all-versions-install-deps.yaml` |
`ARCH` becomes a matrix axis carrying the per-arch differences: `agent` (artemis/gaia), `BACKEND` (kubernetes/docker), and `R_PKG_CACHE_DIR`. Build-all keeps its `SPLIT_INDEX` parallelism, so its matrix is ARCH x SPLIT = 8 rows; only the rows matching the selected arch run.
**Invocation unchanged.** Routing uses the existing task selector `task == "build-all-${ARCH}"`, so `--var task=build-all-amd64 ...` still runs exactly the amd64 rows. `OS` / `OS_VERSION` / `R_VERSION` / `K8S_INSTANCE_TYPE` are still passed via `--var`. (You can drop `--var ARCH=...` now — arch comes from the matrix/task.)
**No cron.** These pipelines are manual-only, so there are no scheduled runs to affect.
## Validation before merge
Same matrix-interpolation-in-`when`/`labels` mechanism validated for #92, plus it now relies on per-row `agent`/`BACKEND` selecting the right execution backend (kubernetes for amd64, docker for arm64). Before merge, on this branch trigger a small `task=build-all-arm64` run and confirm it lands on the gaia/docker backend (and amd64 on artemis/kubernetes). Originals are in git history for rollback.
Note: `archive-missed-packages.yaml` was already a single consolidated matrix file (cron + manual) and is left untouched.
Reviewed-on: #93
## Summary
Collapse **44** per-platform crow pipeline files into **3** matrix-driven files (one per family), using `matrix.include` + the crow #1165 declarative manual `variables:` block:
| Family | Before | After |
|---|---|---|
| process-updates | 14 | `process-updates.yaml` |
| weekly-audit-missing | 16 | `weekly-audit-missing.yaml` |
| weekly-rebuild-missing | 14 | `weekly-rebuild-missing.yaml` |
- **One file per family** (not split by arch). Arch placement is via the agent label as a matrix var (`artemis`=amd64, `gaia`=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-`nodeSelector` placement (no agent label).
- **Matrix axis is named `OS`** (e.g. `redhat-9`) per request, with per-row `R_VERSION`, image tag, codename, and `process_new` (FALSE for alpine).
- **Runs preserved 1:1:**
- *Cron*: each existing `<family>-<os>-<arch>` cron still fires only its matching matrix row via `CI_PIPELINE_CRON`. Server-side cron entries unchanged.
- *Manual*: a #1165 dropdown variable (`process_cran_updates` / `weekly_audit_missing` / `weekly_rebuild_missing`) selects a single `<os>-<arch>` or `all`. weekly-rebuild defaults to `all`, matching its former bare `event: manual` trigger.
## Behavior deltas (intentional, flagged)
To fit one file per family the backend was unified:
- The amd64-only `node.kubernetes.io/instance-type: AX42` nodeSelector pin is **dropped**; placement now relies on the `artemis`/`gaia` agents.
- **process-updates arm64** jobs now share the same `backend_options` as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped.
## Validation required before merge
Relies on crow interpolating matrix variables inside `when.evaluate`, `labels.agent`, and `commands`. Interpolation in image/env/commands is standard crow; the novel bit is `when.evaluate`. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch:
1. Trigger manually with the dropdown = `redhat-9-amd64` -> confirm only that one job runs (image `build-env-redhat:9`, R 4.4.3, agent artemis).
2. Trigger with `all` -> confirm all os/arch jobs schedule.
Crons only fire on the default branch, so no collision while unmerged. Fallback if `${...}` doesn't interpolate in `when`: a runtime `case "$CI_PIPELINE_CRON" in ...` guard, same files otherwise.
Reviewed-on: #92
## Summary
When a `build-all-*` workflow is restarted, the build job re-reads the static `pkgs_to_build.rds` that the install-deps step produced once, so it cycles over every package an interrupted run already built. This adds a DB-based skip filter so a restart only processes what is genuinely left.
- At job start, `build-all.R` queries the `single_builds` metadata table for `(name, tag)` already built successfully (`error_occurred = FALSE`) on this `platform`/`arch`, and drops those pairs from the chunk before the build loop. It logs how many it skipped.
- One indexed query, one round trip, run before the pak forks — no extra S3 listing and no new Python/s3fs memory pressure (`RPostgres`/`DBI` are already used in the container).
- Errored versions are intentionally **not** skipped, so transient failures still get retried on restart.
## Dependency
Correctness depends on a `error_occurred = FALSE` row meaning the binary is actually published. That guarantee is added in rpkgs/bincraft#56 (success row written only after a confirmed S3 upload). This PR should land together with / after a bincraft release including that fix.
Reviewed-on: #91
The sensitive-only multi-R build passes ran under each minor's R but loaded
bincraft from the shared R_LIBS_USER=/mnt/cache/R-pkgs cache, which is compiled
under the primary R and fails to load under a newer minor (e.g. 4.6:
"undefined symbol: SETLENGTH") — so those minor builds silently no-op'd.
Give each non-primary minor its own R_LIBS_USER=/mnt/cache/R-pkgs-<minor> and
install bincraft 4.2.1 there (built for that minor); the rest of bincraft's
deps load from the per-version /opt/R/<minor>/lib. Applied to the
build-all-versions and process-updates sensitive-only passes.
The per-minor upload_package_index() loop invoked each minor's own R, which
loads bincraft from the shared /mnt/cache/R-pkgs cache (compiled under the
primary R). Under a newer minor (e.g. 4.6) that fails to load
("undefined symbol: SETLENGTH") and the index step dies. Indexing is pure S3
work and needs no specific R, so run it under /opt/R/$R_VERSION like the generic
index and build-all-versions already do.
A package's ABI sensitivity is the same across R minors, so for a non-sensitive
package the per-minor loop only ran `ensure_bincraft` (a ~14s bincraft/cranlike
install per minor) before build-one.R classified and skipped it. The primary
pass now writes a .r_minor_sensitive sentinel when it builds a sensitive
package, and the wrapper gates the whole per-minor loop (installs included) on
that file — non-sensitive rebuilds no longer touch other minors' libraries.
- Dockerfile: install bincraft into each R minor, run a primary pass plus a
sensitive-only pass under every other /opt/R/[0-9]* minor (deduped by minor),
probe/skip xvfb, set GIT_TERMINAL_PROMPT=0; extra-minor failures are non-fatal.
- build-one.R: add --sensitive-only mode, log + shallow-clone the ABI classify
step, and clear cranlike's stale ./PACKAGES.db before each index refresh
(workaround for the "table packages already exists" bug; pending cranlike fix).
- Accept TF_VAR_-prefixed B2 keys (from direnv); make GITHUB_PAT optional.
- Stream the remote build with --progress=plain.
- Document creating the artemis/gaia buildx builders with the docker-container
driver (the default remote driver does not work with an ssh:// docker host).
- Pin bincraft v4.2.0 -> v4.2.1 (per-minor archive.rds/index fix) across the
process-updates, weekly-rebuild and archive workflows.
- Restrict the per-R-minor build and index loops to /opt/R/[0-9]* so the
`current` symlink dir is not treated as a distinct minor (avoids double-
building a minor and writing a bogus contrib/current/ index slot).
## Summary
Follow-up fix to the `just rebuild` recipe (#87): the local `.envrc` exports the B2 credentials as `TF_VAR_B2_S3_ACCESS_KEY` / `TF_VAR_B2_S3_SECRET_KEY` (and has no `GITHUB_PAT`), but the recipe required the plain `B2_S3_ACCESS_KEY` / `B2_S3_SECRET_KEY` names and always passed a `github_pat` secret.
- Accept the `TF_VAR_`-prefixed B2 names (falling back to the plain names if set directly).
- Pass the `github_pat` buildx secret only when `GITHUB_PAT` is set, so the build works without it.
Reviewed-on: #88
## Summary
Adds a local `just rebuild` recipe to (re)build specific versions of a single package on a given OS/arch, dispatching to a remote buildx builder (the build runs there, not locally).
- `just rebuild <os> <tag> <arch> <package> <version>...` → `docker buildx build --builder <artemis|gaia> --platform linux/<arch> …` (amd64→artemis, arm64→gaia; names + `R_VERSION` env-overridable).
- `docker/build-one.Dockerfile` runs `build-one.R` as a secret-mounted `RUN`, built `--no-cache --output type=cacheonly` (pure side-effect: the S3 upload; no image kept).
- `local/build-one.R` auto-classifies each version via the ABI classifier (risky → per-minor slot `contrib/<x.y>/`, else generic), force-rebuilds + uploads + stores metadata, then refreshes the touched slot's `PACKAGES` index.
## Prerequisites
- buildx builders named `artemis` (amd64) and `gaia` (arm64) registered (`docker buildx create --name artemis ssh://…`).
- Exported secrets: `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY`, `PGPASS` (`GITHUB_PAT` optional).
- bincraft `v4.2.0` tag must exist (the build installs `@v4.2.0` and uses its classifier + per-minor index API).
Reviewed-on: #87
## Summary
Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered.
## What's in here
**Detection / precompute**
- `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests.
- `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions.
**Full build**
- `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset.
- `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor.
**Iterative build**
- All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added.
**Tooling / housekeeping**
- Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`.
- Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts.
- Design + implementation docs under `docs/superpowers/`.
## Requires before merge
- A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected.
Reviewed-on: #84
## Summary
The `rm -rf /mnt/cache/...` block in 14 `process-updates-*` files had two real issues:
1. `/mnt/cache/pkgcache/R/pkgcache` listed twice — harmless duplicate.
2. `/mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres` — missing a space between two paths, so `rm` walks a nested nonsense path that never exists. The `00LOCK-RPostgres` lock cleanup never actually ran.
Collapses to the same shape `weekly-rebuild-missing-*` already uses:
```
rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/{bincraft,pkgcache} /mnt/cache/pkgcache/R/pkgcache
```
`00LOCK-*` covers every stale lock (pak, bincraft, RPostgres, r-pkg-binaries, …) without enumerating them, so adding a new locked package later doesn't need another YAML touch.
14 files, 1 line changed each.
Reviewed-on: #76
## Summary
While cleaning this up I noticed the script **defined** `purge_cdn_cache()` but never **called** it. Every CI run only declared the helper and exited cleanly without issuing a single curl. The PACKAGES freshness on cran.devxy.io / cran.rpkgs.com has been carried entirely by R's `Cache-Control: no-cache` header on the index files.
Fixes in one shot:
1. **Actually run the purge.** The shifted args (`api_key`, `arch`, `os_id`, `domain…`) are now consumed inline and a curl POST is issued per (domain × resource).
2. **Drop `set -x`** — leaked the `AccessKey:` header into job logs.
3. **Drop the Python URL-encoder.** `curl -G --data-urlencode "url=…" --data "async=false" https://api.bunny.net/purge` does the same with no Python. The workflow purge steps can later switch from `alpine:3.23 + apk add bash curl` to a slimmer curl-only image.
4. **Add `src/contrib/Meta/archive.rds`** to the purged resource list. It's rewritten on every `process_cran_updates` run (see README) and was being served stale.
## Risks
- This is the **first time** the script actually purges anything. If anything else (e.g. a downstream service) relied on the no-op behavior, this PR is the moment it stops being silent. I don't see any such caller.
- Edge cache miss right after a purge means an origin S3 fetch — minor latency uptick on the first request per region per resource.
Reviewed-on: #78
## Summary
`archive-missed-packages.yaml`'s matrix was last touched when redhat-9 and alpine322 were the newest targets. redhat-10 and alpine323 builds are now wired up across `process-updates-*` and `weekly-rebuild-missing-*` workflows, but the daily archive sweep wasn't catching them — so missed-archive packages on those platforms accumulate untouched.
Adds the 4 missing entries:
- `redhat-10` × `{amd64, arm64}`
- `alpine323` × `{amd64, arm64}`
Reviewed-on: #75
## Summary
`local/build-all.R` had a 75-name hardcoded `exclude <- c(...)` vector that had drifted from `local/excluded-packages.json` — `RcmdrPlugin.ROC` was in the R vector but missing from the JSON.
`.crow/weekly-rebuild-missing-*.yaml` already reads the JSON via `jsonlite::fromJSON(...)[["package"]]`. This brings `build-all.R` in line with that pattern.
Changes:
- **`local/excluded-packages.json`**: add the missing `RcmdrPlugin.ROC` entry (reason `"hang"`, matching siblings).
- **`local/build-all.R`**: replace the 16-line hardcoded vector with one `jsonlite::fromJSON(...)` call.
- **`.crow/build-all-versions-install-deps-{amd,arm}64.yaml`**: add `jsonlite` to the install-deps `pak::pak()` list so it's available in `/mnt/cache/R-pkgs` for the build step.
- **`.crow/build-all-versions-arm64.yaml`**: drop the dead base64-encoded `SKIP_PKGS` docs comment that nobody was passing as a `--var` anyway; replace with a one-line pointer to the JSON.
Reviewed-on: #79
## Summary
Both helpers still point at the old Hetzner storage:
- `Justfile` — all 3 recipes (`build-all`, `build-single`, `process-updates`) hit `hel1.your-objectstorage.com` / bucket `devxy-r-package-binaries-hel1` using `HETZNER_S3_*_K3S` env vars.
- `local/manual-package-index-update.R` — same endpoint + bucket, hardcoded into `s3fs::s3_dir_ls()` / `s3_file_delete()` calls.
Storage moved to Backblaze (`s3.eu-central-003.backblazeb2.com` / `devxy-rpkgs-binaries`) a while back, so running either of these today would write to the wrong bucket or fail outright.
No callers reference them in-tree, deleting outright. A one-off rebuild can just call `bincraft::upload_package_index()` directly with current Backblaze settings.
Reviewed-on: #77
## Summary
`block_ips = ["185.172.53.0"]` was duplicated in both `bunnynet_pullzone` resources in `cdn.tf`. Move it to a typed `cdn_block_ips` variable in `vars.tf` with the existing IP as the default.
- No plan diff on apply — same value, just sourced from `var.cdn_block_ips` instead of a literal.
- Adding/removing IPs is now a one-line var override (or a default change) instead of two edits in the resource bodies.
## Interaction with #80 (for_each refactor)
`#80` collapses the two pullzones to `bunnynet_pullzone.this[for_each]`. Either order works; whichever lands second is a trivial one-line rebase on the surviving `block_ips =` line.
Reviewed-on: #81
## Summary
bincraft is baked into the build-env images, but every workflow still unconditionally calls `pak::pak("git::.../bincraft.git")` (~30–60s per job).
Pin bincraft to a specific tag (currently `v4.0.2`) and guard the install on a version check.
```
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.0.2") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.0.2")'
```
- Image matches the pin → skip install entirely.
- Image is missing bincraft or has a different version → install the pinned tag (covers both broken bakes and intentional version bumps).
- Renovate customManager keeps the pin and the version-check string in sync whenever bincraft cuts a new tag (uses `git-tags` datasource against the bincraft repo).
Touches 43 `.crow` YAMLs + `renovate.json`.
Reviewed-on: #72
## Summary
`process-updates-*` already calls `bincraft::upload_package_index()` at the end of every daily delta run, and with #71 the CDN purge happens there too.
The dedicated `update-package-index-*` chain re-ran the same `upload_package_index()` against the same S3 prefixes — once per platform per day — and was serialized via `depends_on:` purely to dodge S3 rate limits.
Net effect: ~40 min cold rebuild × 14 platforms = ~9 h/day of duplicated work, with no consumer waiting on it (process-updates had already published the fresh index).
This deletes the 14 chain files (~1.1k lines). The matching `update-package-index` / `update-package-index-*` cron entries on the Crow side become inert with the YAMLs gone and can be removed in a follow-up cron-config change.
Best reviewed alongside (or after):
- #71 — purge CDN after process-updates
- #72 — guard bincraft reinstall
- #73 — drop redundant cranlike/desc installs
## Risks
- If `process-updates-*` ever fails for a given platform, the chain used to act as a daily safety net for the PACKAGES index. Failures will now surface only via the missing daily push instead of being silently masked. The existing `weekly-audit-missing-*` workflow continues to flag stale state.
- `local/manual-package-index-update.R` still exists for one-off manual rebuilds.
Reviewed-on: #74
## Summary
`bincraft`'s DESCRIPTION already pulls both forks via `Remotes:`:
```
Imports:
...
cranlike,
...
Remotes:
pat-s/cranlike@s3,
pat-s/desc@description-from-remote
```
So `pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")` resolves the forks and installs them transitively. The explicit follow-up `pak::pak("pat-s/cranlike@s3")` and `pak::pak("pat-s/desc@description-from-remote")` calls in `update-package-index-*.yaml` are dead weight.
Drops the two lines from all 14 `update-package-index-*` files.
`process-updates-*.yaml` already only installs `bincraft` — this brings the two workflows in sync.
Reviewed-on: #73
## Summary
- `process_cran_updates()` ends with `upload_package_index()`, refreshing `src/contrib/PACKAGES{,.gz,.rds,.db}` in S3 — but no CDN purge follows.
- The CDN was only purged by the daily `update-package-index-*` chain, so between cron runs the edge kept serving stale indexes. A binary that landed in S3 was effectively invisible to `install.packages()` until the index-update chain caught up.
Adds the same `Purge CDN cache` step that `update-package-index-*` already uses, parameterized per platform/arch. The script and image are unchanged.
Reviewed-on: #71
## Summary
- Switches all 64 `.crow/*.yaml` workflows, the `Justfile`, and the commented-out `build-all-versions-install-deps.yaml` template to the new multi-R-version container images.
- Each image tag drops `-${R_VERSION}` (e.g. `build-env-alpine:3.22-4.5` → `build-env-alpine:3.22`); workflows pick an R version by calling `/opt/R/${R_VERSION}/bin/R` explicitly.
- Pattern-1 workflows (hard-coded image tag) gain a single `R_VERSION:` env var per file.
- Pattern-2 workflows (`build-all-versions-*`, matrix-driven) continue to receive `R_VERSION` via `crow pipeline create --var`.
## Platform → R_VERSION mapping
| Platform | Image | R_VERSION |
| --- | --- | --- |
| alpine 3.22 / 3.23 | `build-env-alpine:3.{22,23}` | 4.5.3 |
| ubuntu jammy / noble | `build-env-ubuntu:{jammy,noble}` | 4.4.3 |
| redhat 8 / 9 | `build-env-redhat:{8,9}` | 4.4.3 |
| redhat 10 | `build-env-redhat:10` | 4.5.3 |
The `alpine-321` audit-only workflows fall back to `build-env-alpine:3.23` since no 3.21 image exists in the new scheme; the audits only query S3/CRAN, so the container OS does not affect correctness.
## Bug fixes folded in
1. All six `weekly-audit-missing-alpine-*.yaml` files were pinned to `alpine:3.23-4.5` regardless of platform.
Now each one points at its own alpine image.
2. All fourteen `update-package-index-*.yaml` files were pinned to `build-env-ubuntu:noble-4.4` (or `noble-4.5` for `redhat-10-amd64`) regardless of platform.
Now each one uses its own platform's image.
3. `process-updates-ubuntu-2404-{amd64,arm64}.yaml` had `R_VERSION` of `4.4` (no patch) baked into the tag.
Aligned to `4.4.3`, matching the audit and rebuild counterparts.
## Smoke targets (post-merge)
One workflow per shape, ready to dry-run:
- `process-updates-alpine-322-amd64.yaml`
- `weekly-rebuild-missing-redhat-9-amd64.yaml`
- `weekly-audit-missing-ubuntu-2204-amd64.yaml`
- `update-package-index-redhat-10-amd64.yaml`
- `archive-missed-packages.yaml`
- `build-all-versions-amd64.yaml` (with its install-deps predecessor)
- `build-r-minor-sensitive-packages.yaml`
## Follow-ups (out of scope, surfaced during review)
- `process-updates-redhat-10-{amd64,arm64}.yaml` set `codename = "rhel9"` in their `upload_package_index(...)` calls.
Pre-existing bug — should be `"rhel10"`. The `update-package-index-redhat-10-*` counterparts already use `"rhel10"` correctly.
- `update-package-index-redhat-8-amd64.yaml` is missing the `packageVersion("bincraft")` diagnostic line that every sibling file has.
Pre-existing asymmetry between amd64 and arm64.
## Spec & plan
- Spec: `docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md`
- Plan: `docs/superpowers/plans/2026-05-25-multi-r-version-images.md`
Reviewed-on: #70
Both pipelines had platform = 'alpine-321' instead of their actual target, causing builds to upload to the wrong S3 path and the PACKAGES index to not reflect newly built packages.
s3fs uses reticulate/Python which significantly increases R process memory.
When pak tries to fork R to install package dependencies, the enlarged
process can't be duplicated within the container memory limit.
Now the install-deps step saves the S3 file listing as s3_cache.rds,
and the build step reads it with readRDS() — no s3fs loading needed.
- Load bincraft eagerly via library() to avoid lazy-load memory spike
- Do bulk S3 listing upfront and pass as s3_package_cache to avoid
per-package S3 calls that accumulate memory and trigger fork failures
- Consolidate amd64/arm64 exclude lists into single script
- Much easier to read and maintain than a YAML-embedded one-liner
The precompute script was constructing S3 paths as "redhat10" while
bincraft uses "rhel10" as the codename. This mismatch caused the
precompute to find no existing packages in S3, producing ~77K false
positive package versions that all get skipped at build time.
Now uses bincraft::set_codename() to ensure path alignment.
Also fixes nrow() vs length() in install-deps summary message.
force=TRUE bypasses the error-DB filter and overwrites build metadata,
causing failed packages to be retried indefinitely. Since the precompute
step already filters to missing versions, force is unnecessary.
packages-to-build.R now saves the full (Package, Version) data.table
so the build step can pass specific tags to build_binary_package(),
eliminating redundant per-package tag discovery and S3 checks.
Also reduces archive versions from 9 to 4 (+ 1 release = 5 total).
Each process-updates workflow now calls upload_package_index at the end to update the PACKAGES files for its platform/arch immediately after builds complete. The full update-package-index cron should be rescheduled to weekly.
All 12 workflows were triggered simultaneously by the update-package-index cron. Now they run one after another via depends_on. Individual cron triggers still allow running one at a time.
Uses install-deps step, precomputed package list, weston compositor, stderr redirect, sequential plan with progress output, and unique X server numbers.
Move packages-to-build.R execution to the single install-deps step and save results as RDS. Matrix jobs read the precomputed list instead of loading the massive CRAN archive/release databases, freeing memory for fork/system calls.
Adds build-all-versions-install-deps pipeline that runs once before the matrix builds, cleaning stale locks and installing bincraft to the shared cache without contention.
The S3 credentials, endpoint, and bucket name were still pointing to the old Hetzner storage after the Backblaze migration, causing silent build failures.
