## Problem
The arm64 `build-all` pipeline fills the macmini (gaia) host disk despite an 8h prune.
Root cause is not images or job volumes: it is the persistent dep-cache volume, specifically `pkgcache/R/pkgcache/_metadata`, which grew to ~165 GB.
`{pkgcache}` mints a new content hash for the "patched" binaries repo on every PACKAGES change, so each per-package build writes a fresh ~70 MB `pkgs-<hash>.rds` (+ `patched-<hash>/`) that is never evicted (2407 snapshots observed).
When the disk hits 100% OrbStack stops and the on-host prune can no longer connect to the daemon, so it never self-heals.
## Change (Workstream A of the disk-fill fix)
- Add `trim_pkgcache_metadata()` to `local/r-minor-helpers.R`: keeps the newest `keep` (default 20) `patched-*`/`pkgs-*.rds` entries under `_metadata`, deleting only entries older than `min_age_secs` (default 600s) so it never races the up-to-4 concurrent split-jobs sharing the volume.
Preserves `pkg/` downloads and the stable CRAN/BioC/INLA repo dirs.
No-op when `R_PKG_CACHE_DIR` is empty (amd64) or `_metadata` is absent (first run).
- Call it every 25 packages inside the build loop in `local/build-all.R`.
- Add a defensive start-of-run cleanup of `_metadata/patched-*` + `pkgs-*.rds` to the two workflows that mount the persistent volume (`build-all-versions.yaml`, `build-all-versions-install-deps.yaml`).
Only these paths are touched; `process-updates.yaml`/`weekly-rebuild-missing.yaml` (no persistent volume) are unchanged.
Follow-ups (separate workstreams): on-host self-healing prune watcher + OrbStack disk cap (ansible), and Prometheus/Grafana alerting (k8s-talos).
Upstream: bincraft patched-repo hash churn is the true source fix.
New unit tests (6) for the helper; full suite 24/24 green.
Reviewed-on: #110
## Motivation
Build jobs were cycling through hundreds of packages that were only ever printed as `Skipping … due to previous build error recorded in metadata DB`, wasting wall-clock on per-package preparation before dropping each one.
## Cause
The prefilter query in `local/build-all.R` selected only successfully-built versions (`error_occurred = FALSE`) into `built`, so line 112 removed only those from the chunk.
Every previously-errored version stayed in the work list and was walked one-by-one, each hitting the internal skip in `build_binary_package()`.
This also explains the misleading `Skipped 0 already-built package versions` line for alphabetical chunks whose leading packages only have error records.
## Changes
- `local/build-all.R`: drop the `AND error_occurred = FALSE` clause so `built` holds every version already attempted (built or errored) for this platform/arch; the existing filter then removes all of them up front.
- Rename the log line to `already-attempted` so the reported count reflects successes and errors.
- Update the surrounding comment to explain why errored versions are excluded.
## Behaviour change
Previously-errored versions are now dropped before the build loop instead of being iterated and individually skipped.
No package that would otherwise build is affected, `build_binary_package()` already skipped these internally.
Retrying errored versions is out of scope and would need a separate opt-in flag on both the prefilter and the in-loop skip.
Reviewed-on: #113
## Problem
The `fs` 2.1.0 binary links **system libuv** (`readelf -d fs.so` shows `NEEDED libuv.so.1`).
fs's `configure` prefers system libuv whenever `pkg-config` resolves it, and our build images ship `libuv-devel` (installed as a pak build-time system requirement), so the resulting binary is dynamically linked against `libuv.so.1`.
That binary fails to load on any consumer machine without runtime libuv:
```text
unable to load shared object '.../fs/libs/fs.so':
libuv.so.1: cannot open shared object file: No such file or directory
```
`install.packages()`/renv do **not** install `SystemRequirements` (only `pak` does, and only inside the build container), so most consumers hit this.
Older fs 1.6.x always vendored libuv, so only the 2.x binaries regressed.
Reproduced in a clean `reg.devxy.io/r/r-alma:4.5-9`.
## Fix
Add `local/patches/fs/force-vendored-libuv.patch`, registered for all platforms.
It short-circuits `configure` to `cp -f src/Makevars.vendor src/Makevars; exit 0` before the pkg-config detection, forcing the bundled static libuv build (`tools/libuv-v1.52.0.tar.gz`, built via cmake).
An env/pkg-config override (`PKG_CONFIG_LIBDIR`) was tried first but the rebuilt binary still linked `libuv.so.1` (the registry `env` tier does not reach fs's configure step), so a source patch is used instead.
## Verification
Built end-to-end inside the real `build-env-redhat:9` image (system libuv present):
- patch fires (`Building static libuv (bincraft: forced vendored)`),
- cmake compiles the vendored libuv,
- resulting `fs.so` has **no `libuv.so.1`** in `NEEDED` (only libR, libstdc++, libm, libgcc_s, libc).
`Rscript local/validate-patches.R` passes (2 entries).
cmake confirmed present in the build-env images.
## Follow-up (not in this PR)
- Rebuild `fs 2.1.0` on every affected platform (rhel8/9/10, ubuntu jammy/noble, alpine 3.22/3.23; amd64 + arm64) and purge the CDN binary paths.
- CDN delivery gap: `purge_cdn_cache.sh` only purges `PACKAGES*`, never package binaries, so rebuilt binaries stay masked until their `.tar.gz` path is purged.
Reviewed-on: #111
## Summary
Stop hardcoding the bincraft version. Every `.crow` workflow and the build-one image pinned `@vX.Y.Z` (and a `packageVersion() != "X.Y.Z"` guard), so each bincraft release meant editing the version in ~8 places — and it was easy to miss one (the Dockerfile lagged at v4.2.1; v4.4.1 shipped without the empty-env fix because of exactly this churn).
## Change
New `local/install-bincraft.R` resolves the **latest release tag dynamically**:
- `git ls-remote --tags` on the public repo (no token),
- keep `vX.Y.Z` tags, pick the highest version (filtered/sorted in R for portability, not via git `--sort`/refspec which behaved inconsistently under `system2()`),
- `pak::pak("git::…@<latest>")` — idempotent on the git ref, so re-runs keep the package unless a newer tag exists.
All call sites now invoke the helper instead of a pinned version:
- `.crow/build-all-versions.yaml` (primary + per-minor pass)
- `.crow/build-all-versions-install-deps.yaml`
- `.crow/process-updates.yaml` (primary + per-minor pass)
- `.crow/weekly-rebuild-missing.yaml`
- `.crow/archive-missed-packages.yaml`
- `docker/build-one.Dockerfile` (ships the helper into the image; `ensure_bincraft` sources it)
## Effect
Tag a new bincraft release → the next CI run / `just rebuild` picks it up automatically. No more pin edits, and no more "forgot to bump the Dockerfile" drift.
## Verified
- Resolver returns the current latest tag (`v4.4.2`) via `git ls-remote` + R-side version sort.
- All five workflow YAMLs parse; helper R parses; air/editorconfig clean.
Note: this tracks the latest **tag**, so cutting a release is still the deliberate gate — CI won't pick up un-tagged main.
Reviewed-on: #107
## Summary
Fixes the RcppParallel patch, which was a **no-op** and left the build hanging.
The previous registry entry set `env: { RCPP_PARALLEL_USE_TBB: "0" }`. But `RCPP_PARALLEL_USE_TBB` is a **compile-time `-D` flag** in RcppParallel's Makevars — it is never read from the environment. So the override did nothing: `USE_TBB=Linux` (hardcoded from `uname`) still triggered the **bundled Intel TBB build**, which hangs/fails on musl (Alpine) and newer toolchains (g++ 15 on ubuntu-2604). The `Applying patch …` log only meant the env was set, not that it had any effect.
## Fix
Replace the env entry with a **source patch** (`local/patches/RcppParallel/disable-tbb.patch`) on `src/Makevars.in` that, on Linux:
- leaves `USE_TBB` unset → the whole bundled-TBB build/link path is skipped (no hang), and
- forces `PKG_CXXFLAGS += -DRCPP_PARALLEL_USE_TBB=0` → the sources compile the **TinyThread** backend (needed because `RcppParallel.h` otherwise auto-defaults TBB on for glibc Linux).
## Verification
In a Linux container, applying the patch and running `R CMD INSTALL RcppParallel`:
```
bundled_TBB_build=0 # bundled TBB build never runs
* DONE (RcppParallel) # installs via TinyThread
```
## Note
bincraft's `apply_source_patch` shells out to `patch`. If a build-env image lacks the `patch` tool (common on Alpine), the patch will report "did not apply cleanly" and fall back to an unpatched (hanging) build. If that happens, the follow-up is to switch bincraft's patch application to `git apply` (git is always present) — happy to do that if needed.
Reviewed-on: #106
## Summary
Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`).
Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading.
## What's included
- `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs.
- `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace).
- `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes).
- `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`.
- `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`.
## ⚠️ Merge ordering (blocker)
This PR adds a `patches = ...` argument to `build_binary_package()` calls.
The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until:
1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and
2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`.
The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then.
Reviewed-on: #103
## Summary
Two changes:
1. **`OS`/`OS_VERSION` manual-run dropdowns** — give these form variables explicit `options:` lists (like `target_arch` and `R_VERSION`), so the manual-run form shows dropdowns instead of free-text, in both `build-all-versions.yaml` and `build-all-versions-install-deps.yaml`. Crow form variables are independent (no cascading), so the operator still has to pick a coherent `OS` + `OS_VERSION` combination (e.g. `redhat` + `9`, not `alpine` + `jammy`).
2. **Repairs `main`** — the crow fix from PR #96 (`bc2f6f1`) was lost when that PR was squashed (only the first commit was captured). As a result `main` currently carries the `OS: ${OS}` env vars that break Crow parsing (`unable to parse variable name`) and the unfixed `build-all.R`. This PR re-applies that fix: drop the env additions and derive `platform`/`arch` inside `build-all.R` from the container (bincraft codename → platform mapping + `Sys.info()` arch).
## Notes
- `OS_VERSION` options are quoted strings so tags like `8`/`9`/`10` aren't parsed as integers.
- Validated: both YAMLs parse, `build-all.R` parses.
Reviewed-on: #97
## Summary
`build-all.R` reads three snapshot files from `/mnt/cache/packages/` that the `build-all-versions-install-deps` step precomputes: `pkgs_to_build.rds`, `r_minor_sensitive_pkgs.rds`, and `s3_cache.rds`.
That cache volume is **per-agent**, so a build job scheduled on a different (fresh) agent than the one that ran install-deps finds the snapshot absent and dies at `readRDS` (`cannot open compressed file '/mnt/cache/packages/pkgs_to_build.rds'`).
This adds a conditional guard at the top of `build-all.R`: when any of the three files is missing, it sources `local/packages-to-build.R` (which has all needed creds via `PGPASS` / `B2_S3_*` env, already present in the build step) and writes the derived `.rds` files — exactly mirroring the install-deps command.
- The first build job on a fresh agent repopulates the shared cache, so subsequent jobs on that agent reuse it.
- Concurrent jobs that also miss simply redo the work (accepted tradeoff vs. slow shared storage like NFS).
- Saves use a temp-file + atomic `file.rename`, so a concurrent reader never sees a half-written `.rds`.
Reviewed-on: #95
## Summary
When a `build-all-*` workflow is restarted, the build job re-reads the static `pkgs_to_build.rds` that the install-deps step produced once, so it cycles over every package an interrupted run already built. This adds a DB-based skip filter so a restart only processes what is genuinely left.
- At job start, `build-all.R` queries the `single_builds` metadata table for `(name, tag)` already built successfully (`error_occurred = FALSE`) on this `platform`/`arch`, and drops those pairs from the chunk before the build loop. It logs how many it skipped.
- One indexed query, one round trip, run before the pak forks — no extra S3 listing and no new Python/s3fs memory pressure (`RPostgres`/`DBI` are already used in the container).
- Errored versions are intentionally **not** skipped, so transient failures still get retried on restart.
## Dependency
Correctness depends on a `error_occurred = FALSE` row meaning the binary is actually published. That guarantee is added in rpkgs/bincraft#56 (success row written only after a confirmed S3 upload). This PR should land together with / after a bincraft release including that fix.
Reviewed-on: #91
A package's ABI sensitivity is the same across R minors, so for a non-sensitive
package the per-minor loop only ran `ensure_bincraft` (a ~14s bincraft/cranlike
install per minor) before build-one.R classified and skipped it. The primary
pass now writes a .r_minor_sensitive sentinel when it builds a sensitive
package, and the wrapper gates the whole per-minor loop (installs included) on
that file — non-sensitive rebuilds no longer touch other minors' libraries.
- Dockerfile: install bincraft into each R minor, run a primary pass plus a
sensitive-only pass under every other /opt/R/[0-9]* minor (deduped by minor),
probe/skip xvfb, set GIT_TERMINAL_PROMPT=0; extra-minor failures are non-fatal.
- build-one.R: add --sensitive-only mode, log + shallow-clone the ABI classify
step, and clear cranlike's stale ./PACKAGES.db before each index refresh
(workaround for the "table packages already exists" bug; pending cranlike fix).
## Summary
Adds a local `just rebuild` recipe to (re)build specific versions of a single package on a given OS/arch, dispatching to a remote buildx builder (the build runs there, not locally).
- `just rebuild <os> <tag> <arch> <package> <version>...` → `docker buildx build --builder <artemis|gaia> --platform linux/<arch> …` (amd64→artemis, arm64→gaia; names + `R_VERSION` env-overridable).
- `docker/build-one.Dockerfile` runs `build-one.R` as a secret-mounted `RUN`, built `--no-cache --output type=cacheonly` (pure side-effect: the S3 upload; no image kept).
- `local/build-one.R` auto-classifies each version via the ABI classifier (risky → per-minor slot `contrib/<x.y>/`, else generic), force-rebuilds + uploads + stores metadata, then refreshes the touched slot's `PACKAGES` index.
## Prerequisites
- buildx builders named `artemis` (amd64) and `gaia` (arm64) registered (`docker buildx create --name artemis ssh://…`).
- Exported secrets: `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY`, `PGPASS` (`GITHUB_PAT` optional).
- bincraft `v4.2.0` tag must exist (the build installs `@v4.2.0` and uses its classifier + per-minor index API).
Reviewed-on: #87
## Summary
Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered.
## What's in here
**Detection / precompute**
- `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests.
- `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions.
**Full build**
- `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset.
- `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor.
**Iterative build**
- All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added.
**Tooling / housekeeping**
- Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`.
- Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts.
- Design + implementation docs under `docs/superpowers/`.
## Requires before merge
- A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected.
Reviewed-on: #84
## Summary
`local/build-all.R` had a 75-name hardcoded `exclude <- c(...)` vector that had drifted from `local/excluded-packages.json` — `RcmdrPlugin.ROC` was in the R vector but missing from the JSON.
`.crow/weekly-rebuild-missing-*.yaml` already reads the JSON via `jsonlite::fromJSON(...)[["package"]]`. This brings `build-all.R` in line with that pattern.
Changes:
- **`local/excluded-packages.json`**: add the missing `RcmdrPlugin.ROC` entry (reason `"hang"`, matching siblings).
- **`local/build-all.R`**: replace the 16-line hardcoded vector with one `jsonlite::fromJSON(...)` call.
- **`.crow/build-all-versions-install-deps-{amd,arm}64.yaml`**: add `jsonlite` to the install-deps `pak::pak()` list so it's available in `/mnt/cache/R-pkgs` for the build step.
- **`.crow/build-all-versions-arm64.yaml`**: drop the dead base64-encoded `SKIP_PKGS` docs comment that nobody was passing as a `--var` anyway; replace with a one-line pointer to the JSON.
Reviewed-on: #79
## Summary
Both helpers still point at the old Hetzner storage:
- `Justfile` — all 3 recipes (`build-all`, `build-single`, `process-updates`) hit `hel1.your-objectstorage.com` / bucket `devxy-r-package-binaries-hel1` using `HETZNER_S3_*_K3S` env vars.
- `local/manual-package-index-update.R` — same endpoint + bucket, hardcoded into `s3fs::s3_dir_ls()` / `s3_file_delete()` calls.
Storage moved to Backblaze (`s3.eu-central-003.backblazeb2.com` / `devxy-rpkgs-binaries`) a while back, so running either of these today would write to the wrong bucket or fail outright.
No callers reference them in-tree, deleting outright. A one-off rebuild can just call `bincraft::upload_package_index()` directly with current Backblaze settings.
Reviewed-on: #77
s3fs uses reticulate/Python which significantly increases R process memory.
When pak tries to fork R to install package dependencies, the enlarged
process can't be duplicated within the container memory limit.
Now the install-deps step saves the S3 file listing as s3_cache.rds,
and the build step reads it with readRDS() — no s3fs loading needed.
- Load bincraft eagerly via library() to avoid lazy-load memory spike
- Do bulk S3 listing upfront and pass as s3_package_cache to avoid
per-package S3 calls that accumulate memory and trigger fork failures
- Consolidate amd64/arm64 exclude lists into single script
- Much easier to read and maintain than a YAML-embedded one-liner
The precompute script was constructing S3 paths as "redhat10" while
bincraft uses "rhel10" as the codename. This mismatch caused the
precompute to find no existing packages in S3, producing ~77K false
positive package versions that all get skipped at build time.
Now uses bincraft::set_codename() to ensure path alignment.
Also fixes nrow() vs length() in install-deps summary message.
packages-to-build.R now saves the full (Package, Version) data.table
so the build step can pass specific tags to build_binary_package(),
eliminating redundant per-package tag discovery and S3 checks.
Also reduces archive versions from 9 to 4 (+ 1 release = 5 total).
The S3 credentials, endpoint, and bucket name were still pointing to the old Hetzner storage after the Backblaze migration, causing silent build failures.
