Author	SHA1	Message	Date
pat-s	aa4c95457f	fix(rebuild): harden split workflow setup (#164) ... ## Motivation Weekly rebuild shards can all hit a transient CRAN DNS/index outage at once, and the dependent CDN purge always fails because it tries to clone over the checkout preserved from the re-index step. ## Changes - Retry `uvr add` resolution up to four times with bounded backoff. - Reuse the existing Crow workspace checkout in the CDN purge step. - Remove the purge step's unused Git package and repository token. ## Validation - `crow lint .crow/` - `shellcheck local/uvr-install.sh scripts/purge_cdn_zone.sh` - `git diff --check` Reviewed-on: #164	2026-08-13 13:38:28 +00:00
pat-s	4b7dc28cc8	feat(rebuild): shard the weekly rebuild and make each shard resumable (#163) ... ## Problem `weekly-rebuild-missing` runs one job per `<os>-<arch>` and walks that slot's list serially in a single `R -q -e` argument. That was cheap while every source fallback was skipped as "already built". Since bincraft #105/#106/#107 and #159 the gate works, and the lists are large: 8 917 source-served records on `amd64/alpine324`, 15 023 on `amd64/resolute`. Pipeline 10910 (`weekly_rebuild_missing:alpine-324-amd64`) ran for two days, reached `[8692/23885] cholera`, and was killed there. Two failures follow from that shape: - **No parallelism.** The work is embarrassingly parallel across packages; one job does all of it. - **No resumability and no clean stopping point.** The loop ends only by exhausting the list, so the only way to stop it is a kill. A restart re-walks from the first entry, paying a CRAN version resolution and an S3 `HEAD` per package before reaching new work. And a kill matches neither `success` nor `failure`, so the `Purge CDN cache` step never ran: the ~4 600 binaries 10910 did publish stayed hidden behind stale edge copies. ## What this changes **Three shards per slot.** Each of the 18 `OS`/`ARCH` rows gains `SPLIT_INTO`/`SPLIT_INDEX`, mirroring `build-all-versions.yaml`. Cron and manual routing are unchanged: both filters already match on `${OS}-${ARCH}`, so they now match all three shards of a slot. **`local/rebuild-missing.R`** replaces the ~1 500-character inline one-liner. The slice is interleaved rather than contiguous, because the list is alphabetical and cost clusters by name (`Rcpp*`, `Bioc*`, `rstan*`). **Resume by re-deriving state from the bucket.** One `s3_dir_info()` listing gives ETags for the slot; a package is outstanding iff its object's ETag equals CRAN's published `MD5sum`, i.e. it is still byte-identical to CRAN's source. That is `check_s3_root_package()` evaluated in bulk. No progress file, no volume, no DB cursor, and correct when a sibling shard or a `process-updates` run completes something concurrently. It reads ETags rather than the index's `Built` field the way `packages-to-build.R` does, because the index is no longer rewritten until the dependent pipeline runs and so cannot reflect the current run's progress. Unknown always means "already a binary", never "rebuild it": a multipart ETag, an unreadable CRAN index or an empty listing can never mass-schedule work. **A 20 h wall-clock budget** per shard. It exits 0, so the re-index and purge always fire and the remainder is picked up next run with no bookkeeping. **`.crow/weekly-rebuild-reindex.yaml`** takes over re-indexing and the purge, with `depends_on: [weekly-rebuild-missing]` and `runs_on: [success, failure]`. Three shards writing one slot's `PACKAGES` concurrently would race: `update_PACKAGES()` lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work. ## Verification `crow lint .crow/` passes on all 11 pipelines. `prek run` passes. 19 assertions in `local/tests/test-rebuild-missing.R`, 0 failures, covering the partition (disjoint, covering, deterministic, short lists, out-of-range index) and the outstanding filter (source ETag kept, binary ETag dropped, absent object kept, multipart and missing-from-CRAN treated as built). One of those tests caught a real bug before it shipped: an empty ETag table indexed to zero length rather than to `NA`, which recycled the result away and reported "nothing to build" — the dangerous direction. Fixed with an explicit `lookup()`. The filter run against the live `amd64/alpine324` index, using its `MD5sum` column as the ETag (established to match the objects): ``` index packages: 24343 outstanding (filter): 8950 no Built stamp: 8917 filter vs no-Built agreement: 8917 of 8917 outstanding but stamped Built: 33 (version drift vs CRAN) shard sizes: 2984/2983/2983 (sum 8950, unique 8950) ``` It reproduces the source-served set exactly. The extra 33 are packages whose slot version differs from CRAN's current one, so no object exists at the CRAN version key: correctly outstanding. ## Notes for review - The 20 h budget is a chosen default, exposed as `REBUILD_BUDGET_HOURS` in the pipeline. - `depends_on` is file-level, not row-level, so on a full cron run no slot is re-indexed until the slowest of all 54 jobs finishes. The budget bounds that at roughly a day. - An explicit cancel still skips the re-index. Recovery is to trigger `weekly-rebuild-reindex` on its own. - The purge runs on every re-index row rather than one designated slot: a cron fires only its own slot's row, so gating on a named slot would leave every other slot unpurged. - Out of scope: `build-all-versions` still cannot rebuild source fallbacks, because `local/build-all.R:113-122` drops every version with any `single_builds` row, which is precisely the source-fallback set. Design: `specs/2026-08-12-shard-weekly-rebuild-design.md` Reviewed-on: #163	2026-08-12 08:30:29 +00:00
automation-bot	caf3276fb9	chore(deps): update dependency nbafrank/uvr to v0.4.6	2026-08-11 00:31:56 +00:00
pat-s	6372f0f928	fix(ci): refresh the apt index before uvr installs system dependencies (#161) ... Unblocks the ubuntu 26.04 builds without waiting for an image rebuild. ## What broke ``` i Installing R package dependencies ! Error in installing dependencies for package huge with tag 2.0.1: uvr sync --install-system-deps --library /mnt/cache/R-pkgs-4.4 failed (exit 1) ! WARN Missing system dependencies for 1 package(s) igraph needs: libglpk-dev > Running: apt-get install -y libglpk-dev E: Unable to locate package libglpk-dev ``` Neither the package name nor the sysreq mapping is wrong. `libglpk-dev` is `5.0-2build1` in `universe`, which `ubuntu:resolute` enables by default. What is missing is the apt index: uvr v0.4.5 runs `apt-get install` with no `apt-get update` before it, and every ubuntu build image ends its apt layers with `rm -rf /var/lib/apt/lists/*`, so the shipped image has no index at all. Not specific to 26.04 — jammy and noble fail identically for any sysreq the image's preinstall line doesn't already cover. 26.04 surfaced it first, through `igraph` and everything depending on it (`huge` was the first to fail). ## Fix `apt-get update` before the sync in `local/uvr-install.sh`. That one site covers every pipeline: `build-all-versions`, `build-all-versions-install-deps`, `weekly-rebuild-missing`, `weekly-audit-missing`, `trial-build-registry`, `auto-apply-patches` and `weekly-patch-proposals` all reach it, either directly or through `install-bincraft.R`. The index it populates persists for the rest of the step, so the bincraft-driven `uvr sync` calls that follow are covered too. Guarded on `apt-get` and non-fatal: `apk add` fetches its index implicitly and dnf refreshes expired metadata on its own, so alpine and redhat are unaffected and skip it. ## Why here as well as in the image The real fix is upstream `e491b2e`, which refreshes the index before the auto-install. It landed one day after v0.4.5 was tagged, so no release carries it ([nbafrank/uvr#250](https://github.com/nbafrank/uvr/issues/250)); [build-env-images#23](https://codefloe.com/rpkgs/build-env-images/pulls/23) pins the ubuntu image to a commit that has it. That only reaches CI after an image rebuild is triggered. This change takes effect on the next pipeline run. It also stays useful afterwards: a baked index goes stale within weeks, and a stale index turns the same install into a 404 on the `.deb`. ## Verified In `ubuntu:resolute`, with the image state reproduced exactly (R installed, then `rm -rf /var/lib/apt/lists/*` to empty the index again): | Check | Result | |---|---| | `apt-get install -y libglpk-dev`, empty index | `E: Unable to locate package libglpk-dev` — the reported failure, reproduced | | Same container after `apt-get update` | `Candidate: 5.0-2build1`, `Components: main universe restricted multiverse` | | `igraph` + `UVR_INSTALL_SYSREQS=1 uvr sync`, empty index, uvr with the refresh | `Setting up libglpk-dev:amd64 (5.0-2build1)`, `System dependencies installed.`, `Installed 11 package(s)` | | `sh -n` and `shellcheck local/uvr-install.sh` | pass | ## Reverting Delete the guarded block. It is marked TEMPORARY with the upstream reference, and can go once the images ship a uvr above v0.4.5. Reviewed-on: #161	2026-08-10 06:30:25 +00:00
pat-s	a126d74cd3	fix(build): keep source fallbacks out of the S3 package cache (#159) ... ## Problem This is the gap flagged in rpkgs/bincraft#106. `build_binary_package()` has a fast path that compares against `s3_package_cache` instead of querying S3 per package, and that cache is produced here: ```r s3_pkgs <- s3fs::s3_dir_ls(".../latest/src/contrib", recurse = TRUE) saveRDS(basename(s3_pkgs), "/mnt/cache/packages/s3_cache.rds") ``` A raw bucket listing cannot tell a binary from a package whose build failed and was published as its CRAN source — the two occupy the same key. So every source fallback reads as "already built" and is skipped for good. That is how `alpine324` accumulated ~13.5k of them. The same listing feeds `s3_dt`, which is subtracted from the build list at line 194 (`pkgs <- pkgs_no_error[!s3_dt]`). That one matters more: it excludes the very packages that need building, before `build_binary_package()` is even called. ## What this changes Drops from the listing every object the slot's own index reports as served from source. bincraft leaves the `Built` stamp off exactly those records (rpkgs/bincraft#105), so the index already carries the answer and no credentials, downloads or extra API calls are needed. Both consumers are fixed: the saved cache and `s3_dt`. The cache stays a plain filename vector, so the build container still needs no `s3fs`/reticulate — that was the point of saving it in the first place. Two deliberately conservative edges: - archived objects have no index record, so they are kept. Unknown means binary, never "rebuild it". - if the index cannot be read, the full listing is kept and a warning is printed, so a CDN blip cannot mass-schedule a rebuild. ## Verification The script parses, and the new block run against the live indices: ``` amd64/alpine324: index=24235 source-served=13542 e.g. AATtools_0.0.3.tar.gz, ABCDscores_7.0.0.tar.gz amd64/noble: index=24681 source-served=0 ``` `alpine324` is re-indexed by bincraft 5.1.1, so 13 542 objects drop out and those packages become buildable. `noble` has not been re-indexed yet, so every record still carries `Built`, nothing is dropped, and its behaviour is exactly what it is today — the safe failure mode this relies on. The new log line makes it visible per run: ``` S3 cache: N objects, M served as CRAN source, K usable binaries ``` ## Sequencing Needs rpkgs/bincraft#106 (and a release) before a rebuild actually builds: this fixes the bulk build path's list, #106 fixes the per-package pre-build skip. Reviewed-on: #159	2026-08-09 16:27:51 +00:00
pat-s	0703969088	fix(audit): count a source fallback as a missing binary (#157) ... ## Problem The audit's definition of "missing" was `missing_dt <- cran_dt[!s3_dt]`, where `s3_dt` came from listing the bucket. But an object being present does not mean a binary was built: when a build fails, bincraft publishes the CRAN source tarball in its place so the package stays installable, and a listing cannot tell the two apart. So every failed build looked done. On `amd64/alpine324` that is 13 547 of 24 134 comparable objects (56%) — all CRAN sources from the June 2026 bootstrap, none ever reported here, so `weekly-rebuild-missing` never retried any of them. `amd64/noble` sits at 4.6% for comparison. ## What this changes Reads the slot's own `PACKAGES` index instead of listing the bucket, and counts only records carrying a `Built` stamp. bincraft writes that stamp only for what it actually built (rpkgs/bincraft#105), so the index answers the question a listing cannot. Side effects of reading the index rather than the bucket: - the audit reports how many records the index holds and how many are served as source, so the gap is visible in the log - `s3fs` is no longer used, so the audit needs no B2 credentials - an object present but absent from the index is now counted as missing, which is correct: clients only see the index ## Sequencing This is inert until the matching bincraft change ships and a slot is re-indexed. A slot last indexed by an older bincraft carries `Built` on every record, including the fallbacks, so the audit reports exactly what it does today. Verified against the live indices: ``` amd64/alpine324: records=24152 built=24152 source-served=0 amd64/noble: records=24681 built=24681 source-served=0 ``` After a re-index those `source-served` counts become the real ones, and the packages behind them start appearing in issue #63 and getting rebuilt. Reviewed-on: #157	2026-08-09 10:37:41 +00:00
pat-s	0da29ab3f6	fix(index): add a slot repair for a broken Built stamp and retire alpine 3.21 (#150) ... ## Problem `arm64/alpine321` and `arm64/alpine322` advertise a broken stamp: ``` arm64/alpine321 :: 22930 Built: R 4.4.0; NA; 2026-07-31 13:35:52 UTC; unix arm64/alpine322 :: 24696 Built: R 4.5.0; NA; 2026-07-31 13:51:54 UTC; unix ``` The per-minor sub-slots (`contrib/4.4`, `4.5`, `4.6`) are affected too. All 18 other slots are correct. uvr matches the stamp's platform triple plus R minor to pick binary over source, so nothing matches `NA` and both slots silently serve as source-only, which is exactly the regression bincraft#85 added the stamp to prevent. `install.packages()` is unaffected, since it reads `Built:` from each tarball's own `DESCRIPTION`. The tarballs are fine (`arm64/alpine322/.../dress.graph_0.8.3.tar.gz` carries `aarch64-unknown-linux-musl`), and so is the R that built them (`r-4.5.0_1_aarch64.apk` ships `R_PLATFORM='aarch64-unknown-linux-musl'`). Only the index is wrong. bincraft#96 stops a stamp like this being written again, but it cannot repair what is already there. ## Why not just re-run the index update `upload_package_index()` reuses the slot's remote `PACKAGES.db`, and cranlike's `update_db()` only reparses files whose md5 changed, so entries already in the database keep the stamp they were written with. Dropping `PACKAGES.db` to force a full reparse does work, and it is what bincraft#85's rollout note suggested, but for an S3 repo cranlike reads each package's metadata from the CRAN *source* mirror on GitHub. A 25k-entry slot is then 25k requests to raw.githubusercontent.com, with a real risk of being rate-limited part-way through and leaving the slot half-written. Only the `Built` column is wrong, so this corrects it in place instead: patch the column in `PACKAGES.db`, put the database back, and let `upload_package_index()` re-emit `PACKAGES*` from it. `update_db()` always rewrites the index files even when nothing was reparsed, so no tarball is re-read and nothing is fetched from GitHub. ## Change - `local/repair-built-stamp.R` — repairs the generic slot plus every per-minor sub-slot. Dry-run by default; `--apply` writes. The replacement comes from `bincraft::built_stamp()` under the R running the script, so it is exactly what a healthy run would have written, and bincraft#96's guard makes a broken build image fail rather than write a second bad stamp. - `.crow/repair-built-stamp.yaml` — manual pipeline, routed by `target_arch` to the matching agent group and platform image, with `dry_run` defaulting to `true`. - `.crow/archive-missed-packages.yaml` — drop the two `alpine321` matrix entries. Alpine 3.21 is EOL: the website advertises only 3.23/3.24 and `process-updates.yaml` already dropped it, so that slot is retired rather than repaired. ## Verification `crow lint .crow/` passes on all nine pipelines. `air format` and `jarl check` are clean; the script parses, and the `/opt/R` minor-version derivation was checked against `4.4.3 / 4.5.3 / 4.6.0 / current` → `4.4 4.5 4.6`. The repair itself is unrun by design — it needs B2 credentials and an arm64 agent. ## Rollout 1. Cut a bincraft release so `local/install-bincraft.R` picks up #96 (it resolves the latest `vX.Y.Z` tag, and #96 is only on `main`). 2. Run this pipeline with `target_arch=arm64`, `OS=alpine`, `OS_VERSION=3.22`, `R_VERSION=4.5.3`, `dry_run=true` and check the reported counts. 3. Re-run with `dry_run=false`. 4. Confirm: `curl -sS https://cran.devxy.io/arm64/alpine322/latest/src/contrib/PACKAGES | grep '^Built:' | sort | uniq -c` `arm64/alpine321` is deliberately left alone. Reviewed-on: #150	2026-08-07 09:15:04 +00:00
automation-bot	e02be7518f	chore(deps): update dependency nbafrank/uvr to v0.4.5	2026-08-04 02:32:01 +00:00
pat-s	ee15c50f53	refactor: migrate package installation from pak to uvr (#147) ... `bincraft` dropped pak in favour of uvr (5.0.x, "Dependencies and their system requirements are now installed with `uvr` instead of pak during `build_binary_package()`"), so the pipelines, helper scripts and images in this repo move with it. ## Approach uvr is project-scoped in a way pak is not: `uvr add` refuses to run outside a project and always installs into `.uvr/library/`, and only `uvr sync` honours `--library`. So there is no one-line `pak::pak(...)` equivalent. `local/uvr-install.sh` encapsulates the dance — bootstrap a pinned uvr, mint a throwaway project under `TMPDIR`, `uvr add --no-install`, then `uvr sync --library <target>`. Keeping the project outside the checkout also keeps `uvr init`'s `.Rprofile` from hijacking `.libPaths()` for every other R call in the pipeline. This matches what bincraft itself does (`uvr sync --install-system-deps --library <lib>`), and bincraft requires `uvr` on `PATH`, which the bootstrap provides: every pipeline that calls `bincraft::` runs `install-bincraft.R` (and therefore the bootstrap) first. ## Changes | File | Change | | --- | --- | | `local/uvr-install.sh` | **New.** The single replacement for `pak::pak(...)`. Bootstraps uvr `v0.4.4`, resolves the R interpreter from `UVR_R_BIN`/`R_VERSION`/`PATH`, pins the manifest to that R's exact version, and syncs into `UVR_TARGET_LIB`/`R_LIBS_USER`. | | `local/install-bincraft.R` | Installs `forgejo::codefloe.com/rpkgs/bincraft@<tag>` instead of a `git::` URL; keeps the `git ls-remote` tag resolution. Exports `UVR_R_BIN`/`UVR_TARGET_LIB` from `R.home()`/`.libPaths()[1]` so the per-R-minor passes target their own R and library. | | `.crow/auto-apply-patches.yaml`, `.crow/weekly-patch-proposals.yaml`, `.crow/weekly-audit-missing.yaml`, `.crow/weekly-rebuild-missing.yaml`, `.crow/build-all-versions-install-deps.yaml` | `pak::pak(...)` → `UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh ...`. The explicit `UVR_R_BIN` matters in `build-all-versions-install-deps.yaml`, which has no `R_VERSION` in its step environment. | | `.crow/build-all-versions.yaml`, `.crow/process-updates.yaml`, `.crow/weekly-rebuild-missing.yaml` | `R_PKG_CACHE_DIR` → `UVR_CACHE_DIR` + `UVR_PACKAGES_DIR` on the same `/mnt/cache` volume, preserving the amd64-off/arm64-on split. Drops the `rm -rf .../pkgcache/_metadata/...` cleanup. | | `local/r-minor-helpers.R`, `local/build-all.R`, `local/tests/test-trim-pkgcache.R` | Removes `trim_pkgcache_metadata()`, its every-25-packages call and its tests. uvr's cache does not mint a fresh ~70 MB snapshot per `PACKAGES` change. | | `.crow/build-all-versions-install-deps.yaml` | Drops `pak::sysreqs_db_update()`; uvr resolves sysreqs from its vendored `r-system-requirements` rules via `--install-system-deps`. | | `docker/Containerfile-shiny-app` | Bootstraps uvr and drives both dependency installs through one uvr project with `UVR_LIBRARY` pointed at the image's R library. | | `docker/build-one.Dockerfile` | Ships `uvr-install.sh` at `/work/local/` so `install-bincraft.R` finds it. | | `docker/reprex/alpine.sh` | Replaces `pak::local_install_deps()` with DESCRIPTION parsing + `uvr add`. | | `local/test-package-loading.R` | Installs via the helper instead of `pak::pkg_install()`. | | `README.md` | Documents uvr for sysreq inference, archived-version installs and cache clearing. | | `renovate.json` | Tracks the `UVR_PIN` in `uvr-install.sh` via `github-releases`. | ## Behaviour notes - **`weekly-audit-missing` still takes bincraft from the default branch**, not the latest release tag, matching what the `git::` pak call did. Called out in a comment rather than silently changed. - **The uvr pin is repo-wide.** bincraft resolves `uvr` from `PATH` and pins no version of its own, so `UVR_PIN` in `uvr-install.sh` governs the whole pipeline. - **Persistent caches now also benefit bincraft**, which reads `UVR_CACHE_DIR`/`UVR_PACKAGES_DIR` from the inherited pipeline environment. - **`uvr sync` will not prune the shared library.** Pruning is disabled whenever `--library` is passed (`do_prune = prune && library_override.is_none()`), so `/mnt/cache/R-pkgs` keeps bincraft and its dependencies. The wipe-on-ABI-mismatch path is *not* similarly guarded, which is why the helper pins the manifest to the active R's exact version. - **`plans/` and `specs/` are untouched** — they are dated records of decisions made in June 2026 and describe bincraft's then-pak-based internals; rewriting them would misstate history. ## Verification `shellcheck`, all pre-commit hooks (on this commit's file range) and the `local/tests/` suite (100 assertions) pass. Not yet exercised in CI: the build-env images do not ship `uvr`, so the per-step `curl install.sh` bootstrap is untested against a real image. Worth a manual `build-all-versions-install-deps` run before merging. Reviewed-on: #147	2026-07-31 12:19:26 +00:00
pat-s	fe2289af56	fix(patches): force RcppParallel to use its bundled oneTBB (#146) ... ## Why `RcppParallel` binaries built on ubuntu 22.04 still record `NEEDED libtbb.so.12` and cannot `dyn.load` without the distro TBB, although #145 landed the link-order patch and build-env-images dropped `TBB_INC`/`TBB_LIB` the same morning. The link-order fix does not survive a build image that exports those variables, and the published images still do: `.crow/images.yaml` in build-env-images triggers on `cron`/`manual` only, so every image in the registry is still the 2026-07-21 build. With `TBB_LIB` set, `configure.R` never reaches the bundled-oneTBB branch at all, `install.libs.R` fills `RcppParallel/lib` with absolute symlinks into the image's library dir, and the plain `-L` from #145 just points at the system TBB instead of the bundled one. Nothing on the pipeline side can undo this: R reads `~/.Renviron` **after** the process environment, so neither a Crow `environment:` block nor the registry `env` field wins. ``` # the image, not the Containerfile docker run --rm reg.devxy.io/rpkgs/build-env-ubuntu:jammy sh -lc 'grep TBB ~/.Renviron' TBB_INC=/usr/include/oneapi TBB_LIB=/usr/lib/x86_64-linux-gnu ``` Rather than couple correctness to image freshness for a third round-trip, the patch now forces the bundled TBB itself. ## Changes - `local/patches/RcppParallel/bundled-tbb-link-order.patch` -> `local/patches/RcppParallel/force-bundled-tbb.patch`. Keeps the `-Wl,-L` -> `-L` link-order hunk (the bundled branch has the same problem when a distro libtbb is installed) and adds an opt-out, `BINCRAFT_ALLOW_SYSTEM_TBB=TRUE`, for the ambient `TBB_ROOT`/`TBB_LIB`/`TBB_INC`. - All four read sites have to move together; patching only `configure.R` builds the bundled TBB and then **segfaults R on the post-install load test**: - `tools/config/configure.R`: branch selection. - `src/install.libs.R`: `R CMD INSTALL` runs it outside the `tbb` rule in `src/Makevars` that passes the configured values, so at install time it still sees the image environment. - `R/tbb.R` `tbbLibraryPath()`: run time. `.onLoad()` would `dyn.load` the system TBB into the global scope next to the bundled one that `RcppParallel.so` needs - two copies of the same symbols, hence the segfault. - `R/tbb.R` `tbbCxxFlags()` / `tbbLdFlags()`: otherwise dependents such as `rstan` get the system TBB handed back to them. - `local/patches/registry.json`: new patch path and a `reason` describing both failure modes. ## Verification Built through `bincraft:::prepare_patched_repo()` on the **current, unfixed** images (all three still export `TBB_INC`/`TBB_LIB`), with bincraft v5.0.1, i.e. what CI deploys: | image | `NEEDED` | rpath | `RcppParallel/lib` | load, all system `libtbb*` moved away | | --- | --- | --- | --- | --- | | `build-env-ubuntu:jammy` | `libtbb.so` | `$ORIGIN/../lib` | real `libtbb.so.2`, 5.0 MB | OK, 12 threads | | `build-env-redhat:9` | `libtbb.so` | `$ORIGIN/../lib` | real `libtbb.so.2`, 5.1 MB | OK, 12 threads | | `build-env-alpine:3.24` | `libtbb.so` | `$ORIGIN/../lib` | real `libtbb.so.2`, 4.7 MB | OK, 12 threads | For contrast, the same jammy build without this change: ``` NEEDED libtbb.so.12 RUNPATH /usr/lib/x86_64-linux-gnu:$ORIGIN/../lib lib/ libtbb.so.12.5 -> /usr/lib/x86_64-linux-gnu/libtbb.so.12.5 (dangling off the image) load libtbb.so.12: cannot open shared object file: No such file or directory ``` With `TBB_LIB` still exported, `RcppParallel::tbbLibraryPath()` resolves to the package's own `lib`, and `CxxFlags()` emits the package's own `include` plus `-DTBB_INTERFACE_NEW` - not `/usr/include/oneapi`. ## Behaviour change Published `RcppParallel` binaries carry their own oneTBB on every platform, whichever image version CI pulls. The patch content changed, so the patched-binary cache key changes with it and no stale entry is reused. The build-env-images fix is still worth rolling out (those images also carry the broken el8 `uvr` and uvr 0.4.1), but RcppParallel no longer waits on it. ## Follow-up, not in this PR The binaries already in B2 are the broken ones; they need a rebuild, and a Bunny `/purge` does not evict Perma-Cache. Reviewed-on: #146	2026-07-31 09:48:41 +00:00
pat-s	98371fb9c5	fix(patches): replace the RcppParallel system-TBB patch with a link-order fix (#145) ... ## Why Every build logs: ``` ! Patch for RcppParallel 6.2.0 did not apply cleanly; skipping patched build. ``` RcppParallel 6.x rewrote its build system. `src/Makevars.in` no longer contains the `USE_TBB=Linux` block that `system-tbb.patch` edited (it is now a short `@VAR@` template driven by `tools/config/configure.R`), so the patch can never apply again. The workaround it implemented is obsolete too. 6.x bundles **oneTBB 2022** and builds it with cmake, which works on musl and with g++ 8-15, so the 5.x reason for linking a system TBB is gone. Linking one is now harmful: `install.libs.R` symlinks the system libraries into `RcppParallel/lib`, so the published binary depends on a TBB the consumer does not have — the same failure mode as `fs` and libuv. What is still broken upstream is the **link order**. `configure.R` names the TBB directory with `-Wl,-L`, and gcc expands its own search dirs into `-L` options ahead of anything forwarded verbatim with `-Wl,`: ``` $ gcc -v -o t t.c -Wl,-L/usr/local/lib64 -ltbb -L/usr/lib/gcc/x86_64-redhat-linux/8 -L/lib/../lib64 -L/usr/lib/../lib64 ... -L/usr/local/lib64 ``` So on any build host with a distro TBB installed, `-ltbb` resolves to that library and `RcppParallel.so` records *its* SONAME (`libtbb.so.12`, or `libtbb.so.2` for the classic Intel TBB on el8/el9) instead of the bundled `libtbb.so`. ## Changes - Replace `RcppParallel/system-tbb.patch` with `RcppParallel/bundled-tbb-link-order.patch`, which changes the three `-Wl,-L` occurrences in `tools/config/configure.R` to plain `-L`. - Scope the RcppParallel entry to `>=6.0.0` and widen `platforms` to `*`. - Drop the `rstan` entry: its `-DTBB_INTERFACE_NEW` is already emitted by `RcppParallel::CxxFlags()` once the bundled oneTBB is used, and its `-I/usr/local/include` was an el8/el9 path applied on every platform. - `.pre-commit-config.yaml`: exclude `local/patches/*.patch` from `trailing-whitespace`, `end-of-file-fixer` and `editorconfig-checker`. They rewrite blank context lines (` ` -> ``) in every diff in the registry; `git apply` happens to tolerate it today, but a patch with meaningful trailing whitespace would be silently corrupted. The excludes are per-hook so `validate patch registry` still runs. - README: the RcppParallel example described the 5.x problem. ## Verification Built in the published images: | image | NEEDED | RPATH | loads with system libtbb removed | | --- | --- | --- | --- | | `build-env-alpine:3.24` | `libtbb.so` | `$ORIGIN/../lib` | yes | | `build-env-redhat:8` | `libtbb.so`, `libtbbmalloc.so` | `$ORIGIN/../lib` | yes | | `build-env-ubuntu:noble` | `libtbb.so` | `$ORIGIN/../lib` | yes | Without the patch the same builds record `libtbb.so.12` (alpine, ubuntu) or `libtbb.so.2` (el8, the classic 2018 TBB) and fall back to the system library at load time. rstan 2.32.7 compiles and loads against the patched RcppParallel on ubuntu noble with no makevars override and with the system libtbb moved away. On el8 it also compiles; loading it there is blocked by an unrelated image bug (see below). ## Behaviour change Requires the paired build-env-images PR (drops `TBB_INC`/`TBB_LIB`) and an image rebuild — with those env vars set, configure still takes the system-TBB branch. After that, RcppParallel binaries ship their own oneTBB and are self-contained. ## Also found, not fixed here - The `uvr lock failed ... GLIBC_2.29 not found` errors in the same log are the `-gnu` uvr artifact on el8's glibc 2.28; fixed in build-env-images. - `build-env-redhat:8` R 4.5.3/4.6.0 cannot load `stats` (`libRlapack.so: undefined symbol: dgemmtr_`): the el8 R RPM symlinks `libRblas.so` to openblas 0.3.15, which predates that symbol. R 4.4.3 and el9 are fine. Belongs in the R RPM build. Reviewed-on: #145	2026-07-31 06:41:46 +00:00
pat-s	8d0c4b7ccd	fix(patches): widen RcppParallel system-tbb scope to the whole redhat family (#139) ... Extends #137. el8/el9 were excluded because they ship only classic Intel TBB. build-env-images now vendors oneTBB 2021 into `/usr/local` on el8/el9, so every platform has oneTBB 2021+. Widen the scope `redhat-10` → `redhat` (family token covers el8/el9/el10) and fix the reason text (no longer 'left on bundled'). **Requires the el8/el9 images rebuilt with the vendored oneTBB (build-env-images PR).** Verified: patched RcppParallel builds + links the vendored oneTBB on both el8 (g++ 8.5) and el9 (g++ 11.5). Reviewed-on: #139	2026-07-28 06:57:08 +00:00
pat-s	71f3f0c601	fix(audit): replace arch subsection in place instead of appending duplicates (#138) ... ## Problem A fresh `weekly-audit-missing` run followed by `weekly-rebuild-missing` reported that almost every package "already exists in the remote bucket", even though the audit had just flagged them as missing. Root cause: the audit's Forgejo-issue update matched the existing `### <arch>` subsection by its **bare** header (`### arm64`) while writing headers with a ` (N missing, M to rebuild)` suffix. The equality test never matched, so every run **appended** a new block instead of replacing the old one. Issue #63 had accumulated 38 arch subsections under `## alpine-323` where 2 are expected (75 total across the alpine platforms; body ~77k lines). `fetch-rebuild-packages-from-issue.R` reads the **first** matching block, which was the oldest snapshot. So the rebuild kept re-checking a months-old list (180 packages, mostly already built), while the genuinely-missing packages, ~1921 for alpine-323/arm64 in the freshest block, were never fed to the rebuild and the backlog grew silently. ## Changes - **`local/weekly-missing-binaries-audit.R`**: match arch subsections by prefix (`^### <arch>( |$)`) and remove **all** blocks for that arch before writing one fresh block. Accumulation now self-heals on every run. - **`local/dedupe-audit-issue.R`** (new): one-off cleanup that collapses each `## platform` section to the freshest block per arch across the three OS-family issues. Supports `DRY_RUN=1`. - **`local/fetch-rebuild-packages-from-issue.R`**: prefer the audit's freshly-written RDS (overwritten each run, immune to issue-body drift), falling back to issue parsing when absent. ## Validation Simulated the dedupe logic against the live #63 body: **75 → 8** arch subsections, body 77k → 28k lines, and the kept alpine-323/arm64 block correctly resolves to `GARCH.X (3.0)` (the stale first block held `2.0`). ## Follow-ups (not in this PR) - `alpine-321` is audited but has no row in the rebuild matrix (7,230 missing, never rebuilt). - An `alpine-324` section exists in the issue but is in neither matrix. Reviewed-on: #138	2026-07-21 08:51:23 +00:00
pat-s	3cc323d99e	fix(patches): widen RcppParallel system-tbb scope to all oneTBB-2021 platforms (#137) ... ## What Widen the RcppParallel `system-tbb` registry patch scope from `[alpine, ubuntu-2604]` to `[alpine, ubuntu, redhat-10]`. ## Why #135 fixed the TBB cluster on alpine + ubuntu-2604 only. A per-image audit showed the problem is broader: RcppParallel's bundled Intel TBB build fails on **every** build image with a modern toolchain (alpine/musl, ubuntu jammy/noble/resolute g++ 11/13/15, redhat-10 GCC 14), and only redhat-8/9 (older toolchains + pre-oneTBB system TBB) tolerate the bundled build. The system-TBB fix applies wherever the image ships **oneTBB 2021+** (has the `tbb::detail::r1` namespace rstan needs): | platform | system TBB | in scope? | |---|---|---| | alpine (all) | oneTBB 2022 | ✅ | | ubuntu 2204/2404/2604 | oneTBB 2021.5 / 2021.11 / 2022 | ✅ (via `ubuntu` family token) | | redhat-10 | oneTBB 2021.11 | ✅ | | redhat-8 | TBB 2018 (classic) | ❌ pre-oneTBB → bundled | | redhat-9 | TBB 2020.3 (classic) | ❌ pre-oneTBB, bundled works → bundled | ## Requires The matching build-env-images env (`TBB_INC`/`TBB_LIB`) extended to all ubuntu + el10: build-env-images #13. Ship together. ## Verified RcppParallel + rstan build and link (no `symbol not found`) on alpine, redhat-10, ubuntu noble and ubuntu resolute; RcppParallel builds on ubuntu jammy. Reviewed-on: #137	2026-07-20 20:59:47 +00:00
pat-s	6df945d147	fix(patches): build RcppParallel against system oneTBB instead of disabling TBB (#135) ... ## What Replace RcppParallel's `disable-tbb` registry patch with `system-tbb`: build RcppParallel against the system oneTBB instead of stripping TBB entirely. ## Why `disable-tbb` skipped the bundled Intel TBB build by forcing the TinyThread backend, which also removed RcppParallel's TBB linkage. That broke every dependent that links TBB through `RcppParallelLibs()` -- `rstan` and the whole Stan cluster -- with `symbol not found: tbb::detail::r1::observe`. This is the root cause behind the large "blocked on RcppParallel" / rstan clusters (issues #115, #120): the packages were not individually broken, they were all waiting on one TBB-linkage regression. ## Change `system-tbb.patch` leaves `USE_TBB` unset (so the bundled build is still skipped on musl and g++ 15) but keeps the TBB backend and links the system oneTBB now shipped in the build-env images: ``` PKG_CXXFLAGS += -DRCPP_PARALLEL_USE_TBB=1 -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -DTBB_INTERFACE_NEW PKG_LIBS += -ltbb -ltbbmalloc ``` Scoped to `alpine` + `ubuntu-2604`, the only platforms where the bundled build fails; redhat and older ubuntus keep the bundled TBB. ## Requires The companion image change that exports `TBB_INC`/`TBB_LIB` so `RcppParallelLibs()` hands the system-TBB flags to dependents: build-env-images PR #12. Both must ship together. ## Verified In `build-env-alpine:3.24` with the new images: - RcppParallel builds against oneTBB 2022, no ABI errors, patch applies cleanly to the target clone. - `RcppParallelLibs()` returns `-L/usr/lib -Wl,-rpath,/usr/lib -ltbb -ltbbmalloc`. - rstan links (`-ltbb -ltbbmalloc`) and loads with no missing symbol -- `* DONE (rstan)`, trial build exit 0. Reviewed-on: #135	2026-07-20 09:56:21 +00:00
pat-s	b4b7d53a83	fix(local): make trial-build-patch.R detect non-throwing build failures (#134) ... `local/trial-build-patch.R` -- the single-package acceptance gate the proposer prints for humans to run (`PGPASS=... Rscript local/trial-build-patch.R <pkg>`) -- reported **`Trial build OK`** for rstan even though rstan.so failed to load (`symbol not found: tbb::detail::r1::observe`). Same false-green as the registry gate before #130: `build_binary_package()` catches failures internally and returns `"error"` rather than throwing, so a `tryCatch` that only treats a thrown exception as failure passes a broken build. Now inspects the return value (mirrors trial-build-registry.R). Found while verifying the rstan/oneTBB unblock. Reviewed-on: #134	2026-07-18 08:37:04 +00:00
pat-s	6c03f278ec	feat(local): aggregate blocked-on-dependency reporting by dependency (#131) ... ## Why With cascade detection (#128) live, the latest `auto-apply-patches` run did exactly the right thing — **proposed nothing** (`No auto-proposable candidates`) because every failure is a dependency cascade, and it surfaced the ~30 root-cause dependencies to fix. But the "Blocked on a dependency" list printed **one line per fingerprint group**, so the same dependency repeated (rstan ×4, lpsymphony ×4, salso ×2, BH ×2, GO.db ×2, RcppCWB ×2, …), burying the priority. ## What Aggregate blocked packages across all groups **by the dependency they wait on**: - Expose `blocked_map` (package → dependency) from `build_triage_report()`. - Add `blocked_by_dependency()` — dedupes dependents (a package in two groups counts once) and ranks dependencies by how many distinct dependents they block. - Proposer and tracker (log + issue) now print one line per dependency, sorted by impact. Replaces the per-group `blocked_summary`. ## Result (same data, aggregated) ``` Blocked on a dependency (3 dependencies block 6 dependents; fix the dependency, not each dependent): RcppParallel 3 dependent(s) rstan 2 dependent(s) sf 1 dependent(s) ``` So the real run becomes a crisp, ranked worklist: RcppParallel (894), sf (128), rstan (~96), Rfast (33), clarabel/DescTools (26), Rglpk (22), xgboost (18), … ## Verified New test covers cross-group aggregation, dedup (a dependent in two groups counted once), the example cap, and ranking. 112 tests pass; hooks pass. Reviewed-on: #131	2026-07-16 21:24:03 +00:00
pat-s	118a92889f	fix(local): make the trial-build gate detect non-throwing build failures (#130) ... ## Critical: the gate was false-green The latest run printed **`3/3 passed on alpine-324`** while all three builds actually **failed** (their `rstan` dependency won't compile). A false-green gate would let broken registry entries merge — worse than no gate. Root cause: `bincraft::build_binary_package()` catches build failures internally and **returns `"error"`** for the failed tag rather than throwing. The gate's `tryCatch` only treated a *thrown* exception as failure, so every non-throwing failure looked like a pass. Fix: inspect the return value. A tag passes only if the flattened result is non-empty and contains no `"error"` sentinel; a thrown error still counts as failure. Verified the verdict against `error`/`skipped`/`TRUE`/`list(success=TRUE)`/`NULL`/mixed inputs. With this, the current rstan-blocked entries will correctly show **0/3 (red)** — which is the right answer until rstan builds. Reviewed-on: #130	2026-07-16 08:24:03 +00:00
pat-s	f9d399fac0	feat(local): detect dependency-cascade failures generally, not just RcppParallel (#128) ... ## Why (from the #127 trial-build gate) The gate did its job: 0/3 passed, merge blocked. The log showed *why* -- BFpack, BayesERtools, GMLTM all fail while building their shared dependency **`rstan`**, not in their own code: ``` Failed to build source package rstan. .../StanHeaders/include/stan/math/prim/core/init_threadpool_tbb.hpp:9:10: fatal error: tbb/tbb_stddef.h: No such file or directory ``` So the per-package `-DTBB_INTERFACE_NEW` makevars entries the classifier proposed are useless for these packages -- they're blocked on `rstan` (which already has a registry entry). This is the **same dependency cascade** the RcppParallel `applies_to` guard catches, but `tbb-stddef-removed` is a generic signature with no such pin, so ~73 Stan packages kept getting proposed. ## What Generalise cascade detection beyond the RcppParallel special case: - `failing_dependency(error_text, package)` -- when the log names a **different** package as the one that failed to compile (`Failed to build source package X`, `compilation failed for package 'X'`, `dependency 'X' ... not available`), that package is the real cause. - `build_triage_report()` now blocks any package whose **every** failing build is such a cascade: reported as `blocked_on` that dependency, never proposed a bogus per-package entry. A package that fails in its **own** compilation is still proposed. - The `applies_to` (RcppParallel) and data-driven (rstan) cases are unified into one `blocked_packages` / `blocked_on` model; the report, proposer, and `blocked_summary` count the actually-blocked packages, and the blocked note shows even when a group also has genuine proposals. ## Effect Next auto-apply run will stop proposing the rstan-blocked Stan packages (and any future dependency cascade) and surface them as "blocked on rstan" instead. Fixing `rstan` once clears the whole cluster. ## Verification - New tests: `failing_dependency` (cascade vs own-compile vs none), and an end-to-end split where BFpack/GMLTM (blocked on rstan) are not proposed while an own-compile package still is. - Full suite: 112 tests pass; all pre-commit hooks pass. Refs #120, #127. (Separate follow-ups: fixing rstan's build itself, and quieting the gate's metadata-DB retry storm -- both root-caused to bincraft.) Reviewed-on: #128	2026-07-16 08:23:54 +00:00
pat-s	ff9f5f5177	fix(local): shell-quote git args in the auto-patch push and trial-build gate (#126) ... ## Problem The first real `auto-apply-patches` run classified, limited to the top-10, and validated the candidate registry cleanly, then died at the push step: ``` Validating candidate registry: Patch registry OK (13 entries). sh: syntax error: unexpected "(" Error in git("commit", "-m", ...): git commit -m feat(patches): auto-propose 10 registry entries ... failed ``` Root cause: R's `system2()` with captured output (`stdout=TRUE`) runs the command through `/bin/sh`, and the arguments were passed **unquoted**. The commit message `feat(patches): ...` contains `()`, which the shell tried to interpret. The same class of bug affects the `^{commit}` and `ref:path` git refs in the trial-build gate. ## Fix `shQuote()` every git argument: - `propose-patches.R` -- the `git()` helper used by `--open-pr` (commit, push, checkout). - `trial-build-registry.R` -- the base-ref reads (`rev-parse ... ^{commit}`, `ls-tree`, `show ref:path`). ## Verification - Reproduced against a real git repo: the unquoted call fails with status 2 (the same `unexpected "("`); the `shQuote`d call commits successfully, and `rev-parse HEAD^{commit}` resolves. - 105 tests pass; all pre-commit hooks pass. Everything else in that run was correct: 842,659 failing builds classified, RcppParallel's 895 dependents correctly reported as blocked (not proposed), top-10 tbb-stddef candidates selected, 69 deferred, registry validated. Only the shell quoting was broken. Reviewed-on: #126	2026-07-15 15:43:02 +00:00
pat-s	1bca6004fd	chore(local): reuse FORGEJO_TOKEN for the auto-patch push (#125) ... Follow-up to #124 (merged): the auto-patch pipeline required a separate write-scoped `REPO_RW_TOKEN` to push the branch. Reuse the existing `FORGEJO_TOKEN` instead. - `propose-patches.R --open-pr` now pushes `auto/registry-patch-proposals` over HTTPS with `FORGEJO_TOKEN` (the same token used for the PR API); the read-only `origin` clone URL can't push, so it builds an authenticated URL explicitly. - Drop `REPO_RW_TOKEN` from `.crow/auto-apply-patches.yaml` and the docs. No new secret needed: the pipeline's secrets are now `PGPASS`, `REPO_RO_TOKEN`, and `FORGEJO_TOKEN` (all existing). `FORGEJO_TOKEN` must have repository write scope for the push to succeed. 105 tests pass; all pre-commit hooks pass. Reviewed-on: #125	2026-07-15 14:50:19 +00:00
pat-s	4bca17e4ac	feat(local): auto-apply registry patches with a build-env trial-build gate (#124) ... Closes the classifier loop (issue #115, step 3): turns the auto-proposable candidates into an actual PR, gated by a real trial build **in our own build-env images**. Chosen model (from the design discussion): **autonomous PR**, **PR-first with a CI trial-build gate**, **bounded top-N batch** per run. ## Creating the patch PR - `propose-patches.R` gains: - `--limit N` -- act on the top-N candidates by failure volume; the rest defer to the next run (logged). - `--open-pr` -- write the entries onto the reused `auto/registry-patch-proposals` branch, push (with `REPO_RW_TOKEN`), and open/update **one** PR via the Forgejo API (so re-runs update the same PR instead of piling up). - `.crow/auto-apply-patches.yaml` -- a single job that runs `--open-pr --limit` on a cron/manual trigger. Needs `FORGEJO_TOKEN` + a write-scoped `REPO_RW_TOKEN`. Novel source diffs and unknown signatures are still never proposed; nothing merges. ## The merge gate (our build-env images) - `.crow/trial-build-registry.yaml` -- matrixed over the real `OS/IMG` build-env matrix (alpine:3.24, redhat:8/9/10, ubuntu:jammy/noble/**resolute** for ubuntu-2604). Each platform runs `local/trial-build-registry.R`, which diffs the branch registry against `main` and trial-builds **only the entries the branch adds** that apply to that platform, inside `reg.devxy.io/rpkgs/build-env-*`. Green only if every new entry builds; a platform with no new entries is a fast no-op. Nothing is uploaded/archived/recorded. - The base-registry read **fails loud** if it can't read `registry.json` at `main`, rather than silently treating the base as empty and trial-building the whole registry. ## Notes / follow-up - The repo uses **no `pull_request` triggers**, so the gate runs manually or on a cron against the branch (`--var patch_branch=...`). Wiring it to fire automatically on the PR needs `event: pull_request` enabled on the Forgejo webhook -- a one-line addition once that's on. - Two new crons to register in the crow UI: `auto-apply-patches` and `trial-build-registry`. New secret needed: `REPO_RW_TOKEN` (write scope) for the push. ## Verification - New pure helpers `entry_applies_to_os()` / `new_registry_packages()` covered by tests (platform codename/family/wildcard matching; added-vs-unchanged entry detection; per-platform filtering). - `--limit` smoke (stubbed DB): top-2 by volume proposed, 3 deferred, candidate registry validates. - Full suite: 105 tests pass; all pre-commit hooks pass (air, prettier, markdownlint, yamllint, validate-patches). Reviewed-on: #124	2026-07-15 08:28:15 +00:00
pat-s	21a2fe9e6c	feat(local): report unclassified and dependency-blocked failures for discovery (#122) ... ## Why Issue #120 (the auto-proposed-patches issue) only lists **auto-proposable** fixes -- currently just the TBB signatures. So a reasonable read of it was "TBB is our only build failure", when in fact three whole categories are simply not shown there: - **Unclassified failures** -- anything that doesn't match a seeded signature is routed to human triage and never appears (we've only seeded TBB and libuv signatures). - **Dependency-blocked failures** -- the ~800 RcppParallel dependents (post #121) are still failing; they only show as a log line. - Human-only signatures (libuv). These blind spots are exactly where the *next* signatures should come from, so they deserve the same visibility as the proposals. ## What Extend the feedback-loop tracker to surface the classifier's blind spots: - **`unclassified_summary()`** -- groups every unknown-signature failure by normalised fingerprint, ranked by build count, capped with an explicit `dropped_groups` count (no silent truncation), each with example packages + platforms. These are the candidates for new `build_signatures()` rules. - **`blocked_summary()`** -- lists each dependency (e.g. RcppParallel) and how many dependents wait on it. - `proposal-tracking.R` prints both sections, and a new **`--open-issue`** mode posts/updates a *"Unclassified build failures (needs signatures) (#115)"* Forgejo issue. - The weekly crow pipeline now runs the tracker with `--open-issue`, so it maintains a second tracking issue alongside the proposals one. Read-only on the DB; the only writes are the two issues. ## Verification - New tests cover `unclassified_summary` (ranking + both caps) and `blocked_summary`. - Tracker smoke with a stubbed DB (proposable + blocked + unclassified mix) prints the hit rate, `Blocked on a dependency: RcppParallel: 2 dependent(s)`, and `Unclassified failures ... [2 builds | 2 pkgs] ld: undefined reference ...`. - Full suite: 95 tests pass; all pre-commit hooks pass (air, prettier, markdownlint, yamllint, validate-patches). Reviewed-on: #122	2026-07-15 07:46:29 +00:00
pat-s	9bc4973159	fix(local): stop proposing per-dependent patches for dependency-cascade failures (#121) ... ## Problem The scheduled `--open-issue` run (issue #120) posted **800+** proposed entries, all `rcppparallel-bundled-tbb` on `ubuntu-2604`, each pre-filling `RcppParallel/disable-tbb.patch` for an unrelated package (`ACEsimFit`, `AovBay`, `AdaptGauss`, ...). Root cause: those packages fail on ubuntu-2604 only because their **RcppParallel dependency** does not build there. bincraft records RcppParallel's TBB error text against each dependent, so every one of them matches the `rcppparallel-bundled-tbb` signature. Proposing a RcppParallel source patch for `ACEsimFit` is not just noise -- it is **broken**: the diff targets RcppParallel's `Makevars.in`, so it cannot apply to a dependent's source. ## Fix Add an optional `applies_to` field that pins a package-specific fix to the package it targets: - A signature whose fix is a curated per-package source patch (`rcppparallel-bundled-tbb` -> `RcppParallel`) is only ever proposed **for that package**. - Other packages matching the signature are downstream failures, reported as **"blocked on `<dependency>`"** and never proposed a bogus entry. Fix the dependency once and the whole cascade clears. Generic levers (the `tbb-stddef-removed` makevars fix) are unaffected -- they carry no `applies_to`, so they still apply to any matching package. ## Changes - gate proposal generation in `build_triage_report()` on `applies_to`; add a `blocked_on` field to group records - surface blocked groups in both `failing-builds-report.R` and `propose-patches.R` instead of the misleading "already registered" note - document `applies_to` in the signature-table header ## Verification - New tests: RcppParallel dependents are blocked (no `proposed_entries`, `blocked_on = "RcppParallel"`), while RcppParallel *itself* is still proposed when it is the failing package. - Reproduced the avalanche end-to-end with a stubbed DB: 5 dependents now report "Blocked on RcppParallel: 5 package(s) fail because RcppParallel does not build" and **zero** candidates are emitted; a genuine `tbb-stddef` proposal in the same run is unaffected. - Full suite: 84 tests pass; all pre-commit hooks pass. Refs #120. Note: the underlying build problem (RcppParallel failing on ubuntu-2604 despite its registry entry) is real and separate -- this PR stops the classifier from spamming broken per-dependent proposals about it. Reviewed-on: #121	2026-07-14 18:40:41 +00:00
pat-s	f1dd661213	fix(local): route multi-signature packages to human triage in the proposer (#119) ... ## Problem A live `propose-patches.R --open-issue` run aborted with: ``` Patch registry validation FAILED: - ambiguous duplicate entries: hmmTMB|ubuntu-2604|*, imt|ubuntu-2604|*, refundBayes|ubuntu-2604|* Error: Candidate registry failed validation; not writing or proposing. ``` Root cause: a package whose failing builds match **more than one** auto-proposable signature (e.g. some logs hit `tbb-stddef-removed`, others `rcppparallel-bundled-tbb`) landed in two separate signature groups, so the proposer emitted two candidate entries with the same `package|platforms|versions` key. The validator's ambiguous-duplicate check then failed the whole candidate set, aborting the run — so even the unambiguous candidates (e.g. `vacalibration`) never got proposed. ## Fix Add a pure, tested `dedupe_candidates()` that: - keeps a package's candidate only when it maps to a **single** signature (collapsing an accidental repeat under the same signature), and - routes a package that maps to **multiple** signatures (conflicting fix tiers -- makevars vs source patch) to **human triage** instead of guessing between them. The proposer prints the skipped ambiguous packages and proceeds with the clean candidates, so one ambiguous package no longer blocks the rest. This matches the issue #115 guardrail: don't auto-pick when the fix is ambiguous. ## Verification - New unit tests in `test-proposal-tracking-lib.R` cover the single-signature, multi-signature (conflict), duplicate-under-one-signature, and empty cases. - Reproduced the original failure end-to-end with a stubbed DB (`hmmTMB` ambiguous + `vacalibration` clean): `hmmTMB` is now listed under "Ambiguous -> human triage", `vacalibration` is proposed, and the candidate registry validates (`Patch registry OK`). - Full suite: 78 tests pass; all pre-commit hooks pass. Reviewed-on: #119	2026-07-14 17:11:54 +00:00
pat-s	af481f5129	ci(crow): add weekly patch-proposal + feedback-loop pipeline (#118) ... Adds the scheduled pipeline for the failure-triage proposer/tracker landed in #117 (issue #115, steps 3 + 4). ## `.crow/weekly-patch-proposals.yaml` A single (non-matrix) job -- the classifier groups over all of `single_builds`, so unlike the per-platform `weekly-audit-missing` it runs once globally. Weekly it: 1. `propose-patches.R --open-issue` -- posts/updates a Forgejo tracking issue with the auto-proposable registry entries (human reviews, trial-builds, opens the PR; nothing merges). 2. `proposal-tracking.R` -- logs the feedback-loop metrics (signature hit rate, proposed-vs-merged, retirement candidates). Low blast radius: clones read-only (`REPO_RO_TOKEN`); the only write is the tracking issue via `FORGEJO_TOKEN`. It builds nothing (no B2 keys, no bincraft). Triggers: `cron: weekly-patch-proposals` (register the cron in the crow UI) or manual with `task=weekly-patch-proposals`, matching the `archive-missed-packages` convention. Documented under "Scheduled run" in `local/patches/README.md`. `yamllint` and the other pre-commit hooks pass. Reviewed-on: #118	2026-07-14 15:17:16 +00:00
pat-s	1c297c01bf	feat(local): auto-propose registry patches and track the feedback loop (#117) ... Implements steps 3 + 4 of #115, building on the classifier merged in #116. Now that bincraft **v4.4.3** applies registry `patch`/`makevars`/`configure_args` to the *target* package build (previously deps-only), a trial patched build is a meaningful acceptance gate, so the "propose" half is viable. ## Step 3 — propose, do not apply - **`local/propose-patches.R`** — for each classified, safe fix affecting a package with no current registry entry, emits a pre-filled `registry.json` entry and validates the candidate set against a *temporary* merged registry (the real one is never touched unless asked). - default: print candidates + validation, **take no action** - `--write`: append entries to `registry.json` + the proposals ledger (you commit + open the PR) - `--open-issue`: post/update a Forgejo tracking issue (reuses the weekly-audit `httr2` + `FORGEJO_TOKEN` pattern) - **`local/trial-build-patch.R`** — isolated bincraft build of one package with the registry applied (no upload/archive/metadata; `patchhash` keeps it out of the real cache). Exit 0/1, so it gates a CI step or manual pre-merge check. The human gate stays: nothing merges. Acceptance = `validate-patches.R` passes (checked automatically) **and** the trial build succeeds. Novel source diffs and unknown signatures are never proposed (they carry `auto = FALSE`). ## Step 4 — feedback loop - **`local/proposal-tracking.R`** (read-only) — signature hit rate (builds/pkgs/addressed/open per signature), proposed-vs-merged (a proposal counts merged once its package is in the registry), and retirement candidates (registry entries whose package no longer fails, i.e. likely fixed upstream). - **`local/proposal-tracking-lib.R`** — the pure metric/ledger helpers. ## Supporting changes - Refactored the classify helpers to expose a pure `build_triage_report()` + a list-returning entry builder; `failing-builds-report.R` now renders from the shared function (no behaviour change). - `validate-patches.R` gains optional `PATCH_DIR`/`REGISTRY_FILE` overrides (backward-compatible) so a candidate registry can be validated in isolation. - Documented the propose/trial-build/tracking workflow in `local/patches/README.md`. ## Verification - 71 unit tests pass (incl. new `test-proposal-tracking-lib.R`) under the Dockerized R 4.5.3 build env. - All pre-commit hooks pass (`air-format`, `validate-patches`, prettier, etc.). - Smoke-tested all three entrypoints end-to-end with a stubbed DB: dry-run, `--write` (produces a registry that passes the canonical validator + a valid ledger, then reverted), and the tracker. Closes #115 Reviewed-on: #117	2026-07-14 15:03:24 +00:00
pat-s	f11ba7172f	feat(local): classify failing binary builds and pre-fill registry suggestions (#116) ... Implements steps 1 + 2 of #115: turn recorded build failures into triaged patch suggestions instead of hand-scraping Crow logs. ## What this adds A **read-only** reporting pipeline over the `single_builds` metadata table. It never writes to the DB or the registry. - `local/failing-builds-classify.R` — pure, DB-free helpers: - `normalise_error()` strips temp paths, version numbers, hex addresses, and the package name so the same root cause collapses to one fingerprint. - `fingerprint_error()` extracts the salient error line and normalises it. - `classify_error()` matches against a seed signature set; unmatched errors are never guessed at. - `propose_registry_entry()` renders a schema-valid `registry.json` entry. - `local/failing-builds-report.R` — entrypoint: queries `single_builds WHERE error_occurred = TRUE AND removed = FALSE`, groups by root cause (signature when classified, fingerprint otherwise), classifies each group, and prints a triaged report. Flags: `--platform`, `--arch`, `--min`, `--json`; `PLATFORM`/`ARCH` env fallbacks. - `local/tests/test-failing-builds-classify.R` — unit tests for the helpers. - `local/patches/README.md` — documents the workflow. ## Seed signatures Each rule carries a fix tier, confidence, and an auto/human-only flag: | Signature | Fix | Disposition | | --- | --- | --- | | `tbb/tbb_stddef.h: No such file` | makevars `-DTBB_INTERFACE_NEW` | auto-proposable | | RcppParallel bundled TBB (musl / new g++) | curated `disable-tbb.patch` | auto-proposable | | system `libuv.so` link leak | force vendored/static lib | **human triage** (novel source diff) | | unmatched | none | **human triage** | ## Guardrails honored - No autonomous novel source diffs: only known env/makevars levers and already-curated package patches are auto-proposable; anything needing a brand-new diff, and any unknown signature, is routed to human triage. - No DB or registry writes; no change to the public `src/contrib` index. - Reuses `single_builds.error_text`; no new failure-capture pipeline. ## Verification - All helper unit tests pass under the Dockerized R 4.5.3 build env. - Pre-commit hooks pass (`air-format`, `validate-patches`, prettier, etc.). - Smoke-tested the full report path with a stubbed DB; generated proposals pass the real `local/validate-patches.R`. Steps 3 (auto-open PRs) and 4 (feedback loop) are intentionally deferred, per the issue's suggestion to validate the signature set first. Closes #115 Reviewed-on: #116	2026-07-14 14:45:25 +00:00
pat-s	04059f20e3	fix(patches): add rstan entry to skip removed tbb_stddef.h (#114) ... ## Motivation `rstan` fails to compile when it is pulled in as a transitive dependency (e.g. of `ctsem`), aborting the dependency install: ``` /mnt/cache/R-pkgs/StanHeaders/include/stan/math/prim/core/init_threadpool_tbb.hpp:9:10: fatal error: tbb/tbb_stddef.h: No such file or directory 9 | #include <tbb/tbb_stddef.h> ``` StanHeaders' `init_threadpool_tbb.hpp` unconditionally includes the legacy `<tbb/tbb_stddef.h>` to detect the TBB version, but that header was removed in oneTBB 2021+, which is the TBB that StanHeaders now bundles. ## Fix Add an `rstan` entry to `local/patches/registry.json` that pre-defines `TBB_INTERFACE_NEW` via `CPPFLAGS`. The header's own guard skips the removed include when that macro is already defined and selects the modern `tbb/global_control.h` + `tbb/task_arena.h` path that the bundled oneTBB actually provides: ```cpp #ifndef TBB_INTERFACE_NEW #include <tbb/tbb_stddef.h> // removed in oneTBB 2021+ #if TBB_VERSION_MAJOR >= 2020 #define TBB_INTERFACE_NEW #endif #endif #ifdef TBB_INTERFACE_NEW #include <tbb/global_control.h> // taken when we pre-define the macro #include <tbb/task_arena.h> ``` This is a makevars-tier override (no source diff), applied through bincraft's `R_MAKEVARS_USER`, so `CPPFLAGS` stays additive and does not clobber rstan's own `PKG_CPPFLAGS`. `-I/usr/local/include` is kept because the override replaces the default `CPPFLAGS`. ## Files changed - `local/patches/registry.json`: add the `rstan` entry (`versions: "*"`, `platforms: ["*"]`, `makevars.CPPFLAGS = "-DTBB_INTERFACE_NEW -I/usr/local/include"`). ## Behaviour change When bincraft resolves `rstan` (direct or transitive) it builds a patched binary with `TBB_INTERFACE_NEW` defined, so `rstan` (and dependents such as `ctsem`) compile against the bundled oneTBB. The scope is `["*"]` because the failure is bound to StanHeaders' bundled TBB version, not the OS or toolchain. Reviewed-on: #114	2026-07-14 13:52:17 +00:00
pat-s	a4b274f281	fix(ci): bound pkgcache _metadata growth to stop macmini disk-fill (#110) ... ## Problem The arm64 `build-all` pipeline fills the macmini (gaia) host disk despite an 8h prune. Root cause is not images or job volumes: it is the persistent dep-cache volume, specifically `pkgcache/R/pkgcache/_metadata`, which grew to ~165 GB. `{pkgcache}` mints a new content hash for the "patched" binaries repo on every PACKAGES change, so each per-package build writes a fresh ~70 MB `pkgs-<hash>.rds` (+ `patched-<hash>/`) that is never evicted (2407 snapshots observed). When the disk hits 100% OrbStack stops and the on-host prune can no longer connect to the daemon, so it never self-heals. ## Change (Workstream A of the disk-fill fix) - Add `trim_pkgcache_metadata()` to `local/r-minor-helpers.R`: keeps the newest `keep` (default 20) `patched-*`/`pkgs-*.rds` entries under `_metadata`, deleting only entries older than `min_age_secs` (default 600s) so it never races the up-to-4 concurrent split-jobs sharing the volume. Preserves `pkg/` downloads and the stable CRAN/BioC/INLA repo dirs. No-op when `R_PKG_CACHE_DIR` is empty (amd64) or `_metadata` is absent (first run). - Call it every 25 packages inside the build loop in `local/build-all.R`. - Add a defensive start-of-run cleanup of `_metadata/patched-*` + `pkgs-*.rds` to the two workflows that mount the persistent volume (`build-all-versions.yaml`, `build-all-versions-install-deps.yaml`). Only these paths are touched; `process-updates.yaml`/`weekly-rebuild-missing.yaml` (no persistent volume) are unchanged. Follow-ups (separate workstreams): on-host self-healing prune watcher + OrbStack disk cap (ansible), and Prometheus/Grafana alerting (k8s-talos). Upstream: bincraft patched-repo hash churn is the true source fix. New unit tests (6) for the helper; full suite 24/24 green. Reviewed-on: #110	2026-07-13 09:31:29 +00:00
pat-s	ffc2319558	fix(build-all): exclude previously-errored versions in prefilter (#113) ... ## Motivation Build jobs were cycling through hundreds of packages that were only ever printed as `Skipping … due to previous build error recorded in metadata DB`, wasting wall-clock on per-package preparation before dropping each one. ## Cause The prefilter query in `local/build-all.R` selected only successfully-built versions (`error_occurred = FALSE`) into `built`, so line 112 removed only those from the chunk. Every previously-errored version stayed in the work list and was walked one-by-one, each hitting the internal skip in `build_binary_package()`. This also explains the misleading `Skipped 0 already-built package versions` line for alphabetical chunks whose leading packages only have error records. ## Changes - `local/build-all.R`: drop the `AND error_occurred = FALSE` clause so `built` holds every version already attempted (built or errored) for this platform/arch; the existing filter then removes all of them up front. - Rename the log line to `already-attempted` so the reported count reflects successes and errors. - Update the surrounding comment to explain why errored versions are excluded. ## Behaviour change Previously-errored versions are now dropped before the build loop instead of being iterated and individually skipped. No package that would otherwise build is affected, `build_binary_package()` already skipped these internally. Retrying errored versions is out of scope and would need a separate opt-in flag on both the prefilter and the in-loop skip. Reviewed-on: #113	2026-07-13 09:28:45 +00:00
pat-s	36d3bc8604	fix(patches): force fs to build vendored static libuv (#111) ... ## Problem The `fs` 2.1.0 binary links **system libuv** (`readelf -d fs.so` shows `NEEDED libuv.so.1`). fs's `configure` prefers system libuv whenever `pkg-config` resolves it, and our build images ship `libuv-devel` (installed as a pak build-time system requirement), so the resulting binary is dynamically linked against `libuv.so.1`. That binary fails to load on any consumer machine without runtime libuv: ```text unable to load shared object '.../fs/libs/fs.so': libuv.so.1: cannot open shared object file: No such file or directory ``` `install.packages()`/renv do **not** install `SystemRequirements` (only `pak` does, and only inside the build container), so most consumers hit this. Older fs 1.6.x always vendored libuv, so only the 2.x binaries regressed. Reproduced in a clean `reg.devxy.io/r/r-alma:4.5-9`. ## Fix Add `local/patches/fs/force-vendored-libuv.patch`, registered for all platforms. It short-circuits `configure` to `cp -f src/Makevars.vendor src/Makevars; exit 0` before the pkg-config detection, forcing the bundled static libuv build (`tools/libuv-v1.52.0.tar.gz`, built via cmake). An env/pkg-config override (`PKG_CONFIG_LIBDIR`) was tried first but the rebuilt binary still linked `libuv.so.1` (the registry `env` tier does not reach fs's configure step), so a source patch is used instead. ## Verification Built end-to-end inside the real `build-env-redhat:9` image (system libuv present): - patch fires (`Building static libuv (bincraft: forced vendored)`), - cmake compiles the vendored libuv, - resulting `fs.so` has **no `libuv.so.1`** in `NEEDED` (only libR, libstdc++, libm, libgcc_s, libc). `Rscript local/validate-patches.R` passes (2 entries). cmake confirmed present in the build-env images. ## Follow-up (not in this PR) - Rebuild `fs 2.1.0` on every affected platform (rhel8/9/10, ubuntu jammy/noble, alpine 3.22/3.23; amd64 + arm64) and purge the CDN binary paths. - CDN delivery gap: `purge_cdn_cache.sh` only purges `PACKAGES*`, never package binaries, so rebuilt binaries stay masked until their `.tar.gz` path is purged. Reviewed-on: #111	2026-07-07 18:12:36 +00:00
pat-s	1ad7a6bfe9	chore: resolve latest bincraft release dynamically (no hardcoded pins) (#107) ... ## Summary Stop hardcoding the bincraft version. Every `.crow` workflow and the build-one image pinned `@vX.Y.Z` (and a `packageVersion() != "X.Y.Z"` guard), so each bincraft release meant editing the version in ~8 places — and it was easy to miss one (the Dockerfile lagged at v4.2.1; v4.4.1 shipped without the empty-env fix because of exactly this churn). ## Change New `local/install-bincraft.R` resolves the **latest release tag dynamically**: - `git ls-remote --tags` on the public repo (no token), - keep `vX.Y.Z` tags, pick the highest version (filtered/sorted in R for portability, not via git `--sort`/refspec which behaved inconsistently under `system2()`), - `pak::pak("git::…@<latest>")` — idempotent on the git ref, so re-runs keep the package unless a newer tag exists. All call sites now invoke the helper instead of a pinned version: - `.crow/build-all-versions.yaml` (primary + per-minor pass) - `.crow/build-all-versions-install-deps.yaml` - `.crow/process-updates.yaml` (primary + per-minor pass) - `.crow/weekly-rebuild-missing.yaml` - `.crow/archive-missed-packages.yaml` - `docker/build-one.Dockerfile` (ships the helper into the image; `ensure_bincraft` sources it) ## Effect Tag a new bincraft release → the next CI run / `just rebuild` picks it up automatically. No more pin edits, and no more "forgot to bump the Dockerfile" drift. ## Verified - Resolver returns the current latest tag (`v4.4.2`) via `git ls-remote` + R-side version sort. - All five workflow YAMLs parse; helper R parses; air/editorconfig clean. Note: this tracks the latest **tag**, so cutting a release is still the deliberate gate — CI won't pick up un-tagged main. Reviewed-on: #107	2026-07-01 08:10:02 +00:00
pat-s	e59590e2d6	fix: RcppParallel disable-TBB source patch (env var was a no-op) (#106) ... ## Summary Fixes the RcppParallel patch, which was a **no-op** and left the build hanging. The previous registry entry set `env: { RCPP_PARALLEL_USE_TBB: "0" }`. But `RCPP_PARALLEL_USE_TBB` is a **compile-time `-D` flag** in RcppParallel's Makevars — it is never read from the environment. So the override did nothing: `USE_TBB=Linux` (hardcoded from `uname`) still triggered the **bundled Intel TBB build**, which hangs/fails on musl (Alpine) and newer toolchains (g++ 15 on ubuntu-2604). The `Applying patch …` log only meant the env was set, not that it had any effect. ## Fix Replace the env entry with a **source patch** (`local/patches/RcppParallel/disable-tbb.patch`) on `src/Makevars.in` that, on Linux: - leaves `USE_TBB` unset → the whole bundled-TBB build/link path is skipped (no hang), and - forces `PKG_CXXFLAGS += -DRCPP_PARALLEL_USE_TBB=0` → the sources compile the **TinyThread** backend (needed because `RcppParallel.h` otherwise auto-defaults TBB on for glibc Linux). ## Verification In a Linux container, applying the patch and running `R CMD INSTALL RcppParallel`: ``` bundled_TBB_build=0 # bundled TBB build never runs * DONE (RcppParallel) # installs via TinyThread ``` ## Note bincraft's `apply_source_patch` shells out to `patch`. If a build-env image lacks the `patch` tool (common on Alpine), the patch will report "did not apply cleanly" and fall back to an unpatched (hanging) build. If that happens, the follow-up is to switch bincraft's patch application to `git apply` (git is always present) — happy to do that if needed. Reviewed-on: #106	2026-06-30 13:27:01 +00:00
pat-s	55a18fd87d	feat: patch registry + wiring for per-package patching (#103) ... ## Summary Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`). Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading. ## What's included - `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs. - `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace). - `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes). - `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`. - `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`. ## ⚠️ Merge ordering (blocker) This PR adds a `patches = ...` argument to `build_binary_package()` calls. The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until: 1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and 2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`. The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then. Reviewed-on: #103	2026-06-30 08:50:19 +00:00
pat-s	83c761b736	feat(ci): OS/OS_VERSION manual dropdowns + restore lost crow fix (#97) ... ## Summary Two changes: 1. **`OS`/`OS_VERSION` manual-run dropdowns** — give these form variables explicit `options:` lists (like `target_arch` and `R_VERSION`), so the manual-run form shows dropdowns instead of free-text, in both `build-all-versions.yaml` and `build-all-versions-install-deps.yaml`. Crow form variables are independent (no cascading), so the operator still has to pick a coherent `OS` + `OS_VERSION` combination (e.g. `redhat` + `9`, not `alpine` + `jammy`). 2. **Repairs `main`** — the crow fix from PR #96 (`bc2f6f1`) was lost when that PR was squashed (only the first commit was captured). As a result `main` currently carries the `OS: ${OS}` env vars that break Crow parsing (`unable to parse variable name`) and the unfixed `build-all.R`. This PR re-applies that fix: drop the env additions and derive `platform`/`arch` inside `build-all.R` from the container (bincraft codename → platform mapping + `Sys.info()` arch). ## Notes - `OS_VERSION` options are quoted strings so tags like `8`/`9`/`10` aren't parsed as integers. - Validated: both YAMLs parse, `build-all.R` parses. Reviewed-on: #97	2026-06-18 11:39:11 +00:00
pat-s	fa955c10bf	fix: recompute package snapshot when missing from cache (#95) ... ## Summary `build-all.R` reads three snapshot files from `/mnt/cache/packages/` that the `build-all-versions-install-deps` step precomputes: `pkgs_to_build.rds`, `r_minor_sensitive_pkgs.rds`, and `s3_cache.rds`. That cache volume is **per-agent**, so a build job scheduled on a different (fresh) agent than the one that ran install-deps finds the snapshot absent and dies at `readRDS` (`cannot open compressed file '/mnt/cache/packages/pkgs_to_build.rds'`). This adds a conditional guard at the top of `build-all.R`: when any of the three files is missing, it sources `local/packages-to-build.R` (which has all needed creds via `PGPASS` / `B2_S3_*` env, already present in the build step) and writes the derived `.rds` files — exactly mirroring the install-deps command. - The first build job on a fresh agent repopulates the shared cache, so subsequent jobs on that agent reuse it. - Concurrent jobs that also miss simply redo the work (accepted tradeoff vs. slow shared storage like NFS). - Saves use a temp-file + atomic `file.rename`, so a concurrent reader never sees a half-written `.rds`. Reviewed-on: #95	2026-06-17 17:49:04 +00:00
pat-s	1cd86b65e1	feat(build): skip already-built package versions on workflow restart (#91) ... ## Summary When a `build-all-*` workflow is restarted, the build job re-reads the static `pkgs_to_build.rds` that the install-deps step produced once, so it cycles over every package an interrupted run already built. This adds a DB-based skip filter so a restart only processes what is genuinely left. - At job start, `build-all.R` queries the `single_builds` metadata table for `(name, tag)` already built successfully (`error_occurred = FALSE`) on this `platform`/`arch`, and drops those pairs from the chunk before the build loop. It logs how many it skipped. - One indexed query, one round trip, run before the pak forks — no extra S3 listing and no new Python/s3fs memory pressure (`RPostgres`/`DBI` are already used in the container). - Errored versions are intentionally **not** skipped, so transient failures still get retried on restart. ## Dependency Correctness depends on a `error_occurred = FALSE` row meaning the binary is actually published. That guarantee is added in rpkgs/bincraft#56 (success row written only after a confirmed S3 upload). This PR should land together with / after a bincraft release including that fix. Reviewed-on: #91	2026-06-16 07:27:20 +00:00
pat-s	7e01f481c0	perf(local): skip per-minor passes for non-r-minor-sensitive packages ... A package's ABI sensitivity is the same across R minors, so for a non-sensitive package the per-minor loop only ran `ensure_bincraft` (a ~14s bincraft/cranlike install per minor) before build-one.R classified and skipped it. The primary pass now writes a .r_minor_sensitive sentinel when it builds a sensitive package, and the wrapper gates the whole per-minor loop (installs included) on that file — non-sensitive rebuilds no longer touch other minors' libraries.	2026-06-14 14:50:40 +02:00
pat-s	a9ad907f9c	chore(local): add S3 migration and CRAN-source helper scripts ... - migrate-s3-hetzner-to-backblaze.sh: rclone-based bucket migration helper. - find-R-api-packages.sh: scan CRAN package sources for a C-API usage pattern. - query-pkgs-without-old-versions.R, test-package-loading.R: ad-hoc helpers.	2026-06-14 13:02:48 +02:00
pat-s	3ae4672d66	feat(local): just rebuild builds across all installed R minors ... - Dockerfile: install bincraft into each R minor, run a primary pass plus a sensitive-only pass under every other /opt/R/[0-9]* minor (deduped by minor), probe/skip xvfb, set GIT_TERMINAL_PROMPT=0; extra-minor failures are non-fatal. - build-one.R: add --sensitive-only mode, log + shallow-clone the ABI classify step, and clear cranlike's stale ./PACKAGES.db before each index refresh (workaround for the "table packages already exists" bug; pending cranlike fix).	2026-06-14 13:02:48 +02:00
pat-s	1224bd74fe	feat(local): just rebuild recipe for targeted builds via remote buildx (#87) ... ## Summary Adds a local `just rebuild` recipe to (re)build specific versions of a single package on a given OS/arch, dispatching to a remote buildx builder (the build runs there, not locally). - `just rebuild <os> <tag> <arch> <package> <version>...` → `docker buildx build --builder <artemis|gaia> --platform linux/<arch> …` (amd64→artemis, arm64→gaia; names + `R_VERSION` env-overridable). - `docker/build-one.Dockerfile` runs `build-one.R` as a secret-mounted `RUN`, built `--no-cache --output type=cacheonly` (pure side-effect: the S3 upload; no image kept). - `local/build-one.R` auto-classifies each version via the ABI classifier (risky → per-minor slot `contrib/<x.y>/`, else generic), force-rebuilds + uploads + stores metadata, then refreshes the touched slot's `PACKAGES` index. ## Prerequisites - buildx builders named `artemis` (amd64) and `gaia` (arm64) registered (`docker buildx create --name artemis ssh://…`). - Exported secrets: `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY`, `PGPASS` (`GITHUB_PAT` optional). - bincraft `v4.2.0` tag must exist (the build installs `@v4.2.0` and uses its classifier + per-minor index API). Reviewed-on: #87	2026-06-14 09:03:45 +00:00
pat-s	d558e27c11	feat: R-minor-sensitive binary builds (full + iterative) (#84) ... ## Summary Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered. ## What's in here **Detection / precompute** - `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests. - `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions. **Full build** - `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset. - `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor. **Iterative build** - All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added. **Tooling / housekeeping** - Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`. - Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts. - Design + implementation docs under `docs/superpowers/`. ## Requires before merge - A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected. Reviewed-on: #84	2026-06-14 08:04:40 +00:00
pat-s	53da568749	refactor: single source of truth for excluded packages (#79) ... ## Summary `local/build-all.R` had a 75-name hardcoded `exclude <- c(...)` vector that had drifted from `local/excluded-packages.json` — `RcmdrPlugin.ROC` was in the R vector but missing from the JSON. `.crow/weekly-rebuild-missing-*.yaml` already reads the JSON via `jsonlite::fromJSON(...)[["package"]]`. This brings `build-all.R` in line with that pattern. Changes: - **`local/excluded-packages.json`**: add the missing `RcmdrPlugin.ROC` entry (reason `"hang"`, matching siblings). - **`local/build-all.R`**: replace the 16-line hardcoded vector with one `jsonlite::fromJSON(...)` call. - **`.crow/build-all-versions-install-deps-{amd,arm}64.yaml`**: add `jsonlite` to the install-deps `pak::pak()` list so it's available in `/mnt/cache/R-pkgs` for the build step. - **`.crow/build-all-versions-arm64.yaml`**: drop the dead base64-encoded `SKIP_PKGS` docs comment that nobody was passing as a `--var` anyway; replace with a one-line pointer to the JSON. Reviewed-on: #79	2026-06-08 08:30:19 +00:00
pat-s	fb95042d16	chore: drop stale Hetzner-pointing helpers (#77) ... ## Summary Both helpers still point at the old Hetzner storage: - `Justfile` — all 3 recipes (`build-all`, `build-single`, `process-updates`) hit `hel1.your-objectstorage.com` / bucket `devxy-r-package-binaries-hel1` using `HETZNER_S3_*_K3S` env vars. - `local/manual-package-index-update.R` — same endpoint + bucket, hardcoded into `s3fs::s3_dir_ls()` / `s3_file_delete()` calls. Storage moved to Backblaze (`s3.eu-central-003.backblazeb2.com` / `devxy-rpkgs-binaries`) a while back, so running either of these today would write to the wrong bucket or fail outright. No callers reference them in-tree, deleting outright. A one-off rebuild can just call `bincraft::upload_package_index()` directly with current Backblaze settings. Reviewed-on: #77	2026-06-08 08:29:40 +00:00
pat-s	84c59d1984	fix: audit workflows	2026-04-21 21:54:48 +02:00
pat-s	8a98cf96a6	fix: save S3 listing in precompute step, avoid loading s3fs in build step ... s3fs uses reticulate/Python which significantly increases R process memory. When pak tries to fork R to install package dependencies, the enlarged process can't be duplicated within the container memory limit. Now the install-deps step saves the S3 file listing as s3_cache.rds, and the build step reads it with readRDS() — no s3fs loading needed.	2026-04-17 20:05:26 +02:00
pat-s	bc3dbd6126	fix: map x86_64/aarch64 to amd64/arm64 for S3 cache path	2026-04-17 19:04:18 +02:00
pat-s	9b27b3e650	refactor: extract build-all R one-liner into local/build-all.R ... - Load bincraft eagerly via library() to avoid lazy-load memory spike - Do bulk S3 listing upfront and pass as s3_package_cache to avoid per-package S3 calls that accumulate memory and trigger fork failures - Consolidate amd64/arm64 exclude lists into single script - Much easier to read and maintain than a YAML-embedded one-liner	2026-04-17 19:01:08 +02:00
pat-s	87d3f54b21	fix: pass NULL to set_codename() which requires an explicit argument	2026-04-17 18:55:40 +02:00