Replace the per-platform process-updates-<platform>-<arch>.yaml files with
process-updates-amd64.yaml and process-updates-arm64.yaml, each driving all
7 platforms via matrix.include. Split by arch because amd64 pins a nodeSelector
instance-type and runs on agent 'artemis' with backend_options, while arm64
runs on agent 'gaia' with none -- a single template can't cleanly express both.
Per-platform variance (R version, image tag, codename, and the process_new
flag which is FALSE for alpine) lives in matrix rows. Current runs are preserved
1:1: each existing per-platform cron still fires only its matching row via
CI_PIPELINE_CRON, and the manual trigger uses a PR-1165 dropdown
(process_cran_updates) for 'all' or a specific <platform>-<arch>. Server-side
cron entries are unchanged.
Resolve .crow conflicts by taking main's content (which moved bincraft pins
to 4.2.1 and added per-minor build/install loops) and re-bumping every
bincraft pin to 4.2.2, so all pinned pipelines use the metadata-after-upload
release. build-all.R auto-merged: the restart skip-filter coexists with
main's r-minor-sensitive build path.
Moves the process-updates-*, weekly-rebuild-missing-* and archive-missed
workflows onto bincraft 4.2.2, which records a build's success metadata only
after the S3 upload is confirmed (rpkgs/bincraft#56). Keeps these pipelines
consistent with the metadata-after-upload behavior the restart-skip filter
relies on.
The sensitive-only multi-R build passes ran under each minor's R but loaded
bincraft from the shared R_LIBS_USER=/mnt/cache/R-pkgs cache, which is compiled
under the primary R and fails to load under a newer minor (e.g. 4.6:
"undefined symbol: SETLENGTH") — so those minor builds silently no-op'd.
Give each non-primary minor its own R_LIBS_USER=/mnt/cache/R-pkgs-<minor> and
install bincraft 4.2.1 there (built for that minor); the rest of bincraft's
deps load from the per-version /opt/R/<minor>/lib. Applied to the
build-all-versions and process-updates sensitive-only passes.
The per-minor upload_package_index() loop invoked each minor's own R, which
loads bincraft from the shared /mnt/cache/R-pkgs cache (compiled under the
primary R). Under a newer minor (e.g. 4.6) that fails to load
("undefined symbol: SETLENGTH") and the index step dies. Indexing is pure S3
work and needs no specific R, so run it under /opt/R/$R_VERSION like the generic
index and build-all-versions already do.
- Pin bincraft v4.2.0 -> v4.2.1 (per-minor archive.rds/index fix) across the
process-updates, weekly-rebuild and archive workflows.
- Restrict the per-R-minor build and index loops to /opt/R/[0-9]* so the
`current` symlink dir is not treated as a distinct minor (avoids double-
building a minor and writing a bogus contrib/current/ index slot).
## Summary
Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered.
## What's in here
**Detection / precompute**
- `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests.
- `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions.
**Full build**
- `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset.
- `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor.
**Iterative build**
- All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added.
**Tooling / housekeeping**
- Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`.
- Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts.
- Design + implementation docs under `docs/superpowers/`.
## Requires before merge
- A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected.
Reviewed-on: #84
## Summary
The `rm -rf /mnt/cache/...` block in 14 `process-updates-*` files had two real issues:
1. `/mnt/cache/pkgcache/R/pkgcache` listed twice — harmless duplicate.
2. `/mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres` — missing a space between two paths, so `rm` walks a nested nonsense path that never exists. The `00LOCK-RPostgres` lock cleanup never actually ran.
Collapses to the same shape `weekly-rebuild-missing-*` already uses:
```
rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/{bincraft,pkgcache} /mnt/cache/pkgcache/R/pkgcache
```
`00LOCK-*` covers every stale lock (pak, bincraft, RPostgres, r-pkg-binaries, …) without enumerating them, so adding a new locked package later doesn't need another YAML touch.
14 files, 1 line changed each.
Reviewed-on: #76
## Summary
`archive-missed-packages.yaml`'s matrix was last touched when redhat-9 and alpine322 were the newest targets. redhat-10 and alpine323 builds are now wired up across `process-updates-*` and `weekly-rebuild-missing-*` workflows, but the daily archive sweep wasn't catching them — so missed-archive packages on those platforms accumulate untouched.
Adds the 4 missing entries:
- `redhat-10` × `{amd64, arm64}`
- `alpine323` × `{amd64, arm64}`
Reviewed-on: #75
## Summary
`local/build-all.R` had a 75-name hardcoded `exclude <- c(...)` vector that had drifted from `local/excluded-packages.json` — `RcmdrPlugin.ROC` was in the R vector but missing from the JSON.
`.crow/weekly-rebuild-missing-*.yaml` already reads the JSON via `jsonlite::fromJSON(...)[["package"]]`. This brings `build-all.R` in line with that pattern.
Changes:
- **`local/excluded-packages.json`**: add the missing `RcmdrPlugin.ROC` entry (reason `"hang"`, matching siblings).
- **`local/build-all.R`**: replace the 16-line hardcoded vector with one `jsonlite::fromJSON(...)` call.
- **`.crow/build-all-versions-install-deps-{amd,arm}64.yaml`**: add `jsonlite` to the install-deps `pak::pak()` list so it's available in `/mnt/cache/R-pkgs` for the build step.
- **`.crow/build-all-versions-arm64.yaml`**: drop the dead base64-encoded `SKIP_PKGS` docs comment that nobody was passing as a `--var` anyway; replace with a one-line pointer to the JSON.
Reviewed-on: #79
## Summary
bincraft is baked into the build-env images, but every workflow still unconditionally calls `pak::pak("git::.../bincraft.git")` (~30–60s per job).
Pin bincraft to a specific tag (currently `v4.0.2`) and guard the install on a version check.
```
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.0.2") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.0.2")'
```
- Image matches the pin → skip install entirely.
- Image is missing bincraft or has a different version → install the pinned tag (covers both broken bakes and intentional version bumps).
- Renovate customManager keeps the pin and the version-check string in sync whenever bincraft cuts a new tag (uses `git-tags` datasource against the bincraft repo).
Touches 43 `.crow` YAMLs + `renovate.json`.
Reviewed-on: #72
## Summary
`process-updates-*` already calls `bincraft::upload_package_index()` at the end of every daily delta run, and with #71 the CDN purge happens there too.
The dedicated `update-package-index-*` chain re-ran the same `upload_package_index()` against the same S3 prefixes — once per platform per day — and was serialized via `depends_on:` purely to dodge S3 rate limits.
Net effect: ~40 min cold rebuild × 14 platforms = ~9 h/day of duplicated work, with no consumer waiting on it (process-updates had already published the fresh index).
This deletes the 14 chain files (~1.1k lines). The matching `update-package-index` / `update-package-index-*` cron entries on the Crow side become inert with the YAMLs gone and can be removed in a follow-up cron-config change.
Best reviewed alongside (or after):
- #71 — purge CDN after process-updates
- #72 — guard bincraft reinstall
- #73 — drop redundant cranlike/desc installs
## Risks
- If `process-updates-*` ever fails for a given platform, the chain used to act as a daily safety net for the PACKAGES index. Failures will now surface only via the missing daily push instead of being silently masked. The existing `weekly-audit-missing-*` workflow continues to flag stale state.
- `local/manual-package-index-update.R` still exists for one-off manual rebuilds.
Reviewed-on: #74
## Summary
`bincraft`'s DESCRIPTION already pulls both forks via `Remotes:`:
```
Imports:
...
cranlike,
...
Remotes:
pat-s/cranlike@s3,
pat-s/desc@description-from-remote
```
So `pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")` resolves the forks and installs them transitively. The explicit follow-up `pak::pak("pat-s/cranlike@s3")` and `pak::pak("pat-s/desc@description-from-remote")` calls in `update-package-index-*.yaml` are dead weight.
Drops the two lines from all 14 `update-package-index-*` files.
`process-updates-*.yaml` already only installs `bincraft` — this brings the two workflows in sync.
Reviewed-on: #73
## Summary
- `process_cran_updates()` ends with `upload_package_index()`, refreshing `src/contrib/PACKAGES{,.gz,.rds,.db}` in S3 — but no CDN purge follows.
- The CDN was only purged by the daily `update-package-index-*` chain, so between cron runs the edge kept serving stale indexes. A binary that landed in S3 was effectively invisible to `install.packages()` until the index-update chain caught up.
Adds the same `Purge CDN cache` step that `update-package-index-*` already uses, parameterized per platform/arch. The script and image are unchanged.
Reviewed-on: #71
## Summary
- Switches all 64 `.crow/*.yaml` workflows, the `Justfile`, and the commented-out `build-all-versions-install-deps.yaml` template to the new multi-R-version container images.
- Each image tag drops `-${R_VERSION}` (e.g. `build-env-alpine:3.22-4.5` → `build-env-alpine:3.22`); workflows pick an R version by calling `/opt/R/${R_VERSION}/bin/R` explicitly.
- Pattern-1 workflows (hard-coded image tag) gain a single `R_VERSION:` env var per file.
- Pattern-2 workflows (`build-all-versions-*`, matrix-driven) continue to receive `R_VERSION` via `crow pipeline create --var`.
## Platform → R_VERSION mapping
| Platform | Image | R_VERSION |
| --- | --- | --- |
| alpine 3.22 / 3.23 | `build-env-alpine:3.{22,23}` | 4.5.3 |
| ubuntu jammy / noble | `build-env-ubuntu:{jammy,noble}` | 4.4.3 |
| redhat 8 / 9 | `build-env-redhat:{8,9}` | 4.4.3 |
| redhat 10 | `build-env-redhat:10` | 4.5.3 |
The `alpine-321` audit-only workflows fall back to `build-env-alpine:3.23` since no 3.21 image exists in the new scheme; the audits only query S3/CRAN, so the container OS does not affect correctness.
## Bug fixes folded in
1. All six `weekly-audit-missing-alpine-*.yaml` files were pinned to `alpine:3.23-4.5` regardless of platform.
Now each one points at its own alpine image.
2. All fourteen `update-package-index-*.yaml` files were pinned to `build-env-ubuntu:noble-4.4` (or `noble-4.5` for `redhat-10-amd64`) regardless of platform.
Now each one uses its own platform's image.
3. `process-updates-ubuntu-2404-{amd64,arm64}.yaml` had `R_VERSION` of `4.4` (no patch) baked into the tag.
Aligned to `4.4.3`, matching the audit and rebuild counterparts.
## Smoke targets (post-merge)
One workflow per shape, ready to dry-run:
- `process-updates-alpine-322-amd64.yaml`
- `weekly-rebuild-missing-redhat-9-amd64.yaml`
- `weekly-audit-missing-ubuntu-2204-amd64.yaml`
- `update-package-index-redhat-10-amd64.yaml`
- `archive-missed-packages.yaml`
- `build-all-versions-amd64.yaml` (with its install-deps predecessor)
- `build-r-minor-sensitive-packages.yaml`
## Follow-ups (out of scope, surfaced during review)
- `process-updates-redhat-10-{amd64,arm64}.yaml` set `codename = "rhel9"` in their `upload_package_index(...)` calls.
Pre-existing bug — should be `"rhel10"`. The `update-package-index-redhat-10-*` counterparts already use `"rhel10"` correctly.
- `update-package-index-redhat-8-amd64.yaml` is missing the `packageVersion("bincraft")` diagnostic line that every sibling file has.
Pre-existing asymmetry between amd64 and arm64.
## Spec & plan
- Spec: `docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md`
- Plan: `docs/superpowers/plans/2026-05-25-multi-r-version-images.md`
Reviewed-on: #70
Both pipelines had platform = 'alpine-321' instead of their actual target, causing builds to upload to the wrong S3 path and the PACKAGES index to not reflect newly built packages.
- Load bincraft eagerly via library() to avoid lazy-load memory spike
- Do bulk S3 listing upfront and pass as s3_package_cache to avoid
per-package S3 calls that accumulate memory and trigger fork failures
- Consolidate amd64/arm64 exclude lists into single script
- Much easier to read and maintain than a YAML-embedded one-liner
The precompute script was constructing S3 paths as "redhat10" while
bincraft uses "rhel10" as the codename. This mismatch caused the
precompute to find no existing packages in S3, producing ~77K false
positive package versions that all get skipped at build time.
Now uses bincraft::set_codename() to ensure path alignment.
Also fixes nrow() vs length() in install-deps summary message.
force=TRUE bypasses the error-DB filter and overwrites build metadata,
causing failed packages to be retried indefinitely. Since the precompute
step already filters to missing versions, force is unnecessary.
Each process-updates workflow now calls upload_package_index at the end to update the PACKAGES files for its platform/arch immediately after builds complete. The full update-package-index cron should be rescheduled to weekly.
All 12 workflows were triggered simultaneously by the update-package-index cron. Now they run one after another via depends_on. Individual cron triggers still allow running one at a time.
Uses install-deps step, precomputed package list, weston compositor, stderr redirect, sequential plan with progress output, and unique X server numbers.
Move packages-to-build.R execution to the single install-deps step and save results as RDS. Matrix jobs read the precomputed list instead of loading the massive CRAN archive/release databases, freeing memory for fork/system calls.
