- Dockerfile: install bincraft into each R minor, run a primary pass plus a
sensitive-only pass under every other /opt/R/[0-9]* minor (deduped by minor),
probe/skip xvfb, set GIT_TERMINAL_PROMPT=0; extra-minor failures are non-fatal.
- build-one.R: add --sensitive-only mode, log + shallow-clone the ABI classify
step, and clear cranlike's stale ./PACKAGES.db before each index refresh
(workaround for the "table packages already exists" bug; pending cranlike fix).
## Summary
Adds a local `just rebuild` recipe to (re)build specific versions of a single package on a given OS/arch, dispatching to a remote buildx builder (the build runs there, not locally).
- `just rebuild <os> <tag> <arch> <package> <version>...` → `docker buildx build --builder <artemis|gaia> --platform linux/<arch> …` (amd64→artemis, arm64→gaia; names + `R_VERSION` env-overridable).
- `docker/build-one.Dockerfile` runs `build-one.R` as a secret-mounted `RUN`, built `--no-cache --output type=cacheonly` (pure side-effect: the S3 upload; no image kept).
- `local/build-one.R` auto-classifies each version via the ABI classifier (risky → per-minor slot `contrib/<x.y>/`, else generic), force-rebuilds + uploads + stores metadata, then refreshes the touched slot's `PACKAGES` index.
## Prerequisites
- buildx builders named `artemis` (amd64) and `gaia` (arm64) registered (`docker buildx create --name artemis ssh://…`).
- Exported secrets: `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY`, `PGPASS` (`GITHUB_PAT` optional).
- bincraft `v4.2.0` tag must exist (the build installs `@v4.2.0` and uses its classifier + per-minor index API).
Reviewed-on: #87
## Summary
Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered.
## What's in here
**Detection / precompute**
- `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests.
- `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions.
**Full build**
- `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset.
- `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor.
**Iterative build**
- All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added.
**Tooling / housekeeping**
- Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`.
- Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts.
- Design + implementation docs under `docs/superpowers/`.
## Requires before merge
- A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected.
Reviewed-on: #84
## Summary
`local/build-all.R` had a 75-name hardcoded `exclude <- c(...)` vector that had drifted from `local/excluded-packages.json` — `RcmdrPlugin.ROC` was in the R vector but missing from the JSON.
`.crow/weekly-rebuild-missing-*.yaml` already reads the JSON via `jsonlite::fromJSON(...)[["package"]]`. This brings `build-all.R` in line with that pattern.
Changes:
- **`local/excluded-packages.json`**: add the missing `RcmdrPlugin.ROC` entry (reason `"hang"`, matching siblings).
- **`local/build-all.R`**: replace the 16-line hardcoded vector with one `jsonlite::fromJSON(...)` call.
- **`.crow/build-all-versions-install-deps-{amd,arm}64.yaml`**: add `jsonlite` to the install-deps `pak::pak()` list so it's available in `/mnt/cache/R-pkgs` for the build step.
- **`.crow/build-all-versions-arm64.yaml`**: drop the dead base64-encoded `SKIP_PKGS` docs comment that nobody was passing as a `--var` anyway; replace with a one-line pointer to the JSON.
Reviewed-on: #79
## Summary
Both helpers still point at the old Hetzner storage:
- `Justfile` — all 3 recipes (`build-all`, `build-single`, `process-updates`) hit `hel1.your-objectstorage.com` / bucket `devxy-r-package-binaries-hel1` using `HETZNER_S3_*_K3S` env vars.
- `local/manual-package-index-update.R` — same endpoint + bucket, hardcoded into `s3fs::s3_dir_ls()` / `s3_file_delete()` calls.
Storage moved to Backblaze (`s3.eu-central-003.backblazeb2.com` / `devxy-rpkgs-binaries`) a while back, so running either of these today would write to the wrong bucket or fail outright.
No callers reference them in-tree, deleting outright. A one-off rebuild can just call `bincraft::upload_package_index()` directly with current Backblaze settings.
Reviewed-on: #77
s3fs uses reticulate/Python which significantly increases R process memory.
When pak tries to fork R to install package dependencies, the enlarged
process can't be duplicated within the container memory limit.
Now the install-deps step saves the S3 file listing as s3_cache.rds,
and the build step reads it with readRDS() — no s3fs loading needed.
- Load bincraft eagerly via library() to avoid lazy-load memory spike
- Do bulk S3 listing upfront and pass as s3_package_cache to avoid
per-package S3 calls that accumulate memory and trigger fork failures
- Consolidate amd64/arm64 exclude lists into single script
- Much easier to read and maintain than a YAML-embedded one-liner
The precompute script was constructing S3 paths as "redhat10" while
bincraft uses "rhel10" as the codename. This mismatch caused the
precompute to find no existing packages in S3, producing ~77K false
positive package versions that all get skipped at build time.
Now uses bincraft::set_codename() to ensure path alignment.
Also fixes nrow() vs length() in install-deps summary message.
packages-to-build.R now saves the full (Package, Version) data.table
so the build step can pass specific tags to build_binary_package(),
eliminating redundant per-package tag discovery and S3 checks.
Also reduces archive versions from 9 to 4 (+ 1 release = 5 total).
The S3 credentials, endpoint, and bucket name were still pointing to the old Hetzner storage after the Backblaze migration, causing silent build failures.
