diff --git a/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md b/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md new file mode 100644 index 0000000..81f828f --- /dev/null +++ b/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md @@ -0,0 +1,148 @@ +# R-minor-sensitive binary builds — design + +## Problem + +CRAN binaries are currently built once, under a single R minor version, and served from a +single generic slot (`…/latest/src/contrib/`). +That is wrong for the minority of packages whose compiled code reaches into volatile R +internals: a binary built under R 4.5 will fail to load under R 4.4 with an +`undefined symbol` error. + +bincraft v4.1.0+ can now detect these packages automatically via the ABI classifier +(`abi_classify()` / `needs_per_minor_recompile()`, added in bincraft PR #49, +). +The exec-env images now ship multiple R minor versions under `/opt/R/`. +This design uses both to build R-minor-specific binaries for the sensitive packages only, +in both the iterative (`process-updates-*`) and full (`build-all-versions-*`) pipelines. + +## Background: how bincraft already behaves + +- `abi_classify(path)` returns a tier: `pure-r` (~78.6%), `safe-compiled` (~7.7%), or + `risky` (~13.6%). `needs_per_minor_recompile(path)` is the boolean wrapper + (`TRUE` iff `risky`). Both need the package source (DESCRIPTION + `src/`). +- `build_binary_package(…, is_r_minor_sensitive = TRUE)` uploads the artifact into a + per-minor slot `…/latest/src/contrib//` and records `r_version` in the + build metadata. With `FALSE` it uses the generic slot. The minor is derived from the + **running interpreter** (`R.version`), so producing a 4.4 binary requires running + `/opt/R/4.4.x/bin/R`. +- `process_cran_updates()` orchestrates the iterative flow with a single + `is_r_minor_sensitive` bool for the whole run and no per-package classification or + R-version loop. +- `upload_package_index()` writes/uploads `PACKAGES*` for the **generic slot only**; it + has no per-minor support. The current standalone r-minor workflow never builds a + per-minor index, so per-minor slots are effectively unservable today. + +## Core model + +Collapse the three tiers to one boolean per package: + +``` +r_minor_sensitive := (abi_classify(pkg)$tier == "risky") +``` + +| group | tiers | built under | slot | +| ------------------------------ | ---------------------- | -------------------- | ---------------------------- | +| non-sensitive (~86%) | pure-r, safe-compiled | primary R only | generic `contrib/` | +| sensitive (~14%) | risky | every installed minor| per-minor `contrib//` | + +- **Primary R** = the existing `R_VERSION` env var in each workflow. Its pass builds the + non-sensitive packages (generic slot) *and* the sensitive packages for its own minor slot. +- **Extra minors** = every other R version discovered by scanning `/opt/R/` at runtime. + Each runs a **sensitive-only** pass. An image with a single R version degrades cleanly + to just the primary pass. +- The loop over minors always lives at the shell/script layer (one `/opt/R//bin/R` + invocation per minor), never inside a single `build_binary_package()` call. + +Classification granularity: classify **once per package** (its release version) and apply +the resulting flag to all archived versions of that package. Tier rarely changes across +recent versions; this avoids multiplying source downloads. + +## Full build — `build-all-versions-*` + `local/` + +### install-deps step (`local/packages-to-build.R`) + +After computing `pkgs_to_build`, add an `r_minor_sensitive` logical column: + +1. Pull `NeedsCompilation` and `LinkingTo` from `tools::CRAN_package_db()` (already loaded). +2. Resolve cheaply, no download: + - `NeedsCompilation != "yes"` → not sensitive (rule 1, pure-r). + - `LinkingTo` references any `bincraft::abi_risky_linking_deps()` entry → sensitive + (rule 2). +3. For the remaining compiled, non-LinkingTo-risky packages only: download the source and + call `bincraft::needs_per_minor_recompile()` (rules 3/4). +4. Join the per-package flag onto every `(Package, Version)` row. + +Outputs: +- `pkgs_to_build.rds` — now carries the `r_minor_sensitive` column. +- `r_minor_sensitive_pkgs.rds` — the sensitive subset, for the extra-minor passes. + +### build step (`local/build-all.R` + `.crow/build-all-versions-{amd64,arm64}.yaml`) + +- `build-all.R` gains an optional `--sensitive-only` mode (or an arg flag). In normal mode + it builds the full chunk, passing `is_r_minor_sensitive = ` per package. In + sensitive-only mode it reads `r_minor_sensitive_pkgs.rds`, intersects with its chunk, and + builds those with `is_r_minor_sensitive = TRUE`. +- The workflow step keeps the existing matrix split. After the primary `Rscript build-all.R` + invocation, a shell loop discovers non-primary `/opt/R/*` minors and runs + `Rscript build-all.R --sensitive-only ` under each. +- Index upload step: generic index as today, plus a per-minor index for each minor slot + that received artifacts (depends on the `upload_package_index()` enhancement below). + +## Iterative build — `process-updates-*` + +Driven by the bincraft enhancement below; the single build step becomes: + +1. Primary-R pass: + `process_cran_updates(…, r_minor_detection = "classifier")`. + Each updated/new package is classified; risky → primary minor slot, rest → generic slot. + Removed-package handling stays as-is. +2. Shell loop over non-primary `/opt/R/*` minors: + `process_cran_updates(…, r_minor_detection = "classifier", r_minor_sensitive_only = TRUE)`. + Builds only risky updates into their respective minor slots. +3. Index upload extended to cover each touched minor slot in addition to the generic slot. + +## Required bincraft enhancements (separate PR, coordinated release) + +1. `build_binary_package()`: + - Accept `is_r_minor_sensitive = "auto"` — classify the source it already clones via + `abi_classify()` and route the artifact to the per-minor slot iff `risky`. + - Add `r_minor_sensitive_only` — when `TRUE`, skip (return `"skipped"`) non-risky + packages early, after classification, before building. +2. `process_cran_updates()`: + - Add `r_minor_detection = c("none", "issue", "classifier")` (default `"none"` to + preserve current behavior; `"issue"` is today's `filter_r_minor_sensitive` path). + - Add `r_minor_sensitive_only`, threaded down to `build_binary_package()`. + - With `"classifier"`, pass `is_r_minor_sensitive = "auto"` per package instead of a + single run-wide bool. +3. `upload_package_index()`: + - Add per-minor slot support: write/upload `PACKAGES*` (and `Meta/archive.rds`) under + `…/contrib//`, mirroring the generic-slot logic. Invocable per minor. + +These three are the only bincraft changes; detection itself (PR #49) is already merged. + +## Scope and cleanup + +- In scope: `.crow/process-updates-*` (iterative) and `.crow/build-all-versions-*` + + `local/build-all.R` + `local/packages-to-build.R` (full), across amd64 and arm64 and all + platforms (alpine, ubuntu, redhat). The per-platform workflow files share the same edit. +- Removed: `.crow/build-r-minor-sensitive-packages.yaml` — superseded by the integrated + flow (it was manual, alpine-3.21-only, and issue-list-driven). +- Out of scope (call out, do not change here): `weekly-rebuild-missing-*`, + `archive-missed-packages`, and the audit workflows. Revisit separately if per-minor + rebuilds are wanted there too. + +## Open risks / notes + +- **R-version discovery**: assumes `/opt/R//bin/R` layout and that the primary + `R_VERSION` is one of the installed versions. Parse minor as `major.minor` from each + discovered version; dedupe by minor (build once per minor even if two patch releases + coexist). +- **Per-minor index correctness**: clients resolving `bin//contrib//` require the + per-minor `PACKAGES` to exist; the `upload_package_index()` enhancement is a hard + dependency for the sensitive artifacts to be usable. Verify against a real client + install before declaring done. +- **Classification cost**: bounded by downloading sources only for the compiled, + non-LinkingTo-risky subset in install-deps. Worth measuring on a full run; if still too + heavy, consider caching classifications keyed by package+version in the metadata DB. +- **Double download**: install-deps classification downloads some sources that the build + step re-downloads. Acceptable for now; the metadata-DB cache above would also remove this.