docs: design for R-minor-sensitive binary builds
Spec for integrating bincraft's ABI classifier (PR #49) into the iterative and full build pipelines so only R-minor-sensitive packages are built per installed R minor, leveraging the multi-R exec-env images.
This commit is contained in:
parent
a02bd5c4ff
commit
59c3e0960c
GPG key ID:
3C6318841EF78925
1 changed files with 148 additions and 0 deletions
|
|
@ -0,0 +1,148 @@
|
|||
# R-minor-sensitive binary builds — design
|
||||
|
||||
## Problem
|
||||
|
||||
CRAN binaries are currently built once, under a single R minor version, and served from a
|
||||
single generic slot (`…/latest/src/contrib/`).
|
||||
That is wrong for the minority of packages whose compiled code reaches into volatile R
|
||||
internals: a binary built under R 4.5 will fail to load under R 4.4 with an
|
||||
`undefined symbol` error.
|
||||
|
||||
bincraft v4.1.0+ can now detect these packages automatically via the ABI classifier
|
||||
(`abi_classify()` / `needs_per_minor_recompile()`, added in bincraft PR #49,
|
||||
<https://codefloe.com/rpkgs/bincraft/pulls/49>).
|
||||
The exec-env images now ship multiple R minor versions under `/opt/R/`.
|
||||
This design uses both to build R-minor-specific binaries for the sensitive packages only,
|
||||
in both the iterative (`process-updates-*`) and full (`build-all-versions-*`) pipelines.
|
||||
|
||||
## Background: how bincraft already behaves
|
||||
|
||||
- `abi_classify(path)` returns a tier: `pure-r` (~78.6%), `safe-compiled` (~7.7%), or
|
||||
`risky` (~13.6%). `needs_per_minor_recompile(path)` is the boolean wrapper
|
||||
(`TRUE` iff `risky`). Both need the package source (DESCRIPTION + `src/`).
|
||||
- `build_binary_package(…, is_r_minor_sensitive = TRUE)` uploads the artifact into a
|
||||
per-minor slot `…/latest/src/contrib/<major.minor>/` and records `r_version` in the
|
||||
build metadata. With `FALSE` it uses the generic slot. The minor is derived from the
|
||||
**running interpreter** (`R.version`), so producing a 4.4 binary requires running
|
||||
`/opt/R/4.4.x/bin/R`.
|
||||
- `process_cran_updates()` orchestrates the iterative flow with a single
|
||||
`is_r_minor_sensitive` bool for the whole run and no per-package classification or
|
||||
R-version loop.
|
||||
- `upload_package_index()` writes/uploads `PACKAGES*` for the **generic slot only**; it
|
||||
has no per-minor support. The current standalone r-minor workflow never builds a
|
||||
per-minor index, so per-minor slots are effectively unservable today.
|
||||
|
||||
## Core model
|
||||
|
||||
Collapse the three tiers to one boolean per package:
|
||||
|
||||
```
|
||||
r_minor_sensitive := (abi_classify(pkg)$tier == "risky")
|
||||
```
|
||||
|
||||
| group | tiers | built under | slot |
|
||||
| ------------------------------ | ---------------------- | -------------------- | ---------------------------- |
|
||||
| non-sensitive (~86%) | pure-r, safe-compiled | primary R only | generic `contrib/` |
|
||||
| sensitive (~14%) | risky | every installed minor| per-minor `contrib/<x.y>/` |
|
||||
|
||||
- **Primary R** = the existing `R_VERSION` env var in each workflow. Its pass builds the
|
||||
non-sensitive packages (generic slot) *and* the sensitive packages for its own minor slot.
|
||||
- **Extra minors** = every other R version discovered by scanning `/opt/R/` at runtime.
|
||||
Each runs a **sensitive-only** pass. An image with a single R version degrades cleanly
|
||||
to just the primary pass.
|
||||
- The loop over minors always lives at the shell/script layer (one `/opt/R/<ver>/bin/R`
|
||||
invocation per minor), never inside a single `build_binary_package()` call.
|
||||
|
||||
Classification granularity: classify **once per package** (its release version) and apply
|
||||
the resulting flag to all archived versions of that package. Tier rarely changes across
|
||||
recent versions; this avoids multiplying source downloads.
|
||||
|
||||
## Full build — `build-all-versions-*` + `local/`
|
||||
|
||||
### install-deps step (`local/packages-to-build.R`)
|
||||
|
||||
After computing `pkgs_to_build`, add an `r_minor_sensitive` logical column:
|
||||
|
||||
1. Pull `NeedsCompilation` and `LinkingTo` from `tools::CRAN_package_db()` (already loaded).
|
||||
2. Resolve cheaply, no download:
|
||||
- `NeedsCompilation != "yes"` → not sensitive (rule 1, pure-r).
|
||||
- `LinkingTo` references any `bincraft::abi_risky_linking_deps()` entry → sensitive
|
||||
(rule 2).
|
||||
3. For the remaining compiled, non-LinkingTo-risky packages only: download the source and
|
||||
call `bincraft::needs_per_minor_recompile()` (rules 3/4).
|
||||
4. Join the per-package flag onto every `(Package, Version)` row.
|
||||
|
||||
Outputs:
|
||||
- `pkgs_to_build.rds` — now carries the `r_minor_sensitive` column.
|
||||
- `r_minor_sensitive_pkgs.rds` — the sensitive subset, for the extra-minor passes.
|
||||
|
||||
### build step (`local/build-all.R` + `.crow/build-all-versions-{amd64,arm64}.yaml`)
|
||||
|
||||
- `build-all.R` gains an optional `--sensitive-only` mode (or an arg flag). In normal mode
|
||||
it builds the full chunk, passing `is_r_minor_sensitive = <row flag>` per package. In
|
||||
sensitive-only mode it reads `r_minor_sensitive_pkgs.rds`, intersects with its chunk, and
|
||||
builds those with `is_r_minor_sensitive = TRUE`.
|
||||
- The workflow step keeps the existing matrix split. After the primary `Rscript build-all.R`
|
||||
invocation, a shell loop discovers non-primary `/opt/R/*` minors and runs
|
||||
`Rscript build-all.R --sensitive-only <split> <index> <ncpus>` under each.
|
||||
- Index upload step: generic index as today, plus a per-minor index for each minor slot
|
||||
that received artifacts (depends on the `upload_package_index()` enhancement below).
|
||||
|
||||
## Iterative build — `process-updates-*`
|
||||
|
||||
Driven by the bincraft enhancement below; the single build step becomes:
|
||||
|
||||
1. Primary-R pass:
|
||||
`process_cran_updates(…, r_minor_detection = "classifier")`.
|
||||
Each updated/new package is classified; risky → primary minor slot, rest → generic slot.
|
||||
Removed-package handling stays as-is.
|
||||
2. Shell loop over non-primary `/opt/R/*` minors:
|
||||
`process_cran_updates(…, r_minor_detection = "classifier", r_minor_sensitive_only = TRUE)`.
|
||||
Builds only risky updates into their respective minor slots.
|
||||
3. Index upload extended to cover each touched minor slot in addition to the generic slot.
|
||||
|
||||
## Required bincraft enhancements (separate PR, coordinated release)
|
||||
|
||||
1. `build_binary_package()`:
|
||||
- Accept `is_r_minor_sensitive = "auto"` — classify the source it already clones via
|
||||
`abi_classify()` and route the artifact to the per-minor slot iff `risky`.
|
||||
- Add `r_minor_sensitive_only` — when `TRUE`, skip (return `"skipped"`) non-risky
|
||||
packages early, after classification, before building.
|
||||
2. `process_cran_updates()`:
|
||||
- Add `r_minor_detection = c("none", "issue", "classifier")` (default `"none"` to
|
||||
preserve current behavior; `"issue"` is today's `filter_r_minor_sensitive` path).
|
||||
- Add `r_minor_sensitive_only`, threaded down to `build_binary_package()`.
|
||||
- With `"classifier"`, pass `is_r_minor_sensitive = "auto"` per package instead of a
|
||||
single run-wide bool.
|
||||
3. `upload_package_index()`:
|
||||
- Add per-minor slot support: write/upload `PACKAGES*` (and `Meta/archive.rds`) under
|
||||
`…/contrib/<major.minor>/`, mirroring the generic-slot logic. Invocable per minor.
|
||||
|
||||
These three are the only bincraft changes; detection itself (PR #49) is already merged.
|
||||
|
||||
## Scope and cleanup
|
||||
|
||||
- In scope: `.crow/process-updates-*` (iterative) and `.crow/build-all-versions-*` +
|
||||
`local/build-all.R` + `local/packages-to-build.R` (full), across amd64 and arm64 and all
|
||||
platforms (alpine, ubuntu, redhat). The per-platform workflow files share the same edit.
|
||||
- Removed: `.crow/build-r-minor-sensitive-packages.yaml` — superseded by the integrated
|
||||
flow (it was manual, alpine-3.21-only, and issue-list-driven).
|
||||
- Out of scope (call out, do not change here): `weekly-rebuild-missing-*`,
|
||||
`archive-missed-packages`, and the audit workflows. Revisit separately if per-minor
|
||||
rebuilds are wanted there too.
|
||||
|
||||
## Open risks / notes
|
||||
|
||||
- **R-version discovery**: assumes `/opt/R/<version>/bin/R` layout and that the primary
|
||||
`R_VERSION` is one of the installed versions. Parse minor as `major.minor` from each
|
||||
discovered version; dedupe by minor (build once per minor even if two patch releases
|
||||
coexist).
|
||||
- **Per-minor index correctness**: clients resolving `bin/<os>/contrib/<x.y>/` require the
|
||||
per-minor `PACKAGES` to exist; the `upload_package_index()` enhancement is a hard
|
||||
dependency for the sensitive artifacts to be usable. Verify against a real client
|
||||
install before declaring done.
|
||||
- **Classification cost**: bounded by downloading sources only for the compiled,
|
||||
non-LinkingTo-risky subset in install-deps. Worth measuring on a full run; if still too
|
||||
heavy, consider caching classifications keyed by package+version in the metadata DB.
|
||||
- **Double download**: install-deps classification downloads some sources that the build
|
||||
step re-downloads. Acceptable for now; the metadata-DB cache above would also remove this.
|
||||
Loading…
Reference in a new issue