pat-s
d328f261d2
Split into two sequenced plans: bincraft 4.2.0 enhancements (per-minor index + classifier-driven process_cran_updates) and the pipeline integration (full + iterative). Spec revised to classify at the orchestration layer instead of a build_binary_package "auto" mode.
155 lines
8.8 KiB
Markdown
155 lines
8.8 KiB
Markdown
# R-minor-sensitive binary builds — design
|
|
|
|
## Problem
|
|
|
|
CRAN binaries are currently built once, under a single R minor version, and served from a
|
|
single generic slot (`…/latest/src/contrib/`).
|
|
That is wrong for the minority of packages whose compiled code reaches into volatile R
|
|
internals: a binary built under R 4.5 will fail to load under R 4.4 with an
|
|
`undefined symbol` error.
|
|
|
|
bincraft v4.1.0+ can now detect these packages automatically via the ABI classifier
|
|
(`abi_classify()` / `needs_per_minor_recompile()`, added in bincraft PR #49,
|
|
<https://codefloe.com/rpkgs/bincraft/pulls/49>).
|
|
The exec-env images now ship multiple R minor versions under `/opt/R/`.
|
|
This design uses both to build R-minor-specific binaries for the sensitive packages only,
|
|
in both the iterative (`process-updates-*`) and full (`build-all-versions-*`) pipelines.
|
|
|
|
## Background: how bincraft already behaves
|
|
|
|
- `abi_classify(path)` returns a tier: `pure-r` (~78.6%), `safe-compiled` (~7.7%), or
|
|
`risky` (~13.6%). `needs_per_minor_recompile(path)` is the boolean wrapper
|
|
(`TRUE` iff `risky`). Both need the package source (DESCRIPTION + `src/`).
|
|
- `build_binary_package(…, is_r_minor_sensitive = TRUE)` uploads the artifact into a
|
|
per-minor slot `…/latest/src/contrib/<major.minor>/` and records `r_version` in the
|
|
build metadata. With `FALSE` it uses the generic slot. The minor is derived from the
|
|
**running interpreter** (`R.version`), so producing a 4.4 binary requires running
|
|
`/opt/R/4.4.x/bin/R`.
|
|
- `process_cran_updates()` orchestrates the iterative flow with a single
|
|
`is_r_minor_sensitive` bool for the whole run and no per-package classification or
|
|
R-version loop.
|
|
- `upload_package_index()` writes/uploads `PACKAGES*` for the **generic slot only**; it
|
|
has no per-minor support. The current standalone r-minor workflow never builds a
|
|
per-minor index, so per-minor slots are effectively unservable today.
|
|
|
|
## Core model
|
|
|
|
Collapse the three tiers to one boolean per package:
|
|
|
|
```
|
|
r_minor_sensitive := (abi_classify(pkg)$tier == "risky")
|
|
```
|
|
|
|
| group | tiers | built under | slot |
|
|
| ------------------------------ | ---------------------- | -------------------- | ---------------------------- |
|
|
| non-sensitive (~86%) | pure-r, safe-compiled | primary R only | generic `contrib/` |
|
|
| sensitive (~14%) | risky | every installed minor| per-minor `contrib/<x.y>/` |
|
|
|
|
- **Primary R** = the existing `R_VERSION` env var in each workflow. Its pass builds the
|
|
non-sensitive packages (generic slot) *and* the sensitive packages for its own minor slot.
|
|
- **Extra minors** = every other R version discovered by scanning `/opt/R/` at runtime.
|
|
Each runs a **sensitive-only** pass. An image with a single R version degrades cleanly
|
|
to just the primary pass.
|
|
- The loop over minors always lives at the shell/script layer (one `/opt/R/<ver>/bin/R`
|
|
invocation per minor), never inside a single `build_binary_package()` call.
|
|
|
|
Classification granularity: classify **once per package** (its release version) and apply
|
|
the resulting flag to all archived versions of that package. Tier rarely changes across
|
|
recent versions; this avoids multiplying source downloads.
|
|
|
|
## Full build — `build-all-versions-*` + `local/`
|
|
|
|
### install-deps step (`local/packages-to-build.R`)
|
|
|
|
After computing `pkgs_to_build`, add an `r_minor_sensitive` logical column:
|
|
|
|
1. Pull `NeedsCompilation` and `LinkingTo` from `tools::CRAN_package_db()` (already loaded).
|
|
2. Resolve cheaply, no download:
|
|
- `NeedsCompilation != "yes"` → not sensitive (rule 1, pure-r).
|
|
- `LinkingTo` references any `bincraft::abi_risky_linking_deps()` entry → sensitive
|
|
(rule 2).
|
|
3. For the remaining compiled, non-LinkingTo-risky packages only: download the source and
|
|
call `bincraft::needs_per_minor_recompile()` (rules 3/4).
|
|
4. Join the per-package flag onto every `(Package, Version)` row.
|
|
|
|
Outputs:
|
|
- `pkgs_to_build.rds` — now carries the `r_minor_sensitive` column.
|
|
- `r_minor_sensitive_pkgs.rds` — the sensitive subset, for the extra-minor passes.
|
|
|
|
### build step (`local/build-all.R` + `.crow/build-all-versions-{amd64,arm64}.yaml`)
|
|
|
|
- `build-all.R` gains an optional `--sensitive-only` mode (or an arg flag). In normal mode
|
|
it builds the full chunk, passing `is_r_minor_sensitive = <row flag>` per package. In
|
|
sensitive-only mode it reads `r_minor_sensitive_pkgs.rds`, intersects with its chunk, and
|
|
builds those with `is_r_minor_sensitive = TRUE`.
|
|
- The workflow step keeps the existing matrix split. After the primary `Rscript build-all.R`
|
|
invocation, a shell loop discovers non-primary `/opt/R/*` minors and runs
|
|
`Rscript build-all.R --sensitive-only <split> <index> <ncpus>` under each.
|
|
- Index upload step: generic index as today, plus a per-minor index for each minor slot
|
|
that received artifacts (depends on the `upload_package_index()` enhancement below).
|
|
|
|
## Iterative build — `process-updates-*`
|
|
|
|
Driven by the bincraft enhancement below; the single build step becomes:
|
|
|
|
1. Primary-R pass:
|
|
`process_cran_updates(…, r_minor_detection = "classifier")`.
|
|
Each updated/new package is classified; risky → primary minor slot, rest → generic slot.
|
|
Removed-package handling stays as-is.
|
|
2. Shell loop over non-primary `/opt/R/*` minors:
|
|
`process_cran_updates(…, r_minor_detection = "classifier", r_minor_sensitive_only = TRUE)`.
|
|
Builds only risky updates into their respective minor slots.
|
|
3. Index upload extended to cover each touched minor slot in addition to the generic slot.
|
|
|
|
## Required bincraft enhancements (separate PR, coordinated release)
|
|
|
|
`build_binary_package()` needs **no change** — it already accepts a concrete
|
|
`is_r_minor_sensitive` logical and routes the slot accordingly. Classification stays at
|
|
the orchestration layer (mirroring the full-build precompute), which keeps
|
|
`build_binary_package`'s pre-build S3 skip-check and source clone untouched.
|
|
|
|
1. `process_cran_updates()`:
|
|
- Add `r_minor_detection = c("none", "issue", "classifier")` (default `"none"` to
|
|
preserve current behavior; `"issue"` is today's `filter_r_minor_sensitive` path).
|
|
- Add `r_minor_sensitive_only` (default `FALSE`).
|
|
- With `"classifier"`: for each candidate `(name, version)`, clone the source to a temp
|
|
dir and call `bincraft::needs_per_minor_recompile()`; pass the resulting concrete
|
|
logical as `is_r_minor_sensitive` to `build_binary_package()`. When
|
|
`r_minor_sensitive_only = TRUE`, drop non-risky candidates before building.
|
|
- Implemented via a small internal helper `classify_r_minor_sensitive(name, tag,
|
|
source_org_url, local_clone_dir)` returning a logical — the unit-testable seam.
|
|
2. `upload_package_index()`:
|
|
- Add an `r_minor = NULL` argument. When non-NULL (e.g. `"4.4"`), point the remote dir
|
|
at `…/contrib/<r_minor>/` and write/upload `PACKAGES*` (and `Meta/archive.rds`) there,
|
|
mirroring the generic-slot logic. Extract the remote-dir construction into a pure
|
|
helper `package_index_remote_dir(s3_bucket, arch, codename, r_minor = NULL)` — the
|
|
unit-testable seam.
|
|
|
|
These two are the only bincraft changes; detection itself (PR #49) is already merged.
|
|
|
|
## Scope and cleanup
|
|
|
|
- In scope: `.crow/process-updates-*` (iterative) and `.crow/build-all-versions-*` +
|
|
`local/build-all.R` + `local/packages-to-build.R` (full), across amd64 and arm64 and all
|
|
platforms (alpine, ubuntu, redhat). The per-platform workflow files share the same edit.
|
|
- Removed: `.crow/build-r-minor-sensitive-packages.yaml` — superseded by the integrated
|
|
flow (it was manual, alpine-3.21-only, and issue-list-driven).
|
|
- Out of scope (call out, do not change here): `weekly-rebuild-missing-*`,
|
|
`archive-missed-packages`, and the audit workflows. Revisit separately if per-minor
|
|
rebuilds are wanted there too.
|
|
|
|
## Open risks / notes
|
|
|
|
- **R-version discovery**: assumes `/opt/R/<version>/bin/R` layout and that the primary
|
|
`R_VERSION` is one of the installed versions. Parse minor as `major.minor` from each
|
|
discovered version; dedupe by minor (build once per minor even if two patch releases
|
|
coexist).
|
|
- **Per-minor index correctness**: clients resolving `bin/<os>/contrib/<x.y>/` require the
|
|
per-minor `PACKAGES` to exist; the `upload_package_index()` enhancement is a hard
|
|
dependency for the sensitive artifacts to be usable. Verify against a real client
|
|
install before declaring done.
|
|
- **Classification cost**: bounded by downloading sources only for the compiled,
|
|
non-LinkingTo-risky subset in install-deps. Worth measuring on a full run; if still too
|
|
heavy, consider caching classifications keyed by package+version in the metadata DB.
|
|
- **Double download**: install-deps classification downloads some sources that the build
|
|
step re-downloads. Acceptable for now; the metadata-DB cache above would also remove this.
|