docs: implementation plans for R-minor-sensitive builds

Split into two sequenced plans: bincraft 4.2.0 enhancements (per-minor index +
classifier-driven process_cran_updates) and the pipeline integration (full +
iterative). Spec revised to classify at the orchestration layer instead of a
build_binary_package "auto" mode.
This commit is contained in:
Patrick Schratz 2026-06-13 19:10:55 +02:00
commit d328f261d2
Signed by: pat-s
GPG key ID: 3C6318841EF78925

View file

@ -0,0 +1,471 @@
# Pipeline R-minor Builds Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make the full (`build-all-versions-*`) and iterative (`process-updates-*`) pipelines build R-minor-sensitive packages once per installed R minor, and everything else once into the generic slot.
**Architecture:** A precompute step classifies packages (cheap CRAN-metadata rules + source download only for the ambiguous compiled subset) and emits a sensitivity flag. Build scripts pass a concrete `is_r_minor_sensitive` per package on the primary-R pass and re-run a sensitive-only pass under each additional `/opt/R/*` minor. Per-minor `PACKAGES` indexes are uploaded for each touched slot.
**Tech Stack:** R scripts under `local/`, crow/woodpecker CI YAML under `.crow/`, bincraft `>= 4.2.0` (see `2026-06-13-bincraft-r-minor-enhancements.md`).
**HARD DEPENDENCY:** bincraft `4.2.0` (the companion plan) must be merged and released first — Tasks 4-6 call `r_minor_detection`/`upload_package_index(r_minor=)` which only exist there. Tasks 1-3 (this repo's R scripts) can be written and unit-tested before the release.
---
### Task 1: pure metadata classifier helper
**Files:**
- Create: `local/r-minor-helpers.R`
- Test: `local/tests/test-r-minor-helpers.R`
- [ ] **Step 1: Write the failing test**
```r
# local/tests/test-r-minor-helpers.R
source(file.path("local", "r-minor-helpers.R"))
test_that("pure-r (NeedsCompilation != yes) is not sensitive", {
expect_identical(classify_from_metadata("no", NA, c("Rcpp")), "not-sensitive")
expect_identical(classify_from_metadata("", "Rcpp", c("Rcpp")), "not-sensitive")
})
test_that("LinkingTo a risky dep is sensitive (version constraints stripped)", {
expect_identical(classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), "sensitive")
expect_identical(classify_from_metadata("yes", "R6,\n cpp11", c("Rcpp", "cpp11")), "sensitive")
})
test_that("compiled but no risky LinkingTo is ambiguous (needs source)", {
expect_identical(classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), "ambiguous")
expect_identical(classify_from_metadata("yes", NA, c("Rcpp")), "ambiguous")
})
```
- [ ] **Step 2: Run test to verify it fails**
Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'`
Expected: FAIL — `cannot open file 'local/r-minor-helpers.R'` / function not found.
- [ ] **Step 3: Write the helper**
```r
# local/r-minor-helpers.R
# Metadata-only ABI triage so the full build avoids downloading every source.
# Mirrors bincraft::abi_classify rules 1-2; "ambiguous" packages still need a
# source grep via bincraft::needs_per_minor_recompile().
classify_from_metadata <- function(needs_compilation, linking_to, risky_deps) {
nc <- if (length(needs_compilation) == 0L || is.na(needs_compilation)) {
""
} else {
tolower(trimws(needs_compilation))
}
if (!identical(nc, "yes")) {
return("not-sensitive")
}
lt <- if (length(linking_to) == 0L || is.na(linking_to)) "" else linking_to
linked <- trimws(unlist(strsplit(lt, "[,\n]")))
linked <- sub("\\s*\\(.*\\)$", "", linked) # strip "(>= x)" constraints
linked <- linked[nzchar(linked)]
if (any(linked %in% risky_deps)) {
return("sensitive")
}
"ambiguous"
}
```
- [ ] **Step 4: Run test to verify it passes**
Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'`
Expected: PASS (3/3).
- [ ] **Step 5: Commit**
```bash
git add local/r-minor-helpers.R local/tests/test-r-minor-helpers.R
git commit -m "feat(local): metadata-only ABI triage helper"
```
---
### Task 2: add `r_minor_sensitive` to `packages-to-build.R`
**Files:**
- Modify: `local/packages-to-build.R` (after `pkgs` is finalized, ~line 142)
- [ ] **Step 1: Append the classification block**
At the end of `local/packages-to-build.R` (after `setorder(pkgs, Package, Version)`), add:
```r
### R-minor sensitivity (classify once per package, applied to all versions)
source(file.path("local", "r-minor-helpers.R"))
risky_deps <- bincraft::abi_risky_linking_deps()
release_meta <- data.table(
Package = cran_release$Package,
NeedsCompilation = cran_release$NeedsCompilation,
LinkingTo = cran_release$LinkingTo
)
meta <- release_meta[Package %in% unique(pkgs$Package)]
meta[, triage := mapply(
classify_from_metadata,
NeedsCompilation,
LinkingTo,
MoreArgs = list(risky_deps = risky_deps)
)]
# Only the "ambiguous" compiled packages need a source grep.
ambiguous <- meta[triage == "ambiguous", Package]
sensitive_ambiguous <- character()
if (length(ambiguous) > 0L) {
tmp_src <- file.path(tempdir(), "abi_src")
dir.create(tmp_src, showWarnings = FALSE, recursive = TRUE)
sens <- vapply(ambiguous, function(pkg) {
out <- tryCatch({
dl <- utils::download.packages(
pkg, destdir = tmp_src,
repos = "https://cloud.r-project.org", quiet = TRUE
)
isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L])))
}, error = function(e) TRUE) # fail safe: treat as sensitive
out
}, logical(1L))
sensitive_ambiguous <- ambiguous[sens]
}
sensitive_pkgs <- unique(c(meta[triage == "sensitive", Package], sensitive_ambiguous))
pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs]
sprintf("R-minor-sensitive packages: %s of %s", length(sensitive_pkgs), uniqueN(pkgs$Package))
```
- [ ] **Step 2: Verify the script parses and the column is added (offline smoke)**
Run:
```bash
R -q -e '
library(data.table)
source("local/r-minor-helpers.R")
pkgs <- data.table(Package = c("A","B"), Version = c("1","1"))
cran_release <- data.frame(Package = c("A","B"),
NeedsCompilation = c("no","yes"), LinkingTo = c(NA,"Rcpp"),
stringsAsFactors = FALSE)
risky_deps <- c("Rcpp")
release_meta <- as.data.table(cran_release)
meta <- release_meta[Package %in% unique(pkgs$Package)]
meta[, triage := mapply(classify_from_metadata, NeedsCompilation, LinkingTo,
MoreArgs = list(risky_deps = risky_deps))]
sensitive_pkgs <- meta[triage == "sensitive", Package]
pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs]
stopifnot(identical(pkgs$r_minor_sensitive, c(FALSE, TRUE)))
cat("OK\n")'
```
Expected: prints `OK` (B classified sensitive via LinkingTo, A not).
- [ ] **Step 3: Persist the sensitive subset in install-deps**
In `.crow/build-all-versions-install-deps-amd64.yaml` and `...-arm64.yaml`, change the precompute command (line ~38) from saving only `pkgs_to_build.rds` to also saving the subset:
```yaml
- /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))"
```
- [ ] **Step 4: Commit**
```bash
git add local/packages-to-build.R .crow/build-all-versions-install-deps-amd64.yaml .crow/build-all-versions-install-deps-arm64.yaml
git commit -m "feat(full): classify r-minor sensitivity in install-deps precompute"
```
---
### Task 3: per-package flag + `--sensitive-only` in `build-all.R`
**Files:**
- Modify: `local/build-all.R`
- Test: `local/tests/test-build-all-args.R` (new — covers the arg/flag parsing seam only)
- [ ] **Step 1: Write the failing test for the arg parser**
```r
# local/tests/test-build-all-args.R
source(file.path("local", "r-minor-helpers.R"))
test_that("parse_build_args splits flags from positionals", {
a <- parse_build_args(c("--sensitive-only", "4", "2", "8"))
expect_true(a$sensitive_only)
expect_identical(a$split_into, 4L)
expect_identical(a$split_index, 2L)
expect_identical(a$ncpus, 8L)
b <- parse_build_args(c("4", "2", "8"))
expect_false(b$sensitive_only)
expect_identical(b$split_into, 4L)
})
```
- [ ] **Step 2: Add `parse_build_args` to the helpers and run the test**
Append to `local/r-minor-helpers.R`:
```r
parse_build_args <- function(args) {
sensitive_only <- "--sensitive-only" %in% args
pos <- args[!startsWith(args, "--")]
list(
sensitive_only = sensitive_only,
split_into = as.integer(pos[1L]),
split_index = as.integer(pos[2L]),
ncpus = as.integer(pos[3L])
)
}
```
Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-build-all-args.R")'`
Expected: PASS (1/1, 5 expectations).
- [ ] **Step 3: Rewire `build-all.R` to use the parser, the flag, and the per-row sensitivity**
Replace the arg-parsing header and the `mapply` build loop of `local/build-all.R`. New top:
```r
sink(stdout(), type = "message")
options(crayon.enabled = TRUE, future.globals.onReference = NULL)
source(file.path("local", "r-minor-helpers.R"))
args <- commandArgs(trailingOnly = TRUE)
parsed <- parse_build_args(args)
split_into <- parsed$split_into
split_index <- parsed$split_index
ncpus <- parsed$ncpus
sensitive_only <- parsed$sensitive_only
options(Ncpus = ncpus)
library(bincraft, quietly = TRUE)
library(future)
plan("sequential")
pkgs <- if (sensitive_only) {
readRDS("/mnt/cache/packages/r_minor_sensitive_pkgs.rds")
} else {
readRDS("/mnt/cache/packages/pkgs_to_build.rds")
}
# Back-compat: tolerate an older RDS without the column (treat all as non-sensitive)
if (is.null(pkgs$r_minor_sensitive)) pkgs$r_minor_sensitive <- FALSE
sprintf("Total# of remaining package versions: %s (sensitive_only=%s)", nrow(pkgs), sensitive_only)
```
Keep the existing chunk-split, exclude-list, and `s3_cache` lines unchanged. Then replace the `mapply(...)` call with:
```r
n <- nrow(chunk)
mapply(function(pkg, ver, sens, i) {
cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens))
bincraft::build_binary_package(
pkg,
tag = ver,
is_r_minor_sensitive = isTRUE(sens),
s3_endpoint = "https://s3.eu-central-003.backblazeb2.com",
s3_region = "eu-central-003",
s3_bucket = "devxy-rpkgs-binaries",
s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"),
s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"),
s3_package_cache = s3_cache,
metadata_db_host = "r-binaries.devxy.io",
metadata_db_name = "build_metadata",
metadata_db_table = "single_builds",
metadata_db_user = "rpkgs",
metadata_db_password = Sys.getenv("PGPASS"),
metadata_db_sslmode = "require",
metadata_db_port = 15432,
archive = TRUE,
upload = TRUE,
store_build_metadata = TRUE
)
}, chunk$Package, chunk$Version, chunk$r_minor_sensitive, seq_len(n))
```
Note: in `--sensitive-only` mode every row already has `r_minor_sensitive == TRUE`, so the same loop builds them into per-minor slots; no special-casing needed.
- [ ] **Step 4: Verify the script parses**
Run: `R -q -e 'invisible(parse(file="local/build-all.R")); cat("parsed OK\n")'`
Expected: `parsed OK`.
- [ ] **Step 5: Commit**
```bash
git add local/build-all.R local/r-minor-helpers.R local/tests/test-build-all-args.R
git commit -m "feat(full): per-package r-minor flag and --sensitive-only mode in build-all.R"
```
---
### Task 4: multi-R loop + per-minor index in `build-all-versions-*.yaml`
> Depends on bincraft 4.2.0 (`upload_package_index(r_minor=)`).
**Files:**
- Modify: `.crow/build-all-versions-amd64.yaml`, `.crow/build-all-versions-arm64.yaml`
- [ ] **Step 1: Add the sensitive-only multi-R pass after the primary build**
In each file's `commands:` (after the existing `Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS` line and before the `process_unarchived_pkgs` line), insert this multiline command:
```yaml
- |
PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
for RBIN in /opt/R/*/bin/R; do
RV=$(basename "$(dirname "$(dirname "$RBIN")")")
RMINOR=$(echo "$RV" | cut -d. -f1-2)
[ "$RMINOR" = "$PRIMARY_MINOR" ] && continue
echo "=== R-minor-sensitive pass under R $RV ==="
R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$RBIN" Rscript local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true
done
```
(`$XVFB`/`$XVFB_ARGS` are already defined on the preceding line in this workflow.) Note the primary pass still runs `build-all.R` without `--sensitive-only`, building everything and routing sensitive packages into the primary minor slot.
- [ ] **Step 2: Add a per-minor index upload step**
Append a new step after the build step (mirroring the existing CDN/index pattern). Add to `commands` of a new `Upload per-minor indexes` step (or extend the existing index handling) the following, which uploads the generic index plus one per discovered minor:
```yaml
- |
CODENAME=$(/opt/R/$R_VERSION/bin/Rscript -e "cat(bincraft::set_codename(NULL))")
# generic slot
/opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))"
for RBIN in /opt/R/*/bin/R; do
RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2)
/opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true
done
```
- [ ] **Step 3: Validate YAML**
Run: `R -q -e 'invisible(lapply(c(".crow/build-all-versions-amd64.yaml",".crow/build-all-versions-arm64.yaml"), yaml::yaml.load_file)); cat("yaml OK\n")'`
Expected: `yaml OK`.
- [ ] **Step 4: Commit**
```bash
git add .crow/build-all-versions-amd64.yaml .crow/build-all-versions-arm64.yaml
git commit -m "feat(full): sensitive-only multi-R passes and per-minor index upload"
```
---
### Task 5: classifier + multi-R in all `process-updates-*.yaml`
> Depends on bincraft 4.2.0 (`r_minor_detection`, `upload_package_index(r_minor=)`).
**Files (14):** every `.crow/process-updates-<plat>-<arch>.yaml`:
`alpine-322`, `alpine-323`, `redhat-8`, `redhat-9`, `redhat-10`, `ubuntu-2204`, `ubuntu-2404` × `amd64`, `arm64`.
The transform is identical in shape; only the already-present `platform=`, `R_VERSION`, `ARCH`, and the index `codename=` differ per file (leave those as-is).
- [ ] **Step 1: Edit the `process_cran_updates` call (primary pass)**
In each file, inside the single long `bincraft::process_cran_updates(...)` argument list, add `r_minor_detection = 'classifier', ` immediately before `s3_endpoint = ...`. Leave all other args unchanged.
- [ ] **Step 2: Add the sensitive-only multi-R pass**
Immediately after the `process_cran_updates(...)` command line, insert (preserve the per-file `'<platform>'` string and `$INTERVAL`):
```yaml
- |
PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
for RBIN in /opt/R/*/bin/R; do
RV=$(basename "$(dirname "$(dirname "$RBIN")")")
RMINOR=$(echo "$RV" | cut -d. -f1-2)
[ "$RMINOR" = "$PRIMARY_MINOR" ] && continue
echo "=== R-minor-sensitive update pass under R $RV ==="
xvfb-run "$RBIN" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '<PLATFORM>', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true
done
```
Replace `<PLATFORM>` with that file's existing platform string (e.g. `alpine-323`, `redhat-9`, `ubuntu-2404`). `process_removed = FALSE` on the extra passes (removals are handled once by the primary pass).
- [ ] **Step 3: Add per-minor index upload**
After the existing `upload_package_index(codename = "<CODENAME>", ...)` line in each file, append (reusing the file's existing `codename`):
```yaml
- |
for RBIN in /opt/R/*/bin/R; do
RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2)
/opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "<CODENAME>", r_minor = "'"$RMINOR"'", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' || true
done
```
Replace `<CODENAME>` with the file's existing codename value (e.g. `alpine323`, `rhel9`, `ubuntu2404`).
- [ ] **Step 4: Verify every file changed and parses**
Run:
```bash
R -q -e 'fs <- Sys.glob(".crow/process-updates-*.yaml"); invisible(lapply(fs, yaml::yaml.load_file)); cat(length(fs), "files parse OK\n")'
grep -L "r_minor_detection = 'classifier'" .crow/process-updates-*.yaml
```
Expected: `14 files parse OK`, and the `grep -L` prints **nothing** (every file contains the classifier arg).
- [ ] **Step 5: Commit**
```bash
git add .crow/process-updates-*.yaml
git commit -m "feat(updates): classifier-driven r-minor builds + per-minor index across platforms"
```
---
### Task 6: pin bincraft 4.2.0, remove the standalone workflow
**Files:**
- Modify: all workflows pinning bincraft (`grep -rln "bincraft.git@v" .crow/`), and `.crow/build-all-versions-install-deps-*.yaml` (which installs from `main`)
- Delete: `.crow/build-r-minor-sensitive-packages.yaml`
- [ ] **Step 1: Bump the pin**
Run to find pins: `grep -rn "bincraft" .crow/ | grep -E "@v4\.1\.1|bincraft.git"`. In every `process-updates-*` file, change `bincraft.git@v4.1.1` (and the `packageVersion("bincraft") != "4.1.1"` guard) to `4.2.0`.
- [ ] **Step 2: Remove the superseded workflow**
```bash
git rm .crow/build-r-minor-sensitive-packages.yaml
```
- [ ] **Step 3: Verify no stale references**
Run: `grep -rn "build-r-minor-sensitive\|@v4.1.1\|!= \"4.1.1\"" .crow/`
Expected: no matches.
- [ ] **Step 4: Commit**
```bash
git add .crow/
git commit -m "chore: pin bincraft 4.2.0 and drop superseded standalone r-minor workflow"
```
---
### Task 7: integration smoke test (manual, gated on bincraft 4.2.0 release)
- [ ] **Step 1: Run one iterative platform manually** via crow against a short interval and confirm: a known risky package (e.g. one LinkingTo Rcpp) lands under `…/contrib/<x.y>/` for each installed minor, and a pure-r package lands only in the generic slot.
- [ ] **Step 2: Confirm a client install resolves the per-minor slot.** From an R `4.4` and an R `4.5` container:
```r
install.packages("<risky_pkg>", repos = "https://cran.devxy.io/<codename>")
library(<risky_pkg>) # must load without "undefined symbol"
```
Expected: loads under both minors. If the per-minor `PACKAGES` is missing, revisit bincraft Task 2.
- [ ] **Step 3: Spot-check the full build** on one platform with a small `SPLIT_INTO`, verifying the sensitive-only extra passes ran and produced per-minor artifacts.
---
## Self-Review
- **Spec coverage:** install-deps precompute → Tasks 1-2; `build-all.R` per-row flag + `--sensitive-only` → Task 3; full-build multi-R loop + per-minor index → Task 4; iterative classifier + multi-R + per-minor index → Task 5; remove standalone workflow + version pin → Task 6; per-minor index client-serviceability risk → Task 7 verification.
- **Type/name consistency:** `classify_from_metadata` and `parse_build_args` live in `local/r-minor-helpers.R` and are used in Tasks 2-3; the RDS column is `r_minor_sensitive` everywhere; the subset file is `/mnt/cache/packages/r_minor_sensitive_pkgs.rds` in install-deps (Task 2) and `build-all.R` (Task 3).
- **Placeholders:** `<PLATFORM>`/`<CODENAME>` in Task 5 are explicit per-file substitutions (the values already exist in each file), not unfilled blanks. No "TBD"/"handle errors" placeholders.
- **Dependency ordering:** Tasks 1-3 are pure R-script work, unit-testable now; Tasks 4-6 are gated on bincraft 4.2.0; Task 7 is post-release verification.