build-cran-binaries/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md

# Pipeline R-minor Builds Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Make the full (`build-all-versions-*`) and iterative (`process-updates-*`) pipelines build R-minor-sensitive packages once per installed R minor, and everything else once into the generic slot.

**Architecture:** A precompute step classifies packages (cheap CRAN-metadata rules + source download only for the ambiguous compiled subset) and emits a sensitivity flag. Build scripts pass a concrete `is_r_minor_sensitive` per package on the primary-R pass and re-run a sensitive-only pass under each additional `/opt/R/*` minor. Per-minor `PACKAGES` indexes are uploaded for each touched slot.

**Tech Stack:** R scripts under `local/`, crow/woodpecker CI YAML under `.crow/`, bincraft `>= 4.2.0` (see `2026-06-13-bincraft-r-minor-enhancements.md`).

**HARD DEPENDENCY:** bincraft `4.2.0` (the companion plan) must be merged and released first — Tasks 4-6 call `r_minor_detection`/`upload_package_index(r_minor=)` which only exist there. Tasks 1-3 (this repo's R scripts) can be written and unit-tested before the release.

---

### Task 1: pure metadata classifier helper

**Files:**
- Create: `local/r-minor-helpers.R`
- Test: `local/tests/test-r-minor-helpers.R`

- [ ] **Step 1: Write the failing test**

```r
# local/tests/test-r-minor-helpers.R
source(file.path("local", "r-minor-helpers.R"))

test_that("pure-r (NeedsCompilation != yes) is not sensitive", {
  expect_identical(classify_from_metadata("no", NA, c("Rcpp")), "not-sensitive")
  expect_identical(classify_from_metadata("", "Rcpp", c("Rcpp")), "not-sensitive")
})

test_that("LinkingTo a risky dep is sensitive (version constraints stripped)", {
  expect_identical(classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), "sensitive")
  expect_identical(classify_from_metadata("yes", "R6,\n  cpp11", c("Rcpp", "cpp11")), "sensitive")
})

test_that("compiled but no risky LinkingTo is ambiguous (needs source)", {
  expect_identical(classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), "ambiguous")
  expect_identical(classify_from_metadata("yes", NA, c("Rcpp")), "ambiguous")
})
```

- [ ] **Step 2: Run test to verify it fails**

Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'`
Expected: FAIL — `cannot open file 'local/r-minor-helpers.R'` / function not found.

- [ ] **Step 3: Write the helper**

```r
# local/r-minor-helpers.R
# Metadata-only ABI triage so the full build avoids downloading every source.
# Mirrors bincraft::abi_classify rules 1-2; "ambiguous" packages still need a
# source grep via bincraft::needs_per_minor_recompile().

classify_from_metadata <- function(needs_compilation, linking_to, risky_deps) {
  nc <- if (length(needs_compilation) == 0L || is.na(needs_compilation)) {
    ""
  } else {
    tolower(trimws(needs_compilation))
  }
  if (!identical(nc, "yes")) {
    return("not-sensitive")
  }
  lt <- if (length(linking_to) == 0L || is.na(linking_to)) "" else linking_to
  linked <- trimws(unlist(strsplit(lt, "[,\n]")))
  linked <- sub("\\s*\\(.*\\)$", "", linked) # strip "(>= x)" constraints
  linked <- linked[nzchar(linked)]
  if (any(linked %in% risky_deps)) {
    return("sensitive")
  }
  "ambiguous"
}
```

- [ ] **Step 4: Run test to verify it passes**

Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'`
Expected: PASS (3/3).

- [ ] **Step 5: Commit**

```bash
git add local/r-minor-helpers.R local/tests/test-r-minor-helpers.R
git commit -m "feat(local): metadata-only ABI triage helper"
```

---

### Task 2: add `r_minor_sensitive` to `packages-to-build.R`

**Files:**
- Modify: `local/packages-to-build.R` (after `pkgs` is finalized, ~line 142)

- [ ] **Step 1: Append the classification block**

At the end of `local/packages-to-build.R` (after `setorder(pkgs, Package, Version)`), add:

```r
### R-minor sensitivity (classify once per package, applied to all versions)
source(file.path("local", "r-minor-helpers.R"))
risky_deps <- bincraft::abi_risky_linking_deps()

release_meta <- data.table(
  Package = cran_release$Package,
  NeedsCompilation = cran_release$NeedsCompilation,
  LinkingTo = cran_release$LinkingTo
)

meta <- release_meta[Package %in% unique(pkgs$Package)]
meta[, triage := mapply(
  classify_from_metadata,
  NeedsCompilation,
  LinkingTo,
  MoreArgs = list(risky_deps = risky_deps)
)]

# Only the "ambiguous" compiled packages need a source grep.
ambiguous <- meta[triage == "ambiguous", Package]
sensitive_ambiguous <- character()
if (length(ambiguous) > 0L) {
  tmp_src <- file.path(tempdir(), "abi_src")
  dir.create(tmp_src, showWarnings = FALSE, recursive = TRUE)
  sens <- vapply(ambiguous, function(pkg) {
    out <- tryCatch({
      dl <- utils::download.packages(
        pkg, destdir = tmp_src,
        repos = "https://cloud.r-project.org", quiet = TRUE
      )
      isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L])))
    }, error = function(e) TRUE) # fail safe: treat as sensitive
    out
  }, logical(1L))
  sensitive_ambiguous <- ambiguous[sens]
}

sensitive_pkgs <- unique(c(meta[triage == "sensitive", Package], sensitive_ambiguous))
pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs]
sprintf("R-minor-sensitive packages: %s of %s", length(sensitive_pkgs), uniqueN(pkgs$Package))
```

- [ ] **Step 2: Verify the script parses and the column is added (offline smoke)**

Run:
```bash
R -q -e '
  library(data.table)
  source("local/r-minor-helpers.R")
  pkgs <- data.table(Package = c("A","B"), Version = c("1","1"))
  cran_release <- data.frame(Package = c("A","B"),
    NeedsCompilation = c("no","yes"), LinkingTo = c(NA,"Rcpp"),
    stringsAsFactors = FALSE)
  risky_deps <- c("Rcpp")
  release_meta <- as.data.table(cran_release)
  meta <- release_meta[Package %in% unique(pkgs$Package)]
  meta[, triage := mapply(classify_from_metadata, NeedsCompilation, LinkingTo,
    MoreArgs = list(risky_deps = risky_deps))]
  sensitive_pkgs <- meta[triage == "sensitive", Package]
  pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs]
  stopifnot(identical(pkgs$r_minor_sensitive, c(FALSE, TRUE)))
  cat("OK\n")'
```
Expected: prints `OK` (B classified sensitive via LinkingTo, A not).

- [ ] **Step 3: Persist the sensitive subset in install-deps**

In `.crow/build-all-versions-install-deps-amd64.yaml` and `...-arm64.yaml`, change the precompute command (line ~38) from saving only `pkgs_to_build.rds` to also saving the subset:

```yaml
      - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))"
```

- [ ] **Step 4: Commit**

```bash
git add local/packages-to-build.R .crow/build-all-versions-install-deps-amd64.yaml .crow/build-all-versions-install-deps-arm64.yaml
git commit -m "feat(full): classify r-minor sensitivity in install-deps precompute"
```

---

### Task 3: per-package flag + `--sensitive-only` in `build-all.R`

**Files:**
- Modify: `local/build-all.R`
- Test: `local/tests/test-build-all-args.R` (new — covers the arg/flag parsing seam only)

- [ ] **Step 1: Write the failing test for the arg parser**

```r
# local/tests/test-build-all-args.R
source(file.path("local", "r-minor-helpers.R"))

test_that("parse_build_args splits flags from positionals", {
  a <- parse_build_args(c("--sensitive-only", "4", "2", "8"))
  expect_true(a$sensitive_only)
  expect_identical(a$split_into, 4L)
  expect_identical(a$split_index, 2L)
  expect_identical(a$ncpus, 8L)

  b <- parse_build_args(c("4", "2", "8"))
  expect_false(b$sensitive_only)
  expect_identical(b$split_into, 4L)
})
```

- [ ] **Step 2: Add `parse_build_args` to the helpers and run the test**

Append to `local/r-minor-helpers.R`:

```r
parse_build_args <- function(args) {
  sensitive_only <- "--sensitive-only" %in% args
  pos <- args[!startsWith(args, "--")]
  list(
    sensitive_only = sensitive_only,
    split_into = as.integer(pos[1L]),
    split_index = as.integer(pos[2L]),
    ncpus = as.integer(pos[3L])
  )
}
```

Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-build-all-args.R")'`
Expected: PASS (1/1, 5 expectations).

- [ ] **Step 3: Rewire `build-all.R` to use the parser, the flag, and the per-row sensitivity**

Replace the arg-parsing header and the `mapply` build loop of `local/build-all.R`. New top:

```r
sink(stdout(), type = "message")
options(crayon.enabled = TRUE, future.globals.onReference = NULL)
source(file.path("local", "r-minor-helpers.R"))

args <- commandArgs(trailingOnly = TRUE)
parsed <- parse_build_args(args)
split_into <- parsed$split_into
split_index <- parsed$split_index
ncpus <- parsed$ncpus
sensitive_only <- parsed$sensitive_only
options(Ncpus = ncpus)

library(bincraft, quietly = TRUE)
library(future)
plan("sequential")

pkgs <- if (sensitive_only) {
  readRDS("/mnt/cache/packages/r_minor_sensitive_pkgs.rds")
} else {
  readRDS("/mnt/cache/packages/pkgs_to_build.rds")
}
# Back-compat: tolerate an older RDS without the column (treat all as non-sensitive)
if (is.null(pkgs$r_minor_sensitive)) pkgs$r_minor_sensitive <- FALSE
sprintf("Total# of remaining package versions: %s (sensitive_only=%s)", nrow(pkgs), sensitive_only)
```

Keep the existing chunk-split, exclude-list, and `s3_cache` lines unchanged. Then replace the `mapply(...)` call with:

```r
n <- nrow(chunk)
mapply(function(pkg, ver, sens, i) {
  cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens))
  bincraft::build_binary_package(
    pkg,
    tag = ver,
    is_r_minor_sensitive = isTRUE(sens),
    s3_endpoint = "https://s3.eu-central-003.backblazeb2.com",
    s3_region = "eu-central-003",
    s3_bucket = "devxy-rpkgs-binaries",
    s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"),
    s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"),
    s3_package_cache = s3_cache,
    metadata_db_host = "r-binaries.devxy.io",
    metadata_db_name = "build_metadata",
    metadata_db_table = "single_builds",
    metadata_db_user = "rpkgs",
    metadata_db_password = Sys.getenv("PGPASS"),
    metadata_db_sslmode = "require",
    metadata_db_port = 15432,
    archive = TRUE,
    upload = TRUE,
    store_build_metadata = TRUE
  )
}, chunk$Package, chunk$Version, chunk$r_minor_sensitive, seq_len(n))
```

Note: in `--sensitive-only` mode every row already has `r_minor_sensitive == TRUE`, so the same loop builds them into per-minor slots; no special-casing needed.

- [ ] **Step 4: Verify the script parses**

Run: `R -q -e 'invisible(parse(file="local/build-all.R")); cat("parsed OK\n")'`
Expected: `parsed OK`.

- [ ] **Step 5: Commit**

```bash
git add local/build-all.R local/r-minor-helpers.R local/tests/test-build-all-args.R
git commit -m "feat(full): per-package r-minor flag and --sensitive-only mode in build-all.R"
```

---

### Task 4: multi-R loop + per-minor index in `build-all-versions-*.yaml`

> Depends on bincraft 4.2.0 (`upload_package_index(r_minor=)`).

**Files:**
- Modify: `.crow/build-all-versions-amd64.yaml`, `.crow/build-all-versions-arm64.yaml`

- [ ] **Step 1: Add the sensitive-only multi-R pass after the primary build**

In each file's `commands:` (after the existing `Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS` line and before the `process_unarchived_pkgs` line), insert this multiline command:

```yaml
      - |
        PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
        for RBIN in /opt/R/*/bin/R; do
          RV=$(basename "$(dirname "$(dirname "$RBIN")")")
          RMINOR=$(echo "$RV" | cut -d. -f1-2)
          [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue
          echo "=== R-minor-sensitive pass under R $RV ==="
          R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$RBIN" Rscript local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true
        done
```

(`$XVFB`/`$XVFB_ARGS` are already defined on the preceding line in this workflow.) Note the primary pass still runs `build-all.R` without `--sensitive-only`, building everything and routing sensitive packages into the primary minor slot.

- [ ] **Step 2: Add a per-minor index upload step**

Append a new step after the build step (mirroring the existing CDN/index pattern). Add to `commands` of a new `Upload per-minor indexes` step (or extend the existing index handling) the following, which uploads the generic index plus one per discovered minor:

```yaml
      - |
        CODENAME=$(/opt/R/$R_VERSION/bin/Rscript -e "cat(bincraft::set_codename(NULL))")
        # generic slot
        /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))"
        for RBIN in /opt/R/*/bin/R; do
          RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2)
          /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true
        done
```

- [ ] **Step 3: Validate YAML**

Run: `R -q -e 'invisible(lapply(c(".crow/build-all-versions-amd64.yaml",".crow/build-all-versions-arm64.yaml"), yaml::yaml.load_file)); cat("yaml OK\n")'`
Expected: `yaml OK`.

- [ ] **Step 4: Commit**

```bash
git add .crow/build-all-versions-amd64.yaml .crow/build-all-versions-arm64.yaml
git commit -m "feat(full): sensitive-only multi-R passes and per-minor index upload"
```

---

### Task 5: classifier + multi-R in all `process-updates-*.yaml`

> Depends on bincraft 4.2.0 (`r_minor_detection`, `upload_package_index(r_minor=)`).

**Files (14):** every `.crow/process-updates-<plat>-<arch>.yaml`:
`alpine-322`, `alpine-323`, `redhat-8`, `redhat-9`, `redhat-10`, `ubuntu-2204`, `ubuntu-2404` × `amd64`, `arm64`.

The transform is identical in shape; only the already-present `platform=`, `R_VERSION`, `ARCH`, and the index `codename=` differ per file (leave those as-is).

- [ ] **Step 1: Edit the `process_cran_updates` call (primary pass)**

In each file, inside the single long `bincraft::process_cran_updates(...)` argument list, add `r_minor_detection = 'classifier', ` immediately before `s3_endpoint = ...`. Leave all other args unchanged.

- [ ] **Step 2: Add the sensitive-only multi-R pass**

Immediately after the `process_cran_updates(...)` command line, insert (preserve the per-file `'<platform>'` string and `$INTERVAL`):

```yaml
      - |
        PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
        for RBIN in /opt/R/*/bin/R; do
          RV=$(basename "$(dirname "$(dirname "$RBIN")")")
          RMINOR=$(echo "$RV" | cut -d. -f1-2)
          [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue
          echo "=== R-minor-sensitive update pass under R $RV ==="
          xvfb-run "$RBIN" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '<PLATFORM>', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true
        done
```

Replace `<PLATFORM>` with that file's existing platform string (e.g. `alpine-323`, `redhat-9`, `ubuntu-2404`). `process_removed = FALSE` on the extra passes (removals are handled once by the primary pass).

- [ ] **Step 3: Add per-minor index upload**

After the existing `upload_package_index(codename = "<CODENAME>", ...)` line in each file, append (reusing the file's existing `codename`):

```yaml
      - |
        for RBIN in /opt/R/*/bin/R; do
          RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2)
          /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "<CODENAME>", r_minor = "'"$RMINOR"'", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' || true
        done
```

Replace `<CODENAME>` with the file's existing codename value (e.g. `alpine323`, `rhel9`, `ubuntu2404`).

- [ ] **Step 4: Verify every file changed and parses**

Run:
```bash
R -q -e 'fs <- Sys.glob(".crow/process-updates-*.yaml"); invisible(lapply(fs, yaml::yaml.load_file)); cat(length(fs), "files parse OK\n")'
grep -L "r_minor_detection = 'classifier'" .crow/process-updates-*.yaml
```
Expected: `14 files parse OK`, and the `grep -L` prints **nothing** (every file contains the classifier arg).

- [ ] **Step 5: Commit**

```bash
git add .crow/process-updates-*.yaml
git commit -m "feat(updates): classifier-driven r-minor builds + per-minor index across platforms"
```

---

### Task 6: pin bincraft 4.2.0, remove the standalone workflow

**Files:**
- Modify: all workflows pinning bincraft (`grep -rln "bincraft.git@v" .crow/`), and `.crow/build-all-versions-install-deps-*.yaml` (which installs from `main`)
- Delete: `.crow/build-r-minor-sensitive-packages.yaml`

- [ ] **Step 1: Bump the pin**

Run to find pins: `grep -rn "bincraft" .crow/ | grep -E "@v4\.1\.1|bincraft.git"`. In every `process-updates-*` file, change `bincraft.git@v4.1.1` (and the `packageVersion("bincraft") != "4.1.1"` guard) to `4.2.0`.

- [ ] **Step 2: Remove the superseded workflow**

```bash
git rm .crow/build-r-minor-sensitive-packages.yaml
```

- [ ] **Step 3: Verify no stale references**

Run: `grep -rn "build-r-minor-sensitive\|@v4.1.1\|!= \"4.1.1\"" .crow/`
Expected: no matches.

- [ ] **Step 4: Commit**

```bash
git add .crow/
git commit -m "chore: pin bincraft 4.2.0 and drop superseded standalone r-minor workflow"
```

---

### Task 7: integration smoke test (manual, gated on bincraft 4.2.0 release)

- [ ] **Step 1: Run one iterative platform manually** via crow against a short interval and confirm: a known risky package (e.g. one LinkingTo Rcpp) lands under `…/contrib/<x.y>/` for each installed minor, and a pure-r package lands only in the generic slot.

- [ ] **Step 2: Confirm a client install resolves the per-minor slot.** From an R `4.4` and an R `4.5` container:
```r
install.packages("<risky_pkg>", repos = "https://cran.devxy.io/<codename>")
library(<risky_pkg>)  # must load without "undefined symbol"
```
Expected: loads under both minors. If the per-minor `PACKAGES` is missing, revisit bincraft Task 2.

- [ ] **Step 3: Spot-check the full build** on one platform with a small `SPLIT_INTO`, verifying the sensitive-only extra passes ran and produced per-minor artifacts.

---

## Self-Review

- **Spec coverage:** install-deps precompute → Tasks 1-2; `build-all.R` per-row flag + `--sensitive-only` → Task 3; full-build multi-R loop + per-minor index → Task 4; iterative classifier + multi-R + per-minor index → Task 5; remove standalone workflow + version pin → Task 6; per-minor index client-serviceability risk → Task 7 verification.
- **Type/name consistency:** `classify_from_metadata` and `parse_build_args` live in `local/r-minor-helpers.R` and are used in Tasks 2-3; the RDS column is `r_minor_sensitive` everywhere; the subset file is `/mnt/cache/packages/r_minor_sensitive_pkgs.rds` in install-deps (Task 2) and `build-all.R` (Task 3).
- **Placeholders:** `<PLATFORM>`/`<CODENAME>` in Task 5 are explicit per-file substitutions (the values already exist in each file), not unfilled blanks. No "TBD"/"handle errors" placeholders.
- **Dependency ordering:** Tasks 1-3 are pure R-script work, unit-testable now; Tasks 4-6 are gated on bincraft 4.2.0; Task 7 is post-release verification.