build-cran-binaries/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md

Pipeline R-minor Builds Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make the full (build-all-versions-*) and iterative (process-updates-*) pipelines build R-minor-sensitive packages once per installed R minor, and everything else once into the generic slot.

Architecture: A precompute step classifies packages (cheap CRAN-metadata rules + source download only for the ambiguous compiled subset) and emits a sensitivity flag. Build scripts pass a concrete is_r_minor_sensitive per package on the primary-R pass and re-run a sensitive-only pass under each additional /opt/R/* minor. Per-minor PACKAGES indexes are uploaded for each touched slot.

Tech Stack: R scripts under local/, crow/woodpecker CI YAML under .crow/, bincraft >= 4.2.0 (see 2026-06-13-bincraft-r-minor-enhancements.md).

HARD DEPENDENCY: bincraft 4.2.0 (the companion plan) must be merged and released first — Tasks 4-6 call r_minor_detection/upload_package_index(r_minor=) which only exist there. Tasks 1-3 (this repo's R scripts) can be written and unit-tested before the release.

---

Task 1: pure metadata classifier helper

Files:

• Create: local/r-minor-helpers.R

• Test: local/tests/test-r-minor-helpers.R

• Step 1: Write the failing test

# local/tests/test-r-minor-helpers.R
source(file.path("local", "r-minor-helpers.R"))

test_that("pure-r (NeedsCompilation != yes) is not sensitive", {
  expect_identical(classify_from_metadata("no", NA, c("Rcpp")), "not-sensitive")
  expect_identical(classify_from_metadata("", "Rcpp", c("Rcpp")), "not-sensitive")
})

test_that("LinkingTo a risky dep is sensitive (version constraints stripped)", {
  expect_identical(classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), "sensitive")
  expect_identical(classify_from_metadata("yes", "R6,\n  cpp11", c("Rcpp", "cpp11")), "sensitive")
})

test_that("compiled but no risky LinkingTo is ambiguous (needs source)", {
  expect_identical(classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), "ambiguous")
  expect_identical(classify_from_metadata("yes", NA, c("Rcpp")), "ambiguous")
})

• Step 2: Run test to verify it fails

Run: R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")' Expected: FAIL — cannot open file 'local/r-minor-helpers.R' / function not found.

• Step 3: Write the helper

# local/r-minor-helpers.R
# Metadata-only ABI triage so the full build avoids downloading every source.
# Mirrors bincraft::abi_classify rules 1-2; "ambiguous" packages still need a
# source grep via bincraft::needs_per_minor_recompile().

classify_from_metadata <- function(needs_compilation, linking_to, risky_deps) {
  nc <- if (length(needs_compilation) == 0L || is.na(needs_compilation)) {
    ""
  } else {
    tolower(trimws(needs_compilation))
  }
  if (!identical(nc, "yes")) {
    return("not-sensitive")
  }
  lt <- if (length(linking_to) == 0L || is.na(linking_to)) "" else linking_to
  linked <- trimws(unlist(strsplit(lt, "[,\n]")))
  linked <- sub("\\s*\\(.*\\)$", "", linked) # strip "(>= x)" constraints
  linked <- linked[nzchar(linked)]
  if (any(linked %in% risky_deps)) {
    return("sensitive")
  }
  "ambiguous"
}

• Step 4: Run test to verify it passes

Run: R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")' Expected: PASS (3/3).

• Step 5: Commit

git add local/r-minor-helpers.R local/tests/test-r-minor-helpers.R
git commit -m "feat(local): metadata-only ABI triage helper"

---

Task 2: add r_minor_sensitive to packages-to-build.R

Files:

• Modify: local/packages-to-build.R (after pkgs is finalized, ~line 142)

• Step 1: Append the classification block

At the end of local/packages-to-build.R (after setorder(pkgs, Package, Version)), add:

### R-minor sensitivity (classify once per package, applied to all versions)
source(file.path("local", "r-minor-helpers.R"))
risky_deps <- bincraft::abi_risky_linking_deps()

release_meta <- data.table(
  Package = cran_release$Package,
  NeedsCompilation = cran_release$NeedsCompilation,
  LinkingTo = cran_release$LinkingTo
)

meta <- release_meta[Package %in% unique(pkgs$Package)]
meta[, triage := mapply(
  classify_from_metadata,
  NeedsCompilation,
  LinkingTo,
  MoreArgs = list(risky_deps = risky_deps)
)]

# Only the "ambiguous" compiled packages need a source grep.
ambiguous <- meta[triage == "ambiguous", Package]
sensitive_ambiguous <- character()
if (length(ambiguous) > 0L) {
  tmp_src <- file.path(tempdir(), "abi_src")
  dir.create(tmp_src, showWarnings = FALSE, recursive = TRUE)
  sens <- vapply(ambiguous, function(pkg) {
    out <- tryCatch({
      dl <- utils::download.packages(
        pkg, destdir = tmp_src,
        repos = "https://cloud.r-project.org", quiet = TRUE
      )
      isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L])))
    }, error = function(e) TRUE) # fail safe: treat as sensitive
    out
  }, logical(1L))
  sensitive_ambiguous <- ambiguous[sens]
}

sensitive_pkgs <- unique(c(meta[triage == "sensitive", Package], sensitive_ambiguous))
pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs]
sprintf("R-minor-sensitive packages: %s of %s", length(sensitive_pkgs), uniqueN(pkgs$Package))

• Step 2: Verify the script parses and the column is added (offline smoke)

Run:

R -q -e '
  library(data.table)
  source("local/r-minor-helpers.R")
  pkgs <- data.table(Package = c("A","B"), Version = c("1","1"))
  cran_release <- data.frame(Package = c("A","B"),
    NeedsCompilation = c("no","yes"), LinkingTo = c(NA,"Rcpp"),
    stringsAsFactors = FALSE)
  risky_deps <- c("Rcpp")
  release_meta <- as.data.table(cran_release)
  meta <- release_meta[Package %in% unique(pkgs$Package)]
  meta[, triage := mapply(classify_from_metadata, NeedsCompilation, LinkingTo,
    MoreArgs = list(risky_deps = risky_deps))]
  sensitive_pkgs <- meta[triage == "sensitive", Package]
  pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs]
  stopifnot(identical(pkgs$r_minor_sensitive, c(FALSE, TRUE)))
  cat("OK\n")'

Expected: prints OK (B classified sensitive via LinkingTo, A not).

• Step 3: Persist the sensitive subset in install-deps

In .crow/build-all-versions-install-deps-amd64.yaml and ...-arm64.yaml, change the precompute command (line ~38) from saving only pkgs_to_build.rds to also saving the subset:

      - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))"

• Step 4: Commit

git add local/packages-to-build.R .crow/build-all-versions-install-deps-amd64.yaml .crow/build-all-versions-install-deps-arm64.yaml
git commit -m "feat(full): classify r-minor sensitivity in install-deps precompute"

---

Task 3: per-package flag + --sensitive-only in build-all.R

Files:

• Modify: local/build-all.R

• Test: local/tests/test-build-all-args.R (new — covers the arg/flag parsing seam only)

• Step 1: Write the failing test for the arg parser

# local/tests/test-build-all-args.R
source(file.path("local", "r-minor-helpers.R"))

test_that("parse_build_args splits flags from positionals", {
  a <- parse_build_args(c("--sensitive-only", "4", "2", "8"))
  expect_true(a$sensitive_only)
  expect_identical(a$split_into, 4L)
  expect_identical(a$split_index, 2L)
  expect_identical(a$ncpus, 8L)

  b <- parse_build_args(c("4", "2", "8"))
  expect_false(b$sensitive_only)
  expect_identical(b$split_into, 4L)
})

• Step 2: Add parse_build_args to the helpers and run the test

Append to local/r-minor-helpers.R:

parse_build_args <- function(args) {
  sensitive_only <- "--sensitive-only" %in% args
  pos <- args[!startsWith(args, "--")]
  list(
    sensitive_only = sensitive_only,
    split_into = as.integer(pos[1L]),
    split_index = as.integer(pos[2L]),
    ncpus = as.integer(pos[3L])
  )
}

Run: R -q -e 'library(testthat); testthat::test_file("local/tests/test-build-all-args.R")' Expected: PASS (1/1, 5 expectations).

• Step 3: Rewire build-all.R to use the parser, the flag, and the per-row sensitivity

Replace the arg-parsing header and the mapply build loop of local/build-all.R. New top:

sink(stdout(), type = "message")
options(crayon.enabled = TRUE, future.globals.onReference = NULL)
source(file.path("local", "r-minor-helpers.R"))

args <- commandArgs(trailingOnly = TRUE)
parsed <- parse_build_args(args)
split_into <- parsed$split_into
split_index <- parsed$split_index
ncpus <- parsed$ncpus
sensitive_only <- parsed$sensitive_only
options(Ncpus = ncpus)

library(bincraft, quietly = TRUE)
library(future)
plan("sequential")

pkgs <- if (sensitive_only) {
  readRDS("/mnt/cache/packages/r_minor_sensitive_pkgs.rds")
} else {
  readRDS("/mnt/cache/packages/pkgs_to_build.rds")
}
# Back-compat: tolerate an older RDS without the column (treat all as non-sensitive)
if (is.null(pkgs$r_minor_sensitive)) pkgs$r_minor_sensitive <- FALSE
sprintf("Total# of remaining package versions: %s (sensitive_only=%s)", nrow(pkgs), sensitive_only)

Keep the existing chunk-split, exclude-list, and s3_cache lines unchanged. Then replace the mapply(...) call with:

n <- nrow(chunk)
mapply(function(pkg, ver, sens, i) {
  cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens))
  bincraft::build_binary_package(
    pkg,
    tag = ver,
    is_r_minor_sensitive = isTRUE(sens),
    s3_endpoint = "https://s3.eu-central-003.backblazeb2.com",
    s3_region = "eu-central-003",
    s3_bucket = "devxy-rpkgs-binaries",
    s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"),
    s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"),
    s3_package_cache = s3_cache,
    metadata_db_host = "r-binaries.devxy.io",
    metadata_db_name = "build_metadata",
    metadata_db_table = "single_builds",
    metadata_db_user = "rpkgs",
    metadata_db_password = Sys.getenv("PGPASS"),
    metadata_db_sslmode = "require",
    metadata_db_port = 15432,
    archive = TRUE,
    upload = TRUE,
    store_build_metadata = TRUE
  )
}, chunk$Package, chunk$Version, chunk$r_minor_sensitive, seq_len(n))

Note: in --sensitive-only mode every row already has r_minor_sensitive == TRUE, so the same loop builds them into per-minor slots; no special-casing needed.

• Step 4: Verify the script parses

Run: R -q -e 'invisible(parse(file="local/build-all.R")); cat("parsed OK\n")' Expected: parsed OK.

• Step 5: Commit

git add local/build-all.R local/r-minor-helpers.R local/tests/test-build-all-args.R
git commit -m "feat(full): per-package r-minor flag and --sensitive-only mode in build-all.R"

---

Task 4: multi-R loop + per-minor index in build-all-versions-*.yaml

Depends on bincraft 4.2.0 (upload_package_index(r_minor=)).

Files:

• Modify: .crow/build-all-versions-amd64.yaml, .crow/build-all-versions-arm64.yaml

• Step 1: Add the sensitive-only multi-R pass after the primary build

In each file's commands: (after the existing Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS line and before the process_unarchived_pkgs line), insert this multiline command:

      - |
        PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
        for RBIN in /opt/R/*/bin/R; do
          RV=$(basename "$(dirname "$(dirname "$RBIN")")")
          RMINOR=$(echo "$RV" | cut -d. -f1-2)
          [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue
          echo "=== R-minor-sensitive pass under R $RV ==="
          R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$RBIN" Rscript local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true
        done

($XVFB/$XVFB_ARGS are already defined on the preceding line in this workflow.) Note the primary pass still runs build-all.R without --sensitive-only, building everything and routing sensitive packages into the primary minor slot.

• Step 2: Add a per-minor index upload step

Append a new step after the build step (mirroring the existing CDN/index pattern). Add to commands of a new Upload per-minor indexes step (or extend the existing index handling) the following, which uploads the generic index plus one per discovered minor:

      - |
        CODENAME=$(/opt/R/$R_VERSION/bin/Rscript -e "cat(bincraft::set_codename(NULL))")
        # generic slot
        /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))"
        for RBIN in /opt/R/*/bin/R; do
          RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2)
          /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true
        done

• Step 3: Validate YAML

Run: R -q -e 'invisible(lapply(c(".crow/build-all-versions-amd64.yaml",".crow/build-all-versions-arm64.yaml"), yaml::yaml.load_file)); cat("yaml OK\n")' Expected: yaml OK.

• Step 4: Commit

git add .crow/build-all-versions-amd64.yaml .crow/build-all-versions-arm64.yaml
git commit -m "feat(full): sensitive-only multi-R passes and per-minor index upload"

---

Task 5: classifier + multi-R in all process-updates-*.yaml

Depends on bincraft 4.2.0 (r_minor_detection, upload_package_index(r_minor=)).

Files (14): every .crow/process-updates-<plat>-<arch>.yaml: alpine-322, alpine-323, redhat-8, redhat-9, redhat-10, ubuntu-2204, ubuntu-2404 × amd64, arm64.

The transform is identical in shape; only the already-present platform=, R_VERSION, ARCH, and the index codename= differ per file (leave those as-is).

• Step 1: Edit the process_cran_updates call (primary pass)

In each file, inside the single long bincraft::process_cran_updates(...) argument list, add r_minor_detection = 'classifier', immediately before s3_endpoint = .... Leave all other args unchanged.

• Step 2: Add the sensitive-only multi-R pass

Immediately after the process_cran_updates(...) command line, insert (preserve the per-file '<platform>' string and $INTERVAL):

      - |
        PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
        for RBIN in /opt/R/*/bin/R; do
          RV=$(basename "$(dirname "$(dirname "$RBIN")")")
          RMINOR=$(echo "$RV" | cut -d. -f1-2)
          [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue
          echo "=== R-minor-sensitive update pass under R $RV ==="
          xvfb-run "$RBIN" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '<PLATFORM>', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true
        done

Replace <PLATFORM> with that file's existing platform string (e.g. alpine-323, redhat-9, ubuntu-2404). process_removed = FALSE on the extra passes (removals are handled once by the primary pass).

• Step 3: Add per-minor index upload

After the existing upload_package_index(codename = "<CODENAME>", ...) line in each file, append (reusing the file's existing codename):

      - |
        for RBIN in /opt/R/*/bin/R; do
          RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2)
          /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "<CODENAME>", r_minor = "'"$RMINOR"'", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' || true
        done

Replace <CODENAME> with the file's existing codename value (e.g. alpine323, rhel9, ubuntu2404).

• Step 4: Verify every file changed and parses

Run:

R -q -e 'fs <- Sys.glob(".crow/process-updates-*.yaml"); invisible(lapply(fs, yaml::yaml.load_file)); cat(length(fs), "files parse OK\n")'
grep -L "r_minor_detection = 'classifier'" .crow/process-updates-*.yaml

Expected: 14 files parse OK, and the grep -L prints nothing (every file contains the classifier arg).

• Step 5: Commit

git add .crow/process-updates-*.yaml
git commit -m "feat(updates): classifier-driven r-minor builds + per-minor index across platforms"

---

Task 6: pin bincraft 4.2.0, remove the standalone workflow

Files:

• Modify: all workflows pinning bincraft (grep -rln "bincraft.git@v" .crow/), and .crow/build-all-versions-install-deps-*.yaml (which installs from main)

• Delete: .crow/build-r-minor-sensitive-packages.yaml

• Step 1: Bump the pin

Run to find pins: grep -rn "bincraft" .crow/ | grep -E "@v4\.1\.1|bincraft.git". In every process-updates-* file, change bincraft.git@v4.1.1 (and the packageVersion("bincraft") != "4.1.1" guard) to 4.2.0.

• Step 2: Remove the superseded workflow

git rm .crow/build-r-minor-sensitive-packages.yaml

• Step 3: Verify no stale references

Run: grep -rn "build-r-minor-sensitive\|@v4.1.1\|!= \"4.1.1\"" .crow/ Expected: no matches.

• Step 4: Commit

git add .crow/
git commit -m "chore: pin bincraft 4.2.0 and drop superseded standalone r-minor workflow"

---

Task 7: integration smoke test (manual, gated on bincraft 4.2.0 release)

• Step 1: Run one iterative platform manually via crow against a short interval and confirm: a known risky package (e.g. one LinkingTo Rcpp) lands under …/contrib/<x.y>/ for each installed minor, and a pure-r package lands only in the generic slot.

• Step 2: Confirm a client install resolves the per-minor slot. From an R 4.4 and an R 4.5 container:

install.packages("<risky_pkg>", repos = "https://cran.devxy.io/<codename>")
library(<risky_pkg>)  # must load without "undefined symbol"

Expected: loads under both minors. If the per-minor PACKAGES is missing, revisit bincraft Task 2.

• Step 3: Spot-check the full build on one platform with a small SPLIT_INTO, verifying the sensitive-only extra passes ran and produced per-minor artifacts.

---

Self-Review

• Spec coverage: install-deps precompute → Tasks 1-2; build-all.R per-row flag + --sensitive-only → Task 3; full-build multi-R loop + per-minor index → Task 4; iterative classifier + multi-R + per-minor index → Task 5; remove standalone workflow + version pin → Task 6; per-minor index client-serviceability risk → Task 7 verification.
• Type/name consistency: classify_from_metadata and parse_build_args live in local/r-minor-helpers.R and are used in Tasks 2-3; the RDS column is r_minor_sensitive everywhere; the subset file is /mnt/cache/packages/r_minor_sensitive_pkgs.rds in install-deps (Task 2) and build-all.R (Task 3).
• Placeholders: <PLATFORM>/<CODENAME> in Task 5 are explicit per-file substitutions (the values already exist in each file), not unfilled blanks. No "TBD"/"handle errors" placeholders.
• Dependency ordering: Tasks 1-3 are pure R-script work, unit-testable now; Tasks 4-6 are gated on bincraft 4.2.0; Task 7 is post-release verification.