From d328f261d248bcaf439b45cfe0bcc6b66873ef2f Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 19:10:55 +0200 Subject: [PATCH] docs: implementation plans for R-minor-sensitive builds Split into two sequenced plans: bincraft 4.2.0 enhancements (per-minor index + classifier-driven process_cran_updates) and the pipeline integration (full + iterative). Spec revised to classify at the orchestration layer instead of a build_binary_package "auto" mode. --- ...026-06-13-bincraft-r-minor-enhancements.md | 470 +++++++++++++++++ .../2026-06-13-pipeline-r-minor-builds.md | 471 ++++++++++++++++++ ...6-06-13-r-minor-sensitive-builds-design.md | 33 +- 3 files changed, 961 insertions(+), 13 deletions(-) create mode 100644 docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md create mode 100644 docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md diff --git a/docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md b/docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md new file mode 100644 index 0000000..b52735a --- /dev/null +++ b/docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md @@ -0,0 +1,470 @@ +# bincraft R-minor Enhancements Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Teach bincraft to (a) build/serve per-R-minor package indexes and (b) drive `process_cran_updates()` from the ABI classifier so only risky packages are built per R minor. + +**Architecture:** Two additive, low-risk changes to the bincraft R package. `upload_package_index()` gains an `r_minor` argument so a per-minor slot (`…/contrib//`) gets its own `PACKAGES*`. `process_cran_updates()` gains `r_minor_detection`/`r_minor_sensitive_only` and classifies each candidate via `needs_per_minor_recompile()` (already in the package), passing a concrete `is_r_minor_sensitive` logical to the unchanged `build_binary_package()`. + +**Tech Stack:** R package, `testthat` (3e) + `mockery` for tests, `roxygen2` for docs, `s3fs`/`cranlike` for the index. Repo: `https://codefloe.com/rpkgs/bincraft` (this plan is executed in a clone of that repo, NOT in build-cran-binaries). + +**Pre-req:** PR #49 (ABI classifier) is already merged on `main`; `abi_classify()`, `needs_per_minor_recompile()`, `abi_risky_linking_deps()` are exported. + +--- + +### Task 1: `package_index_remote_dir()` pure helper + +**Files:** +- Modify: `R/package_index.R` (top of file, before `add_to_package_index`) +- Test: `tests/testthat/test-package_index.R` (new) + +- [ ] **Step 1: Write the failing test** + +```r +# tests/testthat/test-package_index.R +test_that("package_index_remote_dir builds the generic slot when r_minor is NULL", { + expect_identical( + package_index_remote_dir("bucket", "amd64", "alpine323"), + file.path("bucket", "amd64", "alpine323", "latest", "src", "contrib") + ) +}) + +test_that("package_index_remote_dir appends the minor slot when r_minor is set", { + expect_identical( + package_index_remote_dir("bucket", "amd64", "alpine323", r_minor = "4.4"), + file.path("bucket", "amd64", "alpine323", "latest", "src", "contrib", "4.4") + ) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-package_index.R")'` +Expected: FAIL — `could not find function "package_index_remote_dir"`. + +- [ ] **Step 3: Add the helper** + +```r +# R/package_index.R — add near the top, above add_to_package_index() +#' Build the S3 remote contrib dir for a package index +#' +#' @param r_minor Optional `"major.minor"` string (e.g. `"4.4"`). When non-NULL +#' the path points at the per-minor slot. +#' @keywords internal +package_index_remote_dir <- function(s3_bucket, arch, codename, r_minor = NULL) { + base <- file.path(s3_bucket, arch, codename, "latest", "src", "contrib") + if (is.null(r_minor)) { + base + } else { + file.path(base, r_minor) + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-package_index.R")'` +Expected: PASS (2/2). + +- [ ] **Step 5: Commit** + +```bash +git add R/package_index.R tests/testthat/test-package_index.R +git commit -m "feat(index): add package_index_remote_dir helper for per-minor slots" +``` + +--- + +### Task 2: per-minor support in `upload_package_index()` + +**Files:** +- Modify: `R/package_index.R` (`upload_package_index`, signature + the `remote_bin_dir` block at lines ~117-124; `add_to_package_index` for consistency) + +- [ ] **Step 1: Add `r_minor` to the signature and use the helper** + +In `upload_package_index()`, add `r_minor = NULL` to the argument list (after `arch = NULL`). Replace the inline `remote_bin_dir <- file.path(s3_bucket, arch, codename, "latest", "src", "contrib")` block with: + +```r + remote_bin_dir <- package_index_remote_dir(s3_bucket, arch, codename, r_minor) +``` + +Do the same replacement in `add_to_package_index()` (it has the identical inline construction); add `r_minor = NULL` to its signature too. + +- [ ] **Step 2: Add the roxygen param** + +Above `upload_package_index` and `add_to_package_index`, add: + +```r +#' @param r_minor Optional `"major.minor"` string. When set, the index is +#' written/read under the per-minor slot `…/contrib//` instead of the +#' generic `…/contrib/` slot. +``` + +- [ ] **Step 3: Regenerate docs** + +Run: `R -q -e 'devtools::document()'` +Expected: updated `man/upload_package_index.Rd`, `man/add_to_package_index.Rd`, no errors. + +- [ ] **Step 4: Verify package still loads and existing tests pass** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_dir("tests/testthat")'` +Expected: PASS — no regressions; the new path test from Task 1 still green. + +- [ ] **Step 5: Commit** + +```bash +git add R/package_index.R man/ +git commit -m "feat(index): upload/update PACKAGES for a per-minor slot via r_minor" +``` + +--- + +### Task 3: `classify_r_minor_sensitive()` internal helper + +**Files:** +- Modify: `R/process_cran_updates.R` (add helper near the top) +- Test: `tests/testthat/test-process_cran_updates.R` (new) + +- [ ] **Step 1: Write the failing test** + +```r +# tests/testthat/test-process_cran_updates.R +test_that("classify_r_minor_sensitive returns TRUE for a risky package", { + skip_if_not_installed("mockery") + mockery::stub(classify_r_minor_sensitive, "clone_repository", function(pkg, tag, url, dest) { + dir.create(dest, recursive = TRUE, showWarnings = FALSE) + writeLines( + c("Package: dummy", "Version: 1.0", "NeedsCompilation: yes", "LinkingTo: Rcpp"), + file.path(dest, "DESCRIPTION") + ) + dir.create(file.path(dest, "src")) + writeLines("// Rcpp glue", file.path(dest, "src", "x.cpp")) + }) + expect_true(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran")) +}) + +test_that("classify_r_minor_sensitive returns FALSE for a pure-r package", { + skip_if_not_installed("mockery") + mockery::stub(classify_r_minor_sensitive, "clone_repository", function(pkg, tag, url, dest) { + dir.create(dest, recursive = TRUE, showWarnings = FALSE) + writeLines(c("Package: dummy", "Version: 1.0", "NeedsCompilation: no"), + file.path(dest, "DESCRIPTION")) + }) + expect_false(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran")) +}) + +test_that("classify_r_minor_sensitive fails safe to TRUE on clone error", { + skip_if_not_installed("mockery") + mockery::stub(classify_r_minor_sensitive, "clone_repository", function(...) stop("boom")) + expect_true(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran")) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` +Expected: FAIL — `could not find function "classify_r_minor_sensitive"`. + +- [ ] **Step 3: Add the helper** + +```r +# R/process_cran_updates.R — add above process_cran_updates() +#' Classify a single CRAN package for R-minor sensitivity +#' +#' Clones the package source to a temp dir and runs +#' [needs_per_minor_recompile()]. Fails safe to `TRUE` (build per minor) if the +#' clone or classification errors, so a possibly-ABI-fragile binary is never +#' served from the cross-minor generic slot by mistake. +#' @keywords internal +classify_r_minor_sensitive <- function( + package_name, + tag, + source_org_url = "https://github.com/cran", + local_clone_dir = tempdir() +) { + dest <- file.path( + local_clone_dir, + sprintf("classify_%s_%s", package_name, tag) + ) + on.exit(unlink(dest, recursive = TRUE, force = TRUE), add = TRUE) + tryCatch( + { + clone_repository(package_name, tag, source_org_url, dest) + isTRUE(as.logical(needs_per_minor_recompile(dest))) + }, + error = function(e) { + log_warn(sprintf( + "{.fun classify_r_minor_sensitive}: failed for {.pkg %s} {.field %s}: %s. Treating as r-minor-sensitive.", # nolint + package_name, + tag, + conditionMessage(e) + )) + TRUE + } + ) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` +Expected: PASS (3/3). + +- [ ] **Step 5: Commit** + +```bash +git add R/process_cran_updates.R tests/testthat/test-process_cran_updates.R +git commit -m "feat(updates): add classify_r_minor_sensitive helper" +``` + +--- + +### Task 4: wire classifier detection into `process_cran_updates()` + +**Files:** +- Modify: `R/process_cran_updates.R` (`process_cran_updates` signature + the build loop at lines ~262-289) +- Modify: `tests/testthat/test-process_cran_updates.R` (add routing tests) + +- [ ] **Step 1: Write the failing tests** + +```r +# append to tests/testthat/test-process_cran_updates.R +test_that("classifier mode passes per-package is_r_minor_sensitive to build", { + skip_if_not_installed("mockery") + recorded <- list() + mockery::stub(process_cran_updates, "get_updated_cran_packages", + function(...) data.frame(name = c("riskypkg", "purepkg"), + version = c("1.0", "2.0"), stringsAsFactors = FALSE)) + mockery::stub(process_cran_updates, "get_new_cran_packages", + function(...) data.frame(name = character(), version = character())) + mockery::stub(process_cran_updates, "tools::CRAN_package_db", + function(...) data.frame(Package = character(), OS_type = character())) + mockery::stub(process_cran_updates, "classify_r_minor_sensitive", + function(name, tag, ...) name == "riskypkg") + mockery::stub(process_cran_updates, "build_binary_package", + function(name, tag, ..., is_r_minor_sensitive) { + recorded[[name]] <<- is_r_minor_sensitive + invisible(TRUE) + }) + + process_cran_updates( + platform = "alpine-323", process_removed = FALSE, + r_minor_detection = "classifier", + s3_endpoint = "x", s3_region = "x", s3_bucket = "x" + ) + + expect_true(recorded[["riskypkg"]]) + expect_false(recorded[["purepkg"]]) +}) + +test_that("r_minor_sensitive_only drops non-risky candidates", { + skip_if_not_installed("mockery") + built <- character() + mockery::stub(process_cran_updates, "get_updated_cran_packages", + function(...) data.frame(name = c("riskypkg", "purepkg"), + version = c("1.0", "2.0"), stringsAsFactors = FALSE)) + mockery::stub(process_cran_updates, "get_new_cran_packages", + function(...) data.frame(name = character(), version = character())) + mockery::stub(process_cran_updates, "tools::CRAN_package_db", + function(...) data.frame(Package = character(), OS_type = character())) + mockery::stub(process_cran_updates, "classify_r_minor_sensitive", + function(name, tag, ...) name == "riskypkg") + mockery::stub(process_cran_updates, "build_binary_package", + function(name, tag, ..., is_r_minor_sensitive) { built <<- c(built, name); invisible(TRUE) }) + + process_cran_updates( + platform = "alpine-323", process_removed = FALSE, + r_minor_detection = "classifier", r_minor_sensitive_only = TRUE, + s3_endpoint = "x", s3_region = "x", s3_bucket = "x" + ) + + expect_identical(built, "riskypkg") +}) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` +Expected: FAIL — `unused argument (r_minor_detection = ...)`. + +- [ ] **Step 3: Add the parameters** + +In the `process_cran_updates()` signature, after `filter_r_minor_sensitive = FALSE,` add: + +```r + r_minor_detection = c("none", "issue", "classifier"), + r_minor_sensitive_only = FALSE, +``` + +Immediately after the three `stop()` validation lines at the top of the body, add: + +```r + r_minor_detection <- match.arg(r_minor_detection) + # back-compat: the old boolean maps onto the issue-list path + if (isTRUE(filter_r_minor_sensitive) && r_minor_detection == "none") { + r_minor_detection <- "issue" + } +``` + +- [ ] **Step 4: Replace the issue-filter gate** + +The existing block reads `if (filter_r_minor_sensitive) { all_pkgs <- get_r_minor_sensitive_packages(...) }`. Change its condition to: + +```r + if (r_minor_detection == "issue") { + all_pkgs <- get_r_minor_sensitive_packages( + r_minor_packages_forge_type, + r_minor_packages_issue_url, + interval, + updated_packages = updated_pkgs, + new_packages = new_pkgs + ) + } +``` + +Also change the two later `if (filter_r_minor_sensitive)` log branches to `if (r_minor_detection != "none")`. + +- [ ] **Step 5: Replace the build loop** + +Replace the `purrr::walk2(all_pkgs$name, all_pkgs$version, ~ { build_binary_package(... is_r_minor_sensitive = filter_r_minor_sensitive ...) })` block with: + +```r + if (nrow(all_pkgs) > 0L) { + sensitive <- switch( + r_minor_detection, + classifier = vapply( + seq_len(nrow(all_pkgs)), + function(i) { + classify_r_minor_sensitive( + all_pkgs$name[i], + all_pkgs$version[i], + local_clone_dir = local_clone_dir + ) + }, + logical(1L) + ), + issue = rep(TRUE, nrow(all_pkgs)), + none = rep(FALSE, nrow(all_pkgs)) + ) + + if (isTRUE(r_minor_sensitive_only)) { + all_pkgs <- all_pkgs[sensitive, , drop = FALSE] + sensitive <- sensitive[sensitive] + } + + purrr::pwalk( + list(all_pkgs$name, all_pkgs$version, sensitive), + function(.name, .version, .sensitive) { + build_binary_package( + .name, + .version, + platform = platform, + upload = upload, + archive = archive, + force = force, + store_build_metadata = store_build_metadata, + s3_endpoint = s3_endpoint, + s3_bucket = s3_bucket, + s3_region = s3_region, + s3_access_key_id = s3_access_key_id, + s3_secret_access_key = s3_secret_access_key, + is_r_minor_sensitive = .sensitive, + metadata_db_type = metadata_db_type, + metadata_db_host = metadata_db_host, + metadata_db_name = metadata_db_name, + metadata_db_table = metadata_db_table, + metadata_db_port = metadata_db_port, + metadata_db_user = metadata_db_user, + metadata_db_password = metadata_db_password, + metadata_db_sslmode = metadata_db_sslmode + ) + } + ) + } else { + log_info("No packages to process after filtering Windows-only packages") + } +``` + +- [ ] **Step 6: Run tests to verify they pass** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` +Expected: PASS (5/5 — 3 from Task 3 + 2 new). + +- [ ] **Step 7: Commit** + +```bash +git add R/process_cran_updates.R tests/testthat/test-process_cran_updates.R +git commit -m "feat(updates): classifier-driven per-package r-minor routing + sensitive-only pass" +``` + +--- + +### Task 5: docs, roxygen params, version bump, NEWS + +**Files:** +- Create: `man-roxygen/param-r_minor_detection.R`, `man-roxygen/param-r_minor_sensitive_only.R` +- Modify: `R/process_cran_updates.R` (roxygen `@template` lines), `DESCRIPTION`, `NEWS.md` + +- [ ] **Step 1: Add the man templates** + +```r +# man-roxygen/param-r_minor_detection.R +#' @param r_minor_detection How to decide which packages are R-minor-sensitive. +#' `"none"` (default) builds everything into the generic slot. `"issue"` uses +#' the curated tracking issue (the legacy `filter_r_minor_sensitive` path). +#' `"classifier"` classifies each candidate via [needs_per_minor_recompile()] +#' and routes only `risky` packages to the per-minor slot. +``` + +```r +# man-roxygen/param-r_minor_sensitive_only.R +#' @param r_minor_sensitive_only When `TRUE`, only R-minor-sensitive packages are +#' built (used for the additional per-minor passes under non-primary R versions). +``` + +- [ ] **Step 2: Reference the templates** + +In the roxygen block above `process_cran_updates`, add: + +```r +#' @template param-r_minor_detection +#' @template param-r_minor_sensitive_only +``` + +- [ ] **Step 3: Bump version and NEWS** + +In `DESCRIPTION` set `Version: 4.2.0`. Add to the top of `NEWS.md`: + +```markdown +# bincraft 4.2.0 + +* `process_cran_updates()` gains `r_minor_detection` (`"none"`/`"issue"`/`"classifier"`) + and `r_minor_sensitive_only`, classifying each candidate via the ABI classifier and + routing only `risky` packages to per-minor slots. +* `upload_package_index()` / `add_to_package_index()` gain an `r_minor` argument to + write/serve a per-minor `PACKAGES*` index under `…/contrib//`. +``` + +- [ ] **Step 4: Regenerate docs and run full check** + +Run: `R -q -e 'devtools::document()'` +Run: `R -q -e 'devtools::test()'` +Expected: docs regenerate clean; all tests PASS. + +- [ ] **Step 5: Commit, push, open PR** + +```bash +git add DESCRIPTION NEWS.md man-roxygen/ man/ R/process_cran_updates.R +git commit -m "docs(release): bincraft 4.2.0 — per-minor index + classifier-driven updates" +git push -u origin +fj -H codefloe.com pr create --base main --head --body "" "feat: per-minor index + classifier-driven r-minor builds (4.2.0)" +``` + +--- + +## Self-Review + +- **Spec coverage:** §"Required bincraft enhancements" item 1 (process_cran_updates) → Tasks 3-5; item 2 (upload_package_index per-minor) → Tasks 1-2. `build_binary_package` unchanged per the spec — no task, correct. +- **Type consistency:** helper named `package_index_remote_dir` (Tasks 1-2), `classify_r_minor_sensitive` (Tasks 3-4) consistently; params `r_minor_detection`/`r_minor_sensitive_only`/`r_minor` consistent across signature, tests, and docs. +- **Placeholders:** none — every code/edit step shows the actual code; ``/`` in the final push step are deliberate operator inputs. +- **Release dependency:** the pipeline plan pins bincraft `4.2.0`, matching the bump here. diff --git a/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md b/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md new file mode 100644 index 0000000..6123278 --- /dev/null +++ b/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md @@ -0,0 +1,471 @@ +# Pipeline R-minor Builds Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the full (`build-all-versions-*`) and iterative (`process-updates-*`) pipelines build R-minor-sensitive packages once per installed R minor, and everything else once into the generic slot. + +**Architecture:** A precompute step classifies packages (cheap CRAN-metadata rules + source download only for the ambiguous compiled subset) and emits a sensitivity flag. Build scripts pass a concrete `is_r_minor_sensitive` per package on the primary-R pass and re-run a sensitive-only pass under each additional `/opt/R/*` minor. Per-minor `PACKAGES` indexes are uploaded for each touched slot. + +**Tech Stack:** R scripts under `local/`, crow/woodpecker CI YAML under `.crow/`, bincraft `>= 4.2.0` (see `2026-06-13-bincraft-r-minor-enhancements.md`). + +**HARD DEPENDENCY:** bincraft `4.2.0` (the companion plan) must be merged and released first — Tasks 4-6 call `r_minor_detection`/`upload_package_index(r_minor=)` which only exist there. Tasks 1-3 (this repo's R scripts) can be written and unit-tested before the release. + +--- + +### Task 1: pure metadata classifier helper + +**Files:** +- Create: `local/r-minor-helpers.R` +- Test: `local/tests/test-r-minor-helpers.R` + +- [ ] **Step 1: Write the failing test** + +```r +# local/tests/test-r-minor-helpers.R +source(file.path("local", "r-minor-helpers.R")) + +test_that("pure-r (NeedsCompilation != yes) is not sensitive", { + expect_identical(classify_from_metadata("no", NA, c("Rcpp")), "not-sensitive") + expect_identical(classify_from_metadata("", "Rcpp", c("Rcpp")), "not-sensitive") +}) + +test_that("LinkingTo a risky dep is sensitive (version constraints stripped)", { + expect_identical(classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), "sensitive") + expect_identical(classify_from_metadata("yes", "R6,\n cpp11", c("Rcpp", "cpp11")), "sensitive") +}) + +test_that("compiled but no risky LinkingTo is ambiguous (needs source)", { + expect_identical(classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), "ambiguous") + expect_identical(classify_from_metadata("yes", NA, c("Rcpp")), "ambiguous") +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'` +Expected: FAIL — `cannot open file 'local/r-minor-helpers.R'` / function not found. + +- [ ] **Step 3: Write the helper** + +```r +# local/r-minor-helpers.R +# Metadata-only ABI triage so the full build avoids downloading every source. +# Mirrors bincraft::abi_classify rules 1-2; "ambiguous" packages still need a +# source grep via bincraft::needs_per_minor_recompile(). + +classify_from_metadata <- function(needs_compilation, linking_to, risky_deps) { + nc <- if (length(needs_compilation) == 0L || is.na(needs_compilation)) { + "" + } else { + tolower(trimws(needs_compilation)) + } + if (!identical(nc, "yes")) { + return("not-sensitive") + } + lt <- if (length(linking_to) == 0L || is.na(linking_to)) "" else linking_to + linked <- trimws(unlist(strsplit(lt, "[,\n]"))) + linked <- sub("\\s*\\(.*\\)$", "", linked) # strip "(>= x)" constraints + linked <- linked[nzchar(linked)] + if (any(linked %in% risky_deps)) { + return("sensitive") + } + "ambiguous" +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'` +Expected: PASS (3/3). + +- [ ] **Step 5: Commit** + +```bash +git add local/r-minor-helpers.R local/tests/test-r-minor-helpers.R +git commit -m "feat(local): metadata-only ABI triage helper" +``` + +--- + +### Task 2: add `r_minor_sensitive` to `packages-to-build.R` + +**Files:** +- Modify: `local/packages-to-build.R` (after `pkgs` is finalized, ~line 142) + +- [ ] **Step 1: Append the classification block** + +At the end of `local/packages-to-build.R` (after `setorder(pkgs, Package, Version)`), add: + +```r +### R-minor sensitivity (classify once per package, applied to all versions) +source(file.path("local", "r-minor-helpers.R")) +risky_deps <- bincraft::abi_risky_linking_deps() + +release_meta <- data.table( + Package = cran_release$Package, + NeedsCompilation = cran_release$NeedsCompilation, + LinkingTo = cran_release$LinkingTo +) + +meta <- release_meta[Package %in% unique(pkgs$Package)] +meta[, triage := mapply( + classify_from_metadata, + NeedsCompilation, + LinkingTo, + MoreArgs = list(risky_deps = risky_deps) +)] + +# Only the "ambiguous" compiled packages need a source grep. +ambiguous <- meta[triage == "ambiguous", Package] +sensitive_ambiguous <- character() +if (length(ambiguous) > 0L) { + tmp_src <- file.path(tempdir(), "abi_src") + dir.create(tmp_src, showWarnings = FALSE, recursive = TRUE) + sens <- vapply(ambiguous, function(pkg) { + out <- tryCatch({ + dl <- utils::download.packages( + pkg, destdir = tmp_src, + repos = "https://cloud.r-project.org", quiet = TRUE + ) + isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L]))) + }, error = function(e) TRUE) # fail safe: treat as sensitive + out + }, logical(1L)) + sensitive_ambiguous <- ambiguous[sens] +} + +sensitive_pkgs <- unique(c(meta[triage == "sensitive", Package], sensitive_ambiguous)) +pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs] +sprintf("R-minor-sensitive packages: %s of %s", length(sensitive_pkgs), uniqueN(pkgs$Package)) +``` + +- [ ] **Step 2: Verify the script parses and the column is added (offline smoke)** + +Run: +```bash +R -q -e ' + library(data.table) + source("local/r-minor-helpers.R") + pkgs <- data.table(Package = c("A","B"), Version = c("1","1")) + cran_release <- data.frame(Package = c("A","B"), + NeedsCompilation = c("no","yes"), LinkingTo = c(NA,"Rcpp"), + stringsAsFactors = FALSE) + risky_deps <- c("Rcpp") + release_meta <- as.data.table(cran_release) + meta <- release_meta[Package %in% unique(pkgs$Package)] + meta[, triage := mapply(classify_from_metadata, NeedsCompilation, LinkingTo, + MoreArgs = list(risky_deps = risky_deps))] + sensitive_pkgs <- meta[triage == "sensitive", Package] + pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs] + stopifnot(identical(pkgs$r_minor_sensitive, c(FALSE, TRUE))) + cat("OK\n")' +``` +Expected: prints `OK` (B classified sensitive via LinkingTo, A not). + +- [ ] **Step 3: Persist the sensitive subset in install-deps** + +In `.crow/build-all-versions-install-deps-amd64.yaml` and `...-arm64.yaml`, change the precompute command (line ~38) from saving only `pkgs_to_build.rds` to also saving the subset: + +```yaml + - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))" +``` + +- [ ] **Step 4: Commit** + +```bash +git add local/packages-to-build.R .crow/build-all-versions-install-deps-amd64.yaml .crow/build-all-versions-install-deps-arm64.yaml +git commit -m "feat(full): classify r-minor sensitivity in install-deps precompute" +``` + +--- + +### Task 3: per-package flag + `--sensitive-only` in `build-all.R` + +**Files:** +- Modify: `local/build-all.R` +- Test: `local/tests/test-build-all-args.R` (new — covers the arg/flag parsing seam only) + +- [ ] **Step 1: Write the failing test for the arg parser** + +```r +# local/tests/test-build-all-args.R +source(file.path("local", "r-minor-helpers.R")) + +test_that("parse_build_args splits flags from positionals", { + a <- parse_build_args(c("--sensitive-only", "4", "2", "8")) + expect_true(a$sensitive_only) + expect_identical(a$split_into, 4L) + expect_identical(a$split_index, 2L) + expect_identical(a$ncpus, 8L) + + b <- parse_build_args(c("4", "2", "8")) + expect_false(b$sensitive_only) + expect_identical(b$split_into, 4L) +}) +``` + +- [ ] **Step 2: Add `parse_build_args` to the helpers and run the test** + +Append to `local/r-minor-helpers.R`: + +```r +parse_build_args <- function(args) { + sensitive_only <- "--sensitive-only" %in% args + pos <- args[!startsWith(args, "--")] + list( + sensitive_only = sensitive_only, + split_into = as.integer(pos[1L]), + split_index = as.integer(pos[2L]), + ncpus = as.integer(pos[3L]) + ) +} +``` + +Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-build-all-args.R")'` +Expected: PASS (1/1, 5 expectations). + +- [ ] **Step 3: Rewire `build-all.R` to use the parser, the flag, and the per-row sensitivity** + +Replace the arg-parsing header and the `mapply` build loop of `local/build-all.R`. New top: + +```r +sink(stdout(), type = "message") +options(crayon.enabled = TRUE, future.globals.onReference = NULL) +source(file.path("local", "r-minor-helpers.R")) + +args <- commandArgs(trailingOnly = TRUE) +parsed <- parse_build_args(args) +split_into <- parsed$split_into +split_index <- parsed$split_index +ncpus <- parsed$ncpus +sensitive_only <- parsed$sensitive_only +options(Ncpus = ncpus) + +library(bincraft, quietly = TRUE) +library(future) +plan("sequential") + +pkgs <- if (sensitive_only) { + readRDS("/mnt/cache/packages/r_minor_sensitive_pkgs.rds") +} else { + readRDS("/mnt/cache/packages/pkgs_to_build.rds") +} +# Back-compat: tolerate an older RDS without the column (treat all as non-sensitive) +if (is.null(pkgs$r_minor_sensitive)) pkgs$r_minor_sensitive <- FALSE +sprintf("Total# of remaining package versions: %s (sensitive_only=%s)", nrow(pkgs), sensitive_only) +``` + +Keep the existing chunk-split, exclude-list, and `s3_cache` lines unchanged. Then replace the `mapply(...)` call with: + +```r +n <- nrow(chunk) +mapply(function(pkg, ver, sens, i) { + cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens)) + bincraft::build_binary_package( + pkg, + tag = ver, + is_r_minor_sensitive = isTRUE(sens), + s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", + s3_region = "eu-central-003", + s3_bucket = "devxy-rpkgs-binaries", + s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), + s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), + s3_package_cache = s3_cache, + metadata_db_host = "r-binaries.devxy.io", + metadata_db_name = "build_metadata", + metadata_db_table = "single_builds", + metadata_db_user = "rpkgs", + metadata_db_password = Sys.getenv("PGPASS"), + metadata_db_sslmode = "require", + metadata_db_port = 15432, + archive = TRUE, + upload = TRUE, + store_build_metadata = TRUE + ) +}, chunk$Package, chunk$Version, chunk$r_minor_sensitive, seq_len(n)) +``` + +Note: in `--sensitive-only` mode every row already has `r_minor_sensitive == TRUE`, so the same loop builds them into per-minor slots; no special-casing needed. + +- [ ] **Step 4: Verify the script parses** + +Run: `R -q -e 'invisible(parse(file="local/build-all.R")); cat("parsed OK\n")'` +Expected: `parsed OK`. + +- [ ] **Step 5: Commit** + +```bash +git add local/build-all.R local/r-minor-helpers.R local/tests/test-build-all-args.R +git commit -m "feat(full): per-package r-minor flag and --sensitive-only mode in build-all.R" +``` + +--- + +### Task 4: multi-R loop + per-minor index in `build-all-versions-*.yaml` + +> Depends on bincraft 4.2.0 (`upload_package_index(r_minor=)`). + +**Files:** +- Modify: `.crow/build-all-versions-amd64.yaml`, `.crow/build-all-versions-arm64.yaml` + +- [ ] **Step 1: Add the sensitive-only multi-R pass after the primary build** + +In each file's `commands:` (after the existing `Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS` line and before the `process_unarchived_pkgs` line), insert this multiline command: + +```yaml + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive pass under R $RV ===" + R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$RBIN" Rscript local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true + done +``` + +(`$XVFB`/`$XVFB_ARGS` are already defined on the preceding line in this workflow.) Note the primary pass still runs `build-all.R` without `--sensitive-only`, building everything and routing sensitive packages into the primary minor slot. + +- [ ] **Step 2: Add a per-minor index upload step** + +Append a new step after the build step (mirroring the existing CDN/index pattern). Add to `commands` of a new `Upload per-minor indexes` step (or extend the existing index handling) the following, which uploads the generic index plus one per discovered minor: + +```yaml + - | + CODENAME=$(/opt/R/$R_VERSION/bin/Rscript -e "cat(bincraft::set_codename(NULL))") + # generic slot + /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done +``` + +- [ ] **Step 3: Validate YAML** + +Run: `R -q -e 'invisible(lapply(c(".crow/build-all-versions-amd64.yaml",".crow/build-all-versions-arm64.yaml"), yaml::yaml.load_file)); cat("yaml OK\n")'` +Expected: `yaml OK`. + +- [ ] **Step 4: Commit** + +```bash +git add .crow/build-all-versions-amd64.yaml .crow/build-all-versions-arm64.yaml +git commit -m "feat(full): sensitive-only multi-R passes and per-minor index upload" +``` + +--- + +### Task 5: classifier + multi-R in all `process-updates-*.yaml` + +> Depends on bincraft 4.2.0 (`r_minor_detection`, `upload_package_index(r_minor=)`). + +**Files (14):** every `.crow/process-updates--.yaml`: +`alpine-322`, `alpine-323`, `redhat-8`, `redhat-9`, `redhat-10`, `ubuntu-2204`, `ubuntu-2404` × `amd64`, `arm64`. + +The transform is identical in shape; only the already-present `platform=`, `R_VERSION`, `ARCH`, and the index `codename=` differ per file (leave those as-is). + +- [ ] **Step 1: Edit the `process_cran_updates` call (primary pass)** + +In each file, inside the single long `bincraft::process_cran_updates(...)` argument list, add `r_minor_detection = 'classifier', ` immediately before `s3_endpoint = ...`. Leave all other args unchanged. + +- [ ] **Step 2: Add the sensitive-only multi-R pass** + +Immediately after the `process_cran_updates(...)` command line, insert (preserve the per-file `''` string and `$INTERVAL`): + +```yaml + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$RBIN" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done +``` + +Replace `` with that file's existing platform string (e.g. `alpine-323`, `redhat-9`, `ubuntu-2404`). `process_removed = FALSE` on the extra passes (removals are handled once by the primary pass). + +- [ ] **Step 3: Add per-minor index upload** + +After the existing `upload_package_index(codename = "", ...)` line in each file, append (reusing the file's existing `codename`): + +```yaml + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "", r_minor = "'"$RMINOR"'", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' || true + done +``` + +Replace `` with the file's existing codename value (e.g. `alpine323`, `rhel9`, `ubuntu2404`). + +- [ ] **Step 4: Verify every file changed and parses** + +Run: +```bash +R -q -e 'fs <- Sys.glob(".crow/process-updates-*.yaml"); invisible(lapply(fs, yaml::yaml.load_file)); cat(length(fs), "files parse OK\n")' +grep -L "r_minor_detection = 'classifier'" .crow/process-updates-*.yaml +``` +Expected: `14 files parse OK`, and the `grep -L` prints **nothing** (every file contains the classifier arg). + +- [ ] **Step 5: Commit** + +```bash +git add .crow/process-updates-*.yaml +git commit -m "feat(updates): classifier-driven r-minor builds + per-minor index across platforms" +``` + +--- + +### Task 6: pin bincraft 4.2.0, remove the standalone workflow + +**Files:** +- Modify: all workflows pinning bincraft (`grep -rln "bincraft.git@v" .crow/`), and `.crow/build-all-versions-install-deps-*.yaml` (which installs from `main`) +- Delete: `.crow/build-r-minor-sensitive-packages.yaml` + +- [ ] **Step 1: Bump the pin** + +Run to find pins: `grep -rn "bincraft" .crow/ | grep -E "@v4\.1\.1|bincraft.git"`. In every `process-updates-*` file, change `bincraft.git@v4.1.1` (and the `packageVersion("bincraft") != "4.1.1"` guard) to `4.2.0`. + +- [ ] **Step 2: Remove the superseded workflow** + +```bash +git rm .crow/build-r-minor-sensitive-packages.yaml +``` + +- [ ] **Step 3: Verify no stale references** + +Run: `grep -rn "build-r-minor-sensitive\|@v4.1.1\|!= \"4.1.1\"" .crow/` +Expected: no matches. + +- [ ] **Step 4: Commit** + +```bash +git add .crow/ +git commit -m "chore: pin bincraft 4.2.0 and drop superseded standalone r-minor workflow" +``` + +--- + +### Task 7: integration smoke test (manual, gated on bincraft 4.2.0 release) + +- [ ] **Step 1: Run one iterative platform manually** via crow against a short interval and confirm: a known risky package (e.g. one LinkingTo Rcpp) lands under `…/contrib//` for each installed minor, and a pure-r package lands only in the generic slot. + +- [ ] **Step 2: Confirm a client install resolves the per-minor slot.** From an R `4.4` and an R `4.5` container: +```r +install.packages("", repos = "https://cran.devxy.io/") +library() # must load without "undefined symbol" +``` +Expected: loads under both minors. If the per-minor `PACKAGES` is missing, revisit bincraft Task 2. + +- [ ] **Step 3: Spot-check the full build** on one platform with a small `SPLIT_INTO`, verifying the sensitive-only extra passes ran and produced per-minor artifacts. + +--- + +## Self-Review + +- **Spec coverage:** install-deps precompute → Tasks 1-2; `build-all.R` per-row flag + `--sensitive-only` → Task 3; full-build multi-R loop + per-minor index → Task 4; iterative classifier + multi-R + per-minor index → Task 5; remove standalone workflow + version pin → Task 6; per-minor index client-serviceability risk → Task 7 verification. +- **Type/name consistency:** `classify_from_metadata` and `parse_build_args` live in `local/r-minor-helpers.R` and are used in Tasks 2-3; the RDS column is `r_minor_sensitive` everywhere; the subset file is `/mnt/cache/packages/r_minor_sensitive_pkgs.rds` in install-deps (Task 2) and `build-all.R` (Task 3). +- **Placeholders:** ``/`` in Task 5 are explicit per-file substitutions (the values already exist in each file), not unfilled blanks. No "TBD"/"handle errors" placeholders. +- **Dependency ordering:** Tasks 1-3 are pure R-script work, unit-testable now; Tasks 4-6 are gated on bincraft 4.2.0; Task 7 is post-release verification. diff --git a/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md b/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md index 81f828f..248a1a7 100644 --- a/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md +++ b/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md @@ -103,22 +103,29 @@ Driven by the bincraft enhancement below; the single build step becomes: ## Required bincraft enhancements (separate PR, coordinated release) -1. `build_binary_package()`: - - Accept `is_r_minor_sensitive = "auto"` — classify the source it already clones via - `abi_classify()` and route the artifact to the per-minor slot iff `risky`. - - Add `r_minor_sensitive_only` — when `TRUE`, skip (return `"skipped"`) non-risky - packages early, after classification, before building. -2. `process_cran_updates()`: +`build_binary_package()` needs **no change** — it already accepts a concrete +`is_r_minor_sensitive` logical and routes the slot accordingly. Classification stays at +the orchestration layer (mirroring the full-build precompute), which keeps +`build_binary_package`'s pre-build S3 skip-check and source clone untouched. + +1. `process_cran_updates()`: - Add `r_minor_detection = c("none", "issue", "classifier")` (default `"none"` to preserve current behavior; `"issue"` is today's `filter_r_minor_sensitive` path). - - Add `r_minor_sensitive_only`, threaded down to `build_binary_package()`. - - With `"classifier"`, pass `is_r_minor_sensitive = "auto"` per package instead of a - single run-wide bool. -3. `upload_package_index()`: - - Add per-minor slot support: write/upload `PACKAGES*` (and `Meta/archive.rds`) under - `…/contrib//`, mirroring the generic-slot logic. Invocable per minor. + - Add `r_minor_sensitive_only` (default `FALSE`). + - With `"classifier"`: for each candidate `(name, version)`, clone the source to a temp + dir and call `bincraft::needs_per_minor_recompile()`; pass the resulting concrete + logical as `is_r_minor_sensitive` to `build_binary_package()`. When + `r_minor_sensitive_only = TRUE`, drop non-risky candidates before building. + - Implemented via a small internal helper `classify_r_minor_sensitive(name, tag, + source_org_url, local_clone_dir)` returning a logical — the unit-testable seam. +2. `upload_package_index()`: + - Add an `r_minor = NULL` argument. When non-NULL (e.g. `"4.4"`), point the remote dir + at `…/contrib//` and write/upload `PACKAGES*` (and `Meta/archive.rds`) there, + mirroring the generic-slot logic. Extract the remote-dir construction into a pure + helper `package_index_remote_dir(s3_bucket, arch, codename, r_minor = NULL)` — the + unit-testable seam. -These three are the only bincraft changes; detection itself (PR #49) is already merged. +These two are the only bincraft changes; detection itself (PR #49) is already merged. ## Scope and cleanup