feat: R-minor-sensitive binary builds (full + iterative) #84
3 changed files with 0 additions and 1096 deletions
chore: drop superpowers spec/plans from PR (kept locally, gitignored)
commit
ab8f24e067
GPG key ID:
3C6318841EF78925
|
|
@ -1,470 +0,0 @@
|
|||
# bincraft R-minor Enhancements Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Teach bincraft to (a) build/serve per-R-minor package indexes and (b) drive `process_cran_updates()` from the ABI classifier so only risky packages are built per R minor.
|
||||
|
||||
**Architecture:** Two additive, low-risk changes to the bincraft R package. `upload_package_index()` gains an `r_minor` argument so a per-minor slot (`…/contrib/<x.y>/`) gets its own `PACKAGES*`. `process_cran_updates()` gains `r_minor_detection`/`r_minor_sensitive_only` and classifies each candidate via `needs_per_minor_recompile()` (already in the package), passing a concrete `is_r_minor_sensitive` logical to the unchanged `build_binary_package()`.
|
||||
|
||||
**Tech Stack:** R package, `testthat` (3e) + `mockery` for tests, `roxygen2` for docs, `s3fs`/`cranlike` for the index. Repo: `https://codefloe.com/rpkgs/bincraft` (this plan is executed in a clone of that repo, NOT in build-cran-binaries).
|
||||
|
||||
**Pre-req:** PR #49 (ABI classifier) is already merged on `main`; `abi_classify()`, `needs_per_minor_recompile()`, `abi_risky_linking_deps()` are exported.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: `package_index_remote_dir()` pure helper
|
||||
|
||||
**Files:**
|
||||
- Modify: `R/package_index.R` (top of file, before `add_to_package_index`)
|
||||
- Test: `tests/testthat/test-package_index.R` (new)
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```r
|
||||
# tests/testthat/test-package_index.R
|
||||
test_that("package_index_remote_dir builds the generic slot when r_minor is NULL", {
|
||||
expect_identical(
|
||||
package_index_remote_dir("bucket", "amd64", "alpine323"),
|
||||
file.path("bucket", "amd64", "alpine323", "latest", "src", "contrib")
|
||||
)
|
||||
})
|
||||
|
||||
test_that("package_index_remote_dir appends the minor slot when r_minor is set", {
|
||||
expect_identical(
|
||||
package_index_remote_dir("bucket", "amd64", "alpine323", r_minor = "4.4"),
|
||||
file.path("bucket", "amd64", "alpine323", "latest", "src", "contrib", "4.4")
|
||||
)
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-package_index.R")'`
|
||||
Expected: FAIL — `could not find function "package_index_remote_dir"`.
|
||||
|
||||
- [ ] **Step 3: Add the helper**
|
||||
|
||||
```r
|
||||
# R/package_index.R — add near the top, above add_to_package_index()
|
||||
#' Build the S3 remote contrib dir for a package index
|
||||
#'
|
||||
#' @param r_minor Optional `"major.minor"` string (e.g. `"4.4"`). When non-NULL
|
||||
#' the path points at the per-minor slot.
|
||||
#' @keywords internal
|
||||
package_index_remote_dir <- function(s3_bucket, arch, codename, r_minor = NULL) {
|
||||
base <- file.path(s3_bucket, arch, codename, "latest", "src", "contrib")
|
||||
if (is.null(r_minor)) {
|
||||
base
|
||||
} else {
|
||||
file.path(base, r_minor)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-package_index.R")'`
|
||||
Expected: PASS (2/2).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add R/package_index.R tests/testthat/test-package_index.R
|
||||
git commit -m "feat(index): add package_index_remote_dir helper for per-minor slots"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: per-minor support in `upload_package_index()`
|
||||
|
||||
**Files:**
|
||||
- Modify: `R/package_index.R` (`upload_package_index`, signature + the `remote_bin_dir` block at lines ~117-124; `add_to_package_index` for consistency)
|
||||
|
||||
- [ ] **Step 1: Add `r_minor` to the signature and use the helper**
|
||||
|
||||
In `upload_package_index()`, add `r_minor = NULL` to the argument list (after `arch = NULL`). Replace the inline `remote_bin_dir <- file.path(s3_bucket, arch, codename, "latest", "src", "contrib")` block with:
|
||||
|
||||
```r
|
||||
remote_bin_dir <- package_index_remote_dir(s3_bucket, arch, codename, r_minor)
|
||||
```
|
||||
|
||||
Do the same replacement in `add_to_package_index()` (it has the identical inline construction); add `r_minor = NULL` to its signature too.
|
||||
|
||||
- [ ] **Step 2: Add the roxygen param**
|
||||
|
||||
Above `upload_package_index` and `add_to_package_index`, add:
|
||||
|
||||
```r
|
||||
#' @param r_minor Optional `"major.minor"` string. When set, the index is
|
||||
#' written/read under the per-minor slot `…/contrib/<r_minor>/` instead of the
|
||||
#' generic `…/contrib/` slot.
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Regenerate docs**
|
||||
|
||||
Run: `R -q -e 'devtools::document()'`
|
||||
Expected: updated `man/upload_package_index.Rd`, `man/add_to_package_index.Rd`, no errors.
|
||||
|
||||
- [ ] **Step 4: Verify package still loads and existing tests pass**
|
||||
|
||||
Run: `R -q -e 'devtools::load_all("."); testthat::test_dir("tests/testthat")'`
|
||||
Expected: PASS — no regressions; the new path test from Task 1 still green.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add R/package_index.R man/
|
||||
git commit -m "feat(index): upload/update PACKAGES for a per-minor slot via r_minor"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: `classify_r_minor_sensitive()` internal helper
|
||||
|
||||
**Files:**
|
||||
- Modify: `R/process_cran_updates.R` (add helper near the top)
|
||||
- Test: `tests/testthat/test-process_cran_updates.R` (new)
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```r
|
||||
# tests/testthat/test-process_cran_updates.R
|
||||
test_that("classify_r_minor_sensitive returns TRUE for a risky package", {
|
||||
skip_if_not_installed("mockery")
|
||||
mockery::stub(classify_r_minor_sensitive, "clone_repository", function(pkg, tag, url, dest) {
|
||||
dir.create(dest, recursive = TRUE, showWarnings = FALSE)
|
||||
writeLines(
|
||||
c("Package: dummy", "Version: 1.0", "NeedsCompilation: yes", "LinkingTo: Rcpp"),
|
||||
file.path(dest, "DESCRIPTION")
|
||||
)
|
||||
dir.create(file.path(dest, "src"))
|
||||
writeLines("// Rcpp glue", file.path(dest, "src", "x.cpp"))
|
||||
})
|
||||
expect_true(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran"))
|
||||
})
|
||||
|
||||
test_that("classify_r_minor_sensitive returns FALSE for a pure-r package", {
|
||||
skip_if_not_installed("mockery")
|
||||
mockery::stub(classify_r_minor_sensitive, "clone_repository", function(pkg, tag, url, dest) {
|
||||
dir.create(dest, recursive = TRUE, showWarnings = FALSE)
|
||||
writeLines(c("Package: dummy", "Version: 1.0", "NeedsCompilation: no"),
|
||||
file.path(dest, "DESCRIPTION"))
|
||||
})
|
||||
expect_false(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran"))
|
||||
})
|
||||
|
||||
test_that("classify_r_minor_sensitive fails safe to TRUE on clone error", {
|
||||
skip_if_not_installed("mockery")
|
||||
mockery::stub(classify_r_minor_sensitive, "clone_repository", function(...) stop("boom"))
|
||||
expect_true(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran"))
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'`
|
||||
Expected: FAIL — `could not find function "classify_r_minor_sensitive"`.
|
||||
|
||||
- [ ] **Step 3: Add the helper**
|
||||
|
||||
```r
|
||||
# R/process_cran_updates.R — add above process_cran_updates()
|
||||
#' Classify a single CRAN package for R-minor sensitivity
|
||||
#'
|
||||
#' Clones the package source to a temp dir and runs
|
||||
#' [needs_per_minor_recompile()]. Fails safe to `TRUE` (build per minor) if the
|
||||
#' clone or classification errors, so a possibly-ABI-fragile binary is never
|
||||
#' served from the cross-minor generic slot by mistake.
|
||||
#' @keywords internal
|
||||
classify_r_minor_sensitive <- function(
|
||||
package_name,
|
||||
tag,
|
||||
source_org_url = "https://github.com/cran",
|
||||
local_clone_dir = tempdir()
|
||||
) {
|
||||
dest <- file.path(
|
||||
local_clone_dir,
|
||||
sprintf("classify_%s_%s", package_name, tag)
|
||||
)
|
||||
on.exit(unlink(dest, recursive = TRUE, force = TRUE), add = TRUE)
|
||||
tryCatch(
|
||||
{
|
||||
clone_repository(package_name, tag, source_org_url, dest)
|
||||
isTRUE(as.logical(needs_per_minor_recompile(dest)))
|
||||
},
|
||||
error = function(e) {
|
||||
log_warn(sprintf(
|
||||
"{.fun classify_r_minor_sensitive}: failed for {.pkg %s} {.field %s}: %s. Treating as r-minor-sensitive.", # nolint
|
||||
package_name,
|
||||
tag,
|
||||
conditionMessage(e)
|
||||
))
|
||||
TRUE
|
||||
}
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'`
|
||||
Expected: PASS (3/3).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add R/process_cran_updates.R tests/testthat/test-process_cran_updates.R
|
||||
git commit -m "feat(updates): add classify_r_minor_sensitive helper"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: wire classifier detection into `process_cran_updates()`
|
||||
|
||||
**Files:**
|
||||
- Modify: `R/process_cran_updates.R` (`process_cran_updates` signature + the build loop at lines ~262-289)
|
||||
- Modify: `tests/testthat/test-process_cran_updates.R` (add routing tests)
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
```r
|
||||
# append to tests/testthat/test-process_cran_updates.R
|
||||
test_that("classifier mode passes per-package is_r_minor_sensitive to build", {
|
||||
skip_if_not_installed("mockery")
|
||||
recorded <- list()
|
||||
mockery::stub(process_cran_updates, "get_updated_cran_packages",
|
||||
function(...) data.frame(name = c("riskypkg", "purepkg"),
|
||||
version = c("1.0", "2.0"), stringsAsFactors = FALSE))
|
||||
mockery::stub(process_cran_updates, "get_new_cran_packages",
|
||||
function(...) data.frame(name = character(), version = character()))
|
||||
mockery::stub(process_cran_updates, "tools::CRAN_package_db",
|
||||
function(...) data.frame(Package = character(), OS_type = character()))
|
||||
mockery::stub(process_cran_updates, "classify_r_minor_sensitive",
|
||||
function(name, tag, ...) name == "riskypkg")
|
||||
mockery::stub(process_cran_updates, "build_binary_package",
|
||||
function(name, tag, ..., is_r_minor_sensitive) {
|
||||
recorded[[name]] <<- is_r_minor_sensitive
|
||||
invisible(TRUE)
|
||||
})
|
||||
|
||||
process_cran_updates(
|
||||
platform = "alpine-323", process_removed = FALSE,
|
||||
r_minor_detection = "classifier",
|
||||
s3_endpoint = "x", s3_region = "x", s3_bucket = "x"
|
||||
)
|
||||
|
||||
expect_true(recorded[["riskypkg"]])
|
||||
expect_false(recorded[["purepkg"]])
|
||||
})
|
||||
|
||||
test_that("r_minor_sensitive_only drops non-risky candidates", {
|
||||
skip_if_not_installed("mockery")
|
||||
built <- character()
|
||||
mockery::stub(process_cran_updates, "get_updated_cran_packages",
|
||||
function(...) data.frame(name = c("riskypkg", "purepkg"),
|
||||
version = c("1.0", "2.0"), stringsAsFactors = FALSE))
|
||||
mockery::stub(process_cran_updates, "get_new_cran_packages",
|
||||
function(...) data.frame(name = character(), version = character()))
|
||||
mockery::stub(process_cran_updates, "tools::CRAN_package_db",
|
||||
function(...) data.frame(Package = character(), OS_type = character()))
|
||||
mockery::stub(process_cran_updates, "classify_r_minor_sensitive",
|
||||
function(name, tag, ...) name == "riskypkg")
|
||||
mockery::stub(process_cran_updates, "build_binary_package",
|
||||
function(name, tag, ..., is_r_minor_sensitive) { built <<- c(built, name); invisible(TRUE) })
|
||||
|
||||
process_cran_updates(
|
||||
platform = "alpine-323", process_removed = FALSE,
|
||||
r_minor_detection = "classifier", r_minor_sensitive_only = TRUE,
|
||||
s3_endpoint = "x", s3_region = "x", s3_bucket = "x"
|
||||
)
|
||||
|
||||
expect_identical(built, "riskypkg")
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'`
|
||||
Expected: FAIL — `unused argument (r_minor_detection = ...)`.
|
||||
|
||||
- [ ] **Step 3: Add the parameters**
|
||||
|
||||
In the `process_cran_updates()` signature, after `filter_r_minor_sensitive = FALSE,` add:
|
||||
|
||||
```r
|
||||
r_minor_detection = c("none", "issue", "classifier"),
|
||||
r_minor_sensitive_only = FALSE,
|
||||
```
|
||||
|
||||
Immediately after the three `stop()` validation lines at the top of the body, add:
|
||||
|
||||
```r
|
||||
r_minor_detection <- match.arg(r_minor_detection)
|
||||
# back-compat: the old boolean maps onto the issue-list path
|
||||
if (isTRUE(filter_r_minor_sensitive) && r_minor_detection == "none") {
|
||||
r_minor_detection <- "issue"
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Replace the issue-filter gate**
|
||||
|
||||
The existing block reads `if (filter_r_minor_sensitive) { all_pkgs <- get_r_minor_sensitive_packages(...) }`. Change its condition to:
|
||||
|
||||
```r
|
||||
if (r_minor_detection == "issue") {
|
||||
all_pkgs <- get_r_minor_sensitive_packages(
|
||||
r_minor_packages_forge_type,
|
||||
r_minor_packages_issue_url,
|
||||
interval,
|
||||
updated_packages = updated_pkgs,
|
||||
new_packages = new_pkgs
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
Also change the two later `if (filter_r_minor_sensitive)` log branches to `if (r_minor_detection != "none")`.
|
||||
|
||||
- [ ] **Step 5: Replace the build loop**
|
||||
|
||||
Replace the `purrr::walk2(all_pkgs$name, all_pkgs$version, ~ { build_binary_package(... is_r_minor_sensitive = filter_r_minor_sensitive ...) })` block with:
|
||||
|
||||
```r
|
||||
if (nrow(all_pkgs) > 0L) {
|
||||
sensitive <- switch(
|
||||
r_minor_detection,
|
||||
classifier = vapply(
|
||||
seq_len(nrow(all_pkgs)),
|
||||
function(i) {
|
||||
classify_r_minor_sensitive(
|
||||
all_pkgs$name[i],
|
||||
all_pkgs$version[i],
|
||||
local_clone_dir = local_clone_dir
|
||||
)
|
||||
},
|
||||
logical(1L)
|
||||
),
|
||||
issue = rep(TRUE, nrow(all_pkgs)),
|
||||
none = rep(FALSE, nrow(all_pkgs))
|
||||
)
|
||||
|
||||
if (isTRUE(r_minor_sensitive_only)) {
|
||||
all_pkgs <- all_pkgs[sensitive, , drop = FALSE]
|
||||
sensitive <- sensitive[sensitive]
|
||||
}
|
||||
|
||||
purrr::pwalk(
|
||||
list(all_pkgs$name, all_pkgs$version, sensitive),
|
||||
function(.name, .version, .sensitive) {
|
||||
build_binary_package(
|
||||
.name,
|
||||
.version,
|
||||
platform = platform,
|
||||
upload = upload,
|
||||
archive = archive,
|
||||
force = force,
|
||||
store_build_metadata = store_build_metadata,
|
||||
s3_endpoint = s3_endpoint,
|
||||
s3_bucket = s3_bucket,
|
||||
s3_region = s3_region,
|
||||
s3_access_key_id = s3_access_key_id,
|
||||
s3_secret_access_key = s3_secret_access_key,
|
||||
is_r_minor_sensitive = .sensitive,
|
||||
metadata_db_type = metadata_db_type,
|
||||
metadata_db_host = metadata_db_host,
|
||||
metadata_db_name = metadata_db_name,
|
||||
metadata_db_table = metadata_db_table,
|
||||
metadata_db_port = metadata_db_port,
|
||||
metadata_db_user = metadata_db_user,
|
||||
metadata_db_password = metadata_db_password,
|
||||
metadata_db_sslmode = metadata_db_sslmode
|
||||
)
|
||||
}
|
||||
)
|
||||
} else {
|
||||
log_info("No packages to process after filtering Windows-only packages")
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run tests to verify they pass**
|
||||
|
||||
Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'`
|
||||
Expected: PASS (5/5 — 3 from Task 3 + 2 new).
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add R/process_cran_updates.R tests/testthat/test-process_cran_updates.R
|
||||
git commit -m "feat(updates): classifier-driven per-package r-minor routing + sensitive-only pass"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: docs, roxygen params, version bump, NEWS
|
||||
|
||||
**Files:**
|
||||
- Create: `man-roxygen/param-r_minor_detection.R`, `man-roxygen/param-r_minor_sensitive_only.R`
|
||||
- Modify: `R/process_cran_updates.R` (roxygen `@template` lines), `DESCRIPTION`, `NEWS.md`
|
||||
|
||||
- [ ] **Step 1: Add the man templates**
|
||||
|
||||
```r
|
||||
# man-roxygen/param-r_minor_detection.R
|
||||
#' @param r_minor_detection How to decide which packages are R-minor-sensitive.
|
||||
#' `"none"` (default) builds everything into the generic slot. `"issue"` uses
|
||||
#' the curated tracking issue (the legacy `filter_r_minor_sensitive` path).
|
||||
#' `"classifier"` classifies each candidate via [needs_per_minor_recompile()]
|
||||
#' and routes only `risky` packages to the per-minor slot.
|
||||
```
|
||||
|
||||
```r
|
||||
# man-roxygen/param-r_minor_sensitive_only.R
|
||||
#' @param r_minor_sensitive_only When `TRUE`, only R-minor-sensitive packages are
|
||||
#' built (used for the additional per-minor passes under non-primary R versions).
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Reference the templates**
|
||||
|
||||
In the roxygen block above `process_cran_updates`, add:
|
||||
|
||||
```r
|
||||
#' @template param-r_minor_detection
|
||||
#' @template param-r_minor_sensitive_only
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Bump version and NEWS**
|
||||
|
||||
In `DESCRIPTION` set `Version: 4.2.0`. Add to the top of `NEWS.md`:
|
||||
|
||||
```markdown
|
||||
# bincraft 4.2.0
|
||||
|
||||
* `process_cran_updates()` gains `r_minor_detection` (`"none"`/`"issue"`/`"classifier"`)
|
||||
and `r_minor_sensitive_only`, classifying each candidate via the ABI classifier and
|
||||
routing only `risky` packages to per-minor slots.
|
||||
* `upload_package_index()` / `add_to_package_index()` gain an `r_minor` argument to
|
||||
write/serve a per-minor `PACKAGES*` index under `…/contrib/<x.y>/`.
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Regenerate docs and run full check**
|
||||
|
||||
Run: `R -q -e 'devtools::document()'`
|
||||
Run: `R -q -e 'devtools::test()'`
|
||||
Expected: docs regenerate clean; all tests PASS.
|
||||
|
||||
- [ ] **Step 5: Commit, push, open PR**
|
||||
|
||||
```bash
|
||||
git add DESCRIPTION NEWS.md man-roxygen/ man/ R/process_cran_updates.R
|
||||
git commit -m "docs(release): bincraft 4.2.0 — per-minor index + classifier-driven updates"
|
||||
git push -u origin <branch>
|
||||
fj -H codefloe.com pr create --base main --head <branch> --body "<summary>" "feat: per-minor index + classifier-driven r-minor builds (4.2.0)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
- **Spec coverage:** §"Required bincraft enhancements" item 1 (process_cran_updates) → Tasks 3-5; item 2 (upload_package_index per-minor) → Tasks 1-2. `build_binary_package` unchanged per the spec — no task, correct.
|
||||
- **Type consistency:** helper named `package_index_remote_dir` (Tasks 1-2), `classify_r_minor_sensitive` (Tasks 3-4) consistently; params `r_minor_detection`/`r_minor_sensitive_only`/`r_minor` consistent across signature, tests, and docs.
|
||||
- **Placeholders:** none — every code/edit step shows the actual code; `<branch>`/`<summary>` in the final push step are deliberate operator inputs.
|
||||
- **Release dependency:** the pipeline plan pins bincraft `4.2.0`, matching the bump here.
|
||||
|
|
@ -1,471 +0,0 @@
|
|||
# Pipeline R-minor Builds Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make the full (`build-all-versions-*`) and iterative (`process-updates-*`) pipelines build R-minor-sensitive packages once per installed R minor, and everything else once into the generic slot.
|
||||
|
||||
**Architecture:** A precompute step classifies packages (cheap CRAN-metadata rules + source download only for the ambiguous compiled subset) and emits a sensitivity flag. Build scripts pass a concrete `is_r_minor_sensitive` per package on the primary-R pass and re-run a sensitive-only pass under each additional `/opt/R/*` minor. Per-minor `PACKAGES` indexes are uploaded for each touched slot.
|
||||
|
||||
**Tech Stack:** R scripts under `local/`, crow/woodpecker CI YAML under `.crow/`, bincraft `>= 4.2.0` (see `2026-06-13-bincraft-r-minor-enhancements.md`).
|
||||
|
||||
**HARD DEPENDENCY:** bincraft `4.2.0` (the companion plan) must be merged and released first — Tasks 4-6 call `r_minor_detection`/`upload_package_index(r_minor=)` which only exist there. Tasks 1-3 (this repo's R scripts) can be written and unit-tested before the release.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: pure metadata classifier helper
|
||||
|
||||
**Files:**
|
||||
- Create: `local/r-minor-helpers.R`
|
||||
- Test: `local/tests/test-r-minor-helpers.R`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```r
|
||||
# local/tests/test-r-minor-helpers.R
|
||||
source(file.path("local", "r-minor-helpers.R"))
|
||||
|
||||
test_that("pure-r (NeedsCompilation != yes) is not sensitive", {
|
||||
expect_identical(classify_from_metadata("no", NA, c("Rcpp")), "not-sensitive")
|
||||
expect_identical(classify_from_metadata("", "Rcpp", c("Rcpp")), "not-sensitive")
|
||||
})
|
||||
|
||||
test_that("LinkingTo a risky dep is sensitive (version constraints stripped)", {
|
||||
expect_identical(classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), "sensitive")
|
||||
expect_identical(classify_from_metadata("yes", "R6,\n cpp11", c("Rcpp", "cpp11")), "sensitive")
|
||||
})
|
||||
|
||||
test_that("compiled but no risky LinkingTo is ambiguous (needs source)", {
|
||||
expect_identical(classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), "ambiguous")
|
||||
expect_identical(classify_from_metadata("yes", NA, c("Rcpp")), "ambiguous")
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'`
|
||||
Expected: FAIL — `cannot open file 'local/r-minor-helpers.R'` / function not found.
|
||||
|
||||
- [ ] **Step 3: Write the helper**
|
||||
|
||||
```r
|
||||
# local/r-minor-helpers.R
|
||||
# Metadata-only ABI triage so the full build avoids downloading every source.
|
||||
# Mirrors bincraft::abi_classify rules 1-2; "ambiguous" packages still need a
|
||||
# source grep via bincraft::needs_per_minor_recompile().
|
||||
|
||||
classify_from_metadata <- function(needs_compilation, linking_to, risky_deps) {
|
||||
nc <- if (length(needs_compilation) == 0L || is.na(needs_compilation)) {
|
||||
""
|
||||
} else {
|
||||
tolower(trimws(needs_compilation))
|
||||
}
|
||||
if (!identical(nc, "yes")) {
|
||||
return("not-sensitive")
|
||||
}
|
||||
lt <- if (length(linking_to) == 0L || is.na(linking_to)) "" else linking_to
|
||||
linked <- trimws(unlist(strsplit(lt, "[,\n]")))
|
||||
linked <- sub("\\s*\\(.*\\)$", "", linked) # strip "(>= x)" constraints
|
||||
linked <- linked[nzchar(linked)]
|
||||
if (any(linked %in% risky_deps)) {
|
||||
return("sensitive")
|
||||
}
|
||||
"ambiguous"
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'`
|
||||
Expected: PASS (3/3).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add local/r-minor-helpers.R local/tests/test-r-minor-helpers.R
|
||||
git commit -m "feat(local): metadata-only ABI triage helper"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: add `r_minor_sensitive` to `packages-to-build.R`
|
||||
|
||||
**Files:**
|
||||
- Modify: `local/packages-to-build.R` (after `pkgs` is finalized, ~line 142)
|
||||
|
||||
- [ ] **Step 1: Append the classification block**
|
||||
|
||||
At the end of `local/packages-to-build.R` (after `setorder(pkgs, Package, Version)`), add:
|
||||
|
||||
```r
|
||||
### R-minor sensitivity (classify once per package, applied to all versions)
|
||||
source(file.path("local", "r-minor-helpers.R"))
|
||||
risky_deps <- bincraft::abi_risky_linking_deps()
|
||||
|
||||
release_meta <- data.table(
|
||||
Package = cran_release$Package,
|
||||
NeedsCompilation = cran_release$NeedsCompilation,
|
||||
LinkingTo = cran_release$LinkingTo
|
||||
)
|
||||
|
||||
meta <- release_meta[Package %in% unique(pkgs$Package)]
|
||||
meta[, triage := mapply(
|
||||
classify_from_metadata,
|
||||
NeedsCompilation,
|
||||
LinkingTo,
|
||||
MoreArgs = list(risky_deps = risky_deps)
|
||||
)]
|
||||
|
||||
# Only the "ambiguous" compiled packages need a source grep.
|
||||
ambiguous <- meta[triage == "ambiguous", Package]
|
||||
sensitive_ambiguous <- character()
|
||||
if (length(ambiguous) > 0L) {
|
||||
tmp_src <- file.path(tempdir(), "abi_src")
|
||||
dir.create(tmp_src, showWarnings = FALSE, recursive = TRUE)
|
||||
sens <- vapply(ambiguous, function(pkg) {
|
||||
out <- tryCatch({
|
||||
dl <- utils::download.packages(
|
||||
pkg, destdir = tmp_src,
|
||||
repos = "https://cloud.r-project.org", quiet = TRUE
|
||||
)
|
||||
isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L])))
|
||||
}, error = function(e) TRUE) # fail safe: treat as sensitive
|
||||
out
|
||||
}, logical(1L))
|
||||
sensitive_ambiguous <- ambiguous[sens]
|
||||
}
|
||||
|
||||
sensitive_pkgs <- unique(c(meta[triage == "sensitive", Package], sensitive_ambiguous))
|
||||
pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs]
|
||||
sprintf("R-minor-sensitive packages: %s of %s", length(sensitive_pkgs), uniqueN(pkgs$Package))
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify the script parses and the column is added (offline smoke)**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
R -q -e '
|
||||
library(data.table)
|
||||
source("local/r-minor-helpers.R")
|
||||
pkgs <- data.table(Package = c("A","B"), Version = c("1","1"))
|
||||
cran_release <- data.frame(Package = c("A","B"),
|
||||
NeedsCompilation = c("no","yes"), LinkingTo = c(NA,"Rcpp"),
|
||||
stringsAsFactors = FALSE)
|
||||
risky_deps <- c("Rcpp")
|
||||
release_meta <- as.data.table(cran_release)
|
||||
meta <- release_meta[Package %in% unique(pkgs$Package)]
|
||||
meta[, triage := mapply(classify_from_metadata, NeedsCompilation, LinkingTo,
|
||||
MoreArgs = list(risky_deps = risky_deps))]
|
||||
sensitive_pkgs <- meta[triage == "sensitive", Package]
|
||||
pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs]
|
||||
stopifnot(identical(pkgs$r_minor_sensitive, c(FALSE, TRUE)))
|
||||
cat("OK\n")'
|
||||
```
|
||||
Expected: prints `OK` (B classified sensitive via LinkingTo, A not).
|
||||
|
||||
- [ ] **Step 3: Persist the sensitive subset in install-deps**
|
||||
|
||||
In `.crow/build-all-versions-install-deps-amd64.yaml` and `...-arm64.yaml`, change the precompute command (line ~38) from saving only `pkgs_to_build.rds` to also saving the subset:
|
||||
|
||||
```yaml
|
||||
- /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))"
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add local/packages-to-build.R .crow/build-all-versions-install-deps-amd64.yaml .crow/build-all-versions-install-deps-arm64.yaml
|
||||
git commit -m "feat(full): classify r-minor sensitivity in install-deps precompute"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: per-package flag + `--sensitive-only` in `build-all.R`
|
||||
|
||||
**Files:**
|
||||
- Modify: `local/build-all.R`
|
||||
- Test: `local/tests/test-build-all-args.R` (new — covers the arg/flag parsing seam only)
|
||||
|
||||
- [ ] **Step 1: Write the failing test for the arg parser**
|
||||
|
||||
```r
|
||||
# local/tests/test-build-all-args.R
|
||||
source(file.path("local", "r-minor-helpers.R"))
|
||||
|
||||
test_that("parse_build_args splits flags from positionals", {
|
||||
a <- parse_build_args(c("--sensitive-only", "4", "2", "8"))
|
||||
expect_true(a$sensitive_only)
|
||||
expect_identical(a$split_into, 4L)
|
||||
expect_identical(a$split_index, 2L)
|
||||
expect_identical(a$ncpus, 8L)
|
||||
|
||||
b <- parse_build_args(c("4", "2", "8"))
|
||||
expect_false(b$sensitive_only)
|
||||
expect_identical(b$split_into, 4L)
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add `parse_build_args` to the helpers and run the test**
|
||||
|
||||
Append to `local/r-minor-helpers.R`:
|
||||
|
||||
```r
|
||||
parse_build_args <- function(args) {
|
||||
sensitive_only <- "--sensitive-only" %in% args
|
||||
pos <- args[!startsWith(args, "--")]
|
||||
list(
|
||||
sensitive_only = sensitive_only,
|
||||
split_into = as.integer(pos[1L]),
|
||||
split_index = as.integer(pos[2L]),
|
||||
ncpus = as.integer(pos[3L])
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-build-all-args.R")'`
|
||||
Expected: PASS (1/1, 5 expectations).
|
||||
|
||||
- [ ] **Step 3: Rewire `build-all.R` to use the parser, the flag, and the per-row sensitivity**
|
||||
|
||||
Replace the arg-parsing header and the `mapply` build loop of `local/build-all.R`. New top:
|
||||
|
||||
```r
|
||||
sink(stdout(), type = "message")
|
||||
options(crayon.enabled = TRUE, future.globals.onReference = NULL)
|
||||
source(file.path("local", "r-minor-helpers.R"))
|
||||
|
||||
args <- commandArgs(trailingOnly = TRUE)
|
||||
parsed <- parse_build_args(args)
|
||||
split_into <- parsed$split_into
|
||||
split_index <- parsed$split_index
|
||||
ncpus <- parsed$ncpus
|
||||
sensitive_only <- parsed$sensitive_only
|
||||
options(Ncpus = ncpus)
|
||||
|
||||
library(bincraft, quietly = TRUE)
|
||||
library(future)
|
||||
plan("sequential")
|
||||
|
||||
pkgs <- if (sensitive_only) {
|
||||
readRDS("/mnt/cache/packages/r_minor_sensitive_pkgs.rds")
|
||||
} else {
|
||||
readRDS("/mnt/cache/packages/pkgs_to_build.rds")
|
||||
}
|
||||
# Back-compat: tolerate an older RDS without the column (treat all as non-sensitive)
|
||||
if (is.null(pkgs$r_minor_sensitive)) pkgs$r_minor_sensitive <- FALSE
|
||||
sprintf("Total# of remaining package versions: %s (sensitive_only=%s)", nrow(pkgs), sensitive_only)
|
||||
```
|
||||
|
||||
Keep the existing chunk-split, exclude-list, and `s3_cache` lines unchanged. Then replace the `mapply(...)` call with:
|
||||
|
||||
```r
|
||||
n <- nrow(chunk)
|
||||
mapply(function(pkg, ver, sens, i) {
|
||||
cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens))
|
||||
bincraft::build_binary_package(
|
||||
pkg,
|
||||
tag = ver,
|
||||
is_r_minor_sensitive = isTRUE(sens),
|
||||
s3_endpoint = "https://s3.eu-central-003.backblazeb2.com",
|
||||
s3_region = "eu-central-003",
|
||||
s3_bucket = "devxy-rpkgs-binaries",
|
||||
s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"),
|
||||
s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"),
|
||||
s3_package_cache = s3_cache,
|
||||
metadata_db_host = "r-binaries.devxy.io",
|
||||
metadata_db_name = "build_metadata",
|
||||
metadata_db_table = "single_builds",
|
||||
metadata_db_user = "rpkgs",
|
||||
metadata_db_password = Sys.getenv("PGPASS"),
|
||||
metadata_db_sslmode = "require",
|
||||
metadata_db_port = 15432,
|
||||
archive = TRUE,
|
||||
upload = TRUE,
|
||||
store_build_metadata = TRUE
|
||||
)
|
||||
}, chunk$Package, chunk$Version, chunk$r_minor_sensitive, seq_len(n))
|
||||
```
|
||||
|
||||
Note: in `--sensitive-only` mode every row already has `r_minor_sensitive == TRUE`, so the same loop builds them into per-minor slots; no special-casing needed.
|
||||
|
||||
- [ ] **Step 4: Verify the script parses**
|
||||
|
||||
Run: `R -q -e 'invisible(parse(file="local/build-all.R")); cat("parsed OK\n")'`
|
||||
Expected: `parsed OK`.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add local/build-all.R local/r-minor-helpers.R local/tests/test-build-all-args.R
|
||||
git commit -m "feat(full): per-package r-minor flag and --sensitive-only mode in build-all.R"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: multi-R loop + per-minor index in `build-all-versions-*.yaml`
|
||||
|
||||
> Depends on bincraft 4.2.0 (`upload_package_index(r_minor=)`).
|
||||
|
||||
**Files:**
|
||||
- Modify: `.crow/build-all-versions-amd64.yaml`, `.crow/build-all-versions-arm64.yaml`
|
||||
|
||||
- [ ] **Step 1: Add the sensitive-only multi-R pass after the primary build**
|
||||
|
||||
In each file's `commands:` (after the existing `Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS` line and before the `process_unarchived_pkgs` line), insert this multiline command:
|
||||
|
||||
```yaml
|
||||
- |
|
||||
PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
|
||||
for RBIN in /opt/R/*/bin/R; do
|
||||
RV=$(basename "$(dirname "$(dirname "$RBIN")")")
|
||||
RMINOR=$(echo "$RV" | cut -d. -f1-2)
|
||||
[ "$RMINOR" = "$PRIMARY_MINOR" ] && continue
|
||||
echo "=== R-minor-sensitive pass under R $RV ==="
|
||||
R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$RBIN" Rscript local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true
|
||||
done
|
||||
```
|
||||
|
||||
(`$XVFB`/`$XVFB_ARGS` are already defined on the preceding line in this workflow.) Note the primary pass still runs `build-all.R` without `--sensitive-only`, building everything and routing sensitive packages into the primary minor slot.
|
||||
|
||||
- [ ] **Step 2: Add a per-minor index upload step**
|
||||
|
||||
Append a new step after the build step (mirroring the existing CDN/index pattern). Add to `commands` of a new `Upload per-minor indexes` step (or extend the existing index handling) the following, which uploads the generic index plus one per discovered minor:
|
||||
|
||||
```yaml
|
||||
- |
|
||||
CODENAME=$(/opt/R/$R_VERSION/bin/Rscript -e "cat(bincraft::set_codename(NULL))")
|
||||
# generic slot
|
||||
/opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))"
|
||||
for RBIN in /opt/R/*/bin/R; do
|
||||
RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2)
|
||||
/opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true
|
||||
done
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Validate YAML**
|
||||
|
||||
Run: `R -q -e 'invisible(lapply(c(".crow/build-all-versions-amd64.yaml",".crow/build-all-versions-arm64.yaml"), yaml::yaml.load_file)); cat("yaml OK\n")'`
|
||||
Expected: `yaml OK`.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add .crow/build-all-versions-amd64.yaml .crow/build-all-versions-arm64.yaml
|
||||
git commit -m "feat(full): sensitive-only multi-R passes and per-minor index upload"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: classifier + multi-R in all `process-updates-*.yaml`
|
||||
|
||||
> Depends on bincraft 4.2.0 (`r_minor_detection`, `upload_package_index(r_minor=)`).
|
||||
|
||||
**Files (14):** every `.crow/process-updates-<plat>-<arch>.yaml`:
|
||||
`alpine-322`, `alpine-323`, `redhat-8`, `redhat-9`, `redhat-10`, `ubuntu-2204`, `ubuntu-2404` × `amd64`, `arm64`.
|
||||
|
||||
The transform is identical in shape; only the already-present `platform=`, `R_VERSION`, `ARCH`, and the index `codename=` differ per file (leave those as-is).
|
||||
|
||||
- [ ] **Step 1: Edit the `process_cran_updates` call (primary pass)**
|
||||
|
||||
In each file, inside the single long `bincraft::process_cran_updates(...)` argument list, add `r_minor_detection = 'classifier', ` immediately before `s3_endpoint = ...`. Leave all other args unchanged.
|
||||
|
||||
- [ ] **Step 2: Add the sensitive-only multi-R pass**
|
||||
|
||||
Immediately after the `process_cran_updates(...)` command line, insert (preserve the per-file `'<platform>'` string and `$INTERVAL`):
|
||||
|
||||
```yaml
|
||||
- |
|
||||
PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
|
||||
for RBIN in /opt/R/*/bin/R; do
|
||||
RV=$(basename "$(dirname "$(dirname "$RBIN")")")
|
||||
RMINOR=$(echo "$RV" | cut -d. -f1-2)
|
||||
[ "$RMINOR" = "$PRIMARY_MINOR" ] && continue
|
||||
echo "=== R-minor-sensitive update pass under R $RV ==="
|
||||
xvfb-run "$RBIN" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '<PLATFORM>', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true
|
||||
done
|
||||
```
|
||||
|
||||
Replace `<PLATFORM>` with that file's existing platform string (e.g. `alpine-323`, `redhat-9`, `ubuntu-2404`). `process_removed = FALSE` on the extra passes (removals are handled once by the primary pass).
|
||||
|
||||
- [ ] **Step 3: Add per-minor index upload**
|
||||
|
||||
After the existing `upload_package_index(codename = "<CODENAME>", ...)` line in each file, append (reusing the file's existing `codename`):
|
||||
|
||||
```yaml
|
||||
- |
|
||||
for RBIN in /opt/R/*/bin/R; do
|
||||
RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2)
|
||||
/opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "<CODENAME>", r_minor = "'"$RMINOR"'", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' || true
|
||||
done
|
||||
```
|
||||
|
||||
Replace `<CODENAME>` with the file's existing codename value (e.g. `alpine323`, `rhel9`, `ubuntu2404`).
|
||||
|
||||
- [ ] **Step 4: Verify every file changed and parses**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
R -q -e 'fs <- Sys.glob(".crow/process-updates-*.yaml"); invisible(lapply(fs, yaml::yaml.load_file)); cat(length(fs), "files parse OK\n")'
|
||||
grep -L "r_minor_detection = 'classifier'" .crow/process-updates-*.yaml
|
||||
```
|
||||
Expected: `14 files parse OK`, and the `grep -L` prints **nothing** (every file contains the classifier arg).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add .crow/process-updates-*.yaml
|
||||
git commit -m "feat(updates): classifier-driven r-minor builds + per-minor index across platforms"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: pin bincraft 4.2.0, remove the standalone workflow
|
||||
|
||||
**Files:**
|
||||
- Modify: all workflows pinning bincraft (`grep -rln "bincraft.git@v" .crow/`), and `.crow/build-all-versions-install-deps-*.yaml` (which installs from `main`)
|
||||
- Delete: `.crow/build-r-minor-sensitive-packages.yaml`
|
||||
|
||||
- [ ] **Step 1: Bump the pin**
|
||||
|
||||
Run to find pins: `grep -rn "bincraft" .crow/ | grep -E "@v4\.1\.1|bincraft.git"`. In every `process-updates-*` file, change `bincraft.git@v4.1.1` (and the `packageVersion("bincraft") != "4.1.1"` guard) to `4.2.0`.
|
||||
|
||||
- [ ] **Step 2: Remove the superseded workflow**
|
||||
|
||||
```bash
|
||||
git rm .crow/build-r-minor-sensitive-packages.yaml
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify no stale references**
|
||||
|
||||
Run: `grep -rn "build-r-minor-sensitive\|@v4.1.1\|!= \"4.1.1\"" .crow/`
|
||||
Expected: no matches.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add .crow/
|
||||
git commit -m "chore: pin bincraft 4.2.0 and drop superseded standalone r-minor workflow"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: integration smoke test (manual, gated on bincraft 4.2.0 release)
|
||||
|
||||
- [ ] **Step 1: Run one iterative platform manually** via crow against a short interval and confirm: a known risky package (e.g. one LinkingTo Rcpp) lands under `…/contrib/<x.y>/` for each installed minor, and a pure-r package lands only in the generic slot.
|
||||
|
||||
- [ ] **Step 2: Confirm a client install resolves the per-minor slot.** From an R `4.4` and an R `4.5` container:
|
||||
```r
|
||||
install.packages("<risky_pkg>", repos = "https://cran.devxy.io/<codename>")
|
||||
library(<risky_pkg>) # must load without "undefined symbol"
|
||||
```
|
||||
Expected: loads under both minors. If the per-minor `PACKAGES` is missing, revisit bincraft Task 2.
|
||||
|
||||
- [ ] **Step 3: Spot-check the full build** on one platform with a small `SPLIT_INTO`, verifying the sensitive-only extra passes ran and produced per-minor artifacts.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
- **Spec coverage:** install-deps precompute → Tasks 1-2; `build-all.R` per-row flag + `--sensitive-only` → Task 3; full-build multi-R loop + per-minor index → Task 4; iterative classifier + multi-R + per-minor index → Task 5; remove standalone workflow + version pin → Task 6; per-minor index client-serviceability risk → Task 7 verification.
|
||||
- **Type/name consistency:** `classify_from_metadata` and `parse_build_args` live in `local/r-minor-helpers.R` and are used in Tasks 2-3; the RDS column is `r_minor_sensitive` everywhere; the subset file is `/mnt/cache/packages/r_minor_sensitive_pkgs.rds` in install-deps (Task 2) and `build-all.R` (Task 3).
|
||||
- **Placeholders:** `<PLATFORM>`/`<CODENAME>` in Task 5 are explicit per-file substitutions (the values already exist in each file), not unfilled blanks. No "TBD"/"handle errors" placeholders.
|
||||
- **Dependency ordering:** Tasks 1-3 are pure R-script work, unit-testable now; Tasks 4-6 are gated on bincraft 4.2.0; Task 7 is post-release verification.
|
||||
|
|
@ -1,155 +0,0 @@
|
|||
# R-minor-sensitive binary builds — design
|
||||
|
||||
## Problem
|
||||
|
||||
CRAN binaries are currently built once, under a single R minor version, and served from a
|
||||
single generic slot (`…/latest/src/contrib/`).
|
||||
That is wrong for the minority of packages whose compiled code reaches into volatile R
|
||||
internals: a binary built under R 4.5 will fail to load under R 4.4 with an
|
||||
`undefined symbol` error.
|
||||
|
||||
bincraft v4.1.0+ can now detect these packages automatically via the ABI classifier
|
||||
(`abi_classify()` / `needs_per_minor_recompile()`, added in bincraft PR #49,
|
||||
<https://codefloe.com/rpkgs/bincraft/pulls/49>).
|
||||
The exec-env images now ship multiple R minor versions under `/opt/R/`.
|
||||
This design uses both to build R-minor-specific binaries for the sensitive packages only,
|
||||
in both the iterative (`process-updates-*`) and full (`build-all-versions-*`) pipelines.
|
||||
|
||||
## Background: how bincraft already behaves
|
||||
|
||||
- `abi_classify(path)` returns a tier: `pure-r` (~78.6%), `safe-compiled` (~7.7%), or
|
||||
`risky` (~13.6%). `needs_per_minor_recompile(path)` is the boolean wrapper
|
||||
(`TRUE` iff `risky`). Both need the package source (DESCRIPTION + `src/`).
|
||||
- `build_binary_package(…, is_r_minor_sensitive = TRUE)` uploads the artifact into a
|
||||
per-minor slot `…/latest/src/contrib/<major.minor>/` and records `r_version` in the
|
||||
build metadata. With `FALSE` it uses the generic slot. The minor is derived from the
|
||||
**running interpreter** (`R.version`), so producing a 4.4 binary requires running
|
||||
`/opt/R/4.4.x/bin/R`.
|
||||
- `process_cran_updates()` orchestrates the iterative flow with a single
|
||||
`is_r_minor_sensitive` bool for the whole run and no per-package classification or
|
||||
R-version loop.
|
||||
- `upload_package_index()` writes/uploads `PACKAGES*` for the **generic slot only**; it
|
||||
has no per-minor support. The current standalone r-minor workflow never builds a
|
||||
per-minor index, so per-minor slots are effectively unservable today.
|
||||
|
||||
## Core model
|
||||
|
||||
Collapse the three tiers to one boolean per package:
|
||||
|
||||
```
|
||||
r_minor_sensitive := (abi_classify(pkg)$tier == "risky")
|
||||
```
|
||||
|
||||
| group | tiers | built under | slot |
|
||||
| ------------------------------ | ---------------------- | -------------------- | ---------------------------- |
|
||||
| non-sensitive (~86%) | pure-r, safe-compiled | primary R only | generic `contrib/` |
|
||||
| sensitive (~14%) | risky | every installed minor| per-minor `contrib/<x.y>/` |
|
||||
|
||||
- **Primary R** = the existing `R_VERSION` env var in each workflow. Its pass builds the
|
||||
non-sensitive packages (generic slot) *and* the sensitive packages for its own minor slot.
|
||||
- **Extra minors** = every other R version discovered by scanning `/opt/R/` at runtime.
|
||||
Each runs a **sensitive-only** pass. An image with a single R version degrades cleanly
|
||||
to just the primary pass.
|
||||
- The loop over minors always lives at the shell/script layer (one `/opt/R/<ver>/bin/R`
|
||||
invocation per minor), never inside a single `build_binary_package()` call.
|
||||
|
||||
Classification granularity: classify **once per package** (its release version) and apply
|
||||
the resulting flag to all archived versions of that package. Tier rarely changes across
|
||||
recent versions; this avoids multiplying source downloads.
|
||||
|
||||
## Full build — `build-all-versions-*` + `local/`
|
||||
|
||||
### install-deps step (`local/packages-to-build.R`)
|
||||
|
||||
After computing `pkgs_to_build`, add an `r_minor_sensitive` logical column:
|
||||
|
||||
1. Pull `NeedsCompilation` and `LinkingTo` from `tools::CRAN_package_db()` (already loaded).
|
||||
2. Resolve cheaply, no download:
|
||||
- `NeedsCompilation != "yes"` → not sensitive (rule 1, pure-r).
|
||||
- `LinkingTo` references any `bincraft::abi_risky_linking_deps()` entry → sensitive
|
||||
(rule 2).
|
||||
3. For the remaining compiled, non-LinkingTo-risky packages only: download the source and
|
||||
call `bincraft::needs_per_minor_recompile()` (rules 3/4).
|
||||
4. Join the per-package flag onto every `(Package, Version)` row.
|
||||
|
||||
Outputs:
|
||||
- `pkgs_to_build.rds` — now carries the `r_minor_sensitive` column.
|
||||
- `r_minor_sensitive_pkgs.rds` — the sensitive subset, for the extra-minor passes.
|
||||
|
||||
### build step (`local/build-all.R` + `.crow/build-all-versions-{amd64,arm64}.yaml`)
|
||||
|
||||
- `build-all.R` gains an optional `--sensitive-only` mode (or an arg flag). In normal mode
|
||||
it builds the full chunk, passing `is_r_minor_sensitive = <row flag>` per package. In
|
||||
sensitive-only mode it reads `r_minor_sensitive_pkgs.rds`, intersects with its chunk, and
|
||||
builds those with `is_r_minor_sensitive = TRUE`.
|
||||
- The workflow step keeps the existing matrix split. After the primary `Rscript build-all.R`
|
||||
invocation, a shell loop discovers non-primary `/opt/R/*` minors and runs
|
||||
`Rscript build-all.R --sensitive-only <split> <index> <ncpus>` under each.
|
||||
- Index upload step: generic index as today, plus a per-minor index for each minor slot
|
||||
that received artifacts (depends on the `upload_package_index()` enhancement below).
|
||||
|
||||
## Iterative build — `process-updates-*`
|
||||
|
||||
Driven by the bincraft enhancement below; the single build step becomes:
|
||||
|
||||
1. Primary-R pass:
|
||||
`process_cran_updates(…, r_minor_detection = "classifier")`.
|
||||
Each updated/new package is classified; risky → primary minor slot, rest → generic slot.
|
||||
Removed-package handling stays as-is.
|
||||
2. Shell loop over non-primary `/opt/R/*` minors:
|
||||
`process_cran_updates(…, r_minor_detection = "classifier", r_minor_sensitive_only = TRUE)`.
|
||||
Builds only risky updates into their respective minor slots.
|
||||
3. Index upload extended to cover each touched minor slot in addition to the generic slot.
|
||||
|
||||
## Required bincraft enhancements (separate PR, coordinated release)
|
||||
|
||||
`build_binary_package()` needs **no change** — it already accepts a concrete
|
||||
`is_r_minor_sensitive` logical and routes the slot accordingly. Classification stays at
|
||||
the orchestration layer (mirroring the full-build precompute), which keeps
|
||||
`build_binary_package`'s pre-build S3 skip-check and source clone untouched.
|
||||
|
||||
1. `process_cran_updates()`:
|
||||
- Add `r_minor_detection = c("none", "issue", "classifier")` (default `"none"` to
|
||||
preserve current behavior; `"issue"` is today's `filter_r_minor_sensitive` path).
|
||||
- Add `r_minor_sensitive_only` (default `FALSE`).
|
||||
- With `"classifier"`: for each candidate `(name, version)`, clone the source to a temp
|
||||
dir and call `bincraft::needs_per_minor_recompile()`; pass the resulting concrete
|
||||
logical as `is_r_minor_sensitive` to `build_binary_package()`. When
|
||||
`r_minor_sensitive_only = TRUE`, drop non-risky candidates before building.
|
||||
- Implemented via a small internal helper `classify_r_minor_sensitive(name, tag,
|
||||
source_org_url, local_clone_dir)` returning a logical — the unit-testable seam.
|
||||
2. `upload_package_index()`:
|
||||
- Add an `r_minor = NULL` argument. When non-NULL (e.g. `"4.4"`), point the remote dir
|
||||
at `…/contrib/<r_minor>/` and write/upload `PACKAGES*` (and `Meta/archive.rds`) there,
|
||||
mirroring the generic-slot logic. Extract the remote-dir construction into a pure
|
||||
helper `package_index_remote_dir(s3_bucket, arch, codename, r_minor = NULL)` — the
|
||||
unit-testable seam.
|
||||
|
||||
These two are the only bincraft changes; detection itself (PR #49) is already merged.
|
||||
|
||||
## Scope and cleanup
|
||||
|
||||
- In scope: `.crow/process-updates-*` (iterative) and `.crow/build-all-versions-*` +
|
||||
`local/build-all.R` + `local/packages-to-build.R` (full), across amd64 and arm64 and all
|
||||
platforms (alpine, ubuntu, redhat). The per-platform workflow files share the same edit.
|
||||
- Removed: `.crow/build-r-minor-sensitive-packages.yaml` — superseded by the integrated
|
||||
flow (it was manual, alpine-3.21-only, and issue-list-driven).
|
||||
- Out of scope (call out, do not change here): `weekly-rebuild-missing-*`,
|
||||
`archive-missed-packages`, and the audit workflows. Revisit separately if per-minor
|
||||
rebuilds are wanted there too.
|
||||
|
||||
## Open risks / notes
|
||||
|
||||
- **R-version discovery**: assumes `/opt/R/<version>/bin/R` layout and that the primary
|
||||
`R_VERSION` is one of the installed versions. Parse minor as `major.minor` from each
|
||||
discovered version; dedupe by minor (build once per minor even if two patch releases
|
||||
coexist).
|
||||
- **Per-minor index correctness**: clients resolving `bin/<os>/contrib/<x.y>/` require the
|
||||
per-minor `PACKAGES` to exist; the `upload_package_index()` enhancement is a hard
|
||||
dependency for the sensitive artifacts to be usable. Verify against a real client
|
||||
install before declaring done.
|
||||
- **Classification cost**: bounded by downloading sources only for the compiled,
|
||||
non-LinkingTo-risky subset in install-deps. Worth measuring on a full run; if still too
|
||||
heavy, consider caching classifications keyed by package+version in the metadata DB.
|
||||
- **Double download**: install-deps classification downloads some sources that the build
|
||||
step re-downloads. Acceptable for now; the metadata-DB cache above would also remove this.
|
||||
Loading…
Reference in a new issue