Two-phase, TDD plan. Phase A (bincraft): patch registry loading/matching, isolated patched-binary builds, local-repo caching, and wiring into the pak install path, gated by a proof-of-mechanism task. Phase B (this repo): the RcppParallel registry entry, a validator + pre-commit hook, and passing patches through the build entry points.
1477 lines
49 KiB
Markdown
1477 lines
49 KiB
Markdown
# Package Patching Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Let a curated registry of packages be "patched" (env/configure/Makevars overrides and/or source diffs) before `pak` installs them, including when they are transitive dependencies, so compiler-/OS-specific failures like `RcppParallel` stop cascading.
|
||
|
||
**Architecture:** The mechanism lives in `bincraft`. Before `pak::local_install_deps()` runs, bincraft pre-builds each registry-matched package as a binary (reusing `pkgbuild::build(binary=TRUE)` + `cranlike::add_PACKAGES`), caches it, and serves it from a local `file://` repo prepended to `options("repos")` so `pak` installs the patched binary — direct or transitive — without recompiling. The curated registry (`registry.json` + diff files) lives in this repo and is passed in via a new `patches` argument.
|
||
|
||
**Tech Stack:** R, `pak`, `pkgbuild`, `cranlike` (fork `pat-s/cranlike@s3`), `pkgsearch`, `withr`, `jsonlite`, `testthat` (3e), `mockery`. Two repos: `bincraft` at `/Users/pjs/git/codefloe.com/rpkgs/bincraftr`, and this repo `build-cran-binaries`.
|
||
|
||
## Global Constraints
|
||
|
||
- bincraft package name is `bincraft`; repo dir is `…/rpkgs/bincraftr`. Current version `4.2.2.9999`.
|
||
- Logging uses bincraft's wrappers only: `log_info`, `log_warn`, `log_success`, `log_debug`, `log_error`, `log_header` (never bare `message`/`cat` in package code).
|
||
- `cli`-style inline markup is allowed in log messages (e.g. `{.pkg %s}`, `{.path %s}`); curly braces in dynamic/error text must be escaped as already done in `install_helpers.R`.
|
||
- Tests: `testthat` 3rd edition, files at `tests/testthat/test-<topic>.R`, network/build tests guarded with `skip_on_cran()` / `skip_if_offline()`; end-to-end build tests guarded behind `skip_if_not(nzchar(Sys.getenv("BINCRAFT_PATCH_E2E")))`.
|
||
- New exported functions need roxygen with `@keywords internal` for non-user helpers; run `devtools::document()` after adding roxygen.
|
||
- Patched-binary cache dir default: `/mnt/cache/patched-binaries`.
|
||
- Platform tokens for matching = `c(<platform>, <family>, <arch>)`, e.g. `ubuntu-2604` → `c("ubuntu-2604","ubuntu","amd64")`.
|
||
- Registry entry required fields: `package`, `versions`, `platforms`, `reason`. Optional: `env`, `configure_args`, `makevars`, `patch`.
|
||
- One sentence per line in prose/commit messages; do not hard-wrap at 80 columns.
|
||
- Use `fj -H codefloe.com` for any PR operations (Forgejo), not `gh`.
|
||
|
||
---
|
||
|
||
## Phase A — bincraft mechanism
|
||
|
||
All Phase A paths are relative to `/Users/pjs/git/codefloe.com/rpkgs/bincraftr`.
|
||
|
||
### Task A0: Proof of mechanism — pak installs a patched binary from a prepended `file://` repo
|
||
|
||
This de-risks the core assumption before building anything on top: that `pak` installs a binary from a local `file://` repo in preference to CRAN for an equal version, and does so without recompiling. If this fails, the contingency (documented in Step 4) is to serve patched *source* and rely on `pkgcache` build-caching — the rest of the plan changes only inside `build_patched_binary()`.
|
||
|
||
**Files:**
|
||
- Create: `tools/verify-patch-mechanism.R`
|
||
|
||
**Interfaces:**
|
||
- Produces: a runnable script proving `pak::pkg_install()` resolves a local patched binary over CRAN. No package API.
|
||
|
||
- [ ] **Step 1: Write the verification script**
|
||
|
||
```r
|
||
# tools/verify-patch-mechanism.R
|
||
# Proves pak installs a patched binary from a prepended file:// repo instead of
|
||
# CRAN's, without recompiling. Run inside a Linux build-env container:
|
||
# Rscript tools/verify-patch-mechanism.R
|
||
# Exits 0 on success, 1 on failure.
|
||
|
||
pkg <- "glue" # small, pure-R CRAN package
|
||
sentinel <- "PatchMechanismProof"
|
||
|
||
work <- tempfile("verify_")
|
||
repo <- file.path(work, "repo", "src", "contrib")
|
||
lib <- file.path(work, "lib")
|
||
dir.create(repo, recursive = TRUE)
|
||
dir.create(lib, recursive = TRUE)
|
||
|
||
# 1. Download CRAN source for the current version.
|
||
ap <- available.packages(repos = "https://cloud.r-project.org")
|
||
ver <- ap[pkg, "Version"]
|
||
src <- file.path(work, sprintf("%s_%s.tar.gz", pkg, ver))
|
||
download.file(
|
||
sprintf("https://cloud.r-project.org/src/contrib/%s_%s.tar.gz", pkg, ver),
|
||
src, mode = "wb"
|
||
)
|
||
|
||
# 2. Unpack, inject a sentinel field into DESCRIPTION, build a binary.
|
||
untar(src, exdir = work)
|
||
desc <- file.path(work, pkg, "DESCRIPTION")
|
||
writeLines(c(readLines(desc), sprintf("%s: yes", sentinel)), desc)
|
||
pkgbuild::build(
|
||
file.path(work, pkg), binary = TRUE, vignettes = FALSE,
|
||
dest_path = repo, quiet = TRUE
|
||
)
|
||
built <- list.files(repo, pattern = sprintf("^%s_.*\\.tar\\.gz$", pkg), full.names = TRUE)
|
||
file.rename(built[1L], file.path(repo, sprintf("%s_%s.tar.gz", pkg, ver)))
|
||
cranlike::add_PACKAGES(sprintf("%s_%s.tar.gz", pkg, ver), repo)
|
||
|
||
# 3. Install with the local repo prepended; assert our patched build won.
|
||
withr::with_options(
|
||
list(repos = c(patched = sprintf("file://%s", dirname(dirname(repo))),
|
||
CRAN = "https://cloud.r-project.org")),
|
||
pak::pkg_install(pkg, lib = lib, ask = FALSE, upgrade = FALSE)
|
||
)
|
||
|
||
installed_desc <- file.path(lib, pkg, "DESCRIPTION")
|
||
ok <- file.exists(installed_desc) &&
|
||
any(grepl(sentinel, readLines(installed_desc)))
|
||
|
||
if (ok) {
|
||
cat("PROOF PASSED: pak installed the patched local binary.\n")
|
||
quit(status = 0L)
|
||
} else {
|
||
cat("PROOF FAILED: pak did not install the patched local binary.\n")
|
||
quit(status = 1L)
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 2: Run the proof in a build-env container**
|
||
|
||
Run (amd64 example; use any supported build-env image):
|
||
|
||
```bash
|
||
just build-single ubuntu 2604 amd64 4.5.0 glue 1.0.0 1 || true # warms the env
|
||
docker run --rm -v "$PWD":/work -w /work reg.devxy.io/rpkgs/build-env-ubuntu:2604 \
|
||
Rscript tools/verify-patch-mechanism.R
|
||
```
|
||
|
||
Expected: final line `PROOF PASSED: pak installed the patched local binary.` and exit status 0.
|
||
|
||
- [ ] **Step 3: Commit**
|
||
|
||
```bash
|
||
git add tools/verify-patch-mechanism.R
|
||
git commit -m "test(patches): prove pak installs a patched binary from a local file:// repo"
|
||
```
|
||
|
||
- [ ] **Step 4: Record the outcome / contingency**
|
||
|
||
If the proof PASSED, proceed to Task A1 unchanged.
|
||
If it FAILED (pak recompiled or picked CRAN's), the mechanism switches to serving patched *source*: in Task A4 `build_patched_binary()` skips `pkgbuild::build()` and instead repackages the patched source tree with `pkgbuild::build(binary = FALSE)`; everything else (registry, matching, cache, repo prepend) is unchanged because `pak` build-caches the compiled result via `pkgcache`. Note the chosen path in the commit message and continue.
|
||
|
||
---
|
||
|
||
### Task A1: Registry loading and normalization
|
||
|
||
**Files:**
|
||
- Create: `R/patches.R`
|
||
- Test: `tests/testthat/test-patches.R`
|
||
|
||
**Interfaces:**
|
||
- Produces: `load_patch_registry(patches_dir)` → `list()` of normalized entries; each entry is a named list with `package`, `versions`, `platforms` (character vector), `env` (named list), `configure_args` (character), `makevars` (named list), `reason`, and `patch_path` (absolute path or `NULL`). `normalize_patch_entry(entry, patches_dir)` → one normalized entry; errors on missing required field or missing patch file.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
```r
|
||
# tests/testthat/test-patches.R
|
||
test_that("load_patch_registry parses and normalizes entries", {
|
||
dir <- withr::local_tempdir()
|
||
writeLines("--- a patch ---", file.path(dir, "fix.patch"))
|
||
jsonlite::write_json(
|
||
list(list(
|
||
package = "RcppParallel", versions = "*",
|
||
platforms = list("alpine", "ubuntu-2604"),
|
||
env = list(RCPP_PARALLEL_USE_TBB = "0"),
|
||
patch = "fix.patch", reason = "bundled TBB fails"
|
||
)),
|
||
file.path(dir, "registry.json"), auto_unbox = TRUE
|
||
)
|
||
|
||
reg <- load_patch_registry(dir)
|
||
|
||
expect_length(reg, 1L)
|
||
expect_identical(reg[[1L]]$package, "RcppParallel")
|
||
expect_identical(reg[[1L]]$platforms, c("alpine", "ubuntu-2604"))
|
||
expect_identical(reg[[1L]]$env$RCPP_PARALLEL_USE_TBB, "0")
|
||
expect_identical(reg[[1L]]$configure_args, character(0L))
|
||
expect_true(file.exists(reg[[1L]]$patch_path))
|
||
})
|
||
|
||
test_that("load_patch_registry returns empty list when no registry", {
|
||
expect_identical(load_patch_registry(NULL), list())
|
||
expect_identical(load_patch_registry(withr::local_tempdir()), list())
|
||
})
|
||
|
||
test_that("normalize_patch_entry errors on missing required field", {
|
||
expect_error(
|
||
normalize_patch_entry(list(package = "x"), tempdir()),
|
||
"missing required field"
|
||
)
|
||
})
|
||
|
||
test_that("normalize_patch_entry errors on missing patch file", {
|
||
expect_error(
|
||
normalize_patch_entry(
|
||
list(package = "x", versions = "*", platforms = "alpine",
|
||
reason = "r", patch = "nope.patch"),
|
||
tempdir()
|
||
),
|
||
"does not exist"
|
||
)
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: FAIL with "could not find function load_patch_registry".
|
||
|
||
- [ ] **Step 3: Write minimal implementation**
|
||
|
||
```r
|
||
# R/patches.R
|
||
|
||
#' Load and validate the patch registry
|
||
#'
|
||
#' Reads `registry.json` from `patches_dir` and returns normalized entries.
|
||
#'
|
||
#' @param patches_dir Directory containing `registry.json` and any patch files,
|
||
#' or `NULL` to disable patching.
|
||
#' @return A list of normalized patch entries (possibly empty).
|
||
#' @keywords internal
|
||
load_patch_registry <- function(patches_dir) {
|
||
if (is.null(patches_dir)) {
|
||
return(list())
|
||
}
|
||
registry_file <- file.path(patches_dir, "registry.json")
|
||
if (!file.exists(registry_file)) {
|
||
log_warn(sprintf(
|
||
"Patch directory {.path %s} has no registry.json; patching disabled.",
|
||
patches_dir
|
||
))
|
||
return(list())
|
||
}
|
||
raw <- jsonlite::fromJSON(registry_file, simplifyVector = FALSE)
|
||
lapply(raw, normalize_patch_entry, patches_dir = patches_dir)
|
||
}
|
||
|
||
#' Normalize and validate a single patch registry entry
|
||
#'
|
||
#' @param entry A list parsed from `registry.json`.
|
||
#' @param patches_dir Directory used to resolve a relative `patch` path.
|
||
#' @return The entry with defaults filled and `patch_path` resolved.
|
||
#' @keywords internal
|
||
normalize_patch_entry <- function(entry, patches_dir) {
|
||
required <- c("package", "versions", "platforms", "reason")
|
||
missing <- setdiff(required, names(entry))
|
||
if (length(missing) > 0L) {
|
||
stop(
|
||
sprintf("Patch entry is missing required field(s): %s", toString(missing)),
|
||
call. = FALSE
|
||
)
|
||
}
|
||
entry$platforms <- as.character(unlist(entry$platforms))
|
||
entry$env <- if (is.null(entry$env)) list() else entry$env
|
||
entry$configure_args <- if (is.null(entry$configure_args)) {
|
||
character(0L)
|
||
} else {
|
||
as.character(unlist(entry$configure_args))
|
||
}
|
||
entry$makevars <- if (is.null(entry$makevars)) list() else entry$makevars
|
||
if (!is.null(entry$patch)) {
|
||
patch_path <- file.path(patches_dir, entry$patch)
|
||
if (!file.exists(patch_path)) {
|
||
stop(
|
||
sprintf(
|
||
"Patch file '%s' for package '%s' does not exist.",
|
||
patch_path, entry$package
|
||
),
|
||
call. = FALSE
|
||
)
|
||
}
|
||
entry$patch_path <- patch_path
|
||
} else {
|
||
entry$patch_path <- NULL
|
||
}
|
||
entry
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: PASS (4 tests).
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add R/patches.R tests/testthat/test-patches.R
|
||
git commit -m "feat(patches): load and validate the patch registry"
|
||
```
|
||
|
||
---
|
||
|
||
### Task A2: Platform matching and version-constraint satisfaction
|
||
|
||
**Files:**
|
||
- Modify: `R/patches.R`
|
||
- Test: `tests/testthat/test-patches.R`
|
||
|
||
**Interfaces:**
|
||
- Consumes: normalized entries from Task A1.
|
||
- Produces: `build_platform_tokens(platform, arch)` → character vector; `entry_matches_platform(entry, tokens)` → logical; `match_patch_entries(registry, platform, arch)` → filtered list; `version_satisfies(version, constraint)` → logical (constraint forms: `"*"` handled by caller, `"x.y.z"` exact, `">=x"`, `"<=x"`, `">x"`, `"<x"`, `"==x"`).
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
```r
|
||
test_that("build_platform_tokens expands family and arch", {
|
||
expect_setequal(
|
||
build_platform_tokens("ubuntu-2604", "amd64"),
|
||
c("ubuntu-2604", "ubuntu", "amd64")
|
||
)
|
||
expect_setequal(
|
||
build_platform_tokens("alpine-324", "arm64"),
|
||
c("alpine-324", "alpine", "arm64")
|
||
)
|
||
})
|
||
|
||
test_that("match_patch_entries matches family, codename, arch, and wildcard", {
|
||
reg <- list(
|
||
list(package = "A", versions = "*", platforms = "alpine", reason = "r"),
|
||
list(package = "B", versions = "*", platforms = "ubuntu-2604", reason = "r"),
|
||
list(package = "C", versions = "*", platforms = "*", reason = "r"),
|
||
list(package = "D", versions = "*", platforms = "redhat", reason = "r")
|
||
)
|
||
got <- vapply(
|
||
match_patch_entries(reg, "ubuntu-2604", "amd64"),
|
||
function(e) e$package, character(1L)
|
||
)
|
||
expect_setequal(got, c("B", "C"))
|
||
})
|
||
|
||
test_that("version_satisfies handles operators and hyphenated versions", {
|
||
expect_true(version_satisfies("5.1.11-2", "5.1.11-2"))
|
||
expect_true(version_satisfies("5.1.11-2", "==5.1.11-2"))
|
||
expect_true(version_satisfies("5.1.12", ">=5.1.0"))
|
||
expect_false(version_satisfies("5.0.0", ">=5.1.0"))
|
||
expect_true(version_satisfies("5.1.11-2", "<=5.1.11-2"))
|
||
expect_false(version_satisfies("5.1.12", "<=5.1.11-2"))
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: FAIL with "could not find function build_platform_tokens".
|
||
|
||
- [ ] **Step 3: Write minimal implementation (append to `R/patches.R`)**
|
||
|
||
```r
|
||
#' Build platform tokens for patch matching
|
||
#' @keywords internal
|
||
build_platform_tokens <- function(platform, arch) {
|
||
family <- sub("-.*$", "", platform)
|
||
unique(c(platform, family, arch))
|
||
}
|
||
|
||
#' Does a patch entry apply to the current platform tokens?
|
||
#' @keywords internal
|
||
entry_matches_platform <- function(entry, tokens) {
|
||
any(entry$platforms == "*") ||
|
||
length(intersect(entry$platforms, tokens)) > 0L
|
||
}
|
||
|
||
#' Filter registry entries applicable to the current build
|
||
#' @keywords internal
|
||
match_patch_entries <- function(registry, platform, arch) {
|
||
if (length(registry) == 0L) {
|
||
return(list())
|
||
}
|
||
tokens <- build_platform_tokens(platform, arch)
|
||
Filter(function(e) entry_matches_platform(e, tokens), registry)
|
||
}
|
||
|
||
#' Test whether a version satisfies a single constraint
|
||
#'
|
||
#' @param version A version string (CRAN style, may contain `-`).
|
||
#' @param constraint One of `"x.y.z"`, `"==x"`, `">=x"`, `"<=x"`, `">x"`, `"<x"`.
|
||
#' @keywords internal
|
||
version_satisfies <- function(version, constraint) {
|
||
constraint <- trimws(constraint)
|
||
parts <- regmatches(
|
||
constraint,
|
||
regexec("^(>=|<=|==|>|<)?\\s*(.+)$", constraint)
|
||
)[[1L]]
|
||
op <- parts[2L]
|
||
target <- parts[3L]
|
||
v <- package_version(version)
|
||
t <- package_version(target)
|
||
if (op == "" || op == "==") {
|
||
return(v == t)
|
||
}
|
||
switch(
|
||
op,
|
||
">=" = v >= t,
|
||
"<=" = v <= t,
|
||
">" = v > t,
|
||
"<" = v < t,
|
||
FALSE
|
||
)
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add R/patches.R tests/testthat/test-patches.R
|
||
git commit -m "feat(patches): platform matching and version-constraint checks"
|
||
```
|
||
|
||
---
|
||
|
||
### Task A3: Cache key and version resolution
|
||
|
||
**Files:**
|
||
- Modify: `R/patches.R`
|
||
- Test: `tests/testthat/test-patches.R`
|
||
|
||
**Interfaces:**
|
||
- Consumes: normalized entries.
|
||
- Produces: `patch_cache_key(entry, version, platform, arch, r_minor)` → string `"<pkg>_<ver>_<platform>_<arch>_<rminor>_<hash12>"`, where `hash12` covers `env`/`configure_args`/`makevars`/patch bytes; `resolve_patch_version(entry)` → latest CRAN version satisfying `entry$versions`, or `NA_character_`; `describe_patch(entry)` → short human label.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
```r
|
||
test_that("patch_cache_key is stable and sensitive to env/patch changes", {
|
||
e1 <- list(package = "P", env = list(A = "1"),
|
||
configure_args = character(0L), makevars = list(),
|
||
patch_path = NULL)
|
||
e2 <- e1; e2$env <- list(A = "2")
|
||
|
||
k1 <- patch_cache_key(e1, "1.0", "alpine-324", "amd64", "4.5")
|
||
expect_identical(k1, patch_cache_key(e1, "1.0", "alpine-324", "amd64", "4.5"))
|
||
expect_false(identical(
|
||
k1, patch_cache_key(e2, "1.0", "alpine-324", "amd64", "4.5")
|
||
))
|
||
expect_match(k1, "^P_1.0_alpine-324_amd64_4.5_[0-9a-f]{12}$")
|
||
})
|
||
|
||
test_that("resolve_patch_version returns latest for wildcard, NA when unmet", {
|
||
local_mocked_bindings(
|
||
cran_package = function(pkg) list(Version = "5.1.12"),
|
||
.package = "pkgsearch"
|
||
)
|
||
expect_identical(
|
||
resolve_patch_version(list(package = "RcppParallel", versions = "*")),
|
||
"5.1.12"
|
||
)
|
||
expect_identical(
|
||
resolve_patch_version(list(package = "RcppParallel", versions = ">=9.0")),
|
||
NA_character_
|
||
)
|
||
})
|
||
|
||
test_that("describe_patch summarizes the active overrides", {
|
||
expect_match(
|
||
describe_patch(list(env = list(RCPP_PARALLEL_USE_TBB = "0"),
|
||
configure_args = character(0L), makevars = list(),
|
||
patch_path = NULL)),
|
||
"env: RCPP_PARALLEL_USE_TBB=0"
|
||
)
|
||
expect_match(
|
||
describe_patch(list(env = list(), configure_args = character(0L),
|
||
makevars = list(), patch_path = "/x/fix.patch")),
|
||
"source patch"
|
||
)
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: FAIL with "could not find function patch_cache_key".
|
||
|
||
- [ ] **Step 3: Write minimal implementation (append to `R/patches.R`)**
|
||
|
||
```r
|
||
#' Compute the cache key for a patched binary
|
||
#' @keywords internal
|
||
patch_cache_key <- function(entry, version, platform, arch, r_minor) {
|
||
payload <- list(
|
||
env = entry$env,
|
||
configure_args = entry$configure_args,
|
||
makevars = entry$makevars,
|
||
patch = if (!is.null(entry$patch_path)) {
|
||
readBin(entry$patch_path, "raw", file.size(entry$patch_path))
|
||
} else {
|
||
raw(0L)
|
||
}
|
||
)
|
||
tmp <- tempfile()
|
||
on.exit(unlink(tmp), add = TRUE)
|
||
saveRDS(payload, tmp)
|
||
hash <- substr(unname(tools::md5sum(tmp)), 1L, 12L)
|
||
sprintf(
|
||
"%s_%s_%s_%s_%s_%s",
|
||
entry$package, version, platform, arch, r_minor, hash
|
||
)
|
||
}
|
||
|
||
#' Resolve the CRAN version to build for a patch entry
|
||
#'
|
||
#' Returns the latest CRAN version satisfying the entry's `versions` constraint,
|
||
#' or `NA_character_` when CRAN's latest does not satisfy it or lookup fails.
|
||
#' @keywords internal
|
||
resolve_patch_version <- function(entry) {
|
||
latest <- tryCatch(
|
||
pkgsearch::cran_package(entry$package)$Version,
|
||
error = function(e) NA_character_
|
||
)
|
||
if (is.na(latest)) {
|
||
return(NA_character_)
|
||
}
|
||
if (identical(entry$versions, "*") || version_satisfies(latest, entry$versions)) {
|
||
return(latest)
|
||
}
|
||
NA_character_
|
||
}
|
||
|
||
#' Short human label describing a patch entry's overrides
|
||
#' @keywords internal
|
||
describe_patch <- function(entry) {
|
||
bits <- character(0L)
|
||
if (length(entry$env) > 0L) {
|
||
bits <- c(bits, sprintf(
|
||
"env: %s",
|
||
paste(
|
||
names(entry$env),
|
||
unlist(entry$env),
|
||
sep = "=", collapse = ","
|
||
)
|
||
))
|
||
}
|
||
if (length(entry$configure_args) > 0L) {
|
||
bits <- c(bits, sprintf("configure: %s", toString(entry$configure_args)))
|
||
}
|
||
if (length(entry$makevars) > 0L) {
|
||
bits <- c(bits, "makevars")
|
||
}
|
||
if (!is.null(entry$patch_path)) {
|
||
bits <- c(bits, "source patch")
|
||
}
|
||
if (length(bits) == 0L) "no-op" else paste(bits, collapse = "; ")
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: PASS. (Note: `local_mocked_bindings` requires testthat >= 3.1.7; bincraft uses 3e.)
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add R/patches.R tests/testthat/test-patches.R
|
||
git commit -m "feat(patches): cache key, version resolution, and patch description"
|
||
```
|
||
|
||
---
|
||
|
||
### Task A4: Build a patched binary in isolation
|
||
|
||
**Files:**
|
||
- Modify: `R/patches.R`
|
||
- Test: `tests/testthat/test-patches.R`
|
||
|
||
**Interfaces:**
|
||
- Consumes: a normalized entry, a resolved `version`, a `dest_dir`.
|
||
- Produces: `download_cran_source(package, version, dest_dir, cran)` → path or `NULL`; `apply_source_patch(patch_path, pkg_src)` → logical; `configure_args_to_build_args(configure_args)` → character; `build_patched_binary(entry, version, dest_dir)` → path to built binary tarball or `NULL`.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
```r
|
||
test_that("configure_args_to_build_args formats configure args", {
|
||
expect_identical(configure_args_to_build_args(character(0L)), character(0L))
|
||
expect_identical(
|
||
configure_args_to_build_args(c("--with-foo", "--no-bar")),
|
||
"--configure-args=--with-foo --no-bar"
|
||
)
|
||
})
|
||
|
||
test_that("apply_source_patch returns FALSE when patch does not apply", {
|
||
src <- withr::local_tempdir()
|
||
writeLines("unrelated content", file.path(src, "file.txt"))
|
||
bad_patch <- tempfile(fileext = ".patch")
|
||
writeLines(c(
|
||
"--- a/missing.txt", "+++ b/missing.txt",
|
||
"@@ -1 +1 @@", "-nope", "+nope2"
|
||
), bad_patch)
|
||
expect_false(apply_source_patch(bad_patch, src))
|
||
})
|
||
|
||
test_that("build_patched_binary returns NULL when download fails", {
|
||
local_mocked_bindings(download_cran_source = function(...) NULL)
|
||
expect_null(
|
||
build_patched_binary(
|
||
list(package = "P", env = list(), configure_args = character(0L),
|
||
makevars = list(), patch_path = NULL),
|
||
"1.0", withr::local_tempdir()
|
||
)
|
||
)
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: FAIL with "could not find function configure_args_to_build_args".
|
||
|
||
- [ ] **Step 3: Write minimal implementation (append to `R/patches.R`)**
|
||
|
||
```r
|
||
#' Download a CRAN source tarball for an exact version
|
||
#' @keywords internal
|
||
download_cran_source <- function(
|
||
package,
|
||
version,
|
||
dest_dir,
|
||
cran = "https://cloud.r-project.org"
|
||
) {
|
||
fname <- sprintf("%s_%s.tar.gz", package, version)
|
||
urls <- c(
|
||
sprintf("%s/src/contrib/%s", cran, fname),
|
||
sprintf("%s/src/contrib/Archive/%s/%s", cran, package, fname)
|
||
)
|
||
dest <- file.path(dest_dir, fname)
|
||
for (u in urls) {
|
||
ok <- tryCatch(
|
||
{
|
||
utils::download.file(u, dest, mode = "wb", quiet = TRUE)
|
||
file.exists(dest) && file.size(dest) > 0L
|
||
},
|
||
error = function(e) FALSE
|
||
)
|
||
if (isTRUE(ok)) {
|
||
return(dest)
|
||
}
|
||
}
|
||
log_warn(sprintf(
|
||
"Could not download CRAN source for {.pkg %s} %s.",
|
||
package, version
|
||
))
|
||
NULL
|
||
}
|
||
|
||
#' Apply a unified diff to an unpacked source tree
|
||
#'
|
||
#' Uses `patch -p1 --forward` so an already-applied or non-applying patch fails
|
||
#' cleanly (returns FALSE) instead of corrupting the tree.
|
||
#' @keywords internal
|
||
apply_source_patch <- function(patch_path, pkg_src) {
|
||
status <- system2(
|
||
"patch",
|
||
args = c(
|
||
"-p1", "--forward", "--batch",
|
||
"-d", shQuote(pkg_src),
|
||
"-i", shQuote(patch_path)
|
||
),
|
||
stdout = FALSE, stderr = FALSE
|
||
)
|
||
identical(status, 0L)
|
||
}
|
||
|
||
#' Format configure args for `pkgbuild::build(args = ...)`
|
||
#' @keywords internal
|
||
configure_args_to_build_args <- function(configure_args) {
|
||
if (length(configure_args) == 0L) {
|
||
return(character(0L))
|
||
}
|
||
sprintf("--configure-args=%s", paste(configure_args, collapse = " "))
|
||
}
|
||
|
||
#' Build a patched binary for one registry entry, in isolation
|
||
#'
|
||
#' Downloads CRAN source for `version`, applies the source patch (if any), and
|
||
#' builds a binary with the entry's env / configure / Makevars overrides scoped
|
||
#' to this build only. Returns the built tarball path, or `NULL` on any failure.
|
||
#' @keywords internal
|
||
build_patched_binary <- function(entry, version, dest_dir) {
|
||
workdir <- tempfile("patch_build_")
|
||
dir.create(workdir, recursive = TRUE, showWarnings = FALSE)
|
||
on.exit(unlink(workdir, recursive = TRUE, force = TRUE), add = TRUE)
|
||
|
||
src_tarball <- download_cran_source(entry$package, version, workdir)
|
||
if (is.null(src_tarball)) {
|
||
return(NULL)
|
||
}
|
||
|
||
utils::untar(src_tarball, exdir = workdir)
|
||
pkg_src <- file.path(workdir, entry$package)
|
||
|
||
if (!is.null(entry$patch_path)) {
|
||
if (!apply_source_patch(entry$patch_path, pkg_src)) {
|
||
log_warn(sprintf(
|
||
"Patch for {.pkg %s} %s did not apply cleanly; skipping patched build.",
|
||
entry$package, version
|
||
))
|
||
return(NULL)
|
||
}
|
||
}
|
||
|
||
build_env <- entry$env
|
||
if (length(entry$makevars) > 0L) {
|
||
mk <- tempfile(fileext = ".mk")
|
||
writeLines(
|
||
vapply(
|
||
names(entry$makevars),
|
||
function(k) sprintf("%s=%s", k, entry$makevars[[k]]),
|
||
character(1L)
|
||
),
|
||
mk
|
||
)
|
||
build_env$R_MAKEVARS_USER <- mk
|
||
}
|
||
|
||
tryCatch(
|
||
withr::with_envvar(build_env, {
|
||
pkgbuild::build(
|
||
path = pkg_src,
|
||
binary = TRUE,
|
||
vignettes = FALSE,
|
||
dest_path = dest_dir,
|
||
args = configure_args_to_build_args(entry$configure_args),
|
||
quiet = TRUE
|
||
)
|
||
}),
|
||
error = function(e) {
|
||
log_warn(sprintf(
|
||
"Isolated patched build of {.pkg %s} %s failed: %s",
|
||
entry$package, version, conditionMessage(e)
|
||
))
|
||
NULL
|
||
}
|
||
)
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add R/patches.R tests/testthat/test-patches.R
|
||
git commit -m "feat(patches): build a patched binary in isolation from CRAN source"
|
||
```
|
||
|
||
---
|
||
|
||
### Task A5: Orchestrate the local patched repo (cache + index)
|
||
|
||
**Files:**
|
||
- Modify: `R/patches.R`
|
||
- Test: `tests/testthat/test-patches.R`
|
||
|
||
**Interfaces:**
|
||
- Consumes: all helpers above.
|
||
- Produces: `prepare_patched_repo(patches_dir, platform, arch, r_minor, cache_dir, repo_dir)` → path to a `src/contrib`-style dir containing patched binaries + a `PACKAGES` index, or `NULL` when nothing matched/built. On a cache hit it copies the cached tarball into `repo_dir`; on a miss it builds, then writes the result into `cache_dir`.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
```r
|
||
test_that("prepare_patched_repo returns NULL when no entries match", {
|
||
dir <- withr::local_tempdir()
|
||
jsonlite::write_json(
|
||
list(list(package = "A", versions = "*", platforms = list("redhat"),
|
||
reason = "r")),
|
||
file.path(dir, "registry.json"), auto_unbox = TRUE
|
||
)
|
||
expect_null(
|
||
prepare_patched_repo(dir, "ubuntu-2604", "amd64", "4.5",
|
||
cache_dir = withr::local_tempdir(),
|
||
repo_dir = withr::local_tempdir())
|
||
)
|
||
})
|
||
|
||
test_that("prepare_patched_repo serves a cached binary and writes an index", {
|
||
dir <- withr::local_tempdir()
|
||
jsonlite::write_json(
|
||
list(list(package = "glue", versions = "*", platforms = list("*"),
|
||
env = list(A = "1"), reason = "r")),
|
||
file.path(dir, "registry.json"), auto_unbox = TRUE
|
||
)
|
||
cache <- withr::local_tempdir()
|
||
repo <- withr::local_tempdir()
|
||
|
||
local_mocked_bindings(
|
||
resolve_patch_version = function(entry) "1.0.0",
|
||
build_patched_binary = function(entry, version, dest_dir) {
|
||
f <- file.path(dest_dir, sprintf("%s_%s.tar.gz", entry$package, version))
|
||
writeLines("fake binary", f)
|
||
f
|
||
}
|
||
)
|
||
|
||
out <- prepare_patched_repo(dir, "ubuntu-2604", "amd64", "4.5",
|
||
cache_dir = cache, repo_dir = repo)
|
||
|
||
expect_identical(out, repo)
|
||
expect_true(file.exists(file.path(repo, "glue_1.0.0.tar.gz")))
|
||
expect_true(file.exists(file.path(repo, "PACKAGES")))
|
||
# The build result was cached under the key.
|
||
expect_length(list.files(cache, pattern = "^glue_1.0.0_.*\\.tar\\.gz$"), 1L)
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: FAIL with "could not find function prepare_patched_repo".
|
||
|
||
- [ ] **Step 3: Write minimal implementation (append to `R/patches.R`)**
|
||
|
||
```r
|
||
#' Prepare a local repo of patched binaries for the current build
|
||
#'
|
||
#' For each registry entry matching the current platform, ensures a patched
|
||
#' binary is present in `repo_dir` (from `cache_dir` if available, else built
|
||
#' and then cached) and writes a `PACKAGES` index over them.
|
||
#'
|
||
#' @param patches_dir Directory with `registry.json`, or `NULL`.
|
||
#' @param platform Build platform, e.g. `"ubuntu-2604"`.
|
||
#' @param arch Build arch, e.g. `"amd64"`.
|
||
#' @param r_minor R `"major.minor"` string, e.g. `"4.5"`.
|
||
#' @param cache_dir Persistent cache for patched binaries.
|
||
#' @param repo_dir Directory to assemble the local repo in.
|
||
#' @return `repo_dir` if at least one patched binary was produced, else `NULL`.
|
||
#' @keywords internal
|
||
prepare_patched_repo <- function(
|
||
patches_dir,
|
||
platform,
|
||
arch,
|
||
r_minor,
|
||
cache_dir = file.path("/mnt", "cache", "patched-binaries"),
|
||
repo_dir = tempfile("patched_repo_")
|
||
) {
|
||
entries <- match_patch_entries(
|
||
load_patch_registry(patches_dir), platform, arch
|
||
)
|
||
if (length(entries) == 0L) {
|
||
return(NULL)
|
||
}
|
||
|
||
dir.create(repo_dir, recursive = TRUE, showWarnings = FALSE)
|
||
dir.create(cache_dir, recursive = TRUE, showWarnings = FALSE)
|
||
|
||
produced <- 0L
|
||
for (entry in entries) {
|
||
version <- resolve_patch_version(entry)
|
||
if (is.na(version)) {
|
||
log_warn(sprintf(
|
||
"No CRAN version of {.pkg %s} satisfies '%s'; patch skipped.",
|
||
entry$package, entry$versions
|
||
))
|
||
next
|
||
}
|
||
|
||
key <- patch_cache_key(entry, version, platform, arch, r_minor)
|
||
cached <- file.path(cache_dir, sprintf("%s.tar.gz", key))
|
||
target <- file.path(
|
||
repo_dir, sprintf("%s_%s.tar.gz", entry$package, version)
|
||
)
|
||
|
||
if (file.exists(cached)) {
|
||
log_info(sprintf(
|
||
"Using cached patched binary for {.pkg %s} %s.",
|
||
entry$package, version
|
||
))
|
||
file.copy(cached, target, overwrite = TRUE)
|
||
} else {
|
||
log_info(sprintf(
|
||
"Applying patch to {.pkg %s} %s [%s]: %s",
|
||
entry$package, version, describe_patch(entry), entry$reason
|
||
))
|
||
built <- build_patched_binary(entry, version, repo_dir)
|
||
if (is.null(built)) {
|
||
next
|
||
}
|
||
if (!identical(normalizePath(built), normalizePath(target))) {
|
||
file.copy(built, target, overwrite = TRUE)
|
||
}
|
||
file.copy(target, cached, overwrite = TRUE)
|
||
}
|
||
produced <- produced + 1L
|
||
}
|
||
|
||
if (produced == 0L) {
|
||
return(NULL)
|
||
}
|
||
cranlike::add_PACKAGES(
|
||
list.files(repo_dir, pattern = "\\.tar\\.gz$"),
|
||
repo_dir
|
||
)
|
||
repo_dir
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run test to verify it passes**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 5: Update imports and document**
|
||
|
||
Add to `DESCRIPTION` `Imports:` (alphabetical) `jsonlite` if not present, and ensure `pkgsearch`, `pkgbuild`, `cranlike`, `withr` are listed (they are). Then:
|
||
|
||
```bash
|
||
Rscript -e 'devtools::document()'
|
||
git add R/patches.R tests/testthat/test-patches.R DESCRIPTION NAMESPACE
|
||
git commit -m "feat(patches): orchestrate local patched-binary repo with caching"
|
||
```
|
||
|
||
---
|
||
|
||
### Task A6: Wire patched repo into the pak install path
|
||
|
||
**Files:**
|
||
- Modify: `R/install_helpers.R:333-389` (`run_pak_install_with_mutex`)
|
||
- Modify: `R/install-deps.R:23-75` (`install_pkg_sys_deps`)
|
||
- Test: `tests/testthat/test-patches.R`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `prepare_patched_repo()`.
|
||
- Produces: `run_pak_install_with_mutex(local_clone_dir_single, env_vars, patched_repo = NULL)` — prepends `file://<patched_repo>` to `options("repos")` for the install; `install_pkg_sys_deps(package_name, tag, local_clone_dir, platform, aggressive_cleanup = FALSE, patches = NULL, arch = NULL)` — builds the patched repo before installing.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
```r
|
||
test_that("run_pak_install_with_mutex prepends the patched repo to repos", {
|
||
seen <- NULL
|
||
local_mocked_bindings(
|
||
acquire_pak_mutex = function(...) tempfile(),
|
||
release_pak_mutex = function(...) invisible(NULL),
|
||
retry_with_backoff = function(func, ...) func()
|
||
)
|
||
local_mocked_bindings(
|
||
local_install_deps = function(...) {
|
||
seen <<- getOption("repos")
|
||
invisible(TRUE)
|
||
},
|
||
.package = "pak"
|
||
)
|
||
|
||
run_pak_install_with_mutex(
|
||
tempfile(), list(), patched_repo = "/tmp/patched"
|
||
)
|
||
|
||
expect_true(any(grepl("file:///tmp/patched", seen)))
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: FAIL — `patched_repo` argument not yet accepted / repos not prepended.
|
||
|
||
- [ ] **Step 3: Edit `run_pak_install_with_mutex` in `R/install_helpers.R`**
|
||
|
||
Change the signature line:
|
||
|
||
```r
|
||
run_pak_install_with_mutex <- function(local_clone_dir_single, env_vars) {
|
||
```
|
||
|
||
to:
|
||
|
||
```r
|
||
run_pak_install_with_mutex <- function(
|
||
local_clone_dir_single,
|
||
env_vars,
|
||
patched_repo = NULL
|
||
) {
|
||
```
|
||
|
||
Replace the inner `retry_with_backoff(...)` block (the one wrapping `pak::local_install_deps`) with:
|
||
|
||
```r
|
||
retry_with_backoff(function() {
|
||
withr::with_envvar(env_vars, {
|
||
repos <- getOption("repos")
|
||
if (!is.null(patched_repo)) {
|
||
repos <- c(
|
||
patched = sprintf("file://%s", patched_repo),
|
||
repos
|
||
)
|
||
}
|
||
withr::with_options(list(repos = repos), {
|
||
# Default to non-verbose (suppressed messages)
|
||
suppressMessages(pak::local_install_deps(sprintf(
|
||
"%s",
|
||
local_clone_dir_single
|
||
)))
|
||
})
|
||
})
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 4: Edit `install_pkg_sys_deps` in `R/install-deps.R`**
|
||
|
||
Change the signature to add `patches` and `arch`:
|
||
|
||
```r
|
||
install_pkg_sys_deps <- function(
|
||
package_name,
|
||
tag,
|
||
local_clone_dir,
|
||
platform = platform,
|
||
aggressive_cleanup = FALSE,
|
||
patches = NULL,
|
||
arch = NULL
|
||
) {
|
||
```
|
||
|
||
Immediately before the `run_pak_install_with_mutex(...)` call, insert:
|
||
|
||
```r
|
||
# Build a local repo of patched binaries (if any apply) and serve it to pak.
|
||
r_minor <- paste(
|
||
R.version$major,
|
||
strsplit(R.version$minor, ".", fixed = TRUE)[[1L]][1L],
|
||
sep = "."
|
||
)
|
||
patched_repo <- tryCatch(
|
||
prepare_patched_repo(patches, platform, arch, r_minor),
|
||
error = function(e) {
|
||
log_warn(sprintf("Patch preparation failed: %s", conditionMessage(e)))
|
||
NULL
|
||
}
|
||
)
|
||
```
|
||
|
||
and change the call from:
|
||
|
||
```r
|
||
run_pak_install_with_mutex(
|
||
local_clone_dir_single,
|
||
env_vars
|
||
)
|
||
```
|
||
|
||
to:
|
||
|
||
```r
|
||
run_pak_install_with_mutex(
|
||
local_clone_dir_single,
|
||
env_vars,
|
||
patched_repo = patched_repo
|
||
)
|
||
```
|
||
|
||
- [ ] **Step 5: Run test to verify it passes**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add R/install_helpers.R R/install-deps.R tests/testthat/test-patches.R
|
||
git commit -m "feat(patches): serve patched binaries to pak during dep install"
|
||
```
|
||
|
||
---
|
||
|
||
### Task A7: Thread `patches` through the public build API
|
||
|
||
**Files:**
|
||
- Modify: `R/build_binaries.R` (`build_binary_package`, `execute_package_builds`, `build_single_tag`, `handle_system_dependencies`)
|
||
- Create: `man-roxygen/param-patches.R`
|
||
- Test: `tests/testthat/test-patches.R`
|
||
|
||
**Interfaces:**
|
||
- Produces: `build_binary_package(..., patches = NULL)` and the internal chain each carry `patches` down to `install_pkg_sys_deps()`. `handle_system_dependencies(..., patches = NULL)` passes `patches` and `arch` through.
|
||
|
||
- [ ] **Step 1: Write the failing test**
|
||
|
||
```r
|
||
test_that("handle_system_dependencies forwards patches and arch", {
|
||
captured <- list()
|
||
local_mocked_bindings(
|
||
install_pkg_sys_deps = function(package_name, tag, local_clone_dir_single,
|
||
platform, patches = NULL, arch = NULL) {
|
||
captured <<- list(patches = patches, arch = arch)
|
||
invisible(TRUE)
|
||
}
|
||
)
|
||
handle_system_dependencies(
|
||
"RcppParallel", "5.1.11-2", "ubuntu-2604", tempfile(), "amd64",
|
||
NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||
patches = "local/patches"
|
||
)
|
||
expect_identical(captured$patches, "local/patches")
|
||
expect_identical(captured$arch, "amd64")
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 2: Run test to verify it fails**
|
||
|
||
Run: `Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'`
|
||
Expected: FAIL — `handle_system_dependencies` has no `patches` argument.
|
||
|
||
- [ ] **Step 3: Create the roxygen template**
|
||
|
||
```r
|
||
# man-roxygen/param-patches.R
|
||
#' @param patches Optional path to a patch registry directory containing a
|
||
#' `registry.json` (and any referenced diff files). When set, matching
|
||
#' packages are pre-built as patched binaries and served to `pak` during
|
||
#' dependency installation. Defaults to `NULL` (no patching).
|
||
```
|
||
|
||
- [ ] **Step 4: Edit the four functions in `R/build_binaries.R`**
|
||
|
||
In `build_single_tag()`'s call to `handle_system_dependencies(...)`, add `patches = patches` as the final argument, and add `patches = NULL` to `build_single_tag`'s own signature plus `#' @template param-patches` to its roxygen block.
|
||
|
||
Change `handle_system_dependencies` signature to end with `metadata_db_sslmode,` then add `patches = NULL`, and change its inner `install_pkg_sys_deps(...)` call from:
|
||
|
||
```r
|
||
install_pkg_sys_deps(
|
||
package_name,
|
||
tag,
|
||
local_clone_dir_single,
|
||
platform
|
||
)
|
||
```
|
||
|
||
to:
|
||
|
||
```r
|
||
install_pkg_sys_deps(
|
||
package_name,
|
||
tag,
|
||
local_clone_dir_single,
|
||
platform,
|
||
patches = patches,
|
||
arch = arch
|
||
)
|
||
```
|
||
|
||
In `execute_package_builds()`, add `patches = NULL` to the signature and pass `patches = patches` into its `build_single_tag(...)` call inside `worker_function`.
|
||
|
||
In `build_binary_package()`, add `patches = NULL` to the signature (after `s3_package_cache`), add `#' @template param-patches` to its roxygen, and pass `patches = patches` into the `execute_package_builds(...)` call.
|
||
|
||
- [ ] **Step 5: Document, test, and run package check**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
Rscript -e 'devtools::document()'
|
||
Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'
|
||
```
|
||
|
||
Expected: PASS for the new test; `man/build_binary_package.Rd` etc. regenerated.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add R/build_binaries.R man-roxygen/param-patches.R man NAMESPACE tests/testthat/test-patches.R
|
||
git commit -m "feat(patches): thread patches argument through build_binary_package"
|
||
```
|
||
|
||
---
|
||
|
||
### Task A8: End-to-end patch test (guarded) and version bump
|
||
|
||
**Files:**
|
||
- Modify: `tests/testthat/test-patches.R`
|
||
- Modify: `DESCRIPTION` (version), `NEWS.md`
|
||
|
||
**Interfaces:**
|
||
- Produces: a guarded e2e test proving a dependent package builds when its failing dependency is patched.
|
||
|
||
- [ ] **Step 1: Add the guarded e2e test**
|
||
|
||
```r
|
||
test_that("a patched dependency unblocks a dependent build (e2e)", {
|
||
skip_if_not(nzchar(Sys.getenv("BINCRAFT_PATCH_E2E")))
|
||
skip_if_offline()
|
||
|
||
patches_dir <- withr::local_tempdir()
|
||
jsonlite::write_json(
|
||
list(list(
|
||
package = "RcppParallel", versions = "*", platforms = list("*"),
|
||
env = list(RCPP_PARALLEL_USE_TBB = "0"),
|
||
reason = "bundled TBB fails on this toolchain"
|
||
)),
|
||
file.path(patches_dir, "registry.json"), auto_unbox = TRUE
|
||
)
|
||
|
||
out <- withr::local_tempdir()
|
||
result <- build_binary_package(
|
||
"rts2", tag = "latest", local_output_dir_root = out,
|
||
upload = FALSE, archive = FALSE, patches = patches_dir
|
||
)
|
||
expect_true(isTRUE(result) || identical(result, "skipped"))
|
||
})
|
||
```
|
||
|
||
- [ ] **Step 2: Run the e2e test in a container**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
docker run --rm -e BINCRAFT_PATCH_E2E=1 -v "$PWD":/work -w /work \
|
||
reg.devxy.io/rpkgs/build-env-ubuntu:2604 \
|
||
Rscript -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-patches.R")'
|
||
```
|
||
|
||
Expected: the e2e test runs (not skipped) and PASSES; build log shows `Applying patch to RcppParallel`.
|
||
|
||
- [ ] **Step 3: Bump version and changelog**
|
||
|
||
In `DESCRIPTION`, bump `Version:` to `4.3.0.9999`. Prepend to `NEWS.md`:
|
||
|
||
```markdown
|
||
# bincraft 4.3.0
|
||
|
||
* `build_binary_package()` gains a `patches` argument: a registry of
|
||
per-package env / configure / Makevars overrides and source diffs that are
|
||
pre-built into patched binaries and served to pak, fixing compiler- and
|
||
OS-specific failures (e.g. RcppParallel) including for transitive deps.
|
||
```
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add tests/testthat/test-patches.R DESCRIPTION NEWS.md
|
||
git commit -m "test(patches): guarded end-to-end test; bump to 4.3.0.9999"
|
||
```
|
||
|
||
---
|
||
|
||
## Phase B — build-cran-binaries registry and wiring
|
||
|
||
All Phase B paths are relative to this repo (`build-cran-binaries`). Phase B depends on a bincraft build that includes Phase A (install the updated bincraft in the build containers, or bump the pinned version in `.crow/build-all-versions-install-deps.yaml`).
|
||
|
||
### Task B1: Create the patch registry with the RcppParallel entry
|
||
|
||
**Files:**
|
||
- Create: `local/patches/registry.json`
|
||
- Create: `local/patches/README.md`
|
||
|
||
**Interfaces:**
|
||
- Produces: the curated registry consumed by bincraft's `patches` argument.
|
||
|
||
- [ ] **Step 1: Write the registry**
|
||
|
||
```json
|
||
[
|
||
{
|
||
"package": "RcppParallel",
|
||
"versions": "*",
|
||
"platforms": ["alpine", "ubuntu-2604"],
|
||
"env": { "RCPP_PARALLEL_USE_TBB": "0" },
|
||
"configure_args": [],
|
||
"makevars": {},
|
||
"patch": null,
|
||
"reason": "bundled Intel TBB fails to build on musl and on newer toolchains (e.g. g++ 15 on ubuntu-2604); disabling TBB falls back to TinyThread"
|
||
}
|
||
]
|
||
```
|
||
|
||
- [ ] **Step 2: Write the README**
|
||
|
||
`local/patches/README.md` documents the schema (copy the field table from `specs/2026-06-30-package-patching-design.md`), how to add an entry, and that source diffs go in `local/patches/<pkg>/<file>.patch` referenced by the `patch` field.
|
||
|
||
- [ ] **Step 3: Verify it parses**
|
||
|
||
Run: `Rscript -e 'x <- jsonlite::fromJSON("local/patches/registry.json", simplifyVector = FALSE); stopifnot(length(x) == 1L, x[[1]]$package == "RcppParallel"); cat("ok\n")'`
|
||
Expected: `ok`.
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add local/patches/registry.json local/patches/README.md
|
||
git commit -m "feat(patches): add patch registry with RcppParallel TBB workaround"
|
||
```
|
||
|
||
---
|
||
|
||
### Task B2: Registry validator script
|
||
|
||
**Files:**
|
||
- Create: `local/validate-patches.R`
|
||
|
||
**Interfaces:**
|
||
- Consumes: `local/patches/registry.json`.
|
||
- Produces: a script that exits non-zero on schema violations, missing patch files, or ambiguous overlapping entries.
|
||
|
||
- [ ] **Step 1: Write the validator**
|
||
|
||
```r
|
||
#!/usr/bin/env Rscript
|
||
# Validate local/patches/registry.json: schema, referenced patch files, and
|
||
# ambiguous overlaps. Exits 1 on any problem. Used by pre-commit and CI.
|
||
|
||
dir <- "local/patches"
|
||
registry_file <- file.path(dir, "registry.json")
|
||
if (!file.exists(registry_file)) {
|
||
cat("No registry.json found; nothing to validate.\n")
|
||
quit(status = 0L)
|
||
}
|
||
|
||
reg <- jsonlite::fromJSON(registry_file, simplifyVector = FALSE)
|
||
required <- c("package", "versions", "platforms", "reason")
|
||
errs <- character(0L)
|
||
|
||
for (i in seq_along(reg)) {
|
||
e <- reg[[i]]
|
||
missing <- setdiff(required, names(e))
|
||
if (length(missing) > 0L) {
|
||
errs <- c(errs, sprintf(
|
||
"entry %d (%s): missing %s", i,
|
||
if (is.null(e$package)) "?" else e$package, toString(missing)
|
||
))
|
||
}
|
||
if (!is.null(e$patch)) {
|
||
p <- file.path(dir, e$patch)
|
||
if (!file.exists(p)) {
|
||
errs <- c(errs, sprintf("entry %d (%s): patch file '%s' missing",
|
||
i, e$package, p))
|
||
}
|
||
}
|
||
}
|
||
|
||
# Ambiguous overlap: two entries for the same package with identical platforms
|
||
# and versions.
|
||
keys <- vapply(reg, function(e) {
|
||
sprintf("%s|%s|%s", e$package,
|
||
paste(sort(as.character(unlist(e$platforms))), collapse = ","),
|
||
e$versions)
|
||
}, character(1L))
|
||
dups <- keys[duplicated(keys)]
|
||
if (length(dups) > 0L) {
|
||
errs <- c(errs, sprintf("ambiguous duplicate entries: %s", toString(unique(dups))))
|
||
}
|
||
|
||
if (length(errs) > 0L) {
|
||
cat("Patch registry validation FAILED:\n")
|
||
cat(paste0(" - ", errs, "\n"))
|
||
quit(status = 1L)
|
||
}
|
||
cat(sprintf("Patch registry OK (%d entrie(s)).\n", length(reg)))
|
||
```
|
||
|
||
- [ ] **Step 2: Run it (expect success on the B1 registry)**
|
||
|
||
Run: `Rscript local/validate-patches.R`
|
||
Expected: `Patch registry OK (1 entrie(s)).` and exit 0.
|
||
|
||
- [ ] **Step 3: Run it against a broken registry (expect failure)**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
cp local/patches/registry.json /tmp/reg.bak
|
||
Rscript -e 'writeLines("[{\"package\":\"X\"}]", "local/patches/registry.json")'
|
||
Rscript local/validate-patches.R; echo "exit=$?"
|
||
cp /tmp/reg.bak local/patches/registry.json
|
||
```
|
||
|
||
Expected: prints `validation FAILED` with a missing-field message and `exit=1`.
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add local/validate-patches.R
|
||
git commit -m "feat(patches): add registry validator script"
|
||
```
|
||
|
||
---
|
||
|
||
### Task B3: Hook the validator into pre-commit
|
||
|
||
**Files:**
|
||
- Modify: `.pre-commit-config.yaml`
|
||
|
||
**Interfaces:**
|
||
- Produces: a local hook that runs `local/validate-patches.R` when the registry or patch files change.
|
||
|
||
- [ ] **Step 1: Add the hook**
|
||
|
||
Add a `repo: local` hook entry to `.pre-commit-config.yaml`:
|
||
|
||
```yaml
|
||
- repo: local
|
||
hooks:
|
||
- id: validate-patches
|
||
name: validate patch registry
|
||
entry: Rscript local/validate-patches.R
|
||
language: system
|
||
files: ^local/patches/
|
||
pass_filenames: false
|
||
```
|
||
|
||
- [ ] **Step 2: Verify the hook runs**
|
||
|
||
Run: `pre-commit run validate-patches --all-files`
|
||
Expected: hook passes (`Patch registry OK`).
|
||
|
||
- [ ] **Step 3: Commit**
|
||
|
||
```bash
|
||
git add .pre-commit-config.yaml
|
||
git commit -m "ci(patches): validate patch registry in pre-commit"
|
||
```
|
||
|
||
---
|
||
|
||
### Task B4: Pass `patches` through the build entry points
|
||
|
||
**Files:**
|
||
- Modify: `local/build-one.R:99-119` (the `build_binary_package` call)
|
||
- Modify: `local/build-all.R:125-145` (the `build_binary_package` call)
|
||
|
||
**Interfaces:**
|
||
- Consumes: the new bincraft `patches` argument (Phase A) and `local/patches/`.
|
||
- Produces: both entry points pass `patches = "local/patches"`.
|
||
|
||
- [ ] **Step 1: Edit `local/build-one.R`**
|
||
|
||
In the `bincraft::build_binary_package(` call, add as a new argument (e.g. after `archive = TRUE,`):
|
||
|
||
```r
|
||
patches = "local/patches",
|
||
```
|
||
|
||
- [ ] **Step 2: Edit `local/build-all.R`**
|
||
|
||
In the `bincraft::build_binary_package(` call, add:
|
||
|
||
```r
|
||
patches = "local/patches",
|
||
```
|
||
|
||
- [ ] **Step 3: Verify the scripts still parse**
|
||
|
||
Run: `Rscript -e 'invisible(parse("local/build-one.R")); invisible(parse("local/build-all.R")); cat("parse ok\n")'`
|
||
Expected: `parse ok`.
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add local/build-one.R local/build-all.R
|
||
git commit -m "feat(patches): pass patch registry to bincraft build calls"
|
||
```
|
||
|
||
---
|
||
|
||
### Task B5: Document the feature in the README
|
||
|
||
**Files:**
|
||
- Modify: `README.md`
|
||
|
||
**Interfaces:**
|
||
- Produces: a short "Patching packages" section explaining the registry and linking the design spec.
|
||
|
||
- [ ] **Step 1: Add a README section**
|
||
|
||
Add a `## Patching packages` section after the "Build Process" section describing: why patching exists (compiler/OS-specific failures cascading via shared deps like RcppParallel), where the registry lives (`local/patches/registry.json`), the two tiers (env/configure/Makevars overrides vs source diffs), and that bincraft pre-builds patched binaries served to pak. Link `specs/2026-06-30-package-patching-design.md`.
|
||
|
||
- [ ] **Step 2: Commit**
|
||
|
||
```bash
|
||
git add README.md
|
||
git commit -m "docs(patches): document the package patching workflow"
|
||
```
|
||
|
||
---
|
||
|
||
## Self-Review
|
||
|
||
**Spec coverage:**
|
||
- Registry in this repo, passed to bincraft → Task A7 (`patches` arg), B1 (registry), B4 (wiring). ✓
|
||
- Mechanism in bincraft (resolve → pre-build → prepend repo → install) → A4–A6. ✓
|
||
- Both tiers, env-first → A4 (`env`, `configure_args`, `makevars`, `patch` all applied; env is the cheap default and the RcppParallel entry uses only env). ✓
|
||
- Pre-built patched binary served from a prepended local repo → A0 proof + A5/A6. ✓
|
||
- Cache keyed by pkg/version/platform/arch/rminor/patchhash → A3 (`patch_cache_key`) + A5 (cache use). ✓
|
||
- S3 upload of patched binaries → **partial**: A5 caches locally only. Cross-machine S3 reuse is deferred (see note below) to keep the first cut shippable; local `/mnt/cache` reuse already removes per-dependent rebuilds within a container. Recorded as out-of-first-cut, not dropped.
|
||
- Error handling (diff fails → skip+warn; build fails → skip+warn; no match → skip; ambiguous overlap → validation error) → A4 (`apply_source_patch` FALSE path, build tryCatch), A5 (`resolve_patch_version` NA path), B2 (overlap validation). ✓
|
||
- Observability (log line per applied patch) → A5 (`log_info` with `describe_patch`). DB metadata recording of applied patches is deferred with S3 (same note).
|
||
- Testing (registry parse/match, cache key, integration, failure path) → A1–A5 unit tests, A8 e2e, A4 patch-fail test. ✓
|
||
|
||
**Deferred from spec (call out to user):** S3 upload/reuse of patched binaries and recording applied patches in the Postgres build-metadata row. The local `/mnt/cache/patched-binaries` cache already prevents repeated rebuilds within a container; S3 reuse across CI jobs is a follow-up. If you want it in the first cut, add a Task A5b (upload `cached` tarball to a `…/patched/` S3 slot and check there before building) and a metadata column — say so and I'll insert them.
|
||
|
||
**Placeholder scan:** No TBD/TODO; every code step shows complete code. README/registry-README prose steps describe exact content to write (acceptable for docs).
|
||
|
||
**Type consistency:** `prepare_patched_repo` signature is identical across A5 (definition) and A6 (call site, via defaults). `patch_cache_key(entry, version, platform, arch, r_minor)` argument order matches between A3 and A5. `run_pak_install_with_mutex(..., patched_repo)` matches between A6 definition and the A6 test. `handle_system_dependencies(..., patches)` matches between A7 edit and A7 test. `build_patched_binary(entry, version, dest_dir)` matches A4 and A5.
|
||
|
||
**Open risk:** Task A0 gates the whole approach; if it fails, the documented source-serving contingency keeps every later task valid with a localized change in A4.
|