feat: patch registry + wiring for per-package patching (#103)
Some checks failed
ci/crow/cron/process-updates/15 Pipeline failed
ci/crow/cron/process-updates/14 Pipeline failed
ci/crow/cron/process-updates/16 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/5 Pipeline failed
Some checks failed
ci/crow/cron/process-updates/15 Pipeline failed
ci/crow/cron/process-updates/14 Pipeline failed
ci/crow/cron/process-updates/16 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/5 Pipeline failed
## Summary Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`). Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading. ## What's included - `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs. - `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace). - `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes). - `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`. - `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`. ## ⚠️ Merge ordering (blocker) This PR adds a `patches = ...` argument to `build_binary_package()` calls. The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until: 1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and 2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`. The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then. Reviewed-on: #103
This commit is contained in:
parent
1e910bc703
commit
55a18fd87d
16 changed files with 1791 additions and 10 deletions
|
|
@ -62,7 +62,7 @@ steps:
|
|||
GIT_USER: pat-s
|
||||
R_VERSION: 4.5.3
|
||||
commands:
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3", dependencies = TRUE)'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1", dependencies = TRUE)'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'bincraft::process_unarchived_pkgs(Sys.getenv("CODENAME"), Sys.getenv("ARCH"), s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), workers = 2L)'
|
||||
backend_options:
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ steps:
|
|||
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
||||
# Pin the same bincraft version the build steps use, so the precomputed
|
||||
# snapshot and the per-agent library stay consistent across the pipeline.
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'pak::sysreqs_db_update(); pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3"); pak::pak(c("RPostgres", "s3fs", "data.table", "future", "jsonlite")); packageVersion("bincraft")'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'pak::sysreqs_db_update(); pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1"); pak::pak(c("RPostgres", "s3fs", "data.table", "future", "jsonlite")); packageVersion("bincraft")'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))"
|
||||
backend_options:
|
||||
docker:
|
||||
|
|
|
|||
|
|
@ -121,7 +121,7 @@ steps:
|
|||
# to a zero-length value and breaks every metadata query and the sysdeps
|
||||
# install). Pin bincraft here, exactly like the R-minor pass below.
|
||||
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")'
|
||||
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
|
||||
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1
|
||||
- |
|
||||
|
|
@ -133,7 +133,7 @@ steps:
|
|||
echo "=== R-minor-sensitive pass under R $RV ==="
|
||||
LIB="/mnt/cache/R-pkgs-$RMINOR"
|
||||
mkdir -p "$LIB"
|
||||
R_LIBS_USER="$LIB" "$(dirname "$RBIN")/R" -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")' || true
|
||||
R_LIBS_USER="$LIB" "$(dirname "$RBIN")/R" -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")' || true
|
||||
R_LIBS_USER="$LIB" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$(dirname "$RBIN")/Rscript" local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true
|
||||
done
|
||||
# archive missed packages; first arg is the codename (e.g. "alpine324"),
|
||||
|
|
|
|||
|
|
@ -192,13 +192,13 @@ steps:
|
|||
commands:
|
||||
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
||||
- rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/bincraft /mnt/cache/R-pkgs/pkgcache /mnt/cache/pkgcache/R/pkgcache
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")'
|
||||
- mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
|
||||
# rhel-10 ships xwfb-run (Xwayland) instead of xvfb-run; prefer it and start weston, else fall back to xvfb-run
|
||||
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
|
||||
# options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise
|
||||
- $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '${OS}', process_updated = TRUE, process_new = ${PROCESS_NEW}, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)"
|
||||
- $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '${OS}', process_updated = TRUE, process_new = ${PROCESS_NEW}, process_removed = TRUE, patches = 'local/patches', r_minor_detection = 'classifier',s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)"
|
||||
- |
|
||||
PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
|
||||
for RBIN in /opt/R/[0-9]*/bin/R; do
|
||||
|
|
@ -208,8 +208,8 @@ steps:
|
|||
echo "=== R-minor-sensitive update pass under R $RV ==="
|
||||
LIB="/mnt/cache/R-pkgs-$RMINOR"
|
||||
mkdir -p "$LIB"
|
||||
R_LIBS_USER="$LIB" "$(dirname "$RBIN")/R" -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")' || true
|
||||
R_LIBS_USER="$LIB" $XVFB $XVFB_ARGS -- "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '${OS}', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true
|
||||
R_LIBS_USER="$LIB" "$(dirname "$RBIN")/R" -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")' || true
|
||||
R_LIBS_USER="$LIB" $XVFB $XVFB_ARGS -- "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '${OS}', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, patches = 'local/patches', r_minor_detection = 'classifier',r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true
|
||||
done
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "${OS_ID}", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))'
|
||||
- |
|
||||
|
|
|
|||
|
|
@ -129,12 +129,12 @@ steps:
|
|||
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
||||
- mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
|
||||
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")'
|
||||
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")'
|
||||
- /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")'
|
||||
- $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); pkgs <- readLines('/tmp/rebuild_pkgs.txt'); if (length(pkgs) == 0) { cat('Nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s - %s\n', x, conditionMessage(e)))) }" 2>&1
|
||||
- $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); pkgs <- readLines('/tmp/rebuild_pkgs.txt'); if (length(pkgs) == 0) { cat('Nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, patches = 'local/patches', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s - %s\n', x, conditionMessage(e)))) }" 2>&1
|
||||
backend_options:
|
||||
docker:
|
||||
resources:
|
||||
|
|
|
|||
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -99,3 +99,6 @@ docs/
|
|||
local/test.R
|
||||
.DS_Store
|
||||
docs/
|
||||
|
||||
# Superpowers SDD scratch (briefs, reports, ledger)
|
||||
.superpowers/
|
||||
|
|
|
|||
|
|
@ -2,3 +2,6 @@
|
|||
ignores:
|
||||
- LICENSE.md
|
||||
- docs/superpowers/**
|
||||
# Internal design docs (specs/plans) are not user-facing reference material.
|
||||
- specs/**
|
||||
- plans/**
|
||||
|
|
|
|||
|
|
@ -43,3 +43,9 @@ repos:
|
|||
entry: YAML filenames must have .yaml extension.
|
||||
language: fail
|
||||
files: .yml$
|
||||
- id: validate-patches
|
||||
name: validate patch registry
|
||||
entry: Rscript local/validate-patches.R
|
||||
language: system
|
||||
files: ^local/(patches/|validate-patches\.R$)
|
||||
pass_filenames: false
|
||||
|
|
|
|||
22
README.md
22
README.md
|
|
@ -53,6 +53,28 @@ For every package+tag combination:
|
|||
1. Archive old package versions and keep the latest one in the root
|
||||
1. Delete local binaries after successful upload
|
||||
|
||||
## Patching packages
|
||||
|
||||
Some CRAN packages fail to compile on specific platforms due to compiler- or OS-specific issues unrelated to the package itself.
|
||||
The canonical example is `RcppParallel`, whose bundled TBB sources fail on musl (Alpine) and newer compiler/OS combinations.
|
||||
Because such packages are often transitive dependencies of many others, a single failure cascades: all dependents fail even though nothing is wrong with the dependent itself.
|
||||
|
||||
To address this, frequently-failing packages can be "patched" before they are installed — whether as a direct build target or a transitive dependency pulled in by `pak`.
|
||||
|
||||
The patch registry lives in `local/patches/registry.json`.
|
||||
Each entry specifies a package and the platforms/versions it applies to, along with either lightweight build-time overrides (environment variables, configure arguments, Makevars) or a source diff (for deeper fixes).
|
||||
See `local/patches/README.md` for the complete schema.
|
||||
|
||||
Patching uses a two-tier approach:
|
||||
|
||||
1. **Lightweight overrides:** environment variables, configure arguments, or Makevars settings applied during build — typically version-independent and fast.
|
||||
2. **Source diffs:** unified diff patches applied to the unpacked source before building — more powerful but version-pinned.
|
||||
|
||||
The system is implemented in `bincraft`: when a package needs patching, `bincraft` pre-builds it with the patch and serves the patched binary to `pak`, ensuring transitive dependents receive the fixed package.
|
||||
This way, the fix cascades to all packages that depend on it.
|
||||
|
||||
For the design rationale and architecture, see `specs/2026-06-30-package-patching-design.md`.
|
||||
|
||||
## Build Environment
|
||||
|
||||
Binaries are built on a mixed-architecture Kubernetes cluster using CI.
|
||||
|
|
|
|||
|
|
@ -140,6 +140,7 @@ mapply(
|
|||
metadata_db_sslmode = "require",
|
||||
metadata_db_port = 15432,
|
||||
archive = TRUE,
|
||||
patches = "local/patches",
|
||||
upload = TRUE,
|
||||
store_build_metadata = TRUE
|
||||
)
|
||||
|
|
|
|||
|
|
@ -103,6 +103,7 @@ for (ver in versions) {
|
|||
force = TRUE,
|
||||
upload = TRUE,
|
||||
archive = TRUE,
|
||||
patches = "local/patches",
|
||||
store_build_metadata = TRUE,
|
||||
s3_endpoint = s3$s3_endpoint,
|
||||
s3_region = s3$s3_region,
|
||||
|
|
|
|||
43
local/patches/README.md
Normal file
43
local/patches/README.md
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
# Patch Registry
|
||||
|
||||
This directory contains the curated registry of per-package build-time patches consumed by bincraft's `patches` argument.
|
||||
|
||||
## Schema
|
||||
|
||||
The registry is defined in `registry.json` as an array of patch entries. Each entry specifies lightweight build-time overrides (environment variables, configure arguments, Makevars) and optionally a source diff to apply before building.
|
||||
|
||||
### Field semantics
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `package` | string | yes | CRAN package name. |
|
||||
| `versions` | string | yes | `"*"` for any, a constraint such as `">=5.1.0"`, or an exact version `"5.1.11-2"`. Env-tier fixes are typically `"*"`; source diffs are normally exact or lower-bounded because a diff is pinned to the source it was generated against. |
|
||||
| `platforms` | array of strings | yes | Matched against the running build's platform tokens — distro family (`alpine`, `ubuntu`, `redhat`), codename (`ubuntu-2604`, `alpine-324`), and arch (`amd64`, `arm64`). An entry matches if any listed token matches any build token. `["*"]` matches all platforms. |
|
||||
| `env` | object | no | Environment variables exported only for this package's isolated build. |
|
||||
| `configure_args` | array | no | Arguments passed as `--configure-args` to the isolated build. |
|
||||
| `makevars` | object | no | Key/value pairs written into a package-local Makevars for the isolated build. |
|
||||
| `patch` | string or null | no | Path (relative to `local/patches/`) to a unified diff applied to the unpacked CRAN source before building. |
|
||||
| `reason` | string | yes | Human explanation, surfaced in logs and metadata. |
|
||||
|
||||
## Adding an entry
|
||||
|
||||
To add a new patch entry:
|
||||
|
||||
1. Add an object to the array in `registry.json` with the fields documented above.
|
||||
Start with lightweight overrides (environment variables, configure arguments, Makevars) before resorting to source diffs.
|
||||
|
||||
2. If a source diff is needed, place it in `local/patches/<package>/<file>.patch` and reference its path in the `patch` field.
|
||||
For example, a diff for `RcppParallel` would go in `local/patches/RcppParallel/fix.patch` and be referenced as `"patch": "RcppParallel/fix.patch"`.
|
||||
|
||||
3. The `reason` field should clearly explain why the patch is needed and what problem it solves.
|
||||
|
||||
## Validation
|
||||
|
||||
The registry is validated and applied by bincraft during the build process.
|
||||
For manual validation, run the validator from the repo root:
|
||||
|
||||
```bash
|
||||
Rscript local/validate-patches.R
|
||||
```
|
||||
|
||||
This validates the schema, referenced patch-file existence, and checks for duplicate entries across platforms and versions.
|
||||
12
local/patches/registry.json
Normal file
12
local/patches/registry.json
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
[
|
||||
{
|
||||
"package": "RcppParallel",
|
||||
"versions": "*",
|
||||
"platforms": ["alpine", "ubuntu-2604"],
|
||||
"env": { "RCPP_PARALLEL_USE_TBB": "0" },
|
||||
"configure_args": [],
|
||||
"makevars": {},
|
||||
"patch": null,
|
||||
"reason": "bundled Intel TBB fails to build on musl and on newer toolchains (e.g. g++ 15 on ubuntu-2604); disabling TBB falls back to TinyThread"
|
||||
}
|
||||
]
|
||||
56
local/validate-patches.R
Normal file
56
local/validate-patches.R
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
#!/usr/bin/env Rscript
|
||||
# Validate local/patches/registry.json: schema, referenced patch files, and
|
||||
# ambiguous overlaps. Exits 1 on any problem. Used by pre-commit and CI.
|
||||
|
||||
dir <- "local/patches"
|
||||
registry_file <- file.path(dir, "registry.json")
|
||||
if (!file.exists(registry_file)) {
|
||||
cat("No registry.json found; nothing to validate.\n")
|
||||
quit(status = 0L)
|
||||
}
|
||||
|
||||
or_q <- function(x) if (is.null(x)) "?" else x
|
||||
|
||||
reg <- jsonlite::fromJSON(registry_file, simplifyVector = FALSE)
|
||||
required <- c("package", "versions", "platforms", "reason")
|
||||
errs <- character(0L)
|
||||
|
||||
for (i in seq_along(reg)) {
|
||||
e <- reg[[i]]
|
||||
missing <- setdiff(required, names(e))
|
||||
if (length(missing) > 0L) {
|
||||
errs <- c(errs, sprintf(
|
||||
"entry %d (%s): missing %s", i,
|
||||
if (is.null(e$package)) "?" else e$package, toString(missing)
|
||||
))
|
||||
}
|
||||
if (!is.null(e$patch)) {
|
||||
p <- file.path(dir, e$patch)
|
||||
if (!file.exists(p)) {
|
||||
errs <- c(errs, sprintf("entry %d (%s): patch file '%s' missing",
|
||||
i, e$package, p))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Ambiguous overlap: two entries for the same package with identical platforms
|
||||
# and versions.
|
||||
keys <- vapply(reg, function(e) {
|
||||
sprintf(
|
||||
"%s|%s|%s",
|
||||
or_q(e$package),
|
||||
paste(sort(as.character(unlist(e$platforms))), collapse = ","),
|
||||
or_q(e$versions)
|
||||
)
|
||||
}, character(1L))
|
||||
dups <- keys[duplicated(keys)]
|
||||
if (length(dups) > 0L) {
|
||||
errs <- c(errs, sprintf("ambiguous duplicate entries: %s", toString(unique(dups))))
|
||||
}
|
||||
|
||||
if (length(errs) > 0L) {
|
||||
cat("Patch registry validation FAILED:\n")
|
||||
cat(paste0(" - ", errs, "\n"))
|
||||
quit(status = 1L)
|
||||
}
|
||||
cat(sprintf("Patch registry OK (%d %s).\n", length(reg), if (length(reg) == 1L) "entry" else "entries"))
|
||||
1477
plans/2026-06-30-package-patching-implementation.md
Normal file
1477
plans/2026-06-30-package-patching-implementation.md
Normal file
File diff suppressed because it is too large
Load diff
157
specs/2026-06-30-package-patching-design.md
Normal file
157
specs/2026-06-30-package-patching-design.md
Normal file
|
|
@ -0,0 +1,157 @@
|
|||
# Design: Dynamic per-package patching during binary builds
|
||||
|
||||
Date: 2026-06-30
|
||||
Status: Approved (pending spec review)
|
||||
|
||||
## Problem
|
||||
|
||||
Some CRAN packages fail to compile on specific build platforms due to compiler- or OS-specific issues that have nothing to do with the package being built.
|
||||
The canonical example is `RcppParallel`: its bundled Intel TBB sources fail to build on musl (Alpine) and on newer OS/compiler combinations.
|
||||
Observed failure on `ubuntu-2604` ("resolute") with `g++ 15.2.0`:
|
||||
|
||||
```
|
||||
../build/common.inc:74: *** "" is not supported. Add build/.inc file with os-specific settings . Stop.
|
||||
make: *** [Makevars:163: tbb] Error 2
|
||||
ERROR: compilation failed for package 'RcppParallel'
|
||||
```
|
||||
|
||||
Because `RcppParallel` is a dependency of many packages, a single such failure cascades: every dependent package (e.g. `rts2`) also fails, even though nothing is wrong with the dependent itself.
|
||||
|
||||
Today there is no way to intervene.
|
||||
A package can only be **excluded** (`local/excluded-packages.json`), which is all-or-nothing and does not help dependents.
|
||||
|
||||
## Goal
|
||||
|
||||
Allow a curated set of packages to be "patched" — via lightweight build-time overrides or, when necessary, real source diffs — **before** they are installed, whether the package is a direct build target or a transitive dependency pulled in by `pak`.
|
||||
|
||||
## Key constraint that drives the design
|
||||
|
||||
When `RcppParallel` fails here, it is being installed as a **transitive dependency** by `pak`, inside `bincraft::build_binary_package()`.
|
||||
`pak` downloads, configures, and compiles it in one subprocess; this repo never touches that source.
|
||||
For a fix to reach a dependency-of-a-dependency, the fixed package must be visible to `pak` itself, where `pak`'s repositories/sources are configured — which is inside `bincraft`.
|
||||
|
||||
Decisions taken during brainstorming:
|
||||
|
||||
- **Mechanism lives in `bincraft`** (the engine), because only there can transitive deps be influenced.
|
||||
- **Patch tiers: both, env-overrides first.** Support cheap per-package env vars / configure args / Makevars (version-independent) *and* true source diffs (version-pinned), preferring the lightweight override.
|
||||
- **Registry data lives in this repo** (`build-cran-binaries`) and is passed into `bincraft`, keeping `bincraft` as pure mechanism and the frequently-changing policy data with operational config.
|
||||
|
||||
## Approaches considered
|
||||
|
||||
| Approach | How pak sees the fix | Verdict |
|
||||
|---|---|---|
|
||||
| A. Patched **source** repo — drop patched `.tar.gz` source into a local repo, prepend it | pak recompiles from your source | Simple, but env-tier overrides leak globally (one subprocess builds everything) and the dep recompiles on every dependent build |
|
||||
| **B. Pre-built patched binary repo (chosen)** | pak installs a ready binary by repo priority | Per-package scoping is free; no recompile; the binary is a cacheable/uploadable artifact that fits the existing system |
|
||||
| C. pkgdepends per-build hook | intercept each build | No clean per-package pre-compile hook exists; fragile |
|
||||
|
||||
Chosen: **B**.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Registry (this repo)
|
||||
|
||||
```
|
||||
local/patches/
|
||||
registry.json # the manifest
|
||||
RcppParallel/
|
||||
fix.patch # optional source diff, referenced by an entry
|
||||
```
|
||||
|
||||
`registry.json` is an array of entries:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"package": "RcppParallel",
|
||||
"versions": "*",
|
||||
"platforms": ["alpine", "ubuntu-2604"],
|
||||
"env": { "RCPP_PARALLEL_USE_TBB": "0" },
|
||||
"configure_args": [],
|
||||
"makevars": {},
|
||||
"patch": null,
|
||||
"reason": "bundled TBB fails to build on musl / newer compilers"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Field semantics:
|
||||
|
||||
- `package` (string, required): CRAN package name.
|
||||
- `versions` (string, required): `"*"` for any, a constraint such as `">=5.1.0"`, or an exact version `"5.1.11-2"`.
|
||||
Env-tier fixes are typically `"*"`; source diffs are normally exact or lower-bounded because a diff is pinned to the source it was generated against.
|
||||
- `platforms` (array of strings, required): matched against the running build's platform tokens — distro family (`alpine`, `ubuntu`, `redhat`), codename (`ubuntu-2604`, `alpine-324`), and arch (`amd64`, `arm64`).
|
||||
An entry matches if any listed token matches any build token.
|
||||
`["*"]` matches all platforms.
|
||||
- `env` (object, optional): environment variables exported only for this package's isolated build.
|
||||
- `configure_args` (array, optional): passed as `--configure-args` to the isolated build.
|
||||
- `makevars` (object, optional): key/value pairs written into a package-local Makevars for the isolated build.
|
||||
- `patch` (string or null, optional): path (relative to `local/patches/`) to a unified diff applied to the unpacked CRAN source before building.
|
||||
- `reason` (string, required): human explanation, surfaced in logs and metadata.
|
||||
|
||||
A fix is any combination of `env`, `configure_args`, `makevars`, and `patch`.
|
||||
"Env-first" is an authoring guideline (prefer the lightweight override) and an ordering of effort, not a runtime branch — all present fields are applied together for the isolated build.
|
||||
|
||||
### Flow (inside bincraft, around existing pak resolution)
|
||||
|
||||
1. **Resolve** the dependency set (dry-run) to learn the concrete versions `pak` will install.
|
||||
Reuse bincraft's existing resolution where possible (e.g. a `pkgdepends` proposal: `$resolve()` → inspect resolution → ... → `$solve()` / `$install()` after the local repo is prepended).
|
||||
2. For each resolved package that matches a registry entry (name + `versions` + `platforms`): obtain a **patched binary** for the exact `version × platform × arch × R-minor`:
|
||||
- **Cache hit** (local `/mnt/cache/patched-binaries/` or S3): fetch it into the local repo.
|
||||
- **Cache miss**: download the CRAN **source** for that version, apply the source `patch` (if any) to the unpacked tree, build the binary in isolation with `env` / `configure_args` / `makevars` applied, then place the binary in the local repo and write it to the cache (and S3 if uploading is enabled).
|
||||
3. **Prepend** the local binary repo (`file://…`) to `pak`'s repo list, and regenerate its `PACKAGES` index.
|
||||
4. Run the **normal install**.
|
||||
`pak` resolves the patched binary for the matched package — direct or transitive — because it wins on repo priority for an equal version, and installs it without recompiling.
|
||||
|
||||
### Caching (essential)
|
||||
|
||||
`RcppParallel` is a dependency of dozens of packages; without caching the fix would be rebuilt on every dependent build.
|
||||
Patched binaries are keyed by:
|
||||
|
||||
```
|
||||
<package>_<version>_<platform>_<arch>_<rminor>_<patchhash>
|
||||
```
|
||||
|
||||
`patchhash` is a hash of the normalized registry entry plus the referenced diff file contents.
|
||||
Editing a patch therefore changes the hash and auto-invalidates stale cached binaries.
|
||||
|
||||
- Local cache: `/mnt/cache/patched-binaries/`.
|
||||
- Optional S3 cache for cross-build reuse: a dedicated `…/patched/` slot under the existing arch/codename structure, mirroring how normal binaries are stored.
|
||||
|
||||
### S3 upload
|
||||
|
||||
Patched binaries **are** uploaded to S3 (in addition to the local cache) so they are reused across CI jobs and machines, not just within one container.
|
||||
They live in a separate `patched/` slot and are not published into the user-facing `src/contrib` index — they are an internal build accelerator, not a distributed artifact.
|
||||
|
||||
## Error handling
|
||||
|
||||
- **Source diff fails to apply** (CRAN moved past the pinned version): log a clear warning, skip that entry, and proceed.
|
||||
The package builds unpatched (status quo) and may fail.
|
||||
The skipped/failed-to-apply patch is surfaced in build metadata.
|
||||
- **Pre-build of the patched binary fails**: log a warning, skip, proceed.
|
||||
- **No version or platform match**: skip silently (the entry simply does not apply to this build).
|
||||
- **Overlapping entries for one package**: the most specific entry wins (a concrete `platforms`/`versions` beats `"*"`).
|
||||
Genuine ambiguity (two equally specific, conflicting entries) is a validation error reported before the build.
|
||||
|
||||
## Observability
|
||||
|
||||
- One log line per applied patch, e.g.: `Applying patch to RcppParallel 5.1.11-2 [env: RCPP_PARALLEL_USE_TBB=0]: bundled TBB fails on musl / newer compilers`
|
||||
- The set of applied patches (package, version, `patchhash`) is recorded in the Postgres build-metadata row for the build, so it is queryable later.
|
||||
|
||||
## Testing
|
||||
|
||||
- **Unit (registry):** parsing and matching — version constraints, platform token matching, precedence/specificity, and detection of ambiguous overlaps.
|
||||
- **Unit (cache key):** `patchhash` changes when the entry or diff changes; is stable otherwise.
|
||||
- **Integration:** `RcppParallel` on `resolute` (and/or Alpine) fails to build without a registry entry and succeeds with one; a dependent package such as `rts2` succeeds once the dependency is patched.
|
||||
- **Failure path:** an entry pinned to an old version against a newer CRAN release → graceful skip with a warning, build continues.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Shipping a default registry inside `bincraft` (registry is repo-local for now; a baseline-in-engine + repo-override model can come later if needed).
|
||||
- Publishing patched binaries into the public `src/contrib` index.
|
||||
- Automatic detection of which packages need patches — entries are curated by hand.
|
||||
|
||||
## Split of work
|
||||
|
||||
- **bincraft:** the mechanism — registry ingestion, resolution hook, isolated patched-binary build, caching/upload, local-repo prepend, logging, metadata recording.
|
||||
A new `patches` argument on `build_binary_package()`.
|
||||
- **build-cran-binaries (this repo):** the `local/patches/` registry and diffs, passing `patches = "local/patches"` through `build-one.R` / `build-all.R`, and documentation.
|
||||
Loading…
Reference in a new issue