feat: patch registry + wiring for per-package patching (#103)
Some checks failed
ci/crow/cron/process-updates/15 Pipeline failed
ci/crow/cron/process-updates/14 Pipeline failed
ci/crow/cron/process-updates/16 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/5 Pipeline failed

## Summary

Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`).
Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading.

## What's included

- `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs.
- `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace).
- `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes).
- `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`.
- `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`.

## ⚠️ Merge ordering (blocker)

This PR adds a `patches = ...` argument to `build_binary_package()` calls.
The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until:

1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and
2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`.

The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then.

Reviewed-on: #103
This commit is contained in:
Patrick Schratz 2026-06-30 08:50:19 +00:00 committed by Patrick Schratz
commit 55a18fd87d
16 changed files with 1791 additions and 10 deletions

View file

@ -62,7 +62,7 @@ steps:
GIT_USER: pat-s GIT_USER: pat-s
R_VERSION: 4.5.3 R_VERSION: 4.5.3
commands: commands:
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3", dependencies = TRUE)' - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1", dependencies = TRUE)'
- /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")'
- /opt/R/$R_VERSION/bin/R -q -e 'bincraft::process_unarchived_pkgs(Sys.getenv("CODENAME"), Sys.getenv("ARCH"), s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), workers = 2L)' - /opt/R/$R_VERSION/bin/R -q -e 'bincraft::process_unarchived_pkgs(Sys.getenv("CODENAME"), Sys.getenv("ARCH"), s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), workers = 2L)'
backend_options: backend_options:

View file

@ -75,7 +75,7 @@ steps:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
# Pin the same bincraft version the build steps use, so the precomputed # Pin the same bincraft version the build steps use, so the precomputed
# snapshot and the per-agent library stay consistent across the pipeline. # snapshot and the per-agent library stay consistent across the pipeline.
- /opt/R/$R_VERSION/bin/R -q -e 'pak::sysreqs_db_update(); pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3"); pak::pak(c("RPostgres", "s3fs", "data.table", "future", "jsonlite")); packageVersion("bincraft")' - /opt/R/$R_VERSION/bin/R -q -e 'pak::sysreqs_db_update(); pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1"); pak::pak(c("RPostgres", "s3fs", "data.table", "future", "jsonlite")); packageVersion("bincraft")'
- /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))" - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))"
backend_options: backend_options:
docker: docker:

View file

@ -121,7 +121,7 @@ steps:
# to a zero-length value and breaks every metadata query and the sysdeps # to a zero-length value and breaks every metadata query and the sysdeps
# install). Pin bincraft here, exactly like the R-minor pass below. # install). Pin bincraft here, exactly like the R-minor pass below.
- rm -rf /mnt/cache/R-pkgs/00LOCK-* - rm -rf /mnt/cache/R-pkgs/00LOCK-*
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")' - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")'
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 - $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1
- | - |
@ -133,7 +133,7 @@ steps:
echo "=== R-minor-sensitive pass under R $RV ===" echo "=== R-minor-sensitive pass under R $RV ==="
LIB="/mnt/cache/R-pkgs-$RMINOR" LIB="/mnt/cache/R-pkgs-$RMINOR"
mkdir -p "$LIB" mkdir -p "$LIB"
R_LIBS_USER="$LIB" "$(dirname "$RBIN")/R" -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")' || true R_LIBS_USER="$LIB" "$(dirname "$RBIN")/R" -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")' || true
R_LIBS_USER="$LIB" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$(dirname "$RBIN")/Rscript" local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true R_LIBS_USER="$LIB" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$(dirname "$RBIN")/Rscript" local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true
done done
# archive missed packages; first arg is the codename (e.g. "alpine324"), # archive missed packages; first arg is the codename (e.g. "alpine324"),

View file

@ -192,13 +192,13 @@ steps:
commands: commands:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/bincraft /mnt/cache/R-pkgs/pkgcache /mnt/cache/pkgcache/R/pkgcache - rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/bincraft /mnt/cache/R-pkgs/pkgcache /mnt/cache/pkgcache/R/pkgcache
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")' - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")'
- /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")'
- mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
# rhel-10 ships xwfb-run (Xwayland) instead of xvfb-run; prefer it and start weston, else fall back to xvfb-run # rhel-10 ships xwfb-run (Xwayland) instead of xvfb-run; prefer it and start weston, else fall back to xvfb-run
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
# options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise
- $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '${OS}', process_updated = TRUE, process_new = ${PROCESS_NEW}, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" - $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '${OS}', process_updated = TRUE, process_new = ${PROCESS_NEW}, process_removed = TRUE, patches = 'local/patches', r_minor_detection = 'classifier',s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)"
- | - |
PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2)
for RBIN in /opt/R/[0-9]*/bin/R; do for RBIN in /opt/R/[0-9]*/bin/R; do
@ -208,8 +208,8 @@ steps:
echo "=== R-minor-sensitive update pass under R $RV ===" echo "=== R-minor-sensitive update pass under R $RV ==="
LIB="/mnt/cache/R-pkgs-$RMINOR" LIB="/mnt/cache/R-pkgs-$RMINOR"
mkdir -p "$LIB" mkdir -p "$LIB"
R_LIBS_USER="$LIB" "$(dirname "$RBIN")/R" -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")' || true R_LIBS_USER="$LIB" "$(dirname "$RBIN")/R" -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")' || true
R_LIBS_USER="$LIB" $XVFB $XVFB_ARGS -- "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '${OS}', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true R_LIBS_USER="$LIB" $XVFB $XVFB_ARGS -- "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '${OS}', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, patches = 'local/patches', r_minor_detection = 'classifier',r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true
done done
- /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "${OS_ID}", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "${OS_ID}", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))'
- | - |

View file

@ -129,12 +129,12 @@ steps:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
- rm -rf /mnt/cache/R-pkgs/00LOCK-* - rm -rf /mnt/cache/R-pkgs/00LOCK-*
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.3") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.3")' - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.3.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.3.1")'
- /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")'
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
- /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")'
- /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")' - /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")'
- $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); pkgs <- readLines('/tmp/rebuild_pkgs.txt'); if (length(pkgs) == 0) { cat('Nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s - %s\n', x, conditionMessage(e)))) }" 2>&1 - $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); pkgs <- readLines('/tmp/rebuild_pkgs.txt'); if (length(pkgs) == 0) { cat('Nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, patches = 'local/patches', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s - %s\n', x, conditionMessage(e)))) }" 2>&1
backend_options: backend_options:
docker: docker:
resources: resources:

3
.gitignore vendored
View file

@ -99,3 +99,6 @@ docs/
local/test.R local/test.R
.DS_Store .DS_Store
docs/ docs/
# Superpowers SDD scratch (briefs, reports, ledger)
.superpowers/

View file

@ -2,3 +2,6 @@
ignores: ignores:
- LICENSE.md - LICENSE.md
- docs/superpowers/** - docs/superpowers/**
# Internal design docs (specs/plans) are not user-facing reference material.
- specs/**
- plans/**

View file

@ -43,3 +43,9 @@ repos:
entry: YAML filenames must have .yaml extension. entry: YAML filenames must have .yaml extension.
language: fail language: fail
files: .yml$ files: .yml$
- id: validate-patches
name: validate patch registry
entry: Rscript local/validate-patches.R
language: system
files: ^local/(patches/|validate-patches\.R$)
pass_filenames: false

View file

@ -53,6 +53,28 @@ For every package+tag combination:
1. Archive old package versions and keep the latest one in the root 1. Archive old package versions and keep the latest one in the root
1. Delete local binaries after successful upload 1. Delete local binaries after successful upload
## Patching packages
Some CRAN packages fail to compile on specific platforms due to compiler- or OS-specific issues unrelated to the package itself.
The canonical example is `RcppParallel`, whose bundled TBB sources fail on musl (Alpine) and newer compiler/OS combinations.
Because such packages are often transitive dependencies of many others, a single failure cascades: all dependents fail even though nothing is wrong with the dependent itself.
To address this, frequently-failing packages can be "patched" before they are installed — whether as a direct build target or a transitive dependency pulled in by `pak`.
The patch registry lives in `local/patches/registry.json`.
Each entry specifies a package and the platforms/versions it applies to, along with either lightweight build-time overrides (environment variables, configure arguments, Makevars) or a source diff (for deeper fixes).
See `local/patches/README.md` for the complete schema.
Patching uses a two-tier approach:
1. **Lightweight overrides:** environment variables, configure arguments, or Makevars settings applied during build — typically version-independent and fast.
2. **Source diffs:** unified diff patches applied to the unpacked source before building — more powerful but version-pinned.
The system is implemented in `bincraft`: when a package needs patching, `bincraft` pre-builds it with the patch and serves the patched binary to `pak`, ensuring transitive dependents receive the fixed package.
This way, the fix cascades to all packages that depend on it.
For the design rationale and architecture, see `specs/2026-06-30-package-patching-design.md`.
## Build Environment ## Build Environment
Binaries are built on a mixed-architecture Kubernetes cluster using CI. Binaries are built on a mixed-architecture Kubernetes cluster using CI.

View file

@ -140,6 +140,7 @@ mapply(
metadata_db_sslmode = "require", metadata_db_sslmode = "require",
metadata_db_port = 15432, metadata_db_port = 15432,
archive = TRUE, archive = TRUE,
patches = "local/patches",
upload = TRUE, upload = TRUE,
store_build_metadata = TRUE store_build_metadata = TRUE
) )

View file

@ -103,6 +103,7 @@ for (ver in versions) {
force = TRUE, force = TRUE,
upload = TRUE, upload = TRUE,
archive = TRUE, archive = TRUE,
patches = "local/patches",
store_build_metadata = TRUE, store_build_metadata = TRUE,
s3_endpoint = s3$s3_endpoint, s3_endpoint = s3$s3_endpoint,
s3_region = s3$s3_region, s3_region = s3$s3_region,

43
local/patches/README.md Normal file
View file

@ -0,0 +1,43 @@
# Patch Registry
This directory contains the curated registry of per-package build-time patches consumed by bincraft's `patches` argument.
## Schema
The registry is defined in `registry.json` as an array of patch entries. Each entry specifies lightweight build-time overrides (environment variables, configure arguments, Makevars) and optionally a source diff to apply before building.
### Field semantics
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `package` | string | yes | CRAN package name. |
| `versions` | string | yes | `"*"` for any, a constraint such as `">=5.1.0"`, or an exact version `"5.1.11-2"`. Env-tier fixes are typically `"*"`; source diffs are normally exact or lower-bounded because a diff is pinned to the source it was generated against. |
| `platforms` | array of strings | yes | Matched against the running build's platform tokens — distro family (`alpine`, `ubuntu`, `redhat`), codename (`ubuntu-2604`, `alpine-324`), and arch (`amd64`, `arm64`). An entry matches if any listed token matches any build token. `["*"]` matches all platforms. |
| `env` | object | no | Environment variables exported only for this package's isolated build. |
| `configure_args` | array | no | Arguments passed as `--configure-args` to the isolated build. |
| `makevars` | object | no | Key/value pairs written into a package-local Makevars for the isolated build. |
| `patch` | string or null | no | Path (relative to `local/patches/`) to a unified diff applied to the unpacked CRAN source before building. |
| `reason` | string | yes | Human explanation, surfaced in logs and metadata. |
## Adding an entry
To add a new patch entry:
1. Add an object to the array in `registry.json` with the fields documented above.
Start with lightweight overrides (environment variables, configure arguments, Makevars) before resorting to source diffs.
2. If a source diff is needed, place it in `local/patches/<package>/<file>.patch` and reference its path in the `patch` field.
For example, a diff for `RcppParallel` would go in `local/patches/RcppParallel/fix.patch` and be referenced as `"patch": "RcppParallel/fix.patch"`.
3. The `reason` field should clearly explain why the patch is needed and what problem it solves.
## Validation
The registry is validated and applied by bincraft during the build process.
For manual validation, run the validator from the repo root:
```bash
Rscript local/validate-patches.R
```
This validates the schema, referenced patch-file existence, and checks for duplicate entries across platforms and versions.

View file

@ -0,0 +1,12 @@
[
{
"package": "RcppParallel",
"versions": "*",
"platforms": ["alpine", "ubuntu-2604"],
"env": { "RCPP_PARALLEL_USE_TBB": "0" },
"configure_args": [],
"makevars": {},
"patch": null,
"reason": "bundled Intel TBB fails to build on musl and on newer toolchains (e.g. g++ 15 on ubuntu-2604); disabling TBB falls back to TinyThread"
}
]

56
local/validate-patches.R Normal file
View file

@ -0,0 +1,56 @@
#!/usr/bin/env Rscript
# Validate local/patches/registry.json: schema, referenced patch files, and
# ambiguous overlaps. Exits 1 on any problem. Used by pre-commit and CI.
dir <- "local/patches"
registry_file <- file.path(dir, "registry.json")
if (!file.exists(registry_file)) {
cat("No registry.json found; nothing to validate.\n")
quit(status = 0L)
}
or_q <- function(x) if (is.null(x)) "?" else x
reg <- jsonlite::fromJSON(registry_file, simplifyVector = FALSE)
required <- c("package", "versions", "platforms", "reason")
errs <- character(0L)
for (i in seq_along(reg)) {
e <- reg[[i]]
missing <- setdiff(required, names(e))
if (length(missing) > 0L) {
errs <- c(errs, sprintf(
"entry %d (%s): missing %s", i,
if (is.null(e$package)) "?" else e$package, toString(missing)
))
}
if (!is.null(e$patch)) {
p <- file.path(dir, e$patch)
if (!file.exists(p)) {
errs <- c(errs, sprintf("entry %d (%s): patch file '%s' missing",
i, e$package, p))
}
}
}
# Ambiguous overlap: two entries for the same package with identical platforms
# and versions.
keys <- vapply(reg, function(e) {
sprintf(
"%s|%s|%s",
or_q(e$package),
paste(sort(as.character(unlist(e$platforms))), collapse = ","),
or_q(e$versions)
)
}, character(1L))
dups <- keys[duplicated(keys)]
if (length(dups) > 0L) {
errs <- c(errs, sprintf("ambiguous duplicate entries: %s", toString(unique(dups))))
}
if (length(errs) > 0L) {
cat("Patch registry validation FAILED:\n")
cat(paste0(" - ", errs, "\n"))
quit(status = 1L)
}
cat(sprintf("Patch registry OK (%d %s).\n", length(reg), if (length(reg) == 1L) "entry" else "entries"))

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,157 @@
# Design: Dynamic per-package patching during binary builds
Date: 2026-06-30
Status: Approved (pending spec review)
## Problem
Some CRAN packages fail to compile on specific build platforms due to compiler- or OS-specific issues that have nothing to do with the package being built.
The canonical example is `RcppParallel`: its bundled Intel TBB sources fail to build on musl (Alpine) and on newer OS/compiler combinations.
Observed failure on `ubuntu-2604` ("resolute") with `g++ 15.2.0`:
```
../build/common.inc:74: *** "" is not supported. Add build/.inc file with os-specific settings . Stop.
make: *** [Makevars:163: tbb] Error 2
ERROR: compilation failed for package 'RcppParallel'
```
Because `RcppParallel` is a dependency of many packages, a single such failure cascades: every dependent package (e.g. `rts2`) also fails, even though nothing is wrong with the dependent itself.
Today there is no way to intervene.
A package can only be **excluded** (`local/excluded-packages.json`), which is all-or-nothing and does not help dependents.
## Goal
Allow a curated set of packages to be "patched" — via lightweight build-time overrides or, when necessary, real source diffs — **before** they are installed, whether the package is a direct build target or a transitive dependency pulled in by `pak`.
## Key constraint that drives the design
When `RcppParallel` fails here, it is being installed as a **transitive dependency** by `pak`, inside `bincraft::build_binary_package()`.
`pak` downloads, configures, and compiles it in one subprocess; this repo never touches that source.
For a fix to reach a dependency-of-a-dependency, the fixed package must be visible to `pak` itself, where `pak`'s repositories/sources are configured — which is inside `bincraft`.
Decisions taken during brainstorming:
- **Mechanism lives in `bincraft`** (the engine), because only there can transitive deps be influenced.
- **Patch tiers: both, env-overrides first.** Support cheap per-package env vars / configure args / Makevars (version-independent) *and* true source diffs (version-pinned), preferring the lightweight override.
- **Registry data lives in this repo** (`build-cran-binaries`) and is passed into `bincraft`, keeping `bincraft` as pure mechanism and the frequently-changing policy data with operational config.
## Approaches considered
| Approach | How pak sees the fix | Verdict |
|---|---|---|
| A. Patched **source** repo — drop patched `.tar.gz` source into a local repo, prepend it | pak recompiles from your source | Simple, but env-tier overrides leak globally (one subprocess builds everything) and the dep recompiles on every dependent build |
| **B. Pre-built patched binary repo (chosen)** | pak installs a ready binary by repo priority | Per-package scoping is free; no recompile; the binary is a cacheable/uploadable artifact that fits the existing system |
| C. pkgdepends per-build hook | intercept each build | No clean per-package pre-compile hook exists; fragile |
Chosen: **B**.
## Architecture
### Registry (this repo)
```
local/patches/
registry.json # the manifest
RcppParallel/
fix.patch # optional source diff, referenced by an entry
```
`registry.json` is an array of entries:
```json
[
{
"package": "RcppParallel",
"versions": "*",
"platforms": ["alpine", "ubuntu-2604"],
"env": { "RCPP_PARALLEL_USE_TBB": "0" },
"configure_args": [],
"makevars": {},
"patch": null,
"reason": "bundled TBB fails to build on musl / newer compilers"
}
]
```
Field semantics:
- `package` (string, required): CRAN package name.
- `versions` (string, required): `"*"` for any, a constraint such as `">=5.1.0"`, or an exact version `"5.1.11-2"`.
Env-tier fixes are typically `"*"`; source diffs are normally exact or lower-bounded because a diff is pinned to the source it was generated against.
- `platforms` (array of strings, required): matched against the running build's platform tokens — distro family (`alpine`, `ubuntu`, `redhat`), codename (`ubuntu-2604`, `alpine-324`), and arch (`amd64`, `arm64`).
An entry matches if any listed token matches any build token.
`["*"]` matches all platforms.
- `env` (object, optional): environment variables exported only for this package's isolated build.
- `configure_args` (array, optional): passed as `--configure-args` to the isolated build.
- `makevars` (object, optional): key/value pairs written into a package-local Makevars for the isolated build.
- `patch` (string or null, optional): path (relative to `local/patches/`) to a unified diff applied to the unpacked CRAN source before building.
- `reason` (string, required): human explanation, surfaced in logs and metadata.
A fix is any combination of `env`, `configure_args`, `makevars`, and `patch`.
"Env-first" is an authoring guideline (prefer the lightweight override) and an ordering of effort, not a runtime branch — all present fields are applied together for the isolated build.
### Flow (inside bincraft, around existing pak resolution)
1. **Resolve** the dependency set (dry-run) to learn the concrete versions `pak` will install.
Reuse bincraft's existing resolution where possible (e.g. a `pkgdepends` proposal: `$resolve()` → inspect resolution → ... → `$solve()` / `$install()` after the local repo is prepended).
2. For each resolved package that matches a registry entry (name + `versions` + `platforms`): obtain a **patched binary** for the exact `version × platform × arch × R-minor`:
- **Cache hit** (local `/mnt/cache/patched-binaries/` or S3): fetch it into the local repo.
- **Cache miss**: download the CRAN **source** for that version, apply the source `patch` (if any) to the unpacked tree, build the binary in isolation with `env` / `configure_args` / `makevars` applied, then place the binary in the local repo and write it to the cache (and S3 if uploading is enabled).
3. **Prepend** the local binary repo (`file://…`) to `pak`'s repo list, and regenerate its `PACKAGES` index.
4. Run the **normal install**.
`pak` resolves the patched binary for the matched package — direct or transitive — because it wins on repo priority for an equal version, and installs it without recompiling.
### Caching (essential)
`RcppParallel` is a dependency of dozens of packages; without caching the fix would be rebuilt on every dependent build.
Patched binaries are keyed by:
```
<package>_<version>_<platform>_<arch>_<rminor>_<patchhash>
```
`patchhash` is a hash of the normalized registry entry plus the referenced diff file contents.
Editing a patch therefore changes the hash and auto-invalidates stale cached binaries.
- Local cache: `/mnt/cache/patched-binaries/`.
- Optional S3 cache for cross-build reuse: a dedicated `…/patched/` slot under the existing arch/codename structure, mirroring how normal binaries are stored.
### S3 upload
Patched binaries **are** uploaded to S3 (in addition to the local cache) so they are reused across CI jobs and machines, not just within one container.
They live in a separate `patched/` slot and are not published into the user-facing `src/contrib` index — they are an internal build accelerator, not a distributed artifact.
## Error handling
- **Source diff fails to apply** (CRAN moved past the pinned version): log a clear warning, skip that entry, and proceed.
The package builds unpatched (status quo) and may fail.
The skipped/failed-to-apply patch is surfaced in build metadata.
- **Pre-build of the patched binary fails**: log a warning, skip, proceed.
- **No version or platform match**: skip silently (the entry simply does not apply to this build).
- **Overlapping entries for one package**: the most specific entry wins (a concrete `platforms`/`versions` beats `"*"`).
Genuine ambiguity (two equally specific, conflicting entries) is a validation error reported before the build.
## Observability
- One log line per applied patch, e.g.: `Applying patch to RcppParallel 5.1.11-2 [env: RCPP_PARALLEL_USE_TBB=0]: bundled TBB fails on musl / newer compilers`
- The set of applied patches (package, version, `patchhash`) is recorded in the Postgres build-metadata row for the build, so it is queryable later.
## Testing
- **Unit (registry):** parsing and matching — version constraints, platform token matching, precedence/specificity, and detection of ambiguous overlaps.
- **Unit (cache key):** `patchhash` changes when the entry or diff changes; is stable otherwise.
- **Integration:** `RcppParallel` on `resolute` (and/or Alpine) fails to build without a registry entry and succeeds with one; a dependent package such as `rts2` succeeds once the dependency is patched.
- **Failure path:** an entry pinned to an old version against a newer CRAN release → graceful skip with a warning, build continues.
## Out of scope
- Shipping a default registry inside `bincraft` (registry is repo-local for now; a baseline-in-engine + repo-override model can come later if needed).
- Publishing patched binaries into the public `src/contrib` index.
- Automatic detection of which packages need patches — entries are curated by hand.
## Split of work
- **bincraft:** the mechanism — registry ingestion, resolution hook, isolated patched-binary build, caching/upload, local-repo prepend, logging, metadata recording.
A new `patches` argument on `build_binary_package()`.
- **build-cran-binaries (this repo):** the `local/patches/` registry and diffs, passing `patches = "local/patches"` through `build-one.R` / `build-all.R`, and documentation.