From 02e39665239ebb2df54cad7fcf6ace12f15df3c9 Mon Sep 17 00:00:00 2001 From: pat-s Date: Wed, 12 Aug 2026 08:17:49 +0000 Subject: [PATCH 01/31] docs(rebuild): add design for sharding and resuming the weekly rebuild --- .../2026-08-12-shard-weekly-rebuild-design.md | 167 ++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 specs/2026-08-12-shard-weekly-rebuild-design.md diff --git a/specs/2026-08-12-shard-weekly-rebuild-design.md b/specs/2026-08-12-shard-weekly-rebuild-design.md new file mode 100644 index 0000000..f4ccbaa --- /dev/null +++ b/specs/2026-08-12-shard-weekly-rebuild-design.md @@ -0,0 +1,167 @@ +# Design: Sharding and resuming the weekly rebuild + +Date: 2026-08-12 +Status: Approved (pending spec review) + +## Problem + +`weekly-rebuild-missing` runs one job per `-` and walks that slot's rebuild list serially in a single `R -q -e` invocation (`.crow/weekly-rebuild-missing.yaml:165`). +Until 2026-08-09 that was cheap, because every source fallback was skipped as "already built" and the list was effectively empty. +Since bincraft #105/#106/#107 and build-cran-binaries #159 the gate works, and the lists are now large. + +Share of records whose object is byte-identical to CRAN's source, measured against `cran.r-project.org` MD5s on 2026-08-12: + +| slot | records | source-served | share | +| ------------------ | ------: | ------------: | -----------: | +| `amd64/resolute` | 24 212 | 15 023 | 62.1% | +| `arm64/resolute` | 24 291 | 13 670 | 56.3% | +| `arm64/alpine324` | 24 328 | 9 514 | 39.2% | +| `amd64/alpine324` | 24 343 | 8 917 | 36.7% | +| `arm64/rhel10` | 24 695 | 5 384 | 21.9% | +| `amd64/rhel10` | 24 881 | 4 712 | 19.2% | +| 12 remaining slots | ~24 700 | 850 to 2 130 | 3.5% to 8.7% | + +A single serial job cannot absorb that. +Pipeline 10910 (`weekly_rebuild_missing:alpine-324-amd64`) started on 2026-08-09, ran for roughly two days, reached `[8692/23885] cholera`, and was killed there. + +Two distinct failures follow from that shape. + +**No parallelism.** The work is embarrassingly parallel across packages, but one job does all of it. + +**No resumability, and no clean stopping point.** The loop has no terminating condition other than exhausting the list, so the only way to stop it is a kill. +A restarted run re-reads the same list and walks it from the first entry. +It skips completed packages via `check_s3_root_package()`, but that costs a CRAN version resolution and an S3 `HEAD` per package, thousands of times, before it reaches new work. +Worse, a kill is not a pipeline failure: the `Purge CDN cache` step is guarded by `when: status: [success, failure]` (`.crow/weekly-rebuild-missing.yaml:206-207`), and on 10910 it produced no output at all. +So the ~4 600 binaries that run did publish stayed hidden behind stale edge copies. + +## Goal + +Turn each slot's rebuild into bounded, parallel, restartable units, without introducing state that can disagree with the bucket. + +## Design + +### 1. Shard the matrix three ways + +Each of the 18 `OS`/`ARCH` rows in `.crow/weekly-rebuild-missing.yaml` gains `SPLIT_INTO: 3` and `SPLIT_INDEX: 1|2|3`, giving 54 rows. +This mirrors `.crow/build-all-versions.yaml:57-98`, which already shards its matrix four ways per arch. + +Routing needs no change. +The cron filter `cron: weekly-rebuild-missing-${OS}-${ARCH}` and the manual `evaluate: weekly_rebuild_missing == "${OS}-${ARCH}"` both match all three shards of a slot. +Placement stays on the `rpkgs-${ARCH}` group label, so shards queue against available capacity rather than oversubscribing it. + +### 2. Extract the loop into `local/rebuild-missing.R` + +The build is currently a single ~1 500-character `R -q -e` argument. +Shard arithmetic and resume logic do not belong in a YAML string, and none of it is testable there. +The loop moves to `local/rebuild-missing.R`, invoked as `Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX`, mirroring `local/build-all.R`. +Its body is unchanged in substance: read `/tmp/rebuild_pkgs.txt`, subtract `local/excluded-packages.json`, loop with `tryCatch` around `bincraft::build_binary_package()`. + +The slice is **interleaved**, not contiguous: + +```r +# the list is alphabetical and build cost clusters by name (Rcpp*, Bioc*, +# rstan*), so contiguous thirds would be badly unbalanced +mine <- pkgs[seq(split_index, length(pkgs), by = split_into)] +``` + +`local/build-all.R:64` uses contiguous chunks via `cut()`. +That is fine there because its list is every CRAN package and version, so the chunks average out. +Here the list is a filtered backlog in which expensive families sit adjacent, so interleaving is the better default. +Interleaving also makes each shard's `[i/n]` progress representative of the slot as a whole. + +### 3. Resume by re-deriving state from the bucket + +Before the loop, the shard performs one `s3fs::s3_dir_info()` on `devxy-rpkgs-binaries///latest/src/contrib` and reads the `etag` column. +It fetches CRAN's `PACKAGES` once for the latest version and published `MD5sum` of every package. +A package is still outstanding if and only if the object at `_.tar.gz` has an ETag equal to CRAN's `MD5sum` for that version, which is the definition `check_s3_root_package()` already applies one package at a time. + +```r +# one paginated listing instead of ~2900 sequential HEAD requests per shard +info <- s3fs::s3_dir_info(slot_dir) +etag <- setNames(gsub('^"|"$', "", info$etag), basename(info$uri)) + +key <- sprintf("%s_%s.tar.gz", mine, cran_version[mine]) +# keep a package when no object exists yet, or when the object is still +# byte-identical to CRAN's source; drop it once a real binary is published +mine <- mine[is.na(etag[key]) | etag[key] == cran_md5[key]] +``` + +This is the whole resume mechanism. +There is no progress file, no volume, and no database cursor. +A restarted shard recomputes ground truth and continues where it stopped, and it is correct even when a sibling shard, a `process-updates` cron, or a manual `just rebuild` completed something in the meantime. + +Three properties make this the right source of truth: + +- **It is what the build itself checks.** Any other store can disagree with the bucket; this one cannot. +- **It is agent-independent.** `.crow/weekly-rebuild-missing.yaml` mounts no `volumes:`, unlike `.crow/build-all-versions.yaml:132-133`, so `/mnt/cache` is per-job and cannot carry progress anyway. +- **It costs one listing.** `cranlike`'s `s3` fork already does exactly this call against this bucket at ~24 000 objects, so the approach is proven at the required scale. + +It must read ETags rather than the slot index's `Built` field, which is how `local/packages-to-build.R:104-130` answers the same question. +Under this design the index is not rewritten until the dependent re-index pipeline runs (section 5), so mid-run it cannot reflect the current run's progress. + +Packages that genuinely fail to build re-publish their CRAN source, so they stay outstanding and would be retried on every restart. +That is already handled upstream: `bincraft::filter_packages_with_errors()` (`R/build_binaries.R:1018`, `:1143`) drops anything with `error_occurred = TRUE`, and `store_build_metadata = TRUE` is passed on every call. +No additional poison-pill filter is needed here. + +Only the flat `src/contrib` path is considered. +The rebuild call passes no `is_r_minor_sensitive`, so it defaults to `FALSE` and only ever targets the flat path; the resume filter matches that scope deliberately. + +### 4. Give each shard a wall-clock budget + +`local/rebuild-missing.R` takes a budget, defaulting to 20 hours, and breaks out of the loop once it is exceeded: + +```r +# exit cleanly rather than being killed, so the dependent re-index still runs +if (difftime(Sys.time(), started, units = "hours") > budget_hours) { + cat(sprintf("Budget of %sh reached after %d/%d packages; stopping cleanly\n", budget_hours, i, n)) + break +} +``` + +It exits 0 and reports how much of the slice it covered. +Every run then has a terminating condition, the re-index and purge always fire, and the remainder is picked up by the next run with no bookkeeping, because section 3 recomputes the outstanding set from scratch. + +### 5. Move the re-index and purge into `.crow/weekly-rebuild-reindex.yaml` + +Three shards per slot means three concurrent `upload_package_index()` calls on the same S3 prefix. +`cranlike::update_PACKAGES()` lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work. +The re-index steps (`.crow/weekly-rebuild-missing.yaml:171-176`) and the purge step (`:187-207`) therefore leave that file entirely. + +The new file carries: + +```yaml +depends_on: + - weekly-rebuild-missing +runs_on: [success, failure] +``` + +`runs_on: [success, failure]` validates as a workflow-level key under `crow lint`, so a failing shard no longer withholds the re-index. +The file uses the same 18-row matrix and the same `when:` gating as `weekly-rebuild-missing`, so it only re-indexes slots that actually ran. +Each row re-indexes the flat slot and every per-minor slot. +`scripts/purge_cdn_zone.sh` runs once on a single row, because all hostnames share pull zone `3857050` and 18 identical zone purges would be waste. + +## Failure behaviour + +| case | today | after | +| -------------------------- | ----------------------------------- | ----------------------------------------------------- | +| one package errors | `tryCatch` logs, loop continues | unchanged | +| a shard fails outright | purge runs, re-index does not | re-index and purge run via `runs_on` | +| a shard exceeds its budget | cannot happen, runs until killed | exits 0, re-index and purge run | +| a shard is killed | nothing runs | still nothing; trigger the re-index pipeline alone | +| a shard restarts | re-walks the list, HEAD per package | one listing, resumes at the first outstanding package | + +The known cost of `depends_on` being file-level rather than row-level: on the weekly cron no slot is re-indexed until the slowest of all 54 jobs finishes. +The 20-hour budget bounds that at roughly one day. + +## Out of scope + +- `build-all-versions` still cannot rebuild source fallbacks, because `local/build-all.R:113-122` drops every version with any `single_builds` row for the platform and arch, which is precisely the source-fallback set. That is a separate change. +- Bunny Perma-Cache eviction. `scripts/purge_cdn_zone.sh` purges the regular edge cache only; see the note in `CLAUDE.md` and issue history. +- The audit that produces the rebuild list is unchanged. + +## Verification + +- `crow lint .crow/` passes for both pipeline files. +- `local/rebuild-missing.R` gets unit coverage in `local/tests/` for the two pure pieces: the interleaved slice (disjoint, covering, deterministic) and the outstanding-set filter (source-served ETag kept, binary ETag dropped, absent object kept). +- A single-slot manual run of `alpine-324-amd64` shard 1 confirms the listing shortcut against the live bucket, and that the reported outstanding count is close to the 8 917 measured above divided by three. +- Restarting that shard mid-run confirms it resumes rather than replaying, by comparing the outstanding count it reports on the second start. From c072b780b24184331114c62bf041389f4bf47a35 Mon Sep 17 00:00:00 2001 From: pat-s Date: Wed, 12 Aug 2026 08:27:45 +0000 Subject: [PATCH 02/31] feat(rebuild): shard the weekly rebuild and make each shard resumable Split every OS/arch row of weekly-rebuild-missing into three shards and move the build loop out of the inline R one-liner into local/rebuild-missing.R. A shard re-derives its outstanding set from the bucket on every start: an object whose ETag equals CRAN's published MD5sum is still a source fallback and needs building. That is bincraft's check_s3_root_package() evaluated in bulk, so a restart resumes rather than replaying thousands of per-package HEADs, and it stays correct when a sibling shard or a process-updates run finishes work in the meantime. No progress file, no volume, no database cursor. Give each shard a 20h wall-clock budget so it exits cleanly instead of having to be killed. A kill matches neither success nor failure, which is how pipeline 10910 skipped its CDN purge and left ~4600 rebuilt binaries behind stale edge copies. Move re-indexing and the purge into weekly-rebuild-reindex.yaml, which depends on the rebuild and runs on success or failure. Three shards writing one slot's PACKAGES concurrently would race: update_PACKAGES lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work. --- .crow/weekly-rebuild-missing.yaml | 306 +++++++++++++++++++++++++---- .crow/weekly-rebuild-reindex.yaml | 193 ++++++++++++++++++ local/rebuild-missing-helpers.R | 87 ++++++++ local/rebuild-missing.R | 194 ++++++++++++++++++ local/tests/test-rebuild-missing.R | 94 +++++++++ 5 files changed, 838 insertions(+), 36 deletions(-) create mode 100644 .crow/weekly-rebuild-reindex.yaml create mode 100644 local/rebuild-missing-helpers.R create mode 100644 local/rebuild-missing.R create mode 100644 local/tests/test-rebuild-missing.R diff --git a/.crow/weekly-rebuild-missing.yaml b/.crow/weekly-rebuild-missing.yaml index c86fe5a..5afb3c3 100644 --- a/.crow/weekly-rebuild-missing.yaml +++ b/.crow/weekly-rebuild-missing.yaml @@ -1,12 +1,21 @@ # Consolidated weekly-rebuild-missing pipeline (all platforms, both arches). -# One matrix row per OS/arch replaces the former per-platform files. +# Three matrix rows per OS/arch, one per shard of that slot's rebuild list. # Routing is preserved 1:1: # - cron: each existing `weekly-rebuild-missing--` cron fires only -# its matching matrix row (via the per-row `cron:` name filter). +# its matching matrix rows (via the per-row `cron:` name filter), +# which is now all three shards of that slot. # - manual: `weekly_rebuild_missing` dropdown, default "all" (matches the # previous bare manual trigger that ran every os/arch); pick a # single - to run just one. # Arch placement is handled by the group label (rpkgs-amd64, rpkgs-arm64). +# +# The shard picks up its own slice and re-derives what is still outstanding +# from the bucket, so a restart resumes rather than replaying; see +# local/rebuild-missing.R. +# +# Re-indexing and the CDN purge deliberately do NOT live here. Three shards +# writing one slot's PACKAGES concurrently would race, so they moved to +# .crow/weekly-rebuild-reindex.yaml, which depends on this pipeline. variables: # Gates this pipeline. A manual pipeline creation instantiates every file in # .crow/, and a declared default is applied even when the run never passed @@ -53,74 +62,326 @@ matrix: ARCH: amd64 R_VERSION: 4.5.3 IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-322 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-322 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-322 ARCH: arm64 R_VERSION: 4.5.3 IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-322 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-322 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-323 ARCH: amd64 R_VERSION: 4.5.3 IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-323 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-323 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-323 ARCH: arm64 R_VERSION: 4.5.3 IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-323 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-323 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-324 ARCH: amd64 R_VERSION: 4.5.3 IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-324 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-324 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-324 ARCH: arm64 R_VERSION: 4.5.3 IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-324 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-324 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-8 ARCH: amd64 R_VERSION: 4.4.3 IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-8 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-8 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-8 ARCH: arm64 R_VERSION: 4.4.3 IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-8 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-8 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-9 ARCH: amd64 R_VERSION: 4.4.3 IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-9 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-9 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-9 ARCH: arm64 R_VERSION: 4.4.3 IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-9 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-9 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-10 ARCH: amd64 R_VERSION: 4.5.3 IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-10 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-10 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-10 ARCH: arm64 R_VERSION: 4.5.3 IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-10 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-10 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2204 ARCH: amd64 R_VERSION: 4.4.3 IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2204 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2204 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2204 ARCH: arm64 R_VERSION: 4.4.3 IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2204 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2204 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2404 ARCH: amd64 R_VERSION: 4.4.3 IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2404 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2404 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2404 ARCH: arm64 R_VERSION: 4.4.3 IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2404 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2404 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2604 ARCH: amd64 R_VERSION: 4.4.3 IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2604 ARCH: arm64 R_VERSION: 4.4.3 IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 3 steps: - name: 'Rebuild missing binaries' @@ -153,6 +414,12 @@ steps: PLATFORM: ${OS} ARCH: ${ARCH} NCPUS: 2 + SPLIT_INTO: ${SPLIT_INTO} + SPLIT_INDEX: ${SPLIT_INDEX} + # Wall clock after which the shard stops cleanly instead of having to be + # killed. A kill matches neither `success` nor `failure`, so it would skip + # the dependent re-index and leave rebuilt binaries behind a stale edge. + REBUILD_BUDGET_HOURS: 20 commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/uvr/cache /mnt/cache/uvr/packages /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages @@ -162,18 +429,7 @@ steps: - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh httr2 - /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")' - - $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); pkgs <- readLines('/tmp/rebuild_pkgs.txt'); if (length(pkgs) == 0) { cat('Nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, patches = 'local/patches', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s - %s\n', x, conditionMessage(e)))) }" 2>&1 - # A rebuild replaces objects in place, so the slot's index still advertises - # the old MD5 and, for anything that had been served from source, no Built - # stamp. Re-index here rather than waiting for the next process-updates - # run, or the rebuilt binaries stay invisible to clients until then. - # The codename is detected from the image's /etc/os-release. - - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' - - | - for RBIN in /opt/R/[0-9]*/bin/R; do - RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) - /opt/R/$R_VERSION/bin/R -q -e "library(bincraft); upload_package_index(r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true - done + - $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX $REBUILD_BUDGET_HOURS 2>&1 backend_options: docker: resources: @@ -183,25 +439,3 @@ steps: limits: memory: 18Gi cpu: 3000m - - - name: Purge CDN cache - image: reg.devxy.io/docker.io/library/alpine:3.24 - environment: - OTEL_R_TRACES_EXPORTER: none - OTEL_R_LOGS_EXPORTER: none - OTEL_R_METRICS_EXPORTER: none - BUNNYNET_API_KEY: - from_secret: BUNNYNET_API_KEY - REPO_RO_TOKEN: - from_secret: REPO_RO_TOKEN - # All hostnames on the zone share this id, so one purge covers - # cran.devxy.io, cran.allianceswisspass.devxy.io and cran.rpkgs.com. - BUNNY_PULLZONE: '3857050' - commands: - - apk add --no-cache -q bash curl git - - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" "$BUNNY_PULLZONE" - # A rebuild that died part-way still replaced objects, and those are exactly - # the ones a stale edge would keep hiding, so purge either way. - when: - - status: [success, failure] diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml new file mode 100644 index 0000000..224596b --- /dev/null +++ b/.crow/weekly-rebuild-reindex.yaml @@ -0,0 +1,193 @@ +# Re-index and purge after weekly-rebuild-missing. +# +# weekly-rebuild-missing runs three shards per slot. Each of them replaces +# objects in place, so the slot's index still advertises the old MD5 and, for +# anything that had been served from source, no Built stamp. Re-indexing from +# inside a shard would mean three concurrent `upload_package_index()` calls on +# one prefix: `cranlike::update_PACKAGES()` lists the live bucket, so an early +# lister that uploads last publishes an index missing its siblings' work. +# +# So it happens exactly once per slot, here, after every shard has finished. +# `runs_on: [success, failure]` keeps that true when a shard fails; only an +# explicit cancel skips it, and this pipeline can then be triggered on its own. + +variables: + # Mirrors the gate on weekly-rebuild-missing so a manual run re-indexes + # exactly the slots it rebuilt. A manual pipeline creation instantiates every + # file in .crow/, so the default must match no matrix row. + weekly_rebuild_missing: + description: "Manual run target: a specific -, 'all' for every os/arch, or 'none' to run nothing." + options: + - none + - all + - alpine-322-amd64 + - alpine-322-arm64 + - alpine-323-amd64 + - alpine-323-arm64 + - alpine-324-amd64 + - alpine-324-arm64 + - redhat-8-amd64 + - redhat-8-arm64 + - redhat-9-amd64 + - redhat-9-arm64 + - redhat-10-amd64 + - redhat-10-arm64 + - ubuntu-2204-amd64 + - ubuntu-2204-arm64 + - ubuntu-2404-amd64 + - ubuntu-2404-arm64 + - ubuntu-2604-amd64 + - ubuntu-2604-arm64 + default: none + +when: + - event: cron + cron: weekly-rebuild-missing-${OS}-${ARCH} + - event: manual + evaluate: 'weekly_rebuild_missing == "all" || weekly_rebuild_missing == "${OS}-${ARCH}"' + +depends_on: + - weekly-rebuild-missing + +runs_on: [success, failure] + +skip_clone: true + +labels: + group: rpkgs-${ARCH} + +matrix: + include: + - OS: alpine-322 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + - OS: alpine-322 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + - OS: alpine-323 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + - OS: alpine-323 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + - OS: alpine-324 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + - OS: alpine-324 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + - OS: redhat-8 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:8 + - OS: redhat-8 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:8 + - OS: redhat-9 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:9 + - OS: redhat-9 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:9 + - OS: redhat-10 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: redhat:10 + - OS: redhat-10 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: redhat:10 + - OS: ubuntu-2204 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + - OS: ubuntu-2204 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + - OS: ubuntu-2404 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + - OS: ubuntu-2404 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + +steps: + - name: 'Re-index the slot' + image: reg.devxy.io/rpkgs/build-env-${IMG} + pull: true + environment: + OTEL_R_TRACES_EXPORTER: none + OTEL_R_LOGS_EXPORTER: none + OTEL_R_METRICS_EXPORTER: none + RED_HAT_DEV_PW: + from_secret: RED_HAT_DEV_PW + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + GIT_USER: pat-s + R_LIBS_USER: /mnt/cache/R-pkgs + R_VERSION: ${R_VERSION} + PLATFORM: ${OS} + ARCH: ${ARCH} + commands: + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - mkdir -p /mnt/cache/R-pkgs + - rm -rf /mnt/cache/R-pkgs/00LOCK-* + - /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R + # The codename is detected from the image's /etc/os-release. + - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/[0-9]*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e "library(bincraft); upload_package_index(r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done + + - name: Purge CDN cache + image: reg.devxy.io/docker.io/library/alpine:3.24 + environment: + OTEL_R_TRACES_EXPORTER: none + OTEL_R_LOGS_EXPORTER: none + OTEL_R_METRICS_EXPORTER: none + BUNNYNET_API_KEY: + from_secret: BUNNYNET_API_KEY + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + # All hostnames on the zone share this id, so one purge covers + # cran.devxy.io, cran.allianceswisspass.devxy.io and cran.rpkgs.com. + BUNNY_PULLZONE: '3857050' + commands: + - apk add --no-cache -q bash curl git + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" "$BUNNY_PULLZONE" + # Runs on every row rather than on one designated slot: a cron fires only + # its own slot's row, so gating on a named slot would leave every other + # slot unpurged. A manual "all" run therefore purges the zone 18 times, + # which is a cheap API call and rare. + # + # Run it even when the re-index above failed: the objects were still + # replaced, and a stale edge is exactly what keeps them hidden. + when: + - status: [success, failure] diff --git a/local/rebuild-missing-helpers.R b/local/rebuild-missing-helpers.R new file mode 100644 index 0000000..16a588e --- /dev/null +++ b/local/rebuild-missing-helpers.R @@ -0,0 +1,87 @@ +# Pure helpers for local/rebuild-missing.R, kept separate so local/tests can +# source them without executing a rebuild. + +# Interleaved slice of the rebuild list. +# +# The list is alphabetical and build cost clusters by name (Rcpp*, Bioc*, +# rstan*), so contiguous thirds would be badly unbalanced. Interleaving also +# makes each shard's progress counter representative of the slot as a whole. +shard_slice <- function(pkgs, split_into, split_index) { + split_into <- as.integer(split_into) + split_index <- as.integer(split_index) + if (is.na(split_into) || is.na(split_index)) { + stop("shard_slice(): split_into and split_index must be integers") + } + if (split_into < 1L || split_index < 1L || split_index > split_into) { + stop(sprintf( + "shard_slice(): need 1 <= split_index <= split_into, got %s of %s", + split_index, + split_into + )) + } + # seq() errors on a descending range, which is what an empty list or a shard + # index past the end would produce. + if (length(pkgs) < split_index) { + return(pkgs[0L]) + } + pkgs[seq.int(split_index, length(pkgs), by = split_into)] +} + +# Packages that still need building, decided from the bucket rather than from +# remembered progress. +# +# This is bincraft's `check_s3_root_package()` evaluated in bulk: an object +# whose ETag equals CRAN's published MD5sum is byte-identical to CRAN's source, +# so the build that was supposed to replace it has not happened yet. +# +# `etag_by_file` named by `_.tar.gz`, values are unquoted ETags +# `cran_version` named by package +# `cran_md5` named by `_` +# +# Unknown always means "already a binary", never "rebuild it", so an unreadable +# CRAN index or a multipart ETag can never mass-schedule work. +outstanding_packages <- function(pkgs, etag_by_file, cran_version, cran_md5) { + if (length(pkgs) == 0L) { + return(pkgs) + } + + # An empty table indexes to zero length rather than to NA, which would + # recycle the whole result away and silently report "nothing to build". + lookup <- function(table, key) { + if (length(table) == 0L) { + return(rep(NA_character_, length(key))) + } + unname(as.character(table[key])) + } + + version <- lookup(cran_version, pkgs) + file <- sprintf("%s_%s.tar.gz", pkgs, version) + etag <- lookup(etag_by_file, file) + md5 <- lookup(cran_md5, paste(pkgs, version, sep = "_")) + + # No CRAN version means the package cannot be resolved to a tarball at all; + # leave it in and let bincraft report why. + unresolved <- is.na(version) + # No object at the key: never built, so it is outstanding by definition. + absent <- !unresolved & is.na(etag) + # A multipart upload carries a compound ETag rather than an MD5. + unknown <- !is.na(etag) & grepl("-", etag, fixed = TRUE) + + is_source <- !unresolved & + !is.na(etag) & + !unknown & + !is.na(md5) & + etag == md5 + + pkgs[unresolved | absent | is_source] +} + +parse_rebuild_args <- function(args) { + pos <- args[!startsWith(args, "--")] + budget <- as.numeric(pos[3L]) + list( + split_into = as.integer(pos[1L]), + split_index = as.integer(pos[2L]), + budget_hours = if (is.na(budget)) 20 else budget + ) +} diff --git a/local/rebuild-missing.R b/local/rebuild-missing.R new file mode 100644 index 0000000..369024e --- /dev/null +++ b/local/rebuild-missing.R @@ -0,0 +1,194 @@ +### Rebuild one shard of a slot's missing-binary list. +# +# Usage: Rscript local/rebuild-missing.R [budget_hours] +# +# The list itself comes from local/fetch-rebuild-packages-from-issue.R, which +# writes $REBUILD_PKG_LIST (default /tmp/rebuild_pkgs.txt). +# +# Two properties matter here and are the reason this is a script rather than an +# `R -q -e` argument in the pipeline: +# +# * it is restartable. The outstanding set is re-derived from the bucket on +# every start, so a shard that died resumes where it stopped without any +# progress file, and without replaying thousands of per-package HEADs. +# * it terminates. A wall-clock budget stops the loop cleanly instead of the +# run having to be killed, which is what previously skipped the re-index and +# CDN purge and left rebuilt binaries hidden behind stale edge copies. + +options(error = function() { + cat("ERROR:", geterrmessage(), "\n", file = stdout()) + traceback(2) + q(status = 1) +}) + +library(bincraft, quietly = TRUE) + +source(file.path("local", "rebuild-missing-helpers.R")) + +args <- parse_rebuild_args(commandArgs(trailingOnly = TRUE)) +if (is.na(args$split_into) || is.na(args$split_index)) { + stop("usage: rebuild-missing.R [budget_hours]") +} + +list_file <- Sys.getenv("REBUILD_PKG_LIST", "/tmp/rebuild_pkgs.txt") +pkgs <- if (file.exists(list_file)) readLines(list_file) else character(0) +pkgs <- pkgs[nzchar(pkgs)] +if (length(pkgs) == 0L) { + cat("Nothing to rebuild\n") + q("no") +} + +excluded <- jsonlite::fromJSON("local/excluded-packages.json")[["package"]] +pkgs <- setdiff(pkgs, excluded) + +mine <- shard_slice(pkgs, args$split_into, args$split_index) +cat(sprintf( + "Shard %s/%s: %s of %s listed packages\n", + args$split_index, + args$split_into, + length(mine), + length(pkgs) +)) + +### Resume: ask the bucket what is still outstanding + +codename <- bincraft::set_codename(NULL) +local_machine <- Sys.info()[["machine"]] +arch <- if (grepl("arm64|aarch64", local_machine)) "arm64" else "amd64" +slot_dir <- sprintf( + "devxy-rpkgs-binaries/%s/%s/latest/src/contrib", + arch, + codename +) + +s3fs::s3_file_system( + aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), + aws_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), + endpoint = "https://s3.eu-central-003.backblazeb2.com", + region_name = "eu-central-003", + refresh = TRUE +) + +# One paginated listing instead of a HEAD per package. Not recursed: the +# rebuild passes no `is_r_minor_sensitive`, so it only ever targets the flat +# path, and the resume filter matches that scope deliberately. +info <- tryCatch(s3fs::s3_dir_info(slot_dir), error = function(e) NULL) +etag_by_file <- if (is.null(info) || nrow(info) == 0L) { + cat(sprintf( + "WARNING: could not list %s; building the whole shard\n", + slot_dir + )) + stats::setNames(character(), character()) +} else { + stats::setNames( + gsub('^"|"$', "", as.character(info$etag)), + basename(as.character(info$uri)) + ) +} + +cran <- tryCatch( + { + con <- gzcon(url( + "https://cloud.r-project.org/src/contrib/PACKAGES.gz", + open = "rb" + )) + on.exit(close(con), add = TRUE) + read.dcf(con, fields = c("Package", "Version", "MD5sum")) + }, + error = function(e) { + cat(sprintf( + "WARNING: could not read CRAN's index (%s)\n", + conditionMessage(e) + )) + NULL + } +) +cran_version <- stats::setNames(character(), character()) +cran_md5 <- stats::setNames(character(), character()) +if (!is.null(cran)) { + cran_version <- stats::setNames( + as.character(cran[, "Version"]), + as.character(cran[, "Package"]) + ) + keep <- !is.na(cran[, "MD5sum"]) + cran_md5 <- stats::setNames( + as.character(cran[keep, "MD5sum"]), + paste(cran[keep, "Package"], cran[keep, "Version"], sep = "_") + ) +} + +before <- length(mine) +mine <- outstanding_packages(mine, etag_by_file, cran_version, cran_md5) +cat(sprintf( + "Resume: %s of %s already carry a binary; %s outstanding\n", + before - length(mine), + before, + length(mine) +)) + +if (length(mine) == 0L) { + cat("Nothing outstanding for this shard\n") + q("no") +} + +### Build + +options( + crayon.enabled = TRUE, + Ncpus = as.integer(Sys.getenv("NCPUS", "2")), + future.globals.onReference = NULL +) + +started <- Sys.time() +n <- length(mine) +completed <- 0L +for (i in seq_along(mine)) { + elapsed <- as.numeric(difftime(Sys.time(), started, units = "hours")) + if (elapsed > args$budget_hours) { + cat(sprintf( + "Budget of %sh reached after %d/%d packages; stopping cleanly. The next run resumes from the bucket.\n", + args$budget_hours, + completed, + n + )) + break + } + + x <- mine[i] + cat(sprintf("[%d/%d] %s\n", i, n, x)) + tryCatch( + bincraft::build_binary_package( + x, + tag_limit = 1L, + patches = "local/patches", + s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", + s3_region = "eu-central-003", + s3_bucket = "devxy-rpkgs-binaries", + s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), + s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), + metadata_db_host = "r-binaries.devxy.io", + metadata_db_name = "build_metadata", + metadata_db_table = "single_builds", + metadata_db_user = "rpkgs", + metadata_db_password = Sys.getenv("PGPASS"), + metadata_db_sslmode = "require", + metadata_db_port = 15432, + archive = TRUE, + upload = TRUE, + store_build_metadata = TRUE + ), + error = function(e) { + cat(sprintf("ERROR building %s - %s\n", x, conditionMessage(e))) + } + ) + completed <- completed + 1L +} + +cat(sprintf( + "Shard %s/%s finished: %d/%d packages processed in %.1fh\n", + args$split_index, + args$split_into, + completed, + n, + as.numeric(difftime(Sys.time(), started, units = "hours")) +)) diff --git a/local/tests/test-rebuild-missing.R b/local/tests/test-rebuild-missing.R new file mode 100644 index 0000000..a5e70b9 --- /dev/null +++ b/local/tests/test-rebuild-missing.R @@ -0,0 +1,94 @@ +source(file.path("..", "rebuild-missing-helpers.R")) + +test_that("shard_slice partitions the list without gaps or overlap", { + pkgs <- letters[1:10] + parts <- lapply(1:3, function(i) shard_slice(pkgs, 3, i)) + + expect_identical(parts[[1]], c("a", "d", "g", "j")) + expect_identical(parts[[2]], c("b", "e", "h")) + expect_identical(parts[[3]], c("c", "f", "i")) + + expect_identical(sort(unlist(parts)), sort(pkgs)) + expect_identical(anyDuplicated(unlist(parts)), 0L) +}) + +test_that("shard_slice is deterministic and survives short lists", { + expect_identical( + shard_slice(letters[1:10], 3, 2), + shard_slice(letters[1:10], 3, 2) + ) + expect_identical(shard_slice(character(0), 3, 1), character(0)) + # more shards than packages: the tail shards get nothing rather than erroring + expect_identical(shard_slice(c("a"), 3, 1), "a") + expect_identical(shard_slice(c("a"), 3, 2), character(0)) +}) + +test_that("shard_slice rejects an out-of-range index", { + expect_error(shard_slice(letters, 3, 4), "split_index") + expect_error(shard_slice(letters, 3, 0), "split_index") +}) + +test_that("outstanding_packages keeps source fallbacks and drops real binaries", { + cran_version <- c(httr = "1.4.8", R6 = "2.6.1", curl = "7.1.0") + cran_md5 <- c( + httr_1.4.8 = "8756015b94a9cff6f410ca4de8557f12", + R6_2.6.1 = "f01b1787f12797c29194d63c9afd5d70", + curl_7.1.0 = "8af2ccbf5d85dc18866f45f1f26f348d" + ) + etag <- c( + # byte-identical to CRAN: the build never happened + "httr_1.4.8.tar.gz" = "8756015b94a9cff6f410ca4de8557f12", + # a real binary was published + "R6_2.6.1.tar.gz" = "9d6087ee9adda3f0a3b8067cfc652c05" + # curl has no object at all + ) + + out <- outstanding_packages( + c("httr", "R6", "curl"), + etag, + cran_version, + cran_md5 + ) + expect_identical(out, c("httr", "curl")) +}) + +test_that("outstanding_packages treats unknowns as already built", { + cran_version <- c(a = "1.0", b = "1.0") + cran_md5 <- c(a_1.0 = "aaaa") + + # a multipart ETag carries no MD5, and `b` is missing from CRAN's index: + # neither may schedule a rebuild + etag <- c("a_1.0.tar.gz" = "abc-3", "b_1.0.tar.gz" = "bbbb") + + expect_identical( + outstanding_packages(c("a", "b"), etag, cran_version, cran_md5), + character(0) + ) +}) + +test_that("outstanding_packages keeps a package CRAN has no version for", { + out <- outstanding_packages( + "ghost", + c(), + c(other = "1.0"), + c(other_1.0 = "aaaa") + ) + expect_identical(out, "ghost") +}) + +test_that("outstanding_packages handles an empty list", { + expect_identical( + outstanding_packages(character(0), c(), c(), c()), + character(0) + ) +}) + +test_that("parse_rebuild_args defaults the budget", { + a <- parse_rebuild_args(c("3", "2")) + expect_identical(a$split_into, 3L) + expect_identical(a$split_index, 2L) + expect_identical(a$budget_hours, 20) + + b <- parse_rebuild_args(c("3", "2", "1.5")) + expect_identical(b$budget_hours, 1.5) +}) From 4b7dc28cc83784fd350731b87f0d515df7c21299 Mon Sep 17 00:00:00 2001 From: pat-s Date: Wed, 12 Aug 2026 08:30:29 +0000 Subject: [PATCH 03/31] feat(rebuild): shard the weekly rebuild and make each shard resumable (#163) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Problem `weekly-rebuild-missing` runs one job per `-` and walks that slot's list serially in a single `R -q -e` argument. That was cheap while every source fallback was skipped as "already built". Since bincraft #105/#106/#107 and #159 the gate works, and the lists are large: 8 917 source-served records on `amd64/alpine324`, 15 023 on `amd64/resolute`. Pipeline 10910 (`weekly_rebuild_missing:alpine-324-amd64`) ran for two days, reached `[8692/23885] cholera`, and was killed there. Two failures follow from that shape: - **No parallelism.** The work is embarrassingly parallel across packages; one job does all of it. - **No resumability and no clean stopping point.** The loop ends only by exhausting the list, so the only way to stop it is a kill. A restart re-walks from the first entry, paying a CRAN version resolution and an S3 `HEAD` per package before reaching new work. And a kill matches neither `success` nor `failure`, so the `Purge CDN cache` step never ran: the ~4 600 binaries 10910 did publish stayed hidden behind stale edge copies. ## What this changes **Three shards per slot.** Each of the 18 `OS`/`ARCH` rows gains `SPLIT_INTO`/`SPLIT_INDEX`, mirroring `build-all-versions.yaml`. Cron and manual routing are unchanged: both filters already match on `${OS}-${ARCH}`, so they now match all three shards of a slot. **`local/rebuild-missing.R`** replaces the ~1 500-character inline one-liner. The slice is interleaved rather than contiguous, because the list is alphabetical and cost clusters by name (`Rcpp*`, `Bioc*`, `rstan*`). **Resume by re-deriving state from the bucket.** One `s3_dir_info()` listing gives ETags for the slot; a package is outstanding iff its object's ETag equals CRAN's published `MD5sum`, i.e. it is still byte-identical to CRAN's source. That is `check_s3_root_package()` evaluated in bulk. No progress file, no volume, no DB cursor, and correct when a sibling shard or a `process-updates` run completes something concurrently. It reads ETags rather than the index's `Built` field the way `packages-to-build.R` does, because the index is no longer rewritten until the dependent pipeline runs and so cannot reflect the current run's progress. Unknown always means "already a binary", never "rebuild it": a multipart ETag, an unreadable CRAN index or an empty listing can never mass-schedule work. **A 20 h wall-clock budget** per shard. It exits 0, so the re-index and purge always fire and the remainder is picked up next run with no bookkeeping. **`.crow/weekly-rebuild-reindex.yaml`** takes over re-indexing and the purge, with `depends_on: [weekly-rebuild-missing]` and `runs_on: [success, failure]`. Three shards writing one slot's `PACKAGES` concurrently would race: `update_PACKAGES()` lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work. ## Verification `crow lint .crow/` passes on all 11 pipelines. `prek run` passes. 19 assertions in `local/tests/test-rebuild-missing.R`, 0 failures, covering the partition (disjoint, covering, deterministic, short lists, out-of-range index) and the outstanding filter (source ETag kept, binary ETag dropped, absent object kept, multipart and missing-from-CRAN treated as built). One of those tests caught a real bug before it shipped: an empty ETag table indexed to zero length rather than to `NA`, which recycled the result away and reported "nothing to build" — the dangerous direction. Fixed with an explicit `lookup()`. The filter run against the live `amd64/alpine324` index, using its `MD5sum` column as the ETag (established to match the objects): ``` index packages: 24343 outstanding (filter): 8950 no Built stamp: 8917 filter vs no-Built agreement: 8917 of 8917 outstanding but stamped Built: 33 (version drift vs CRAN) shard sizes: 2984/2983/2983 (sum 8950, unique 8950) ``` It reproduces the source-served set exactly. The extra 33 are packages whose slot version differs from CRAN's current one, so no object exists at the CRAN version key: correctly outstanding. ## Notes for review - The 20 h budget is a chosen default, exposed as `REBUILD_BUDGET_HOURS` in the pipeline. - `depends_on` is file-level, not row-level, so on a full cron run no slot is re-indexed until the slowest of all 54 jobs finishes. The budget bounds that at roughly a day. - An explicit cancel still skips the re-index. Recovery is to trigger `weekly-rebuild-reindex` on its own. - The purge runs on every re-index row rather than one designated slot: a cron fires only its own slot's row, so gating on a named slot would leave every other slot unpurged. - Out of scope: `build-all-versions` still cannot rebuild source fallbacks, because `local/build-all.R:113-122` drops every version with any `single_builds` row, which is precisely the source-fallback set. Design: `specs/2026-08-12-shard-weekly-rebuild-design.md` Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/163 --- .crow/weekly-rebuild-missing.yaml | 306 +++++++++++++++--- .crow/weekly-rebuild-reindex.yaml | 193 +++++++++++ local/rebuild-missing-helpers.R | 87 +++++ local/rebuild-missing.R | 194 +++++++++++ local/tests/test-rebuild-missing.R | 94 ++++++ .../2026-08-12-shard-weekly-rebuild-design.md | 167 ++++++++++ 6 files changed, 1005 insertions(+), 36 deletions(-) create mode 100644 .crow/weekly-rebuild-reindex.yaml create mode 100644 local/rebuild-missing-helpers.R create mode 100644 local/rebuild-missing.R create mode 100644 local/tests/test-rebuild-missing.R create mode 100644 specs/2026-08-12-shard-weekly-rebuild-design.md diff --git a/.crow/weekly-rebuild-missing.yaml b/.crow/weekly-rebuild-missing.yaml index c86fe5a..5afb3c3 100644 --- a/.crow/weekly-rebuild-missing.yaml +++ b/.crow/weekly-rebuild-missing.yaml @@ -1,12 +1,21 @@ # Consolidated weekly-rebuild-missing pipeline (all platforms, both arches). -# One matrix row per OS/arch replaces the former per-platform files. +# Three matrix rows per OS/arch, one per shard of that slot's rebuild list. # Routing is preserved 1:1: # - cron: each existing `weekly-rebuild-missing--` cron fires only -# its matching matrix row (via the per-row `cron:` name filter). +# its matching matrix rows (via the per-row `cron:` name filter), +# which is now all three shards of that slot. # - manual: `weekly_rebuild_missing` dropdown, default "all" (matches the # previous bare manual trigger that ran every os/arch); pick a # single - to run just one. # Arch placement is handled by the group label (rpkgs-amd64, rpkgs-arm64). +# +# The shard picks up its own slice and re-derives what is still outstanding +# from the bucket, so a restart resumes rather than replaying; see +# local/rebuild-missing.R. +# +# Re-indexing and the CDN purge deliberately do NOT live here. Three shards +# writing one slot's PACKAGES concurrently would race, so they moved to +# .crow/weekly-rebuild-reindex.yaml, which depends on this pipeline. variables: # Gates this pipeline. A manual pipeline creation instantiates every file in # .crow/, and a declared default is applied even when the run never passed @@ -53,74 +62,326 @@ matrix: ARCH: amd64 R_VERSION: 4.5.3 IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-322 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-322 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-322 ARCH: arm64 R_VERSION: 4.5.3 IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-322 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-322 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-323 ARCH: amd64 R_VERSION: 4.5.3 IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-323 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-323 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-323 ARCH: arm64 R_VERSION: 4.5.3 IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-323 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-323 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-324 ARCH: amd64 R_VERSION: 4.5.3 IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-324 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-324 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: alpine-324 ARCH: arm64 R_VERSION: 4.5.3 IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: alpine-324 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: alpine-324 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-8 ARCH: amd64 R_VERSION: 4.4.3 IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-8 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-8 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-8 ARCH: arm64 R_VERSION: 4.4.3 IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-8 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-8 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:8 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-9 ARCH: amd64 R_VERSION: 4.4.3 IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-9 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-9 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-9 ARCH: arm64 R_VERSION: 4.4.3 IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-9 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-9 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:9 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-10 ARCH: amd64 R_VERSION: 4.5.3 IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-10 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-10 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: redhat-10 ARCH: arm64 R_VERSION: 4.5.3 IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: redhat-10 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: redhat-10 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: redhat:10 + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2204 ARCH: amd64 R_VERSION: 4.4.3 IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2204 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2204 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2204 ARCH: arm64 R_VERSION: 4.4.3 IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2204 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2204 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2404 ARCH: amd64 R_VERSION: 4.4.3 IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2404 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2404 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2404 ARCH: arm64 R_VERSION: 4.4.3 IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2404 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2404 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2604 ARCH: amd64 R_VERSION: 4.4.3 IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 3 - OS: ubuntu-2604 ARCH: arm64 R_VERSION: 4.4.3 IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 3 steps: - name: 'Rebuild missing binaries' @@ -153,6 +414,12 @@ steps: PLATFORM: ${OS} ARCH: ${ARCH} NCPUS: 2 + SPLIT_INTO: ${SPLIT_INTO} + SPLIT_INDEX: ${SPLIT_INDEX} + # Wall clock after which the shard stops cleanly instead of having to be + # killed. A kill matches neither `success` nor `failure`, so it would skip + # the dependent re-index and leave rebuilt binaries behind a stale edge. + REBUILD_BUDGET_HOURS: 20 commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/uvr/cache /mnt/cache/uvr/packages /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages @@ -162,18 +429,7 @@ steps: - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh httr2 - /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")' - - $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); pkgs <- readLines('/tmp/rebuild_pkgs.txt'); if (length(pkgs) == 0) { cat('Nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, patches = 'local/patches', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s - %s\n', x, conditionMessage(e)))) }" 2>&1 - # A rebuild replaces objects in place, so the slot's index still advertises - # the old MD5 and, for anything that had been served from source, no Built - # stamp. Re-index here rather than waiting for the next process-updates - # run, or the rebuilt binaries stay invisible to clients until then. - # The codename is detected from the image's /etc/os-release. - - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' - - | - for RBIN in /opt/R/[0-9]*/bin/R; do - RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) - /opt/R/$R_VERSION/bin/R -q -e "library(bincraft); upload_package_index(r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true - done + - $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX $REBUILD_BUDGET_HOURS 2>&1 backend_options: docker: resources: @@ -183,25 +439,3 @@ steps: limits: memory: 18Gi cpu: 3000m - - - name: Purge CDN cache - image: reg.devxy.io/docker.io/library/alpine:3.24 - environment: - OTEL_R_TRACES_EXPORTER: none - OTEL_R_LOGS_EXPORTER: none - OTEL_R_METRICS_EXPORTER: none - BUNNYNET_API_KEY: - from_secret: BUNNYNET_API_KEY - REPO_RO_TOKEN: - from_secret: REPO_RO_TOKEN - # All hostnames on the zone share this id, so one purge covers - # cran.devxy.io, cran.allianceswisspass.devxy.io and cran.rpkgs.com. - BUNNY_PULLZONE: '3857050' - commands: - - apk add --no-cache -q bash curl git - - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" "$BUNNY_PULLZONE" - # A rebuild that died part-way still replaced objects, and those are exactly - # the ones a stale edge would keep hiding, so purge either way. - when: - - status: [success, failure] diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml new file mode 100644 index 0000000..224596b --- /dev/null +++ b/.crow/weekly-rebuild-reindex.yaml @@ -0,0 +1,193 @@ +# Re-index and purge after weekly-rebuild-missing. +# +# weekly-rebuild-missing runs three shards per slot. Each of them replaces +# objects in place, so the slot's index still advertises the old MD5 and, for +# anything that had been served from source, no Built stamp. Re-indexing from +# inside a shard would mean three concurrent `upload_package_index()` calls on +# one prefix: `cranlike::update_PACKAGES()` lists the live bucket, so an early +# lister that uploads last publishes an index missing its siblings' work. +# +# So it happens exactly once per slot, here, after every shard has finished. +# `runs_on: [success, failure]` keeps that true when a shard fails; only an +# explicit cancel skips it, and this pipeline can then be triggered on its own. + +variables: + # Mirrors the gate on weekly-rebuild-missing so a manual run re-indexes + # exactly the slots it rebuilt. A manual pipeline creation instantiates every + # file in .crow/, so the default must match no matrix row. + weekly_rebuild_missing: + description: "Manual run target: a specific -, 'all' for every os/arch, or 'none' to run nothing." + options: + - none + - all + - alpine-322-amd64 + - alpine-322-arm64 + - alpine-323-amd64 + - alpine-323-arm64 + - alpine-324-amd64 + - alpine-324-arm64 + - redhat-8-amd64 + - redhat-8-arm64 + - redhat-9-amd64 + - redhat-9-arm64 + - redhat-10-amd64 + - redhat-10-arm64 + - ubuntu-2204-amd64 + - ubuntu-2204-arm64 + - ubuntu-2404-amd64 + - ubuntu-2404-arm64 + - ubuntu-2604-amd64 + - ubuntu-2604-arm64 + default: none + +when: + - event: cron + cron: weekly-rebuild-missing-${OS}-${ARCH} + - event: manual + evaluate: 'weekly_rebuild_missing == "all" || weekly_rebuild_missing == "${OS}-${ARCH}"' + +depends_on: + - weekly-rebuild-missing + +runs_on: [success, failure] + +skip_clone: true + +labels: + group: rpkgs-${ARCH} + +matrix: + include: + - OS: alpine-322 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + - OS: alpine-322 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + - OS: alpine-323 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + - OS: alpine-323 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + - OS: alpine-324 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + - OS: alpine-324 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + - OS: redhat-8 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:8 + - OS: redhat-8 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:8 + - OS: redhat-9 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:9 + - OS: redhat-9 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:9 + - OS: redhat-10 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: redhat:10 + - OS: redhat-10 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: redhat:10 + - OS: ubuntu-2204 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + - OS: ubuntu-2204 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + - OS: ubuntu-2404 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + - OS: ubuntu-2404 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + +steps: + - name: 'Re-index the slot' + image: reg.devxy.io/rpkgs/build-env-${IMG} + pull: true + environment: + OTEL_R_TRACES_EXPORTER: none + OTEL_R_LOGS_EXPORTER: none + OTEL_R_METRICS_EXPORTER: none + RED_HAT_DEV_PW: + from_secret: RED_HAT_DEV_PW + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + GIT_USER: pat-s + R_LIBS_USER: /mnt/cache/R-pkgs + R_VERSION: ${R_VERSION} + PLATFORM: ${OS} + ARCH: ${ARCH} + commands: + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - mkdir -p /mnt/cache/R-pkgs + - rm -rf /mnt/cache/R-pkgs/00LOCK-* + - /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R + # The codename is detected from the image's /etc/os-release. + - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/[0-9]*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e "library(bincraft); upload_package_index(r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done + + - name: Purge CDN cache + image: reg.devxy.io/docker.io/library/alpine:3.24 + environment: + OTEL_R_TRACES_EXPORTER: none + OTEL_R_LOGS_EXPORTER: none + OTEL_R_METRICS_EXPORTER: none + BUNNYNET_API_KEY: + from_secret: BUNNYNET_API_KEY + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + # All hostnames on the zone share this id, so one purge covers + # cran.devxy.io, cran.allianceswisspass.devxy.io and cran.rpkgs.com. + BUNNY_PULLZONE: '3857050' + commands: + - apk add --no-cache -q bash curl git + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" "$BUNNY_PULLZONE" + # Runs on every row rather than on one designated slot: a cron fires only + # its own slot's row, so gating on a named slot would leave every other + # slot unpurged. A manual "all" run therefore purges the zone 18 times, + # which is a cheap API call and rare. + # + # Run it even when the re-index above failed: the objects were still + # replaced, and a stale edge is exactly what keeps them hidden. + when: + - status: [success, failure] diff --git a/local/rebuild-missing-helpers.R b/local/rebuild-missing-helpers.R new file mode 100644 index 0000000..16a588e --- /dev/null +++ b/local/rebuild-missing-helpers.R @@ -0,0 +1,87 @@ +# Pure helpers for local/rebuild-missing.R, kept separate so local/tests can +# source them without executing a rebuild. + +# Interleaved slice of the rebuild list. +# +# The list is alphabetical and build cost clusters by name (Rcpp*, Bioc*, +# rstan*), so contiguous thirds would be badly unbalanced. Interleaving also +# makes each shard's progress counter representative of the slot as a whole. +shard_slice <- function(pkgs, split_into, split_index) { + split_into <- as.integer(split_into) + split_index <- as.integer(split_index) + if (is.na(split_into) || is.na(split_index)) { + stop("shard_slice(): split_into and split_index must be integers") + } + if (split_into < 1L || split_index < 1L || split_index > split_into) { + stop(sprintf( + "shard_slice(): need 1 <= split_index <= split_into, got %s of %s", + split_index, + split_into + )) + } + # seq() errors on a descending range, which is what an empty list or a shard + # index past the end would produce. + if (length(pkgs) < split_index) { + return(pkgs[0L]) + } + pkgs[seq.int(split_index, length(pkgs), by = split_into)] +} + +# Packages that still need building, decided from the bucket rather than from +# remembered progress. +# +# This is bincraft's `check_s3_root_package()` evaluated in bulk: an object +# whose ETag equals CRAN's published MD5sum is byte-identical to CRAN's source, +# so the build that was supposed to replace it has not happened yet. +# +# `etag_by_file` named by `_.tar.gz`, values are unquoted ETags +# `cran_version` named by package +# `cran_md5` named by `_` +# +# Unknown always means "already a binary", never "rebuild it", so an unreadable +# CRAN index or a multipart ETag can never mass-schedule work. +outstanding_packages <- function(pkgs, etag_by_file, cran_version, cran_md5) { + if (length(pkgs) == 0L) { + return(pkgs) + } + + # An empty table indexes to zero length rather than to NA, which would + # recycle the whole result away and silently report "nothing to build". + lookup <- function(table, key) { + if (length(table) == 0L) { + return(rep(NA_character_, length(key))) + } + unname(as.character(table[key])) + } + + version <- lookup(cran_version, pkgs) + file <- sprintf("%s_%s.tar.gz", pkgs, version) + etag <- lookup(etag_by_file, file) + md5 <- lookup(cran_md5, paste(pkgs, version, sep = "_")) + + # No CRAN version means the package cannot be resolved to a tarball at all; + # leave it in and let bincraft report why. + unresolved <- is.na(version) + # No object at the key: never built, so it is outstanding by definition. + absent <- !unresolved & is.na(etag) + # A multipart upload carries a compound ETag rather than an MD5. + unknown <- !is.na(etag) & grepl("-", etag, fixed = TRUE) + + is_source <- !unresolved & + !is.na(etag) & + !unknown & + !is.na(md5) & + etag == md5 + + pkgs[unresolved | absent | is_source] +} + +parse_rebuild_args <- function(args) { + pos <- args[!startsWith(args, "--")] + budget <- as.numeric(pos[3L]) + list( + split_into = as.integer(pos[1L]), + split_index = as.integer(pos[2L]), + budget_hours = if (is.na(budget)) 20 else budget + ) +} diff --git a/local/rebuild-missing.R b/local/rebuild-missing.R new file mode 100644 index 0000000..369024e --- /dev/null +++ b/local/rebuild-missing.R @@ -0,0 +1,194 @@ +### Rebuild one shard of a slot's missing-binary list. +# +# Usage: Rscript local/rebuild-missing.R [budget_hours] +# +# The list itself comes from local/fetch-rebuild-packages-from-issue.R, which +# writes $REBUILD_PKG_LIST (default /tmp/rebuild_pkgs.txt). +# +# Two properties matter here and are the reason this is a script rather than an +# `R -q -e` argument in the pipeline: +# +# * it is restartable. The outstanding set is re-derived from the bucket on +# every start, so a shard that died resumes where it stopped without any +# progress file, and without replaying thousands of per-package HEADs. +# * it terminates. A wall-clock budget stops the loop cleanly instead of the +# run having to be killed, which is what previously skipped the re-index and +# CDN purge and left rebuilt binaries hidden behind stale edge copies. + +options(error = function() { + cat("ERROR:", geterrmessage(), "\n", file = stdout()) + traceback(2) + q(status = 1) +}) + +library(bincraft, quietly = TRUE) + +source(file.path("local", "rebuild-missing-helpers.R")) + +args <- parse_rebuild_args(commandArgs(trailingOnly = TRUE)) +if (is.na(args$split_into) || is.na(args$split_index)) { + stop("usage: rebuild-missing.R [budget_hours]") +} + +list_file <- Sys.getenv("REBUILD_PKG_LIST", "/tmp/rebuild_pkgs.txt") +pkgs <- if (file.exists(list_file)) readLines(list_file) else character(0) +pkgs <- pkgs[nzchar(pkgs)] +if (length(pkgs) == 0L) { + cat("Nothing to rebuild\n") + q("no") +} + +excluded <- jsonlite::fromJSON("local/excluded-packages.json")[["package"]] +pkgs <- setdiff(pkgs, excluded) + +mine <- shard_slice(pkgs, args$split_into, args$split_index) +cat(sprintf( + "Shard %s/%s: %s of %s listed packages\n", + args$split_index, + args$split_into, + length(mine), + length(pkgs) +)) + +### Resume: ask the bucket what is still outstanding + +codename <- bincraft::set_codename(NULL) +local_machine <- Sys.info()[["machine"]] +arch <- if (grepl("arm64|aarch64", local_machine)) "arm64" else "amd64" +slot_dir <- sprintf( + "devxy-rpkgs-binaries/%s/%s/latest/src/contrib", + arch, + codename +) + +s3fs::s3_file_system( + aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), + aws_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), + endpoint = "https://s3.eu-central-003.backblazeb2.com", + region_name = "eu-central-003", + refresh = TRUE +) + +# One paginated listing instead of a HEAD per package. Not recursed: the +# rebuild passes no `is_r_minor_sensitive`, so it only ever targets the flat +# path, and the resume filter matches that scope deliberately. +info <- tryCatch(s3fs::s3_dir_info(slot_dir), error = function(e) NULL) +etag_by_file <- if (is.null(info) || nrow(info) == 0L) { + cat(sprintf( + "WARNING: could not list %s; building the whole shard\n", + slot_dir + )) + stats::setNames(character(), character()) +} else { + stats::setNames( + gsub('^"|"$', "", as.character(info$etag)), + basename(as.character(info$uri)) + ) +} + +cran <- tryCatch( + { + con <- gzcon(url( + "https://cloud.r-project.org/src/contrib/PACKAGES.gz", + open = "rb" + )) + on.exit(close(con), add = TRUE) + read.dcf(con, fields = c("Package", "Version", "MD5sum")) + }, + error = function(e) { + cat(sprintf( + "WARNING: could not read CRAN's index (%s)\n", + conditionMessage(e) + )) + NULL + } +) +cran_version <- stats::setNames(character(), character()) +cran_md5 <- stats::setNames(character(), character()) +if (!is.null(cran)) { + cran_version <- stats::setNames( + as.character(cran[, "Version"]), + as.character(cran[, "Package"]) + ) + keep <- !is.na(cran[, "MD5sum"]) + cran_md5 <- stats::setNames( + as.character(cran[keep, "MD5sum"]), + paste(cran[keep, "Package"], cran[keep, "Version"], sep = "_") + ) +} + +before <- length(mine) +mine <- outstanding_packages(mine, etag_by_file, cran_version, cran_md5) +cat(sprintf( + "Resume: %s of %s already carry a binary; %s outstanding\n", + before - length(mine), + before, + length(mine) +)) + +if (length(mine) == 0L) { + cat("Nothing outstanding for this shard\n") + q("no") +} + +### Build + +options( + crayon.enabled = TRUE, + Ncpus = as.integer(Sys.getenv("NCPUS", "2")), + future.globals.onReference = NULL +) + +started <- Sys.time() +n <- length(mine) +completed <- 0L +for (i in seq_along(mine)) { + elapsed <- as.numeric(difftime(Sys.time(), started, units = "hours")) + if (elapsed > args$budget_hours) { + cat(sprintf( + "Budget of %sh reached after %d/%d packages; stopping cleanly. The next run resumes from the bucket.\n", + args$budget_hours, + completed, + n + )) + break + } + + x <- mine[i] + cat(sprintf("[%d/%d] %s\n", i, n, x)) + tryCatch( + bincraft::build_binary_package( + x, + tag_limit = 1L, + patches = "local/patches", + s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", + s3_region = "eu-central-003", + s3_bucket = "devxy-rpkgs-binaries", + s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), + s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), + metadata_db_host = "r-binaries.devxy.io", + metadata_db_name = "build_metadata", + metadata_db_table = "single_builds", + metadata_db_user = "rpkgs", + metadata_db_password = Sys.getenv("PGPASS"), + metadata_db_sslmode = "require", + metadata_db_port = 15432, + archive = TRUE, + upload = TRUE, + store_build_metadata = TRUE + ), + error = function(e) { + cat(sprintf("ERROR building %s - %s\n", x, conditionMessage(e))) + } + ) + completed <- completed + 1L +} + +cat(sprintf( + "Shard %s/%s finished: %d/%d packages processed in %.1fh\n", + args$split_index, + args$split_into, + completed, + n, + as.numeric(difftime(Sys.time(), started, units = "hours")) +)) diff --git a/local/tests/test-rebuild-missing.R b/local/tests/test-rebuild-missing.R new file mode 100644 index 0000000..a5e70b9 --- /dev/null +++ b/local/tests/test-rebuild-missing.R @@ -0,0 +1,94 @@ +source(file.path("..", "rebuild-missing-helpers.R")) + +test_that("shard_slice partitions the list without gaps or overlap", { + pkgs <- letters[1:10] + parts <- lapply(1:3, function(i) shard_slice(pkgs, 3, i)) + + expect_identical(parts[[1]], c("a", "d", "g", "j")) + expect_identical(parts[[2]], c("b", "e", "h")) + expect_identical(parts[[3]], c("c", "f", "i")) + + expect_identical(sort(unlist(parts)), sort(pkgs)) + expect_identical(anyDuplicated(unlist(parts)), 0L) +}) + +test_that("shard_slice is deterministic and survives short lists", { + expect_identical( + shard_slice(letters[1:10], 3, 2), + shard_slice(letters[1:10], 3, 2) + ) + expect_identical(shard_slice(character(0), 3, 1), character(0)) + # more shards than packages: the tail shards get nothing rather than erroring + expect_identical(shard_slice(c("a"), 3, 1), "a") + expect_identical(shard_slice(c("a"), 3, 2), character(0)) +}) + +test_that("shard_slice rejects an out-of-range index", { + expect_error(shard_slice(letters, 3, 4), "split_index") + expect_error(shard_slice(letters, 3, 0), "split_index") +}) + +test_that("outstanding_packages keeps source fallbacks and drops real binaries", { + cran_version <- c(httr = "1.4.8", R6 = "2.6.1", curl = "7.1.0") + cran_md5 <- c( + httr_1.4.8 = "8756015b94a9cff6f410ca4de8557f12", + R6_2.6.1 = "f01b1787f12797c29194d63c9afd5d70", + curl_7.1.0 = "8af2ccbf5d85dc18866f45f1f26f348d" + ) + etag <- c( + # byte-identical to CRAN: the build never happened + "httr_1.4.8.tar.gz" = "8756015b94a9cff6f410ca4de8557f12", + # a real binary was published + "R6_2.6.1.tar.gz" = "9d6087ee9adda3f0a3b8067cfc652c05" + # curl has no object at all + ) + + out <- outstanding_packages( + c("httr", "R6", "curl"), + etag, + cran_version, + cran_md5 + ) + expect_identical(out, c("httr", "curl")) +}) + +test_that("outstanding_packages treats unknowns as already built", { + cran_version <- c(a = "1.0", b = "1.0") + cran_md5 <- c(a_1.0 = "aaaa") + + # a multipart ETag carries no MD5, and `b` is missing from CRAN's index: + # neither may schedule a rebuild + etag <- c("a_1.0.tar.gz" = "abc-3", "b_1.0.tar.gz" = "bbbb") + + expect_identical( + outstanding_packages(c("a", "b"), etag, cran_version, cran_md5), + character(0) + ) +}) + +test_that("outstanding_packages keeps a package CRAN has no version for", { + out <- outstanding_packages( + "ghost", + c(), + c(other = "1.0"), + c(other_1.0 = "aaaa") + ) + expect_identical(out, "ghost") +}) + +test_that("outstanding_packages handles an empty list", { + expect_identical( + outstanding_packages(character(0), c(), c(), c()), + character(0) + ) +}) + +test_that("parse_rebuild_args defaults the budget", { + a <- parse_rebuild_args(c("3", "2")) + expect_identical(a$split_into, 3L) + expect_identical(a$split_index, 2L) + expect_identical(a$budget_hours, 20) + + b <- parse_rebuild_args(c("3", "2", "1.5")) + expect_identical(b$budget_hours, 1.5) +}) diff --git a/specs/2026-08-12-shard-weekly-rebuild-design.md b/specs/2026-08-12-shard-weekly-rebuild-design.md new file mode 100644 index 0000000..f4ccbaa --- /dev/null +++ b/specs/2026-08-12-shard-weekly-rebuild-design.md @@ -0,0 +1,167 @@ +# Design: Sharding and resuming the weekly rebuild + +Date: 2026-08-12 +Status: Approved (pending spec review) + +## Problem + +`weekly-rebuild-missing` runs one job per `-` and walks that slot's rebuild list serially in a single `R -q -e` invocation (`.crow/weekly-rebuild-missing.yaml:165`). +Until 2026-08-09 that was cheap, because every source fallback was skipped as "already built" and the list was effectively empty. +Since bincraft #105/#106/#107 and build-cran-binaries #159 the gate works, and the lists are now large. + +Share of records whose object is byte-identical to CRAN's source, measured against `cran.r-project.org` MD5s on 2026-08-12: + +| slot | records | source-served | share | +| ------------------ | ------: | ------------: | -----------: | +| `amd64/resolute` | 24 212 | 15 023 | 62.1% | +| `arm64/resolute` | 24 291 | 13 670 | 56.3% | +| `arm64/alpine324` | 24 328 | 9 514 | 39.2% | +| `amd64/alpine324` | 24 343 | 8 917 | 36.7% | +| `arm64/rhel10` | 24 695 | 5 384 | 21.9% | +| `amd64/rhel10` | 24 881 | 4 712 | 19.2% | +| 12 remaining slots | ~24 700 | 850 to 2 130 | 3.5% to 8.7% | + +A single serial job cannot absorb that. +Pipeline 10910 (`weekly_rebuild_missing:alpine-324-amd64`) started on 2026-08-09, ran for roughly two days, reached `[8692/23885] cholera`, and was killed there. + +Two distinct failures follow from that shape. + +**No parallelism.** The work is embarrassingly parallel across packages, but one job does all of it. + +**No resumability, and no clean stopping point.** The loop has no terminating condition other than exhausting the list, so the only way to stop it is a kill. +A restarted run re-reads the same list and walks it from the first entry. +It skips completed packages via `check_s3_root_package()`, but that costs a CRAN version resolution and an S3 `HEAD` per package, thousands of times, before it reaches new work. +Worse, a kill is not a pipeline failure: the `Purge CDN cache` step is guarded by `when: status: [success, failure]` (`.crow/weekly-rebuild-missing.yaml:206-207`), and on 10910 it produced no output at all. +So the ~4 600 binaries that run did publish stayed hidden behind stale edge copies. + +## Goal + +Turn each slot's rebuild into bounded, parallel, restartable units, without introducing state that can disagree with the bucket. + +## Design + +### 1. Shard the matrix three ways + +Each of the 18 `OS`/`ARCH` rows in `.crow/weekly-rebuild-missing.yaml` gains `SPLIT_INTO: 3` and `SPLIT_INDEX: 1|2|3`, giving 54 rows. +This mirrors `.crow/build-all-versions.yaml:57-98`, which already shards its matrix four ways per arch. + +Routing needs no change. +The cron filter `cron: weekly-rebuild-missing-${OS}-${ARCH}` and the manual `evaluate: weekly_rebuild_missing == "${OS}-${ARCH}"` both match all three shards of a slot. +Placement stays on the `rpkgs-${ARCH}` group label, so shards queue against available capacity rather than oversubscribing it. + +### 2. Extract the loop into `local/rebuild-missing.R` + +The build is currently a single ~1 500-character `R -q -e` argument. +Shard arithmetic and resume logic do not belong in a YAML string, and none of it is testable there. +The loop moves to `local/rebuild-missing.R`, invoked as `Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX`, mirroring `local/build-all.R`. +Its body is unchanged in substance: read `/tmp/rebuild_pkgs.txt`, subtract `local/excluded-packages.json`, loop with `tryCatch` around `bincraft::build_binary_package()`. + +The slice is **interleaved**, not contiguous: + +```r +# the list is alphabetical and build cost clusters by name (Rcpp*, Bioc*, +# rstan*), so contiguous thirds would be badly unbalanced +mine <- pkgs[seq(split_index, length(pkgs), by = split_into)] +``` + +`local/build-all.R:64` uses contiguous chunks via `cut()`. +That is fine there because its list is every CRAN package and version, so the chunks average out. +Here the list is a filtered backlog in which expensive families sit adjacent, so interleaving is the better default. +Interleaving also makes each shard's `[i/n]` progress representative of the slot as a whole. + +### 3. Resume by re-deriving state from the bucket + +Before the loop, the shard performs one `s3fs::s3_dir_info()` on `devxy-rpkgs-binaries///latest/src/contrib` and reads the `etag` column. +It fetches CRAN's `PACKAGES` once for the latest version and published `MD5sum` of every package. +A package is still outstanding if and only if the object at `_.tar.gz` has an ETag equal to CRAN's `MD5sum` for that version, which is the definition `check_s3_root_package()` already applies one package at a time. + +```r +# one paginated listing instead of ~2900 sequential HEAD requests per shard +info <- s3fs::s3_dir_info(slot_dir) +etag <- setNames(gsub('^"|"$', "", info$etag), basename(info$uri)) + +key <- sprintf("%s_%s.tar.gz", mine, cran_version[mine]) +# keep a package when no object exists yet, or when the object is still +# byte-identical to CRAN's source; drop it once a real binary is published +mine <- mine[is.na(etag[key]) | etag[key] == cran_md5[key]] +``` + +This is the whole resume mechanism. +There is no progress file, no volume, and no database cursor. +A restarted shard recomputes ground truth and continues where it stopped, and it is correct even when a sibling shard, a `process-updates` cron, or a manual `just rebuild` completed something in the meantime. + +Three properties make this the right source of truth: + +- **It is what the build itself checks.** Any other store can disagree with the bucket; this one cannot. +- **It is agent-independent.** `.crow/weekly-rebuild-missing.yaml` mounts no `volumes:`, unlike `.crow/build-all-versions.yaml:132-133`, so `/mnt/cache` is per-job and cannot carry progress anyway. +- **It costs one listing.** `cranlike`'s `s3` fork already does exactly this call against this bucket at ~24 000 objects, so the approach is proven at the required scale. + +It must read ETags rather than the slot index's `Built` field, which is how `local/packages-to-build.R:104-130` answers the same question. +Under this design the index is not rewritten until the dependent re-index pipeline runs (section 5), so mid-run it cannot reflect the current run's progress. + +Packages that genuinely fail to build re-publish their CRAN source, so they stay outstanding and would be retried on every restart. +That is already handled upstream: `bincraft::filter_packages_with_errors()` (`R/build_binaries.R:1018`, `:1143`) drops anything with `error_occurred = TRUE`, and `store_build_metadata = TRUE` is passed on every call. +No additional poison-pill filter is needed here. + +Only the flat `src/contrib` path is considered. +The rebuild call passes no `is_r_minor_sensitive`, so it defaults to `FALSE` and only ever targets the flat path; the resume filter matches that scope deliberately. + +### 4. Give each shard a wall-clock budget + +`local/rebuild-missing.R` takes a budget, defaulting to 20 hours, and breaks out of the loop once it is exceeded: + +```r +# exit cleanly rather than being killed, so the dependent re-index still runs +if (difftime(Sys.time(), started, units = "hours") > budget_hours) { + cat(sprintf("Budget of %sh reached after %d/%d packages; stopping cleanly\n", budget_hours, i, n)) + break +} +``` + +It exits 0 and reports how much of the slice it covered. +Every run then has a terminating condition, the re-index and purge always fire, and the remainder is picked up by the next run with no bookkeeping, because section 3 recomputes the outstanding set from scratch. + +### 5. Move the re-index and purge into `.crow/weekly-rebuild-reindex.yaml` + +Three shards per slot means three concurrent `upload_package_index()` calls on the same S3 prefix. +`cranlike::update_PACKAGES()` lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work. +The re-index steps (`.crow/weekly-rebuild-missing.yaml:171-176`) and the purge step (`:187-207`) therefore leave that file entirely. + +The new file carries: + +```yaml +depends_on: + - weekly-rebuild-missing +runs_on: [success, failure] +``` + +`runs_on: [success, failure]` validates as a workflow-level key under `crow lint`, so a failing shard no longer withholds the re-index. +The file uses the same 18-row matrix and the same `when:` gating as `weekly-rebuild-missing`, so it only re-indexes slots that actually ran. +Each row re-indexes the flat slot and every per-minor slot. +`scripts/purge_cdn_zone.sh` runs once on a single row, because all hostnames share pull zone `3857050` and 18 identical zone purges would be waste. + +## Failure behaviour + +| case | today | after | +| -------------------------- | ----------------------------------- | ----------------------------------------------------- | +| one package errors | `tryCatch` logs, loop continues | unchanged | +| a shard fails outright | purge runs, re-index does not | re-index and purge run via `runs_on` | +| a shard exceeds its budget | cannot happen, runs until killed | exits 0, re-index and purge run | +| a shard is killed | nothing runs | still nothing; trigger the re-index pipeline alone | +| a shard restarts | re-walks the list, HEAD per package | one listing, resumes at the first outstanding package | + +The known cost of `depends_on` being file-level rather than row-level: on the weekly cron no slot is re-indexed until the slowest of all 54 jobs finishes. +The 20-hour budget bounds that at roughly one day. + +## Out of scope + +- `build-all-versions` still cannot rebuild source fallbacks, because `local/build-all.R:113-122` drops every version with any `single_builds` row for the platform and arch, which is precisely the source-fallback set. That is a separate change. +- Bunny Perma-Cache eviction. `scripts/purge_cdn_zone.sh` purges the regular edge cache only; see the note in `CLAUDE.md` and issue history. +- The audit that produces the rebuild list is unchanged. + +## Verification + +- `crow lint .crow/` passes for both pipeline files. +- `local/rebuild-missing.R` gets unit coverage in `local/tests/` for the two pure pieces: the interleaved slice (disjoint, covering, deterministic) and the outstanding-set filter (source-served ETag kept, binary ETag dropped, absent object kept). +- A single-slot manual run of `alpine-324-amd64` shard 1 confirms the listing shortcut against the live bucket, and that the reported outstanding count is close to the 8 917 measured above divided by three. +- Restarting that shard mid-run confirms it resumes rather than replaying, by comparing the outstanding count it reports on the second start. From faa678c0c729eaa87bda550badc2037710df8ec8 Mon Sep 17 00:00:00 2001 From: pat-s Date: Wed, 12 Aug 2026 09:18:48 +0000 Subject: [PATCH 04/31] docs(rebuild): record the matrix size against the Crow permutation cap --- .crow/weekly-rebuild-missing.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.crow/weekly-rebuild-missing.yaml b/.crow/weekly-rebuild-missing.yaml index 5afb3c3..a12676d 100644 --- a/.crow/weekly-rebuild-missing.yaml +++ b/.crow/weekly-rebuild-missing.yaml @@ -9,6 +9,12 @@ # single - to run just one. # Arch placement is handled by the group label (rpkgs-amd64, rpkgs-arm64). # +# 9 OS versions x 2 arches x 3 shards = 54 rows. Crow counts the *declared* +# matrix against CROW_MAX_MATRIX_SIZE before any `when:` gate is applied, so a +# single-slot manual run expands all 54 too. The server default is 50 and was +# raised for this; `crow lint` does not check the limit, so adding an OS +# version here is only caught when a pipeline is triggered. +# # The shard picks up its own slice and re-derives what is still outstanding # from the bucket, so a restart resumes rather than replaying; see # local/rebuild-missing.R. From aa4c95457f0ce7f79adea6705d54b098a10c50dd Mon Sep 17 00:00:00 2001 From: pat-s Date: Thu, 13 Aug 2026 13:38:28 +0000 Subject: [PATCH 05/31] fix(rebuild): harden split workflow setup (#164) ## Motivation Weekly rebuild shards can all hit a transient CRAN DNS/index outage at once, and the dependent CDN purge always fails because it tries to clone over the checkout preserved from the re-index step. ## Changes - Retry `uvr add` resolution up to four times with bounded backoff. - Reuse the existing Crow workspace checkout in the CDN purge step. - Remove the purge step's unused Git package and repository token. ## Validation - `crow lint .crow/` - `shellcheck local/uvr-install.sh scripts/purge_cdn_zone.sh` - `git diff --check` Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/164 --- .crow/weekly-rebuild-reindex.yaml | 6 ++---- local/uvr-install.sh | 17 ++++++++++++++--- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml index 224596b..a67875d 100644 --- a/.crow/weekly-rebuild-reindex.yaml +++ b/.crow/weekly-rebuild-reindex.yaml @@ -173,14 +173,12 @@ steps: OTEL_R_METRICS_EXPORTER: none BUNNYNET_API_KEY: from_secret: BUNNYNET_API_KEY - REPO_RO_TOKEN: - from_secret: REPO_RO_TOKEN # All hostnames on the zone share this id, so one purge covers # cran.devxy.io, cran.allianceswisspass.devxy.io and cran.rpkgs.com. BUNNY_PULLZONE: '3857050' commands: - - apk add --no-cache -q bash curl git - - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - apk add --no-cache -q bash curl + # Crow carries the checkout from the re-index step into this step. - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" "$BUNNY_PULLZONE" # Runs on every row rather than on one designated slot: a cron fires only # its own slot's row, so gating on a named slot would leave every other diff --git a/local/uvr-install.sh b/local/uvr-install.sh index 1e25e47..3966e85 100755 --- a/local/uvr-install.sh +++ b/local/uvr-install.sh @@ -84,9 +84,20 @@ trap 'rm -rf "$project_dir"' EXIT cd "$project_dir" "$uvr_bin" init --here --r-version "$r_full" -# --no-install: resolve and lock only. The install happens in the sync below, -# which is the only command that honours --library. -"$uvr_bin" add --no-install "$@" +# --no-install resolves and locks only; retry because concurrent shards can +# expose short-lived DNS or CRAN-index failures and uvr rolls the manifest back +# cleanly after an unsuccessful resolution. +add_attempt=1 +while ! "$uvr_bin" add --no-install "$@"; do + if [ "$add_attempt" -ge 4 ]; then + echo "error: uvr add failed after ${add_attempt} attempts" >&2 + exit 1 + fi + add_delay=$((add_attempt * 10)) + echo "warning: uvr add attempt ${add_attempt} failed; retrying in ${add_delay}s" >&2 + sleep "$add_delay" + add_attempt=$((add_attempt + 1)) +done # TEMPORARY (drop once the images ship a uvr above v0.4.5): the sync below runs # `apt-get install` for every resolved system dependency without refreshing the From a1c1f5e78f2e22db0497fd749067661f063f35ff Mon Sep 17 00:00:00 2001 From: pat-s Date: Thu, 13 Aug 2026 14:08:10 +0000 Subject: [PATCH 06/31] fix(cdn): align repository routing across pull zones (#165) ## Motivation `cran.rpkgs.com` and `cran.allianceswisspass.devxy.io` serve the same B2 repository through separate Bunny pull zones, but only the first zone was managed and purged after weekly reindexing. This allowed the Alliance endpoint to retain stale repository metadata and left locked `renv` restores unable to retrieve versions whose binary archive object was absent. ## Changes - Adopt the Alliance SwissPass pull zone `3265648` into OpenTofu and configure it with the shared B2 origin and middleware script. - Purge both Bunny pull zones after the weekly rebuild reindex. - Preserve the requested public hostname in middleware redirects. - Redirect missing archived binaries to the corresponding CRAN source package, checking whether the version is archived or still current. - Cover the existing archived-binary passthrough behavior in the edge routing matrix. ## Verification - `prek run -a` - `just edge-test` - `crow lint .crow/` - `tofu validate` - `bash -n scripts/purge_cdn_zone.sh` ## Deployment Run `tofu apply` to adopt pull zone `3265648`, publish the middleware release, and align both pull zones. After the apply, rerun the Alliance SwissPass CI restore that requested `cli 3.6.5` and `AzureStor 3.7.1`. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/165 --- .crow/weekly-rebuild-reindex.yaml | 8 ++--- cdn.tf | 56 +++++++++++++++++++++++++++++-- edge/rpkgs-router.test.ts | 7 ++++ edge/rpkgs-router.ts | 35 +++++++++++++++---- scripts/purge_cdn_zone.sh | 39 +++++++++++---------- 5 files changed, 115 insertions(+), 30 deletions(-) diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml index a67875d..bb00332 100644 --- a/.crow/weekly-rebuild-reindex.yaml +++ b/.crow/weekly-rebuild-reindex.yaml @@ -173,13 +173,13 @@ steps: OTEL_R_METRICS_EXPORTER: none BUNNYNET_API_KEY: from_secret: BUNNYNET_API_KEY - # All hostnames on the zone share this id, so one purge covers - # cran.devxy.io, cran.allianceswisspass.devxy.io and cran.rpkgs.com. - BUNNY_PULLZONE: '3857050' + # cran.rpkgs.com and cran.allianceswisspass.devxy.io are on separate + # Bunny pull zones, so both must be purged after the shared origin changes. + BUNNY_PULLZONES: '3857050 3265648' commands: - apk add --no-cache -q bash curl # Crow carries the checkout from the re-index step into this step. - - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" "$BUNNY_PULLZONE" + - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" $BUNNY_PULLZONES # Runs on every row rather than on one designated slot: a cron fires only # its own slot's row, so gating on a named slot would leave every other # slot unpurged. A manual "all" run therefore purges the zone 18 times, diff --git a/cdn.tf b/cdn.tf index 5e8899d..01191a5 100644 --- a/cdn.tf +++ b/cdn.tf @@ -32,7 +32,7 @@ # cache_stale = ["offline", "updating"] # use_background_update = true - # block_ips = var.cdn_block_ips +# block_ips = var.cdn_block_ips # # 50 TB # limit_bandwidth = 50000000000000 @@ -82,7 +82,7 @@ resource "bunnynet_pullzone" "cran_rpkgs_com" { cache_expiration_time = 31919000 websockets_enabled = false - errorpage_whitelabel = true + errorpage_whitelabel = true origin { type = "OriginUrl" @@ -147,6 +147,58 @@ resource "bunnynet_pullzone_hostname" "cran_rpkgs_com" { tls_enabled = true } +# Alliance SwissPass historically used a separate, manually configured pull +# zone. Adopt it so both public repositories use the same B2 origin, middleware +# release and cache behavior. +import { + to = bunnynet_pullzone.cran_allianceswisspass + id = "3265648" +} + +resource "bunnynet_pullzone" "cran_allianceswisspass" { + name = "cran-allianceswisspass" + + cache_errors = false + cache_expiration_time = 31919000 + websockets_enabled = false + errorpage_whitelabel = true + + origin { + type = "OriginUrl" + url = "https://devxy-rpkgs-binaries.s3.eu-central-003.backblazeb2.com" + middleware_script = bunnynet_compute_script.rpkgs_router.id + } + + routing { + filters = [ + "scripting", + ] + } + + s3_auth_enabled = true + s3_auth_key = var.B2_S3_ACCESS_KEY + s3_auth_secret = var.B2_S3_SECRET_KEY + s3_auth_region = "eu-central-003" + + cache_enabled = true + request_coalescing_enabled = true + block_post_requests = true + cache_vary_headers = ["User-Agent"] + + limit_requests = 5000 + limit_connections = 1000 + + safehop_enabled = true + add_canonical_header = true + cache_stale = ["offline", "updating"] + block_ips = var.cdn_block_ips + + # 50 TB + limit_bandwidth = 50000000000000 + + block_root_path = true +} + # resource "bunnynet_storage_zone" "devxy-r-binaries" { # name = "devxy-r-binaries-storage" # region = "DE" diff --git a/edge/rpkgs-router.test.ts b/edge/rpkgs-router.test.ts index 553185d..9493f8a 100644 --- a/edge/rpkgs-router.test.ts +++ b/edge/rpkgs-router.test.ts @@ -117,6 +117,13 @@ Deno.test('rpkgs-router', async (t) => { assertEquals(res.status, 200); }); + await t.step('serves an archived binary when it exists', async () => { + const path = `${SLOT}/Archive/xml2/xml2_1.5.2.tar.gz`; + const res = await probe(path, UA_R45_MUSL); + assertEquals(res.status, 200); + assertEquals(res.location, null); + }); + await t.step('does not redirect a path already under a minor', async () => { const res = await probe(`${SLOT}/4.5/PACKAGES.gz`, UA_R45_MUSL); assertEquals(res.location, null); diff --git a/edge/rpkgs-router.ts b/edge/rpkgs-router.ts index 9cd410b..9278d4a 100644 --- a/edge/rpkgs-router.ts +++ b/edge/rpkgs-router.ts @@ -27,6 +27,7 @@ import * as BunnySDK from 'https://esm.sh/@bunny.net/edgescript-sdk@0.12'; const PUBLIC_CDN_ORIGIN = 'https://cran.rpkgs.com'; const CRAN_ORIGIN = 'https://cran.r-project.org'; +const PUBLIC_CDN_HOSTS = new Set(['cran.rpkgs.com', 'cran.allianceswisspass.devxy.io']); /** Slots ("/", comma separated) whose per-minor index is a union. */ const UNION_SLOTS = new Set( @@ -47,6 +48,10 @@ const INDEX_FILE_REGEX = /^PACKAGES(\.gz|\.rds)?$/; const SRC_CONTRIB_REGEX = /^\/src\/contrib\/(.+)$/; +/** A binary archive URL whose upstream source counterpart CRAN can serve. */ +const ARCHIVE_TARBALL_REGEX = + /^\/(?:amd64|arm64)\/[a-z0-9._-]+\/latest\/src\/contrib\/Archive\/([^/]+)\/([^/]+\.tar\.gz)$/; + const MACOS_BIN_REGEX = /^\/bin\/macosx\/(big-sur-arm64|big-sur-x86_64|monterey-arm64|monterey-x86_64)\/contrib\/([0-9.]+)\/(.+)$/; @@ -85,6 +90,10 @@ function redirectTo(location: string, status = 302): Response { }); } +function publicCdnOrigin(url: URL): string { + return PUBLIC_CDN_HOSTS.has(url.hostname) ? url.origin : PUBLIC_CDN_ORIGIN; +} + function extractRMinor(userAgent: string): string | null { for (const regex of R_MINOR_REGEXES) { const match = userAgent.match(regex); @@ -181,15 +190,14 @@ BunnySDK.net.http const url = new URL(ctx.request.url); const path = normalizePathname(url.pathname); const userAgent = ctx.request.headers.get('User-Agent') || ''; + const publicOrigin = publicCdnOrigin(url); // macOS clients are served from CRAN's own binary tree. const srcContrib = path.match(SRC_CONTRIB_REGEX); if (srcContrib && /darwin/.test(userAgent)) { const mac = parseMacUserAgent(userAgent); if (mac) { - return Promise.resolve( - redirectTo(`${PUBLIC_CDN_ORIGIN}/bin/macosx/${mac.os}/contrib/${mac.rver}/${srcContrib[1]}`), - ); + return Promise.resolve(redirectTo(`${publicOrigin}/bin/macosx/${mac.os}/contrib/${mac.rver}/${srcContrib[1]}`)); } } @@ -213,7 +221,7 @@ BunnySDK.net.http if (target === path) { return Promise.resolve(ctx.request); } - return Promise.resolve(redirectTo(`${PUBLIC_CDN_ORIGIN}${target}`)); + return Promise.resolve(redirectTo(`${publicOrigin}${target}`)); } // The bare `https://cran.rpkgs.com` form, resolved from the User-Agent. @@ -224,12 +232,27 @@ BunnySDK.net.http } const rest = srcContrib ? srcContrib[1] : ''; - return Promise.resolve(redirectTo(`${PUBLIC_CDN_ORIGIN}${contribPath(slot, rest, userAgent)}`)); + return Promise.resolve(redirectTo(`${publicOrigin}${contribPath(slot, rest, userAgent)}`)); } return Promise.resolve(ctx.request); }) - .onOriginResponse((ctx) => { + .onOriginResponse(async (ctx) => { + const path = normalizePathname(new URL(ctx.request.url).pathname); + const archive = path.match(ARCHIVE_TARBALL_REGEX); + + // Binary archives can be incomplete when an older build never succeeded. + // Preserve renv/remotes version restores by falling back to CRAN's source + // package only for an absent archived tarball. A requested version can be + // either archived upstream or still current, so probe the archive first. + // Other 404s remain visible. + if (ctx.response.status === 404 && archive) { + const archiveUrl = `${CRAN_ORIGIN}/src/contrib/Archive/${archive[1]}/${archive[2]}`; + const archiveResponse = await fetch(archiveUrl, { method: 'HEAD' }); + const sourceUrl = archiveResponse.ok ? archiveUrl : `${CRAN_ORIGIN}/src/contrib/${archive[2]}`; + return redirectTo(sourceUrl); + } + ctx.response.headers.append('X-Via', 'MyMiddleware'); return Promise.resolve(ctx.response); }); diff --git a/scripts/purge_cdn_zone.sh b/scripts/purge_cdn_zone.sh index 391a814..4853245 100755 --- a/scripts/purge_cdn_zone.sh +++ b/scripts/purge_cdn_zone.sh @@ -18,35 +18,38 @@ # objects were replaced. The cost is a cold cache for everything else, which is # why this is not used by the daily update path. # -# All hostnames on the zone (cran.devxy.io, cran.allianceswisspass.devxy.io, -# cran.rpkgs.com) share pull zone 3857050, so one purge covers all of them. +# The public hostnames currently use separate pull zones, so callers must pass +# every zone that serves the repository. # # Usage: -# purge_cdn_zone.sh +# purge_cdn_zone.sh [...] # set -euo pipefail if (($# < 2)); then - echo "usage: $0 " >&2 + echo "usage: $0 [...]" >&2 exit 2 fi api_key="$1" -zone_id="$2" +shift -echo "Purging BunnyCDN pull zone ${zone_id}" +for zone_id in "$@"; do + echo "Purging BunnyCDN pull zone ${zone_id}" -status=$( - curl -sS -o /tmp/purge_zone_response.txt -w '%{http_code}' -X POST \ - -H "AccessKey: ${api_key}" \ - -H "Content-Length: 0" \ - "https://api.bunny.net/pullzone/${zone_id}/purgeCache" -) + response_file="/tmp/purge_zone_response_${zone_id}.txt" + status=$( + curl -sS -o "${response_file}" -w '%{http_code}' -X POST \ + -H "AccessKey: ${api_key}" \ + -H "Content-Length: 0" \ + "https://api.bunny.net/pullzone/${zone_id}/purgeCache" + ) -if [[ "${status}" != "200" && "${status}" != "204" ]]; then - echo "Purge of pull zone ${zone_id} failed with HTTP ${status}:" >&2 - cat /tmp/purge_zone_response.txt >&2 - exit 1 -fi + if [[ "${status}" != "200" && "${status}" != "204" ]]; then + echo "Purge of pull zone ${zone_id} failed with HTTP ${status}:" >&2 + cat "${response_file}" >&2 + exit 1 + fi -echo "Purged pull zone ${zone_id} (HTTP ${status})" + echo "Purged pull zone ${zone_id} (HTTP ${status})" +done From 9bded261eecba451d4d544fbde68a68cd1180873 Mon Sep 17 00:00:00 2001 From: pat-s Date: Thu, 13 Aug 2026 14:13:04 +0000 Subject: [PATCH 07/31] fix(cdn): restore Alliance pull-zone hostname (#166) ## Motivation Applying #165 recreated the Alliance SwissPass pull zone without its custom hostname because the hostname association was not represented in OpenTofu. The recreated zone also received a new numeric ID, making the weekly purge configuration stale. ## Changes - Manage `cran.allianceswisspass.devxy.io` as a pull-zone hostname with TLS and forced HTTPS. - Resolve the Alliance pull-zone ID from its hostname before purging instead of persisting a replaceable numeric ID. - Install `jq` in the purge step for the Bunny API lookup. ## Verification - Targeted `prek` hooks pass. - `tofu validate` passes. - `crow lint .crow/` passes. - `just edge-test` passes all 14 routing steps. - `bash -n scripts/purge_cdn_zone.sh` passes. ## Deployment Run `tofu apply` to restore the Alliance hostname on the recreated pull zone. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/166 --- .crow/weekly-rebuild-reindex.yaml | 4 ++-- cdn.tf | 7 ++++++ scripts/purge_cdn_zone.sh | 40 +++++++++++++++++++++++++++---- 3 files changed, 45 insertions(+), 6 deletions(-) diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml index bb00332..250b225 100644 --- a/.crow/weekly-rebuild-reindex.yaml +++ b/.crow/weekly-rebuild-reindex.yaml @@ -175,9 +175,9 @@ steps: from_secret: BUNNYNET_API_KEY # cran.rpkgs.com and cran.allianceswisspass.devxy.io are on separate # Bunny pull zones, so both must be purged after the shared origin changes. - BUNNY_PULLZONES: '3857050 3265648' + BUNNY_PULLZONES: '3857050 cran.allianceswisspass.devxy.io' commands: - - apk add --no-cache -q bash curl + - apk add --no-cache -q bash curl jq # Crow carries the checkout from the re-index step into this step. - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" $BUNNY_PULLZONES # Runs on every row rather than on one designated slot: a cron fires only diff --git a/cdn.tf b/cdn.tf index 01191a5..96877b2 100644 --- a/cdn.tf +++ b/cdn.tf @@ -199,6 +199,13 @@ resource "bunnynet_pullzone" "cran_allianceswisspass" { block_root_path = true } +resource "bunnynet_pullzone_hostname" "cran_allianceswisspass" { + pullzone = bunnynet_pullzone.cran_allianceswisspass.id + name = "cran.allianceswisspass.devxy.io" + force_ssl = true + tls_enabled = true +} + # resource "bunnynet_storage_zone" "devxy-r-binaries" { # name = "devxy-r-binaries-storage" # region = "DE" diff --git a/scripts/purge_cdn_zone.sh b/scripts/purge_cdn_zone.sh index 4853245..648dfc3 100755 --- a/scripts/purge_cdn_zone.sh +++ b/scripts/purge_cdn_zone.sh @@ -19,22 +19,54 @@ # why this is not used by the daily update path. # # The public hostnames currently use separate pull zones, so callers must pass -# every zone that serves the repository. +# every zone that serves the repository. A zone can be identified by its +# numeric ID or by one of its hostnames; hostname lookup avoids persisting IDs +# that change when a zone is recreated. # # Usage: -# purge_cdn_zone.sh [...] +# purge_cdn_zone.sh [...] # set -euo pipefail if (($# < 2)); then - echo "usage: $0 [...]" >&2 + echo "usage: $0 [...]" >&2 exit 2 fi api_key="$1" shift -for zone_id in "$@"; do +resolve_zone_id() { + local zone="$1" + local response_file + local zone_id + + if [[ "${zone}" =~ ^[0-9]+$ ]]; then + echo "${zone}" + return + fi + + response_file=$(mktemp) + curl -sS -o "${response_file}" \ + -H "AccessKey: ${api_key}" \ + "https://api.bunny.net/pullzone" + zone_id=$( + jq -r --arg hostname "${zone}" \ + '(.Items // .)[] | select(any(.Hostnames[]?; .Value == $hostname)) | .Id' \ + "${response_file}" + ) + rm -f "${response_file}" + + if [[ -z "${zone_id}" ]]; then + echo "Could not find BunnyCDN pull zone for hostname ${zone}" >&2 + exit 1 + fi + + echo "${zone_id}" +} + +for zone in "$@"; do + zone_id=$(resolve_zone_id "${zone}") echo "Purging BunnyCDN pull zone ${zone_id}" response_file="/tmp/purge_zone_response_${zone_id}.txt" From 706fd10d79206ed2784cd7f6bbac1a5e897e390b Mon Sep 17 00:00:00 2001 From: automation-bot Date: Fri, 14 Aug 2026 00:32:00 +0000 Subject: [PATCH 08/31] chore(deps): update terraform bunnynet to ~> 0.18 --- .terraform.lock.hcl | 72 ++++++++++++++++++++++----------------------- provider.tf | 2 +- 2 files changed, 37 insertions(+), 37 deletions(-) diff --git a/.terraform.lock.hcl b/.terraform.lock.hcl index aca8ce1..a21cf43 100644 --- a/.terraform.lock.hcl +++ b/.terraform.lock.hcl @@ -38,43 +38,43 @@ provider "registry.opentofu.org/hashicorp/http" { } provider "registry.terraform.io/bunnyway/bunnynet" { - version = "0.17.0" - constraints = "~> 0.17" + version = "0.18.0" + constraints = "~> 0.18" hashes = [ - "h1:+qDt35lVSK7acw6a1xHuPYrqmZEcHSmtd+6n1TxNuYw=", - "h1:1dCu2l4DhPBjizVAH/WwAjT1Xbo52K4PMvHoD5zUhuU=", - "h1:Dvn46Auwuel4jqrqZXs2D7kdujNhs17LEmqhuY0k4/4=", - "h1:M5eDL3m2uSEr1XATJW0foHzKl8pFhCtgKuOM24bJRwU=", - "h1:PddaC7nM/gY4x9i3xy6TxOs9MAu2/6g58Xs/gv4DRV8=", - "h1:QVIKiZluI+NQAKu8NpFBl3Nvyx+d81vW9btEUdIQREc=", - "h1:S6TnzXHsRoGYvC1vJBkDiVEc0spceksY4n6x5WN5iYw=", - "h1:VcxZDWqCWMSjcUsC1K4sB6uYEoeoou+BC0ePoJXmf3A=", - "h1:W0y/agBVqls1cJlFGFYMu2VnqoPXFzxVHPIYe3OqfYQ=", - "h1:XmNd5fP9a0O77ve5BMQP2vARExgIa7rYl6KvyUYXPSs=", - "h1:e0EFKrWSQwaa/kGhnha4DXk4T68Av8QxP84mRSdWC9M=", - "h1:eM+/lUiU0pNSgQKoqKPgE3xJrJ0MHIpKG+yhaGB/P0M=", - "h1:fPWWA4T0/y7GX+tCGN23l1jODhZ3uCdR/MKgZDXYpAE=", - "h1:g+r2GVi4gVC4DuQg3PL70gW9BDskgWUzCBIMXTUq63A=", - "h1:gaZ8eALDtVHqykVDHav8004gHiMGaYR/3KwET0FUgao=", - "h1:kbqW25eaiv4N/N/z+sxLdJZ15yh5cgnRD/q6RclPMLc=", - "h1:rGjxue3mXRyQQqpywTXC4zK//JAtf0Cz7RP+uPMMJjw=", - "zh:05943fef14c2028f4722bf078aa1889229e94302f7678cc6f63adb669d8ea612", - "zh:26a163930a92a7408f7bbd0130064b84df8a232b500d8c6c3989952986308539", - "zh:41305feaaade55391447521ec309f3c038b631ca542907ad95132fab71a7e116", - "zh:606919a930f0299948504adbdcd0f239a8af5c418f85741c48f8add370a3d038", - "zh:66963d5b445639511939fc508513fd31da3ee1d4ee1a565ee396c9532897a349", - "zh:6c981ec0c8545556395c43e2511861ab65ee9ecf2a960480e7889c3af0d23af3", - "zh:7334a1bdb726ce1f1bf0a3155f30f84f65206980c229c832ff5f0b0718c44e0b", - "zh:75f6c86bf74511e605423332d113711c76c8028361a32282fb3359d6c7ecae9e", - "zh:7aebb1a01cfe8be54903853202ae06eba14ad99c37d230ed93ce7d6633e05e9b", + "h1:+6VXZmSSeIchab8WY+UFZxaXu6poxZbiJenYwog+Z2M=", + "h1:Bz94gADR83NmrsZC50LheryVcS9erG6tF052hNPqz0U=", + "h1:FGkcn6ieyNPr56Riv75IsCa6okZ9ZHqqFd08Y4i5cPI=", + "h1:GqPjTKaOlhhNIxpPHWJpWdGkEtvUDtuOkjYbVIInfgU=", + "h1:NMDWYRisSFwepjfYsf4MrcyP7Ihjlty937l1BW1Cztg=", + "h1:OusNMAZdIIbxWwHXwH8tYzdbXZPRS8dW7cncSYcHfNI=", + "h1:OzgasS4oZCZjYAOrqGy7RpPNbTPX5wGaaw7TDnasSJM=", + "h1:RyK3DC7cM4T2I83OvnO+UXU+eCXLG5vDlYZ/bWKHq7g=", + "h1:U/JKg4BNWii2mK3bNsOWYMBTpkXqGsWb1ypSJvSKl9Y=", + "h1:UjSUm1AU2wZ8eyC3LDRwWatW9cKD/I0rk7MOeUDDkWc=", + "h1:a888ExTeqWKxaB2GBsDIRSm8jOOMLkPFcABrJxxA8qc=", + "h1:aShYLfSapfo1Ozy33zL5WfmYt5UF2m5EziespPOGNAk=", + "h1:bHdpp2ecmvDoezqg8TirE32SnuDb108/NZ4xBGvJy98=", + "h1:lLYZN5cetXgLPJjo+eR+kwodOYGsQetzV6bg5Ki58tk=", + "h1:szi71DaqI4yB6fI55hBbtVZg1GDWJVkO9+WF/1sZ+IE=", + "h1:vK4jZYZZhD6M4cRmX+171mxoJgb2FSNiBNAUhtm/TNs=", + "h1:yNnSUskn43648XU/YS0e9NUdoHImo7pFMIlBzujBoxY=", + "zh:0c3adf039df2fead1e36b8e7887d965223f03b6b5cdd921ec0c98beaa04fdec0", + "zh:12a3db29733e6619e216fac9aa774cbf67da2105f48c5c4dee8df046d69656fa", + "zh:1330b83c949165434e4d0db5e097a1741376cdb66c06c37b2a97ecea8e06b0f0", + "zh:186113259ab0f80ab9079f0a8a810e93ee097f6740100a0a444f805e8772c12e", + "zh:242fe10185da8a700b0e9a6a52e3d1f6592e8e189aaf5e9a79cf515ad5c7ff25", + "zh:28b005b0aa9485c492326fbb4287d3cb465f25a8d572b8c740e041e040d80aa2", + "zh:3b2cd0daa767dfff67dd39c98d1c82b25e70eb81df77c0d6dd762a0dedec16e1", + "zh:3f14b244740f6c6420d298846708e9f3853c8eda685d3a4d71fb0ffec021ad9c", + "zh:469bfd08deefd90e89c930e5e45bfd06a58a1eadb85dfbee9fc48ae296e8a8c9", + "zh:634d96b84b2e77d25baa074c357e831252c8b8114926ef72e1afb68a4bc9eb65", + "zh:6e837b9485e56f84b5e2bb396fa549b628b8e075257fa01bf8c796ce91d20e58", "zh:890df766e9b839623b1f0437355032a3c006226a6c200cd911e15ee1a9014e9f", - "zh:9041d0e20c9ceea532de6eebf5cb3a27dad0bb49d3f5b5154be2a08d68fbbf1f", - "zh:a6bbf65431a02be4df0ebb1cbe01185ad357ff6e33c01bd0558f59bed90c8f36", - "zh:c6d075a31096f080c388dfe46036f451c0cc114c3311a4f46ab8dbe1938a202f", - "zh:dd8703f7b55b8bc8e10f8718bea889781100b18e932b04898995b63178c3d36e", - "zh:dd92a5cd4e133a4000e7e5bc8cce876ae0ed803543cedd2f3d590661ba244d04", - "zh:e024fdf121bebc48c1e6debea344c6d4f174117f3ae605fca6e13b9705d92d22", - "zh:ee0e80c31b438e35fa1608f6a2f5824d2806db1e5e8b9f7a90986585c7bcb895", - "zh:fc2d4b705411b48f8c045981f9368a3ea2f74969dd6302008c31ff0bedd51f0a", + "zh:89e02979311e6727a50586b04cb343e9bbec74a6f4d1dba9352971e4334d64c9", + "zh:a6796143fe61ae52d236b7ea96bddbc6317851a0e2117eb2f55cd73c803b4b6f", + "zh:ad2542160e7b57ee8a016cddbf1d32663baf74fe30fcf24df64da75cb13128b0", + "zh:bfee81c153e8b50121fd58eedeeb84bfa64a3d17f23900e35143c4a86474cbd2", + "zh:eeba652697908712cd1ce2c7d2925f6e6bb2d18ea5d3e6ca185e88eeb80f3c80", + "zh:f0b5b8fd6942647c358440591d56a5cb82fda92f7866898654a66ec901174f1f", ] } diff --git a/provider.tf b/provider.tf index badbbc1..d4f2564 100644 --- a/provider.tf +++ b/provider.tf @@ -2,7 +2,7 @@ terraform { required_providers { bunnynet = { source = "registry.terraform.io/BunnyWay/bunnynet" - version = "~> 0.17" + version = "~> 0.18" } } } From 132d1d2d3cd37611db98733a57d8345a218e152b Mon Sep 17 00:00:00 2001 From: pat-s Date: Fri, 14 Aug 2026 06:44:10 +0000 Subject: [PATCH 09/31] docs(ci): clarify parallel manual matrix runs --- .crow/process-updates.yaml | 6 +++--- .crow/weekly-audit-missing.yaml | 6 +++--- .crow/weekly-rebuild-missing.yaml | 8 ++++---- .crow/weekly-rebuild-reindex.yaml | 2 +- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/.crow/process-updates.yaml b/.crow/process-updates.yaml index 7600876..021d6cb 100644 --- a/.crow/process-updates.yaml +++ b/.crow/process-updates.yaml @@ -3,15 +3,15 @@ # Routing is preserved 1:1: # - cron: each existing `process-cran-updates--` cron fires only # its matching matrix row (via the per-row `cron:` name filter). -# - manual: pick a target from the `process_cran_updates` dropdown -# ("all" = every os/arch). +# - manual: pick a target from the `process_cran_updates` dropdown; +# "all" fans out every os/arch as parallel matrix workflows. # Arch placement is handled by the group label (rpkgs-amd64, rpkgs-arm64). variables: # Gates this pipeline. A manual pipeline creation instantiates every file in # .crow/, and a declared default is applied even when the run never passed # this variable, so the default must be a value that matches no matrix row. process_cran_updates: - description: "Manual run target: a specific -, 'all' for every os/arch, or 'none' to run nothing." + description: "Manual run target: a specific -, 'all' to run every os/arch in parallel, or 'none' to run nothing." options: - none - all diff --git a/.crow/weekly-audit-missing.yaml b/.crow/weekly-audit-missing.yaml index 3a9d98f..428aa04 100644 --- a/.crow/weekly-audit-missing.yaml +++ b/.crow/weekly-audit-missing.yaml @@ -3,15 +3,15 @@ # Routing is preserved 1:1: # - cron: each existing `weekly-audit-missing--` cron fires only # its matching matrix row (via the per-row `cron:` name filter). -# - manual: pick a target from the `weekly_audit_missing` dropdown -# ("all" = every os/arch). +# - manual: pick a target from the `weekly_audit_missing` dropdown; +# "all" fans out every os/arch as parallel matrix workflows. # Arch placement is via the group label (rpkgs-amd64, rpkgs-arm64). variables: # Gates this pipeline. A manual pipeline creation instantiates every file in # .crow/, and a declared default is applied even when the run never passed # this variable, so the default must be a value that matches no matrix row. weekly_audit_missing: - description: "Manual run target: a specific -, 'all' for every os/arch, or 'none' to run nothing." + description: "Manual run target: a specific -, 'all' to run every os/arch in parallel, or 'none' to run nothing." options: - none - all diff --git a/.crow/weekly-rebuild-missing.yaml b/.crow/weekly-rebuild-missing.yaml index 5afb3c3..9639792 100644 --- a/.crow/weekly-rebuild-missing.yaml +++ b/.crow/weekly-rebuild-missing.yaml @@ -4,9 +4,9 @@ # - cron: each existing `weekly-rebuild-missing--` cron fires only # its matching matrix rows (via the per-row `cron:` name filter), # which is now all three shards of that slot. -# - manual: `weekly_rebuild_missing` dropdown, default "all" (matches the -# previous bare manual trigger that ran every os/arch); pick a -# single - to run just one. +# - manual: pick a target from the `weekly_rebuild_missing` dropdown; +# "all" fans out every os/arch and shard as parallel matrix +# workflows, while a single - runs its three shards. # Arch placement is handled by the group label (rpkgs-amd64, rpkgs-arm64). # # The shard picks up its own slice and re-derives what is still outstanding @@ -21,7 +21,7 @@ variables: # .crow/, and a declared default is applied even when the run never passed # this variable, so the default must be a value that matches no matrix row. weekly_rebuild_missing: - description: "Manual run target: a specific -, 'all' for every os/arch, or 'none' to run nothing." + description: "Manual run target: a specific -, 'all' to run every os/arch in parallel, or 'none' to run nothing." options: - none - all diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml index 250b225..2e0f2a0 100644 --- a/.crow/weekly-rebuild-reindex.yaml +++ b/.crow/weekly-rebuild-reindex.yaml @@ -16,7 +16,7 @@ variables: # exactly the slots it rebuilt. A manual pipeline creation instantiates every # file in .crow/, so the default must match no matrix row. weekly_rebuild_missing: - description: "Manual run target: a specific -, 'all' for every os/arch, or 'none' to run nothing." + description: "Manual run target: a specific -, 'all' to run every os/arch in parallel, or 'none' to run nothing." options: - none - all From b75fd2f1c43cf0355e3e2eb9e51d73f16a1321eb Mon Sep 17 00:00:00 2001 From: pat-s Date: Fri, 14 Aug 2026 06:56:41 +0000 Subject: [PATCH 10/31] fix(ci): split oversized weekly rebuild matrix --- .crow/weekly-rebuild-missing-ubuntu-2604.yaml | 140 ++++++++++++++++++ .crow/weekly-rebuild-missing.yaml | 41 +---- .crow/weekly-rebuild-reindex-ubuntu-2604.yaml | 110 ++++++++++++++ .crow/weekly-rebuild-reindex.yaml | 9 -- 4 files changed, 253 insertions(+), 47 deletions(-) create mode 100644 .crow/weekly-rebuild-missing-ubuntu-2604.yaml create mode 100644 .crow/weekly-rebuild-reindex-ubuntu-2604.yaml diff --git a/.crow/weekly-rebuild-missing-ubuntu-2604.yaml b/.crow/weekly-rebuild-missing-ubuntu-2604.yaml new file mode 100644 index 0000000..0082e0e --- /dev/null +++ b/.crow/weekly-rebuild-missing-ubuntu-2604.yaml @@ -0,0 +1,140 @@ +# ubuntu-2604 rows split from weekly-rebuild-missing.yaml. +# +# The complete rebuild matrix has 54 permutations, while Crow accepts at most +# 50 per workflow. Keeping these six rows in a companion workflow lets a manual +# "all" run fan out every slot and shard without exceeding that compiler limit. + +variables: + # This is the same gate as the main rebuild workflow. The safe default must + # match no row because every file in .crow/ is evaluated on a manual run. + weekly_rebuild_missing: + description: "Manual run target: a specific -, 'all' to run every os/arch in parallel, or 'none' to run nothing." + options: + - none + - all + - alpine-322-amd64 + - alpine-322-arm64 + - alpine-323-amd64 + - alpine-323-arm64 + - alpine-324-amd64 + - alpine-324-arm64 + - redhat-8-amd64 + - redhat-8-arm64 + - redhat-9-amd64 + - redhat-9-arm64 + - redhat-10-amd64 + - redhat-10-arm64 + - ubuntu-2204-amd64 + - ubuntu-2204-arm64 + - ubuntu-2404-amd64 + - ubuntu-2404-arm64 + - ubuntu-2604-amd64 + - ubuntu-2604-arm64 + default: none + +when: + - event: cron + cron: weekly-rebuild-missing-${OS}-${ARCH} + - event: manual + evaluate: 'weekly_rebuild_missing == "all" || weekly_rebuild_missing == "${OS}-${ARCH}"' + +skip_clone: true + +labels: + group: rpkgs-${ARCH} + +matrix: + include: + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 3 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 3 + +steps: + - name: 'Rebuild missing binaries' + image: reg.devxy.io/rpkgs/build-env-${IMG} + pull: true + environment: + OTEL_R_TRACES_EXPORTER: none + OTEL_R_LOGS_EXPORTER: none + OTEL_R_METRICS_EXPORTER: none + RED_HAT_DEV_PW: + from_secret: RED_HAT_DEV_PW + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + PGPASS: + from_secret: PGPASS + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + GITHUB_PAT: + from_secret: GITHUB_PAT + FORGEJO_TOKEN: + from_secret: FORGEJO_TOKEN + GIT_USER: pat-s + UVR_CACHE_DIR: /mnt/cache/uvr/cache + UVR_PACKAGES_DIR: /mnt/cache/uvr/packages + R_LIBS_USER: /mnt/cache/R-pkgs + R_VERSION: ${R_VERSION} + CCACHE_DIR: /mnt/cache/ccache + PLATFORM: ${OS} + ARCH: ${ARCH} + NCPUS: 2 + SPLIT_INTO: ${SPLIT_INTO} + SPLIT_INDEX: ${SPLIT_INDEX} + # Wall clock after which the shard stops cleanly instead of having to be + # killed. A kill matches neither `success` nor `failure`, so it would skip + # the dependent re-index and leave rebuilt binaries behind a stale edge. + REBUILD_BUDGET_HOURS: 20 + commands: + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - mkdir -p /mnt/cache/uvr/cache /mnt/cache/uvr/packages /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages + - rm -rf /mnt/cache/R-pkgs/00LOCK-* + - /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R + - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' + - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi + - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh httr2 + - /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")' + - $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX $REBUILD_BUDGET_HOURS 2>&1 + backend_options: + docker: + resources: + requests: + memory: 5Gi + cpu: 3000m + limits: + memory: 18Gi + cpu: 3000m diff --git a/.crow/weekly-rebuild-missing.yaml b/.crow/weekly-rebuild-missing.yaml index 9639792..aeb2f50 100644 --- a/.crow/weekly-rebuild-missing.yaml +++ b/.crow/weekly-rebuild-missing.yaml @@ -1,5 +1,7 @@ -# Consolidated weekly-rebuild-missing pipeline (all platforms, both arches). +# Main weekly-rebuild-missing workflow (all slots except ubuntu-2604). # Three matrix rows per OS/arch, one per shard of that slot's rebuild list. +# ubuntu-2604 lives in weekly-rebuild-missing-ubuntu-2604.yaml so a manual +# "all" run stays below Crow's 50-permutation matrix limit. # Routing is preserved 1:1: # - cron: each existing `weekly-rebuild-missing--` cron fires only # its matching matrix rows (via the per-row `cron:` name filter), @@ -346,43 +348,6 @@ matrix: IMG: ubuntu:noble SPLIT_INTO: 3 SPLIT_INDEX: 3 - - OS: ubuntu-2604 - ARCH: amd64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 1 - - OS: ubuntu-2604 - ARCH: amd64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 2 - - OS: ubuntu-2604 - ARCH: amd64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 3 - - OS: ubuntu-2604 - ARCH: arm64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 1 - - OS: ubuntu-2604 - ARCH: arm64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 2 - - OS: ubuntu-2604 - ARCH: arm64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 3 - steps: - name: 'Rebuild missing binaries' image: reg.devxy.io/rpkgs/build-env-${IMG} diff --git a/.crow/weekly-rebuild-reindex-ubuntu-2604.yaml b/.crow/weekly-rebuild-reindex-ubuntu-2604.yaml new file mode 100644 index 0000000..3ab086a --- /dev/null +++ b/.crow/weekly-rebuild-reindex-ubuntu-2604.yaml @@ -0,0 +1,110 @@ +# Re-index ubuntu-2604 after its split rebuild workflow finishes. + +variables: + # Mirrors the shared rebuild gate. The default must match no matrix row so + # unrelated manual runs do not rewrite package indexes. + weekly_rebuild_missing: + description: "Manual run target: a specific -, 'all' to run every os/arch in parallel, or 'none' to run nothing." + options: + - none + - all + - alpine-322-amd64 + - alpine-322-arm64 + - alpine-323-amd64 + - alpine-323-arm64 + - alpine-324-amd64 + - alpine-324-arm64 + - redhat-8-amd64 + - redhat-8-arm64 + - redhat-9-amd64 + - redhat-9-arm64 + - redhat-10-amd64 + - redhat-10-arm64 + - ubuntu-2204-amd64 + - ubuntu-2204-arm64 + - ubuntu-2404-amd64 + - ubuntu-2404-arm64 + - ubuntu-2604-amd64 + - ubuntu-2604-arm64 + default: none + +when: + - event: cron + cron: weekly-rebuild-missing-${OS}-${ARCH} + - event: manual + evaluate: 'weekly_rebuild_missing == "all" || weekly_rebuild_missing == "${OS}-${ARCH}"' + +depends_on: + - weekly-rebuild-missing-ubuntu-2604 + +runs_on: [success, failure] + +skip_clone: true + +labels: + group: rpkgs-${ARCH} + +matrix: + include: + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + +steps: + - name: 'Re-index the slot' + image: reg.devxy.io/rpkgs/build-env-${IMG} + pull: true + environment: + OTEL_R_TRACES_EXPORTER: none + OTEL_R_LOGS_EXPORTER: none + OTEL_R_METRICS_EXPORTER: none + RED_HAT_DEV_PW: + from_secret: RED_HAT_DEV_PW + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + GIT_USER: pat-s + R_LIBS_USER: /mnt/cache/R-pkgs + R_VERSION: ${R_VERSION} + PLATFORM: ${OS} + ARCH: ${ARCH} + commands: + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - mkdir -p /mnt/cache/R-pkgs + - rm -rf /mnt/cache/R-pkgs/00LOCK-* + - /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R + # The codename is detected from the image's /etc/os-release. + - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/[0-9]*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e "library(bincraft); upload_package_index(r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done + + - name: Purge CDN cache + image: reg.devxy.io/docker.io/library/alpine:3.24 + environment: + OTEL_R_TRACES_EXPORTER: none + OTEL_R_LOGS_EXPORTER: none + OTEL_R_METRICS_EXPORTER: none + BUNNYNET_API_KEY: + from_secret: BUNNYNET_API_KEY + # cran.rpkgs.com and cran.allianceswisspass.devxy.io are on separate + # Bunny pull zones, so both must be purged after the shared origin changes. + BUNNY_PULLZONES: '3857050 cran.allianceswisspass.devxy.io' + commands: + - apk add --no-cache -q bash curl jq + # Crow carries the checkout from the re-index step into this step. + - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" $BUNNY_PULLZONES + # Run even when the re-index above failed: the objects were still replaced, + # and a stale edge is exactly what keeps them hidden. + when: + - status: [success, failure] diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml index 2e0f2a0..f38f6d5 100644 --- a/.crow/weekly-rebuild-reindex.yaml +++ b/.crow/weekly-rebuild-reindex.yaml @@ -122,15 +122,6 @@ matrix: ARCH: arm64 R_VERSION: 4.4.3 IMG: ubuntu:noble - - OS: ubuntu-2604 - ARCH: amd64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - - OS: ubuntu-2604 - ARCH: arm64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - steps: - name: 'Re-index the slot' image: reg.devxy.io/rpkgs/build-env-${IMG} From 50495f5c3bb6e3e97d9d42d639c5f9828b6207b6 Mon Sep 17 00:00:00 2001 From: pat-s Date: Fri, 14 Aug 2026 07:01:56 +0000 Subject: [PATCH 11/31] Revert "fix(ci): split oversized weekly rebuild matrix" This reverts commit b75fd2f1c43cf0355e3e2eb9e51d73f16a1321eb. --- .crow/weekly-rebuild-missing-ubuntu-2604.yaml | 140 ------------------ .crow/weekly-rebuild-missing.yaml | 41 ++++- .crow/weekly-rebuild-reindex-ubuntu-2604.yaml | 110 -------------- .crow/weekly-rebuild-reindex.yaml | 9 ++ 4 files changed, 47 insertions(+), 253 deletions(-) delete mode 100644 .crow/weekly-rebuild-missing-ubuntu-2604.yaml delete mode 100644 .crow/weekly-rebuild-reindex-ubuntu-2604.yaml diff --git a/.crow/weekly-rebuild-missing-ubuntu-2604.yaml b/.crow/weekly-rebuild-missing-ubuntu-2604.yaml deleted file mode 100644 index 0082e0e..0000000 --- a/.crow/weekly-rebuild-missing-ubuntu-2604.yaml +++ /dev/null @@ -1,140 +0,0 @@ -# ubuntu-2604 rows split from weekly-rebuild-missing.yaml. -# -# The complete rebuild matrix has 54 permutations, while Crow accepts at most -# 50 per workflow. Keeping these six rows in a companion workflow lets a manual -# "all" run fan out every slot and shard without exceeding that compiler limit. - -variables: - # This is the same gate as the main rebuild workflow. The safe default must - # match no row because every file in .crow/ is evaluated on a manual run. - weekly_rebuild_missing: - description: "Manual run target: a specific -, 'all' to run every os/arch in parallel, or 'none' to run nothing." - options: - - none - - all - - alpine-322-amd64 - - alpine-322-arm64 - - alpine-323-amd64 - - alpine-323-arm64 - - alpine-324-amd64 - - alpine-324-arm64 - - redhat-8-amd64 - - redhat-8-arm64 - - redhat-9-amd64 - - redhat-9-arm64 - - redhat-10-amd64 - - redhat-10-arm64 - - ubuntu-2204-amd64 - - ubuntu-2204-arm64 - - ubuntu-2404-amd64 - - ubuntu-2404-arm64 - - ubuntu-2604-amd64 - - ubuntu-2604-arm64 - default: none - -when: - - event: cron - cron: weekly-rebuild-missing-${OS}-${ARCH} - - event: manual - evaluate: 'weekly_rebuild_missing == "all" || weekly_rebuild_missing == "${OS}-${ARCH}"' - -skip_clone: true - -labels: - group: rpkgs-${ARCH} - -matrix: - include: - - OS: ubuntu-2604 - ARCH: amd64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 1 - - OS: ubuntu-2604 - ARCH: amd64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 2 - - OS: ubuntu-2604 - ARCH: amd64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 3 - - OS: ubuntu-2604 - ARCH: arm64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 1 - - OS: ubuntu-2604 - ARCH: arm64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 2 - - OS: ubuntu-2604 - ARCH: arm64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - SPLIT_INTO: 3 - SPLIT_INDEX: 3 - -steps: - - name: 'Rebuild missing binaries' - image: reg.devxy.io/rpkgs/build-env-${IMG} - pull: true - environment: - OTEL_R_TRACES_EXPORTER: none - OTEL_R_LOGS_EXPORTER: none - OTEL_R_METRICS_EXPORTER: none - RED_HAT_DEV_PW: - from_secret: RED_HAT_DEV_PW - B2_S3_ACCESS_KEY: - from_secret: B2_S3_ACCESS_KEY - B2_S3_SECRET_KEY: - from_secret: B2_S3_SECRET_KEY - PGPASS: - from_secret: PGPASS - REPO_RO_TOKEN: - from_secret: REPO_RO_TOKEN - GITHUB_PAT: - from_secret: GITHUB_PAT - FORGEJO_TOKEN: - from_secret: FORGEJO_TOKEN - GIT_USER: pat-s - UVR_CACHE_DIR: /mnt/cache/uvr/cache - UVR_PACKAGES_DIR: /mnt/cache/uvr/packages - R_LIBS_USER: /mnt/cache/R-pkgs - R_VERSION: ${R_VERSION} - CCACHE_DIR: /mnt/cache/ccache - PLATFORM: ${OS} - ARCH: ${ARCH} - NCPUS: 2 - SPLIT_INTO: ${SPLIT_INTO} - SPLIT_INDEX: ${SPLIT_INDEX} - # Wall clock after which the shard stops cleanly instead of having to be - # killed. A kill matches neither `success` nor `failure`, so it would skip - # the dependent re-index and leave rebuilt binaries behind a stale edge. - REBUILD_BUDGET_HOURS: 20 - commands: - - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - - mkdir -p /mnt/cache/uvr/cache /mnt/cache/uvr/packages /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R - - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh httr2 - - /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")' - - $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX $REBUILD_BUDGET_HOURS 2>&1 - backend_options: - docker: - resources: - requests: - memory: 5Gi - cpu: 3000m - limits: - memory: 18Gi - cpu: 3000m diff --git a/.crow/weekly-rebuild-missing.yaml b/.crow/weekly-rebuild-missing.yaml index aeb2f50..9639792 100644 --- a/.crow/weekly-rebuild-missing.yaml +++ b/.crow/weekly-rebuild-missing.yaml @@ -1,7 +1,5 @@ -# Main weekly-rebuild-missing workflow (all slots except ubuntu-2604). +# Consolidated weekly-rebuild-missing pipeline (all platforms, both arches). # Three matrix rows per OS/arch, one per shard of that slot's rebuild list. -# ubuntu-2604 lives in weekly-rebuild-missing-ubuntu-2604.yaml so a manual -# "all" run stays below Crow's 50-permutation matrix limit. # Routing is preserved 1:1: # - cron: each existing `weekly-rebuild-missing--` cron fires only # its matching matrix rows (via the per-row `cron:` name filter), @@ -348,6 +346,43 @@ matrix: IMG: ubuntu:noble SPLIT_INTO: 3 SPLIT_INDEX: 3 + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 3 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 1 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 2 + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + SPLIT_INTO: 3 + SPLIT_INDEX: 3 + steps: - name: 'Rebuild missing binaries' image: reg.devxy.io/rpkgs/build-env-${IMG} diff --git a/.crow/weekly-rebuild-reindex-ubuntu-2604.yaml b/.crow/weekly-rebuild-reindex-ubuntu-2604.yaml deleted file mode 100644 index 3ab086a..0000000 --- a/.crow/weekly-rebuild-reindex-ubuntu-2604.yaml +++ /dev/null @@ -1,110 +0,0 @@ -# Re-index ubuntu-2604 after its split rebuild workflow finishes. - -variables: - # Mirrors the shared rebuild gate. The default must match no matrix row so - # unrelated manual runs do not rewrite package indexes. - weekly_rebuild_missing: - description: "Manual run target: a specific -, 'all' to run every os/arch in parallel, or 'none' to run nothing." - options: - - none - - all - - alpine-322-amd64 - - alpine-322-arm64 - - alpine-323-amd64 - - alpine-323-arm64 - - alpine-324-amd64 - - alpine-324-arm64 - - redhat-8-amd64 - - redhat-8-arm64 - - redhat-9-amd64 - - redhat-9-arm64 - - redhat-10-amd64 - - redhat-10-arm64 - - ubuntu-2204-amd64 - - ubuntu-2204-arm64 - - ubuntu-2404-amd64 - - ubuntu-2404-arm64 - - ubuntu-2604-amd64 - - ubuntu-2604-arm64 - default: none - -when: - - event: cron - cron: weekly-rebuild-missing-${OS}-${ARCH} - - event: manual - evaluate: 'weekly_rebuild_missing == "all" || weekly_rebuild_missing == "${OS}-${ARCH}"' - -depends_on: - - weekly-rebuild-missing-ubuntu-2604 - -runs_on: [success, failure] - -skip_clone: true - -labels: - group: rpkgs-${ARCH} - -matrix: - include: - - OS: ubuntu-2604 - ARCH: amd64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - - OS: ubuntu-2604 - ARCH: arm64 - R_VERSION: 4.4.3 - IMG: ubuntu:resolute - -steps: - - name: 'Re-index the slot' - image: reg.devxy.io/rpkgs/build-env-${IMG} - pull: true - environment: - OTEL_R_TRACES_EXPORTER: none - OTEL_R_LOGS_EXPORTER: none - OTEL_R_METRICS_EXPORTER: none - RED_HAT_DEV_PW: - from_secret: RED_HAT_DEV_PW - B2_S3_ACCESS_KEY: - from_secret: B2_S3_ACCESS_KEY - B2_S3_SECRET_KEY: - from_secret: B2_S3_SECRET_KEY - REPO_RO_TOKEN: - from_secret: REPO_RO_TOKEN - GIT_USER: pat-s - R_LIBS_USER: /mnt/cache/R-pkgs - R_VERSION: ${R_VERSION} - PLATFORM: ${OS} - ARCH: ${ARCH} - commands: - - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - - mkdir -p /mnt/cache/R-pkgs - - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R - # The codename is detected from the image's /etc/os-release. - - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' - - | - for RBIN in /opt/R/[0-9]*/bin/R; do - RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) - /opt/R/$R_VERSION/bin/R -q -e "library(bincraft); upload_package_index(r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true - done - - - name: Purge CDN cache - image: reg.devxy.io/docker.io/library/alpine:3.24 - environment: - OTEL_R_TRACES_EXPORTER: none - OTEL_R_LOGS_EXPORTER: none - OTEL_R_METRICS_EXPORTER: none - BUNNYNET_API_KEY: - from_secret: BUNNYNET_API_KEY - # cran.rpkgs.com and cran.allianceswisspass.devxy.io are on separate - # Bunny pull zones, so both must be purged after the shared origin changes. - BUNNY_PULLZONES: '3857050 cran.allianceswisspass.devxy.io' - commands: - - apk add --no-cache -q bash curl jq - # Crow carries the checkout from the re-index step into this step. - - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" $BUNNY_PULLZONES - # Run even when the re-index above failed: the objects were still replaced, - # and a stale edge is exactly what keeps them hidden. - when: - - status: [success, failure] diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml index f38f6d5..2e0f2a0 100644 --- a/.crow/weekly-rebuild-reindex.yaml +++ b/.crow/weekly-rebuild-reindex.yaml @@ -122,6 +122,15 @@ matrix: ARCH: arm64 R_VERSION: 4.4.3 IMG: ubuntu:noble + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + steps: - name: 'Re-index the slot' image: reg.devxy.io/rpkgs/build-env-${IMG} From c7b4dca6e2209c1b56932f37843ef586497fe4e9 Mon Sep 17 00:00:00 2001 From: automation-bot Date: Mon, 17 Aug 2026 00:31:56 +0000 Subject: [PATCH 12/31] chore(deps): lock file maintenance --- .terraform.lock.hcl | 62 ++++++++++++++++++++++----------------------- 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/.terraform.lock.hcl b/.terraform.lock.hcl index a21cf43..ff935fa 100644 --- a/.terraform.lock.hcl +++ b/.terraform.lock.hcl @@ -2,38 +2,38 @@ # Manual edits may be lost in future updates. provider "registry.opentofu.org/hashicorp/http" { - version = "3.6.0" + version = "3.6.1" hashes = [ - "h1:0n4RBz9zNw6TTddh5+x7E8L2+qzPXNwKhK4uoZ/DUwE=", - "h1:22Ob7lpzMBSqdrCvoFN5EgmhGPHPBovV/9qo0c/Cd+A=", - "h1:2IRBvmWOYrq/ooaYYn2i86jZb7iIUvlg0KlmOMfDHoQ=", - "h1:5mucXikk4OcW3un3u94QnMx4AB4Wfih+sXeMd5QxSNk=", - "h1:5oU7Zm+2gAVGmxqtJ9E8uTudUkYy/DEn/y3IWphdv4k=", - "h1:5w0R4b1/VSzpqQF1tXXPr/qmaQLPVRXamOmPKWFcTk4=", - "h1:AEVeJr8xGmwad+JUUQ833C3x5d4W+W2szF5DfwxYppw=", - "h1:CPHJ+0zQbS/cX1m55Y90jIOgf1jV3ocUUnqsXAh+9Eg=", - "h1:JPewnGDOJudNer5+ghqwXoaJkfot3QRq9uiEYvo+JHU=", - "h1:QzbluV2vQLxsJYxjpziQCmPndIoJ/UGS4/UHH/GpwUM=", - "h1:TjUNbUdqweRBq/ycQ4ixpNkx5qaYwpXEOn9QCpqNZP8=", - "h1:XNbcODP60ajj21N/OO7af8bBg1ltIsYkq9egn7BYbiY=", - "h1:tgrbgmX7WYQz9G9ncgu7TkpVB+RlLjJA/Rvp9KPlZH8=", - "h1:vLxthX/ZWsOZ+aHKbAMqmNKqD0K5f4nJ8ppy0Ioyup0=", - "h1:wZOdGBAZkY8OKEPjKz82j1HloAKOmmvtjWyTxM+I110=", - "zh:0f719fa5426bc883e9fa6abf7f6498e48025edafbc29015e2f5c028f1cca3b9d", - "zh:1b4d7dafefd6c61764b2f9ed6943ceb9a200dee3590d18747e3a5f6b20ce85e0", - "zh:1d23a712984866d29f7b07028a4e99c783c71f1a5dddf08bc3d4e7da9d91a1fa", - "zh:257d23d58c3bb024b6bc8eb88736eaf912e934ad47c639d0c3c742bddda849a1", - "zh:479860e1a5468f5e04013b9364c9496d7ed0804bf9a1acd8e07558d57609993d", - "zh:4cb5e681bf599b411b27c4a2c4066a5fb2ed79aaa3a1a3cb5a30002fec062ce9", - "zh:4fb35c3f643dae9f3670d719397a415f815a0b95f8ed7bd8a72f27a94ba78092", - "zh:59ba40825ab38db5b4a0989a2db0df35cc15d8984f898176011ba352f27d77b7", - "zh:61fc1252eb88088638f4c69ea4e2171cde2e5089fa632ac1e943b13787348f73", - "zh:7c5d6dd5f7cbc460e95d368be35c29b4e0402069b8912dbd5d1cd7fa9acef216", - "zh:7f76d756240d4284642f359ad470226e5378670239aadc366ef54d9d914d4d2e", - "zh:8133ad0814098177e0d067c816ccf1bf48bbadacd18f6f2c808c90447505723b", - "zh:c93be06269bb728f1968f8c50506de56c887017ac1d6e4be1f925651d8437eb6", - "zh:ef47b78a10a82e6cf53344a6a85a94041c28286c10a70541c564d762f1cfede0", - "zh:f5796a53a74999135bd9087aff50fddda59129d09b2f9b1902ff8c0c1e047e48", + "h1:7fra+jbUXbG5wMaz5L6RKMBv6gIuenJcBiIww87GoXo=", + "h1:BzSV3Ie9XMXF7sZHKAS54CzV95v5GBZNhQ4nrprUgfQ=", + "h1:CkrbSKS+pNVgvP3bMe2WoYHaFCIWJUkCtlC5vyTAdLI=", + "h1:FboJEwgVIRmqUJkjEoSRpfavVCJotUTe1zzT+pBzcV0=", + "h1:GlXELDLSZrdV3Svx1jjEBAXiJFkkdF/Hgx1qrmRK5hE=", + "h1:VuXFI2IcnZ6t4sDqtvkuIzbPK1CJQa0CkaM0MBuOlSU=", + "h1:WmL2nFQbSzRiDsDiwUbZbBp/cxGQrXrZnB7A4LGSvJU=", + "h1:Zdj26awWJ+m8kMoAMhItsIDcDFg81PWgKKJrvNi3WOI=", + "h1:lHvYYIumeZ+KJgCrmhCLnRGzrvNMjSHBTdV24coyMEc=", + "h1:pAOYMwA6Zki3ujAbG20b49u1IYXdBz56pW1JHqKdX5U=", + "h1:qi9GUp2+g69C8zY6Z68u4fWPwcZlDTa/CtdhvPgWbMA=", + "h1:w5A3xJ2mowj2wgiE3oNfOI0lFJf5X9IgxOJ6SErMczA=", + "h1:xAO03iJyuNGSOqolIcXcofH8cocgUb6Cnzq6yivbWcI=", + "h1:xXigGPwW8MlrB6Br2ce+Bf35BbdzdPKa97T/q/xrrcA=", + "h1:yDYzQ2ncNE9q1288xAgflIPq98bOOYsAb9tq6vkbFzw=", + "zh:129d7d5944b31f40916b1ca86b31cef65a6b02fd36008809d13c561894bfedb9", + "zh:24631608288b0bcd35c1fc63dc5839572254d881c0589ebba036be52b2fc04d6", + "zh:5a0f100d7eb256463fe5a2aa1a7128391147b2c5fc895ff1b1ef54fc5b8f15ab", + "zh:6a8a1126ab9ca61be3b62ec184f6b2e7cbf01cde810acc548cee27d71277b09b", + "zh:6fffef54fd3aada85c074e34d41386aa09c79a308a4679132da31c7272733c6c", + "zh:899c992d2aa290ebe1304da0289c5104a630bca421cc6a88ce55bf0960aab1b4", + "zh:960fd6c2847859a843dd9dbfc95a0037a470aa744094d155a38a057175cf1502", + "zh:9b032b685a644634158ace5529e260dfc4447a280056f02858d205ea26753f69", + "zh:bba5477c97020c28ed12d4f5b36be2c1bf14d946d7e44b3690e5c23cd7ddf5e6", + "zh:c2ff6c33efef52441fa3485137972792031626dcabca2b1d8b6527d45f185279", + "zh:cd492b3dfd150de6bef8ad505293d3d53c6c907706f36d0e497b4fc027d8edb6", + "zh:d1f832bc33c42781454dc020c6937e7d0133155a5a9f64335309d64a34b36bb7", + "zh:d42e9cbebc77643556853b1ebbec14cefe70c57ee86cd3b8c71fbe7f523f07df", + "zh:d4c0466f578d7f990646bb0847e31ba3797f2100b6380ee1ca736887546c7621", + "zh:d9d81ecebfe6edabdd4c527f3f4debde3e052ff87c5ef4c67497ab3d7539e424", ] } From d1da0c6cb46447d9b295389e4cda211d014d10fb Mon Sep 17 00:00:00 2001 From: automation-bot Date: Sat, 22 Aug 2026 00:32:09 +0000 Subject: [PATCH 13/31] chore(deps): update terraform bunnynet to v0.18.1 --- .terraform.lock.hcl | 70 ++++++++++++++++++++++----------------------- 1 file changed, 35 insertions(+), 35 deletions(-) diff --git a/.terraform.lock.hcl b/.terraform.lock.hcl index ff935fa..ef9e47d 100644 --- a/.terraform.lock.hcl +++ b/.terraform.lock.hcl @@ -38,43 +38,43 @@ provider "registry.opentofu.org/hashicorp/http" { } provider "registry.terraform.io/bunnyway/bunnynet" { - version = "0.18.0" + version = "0.18.1" constraints = "~> 0.18" hashes = [ - "h1:+6VXZmSSeIchab8WY+UFZxaXu6poxZbiJenYwog+Z2M=", - "h1:Bz94gADR83NmrsZC50LheryVcS9erG6tF052hNPqz0U=", - "h1:FGkcn6ieyNPr56Riv75IsCa6okZ9ZHqqFd08Y4i5cPI=", - "h1:GqPjTKaOlhhNIxpPHWJpWdGkEtvUDtuOkjYbVIInfgU=", - "h1:NMDWYRisSFwepjfYsf4MrcyP7Ihjlty937l1BW1Cztg=", - "h1:OusNMAZdIIbxWwHXwH8tYzdbXZPRS8dW7cncSYcHfNI=", - "h1:OzgasS4oZCZjYAOrqGy7RpPNbTPX5wGaaw7TDnasSJM=", - "h1:RyK3DC7cM4T2I83OvnO+UXU+eCXLG5vDlYZ/bWKHq7g=", - "h1:U/JKg4BNWii2mK3bNsOWYMBTpkXqGsWb1ypSJvSKl9Y=", - "h1:UjSUm1AU2wZ8eyC3LDRwWatW9cKD/I0rk7MOeUDDkWc=", - "h1:a888ExTeqWKxaB2GBsDIRSm8jOOMLkPFcABrJxxA8qc=", - "h1:aShYLfSapfo1Ozy33zL5WfmYt5UF2m5EziespPOGNAk=", - "h1:bHdpp2ecmvDoezqg8TirE32SnuDb108/NZ4xBGvJy98=", - "h1:lLYZN5cetXgLPJjo+eR+kwodOYGsQetzV6bg5Ki58tk=", - "h1:szi71DaqI4yB6fI55hBbtVZg1GDWJVkO9+WF/1sZ+IE=", - "h1:vK4jZYZZhD6M4cRmX+171mxoJgb2FSNiBNAUhtm/TNs=", - "h1:yNnSUskn43648XU/YS0e9NUdoHImo7pFMIlBzujBoxY=", - "zh:0c3adf039df2fead1e36b8e7887d965223f03b6b5cdd921ec0c98beaa04fdec0", - "zh:12a3db29733e6619e216fac9aa774cbf67da2105f48c5c4dee8df046d69656fa", - "zh:1330b83c949165434e4d0db5e097a1741376cdb66c06c37b2a97ecea8e06b0f0", - "zh:186113259ab0f80ab9079f0a8a810e93ee097f6740100a0a444f805e8772c12e", - "zh:242fe10185da8a700b0e9a6a52e3d1f6592e8e189aaf5e9a79cf515ad5c7ff25", - "zh:28b005b0aa9485c492326fbb4287d3cb465f25a8d572b8c740e041e040d80aa2", - "zh:3b2cd0daa767dfff67dd39c98d1c82b25e70eb81df77c0d6dd762a0dedec16e1", - "zh:3f14b244740f6c6420d298846708e9f3853c8eda685d3a4d71fb0ffec021ad9c", - "zh:469bfd08deefd90e89c930e5e45bfd06a58a1eadb85dfbee9fc48ae296e8a8c9", - "zh:634d96b84b2e77d25baa074c357e831252c8b8114926ef72e1afb68a4bc9eb65", - "zh:6e837b9485e56f84b5e2bb396fa549b628b8e075257fa01bf8c796ce91d20e58", + "h1:1nbjHMc5QBwiAfkriGGuO+wLkrgwKjbeAK70sNQRDa4=", + "h1:2AkMQZEckeGl8efzFzpGUqIR5o4rhOt4GSmOOMBX8wg=", + "h1:4D42uIgbm5jI0TQWRfVXJZoG6WOyO3Mi0ZgC8KZjPmc=", + "h1:79iaWeho9vn6p5oFHx7cXCzrX9k0R2yS9iF7rNchX7E=", + "h1:7GKjnlsRnA88qGIZ+V4HseEWB1YmuPXkyVen4DOxRBM=", + "h1:9ZpZHafhWNCDM91hE+GK6ikvy9THlU4/opQoC78B6TU=", + "h1:J4eKuqfDRI+e1PRGkt6MJ6UT+ym3pFxYKN+jIm105Es=", + "h1:LvrSDB8WWxh6NcR7X97oxJ9FrB/UVyWau9V+rdxBkxU=", + "h1:NO36kn/RhmKhWVcM0qRUcbZveSv2idT6LUwOREBqBc4=", + "h1:UPrn5yfuJwaEggnIbKqd9eyNCzO9DRJhA0279J1LYLM=", + "h1:b+tjNcFfxqwWGDgumzm4QImqJLyemGanHbZ+F2qw6vU=", + "h1:gppmt6Jbng9QH8ulpSujYeCHSlC5kWEkYoiYzgFISNQ=", + "h1:kZkf+9F8Be+Ztt10sLUWIxnwgfBCHpxi3V66F3+HV5s=", + "h1:mZvRXynxrx0/Q0txogt/5fTVhKmRWX3rWoqLiwLdamw=", + "h1:ogkHPOIbdDgdmfoa0LaPIyX8QTJvWik3andNAJoR6pk=", + "h1:pCkHlvcWgM5FkwEsrxzT9L7S7ZDNfyWagWM0KMXYFBE=", + "h1:rbKmEiaOKwrbrzEf9iihpwFCwvlVVkyxZUfFeVuX180=", + "zh:198f5aab9e8bbbb6fa96e41b7d33a997e72666dfc7369f08849ff12a0f91f7d4", + "zh:1f152ab9c51353422a79d4c4ee965112972b0d4b3246c62a8e6848422f4cac26", + "zh:4776e4fc6f38b1eb4a64c866e617cebdb0344af51c4685aa3e47356a5502707f", + "zh:66828334af0bdecbde3c5913b9c48371ea119569631f43094e28e03ab1b5fa4a", + "zh:800a94528ed366242fb83ba8d22ca1407bb02826ac8805a83a0ab04ab5efb582", + "zh:862f5e7db0f81d5e57292f0cd44b4f530eaa59510756991037d1ed6fbff91e05", "zh:890df766e9b839623b1f0437355032a3c006226a6c200cd911e15ee1a9014e9f", - "zh:89e02979311e6727a50586b04cb343e9bbec74a6f4d1dba9352971e4334d64c9", - "zh:a6796143fe61ae52d236b7ea96bddbc6317851a0e2117eb2f55cd73c803b4b6f", - "zh:ad2542160e7b57ee8a016cddbf1d32663baf74fe30fcf24df64da75cb13128b0", - "zh:bfee81c153e8b50121fd58eedeeb84bfa64a3d17f23900e35143c4a86474cbd2", - "zh:eeba652697908712cd1ce2c7d2925f6e6bb2d18ea5d3e6ca185e88eeb80f3c80", - "zh:f0b5b8fd6942647c358440591d56a5cb82fda92f7866898654a66ec901174f1f", + "zh:91f8b0f92b7dd1e131e391c170fcb329c795747553d5cc71296b8803496bc33c", + "zh:95446852691ebbbfcd4998f8941d29577d61b15ac320bc4590ae48f4542aee39", + "zh:9ccb3382c4fc20735c3ee016240afa82d6cf745b8ec458a9e5159d1a03e4cd20", + "zh:a11991312edc9fe2567ecedadb2e87de8b1a41e97a0d117d1a1c58eaa68ae645", + "zh:ab729171af18063c34f5a7d33b598b870e082f2e7ba8ee60ca77dc4dede84432", + "zh:c2b58950da56179b93a7d590d0cf3a488070b973826d788401a94914eaeae072", + "zh:d1b6a91581f1580fad12f11b9593461f566153b6760299ba592746716d595583", + "zh:dad0ce1ac0fc934ab18fe4a744b768d8f9969099488b25d6c18c782b40478bfe", + "zh:e2cb8dfebfa0a358d0b52e15b04cfd82407a4c578c39b6ac62ac919b23ef97a6", + "zh:fc648d9464bcb6b360eb4885791675c2437f0d1c0e1b1b8f34fbb6d93a880ea9", + "zh:fd8250944e3794b9440cdaf4ac06f7a1672744e4bcb2e8a33b94c2ddf32ab988", ] } From f4ab6f9dc55d2e33822abf5d48e6517a33bd46e9 Mon Sep 17 00:00:00 2001 From: automation-bot Date: Wed, 26 Aug 2026 00:32:02 +0000 Subject: [PATCH 14/31] chore(deps): update pre-commit hook editorconfig-checker/editorconfig-checker to v3.11.2 --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 82461cb..2bf38b9 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -36,7 +36,7 @@ repos: hooks: - id: air-format - repo: https://github.com/editorconfig-checker/editorconfig-checker - rev: v3.11.1 + rev: v3.11.2 hooks: - id: editorconfig-checker exclude: ^local/patches/.*\.patch$ From d2333c6cbe599fc5e43c7534d55bb2120e104245 Mon Sep 17 00:00:00 2001 From: automation-bot Date: Thu, 27 Aug 2026 00:33:08 +0000 Subject: [PATCH 15/31] chore(deps): update terraform bunnynet to v0.18.2 --- .terraform.lock.hcl | 70 ++++++++++++++++++++++----------------------- 1 file changed, 35 insertions(+), 35 deletions(-) diff --git a/.terraform.lock.hcl b/.terraform.lock.hcl index ef9e47d..9591b83 100644 --- a/.terraform.lock.hcl +++ b/.terraform.lock.hcl @@ -38,43 +38,43 @@ provider "registry.opentofu.org/hashicorp/http" { } provider "registry.terraform.io/bunnyway/bunnynet" { - version = "0.18.1" + version = "0.18.2" constraints = "~> 0.18" hashes = [ - "h1:1nbjHMc5QBwiAfkriGGuO+wLkrgwKjbeAK70sNQRDa4=", - "h1:2AkMQZEckeGl8efzFzpGUqIR5o4rhOt4GSmOOMBX8wg=", - "h1:4D42uIgbm5jI0TQWRfVXJZoG6WOyO3Mi0ZgC8KZjPmc=", - "h1:79iaWeho9vn6p5oFHx7cXCzrX9k0R2yS9iF7rNchX7E=", - "h1:7GKjnlsRnA88qGIZ+V4HseEWB1YmuPXkyVen4DOxRBM=", - "h1:9ZpZHafhWNCDM91hE+GK6ikvy9THlU4/opQoC78B6TU=", - "h1:J4eKuqfDRI+e1PRGkt6MJ6UT+ym3pFxYKN+jIm105Es=", - "h1:LvrSDB8WWxh6NcR7X97oxJ9FrB/UVyWau9V+rdxBkxU=", - "h1:NO36kn/RhmKhWVcM0qRUcbZveSv2idT6LUwOREBqBc4=", - "h1:UPrn5yfuJwaEggnIbKqd9eyNCzO9DRJhA0279J1LYLM=", - "h1:b+tjNcFfxqwWGDgumzm4QImqJLyemGanHbZ+F2qw6vU=", - "h1:gppmt6Jbng9QH8ulpSujYeCHSlC5kWEkYoiYzgFISNQ=", - "h1:kZkf+9F8Be+Ztt10sLUWIxnwgfBCHpxi3V66F3+HV5s=", - "h1:mZvRXynxrx0/Q0txogt/5fTVhKmRWX3rWoqLiwLdamw=", - "h1:ogkHPOIbdDgdmfoa0LaPIyX8QTJvWik3andNAJoR6pk=", - "h1:pCkHlvcWgM5FkwEsrxzT9L7S7ZDNfyWagWM0KMXYFBE=", - "h1:rbKmEiaOKwrbrzEf9iihpwFCwvlVVkyxZUfFeVuX180=", - "zh:198f5aab9e8bbbb6fa96e41b7d33a997e72666dfc7369f08849ff12a0f91f7d4", - "zh:1f152ab9c51353422a79d4c4ee965112972b0d4b3246c62a8e6848422f4cac26", - "zh:4776e4fc6f38b1eb4a64c866e617cebdb0344af51c4685aa3e47356a5502707f", - "zh:66828334af0bdecbde3c5913b9c48371ea119569631f43094e28e03ab1b5fa4a", - "zh:800a94528ed366242fb83ba8d22ca1407bb02826ac8805a83a0ab04ab5efb582", - "zh:862f5e7db0f81d5e57292f0cd44b4f530eaa59510756991037d1ed6fbff91e05", + "h1:3rZl+Co3WMpwj8SciPaCNXoGA31aSoqp6iweLarr5m4=", + "h1:6d9cKLhz8QOZ4R5yVX1G0TsWL+K1Abtfbm3xngndxto=", + "h1:EBjjkfp5Gx7nXP1DVO+tLhsow6fEUvaIjsCEFRT2fY8=", + "h1:Nu2DoHGOv2YN7ag4kFGpfnPeRDh6bzWqY5anW+ETGpM=", + "h1:OnvZxg28m4/UJeEhHVLU4kM2MZ704sxRzYfLWlLxnhA=", + "h1:PiCse2/UcB7nkPxosveHsJN/jKdBC8AH6tKTxcHSYKw=", + "h1:QAahdtlDBUon7eMwNN0D2V6CxgasOXIi+9/UExik6Sg=", + "h1:Su5z0A7/UaSm/E7FJnFjpDVQaa1Ju5+fZ8Mirf8E+k8=", + "h1:UA3a78FJAPAGqCCvlIg9ekPltpVsrmEhwFLalWCFnew=", + "h1:XAlCTNHRtgUkNjdUItkiak6ajjT7wFJzJN8frXKD5Ms=", + "h1:ZgLBOPebYxH059z1cGHmjYO8CTf+tbWPb3VbO97S2YM=", + "h1:anR91C2F6NDJoQQQIy6KHChodnTaSKnApSWSGM4jSX0=", + "h1:gVmaNmIu4gEiITM+CAb66e+zncAqzNBYkniTZfvxZ5Y=", + "h1:pODlGrkPqHV4yhXiO7LLLu11HtcuxOAB2zUx3B8w1vI=", + "h1:qEYeHEKVRcc78q5xiRGJSY8DGQpLj40KafEXUxFfaQc=", + "h1:qdVz+O0lLHhyf5YX3ujmoVvAGlKqvi+YOPUzVTqpKzY=", + "h1:yTrPkdc9eQkxfPLBYydFf0fpcjarP5w0sdLPzekD9RQ=", + "zh:0fe3987c927d81196c97504470ce4d26c3ad0014f8ee3d0c1be422d08cfcf49c", + "zh:15c36dc69e058876921ac887213e1716217d159b7ee7f0f233e21fb35be85178", + "zh:29d58d7b76dcb142a06d4edd15b8500fe6c1afb7f7c056ada17e2d42bb999fbd", + "zh:33d313836c0e985186b3456c0946e062b27cacfcb08611d0a394f36db9ee1aef", + "zh:47e085e52e9b24ad85fa2988dbb8604256a970a6f53f7fa6aab04d8ae756a738", + "zh:4ba4f87571ca72fbc6c24ab71f2f7b5a086938262e2d8e5c0b39701ed52f8bbc", + "zh:4c6bae97b543c5b328e1ecbcf7c976351b4b381654e9d3e569270dcab3ba816c", "zh:890df766e9b839623b1f0437355032a3c006226a6c200cd911e15ee1a9014e9f", - "zh:91f8b0f92b7dd1e131e391c170fcb329c795747553d5cc71296b8803496bc33c", - "zh:95446852691ebbbfcd4998f8941d29577d61b15ac320bc4590ae48f4542aee39", - "zh:9ccb3382c4fc20735c3ee016240afa82d6cf745b8ec458a9e5159d1a03e4cd20", - "zh:a11991312edc9fe2567ecedadb2e87de8b1a41e97a0d117d1a1c58eaa68ae645", - "zh:ab729171af18063c34f5a7d33b598b870e082f2e7ba8ee60ca77dc4dede84432", - "zh:c2b58950da56179b93a7d590d0cf3a488070b973826d788401a94914eaeae072", - "zh:d1b6a91581f1580fad12f11b9593461f566153b6760299ba592746716d595583", - "zh:dad0ce1ac0fc934ab18fe4a744b768d8f9969099488b25d6c18c782b40478bfe", - "zh:e2cb8dfebfa0a358d0b52e15b04cfd82407a4c578c39b6ac62ac919b23ef97a6", - "zh:fc648d9464bcb6b360eb4885791675c2437f0d1c0e1b1b8f34fbb6d93a880ea9", - "zh:fd8250944e3794b9440cdaf4ac06f7a1672744e4bcb2e8a33b94c2ddf32ab988", + "zh:9ba7ab56537963db2449d217528a751469c9dc4e413dec3e3d63fd7daf3db4ef", + "zh:a3c48eda7e11b03b831f2a639797524bb335f155f0dff0e999cf3496994da8b3", + "zh:aab8f4814d55ef8c6c285d2496ae412437017d0fd1be70106f7b3a4a6e764feb", + "zh:b92b9beacf71ae894717c2036ceb68db52c9c43af4a01b8209eceae9f91a2c8e", + "zh:da389285938e22e1249e6a00cebf12a9f67334743f0b3f66399e6881028bda11", + "zh:dadcc33d06e6f64a17d1965478af5e8bbdc971e92ec9b14e384c5d43861d63f7", + "zh:e090c916e6da685125194af4f0a1fd772494a0c63f3f16ab3741782e17f4a8f9", + "zh:e5881e00fa970c08e66e8079b47d69b76def6e7ff3bdc35b68d7811e5ece55d1", + "zh:eeebb25a066a6287d545c91c0fc264acee5b28174d0979faeebdac3bd14f0fff", + "zh:f368195116c9ce0181aa7527c51ae5e7ab23d42fb966acf4eddca344621ae339", ] } From e9782bb529378d2b86e1bf87b1a32d49a1618ea8 Mon Sep 17 00:00:00 2001 From: pat-s Date: Fri, 28 Aug 2026 08:48:58 +0000 Subject: [PATCH 16/31] fix(ci): install RPostgres for metadata updates --- .crow/process-updates.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.crow/process-updates.yaml b/.crow/process-updates.yaml index 021d6cb..73bfff7 100644 --- a/.crow/process-updates.yaml +++ b/.crow/process-updates.yaml @@ -203,6 +203,7 @@ steps: - rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/bincraft - mkdir -p /mnt/cache/uvr/cache /mnt/cache/uvr/packages /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R + - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh RPostgres - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' # rhel-10 ships xwfb-run (Xwayland) instead of xvfb-run; prefer it and start weston, else fall back to xvfb-run - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi @@ -218,6 +219,7 @@ steps: LIB="/mnt/cache/R-pkgs-$RMINOR" mkdir -p "$LIB" R_LIBS_USER="$LIB" "$(dirname "$RBIN")/Rscript" local/install-bincraft.R || true + R_LIBS_USER="$LIB" UVR_R_BIN="$RBIN" local/uvr-install.sh RPostgres || true R_LIBS_USER="$LIB" $XVFB $XVFB_ARGS -- "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '${OS}', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, patches = 'local/patches', r_minor_detection = 'classifier',r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "${OS_ID}", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' From 77a2f1f04a645f5001755c477c34e285e73215e0 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sun, 30 Aug 2026 07:31:28 +0000 Subject: [PATCH 17/31] fix(ci): install RPostgres for audit workflows (#172) ## Summary - Install RPostgres before running the missing-binaries audit. - Install RPostgres before running weekly patch proposal and automatic patch workflows. ## Validation - `prek run --files .crow/auto-apply-patches.yaml .crow/weekly-audit-missing.yaml .crow/weekly-patch-proposals.yaml` - `crow lint .crow/` - `git diff --check` Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/172 --- .crow/auto-apply-patches.yaml | 2 +- .crow/weekly-audit-missing.yaml | 2 +- .crow/weekly-patch-proposals.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.crow/auto-apply-patches.yaml b/.crow/auto-apply-patches.yaml index 7c33c5b..5d6e6fe 100644 --- a/.crow/auto-apply-patches.yaml +++ b/.crow/auto-apply-patches.yaml @@ -58,7 +58,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/R-pkgs - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh httr2 jsonlite + - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh RPostgres httr2 jsonlite - /opt/R/$R_VERSION/bin/Rscript local/propose-patches.R --open-pr --limit $PATCH_LIMIT backend_options: kubernetes: diff --git a/.crow/weekly-audit-missing.yaml b/.crow/weekly-audit-missing.yaml index 428aa04..25875da 100644 --- a/.crow/weekly-audit-missing.yaml +++ b/.crow/weekly-audit-missing.yaml @@ -147,7 +147,7 @@ steps: - mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs - rm -rf /mnt/cache/R-pkgs/00LOCK-* - /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R - - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh httr2 jsonlite + - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh RPostgres httr2 jsonlite - /opt/R/$R_VERSION/bin/R -q -e 'source("local/weekly-missing-binaries-audit.R")' backend_options: docker: diff --git a/.crow/weekly-patch-proposals.yaml b/.crow/weekly-patch-proposals.yaml index 712e87d..7ea6c97 100644 --- a/.crow/weekly-patch-proposals.yaml +++ b/.crow/weekly-patch-proposals.yaml @@ -53,7 +53,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/R-pkgs - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh httr2 jsonlite + - UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh RPostgres httr2 jsonlite - /opt/R/$R_VERSION/bin/Rscript local/propose-patches.R --open-issue - /opt/R/$R_VERSION/bin/Rscript local/proposal-tracking.R --open-issue backend_options: From a8820e6e90a14469d4d566b3ee295aaeb5c51eff Mon Sep 17 00:00:00 2001 From: pat-s Date: Sun, 30 Aug 2026 12:19:52 +0000 Subject: [PATCH 18/31] fix(cdn): route Ubuntu 26.04 to resolute (#173) ## Motivation The generic `https://cran.rpkgs.com/` repository falls back to upstream CRAN on Ubuntu 26.04 because the edge router does not recognize its release version. This causes Resolute users to download and compile source packages even when matching binaries exist. ## Changes - Map Ubuntu 26.04 user agents to the `resolute` repository slot. - Cover ARM64 Ubuntu 26.04 routing with an edge middleware regression test. ## Verification - `just edge-test`, 1 test with 15 steps passed. - Confirmed the current live generic route redirects the Resolute user agent to upstream CRAN. - Confirmed the explicit `amd64/resolute/latest` repository installs `rlang` as a binary in `reg.devxy.io/r/r-ubuntu:4.6-resolute`. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/173 --- edge/rpkgs-router.test.ts | 9 +++++++++ edge/rpkgs-router.ts | 1 + 2 files changed, 10 insertions(+) diff --git a/edge/rpkgs-router.test.ts b/edge/rpkgs-router.test.ts index 9493f8a..e9bbcc9 100644 --- a/edge/rpkgs-router.test.ts +++ b/edge/rpkgs-router.test.ts @@ -18,6 +18,7 @@ const UNION_SLOTS = 'amd64/alpine324'; const UA_R45_MUSL = 'R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl)'; const UA_R46_MUSL = 'R (4.6.0 x86_64-pc-linux-musl x86_64 linux-musl)'; const UA_R45_ALPINE = 'R/4.5.3 R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl) Alpine Linux 3.24'; +const UA_R45_RESOLUTE = 'R/4.5.3 (Ubuntu 26.04) (aarch64-unknown-linux-gnu aarch64 linux-gnu)'; const UA_R45_DARWIN = 'R (4.5.1 aarch64-apple-darwin20 aarch64 darwin20)'; const UA_CURL = 'curl/8.0.1'; @@ -141,6 +142,14 @@ Deno.test('rpkgs-router', async (t) => { assertEquals(res.location, `https://cran.rpkgs.com${SLOT}/4.5/PACKAGES.gz`); }); + await t.step('resolves Ubuntu 26.04 to the resolute slot', async () => { + const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_RESOLUTE); + assertEquals( + res.location, + 'https://cran.rpkgs.com/arm64/resolute/latest/src/contrib/PACKAGES.gz', + ); + }); + await t.step('sends an unidentifiable distro to CRAN', async () => { const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_MUSL); assertEquals(res.location, 'https://cran.r-project.org/src/contrib/PACKAGES.gz'); diff --git a/edge/rpkgs-router.ts b/edge/rpkgs-router.ts index 9278d4a..c9195a4 100644 --- a/edge/rpkgs-router.ts +++ b/edge/rpkgs-router.ts @@ -59,6 +59,7 @@ const RHEL_REGEX = /(almalinux|rocky)[^\d]*(\d+)/i; const UBUNTU_REGEX = /Ubuntu ([\d.]+)/i; const UBUNTU_CODENAMES: Record = { + '26.04': 'resolute', '24.04': 'noble', '22.04': 'jammy', }; From b0d58f3f7c7f3577727be25da9368af3f45b7eff Mon Sep 17 00:00:00 2001 From: pat-s Date: Sun, 30 Aug 2026 12:25:22 +0000 Subject: [PATCH 19/31] feat(cdn): derive Ubuntu routes from codenames (#174) ## Motivation Ubuntu release codenames cannot be derived from version numbers, so the edge router currently needs a code change for every Ubuntu release. The generic repository setup can already read `VERSION_CODENAME` from `/etc/os-release` and send it in the user agent. ## Changes - Prefer a strictly validated `codename=` token for Ubuntu slot routing. - Retain the existing version-to-codename table for clients using the previous user-agent format. - Add a future Ubuntu 28.04 regression case whose codename does not exist in the middleware table. ## User-agent format `R/4.6.1 (Ubuntu 26.04; codename=resolute) (aarch64-unknown-linux-gnu)` ## Verification - `just edge-test`, 1 test with 16 steps passed. - `git diff --check` Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/174 --- edge/rpkgs-router.test.ts | 10 ++++++++++ edge/rpkgs-router.ts | 6 ++++++ 2 files changed, 16 insertions(+) diff --git a/edge/rpkgs-router.test.ts b/edge/rpkgs-router.test.ts index e9bbcc9..3020b2d 100644 --- a/edge/rpkgs-router.test.ts +++ b/edge/rpkgs-router.test.ts @@ -19,6 +19,8 @@ const UA_R45_MUSL = 'R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl)'; const UA_R46_MUSL = 'R (4.6.0 x86_64-pc-linux-musl x86_64 linux-musl)'; const UA_R45_ALPINE = 'R/4.5.3 R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl) Alpine Linux 3.24'; const UA_R45_RESOLUTE = 'R/4.5.3 (Ubuntu 26.04) (aarch64-unknown-linux-gnu aarch64 linux-gnu)'; +const UA_R45_FUTURE_UBUNTU = + 'R/4.5.3 (Ubuntu 28.04; codename=dynamic-dugong) (aarch64-unknown-linux-gnu aarch64 linux-gnu)'; const UA_R45_DARWIN = 'R (4.5.1 aarch64-apple-darwin20 aarch64 darwin20)'; const UA_CURL = 'curl/8.0.1'; @@ -150,6 +152,14 @@ Deno.test('rpkgs-router', async (t) => { ); }); + await t.step('resolves a future Ubuntu release from its codename', async () => { + const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_FUTURE_UBUNTU); + assertEquals( + res.location, + 'https://cran.rpkgs.com/arm64/dynamic-dugong/latest/src/contrib/PACKAGES.gz', + ); + }); + await t.step('sends an unidentifiable distro to CRAN', async () => { const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_MUSL); assertEquals(res.location, 'https://cran.r-project.org/src/contrib/PACKAGES.gz'); diff --git a/edge/rpkgs-router.ts b/edge/rpkgs-router.ts index c9195a4..cdeac44 100644 --- a/edge/rpkgs-router.ts +++ b/edge/rpkgs-router.ts @@ -58,6 +58,7 @@ const MACOS_BIN_REGEX = const RHEL_REGEX = /(almalinux|rocky)[^\d]*(\d+)/i; const UBUNTU_REGEX = /Ubuntu ([\d.]+)/i; +const UBUNTU_CODENAME_REGEX = /Ubuntu [\d.]+;\s*codename=([a-z][a-z0-9-]*)/i; const UBUNTU_CODENAMES: Record = { '26.04': 'resolute', '24.04': 'noble', @@ -137,6 +138,11 @@ function parseSlot(userAgent: string): string | null { const ubuntu = userAgent.match(UBUNTU_REGEX); if (ubuntu) { + const codenameMatch = userAgent.match(UBUNTU_CODENAME_REGEX); + if (codenameMatch) { + return `${arch}/${codenameMatch[1].toLowerCase()}`; + } + const codename = UBUNTU_CODENAMES[ubuntu[1]]; if (codename) { return `${arch}/${codename}`; From eeebef8edb8fb166cfb8125c71a690340560dfa3 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sun, 30 Aug 2026 14:26:37 +0000 Subject: [PATCH 20/31] fix(patches): support RcppParallel 6.2.1 This change will:\n\n- Relax the TBB include hunk context to tolerate the upstream TBB_CXXFLAGS block.\n- Pin the source patch to the verified RcppParallel version. --- local/patches/RcppParallel/force-bundled-tbb.patch | 6 +----- local/patches/registry.json | 2 +- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/local/patches/RcppParallel/force-bundled-tbb.patch b/local/patches/RcppParallel/force-bundled-tbb.patch index c46a05e..5c37fa6 100644 --- a/local/patches/RcppParallel/force-bundled-tbb.patch +++ b/local/patches/RcppParallel/force-bundled-tbb.patch @@ -40,15 +40,11 @@ index 6f6a745..e407986 100644 if (is.null(name)) return(tbbRoot) -@@ -58,7 +58,7 @@ tbbCxxFlags <- function() { - flags <- c("-DRCPP_PARALLEL_USE_TBB=1") - +@@ -58,3 +58,3 @@ tbbCxxFlags <- function() { # if TBB_INC is set, apply those library paths - tbbInc <- Sys.getenv("TBB_INC", unset = TBB_INC) + tbbInc <- bincraftGetenv("TBB_INC", unset = TBB_INC) if (!file.exists(tbbInc)) { - tbbInc <- system.file("include", package = "RcppParallel") - } @@ -117,7 +117,7 @@ tbbLdFlags <- function() { } diff --git a/local/patches/registry.json b/local/patches/registry.json index a7e29dd..2457304 100644 --- a/local/patches/registry.json +++ b/local/patches/registry.json @@ -1,7 +1,7 @@ [ { "package": "RcppParallel", - "versions": ">=6.0.0", + "versions": "6.2.1", "platforms": ["*"], "env": {}, "configure_args": [], From aba2063ea0ce8838239a1307a751b1e655916272 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sun, 30 Aug 2026 21:20:16 +0000 Subject: [PATCH 21/31] feat(edge): gate per-minor routing on published minors and add a staging zone (#175) ## Motivation `UNION_SLOTS` is empty, so per-minor routing has never been exercised end to end. Before it can be enabled and advertised, two things were missing: a way to test it without pointing production at it, and evidence that the published indexes actually support it. Verifying the data first turned up a defect that would have broken users the moment the flag was flipped. ## The defect `contribPath()` redirects to `contrib//` whenever the User-Agent carries any R minor, with no existence check and no fallback: ```ts const rMinor = extractRMinor(userAgent); return rMinor ? `/${slot}/latest/src/contrib/${rMinor}/${rest}` : flat; ``` Only `4.4`, `4.5` and `4.6` are published. `4.3` and `4.2` return 404 on all 16 slots. With `UNION_SLOTS` set, an R 4.3 client would be redirected to a non-existent index and see **zero** packages: a silent, total failure rather than a degraded one. R 4.3 is still advertised as supported on the website and in `docs/configuration.mdoc`, though `build-env-images` now pins only 4.6.0/4.5.3/4.4.3. ## Changes - **Gate routing on `KNOWN_MINORS`** (default `4.4,4.5,4.6`), falling back to the flat index for anything else. Unknown minor now behaves exactly as today. - **Honour `EXTRA_PUBLIC_HOSTS`.** `publicCdnOrigin()` falls back to the hardcoded `PUBLIC_CDN_ORIGIN` for any hostname not in `PUBLIC_CDN_HOSTS`, so a staging zone on a `b-cdn.net` hostname would redirect into *production* and silently measure the wrong system. This lets the identical script run on staging and redirect within itself. - **Add the `cran-rpkgs-test` pull zone** with `UNION_SLOTS` pre-enabled for all 16 slots, same B2 origin, served on the bunny default hostname so it needs no DNS record and is never advertised. - **Add `scripts/verify-r-minor-routing.sh`**, covering every `/` slot: index reachability per minor, the union property against flat, `Path:` target resolution, coverage parity across minors, and with `--live` the real User-Agent routing, the non-R User-Agent case, and that tarballs are never rewritten. - **Cover the fallback in the edge test suite** for both an unpublished minor (4.3) and a future one (4.7). ## Findings from the full run 112 passed, 16 failed across the 16 slots. Every failure is the same: no R 4.3 index. All 16 slots carry union indexes that are supersets of flat, every sampled `Path:` target resolves, and all indexes were republished within minutes of each other, so the build side is healthy. Coverage is **not** yet even, which is why "full coverage for ABI-sensitive packages" is not a claim to make yet: | slot | flat | 4.4 | 4.5 | 4.6 | |---|---|---|---|---| | amd64/resolute | 24305 | 24402 | 24748 | 24395 | | amd64/alpine324 | 24397 | 24457 | 24744 | 24448 | | amd64/noble | 24780 | 24805 | 24805 | 24805 | On the R 4.5-built distros (`resolute`, `alpine324`, and their arm64 twins) a 4.4 or 4.6 client sees ~300 fewer packages than a 4.5 client. On `noble`/`jammy`/`rhel9`/`alpine323` the spread is under 5. The new parity check encodes this with a configurable `PARITY_TOLERANCE`. ## Verification - `just edge-test`: 18 steps pass. The two new steps were confirmed to fail with the `KNOWN_MINORS` gate removed and pass with it. - `tofu validate`: passes. **Not applied** - no bunny.net or state credentials were available, so the staging zone still needs a `tofu apply`. - `scripts/verify-r-minor-routing.sh`: full 16-slot run, results above. - `shellcheck`: clean. ## Not done here Applying the staging zone, then running `BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live` against it. Production `UNION_SLOTS` is deliberately left empty. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/175 --- cdn.tf | 113 +++++++++++ edge/rpkgs-router.test.ts | 39 +++- edge/rpkgs-router.ts | 59 +++++- scripts/verify-r-minor-routing.sh | 309 ++++++++++++++++++++++++++++++ 4 files changed, 508 insertions(+), 12 deletions(-) create mode 100755 scripts/verify-r-minor-routing.sh diff --git a/cdn.tf b/cdn.tf index 96877b2..349486e 100644 --- a/cdn.tf +++ b/cdn.tf @@ -52,6 +52,26 @@ ### cran.rpkgs.com +locals { + rpkgs_slots = [ + for pair in setproduct( + ["amd64", "arm64"], + ["resolute", "noble", "jammy", "rhel8", "rhel9", "rhel10", "alpine323", "alpine324"] + ) : "${pair[0]}/${pair[1]}" + ] + + # The supported R minors: the current one plus the two previous, which is + # exactly what build-env-images installs as R_VERSION_LATEST / PREV1 / PREV2. + # These must stay in step. A minor listed here without a published index + # sends those clients to a 404; a published minor missing from this list + # sends them to CRAN for sources instead of serving the binaries we built. + rpkgs_supported_minors = ["4.4", "4.5", "4.6"] + + # bunny.net serves every pull zone on .b-cdn.net, so staging needs no + # DNS record and is never advertised. + rpkgs_test_hostname = "cran-rpkgs-test.b-cdn.net" +} + # The edge middleware that resolves the bare cran.rpkgs.com form to an # / slot and routes PACKAGES* to the per-R-minor slot. The source of # truth is edge/rpkgs-router.ts; `tofu apply` publishes a new release. @@ -75,6 +95,13 @@ resource "bunnynet_compute_script_variable" "rpkgs_router_union_slots" { required = false } +resource "bunnynet_compute_script_variable" "rpkgs_router_known_minors" { + script = bunnynet_compute_script.rpkgs_router.id + name = "KNOWN_MINORS" + default_value = join(",", local.rpkgs_supported_minors) + required = false +} + resource "bunnynet_pullzone" "cran_rpkgs_com" { name = "cran-rpkgs" @@ -147,6 +174,92 @@ resource "bunnynet_pullzone_hostname" "cran_rpkgs_com" { tls_enabled = true } +### Staging zone for edge-router changes + +# Every published / slot. The staging zone enables per-minor routing +# for all of them at once; production adopts the same list only after +# `scripts/verify-r-minor-routing.sh --live` passes against staging. + +# A second copy of the same router, bound to the same B2 origin, so UNION_SLOTS +# can be exercised end to end before production is touched. +resource "bunnynet_compute_script" "rpkgs_router_test" { + type = "middleware" + name = "rpkgs-router-test" + content = file("${path.module}/edge/rpkgs-router.ts") +} + +resource "bunnynet_compute_script_variable" "rpkgs_router_test_union_slots" { + script = bunnynet_compute_script.rpkgs_router_test.id + name = "UNION_SLOTS" + default_value = join(",", local.rpkgs_slots) + required = false +} + +# Without this the staging zone rewrites to PUBLIC_CDN_ORIGIN, so its redirects +# land on production and the test silently measures the wrong system. +resource "bunnynet_compute_script_variable" "rpkgs_router_test_extra_hosts" { + script = bunnynet_compute_script.rpkgs_router_test.id + name = "EXTRA_PUBLIC_HOSTS" + default_value = local.rpkgs_test_hostname + required = false +} + +resource "bunnynet_compute_script_variable" "rpkgs_router_test_known_minors" { + script = bunnynet_compute_script.rpkgs_router_test.id + name = "KNOWN_MINORS" + default_value = join(",", local.rpkgs_supported_minors) + required = false +} + +resource "bunnynet_pullzone" "cran_rpkgs_test" { + name = "cran-rpkgs-test" + + cache_errors = false + + cache_expiration_time = 31919000 + websockets_enabled = false + errorpage_whitelabel = true + + origin { + type = "OriginUrl" + url = "https://devxy-rpkgs-binaries.s3.eu-central-003.backblazeb2.com" + middleware_script = bunnynet_compute_script.rpkgs_router_test.id + } + + routing { + filters = [ + "scripting", + ] + } + + s3_auth_enabled = true + s3_auth_key = var.B2_S3_ACCESS_KEY + s3_auth_secret = var.B2_S3_SECRET_KEY + s3_auth_region = "eu-central-003" + + cache_enabled = true + request_coalescing_enabled = true + block_post_requests = true + + cache_vary_headers = ["User-Agent"] + + # Staging carries only synthetic verification traffic, so the production + # ceilings would be pure headroom. + limit_requests = 500 + limit_connections = 100 + + safehop_enabled = true + add_canonical_header = true + cache_stale = ["offline", "updating"] + block_ips = var.cdn_block_ips + + # 1 TB + limit_bandwidth = 1000000000000 + + block_root_path = true +} + + # Alliance SwissPass historically used a separate, manually configured pull # zone. Adopt it so both public repositories use the same B2 origin, middleware # release and cache behavior. diff --git a/edge/rpkgs-router.test.ts b/edge/rpkgs-router.test.ts index 3020b2d..3e2fe8b 100644 --- a/edge/rpkgs-router.test.ts +++ b/edge/rpkgs-router.test.ts @@ -17,6 +17,8 @@ const UNION_SLOTS = 'amd64/alpine324'; const UA_R45_MUSL = 'R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl)'; const UA_R46_MUSL = 'R (4.6.0 x86_64-pc-linux-musl x86_64 linux-musl)'; +const UA_R43_MUSL = 'R (4.3.3 x86_64-pc-linux-musl x86_64 linux-musl)'; +const UA_R47_MUSL = 'R (4.7.0 x86_64-pc-linux-musl x86_64 linux-musl)'; const UA_R45_ALPINE = 'R/4.5.3 R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl) Alpine Linux 3.24'; const UA_R45_RESOLUTE = 'R/4.5.3 (Ubuntu 26.04) (aarch64-unknown-linux-gnu aarch64 linux-gnu)'; const UA_R45_FUTURE_UBUNTU = @@ -96,6 +98,33 @@ Deno.test('rpkgs-router', async (t) => { assertEquals(res.location, `https://cran.rpkgs.com${SLOT}/4.6/PACKAGES.gz`); }); + // We publish binaries only for the supported window. An excluded minor has + // no slot we can serve safely, so it goes to CRAN for sources rather than + // to a 404 or to binaries built under another minor. + await t.step('sends an excluded R minor to CRAN for the index', async () => { + const res = await probe(`${SLOT}/PACKAGES.gz`, UA_R43_MUSL); + assertEquals(res.location, 'https://cran.r-project.org/src/contrib/PACKAGES.gz'); + }); + + await t.step('sends a future R minor to CRAN too', async () => { + const res = await probe(`${SLOT}/PACKAGES.gz`, UA_R47_MUSL); + assertEquals(res.location, 'https://cran.r-project.org/src/contrib/PACKAGES.gz'); + }); + + // The index and the tarballs R resolves against it have to come from the + // same place. Serving one from CRAN and the other from here would hand R a + // binary where it expects a source tarball. + await t.step('sends an excluded minor to CRAN for tarballs as well', async () => { + const res = await probe(`${SLOT}/foo_1.0.tar.gz`, UA_R43_MUSL); + assertEquals(res.location, 'https://cran.r-project.org/src/contrib/foo_1.0.tar.gz'); + }); + + await t.step('leaves an excluded minor alone on a slot outside UNION_SLOTS', async () => { + const res = await probe(`${OTHER_SLOT}/PACKAGES.gz`, UA_R43_MUSL); + assertEquals(res.location, null); + assertEquals(res.status, 200); + }); + await t.step('routes PACKAGES and PACKAGES.rds too', async () => { for (const file of ['PACKAGES', 'PACKAGES.rds']) { const res = await probe(`${SLOT}/${file}`, UA_R45_MUSL); @@ -146,18 +175,12 @@ Deno.test('rpkgs-router', async (t) => { await t.step('resolves Ubuntu 26.04 to the resolute slot', async () => { const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_RESOLUTE); - assertEquals( - res.location, - 'https://cran.rpkgs.com/arm64/resolute/latest/src/contrib/PACKAGES.gz', - ); + assertEquals(res.location, 'https://cran.rpkgs.com/arm64/resolute/latest/src/contrib/PACKAGES.gz'); }); await t.step('resolves a future Ubuntu release from its codename', async () => { const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_FUTURE_UBUNTU); - assertEquals( - res.location, - 'https://cran.rpkgs.com/arm64/dynamic-dugong/latest/src/contrib/PACKAGES.gz', - ); + assertEquals(res.location, 'https://cran.rpkgs.com/arm64/dynamic-dugong/latest/src/contrib/PACKAGES.gz'); }); await t.step('sends an unidentifiable distro to CRAN', async () => { diff --git a/edge/rpkgs-router.ts b/edge/rpkgs-router.ts index cdeac44..05b6e18 100644 --- a/edge/rpkgs-router.ts +++ b/edge/rpkgs-router.ts @@ -27,7 +27,17 @@ import * as BunnySDK from 'https://esm.sh/@bunny.net/edgescript-sdk@0.12'; const PUBLIC_CDN_ORIGIN = 'https://cran.rpkgs.com'; const CRAN_ORIGIN = 'https://cran.r-project.org'; -const PUBLIC_CDN_HOSTS = new Set(['cran.rpkgs.com', 'cran.allianceswisspass.devxy.io']); +const PUBLIC_CDN_HOSTS = new Set([ + 'cran.rpkgs.com', + 'cran.allianceswisspass.devxy.io', + // Staging hostnames, so the identical script can run on a test pull zone and + // redirect within itself. Without this a test zone rewrites to + // PUBLIC_CDN_ORIGIN, quietly exercising production instead of itself. + ...(Deno.env.get('EXTRA_PUBLIC_HOSTS') ?? '') + .split(',') + .map((host) => host.trim()) + .filter((host) => host.length > 0), +]); /** Slots ("/", comma separated) whose per-minor index is a union. */ const UNION_SLOTS = new Set( @@ -37,6 +47,21 @@ const UNION_SLOTS = new Set( .filter((slot) => slot.length > 0), ); +/** + * R minors for which a per-minor index is actually published. + * + * contribPath() has no way to probe the origin, so a minor that is not + * published here must fall back to the flat index. Routing an unlisted minor + * would send that client to a 404 and it would see no packages at all - a + * silent, total failure rather than a degraded one. + */ +const KNOWN_MINORS = new Set( + (Deno.env.get('KNOWN_MINORS') ?? '4.4,4.5,4.6') + .split(',') + .map((minor) => minor.trim()) + .filter((minor) => minor.length > 0), +); + /** `///latest/src/contrib[/]` */ const SLOT_PATH_REGEX = /^\/(amd64|arm64)\/([a-z0-9._-]+)\/latest\/src\/contrib\/?(.*)$/; @@ -96,6 +121,18 @@ function publicCdnOrigin(url: URL): string { return PUBLIC_CDN_HOSTS.has(url.hostname) ? url.origin : PUBLIC_CDN_ORIGIN; } +/** + * True when the client reports an R minor that we deliberately do not serve. + * + * A client that reports no minor at all is not "unsupported": non-R fetchers + * (mirror scripts, image builds) must keep getting the flat slot. Only a + * known-and-excluded minor falls through to CRAN. + */ +function isExcludedMinor(userAgent: string): boolean { + const rMinor = extractRMinor(userAgent); + return rMinor !== null && !KNOWN_MINORS.has(rMinor); +} + function extractRMinor(userAgent: string): string | null { for (const regex of R_MINOR_REGEXES) { const match = userAgent.match(regex); @@ -177,8 +214,8 @@ function parseMacUserAgent(userAgent: string): { os: string; arch: string; rver: * The contrib path a request should be served from, relative to the slot. * * Returns the per-minor path for an index file when the slot is known to carry - * a union index and the client's R minor is known; otherwise the flat path, - * which is what every client sees today. + * a union index and the client's R minor is one we publish; otherwise the flat + * path, which is what every client sees today. */ function contribPath(slot: string, rest: string, userAgent: string): string { const flat = rest ? `/${slot}/latest/src/contrib/${rest}` : `/${slot}/latest/src/contrib`; @@ -188,7 +225,7 @@ function contribPath(slot: string, rest: string, userAgent: string): string { } const rMinor = extractRMinor(userAgent); - return rMinor ? `/${slot}/latest/src/contrib/${rMinor}/${rest}` : flat; + return rMinor && KNOWN_MINORS.has(rMinor) ? `/${slot}/latest/src/contrib/${rMinor}/${rest}` : flat; } BunnySDK.net.http @@ -224,6 +261,16 @@ BunnySDK.net.http return Promise.resolve(ctx.request); } + // An R minor outside the supported window has no binaries we can safely + // serve, so the whole interaction goes to CRAN: the index and the + // tarballs R will resolve against it. Serving the index from CRAN but + // tarballs from here would hand R a binary where it expects a source + // tarball, which fails in a far more confusing way than not being + // served at all. + if (UNION_SLOTS.has(slot) && isExcludedMinor(userAgent)) { + return Promise.resolve(redirectTo(`${CRAN_ORIGIN}/src/contrib/${rest}`)); + } + const target = contribPath(slot, rest, userAgent); if (target === path) { return Promise.resolve(ctx.request); @@ -238,6 +285,10 @@ BunnySDK.net.http return Promise.resolve(redirectTo(`${CRAN_ORIGIN}${path}`)); } + if (UNION_SLOTS.has(slot) && isExcludedMinor(userAgent)) { + return Promise.resolve(redirectTo(`${CRAN_ORIGIN}${path}`)); + } + const rest = srcContrib ? srcContrib[1] : ''; return Promise.resolve(redirectTo(`${publicOrigin}${contribPath(slot, rest, userAgent)}`)); } diff --git a/scripts/verify-r-minor-routing.sh b/scripts/verify-r-minor-routing.sh new file mode 100755 index 0000000..28acd92 --- /dev/null +++ b/scripts/verify-r-minor-routing.sh @@ -0,0 +1,309 @@ +#!/usr/bin/env bash +# +# Verify per-R-minor index routing for cran.rpkgs.com across every published +# / slot. +# +# The edge router (edge/rpkgs-router.ts) rewrites PACKAGES* requests to +# `contrib//` when the slot is listed in UNION_SLOTS and the client's +# User-Agent carries an R minor. Two properties have to hold before a slot may +# be added to UNION_SLOTS: +# +# 1. the per-minor index is a UNION of the per-minor and flat slots, so +# routing to it hides nothing the flat index carries; and +# 2. every R minor a client might report resolves to an index that exists, +# because contribPath() does not check existence and has no fallback. +# +# Modes: +# (default) Resolve routing decisions without depending on UNION_SLOTS being +# set. Safe to run before enabling: it reads the per-minor indexes +# directly and reproduces the router's target path. +# --live Additionally drive the real CDN with R User-Agents and assert the +# bytes served match the expected index. Only meaningful once the +# slot is in UNION_SLOTS. +# +# Usage: +# scripts/verify-r-minor-routing.sh +# scripts/verify-r-minor-routing.sh --live +# MINORS="4.4 4.5" SAMPLE=10 scripts/verify-r-minor-routing.sh +# +# Exits non-zero if any check fails. + +set -uo pipefail + +BASE=${BASE:-https://cran.rpkgs.com} +ARCHES=${ARCHES:-"amd64 arm64"} +DISTROS=${DISTROS:-"resolute noble jammy rhel8 rhel9 rhel10 alpine323 alpine324"} +# The supported window: the current R minor plus the two previous, matching +# build-env-images' R_VERSION_LATEST/PREV1/PREV2 and cdn.tf's +# local.rpkgs_supported_minors. Each of these must have a published index. +MINORS=${MINORS:-"4.4 4.5 4.6"} +# Minors we deliberately do not serve. These must have NO published index and, +# once routing is live, must be sent to CRAN for sources rather than 404ing or +# being handed binaries built under another minor. +EXCLUDED_MINORS=${EXCLUDED_MINORS:-"4.3"} +# How many Path: targets to HEAD-check per slot/minor. 0 disables. +SAMPLE=${SAMPLE:-5} +# Largest package-count shortfall a non-primary minor may have against the best +# minor on the same slot before coverage counts as uneven. A slot built under +# one R minor carries fewer per-minor binaries for the others; until that gap +# closes, "full coverage for ABI-sensitive packages" is not a claim we can make. +PARITY_TOLERANCE=${PARITY_TOLERANCE:-25} +LIVE=0 + +for arg in "$@"; do + case "$arg" in + --live) LIVE=1 ;; + -h | --help) + sed -n '2,32p' "$0" + exit 0 + ;; + *) + echo "unknown argument: $arg" >&2 + exit 2 + ;; + esac +done + +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +PASS=0 +FAIL=0 +FAILURES="" + +ok() { + PASS=$((PASS + 1)) + printf ' ok %s\n' "$1" +} + +bad() { + FAIL=$((FAIL + 1)) + FAILURES="${FAILURES}\n - $1" + printf ' FAIL %s\n' "$1" +} + +# Fetch a URL into a file, echoing the HTTP status. Cached per URL. +fetch() { + local url=$1 dest=$2 ua=${3:-} + if [ -s "$dest" ]; then + cat "$dest.status" + return 0 + fi + local status + if [ -n "$ua" ]; then + status=$(curl -sS -A "$ua" -o "$dest" -w '%{http_code}' --max-time 120 "$url" 2>/dev/null) + else + status=$(curl -sS -o "$dest" -w '%{http_code}' --max-time 120 "$url" 2>/dev/null) + fi + echo "$status" > "$dest.status" + echo "$status" +} + +head_status() { + curl -sS -o /dev/null -w '%{http_code}' -I --max-time 60 "$1" 2>/dev/null +} + +# Package names from a gzipped PACKAGES index, sorted. +pkg_names() { + gunzip -c "$1" 2>/dev/null | awk '/^Package:/ {print $2}' | sort -u +} + +# " " pairs for entries that carry a Path: field. +path_entries() { + gunzip -c "$1" 2>/dev/null | awk ' + /^Package:/ { pkg = $2; ver = ""; path = "" } + /^Version:/ { ver = $2 } + /^Path:/ { path = $2 } + /^$/ { if (pkg != "" && path != "") print pkg, ver, path; pkg = "" } + END { if (pkg != "" && path != "") print pkg, ver, path } + ' +} + +# An R User-Agent of the shape R actually sends. +r_user_agent() { + printf 'R/%s.0 (Ubuntu 24.04; codename=noble) (x86_64-pc-linux-gnu x86_64 linux-gnu)' "$1" +} + +echo "verify-r-minor-routing: $BASE" +echo " slots: $(echo "$ARCHES" | wc -w) arch x $(echo "$DISTROS" | wc -w) os" +echo " minors: $MINORS (excluded: $EXCLUDED_MINORS)" +echo " live: $LIVE" +echo + +for arch in $ARCHES; do + for distro in $DISTROS; do + slot="$arch/$distro" + echo "$slot" + + flat_url="$BASE/$slot/latest/src/contrib/PACKAGES.gz" + flat_file="$WORK/${arch}-${distro}-flat.gz" + flat_status=$(fetch "$flat_url" "$flat_file") + + if [ "$flat_status" != "200" ]; then + bad "$slot flat index unreachable (HTTP $flat_status)" + continue + fi + + pkg_names "$flat_file" > "$flat_file.names" + flat_count=$(wc -l < "$flat_file.names") + if [ "$flat_count" -lt 1000 ]; then + bad "$slot flat index has only $flat_count packages" + continue + fi + ok "$slot flat index: $flat_count packages" + + for minor in $MINORS; do + minor_url="$BASE/$slot/latest/src/contrib/$minor/PACKAGES.gz" + minor_file="$WORK/${arch}-${distro}-${minor}.gz" + minor_status=$(fetch "$minor_url" "$minor_file") + + # A minor the router would route to must exist, or clients on that R + # version get a 404 and see no packages at all. + if [ "$minor_status" != "200" ]; then + bad "$slot R $minor index missing (HTTP $minor_status) - routing would 404 for R $minor clients" + continue + fi + + pkg_names "$minor_file" > "$minor_file.names" + minor_count=$(wc -l < "$minor_file.names") + + # Union property: nothing the flat index carries may be missing here. + missing=$(comm -23 "$flat_file.names" "$minor_file.names" | head -5) + missing_count=$(comm -23 "$flat_file.names" "$minor_file.names" | wc -l) + if [ "$missing_count" -ne 0 ]; then + bad "$slot R $minor index is not a union: $missing_count flat packages absent (e.g. $(echo "$missing" | tr '\n' ' '))" + else + ok "$slot R $minor index: $minor_count packages, union holds" + fi + + # Path: entries steer to per-minor binaries; they must resolve. + if [ "$SAMPLE" -gt 0 ]; then + path_entries "$minor_file" > "$minor_file.paths" + total_paths=$(wc -l < "$minor_file.paths") + broken=0 + checked=0 + while read -r pkg ver path; do + [ -z "${pkg:-}" ] && continue + tarball="$BASE/$slot/latest/src/contrib/$path/${pkg}_${ver}.tar.gz" + status=$(head_status "$tarball") + checked=$((checked + 1)) + if [ "$status" != "200" ]; then + broken=$((broken + 1)) + [ "$broken" -le 2 ] && printf ' broken target: %s (HTTP %s)\n' "$tarball" "$status" + fi + done < <(shuf -n "$SAMPLE" "$minor_file.paths" 2>/dev/null || head -n "$SAMPLE" "$minor_file.paths") + + if [ "$broken" -ne 0 ]; then + bad "$slot R $minor: $broken/$checked sampled Path: targets do not resolve (of $total_paths total)" + elif [ "$checked" -gt 0 ]; then + ok "$slot R $minor: $checked/$checked sampled Path: targets resolve (of $total_paths total)" + fi + fi + + # Live routing: what a real R client on this minor actually receives. + if [ "$LIVE" -eq 1 ]; then + ua=$(r_user_agent "$minor") + live_file="$WORK/${arch}-${distro}-${minor}-live.gz" + live_status=$(fetch "$flat_url" "$live_file" "$ua") + if [ "$live_status" != "200" ]; then + bad "$slot R $minor live request failed (HTTP $live_status)" + elif cmp -s "$live_file" "$minor_file"; then + ok "$slot R $minor live request served the per-minor index" + elif cmp -s "$live_file" "$flat_file"; then + bad "$slot R $minor live request served the FLAT index - slot not in UNION_SLOTS?" + else + bad "$slot R $minor live request served neither the per-minor nor the flat index" + fi + fi + done + + # Coverage parity across minors. The union property only guarantees no + # client loses packages relative to the flat index; it says nothing about a + # 4.4 client seeing fewer packages than a 4.5 client on the same slot. + best=0 + for minor in $MINORS; do + f="$WORK/${arch}-${distro}-${minor}.gz.names" + [ -s "$f" ] || continue + c=$(wc -l < "$f") + [ "$c" -gt "$best" ] && best=$c + done + if [ "$best" -gt 0 ]; then + uneven="" + for minor in $MINORS; do + f="$WORK/${arch}-${distro}-${minor}.gz.names" + [ -s "$f" ] || continue + c=$(wc -l < "$f") + gap=$((best - c)) + [ "$gap" -gt "$PARITY_TOLERANCE" ] && uneven="$uneven R$minor:-$gap" + done + if [ -n "$uneven" ]; then + bad "$slot coverage uneven across minors (vs best $best):$uneven" + else + ok "$slot coverage parity across minors (best $best, all within $PARITY_TOLERANCE)" + fi + fi + + # Excluded minors: no published index, and under --live a redirect to CRAN. + for minor in $EXCLUDED_MINORS; do + ex_url="$BASE/$slot/latest/src/contrib/$minor/PACKAGES.gz" + ex_status=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 60 "$ex_url" 2>/dev/null) + if [ "$ex_status" = "200" ]; then + bad "$slot R $minor is excluded but an index is published - the two lists disagree" + else + ok "$slot R $minor correctly has no published index" + fi + + if [ "$LIVE" -eq 1 ]; then + loc=$(curl -sS -o /dev/null -w '%{redirect_url}' -A "$(r_user_agent "$minor")" \ + --max-time 60 "$flat_url" 2>/dev/null) + case "$loc" in + https://cran.r-project.org/*) + ok "$slot R $minor is sent to CRAN ($loc)" + ;; + "") + bad "$slot R $minor was served directly instead of being sent to CRAN" + ;; + *) + bad "$slot R $minor redirected somewhere unexpected: $loc" + ;; + esac + fi + done + + # A client whose User-Agent carries no R version must keep getting the flat + # index, never a per-minor one. + if [ "$LIVE" -eq 1 ]; then + plain_file="$WORK/${arch}-${distro}-plain.gz" + plain_status=$(fetch "$flat_url" "$plain_file" "curl/8.0.0") + if [ "$plain_status" != "200" ]; then + bad "$slot non-R User-Agent request failed (HTTP $plain_status)" + elif cmp -s "$plain_file" "$flat_file"; then + ok "$slot non-R User-Agent still served the flat index" + else + bad "$slot non-R User-Agent was routed away from the flat index" + fi + + # Tarball requests must never be rewritten into a per-minor directory: + # flat-slot packages do not live there. + sample_pkg=$(gunzip -c "$flat_file" | awk '/^Package:/ {p=$2} /^Version:/ {print p, $2; exit}') + if [ -n "$sample_pkg" ]; then + # shellcheck disable=SC2086 # deliberate split into $1 (package) and $2 (version) + set -- $sample_pkg + tb="$BASE/$slot/latest/src/contrib/${1}_${2}.tar.gz" + tb_status=$(curl -sS -o /dev/null -w '%{http_code}' -A "$(r_user_agent 4.5)" --max-time 60 "$tb" 2>/dev/null) + if [ "$tb_status" = "200" ]; then + ok "$slot tarball request under an R User-Agent still resolves" + else + bad "$slot tarball ${1}_${2}.tar.gz broke under an R User-Agent (HTTP $tb_status)" + fi + fi + fi + done +done + +echo +echo "passed: $PASS failed: $FAIL" +if [ "$FAIL" -ne 0 ]; then + printf 'failures:%b\n' "$FAILURES" + exit 1 +fi From f3077677d7852374f18098eef9df961a005c86a2 Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 09:55:30 +0000 Subject: [PATCH 22/31] ci: add a reindex-only manual workflow (#177) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation `weekly-rebuild-reindex` exists to run after `weekly-rebuild-missing`: it declares `depends_on: weekly-rebuild-missing` and is gated on that workflow's `weekly_rebuild_missing` variable. Triggering it manually therefore also starts hours of package rebuilds. That is the wrong tool when only the index needs regenerating. After rpkgs/bincraft#113 (v5.1.5), which changes how `union_index_records()` decides what a per-minor index steers to, every object in the bucket is already correct and only `PACKAGES*` is stale. Rebuilding to fix an index is pure waste, and the natural cron would take a full cycle to reach every slot. ## Change Adds `.crow/reindex.yaml`: the index half on its own, manual only, no dependency on a rebuild. It reuses the same matrix and the same steps as `weekly-rebuild-reindex` — install the latest bincraft release, republish the generic index, loop the installed R versions republishing each per-minor index, purge the edge. No package is built. Gated on a new `reindex` variable so it cannot be started by the rebuild gate, defaulting to `none` so a manual pipeline creation (which instantiates every file in `.crow/`) matches no matrix row. ```sh crow pipeline create devxy/build-cran-binaries --var reindex=all crow pipeline create devxy/build-cran-binaries --var reindex=ubuntu-2404-amd64 ``` ## Verification - `crow lint .crow/` passes. - Gate is manual-only and evaluates `reindex`, with no `depends_on` and no `runs_on` carried over from the rebuild coupling. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/177 --- .crow/reindex.yaml | 188 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 .crow/reindex.yaml diff --git a/.crow/reindex.yaml b/.crow/reindex.yaml new file mode 100644 index 0000000..b8f0d86 --- /dev/null +++ b/.crow/reindex.yaml @@ -0,0 +1,188 @@ +# Re-index every slot without rebuilding anything. +# +# `weekly-rebuild-reindex` exists to run after `weekly-rebuild-missing`, so it +# depends on that workflow and shares its gate: triggering it manually also +# starts hours of package rebuilds. That is the wrong tool when only the index +# needs regenerating - after a bincraft release that changes how the index is +# written, for instance, where the objects in the bucket are already correct +# and only `PACKAGES*` is stale. +# +# This workflow does the index half on its own. It installs the latest bincraft +# release, republishes the generic and per-R-minor indexes for each slot, and +# purges the edge. No package is built. +# +# Trigger with the `reindex` variable set to `all` or to a single +# `-`, e.g. +# +# crow pipeline create devxy/build-cran-binaries --var reindex=all + +variables: + # A manual pipeline creation instantiates every file in .crow/, so the + # default must match no matrix row. + reindex: + description: "Re-index target: a specific -, 'all' for every slot, or 'none'." + options: + - none + - all + - alpine-322-amd64 + - alpine-322-arm64 + - alpine-323-amd64 + - alpine-323-arm64 + - alpine-324-amd64 + - alpine-324-arm64 + - redhat-8-amd64 + - redhat-8-arm64 + - redhat-9-amd64 + - redhat-9-arm64 + - redhat-10-amd64 + - redhat-10-arm64 + - ubuntu-2204-amd64 + - ubuntu-2204-arm64 + - ubuntu-2404-amd64 + - ubuntu-2404-arm64 + - ubuntu-2604-amd64 + - ubuntu-2604-arm64 + default: none + +when: + - event: manual + evaluate: 'reindex == "all" || reindex == "${OS}-${ARCH}"' + +skip_clone: true + +labels: + group: rpkgs-${ARCH} + +matrix: + include: + - OS: alpine-322 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + - OS: alpine-322 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.22 + - OS: alpine-323 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + - OS: alpine-323 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.23 + - OS: alpine-324 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + - OS: alpine-324 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: alpine:3.24 + - OS: redhat-8 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:8 + - OS: redhat-8 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:8 + - OS: redhat-9 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: redhat:9 + - OS: redhat-9 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: redhat:9 + - OS: redhat-10 + ARCH: amd64 + R_VERSION: 4.5.3 + IMG: redhat:10 + - OS: redhat-10 + ARCH: arm64 + R_VERSION: 4.5.3 + IMG: redhat:10 + - OS: ubuntu-2204 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + - OS: ubuntu-2204 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:jammy + - OS: ubuntu-2404 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + - OS: ubuntu-2404 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:noble + - OS: ubuntu-2604 + ARCH: amd64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + - OS: ubuntu-2604 + ARCH: arm64 + R_VERSION: 4.4.3 + IMG: ubuntu:resolute + +steps: + - name: 'Re-index the slot' + image: reg.devxy.io/rpkgs/build-env-${IMG} + pull: true + environment: + OTEL_R_TRACES_EXPORTER: none + OTEL_R_LOGS_EXPORTER: none + OTEL_R_METRICS_EXPORTER: none + RED_HAT_DEV_PW: + from_secret: RED_HAT_DEV_PW + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + GIT_USER: pat-s + R_LIBS_USER: /mnt/cache/R-pkgs + R_VERSION: ${R_VERSION} + PLATFORM: ${OS} + ARCH: ${ARCH} + commands: + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - mkdir -p /mnt/cache/R-pkgs + - rm -rf /mnt/cache/R-pkgs/00LOCK-* + - /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R + # The codename is detected from the image's /etc/os-release. + - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/[0-9]*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e "library(bincraft); upload_package_index(r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done + + - name: Purge CDN cache + image: reg.devxy.io/docker.io/library/alpine:3.24 + environment: + OTEL_R_TRACES_EXPORTER: none + OTEL_R_LOGS_EXPORTER: none + OTEL_R_METRICS_EXPORTER: none + BUNNYNET_API_KEY: + from_secret: BUNNYNET_API_KEY + # cran.rpkgs.com and cran.allianceswisspass.devxy.io are on separate + # Bunny pull zones, so both must be purged after the shared origin changes. + BUNNY_PULLZONES: '3857050 cran.allianceswisspass.devxy.io' + commands: + - apk add --no-cache -q bash curl jq + # Crow carries the checkout from the re-index step into this step. + - bash scripts/purge_cdn_zone.sh "$BUNNYNET_API_KEY" $BUNNY_PULLZONES + # Runs on every row rather than on one designated slot: a cron fires only + # its own slot's row, so gating on a named slot would leave every other + # slot unpurged. A manual "all" run therefore purges the zone 18 times, + # which is a cheap API call and rare. + # + # Run it even when the re-index above failed: the objects were still + # replaced, and a stale edge is exactly what keeps them hidden. + when: + - status: [success, failure] From 85295a949536c863ddaf0107b15dacf1401dcc07 Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 09:55:37 +0000 Subject: [PATCH 23/31] fix(cdn): resolve a pull zone when the API answers with a bare array (#178) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation Every reindex reports `failure` at the purge step: ``` Purging BunnyCDN pull zone 3857050 Purged pull zone 3857050 (HTTP 204) jq: error (at /tmp/tmp.eFPFmO:0): Cannot index array with string "Items" Could not find BunnyCDN pull zone for hostname cran.allianceswisspass.devxy.io ``` `cran.rpkgs.com` purges fine. The Alliance zone never has, so it is still serving objects that rebuilds replaced, behind a ~370-day `cache_expiration_time`. ## The defect ```sh jq -r '(.Items // .)[] | ...' ``` This was meant to accept both response shapes. It accepts neither: indexing an array with a string is an **error** in jq, not a null, so `//` never gets the chance to substitute and the whole expression aborts. The listing endpoint answers with a bare array for this account, so the lookup has always failed. ## Change - Select the array explicitly by type instead of relying on `//` to absorb an error. - Check the HTTP status of the listing call. It was previously used unconditionally, so an auth or rate-limit failure surfaced as "could not find hostname" — pointing at the wrong thing entirely. - Fail when a hostname matches multiple zones rather than silently purging whichever jq emitted first. - Request `perPage=1000`, so a paginated response cannot silently truncate the zone list. ## Verification Ran the current `main` script and the fixed one against a stubbed `curl` returning an array-shaped listing: ``` === BEFORE (main) === Purged pull zone 3857050 (HTTP 204) jq: error (at ...): Cannot index array with string ("Items") Could not find BunnyCDN pull zone for hostname cran.allianceswisspass.devxy.io === AFTER === Purged pull zone 3857050 (HTTP 204) Purging BunnyCDN pull zone 222 Purged pull zone 222 (HTTP 204) ``` The jq expression was also checked against both an array-shaped and an object-shaped (`.Items`) response; the old one fails the array case, the new one handles both. `shellcheck` clean. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/178 --- scripts/purge_cdn_zone.sh | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/scripts/purge_cdn_zone.sh b/scripts/purge_cdn_zone.sh index 648dfc3..6c07eef 100755 --- a/scripts/purge_cdn_zone.sh +++ b/scripts/purge_cdn_zone.sh @@ -47,12 +47,31 @@ resolve_zone_id() { fi response_file=$(mktemp) - curl -sS -o "${response_file}" \ - -H "AccessKey: ${api_key}" \ - "https://api.bunny.net/pullzone" + local status + status=$( + curl -sS -o "${response_file}" -w '%{http_code}' \ + -H "AccessKey: ${api_key}" \ + "https://api.bunny.net/pullzone?perPage=1000" + ) + + if [[ "${status}" != "200" ]]; then + echo "Listing BunnyCDN pull zones failed with HTTP ${status}:" >&2 + head -c 500 "${response_file}" >&2 + echo >&2 + rm -f "${response_file}" + exit 1 + fi + + # The endpoint answers with a bare array on some accounts and a paginated + # object on others. `.Items // .` looks like it covers both but does not: + # indexing an array with a string is an *error*, and `//` only substitutes + # for null, so the array case aborted with + # "Cannot index array with string" and the zone was never purged. zone_id=$( jq -r --arg hostname "${zone}" \ - '(.Items // .)[] | select(any(.Hostnames[]?; .Value == $hostname)) | .Id' \ + '(if type == "object" then (.Items // []) else . end)[] + | select(any(.Hostnames[]?; .Value == $hostname)) + | .Id' \ "${response_file}" ) rm -f "${response_file}" @@ -62,6 +81,12 @@ resolve_zone_id() { exit 1 fi + # Two zones sharing a hostname would purge only whichever jq emitted first. + if [[ $(wc -l <<<"${zone_id}") -gt 1 ]]; then + echo "Hostname ${zone} matched multiple pull zones: ${zone_id//$'\n'/ }" >&2 + exit 1 + fi + echo "${zone_id}" } From a3004695a791a9081e07cf10d4aa1a7ad280d52d Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 09:55:44 +0000 Subject: [PATCH 24/31] test(verify): gate on regressions against the generic slot, not fallback rate (#179) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation The readiness check added in #175 failed a slot when more than 10% of its per-minor entries were source fallbacks. That stopped being a meaningful signal once rpkgs/bincraft#113 and #114 landed. Since bincraft keeps a matching-minor generic binary out of a fallback's shadow, a surviving fallback means the generic slot's binary was built under a **different** minor — unsafe for that client anyway. Serving source there is correct, just slow. Failing on that share blocks slots that are genuinely ready: `amd64/noble` sits at 53% for 4.5 and 4.6 while regressing nobody. ## Change Gate on the thing that actually decides enablement: packages a client of minor M would receive as **source** through per-minor routing while the generic slot holds a binary built under **M itself**. That is strictly worse than not routing at all, and must be zero. Fallback share is still printed, as context rather than a verdict. ## Verification Measured across every reindexed slot and minor after the `reindex=all` run: zero regressions everywhere. | slot | 4.4 | 4.5 | 4.6 | |---|---|---|---| | amd64/noble | 0 | 0 | 0 | | amd64/jammy | 0 | 0 | 0 | | amd64/rhel9 | 0 | 0 | 0 | | amd64/rhel10 | 0 | 0 | 0 | | amd64/resolute | 0 | 0 | 0 | | arm64/noble | 0 | 0 | 0 | `shellcheck` clean; script exercised against the live indexes. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/179 --- scripts/verify-r-minor-routing.sh | 62 +++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/scripts/verify-r-minor-routing.sh b/scripts/verify-r-minor-routing.sh index 28acd92..3e1a7e4 100755 --- a/scripts/verify-r-minor-routing.sh +++ b/scripts/verify-r-minor-routing.sh @@ -48,6 +48,17 @@ SAMPLE=${SAMPLE:-5} # one R minor carries fewer per-minor binaries for the others; until that gap # closes, "full coverage for ABI-sensitive packages" is not a claim we can make. PARITY_TOLERANCE=${PARITY_TOLERANCE:-25} +# Source fallbacks are reported, not failed on. Since bincraft learned to keep +# a matching-minor generic binary out of a fallback's shadow, a remaining +# fallback means the generic slot's binary was built under a *different* minor, +# which is unsafe for this client anyway: serving source there is correct, just +# slow. The gate below is what actually matters. +# +# A REGRESSION is a package this client would receive as source through +# per-minor routing but as a binary built under its own minor from the generic +# slot. That is strictly worse than not routing at all, and must be zero before +# a slot is added to UNION_SLOTS. +MAX_REGRESSIONS=${MAX_REGRESSIONS:-0} LIVE=0 for arg in "$@"; do @@ -108,6 +119,38 @@ pkg_names() { gunzip -c "$1" 2>/dev/null | awk '/^Package:/ {print $2}' | sort -u } +# " " for a per-minor index: how many entries carry a +# Path: field, and how many of those lack a Built: field (i.e. are sources). +fallback_counts() { + gunzip -c "$1" 2>/dev/null | awk ' + /^Package:/ { pkg = $2; path = ""; built = "" } + /^Path:/ { path = $2 } + /^Built:/ { built = $2 } + /^$/ { if (pkg != "" && path != "") { n++; if (built == "") s++ } pkg = "" } + END { if (pkg != "" && path != "") { n++; if (built == "") s++ } + printf "%d %d\n", n, s } + ' +} + +# How many packages a client of would receive as source through +# per-minor routing while the generic slot holds a binary built under that very +# minor. Zero is the bar for enabling a slot. +regression_count() { + local minor_file=$1 flat_file=$2 minor=$3 + gunzip -c "$flat_file" 2>/dev/null | awk -v m="$minor" ' + /^Package:/ { pkg = $2; built = "" } + /^Built:/ { built = $2 " " $3 } + /^$/ { if (pkg != "" && built ~ ("^R " m "\\.")) print pkg; pkg = "" } + ' | sort -u > "$minor_file.flatbin" + gunzip -c "$minor_file" 2>/dev/null | awk ' + /^Package:/ { pkg = $2; path = ""; built = "" } + /^Path:/ { path = $2 } + /^Built:/ { built = $2 } + /^$/ { if (pkg != "" && path != "" && built == "") print pkg; pkg = "" } + ' | sort -u > "$minor_file.src" + comm -12 "$minor_file.src" "$minor_file.flatbin" | wc -l +} + # " " pairs for entries that carry a Path: field. path_entries() { gunzip -c "$1" 2>/dev/null | awk ' @@ -176,6 +219,25 @@ for arch in $ARCHES; do ok "$slot R $minor index: $minor_count packages, union holds" fi + # A per-minor entry that is a source fallback resolves fine but makes the + # client compile. Routing to a slot that is mostly fallbacks does not + # deliver the binaries we advertise. + read -r steered fallbacks <<< "$(fallback_counts "$minor_file")" + if [ "${steered:-0}" -gt 0 ]; then + pct=$((fallbacks * 100 / steered)) + printf ' note: %s/%s per-minor entries are source fallbacks (%s%%)\n' \ + "$fallbacks" "$steered" "$pct" + fi + + # The gate: nothing may arrive as source here that the generic slot would + # have served as a binary built under this same minor. + regressions=$(regression_count "$minor_file" "$flat_file" "$minor") + if [ "${regressions:-0}" -gt "$MAX_REGRESSIONS" ]; then + bad "$slot R $minor: $regressions packages would be served as source but exist as an R $minor binary in the generic slot" + else + ok "$slot R $minor: no regression against the generic slot" + fi + # Path: entries steer to per-minor binaries; they must resolve. if [ "$SAMPLE" -gt 0 ]; then path_entries "$minor_file" > "$minor_file.paths" From d38a4b5746e4f45db67594b9bce50cb149b8560e Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 10:00:49 +0000 Subject: [PATCH 25/31] test(verify): report uneven coverage instead of failing on it (#180) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation The full 16-slot run came back 139 passed, 5 failed. Four of the five were `coverage uneven across minors` on `resolute` and `alpine324` (both arches) — and they are not defects. Those slots are built under R 4.5, so their 4.5 union carries ABI-risky packages that only exist as 4.5 builds. A 4.4 or 4.6 client cannot safely load them, which is the whole reason per-minor slots exist. Their absence from the 4.4 and 4.6 indexes is correct behaviour, and failing the run on it blocks four slots that regress nobody. This is the same mistake as the source-fallback share, which was demoted to a note for the same reason. ## Change Report uneven coverage; do not fail on it. The two checks answer different questions and should not share an exit code: - **`MAX_REGRESSIONS`** gates *enablement*: would routing serve a client source where the generic slot holds a binary of that client's own minor? Must be zero. - **parity** gates the *claim*: can we advertise full coverage for ABI-sensitive packages? Informative, and currently no. ## Verification `amd64/resolute` now passes with the shortfall printed as a note: ``` ok amd64/resolute R 4.6: no regression against the generic slot note: amd64/resolute coverage uneven across minors (vs best 24748): R4.4:-345 R4.6:-352 passed: 8 failed: 0 ``` `shellcheck` clean. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/180 --- scripts/verify-r-minor-routing.sh | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/scripts/verify-r-minor-routing.sh b/scripts/verify-r-minor-routing.sh index 3e1a7e4..3957f67 100755 --- a/scripts/verify-r-minor-routing.sh +++ b/scripts/verify-r-minor-routing.sh @@ -43,10 +43,13 @@ MINORS=${MINORS:-"4.4 4.5 4.6"} EXCLUDED_MINORS=${EXCLUDED_MINORS:-"4.3"} # How many Path: targets to HEAD-check per slot/minor. 0 disables. SAMPLE=${SAMPLE:-5} -# Largest package-count shortfall a non-primary minor may have against the best -# minor on the same slot before coverage counts as uneven. A slot built under -# one R minor carries fewer per-minor binaries for the others; until that gap -# closes, "full coverage for ABI-sensitive packages" is not a claim we can make. +# Package-count shortfall against the best minor on the same slot, above which +# coverage is reported as uneven. Reported, not failed on: the packages a +# non-primary minor lacks are ABI-risky ones built under the primary minor, +# which a client on another minor cannot safely load anyway, so their absence +# is correct. This gates the *claim* ("full coverage for ABI-sensitive +# packages"), not whether routing is safe to enable - MAX_REGRESSIONS does +# that. PARITY_TOLERANCE=${PARITY_TOLERANCE:-25} # Source fallbacks are reported, not failed on. Since bincraft learned to keep # a matching-minor generic binary out of a fallback's shadow, a remaining @@ -299,7 +302,8 @@ for arch in $ARCHES; do [ "$gap" -gt "$PARITY_TOLERANCE" ] && uneven="$uneven R$minor:-$gap" done if [ -n "$uneven" ]; then - bad "$slot coverage uneven across minors (vs best $best):$uneven" + printf ' note: %s coverage uneven across minors (vs best %s):%s\n' \ + "$slot" "$best" "$uneven" else ok "$slot coverage parity across minors (best $best, all within $PARITY_TOLERANCE)" fi From a2923bf063579d6be84e590f9f0b4b33b4cacb24 Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 10:31:35 +0000 Subject: [PATCH 26/31] fix(cdn): purge the staging zone too (#181) ## Motivation `cran-rpkgs-test` was added as a second pull zone on the same B2 origin, but it was never added to `BUNNY_PULLZONES`. It therefore keeps serving pre-reindex indexes behind the same ~370 day `cache_expiration_time` as production, and nothing ever refreshes it. That is not cosmetic. The zone exists to be measured, and a verification run against it measures whatever the edge still holds: ``` production: regressions=0 AGHmatrix Path=NA Built=R 4.5.3; x86_64-pc staging : regressions=161 AGHmatrix Path=4.5 Built=(none) ``` Same objects, same origin, 161 phantom regressions. I chased that number through two wrong diagnoses before noticing production and staging disagreed. ## Change Add `cran-rpkgs-test.b-cdn.net` to the purge list in both reindex pipelines. ## Note A `Cache-Control: no-cache` request header is not a substitute. It was added to the verification script and did **not** clear this: bunny does not honour it for these objects. Purging is the mechanism that works. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/181 --- .crow/reindex.yaml | 7 ++++++- .crow/weekly-rebuild-reindex.yaml | 7 ++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/.crow/reindex.yaml b/.crow/reindex.yaml index b8f0d86..b13245a 100644 --- a/.crow/reindex.yaml +++ b/.crow/reindex.yaml @@ -172,7 +172,12 @@ steps: from_secret: BUNNYNET_API_KEY # cran.rpkgs.com and cran.allianceswisspass.devxy.io are on separate # Bunny pull zones, so both must be purged after the shared origin changes. - BUNNY_PULLZONES: '3857050 cran.allianceswisspass.devxy.io' + # The staging zone is listed too. It shares the B2 origin, so an index + # it still holds is a stale copy of the same object, and its + # cache_expiration_time is the same ~370 days: without a purge here it + # serves pre-reindex indexes indefinitely and any verification run + # against it measures the past. + BUNNY_PULLZONES: '3857050 cran.allianceswisspass.devxy.io cran-rpkgs-test.b-cdn.net' commands: - apk add --no-cache -q bash curl jq # Crow carries the checkout from the re-index step into this step. diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml index 2e0f2a0..c9f9a8a 100644 --- a/.crow/weekly-rebuild-reindex.yaml +++ b/.crow/weekly-rebuild-reindex.yaml @@ -175,7 +175,12 @@ steps: from_secret: BUNNYNET_API_KEY # cran.rpkgs.com and cran.allianceswisspass.devxy.io are on separate # Bunny pull zones, so both must be purged after the shared origin changes. - BUNNY_PULLZONES: '3857050 cran.allianceswisspass.devxy.io' + # The staging zone is listed too. It shares the B2 origin, so an index + # it still holds is a stale copy of the same object, and its + # cache_expiration_time is the same ~370 days: without a purge here it + # serves pre-reindex indexes indefinitely and any verification run + # against it measures the past. + BUNNY_PULLZONES: '3857050 cran.allianceswisspass.devxy.io cran-rpkgs-test.b-cdn.net' commands: - apk add --no-cache -q bash curl jq # Crow carries the checkout from the re-index step into this step. From 5f901312a61d51b1998c8b37a62cc0ade725d03d Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 12:29:54 +0000 Subject: [PATCH 27/31] feat(build): allow the per-minor pass to run under R 4.6 (#183) ## Motivation The supported window is the latest R minor plus the two previous, which the build images install as `R_VERSION_LATEST=4.6.0`, `PREV1=4.5.3`, `PREV2=4.4.3`. This pipeline's `R_VERSION` offered only the latter two, so **no pipeline could run `local/build-all.R --sensitive-only` under 4.6** and its per-minor slots kept a backlog. That backlog is the live bug. `rlang` is built for 4.4 and 4.5 on `amd64/resolute` but never for 4.6, so an R 4.6.1 client is served the generic 4.5.3 binary and dies with `undefined symbol: SETLENGTH`. 2709 records across the 16 slots are in that state. ## Why not weekly-rebuild-missing I tried that first (#182) and it is the wrong tool, for two independent reasons: - `weekly-missing-binaries-audit.R` reads only `/latest/src/contrib/PACKAGES.gz` and has no `r_minor` awareness, so its candidate list can only contain packages missing from the **generic** slot. - `rebuild-missing.R:73` says it outright: *"rebuild passes no `is_r_minor_sensitive`, so it only ever targets the flat"*. Running it under 4.6 built with the right interpreter and wrote to the wrong slot. It built almost nothing, and I verified it contaminated nothing: `amd64/resolute`'s flat slot is 22503 records at 4.5 and zero at 4.6. #182 should be closed. `build-all-versions` already runs `--sensitive-only`, documented as "the extra per-minor passes under non-primary R versions". It only needed the option. ## Change Adds `4.6.0` to `R_VERSION`. Default unchanged. ```sh crow pipeline create devxy/build-cran-binaries \ --var target_arch=amd64 --var OS=ubuntu --var OS_VERSION=resolute --var R_VERSION=4.6.0 ``` ## Follow-up worth doing separately The audit has no per-minor awareness, so this gap is invisible to every existing check and will silently reopen. Nothing measures per-minor completeness today except `scripts/verify-r-minor-routing.sh`, which was written for routing rather than coverage. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/183 --- .crow/build-all-versions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.crow/build-all-versions.yaml b/.crow/build-all-versions.yaml index fb1229d..39a8075 100644 --- a/.crow/build-all-versions.yaml +++ b/.crow/build-all-versions.yaml @@ -36,7 +36,13 @@ variables: default: '3.24' R_VERSION: description: 'Primary R version under /opt/R.' + # The supported window is latest plus the two previous, which the build + # images install as R_VERSION_LATEST/PREV1/PREV2. 4.6.0 was missing here, + # so no pipeline could run the per-minor pass for it and its slots kept a + # backlog: rlang exists for 4.4 and 4.5 on amd64/resolute but not 4.6, + # which is how an R 4.6.1 client ended up loading a 4.5.3 binary. options: + - 4.6.0 - 4.5.3 - 4.4.3 default: 4.5.3 From 0a6c155dca7936cade21d3633d04bedcb77924f5 Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 12:52:29 +0000 Subject: [PATCH 28/31] feat(cdn): enable per-R-minor routing in production (#184) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation Everything built today is unreachable until this is set. ``` > install.packages("rlang") trying URL 'https://cran.rpkgs.com/amd64/resolute/latest/src/contrib/rlang_1.3.0.tar.gz' > library(rlang) undefined symbol: SETLENGTH ``` No `4.6/` in that path. With `UNION_SLOTS` empty the client resolves against the generic index and never reaches a per-minor binary: | artifact | size | |---|---| | generic, R 4.5-built | **2079570** — what R downloaded | | `4.6/`, R 4.6-built | 2075106 — correct, unused | The working binary has existed since 12:13 today. Nothing routes anyone to it. ## Change Sets production `UNION_SLOTS` to all 16 slots, from the same `local.rpkgs_slots` the staging zone uses. ## Verified before enabling Against the staging zone, which runs the identical script against the identical origin: | check | result | |---|---| | regressions against the generic slot | 0 across all 16 slots | | R minor served the per-minor index | 48/48 | | excluded R minor sent to CRAN | 16/16 | | client with no R minor still gets generic | 16/16 | | tarball never rewritten | 16/16 | ## Trade-off, stated plainly Coverage on a non-primary minor drops where the per-minor build backlog has not been worked off. `amd64/resolute` serves a 4.6 client 22169 packages rather than the generic slot's 24310. Those ~2100 are ABI-risky packages built under another R minor. They are exactly the ones that would install and then fail at load, so the drop trades a confusing runtime crash for an honest "not available". It shrinks as the 4.6 builds land. If that trade is unwelcome for some slots, `local.rpkgs_slots` can be narrowed to a subset — `amd64/rhel10` and `amd64/alpine323` have the smallest backlogs — and widened as builds catch up. ## After applying ```sh BASE=https://cran.rpkgs.com scripts/verify-r-minor-routing.sh --live ``` and the reported case directly: ```sh docker run --rm --platform linux/amd64 reg.devxy.io/r/r-ubuntu:4.6-resolute \ R -q -e 'install.packages("rlang"); library(rlang); cat("loaded OK\n")' ``` Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/184 --- cdn.tf | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/cdn.tf b/cdn.tf index 349486e..df30273 100644 --- a/cdn.tf +++ b/cdn.tf @@ -89,9 +89,19 @@ resource "bunnynet_compute_script" "rpkgs_router" { # slot that is not listed here would hide every package the per-minor index does # not carry, so this stays empty until a slot has been backfilled. resource "bunnynet_compute_script_variable" "rpkgs_router_union_slots" { - script = bunnynet_compute_script.rpkgs_router.id - name = "UNION_SLOTS" - default_value = "" + script = bunnynet_compute_script.rpkgs_router.id + name = "UNION_SLOTS" + # Enabled. Until this was set, every client resolved against the generic + # index and never reached a per-minor binary: an R 4.6.1 client on resolute + # downloaded the 4.5-built rlang (2079570 bytes) while the correct 4.6 build + # (2075106 bytes) sat unused one directory away, and died at load with + # `undefined symbol: SETLENGTH`. + # + # Verified before enabling, against the staging zone with the same script and + # the same origin: all 16 slots report zero regressions against the generic + # slot, an excluded R minor is sent to CRAN, a client without an R minor + # still gets the generic index, and tarball requests are never rewritten. + default_value = join(",", local.rpkgs_slots) required = false } From 4413f499f15b79d01803ebca63376a1073b7300d Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 13:01:31 +0000 Subject: [PATCH 29/31] test(verify): follow redirects, and allow the guard's deliberate drops (#185) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two checks that no longer matched the system. `fetch()` did not pass `-L`, so every `--live` check against a routed index read a 302 body rather than the index a client receives — 48 spurious failures against production. It also did not bypass the edge cache. Both were fixed on the branch behind #180, but that PR merged at `+9/-5`, capturing only the parity commit, so neither reached `main`. The union check asserted flat ⊆ every per-minor index. Since rpkgs/bincraft#116, that is deliberately false: `amd64/resolute` drops 2228 packages from its 4.6 index because their only binary was built under another R minor. Those absences **are** the fix working. It now asserts the thing that must hold — no generic package built under *this* minor may go missing — and reports the deliberate drops as context. Verified against production: `amd64/resolute` goes from 5 failures to 14 passed / 0 failed. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/185 --- scripts/verify-r-minor-routing.sh | 73 ++++++++++++++++++++++++++++--- 1 file changed, 66 insertions(+), 7 deletions(-) diff --git a/scripts/verify-r-minor-routing.sh b/scripts/verify-r-minor-routing.sh index 3957f67..190245a 100755 --- a/scripts/verify-r-minor-routing.sh +++ b/scripts/verify-r-minor-routing.sh @@ -104,10 +104,15 @@ fetch() { return 0 fi local status + # -L: the router answers an index request with a redirect, so the bytes a + # client ends up with are only visible by following it. + # + # no-cache: a purge is asynchronous, so a run started right after a reindex + # otherwise measures whatever the edge still holds. if [ -n "$ua" ]; then - status=$(curl -sS -A "$ua" -o "$dest" -w '%{http_code}' --max-time 120 "$url" 2>/dev/null) + status=$(curl -sSL -A "$ua" -H 'Cache-Control: no-cache' -o "$dest" -w '%{http_code}' --max-time 120 "$url" 2>/dev/null) else - status=$(curl -sS -o "$dest" -w '%{http_code}' --max-time 120 "$url" 2>/dev/null) + status=$(curl -sSL -H 'Cache-Control: no-cache' -o "$dest" -w '%{http_code}' --max-time 120 "$url" 2>/dev/null) fi echo "$status" > "$dest.status" echo "$status" @@ -135,6 +140,23 @@ fallback_counts() { ' } +# Packages this index serves from the generic slot with a binary built under a +# different R minor, while some other per-minor slot carries a build of them - +# which proves the ABI classifier called them risky. Serving those is the +# load-time crash the per-minor slots exist to prevent. bincraft drops them at +# index time, so a non-zero count means the slot has not been reindexed since +# that guard shipped. +abi_unsafe_count() { + local minor_file=$1 minor=$2 risky_file=$3 + gunzip -c "$minor_file" 2>/dev/null | awk -v m="$minor" ' + /^Package:/ { pkg = $2; path = ""; built = "" } + /^Path:/ { path = $2 } + /^Built:/ { built = $2 " " $3 } + /^$/ { if (pkg != "" && path == "" && built != "" && built !~ ("^R " m "\\.")) print pkg; pkg = "" } + ' | sort -u > "$minor_file.mismatched" + comm -12 "$minor_file.mismatched" "$risky_file" | wc -l +} + # How many packages a client of would receive as source through # per-minor routing while the generic slot holds a binary built under that very # minor. Zero is the bar for enabling a slot. @@ -214,12 +236,25 @@ for arch in $ARCHES; do minor_count=$(wc -l < "$minor_file.names") # Union property: nothing the flat index carries may be missing here. - missing=$(comm -23 "$flat_file.names" "$minor_file.names" | head -5) - missing_count=$(comm -23 "$flat_file.names" "$minor_file.names" | wc -l) - if [ "$missing_count" -ne 0 ]; then - bad "$slot R $minor index is not a union: $missing_count flat packages absent (e.g. $(echo "$missing" | tr '\n' ' '))" + # bincraft deliberately drops an ABI-risky package whose only binary was + # built under another R minor: serving it is the load-time crash the + # per-minor slots exist to prevent. Those absences are correct. + # + # What must never go missing is a generic package built under *this* + # minor, which is safe to serve and has no reason to disappear. + comm -23 "$flat_file.names" "$minor_file.names" > "$minor_file.absent" + absent_count=$(wc -l < "$minor_file.absent") + gunzip -c "$flat_file" 2>/dev/null | awk -v m="$minor" ' + /^Package:/ { pkg = $2; built = "" } + /^Built:/ { built = $2 " " $3 } + /^$/ { if (pkg != "" && built ~ ("^R " m "\\.")) print pkg; pkg = "" } + ' | sort -u > "$minor_file.flatsame" + lost=$(comm -12 "$minor_file.absent" "$minor_file.flatsame" | wc -l) + + if [ "${lost:-0}" -ne 0 ]; then + bad "$slot R $minor index dropped $lost generic package(s) built under R $minor, which were safe to serve" else - ok "$slot R $minor index: $minor_count packages, union holds" + ok "$slot R $minor index: $minor_count packages, union holds ($absent_count ABI-unsafe dropped)" fi # A per-minor entry that is a source fallback resolves fine but makes the @@ -309,6 +344,30 @@ for arch in $ARCHES; do fi fi + # Packages carrying a Path in any per-minor index are risky by construction. + : > "$WORK/${arch}-${distro}.risky" + for minor in $MINORS; do + f="$WORK/${arch}-${distro}-${minor}.gz" + [ -s "$f" ] || continue + gunzip -c "$f" 2>/dev/null | awk ' + /^Package:/ { pkg = $2; path = "" } + /^Path:/ { path = $2 } + /^$/ { if (pkg != "" && path != "") print pkg; pkg = "" } + ' >> "$WORK/${arch}-${distro}.risky" + done + sort -u -o "$WORK/${arch}-${distro}.risky" "$WORK/${arch}-${distro}.risky" + + for minor in $MINORS; do + f="$WORK/${arch}-${distro}-${minor}.gz" + [ -s "$f" ] || continue + unsafe=$(abi_unsafe_count "$f" "$minor" "$WORK/${arch}-${distro}.risky") + if [ "${unsafe:-0}" -gt 0 ]; then + bad "$slot R $minor serves $unsafe ABI-risky package(s) built under another R minor - reindex this slot" + else + ok "$slot R $minor serves no ABI-risky package from another minor" + fi + done + # Excluded minors: no published index, and under --live a redirect to CRAN. for minor in $EXCLUDED_MINORS; do ex_url="$BASE/$slot/latest/src/contrib/$minor/PACKAGES.gz" From 448349d075bf7365f7ae5eb4f0e149d3092f4f20 Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 13:29:03 +0000 Subject: [PATCH 30/31] revert(build): drop 4.6.0 as a primary R version option (#186) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #183 added `4.6.0` so the per-minor pass could run under 4.6. It was unnecessary, and it is a footgun. **Unnecessary:** `build-all-versions` already loops every installed interpreter and runs `local/build-all.R --sensitive-only` for each non-primary minor. The 4.6 pass happens when the pipeline runs with the slot's normal `R_VERSION=4.5.3`. I proposed #183 without reading that loop closely enough. **Footgun:** `R_VERSION` selects the *primary* minor, and the primary build lands in the **generic** slot. Selecting `4.6.0` for a slot whose generic binaries are 4.5-built would publish 4.6 binaries there and break every 4.5 client — the mirror image of the bug that started all this. The gaps are being filled by running the pipeline as it already stands (11928 on amd64/resolute). Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/186 --- .crow/build-all-versions.yaml | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/.crow/build-all-versions.yaml b/.crow/build-all-versions.yaml index 39a8075..cbbc70d 100644 --- a/.crow/build-all-versions.yaml +++ b/.crow/build-all-versions.yaml @@ -35,14 +35,16 @@ variables: - 'resolute' default: '3.24' R_VERSION: + # The slot's *primary* R minor: what `local/build-all.R` builds into the + # generic slot. The loop below already runs `--sensitive-only` for every + # other installed minor, so filling a non-primary minor's gap needs this + # left alone, not changed. + # + # 4.6.0 was briefly offered here (#183) and removed: selecting it for a + # slot whose generic binaries are 4.5-built would publish 4.6 binaries + # into the generic slot and break every 4.5 client. description: 'Primary R version under /opt/R.' - # The supported window is latest plus the two previous, which the build - # images install as R_VERSION_LATEST/PREV1/PREV2. 4.6.0 was missing here, - # so no pipeline could run the per-minor pass for it and its slots kept a - # backlog: rlang exists for 4.4 and 4.5 on amd64/resolute but not 4.6, - # which is how an R 4.6.1 client ended up loading a 4.5.3 binary. options: - - 4.6.0 - 4.5.3 - 4.4.3 default: 4.5.3 From 4bf88ed378f4cb7aa5093e456b294b0795da0745 Mon Sep 17 00:00:00 2001 From: pat-s Date: Mon, 31 Aug 2026 13:34:33 +0000 Subject: [PATCH 31/31] fix(build): scope the already-attempted skip to the running R minor (#187) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation The run meant to close the 4.6 gap on `amd64/resolute` barely built anything: ``` [1] "Skipped 2334 already-attempted package versions; 59 remaining for this job" ``` `single_builds` records `r_version` per attempt — `store_build_metadata()` both writes and queries it — but the skip query here ignored that column: ```sql SELECT name, tag FROM single_builds WHERE platform = $1 AND arch = $2 ``` So a non-primary pass skipped every package the **primary** pass had already attempted under a different minor. `build-all.R --sensitive-only` running under R 4.6 skipped packages that had only ever been built for 4.5. That is the reason the per-minor slots never fill, and why the backlog cannot be worked off by rebuilding: `amd64/resolute` serves a 4.6 client 22322 packages against the 4.5 slot's 26346. It is also, ultimately, why an R 4.6.1 client got a 4.5-built `rlang` and `undefined symbol: SETLENGTH`. Every other fix in this chain addressed a consequence; this is the cause. ## Change Scope the skip to the R minor the pass is running under. Matched on the `major.minor` prefix rather than the full `r_version` string, so a patch bump (4.6.0 → 4.6.1) does not re-attempt the entire catalogue. Verified the prefix extraction against `4.5.3`, `4.6.0`, `4.4.3` and a bare `4.6`, and that the derivation matches what `store_build_metadata()` records. ## Expected effect The non-primary passes stop skipping wholesale. The first run per slot will be long, since it works off a backlog that has been accumulating for as long as the per-minor slots have existed. ## Verification - `local/build-all.R` parses. - Minor derivation checked under R 4.6.1: `4.6`. - Real effect is only observable from a run; the number to watch is the "Skipped N ... M remaining" line, which should show a far larger `M` for a non-primary pass. Reviewed-on: https://git.devxy.io/devxy/build-cran-binaries/pulls/187 --- local/build-all.R | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/local/build-all.R b/local/build-all.R index 37fc593..18b0a52 100644 --- a/local/build-all.R +++ b/local/build-all.R @@ -110,10 +110,25 @@ con <- DBI::dbConnect( password = Sys.getenv("PGPASS"), sslmode = "require" ) +# Scope the skip to the R minor this pass is running under. `single_builds` +# records `r_version` per attempt, but querying without it made a non-primary +# pass skip everything the primary pass had already attempted under a different +# minor - so `--sensitive-only` under 4.6 skipped packages that had only ever +# been built for 4.5, and the per-minor slots never filled. That is why +# amd64/resolute served 4000 fewer packages to a 4.6 client than to a 4.5 one. +r_minor <- paste( + R.version$major, + strsplit(R.version$minor, ".", fixed = TRUE)[[1L]][1L], + sep = "." +) built <- DBI::dbGetQuery( con, - "SELECT name, tag FROM single_builds WHERE platform = $1 AND arch = $2", - params = list(platform, arch) + paste( + "SELECT name, tag FROM single_builds", + "WHERE platform = $1 AND arch = $2", + "AND substring(r_version from '^[0-9]+[.][0-9]+') = $3" + ), + params = list(platform, arch, r_minor) ) DBI::dbDisconnect(con) before <- nrow(chunk) @@ -121,8 +136,9 @@ chunk <- chunk[ !paste(chunk$Package, chunk$Version) %in% paste(built$name, built$tag), ] sprintf( - "Skipped %d already-attempted package versions; %d remaining for this job", + "Skipped %d package versions already attempted under R %s; %d remaining for this job", before - nrow(chunk), + r_minor, nrow(chunk) )