build-cran-binaries/specs/2026-08-12-shard-weekly-rebuild-design.md

Design: Sharding and resuming the weekly rebuild

Date: 2026-08-12 Status: Approved (pending spec review)

Problem

weekly-rebuild-missing runs one job per <os>-<arch> and walks that slot's rebuild list serially in a single R -q -e invocation (.crow/weekly-rebuild-missing.yaml:165). Until 2026-08-09 that was cheap, because every source fallback was skipped as "already built" and the list was effectively empty. Since bincraft #105/#106/#107 and build-cran-binaries #159 the gate works, and the lists are now large.

Share of records whose object is byte-identical to CRAN's source, measured against cran.r-project.org MD5s on 2026-08-12:

slot	records	source-served	share
amd64/resolute	24 212	15 023	62.1%
arm64/resolute	24 291	13 670	56.3%
arm64/alpine324	24 328	9 514	39.2%
amd64/alpine324	24 343	8 917	36.7%
arm64/rhel10	24 695	5 384	21.9%
amd64/rhel10	24 881	4 712	19.2%
12 remaining slots	~24 700	850 to 2 130	3.5% to 8.7%

A single serial job cannot absorb that. Pipeline 10910 (weekly_rebuild_missing:alpine-324-amd64) started on 2026-08-09, ran for roughly two days, reached [8692/23885] cholera, and was killed there.

Two distinct failures follow from that shape.

No parallelism. The work is embarrassingly parallel across packages, but one job does all of it.

No resumability, and no clean stopping point. The loop has no terminating condition other than exhausting the list, so the only way to stop it is a kill. A restarted run re-reads the same list and walks it from the first entry. It skips completed packages via check_s3_root_package(), but that costs a CRAN version resolution and an S3 HEAD per package, thousands of times, before it reaches new work. Worse, a kill is not a pipeline failure: the Purge CDN cache step is guarded by when: status: [success, failure] (.crow/weekly-rebuild-missing.yaml:206-207), and on 10910 it produced no output at all. So the ~4 600 binaries that run did publish stayed hidden behind stale edge copies.

Goal

Turn each slot's rebuild into bounded, parallel, restartable units, without introducing state that can disagree with the bucket.

Design

1. Shard the matrix three ways

Each of the 18 OS/ARCH rows in .crow/weekly-rebuild-missing.yaml gains SPLIT_INTO: 3 and SPLIT_INDEX: 1|2|3, giving 54 rows. This mirrors .crow/build-all-versions.yaml:57-98, which already shards its matrix four ways per arch.

Routing needs no change. The cron filter cron: weekly-rebuild-missing-${OS}-${ARCH} and the manual evaluate: weekly_rebuild_missing == "${OS}-${ARCH}" both match all three shards of a slot. Placement stays on the rpkgs-${ARCH} group label, so shards queue against available capacity rather than oversubscribing it.

2. Extract the loop into local/rebuild-missing.R

The build is currently a single ~1 500-character R -q -e argument. Shard arithmetic and resume logic do not belong in a YAML string, and none of it is testable there. The loop moves to local/rebuild-missing.R, invoked as Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX, mirroring local/build-all.R. Its body is unchanged in substance: read /tmp/rebuild_pkgs.txt, subtract local/excluded-packages.json, loop with tryCatch around bincraft::build_binary_package().

The slice is interleaved, not contiguous:

# the list is alphabetical and build cost clusters by name (Rcpp*, Bioc*,
# rstan*), so contiguous thirds would be badly unbalanced
mine <- pkgs[seq(split_index, length(pkgs), by = split_into)]

local/build-all.R:64 uses contiguous chunks via cut(). That is fine there because its list is every CRAN package and version, so the chunks average out. Here the list is a filtered backlog in which expensive families sit adjacent, so interleaving is the better default. Interleaving also makes each shard's [i/n] progress representative of the slot as a whole.

3. Resume by re-deriving state from the bucket

Before the loop, the shard performs one s3fs::s3_dir_info() on devxy-rpkgs-binaries/<arch>/<codename>/latest/src/contrib and reads the etag column. It fetches CRAN's PACKAGES once for the latest version and published MD5sum of every package. A package is still outstanding if and only if the object at <pkg>_<version>.tar.gz has an ETag equal to CRAN's MD5sum for that version, which is the definition check_s3_root_package() already applies one package at a time.

# one paginated listing instead of ~2900 sequential HEAD requests per shard
info <- s3fs::s3_dir_info(slot_dir)
etag <- setNames(gsub('^"|"$', "", info$etag), basename(info$uri))

key <- sprintf("%s_%s.tar.gz", mine, cran_version[mine])
# keep a package when no object exists yet, or when the object is still
# byte-identical to CRAN's source; drop it once a real binary is published
mine <- mine[is.na(etag[key]) | etag[key] == cran_md5[key]]

This is the whole resume mechanism. There is no progress file, no volume, and no database cursor. A restarted shard recomputes ground truth and continues where it stopped, and it is correct even when a sibling shard, a process-updates cron, or a manual just rebuild completed something in the meantime.

Three properties make this the right source of truth:

• It is what the build itself checks. Any other store can disagree with the bucket; this one cannot.
• It is agent-independent. .crow/weekly-rebuild-missing.yaml mounts no volumes:, unlike .crow/build-all-versions.yaml:132-133, so /mnt/cache is per-job and cannot carry progress anyway.
• It costs one listing. cranlike's s3 fork already does exactly this call against this bucket at ~24 000 objects, so the approach is proven at the required scale.

It must read ETags rather than the slot index's Built field, which is how local/packages-to-build.R:104-130 answers the same question. Under this design the index is not rewritten until the dependent re-index pipeline runs (section 5), so mid-run it cannot reflect the current run's progress.

Packages that genuinely fail to build re-publish their CRAN source, so they stay outstanding and would be retried on every restart. That is already handled upstream: bincraft::filter_packages_with_errors() (R/build_binaries.R:1018, :1143) drops anything with error_occurred = TRUE, and store_build_metadata = TRUE is passed on every call. No additional poison-pill filter is needed here.

Only the flat src/contrib path is considered. The rebuild call passes no is_r_minor_sensitive, so it defaults to FALSE and only ever targets the flat path; the resume filter matches that scope deliberately.

4. Give each shard a wall-clock budget

local/rebuild-missing.R takes a budget, defaulting to 20 hours, and breaks out of the loop once it is exceeded:

# exit cleanly rather than being killed, so the dependent re-index still runs
if (difftime(Sys.time(), started, units = "hours") > budget_hours) {
  cat(sprintf("Budget of %sh reached after %d/%d packages; stopping cleanly\n", budget_hours, i, n))
  break
}

It exits 0 and reports how much of the slice it covered. Every run then has a terminating condition, the re-index and purge always fire, and the remainder is picked up by the next run with no bookkeeping, because section 3 recomputes the outstanding set from scratch.

5. Move the re-index and purge into .crow/weekly-rebuild-reindex.yaml

Three shards per slot means three concurrent upload_package_index() calls on the same S3 prefix. cranlike::update_PACKAGES() lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work. The re-index steps (.crow/weekly-rebuild-missing.yaml:171-176) and the purge step (:187-207) therefore leave that file entirely.

The new file carries:

depends_on:
  - weekly-rebuild-missing
runs_on: [success, failure]

runs_on: [success, failure] validates as a workflow-level key under crow lint, so a failing shard no longer withholds the re-index. The file uses the same 18-row matrix and the same when: gating as weekly-rebuild-missing, so it only re-indexes slots that actually ran. Each row re-indexes the flat slot and every per-minor slot. scripts/purge_cdn_zone.sh runs once on a single row, because all hostnames share pull zone 3857050 and 18 identical zone purges would be waste.

Failure behaviour

case	today	after
one package errors	tryCatch logs, loop continues	unchanged
a shard fails outright	purge runs, re-index does not	re-index and purge run via runs_on
a shard exceeds its budget	cannot happen, runs until killed	exits 0, re-index and purge run
a shard is killed	nothing runs	still nothing; trigger the re-index pipeline alone
a shard restarts	re-walks the list, HEAD per package	one listing, resumes at the first outstanding package

The known cost of depends_on being file-level rather than row-level: on the weekly cron no slot is re-indexed until the slowest of all 54 jobs finishes. The 20-hour budget bounds that at roughly one day.

Out of scope

• build-all-versions still cannot rebuild source fallbacks, because local/build-all.R:113-122 drops every version with any single_builds row for the platform and arch, which is precisely the source-fallback set. That is a separate change.
• Bunny Perma-Cache eviction. scripts/purge_cdn_zone.sh purges the regular edge cache only; see the note in CLAUDE.md and issue history.
• The audit that produces the rebuild list is unchanged.

Verification

• crow lint .crow/ passes for both pipeline files.
• local/rebuild-missing.R gets unit coverage in local/tests/ for the two pure pieces: the interleaved slice (disjoint, covering, deterministic) and the outstanding-set filter (source-served ETag kept, binary ETag dropped, absent object kept).
• A single-slot manual run of alpine-324-amd64 shard 1 confirms the listing shortcut against the live bucket, and that the reported outstanding count is close to the 8 917 measured above divided by three.
• Restarting that shard mid-run confirms it resumes rather than replaying, by comparing the outstanding count it reports on the second start.