feat(rebuild): shard the weekly rebuild and make each shard resumable #163
1 changed files with 167 additions and 0 deletions
docs(rebuild): add design for sharding and resuming the weekly rebuild
commit
02e3966523
167
specs/2026-08-12-shard-weekly-rebuild-design.md
Normal file
167
specs/2026-08-12-shard-weekly-rebuild-design.md
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
# Design: Sharding and resuming the weekly rebuild
|
||||
|
||||
Date: 2026-08-12
|
||||
Status: Approved (pending spec review)
|
||||
|
||||
## Problem
|
||||
|
||||
`weekly-rebuild-missing` runs one job per `<os>-<arch>` and walks that slot's rebuild list serially in a single `R -q -e` invocation (`.crow/weekly-rebuild-missing.yaml:165`).
|
||||
Until 2026-08-09 that was cheap, because every source fallback was skipped as "already built" and the list was effectively empty.
|
||||
Since bincraft #105/#106/#107 and build-cran-binaries #159 the gate works, and the lists are now large.
|
||||
|
||||
Share of records whose object is byte-identical to CRAN's source, measured against `cran.r-project.org` MD5s on 2026-08-12:
|
||||
|
||||
| slot | records | source-served | share |
|
||||
| ------------------ | ------: | ------------: | -----------: |
|
||||
| `amd64/resolute` | 24 212 | 15 023 | 62.1% |
|
||||
| `arm64/resolute` | 24 291 | 13 670 | 56.3% |
|
||||
| `arm64/alpine324` | 24 328 | 9 514 | 39.2% |
|
||||
| `amd64/alpine324` | 24 343 | 8 917 | 36.7% |
|
||||
| `arm64/rhel10` | 24 695 | 5 384 | 21.9% |
|
||||
| `amd64/rhel10` | 24 881 | 4 712 | 19.2% |
|
||||
| 12 remaining slots | ~24 700 | 850 to 2 130 | 3.5% to 8.7% |
|
||||
|
||||
A single serial job cannot absorb that.
|
||||
Pipeline 10910 (`weekly_rebuild_missing:alpine-324-amd64`) started on 2026-08-09, ran for roughly two days, reached `[8692/23885] cholera`, and was killed there.
|
||||
|
||||
Two distinct failures follow from that shape.
|
||||
|
||||
**No parallelism.** The work is embarrassingly parallel across packages, but one job does all of it.
|
||||
|
||||
**No resumability, and no clean stopping point.** The loop has no terminating condition other than exhausting the list, so the only way to stop it is a kill.
|
||||
A restarted run re-reads the same list and walks it from the first entry.
|
||||
It skips completed packages via `check_s3_root_package()`, but that costs a CRAN version resolution and an S3 `HEAD` per package, thousands of times, before it reaches new work.
|
||||
Worse, a kill is not a pipeline failure: the `Purge CDN cache` step is guarded by `when: status: [success, failure]` (`.crow/weekly-rebuild-missing.yaml:206-207`), and on 10910 it produced no output at all.
|
||||
So the ~4 600 binaries that run did publish stayed hidden behind stale edge copies.
|
||||
|
||||
## Goal
|
||||
|
||||
Turn each slot's rebuild into bounded, parallel, restartable units, without introducing state that can disagree with the bucket.
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Shard the matrix three ways
|
||||
|
||||
Each of the 18 `OS`/`ARCH` rows in `.crow/weekly-rebuild-missing.yaml` gains `SPLIT_INTO: 3` and `SPLIT_INDEX: 1|2|3`, giving 54 rows.
|
||||
This mirrors `.crow/build-all-versions.yaml:57-98`, which already shards its matrix four ways per arch.
|
||||
|
||||
Routing needs no change.
|
||||
The cron filter `cron: weekly-rebuild-missing-${OS}-${ARCH}` and the manual `evaluate: weekly_rebuild_missing == "${OS}-${ARCH}"` both match all three shards of a slot.
|
||||
Placement stays on the `rpkgs-${ARCH}` group label, so shards queue against available capacity rather than oversubscribing it.
|
||||
|
||||
### 2. Extract the loop into `local/rebuild-missing.R`
|
||||
|
||||
The build is currently a single ~1 500-character `R -q -e` argument.
|
||||
Shard arithmetic and resume logic do not belong in a YAML string, and none of it is testable there.
|
||||
The loop moves to `local/rebuild-missing.R`, invoked as `Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX`, mirroring `local/build-all.R`.
|
||||
Its body is unchanged in substance: read `/tmp/rebuild_pkgs.txt`, subtract `local/excluded-packages.json`, loop with `tryCatch` around `bincraft::build_binary_package()`.
|
||||
|
||||
The slice is **interleaved**, not contiguous:
|
||||
|
||||
```r
|
||||
# the list is alphabetical and build cost clusters by name (Rcpp*, Bioc*,
|
||||
# rstan*), so contiguous thirds would be badly unbalanced
|
||||
mine <- pkgs[seq(split_index, length(pkgs), by = split_into)]
|
||||
```
|
||||
|
||||
`local/build-all.R:64` uses contiguous chunks via `cut()`.
|
||||
That is fine there because its list is every CRAN package and version, so the chunks average out.
|
||||
Here the list is a filtered backlog in which expensive families sit adjacent, so interleaving is the better default.
|
||||
Interleaving also makes each shard's `[i/n]` progress representative of the slot as a whole.
|
||||
|
||||
### 3. Resume by re-deriving state from the bucket
|
||||
|
||||
Before the loop, the shard performs one `s3fs::s3_dir_info()` on `devxy-rpkgs-binaries/<arch>/<codename>/latest/src/contrib` and reads the `etag` column.
|
||||
It fetches CRAN's `PACKAGES` once for the latest version and published `MD5sum` of every package.
|
||||
A package is still outstanding if and only if the object at `<pkg>_<version>.tar.gz` has an ETag equal to CRAN's `MD5sum` for that version, which is the definition `check_s3_root_package()` already applies one package at a time.
|
||||
|
||||
```r
|
||||
# one paginated listing instead of ~2900 sequential HEAD requests per shard
|
||||
info <- s3fs::s3_dir_info(slot_dir)
|
||||
etag <- setNames(gsub('^"|"$', "", info$etag), basename(info$uri))
|
||||
|
||||
key <- sprintf("%s_%s.tar.gz", mine, cran_version[mine])
|
||||
# keep a package when no object exists yet, or when the object is still
|
||||
# byte-identical to CRAN's source; drop it once a real binary is published
|
||||
mine <- mine[is.na(etag[key]) | etag[key] == cran_md5[key]]
|
||||
```
|
||||
|
||||
This is the whole resume mechanism.
|
||||
There is no progress file, no volume, and no database cursor.
|
||||
A restarted shard recomputes ground truth and continues where it stopped, and it is correct even when a sibling shard, a `process-updates` cron, or a manual `just rebuild` completed something in the meantime.
|
||||
|
||||
Three properties make this the right source of truth:
|
||||
|
||||
- **It is what the build itself checks.** Any other store can disagree with the bucket; this one cannot.
|
||||
- **It is agent-independent.** `.crow/weekly-rebuild-missing.yaml` mounts no `volumes:`, unlike `.crow/build-all-versions.yaml:132-133`, so `/mnt/cache` is per-job and cannot carry progress anyway.
|
||||
- **It costs one listing.** `cranlike`'s `s3` fork already does exactly this call against this bucket at ~24 000 objects, so the approach is proven at the required scale.
|
||||
|
||||
It must read ETags rather than the slot index's `Built` field, which is how `local/packages-to-build.R:104-130` answers the same question.
|
||||
Under this design the index is not rewritten until the dependent re-index pipeline runs (section 5), so mid-run it cannot reflect the current run's progress.
|
||||
|
||||
Packages that genuinely fail to build re-publish their CRAN source, so they stay outstanding and would be retried on every restart.
|
||||
That is already handled upstream: `bincraft::filter_packages_with_errors()` (`R/build_binaries.R:1018`, `:1143`) drops anything with `error_occurred = TRUE`, and `store_build_metadata = TRUE` is passed on every call.
|
||||
No additional poison-pill filter is needed here.
|
||||
|
||||
Only the flat `src/contrib` path is considered.
|
||||
The rebuild call passes no `is_r_minor_sensitive`, so it defaults to `FALSE` and only ever targets the flat path; the resume filter matches that scope deliberately.
|
||||
|
||||
### 4. Give each shard a wall-clock budget
|
||||
|
||||
`local/rebuild-missing.R` takes a budget, defaulting to 20 hours, and breaks out of the loop once it is exceeded:
|
||||
|
||||
```r
|
||||
# exit cleanly rather than being killed, so the dependent re-index still runs
|
||||
if (difftime(Sys.time(), started, units = "hours") > budget_hours) {
|
||||
cat(sprintf("Budget of %sh reached after %d/%d packages; stopping cleanly\n", budget_hours, i, n))
|
||||
break
|
||||
}
|
||||
```
|
||||
|
||||
It exits 0 and reports how much of the slice it covered.
|
||||
Every run then has a terminating condition, the re-index and purge always fire, and the remainder is picked up by the next run with no bookkeeping, because section 3 recomputes the outstanding set from scratch.
|
||||
|
||||
### 5. Move the re-index and purge into `.crow/weekly-rebuild-reindex.yaml`
|
||||
|
||||
Three shards per slot means three concurrent `upload_package_index()` calls on the same S3 prefix.
|
||||
`cranlike::update_PACKAGES()` lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work.
|
||||
The re-index steps (`.crow/weekly-rebuild-missing.yaml:171-176`) and the purge step (`:187-207`) therefore leave that file entirely.
|
||||
|
||||
The new file carries:
|
||||
|
||||
```yaml
|
||||
depends_on:
|
||||
- weekly-rebuild-missing
|
||||
runs_on: [success, failure]
|
||||
```
|
||||
|
||||
`runs_on: [success, failure]` validates as a workflow-level key under `crow lint`, so a failing shard no longer withholds the re-index.
|
||||
The file uses the same 18-row matrix and the same `when:` gating as `weekly-rebuild-missing`, so it only re-indexes slots that actually ran.
|
||||
Each row re-indexes the flat slot and every per-minor slot.
|
||||
`scripts/purge_cdn_zone.sh` runs once on a single row, because all hostnames share pull zone `3857050` and 18 identical zone purges would be waste.
|
||||
|
||||
## Failure behaviour
|
||||
|
||||
| case | today | after |
|
||||
| -------------------------- | ----------------------------------- | ----------------------------------------------------- |
|
||||
| one package errors | `tryCatch` logs, loop continues | unchanged |
|
||||
| a shard fails outright | purge runs, re-index does not | re-index and purge run via `runs_on` |
|
||||
| a shard exceeds its budget | cannot happen, runs until killed | exits 0, re-index and purge run |
|
||||
| a shard is killed | nothing runs | still nothing; trigger the re-index pipeline alone |
|
||||
| a shard restarts | re-walks the list, HEAD per package | one listing, resumes at the first outstanding package |
|
||||
|
||||
The known cost of `depends_on` being file-level rather than row-level: on the weekly cron no slot is re-indexed until the slowest of all 54 jobs finishes.
|
||||
The 20-hour budget bounds that at roughly one day.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- `build-all-versions` still cannot rebuild source fallbacks, because `local/build-all.R:113-122` drops every version with any `single_builds` row for the platform and arch, which is precisely the source-fallback set. That is a separate change.
|
||||
- Bunny Perma-Cache eviction. `scripts/purge_cdn_zone.sh` purges the regular edge cache only; see the note in `CLAUDE.md` and issue history.
|
||||
- The audit that produces the rebuild list is unchanged.
|
||||
|
||||
## Verification
|
||||
|
||||
- `crow lint .crow/` passes for both pipeline files.
|
||||
- `local/rebuild-missing.R` gets unit coverage in `local/tests/` for the two pure pieces: the interleaved slice (disjoint, covering, deterministic) and the outstanding-set filter (source-served ETag kept, binary ETag dropped, absent object kept).
|
||||
- A single-slot manual run of `alpine-324-amd64` shard 1 confirms the listing shortcut against the live bucket, and that the reported outstanding count is close to the 8 917 measured above divided by three.
|
||||
- Restarting that shard mid-run confirms it resumes rather than replaying, by comparing the outstanding count it reports on the second start.
|
||||
Loading…
Reference in a new issue