# Design: Sharding and resuming the weekly rebuild

Date: 2026-08-12
Status: Approved (pending spec review)

## Problem

`weekly-rebuild-missing` runs one job per `<os>-<arch>` and walks that slot's rebuild list serially in a single `R -q -e` invocation (`.crow/weekly-rebuild-missing.yaml:165`).
Until 2026-08-09 that was cheap, because every source fallback was skipped as "already built" and the list was effectively empty.
Since bincraft #105/#106/#107 and build-cran-binaries #159 the gate works, and the lists are now large.

Share of records whose object is byte-identical to CRAN's source, measured against `cran.r-project.org` MD5s on 2026-08-12:

| slot               | records | source-served |        share |
| ------------------ | ------: | ------------: | -----------: |
| `amd64/resolute`   |  24 212 |        15 023 |        62.1% |
| `arm64/resolute`   |  24 291 |        13 670 |        56.3% |
| `arm64/alpine324`  |  24 328 |         9 514 |        39.2% |
| `amd64/alpine324`  |  24 343 |         8 917 |        36.7% |
| `arm64/rhel10`     |  24 695 |         5 384 |        21.9% |
| `amd64/rhel10`     |  24 881 |         4 712 |        19.2% |
| 12 remaining slots | ~24 700 |  850 to 2 130 | 3.5% to 8.7% |

A single serial job cannot absorb that.
Pipeline 10910 (`weekly_rebuild_missing:alpine-324-amd64`) started on 2026-08-09, ran for roughly two days, reached `[8692/23885] cholera`, and was killed there.

Two distinct failures follow from that shape.

**No parallelism.** The work is embarrassingly parallel across packages, but one job does all of it.

**No resumability, and no clean stopping point.** The loop has no terminating condition other than exhausting the list, so the only way to stop it is a kill.
A restarted run re-reads the same list and walks it from the first entry.
It skips completed packages via `check_s3_root_package()`, but that costs a CRAN version resolution and an S3 `HEAD` per package, thousands of times, before it reaches new work.
Worse, a kill is not a pipeline failure: the `Purge CDN cache` step is guarded by `when: status: [success, failure]` (`.crow/weekly-rebuild-missing.yaml:206-207`), and on 10910 it produced no output at all.
So the ~4 600 binaries that run did publish stayed hidden behind stale edge copies.

## Goal

Turn each slot's rebuild into bounded, parallel, restartable units, without introducing state that can disagree with the bucket.

## Design

### 1. Shard the matrix three ways

Each of the 18 `OS`/`ARCH` rows in `.crow/weekly-rebuild-missing.yaml` gains `SPLIT_INTO: 3` and `SPLIT_INDEX: 1|2|3`, giving 54 rows.
This mirrors `.crow/build-all-versions.yaml:57-98`, which already shards its matrix four ways per arch.

Routing needs no change.
The cron filter `cron: weekly-rebuild-missing-${OS}-${ARCH}` and the manual `evaluate: weekly_rebuild_missing == "${OS}-${ARCH}"` both match all three shards of a slot.
Placement stays on the `rpkgs-${ARCH}` group label, so shards queue against available capacity rather than oversubscribing it.

### 2. Extract the loop into `local/rebuild-missing.R`

The build is currently a single ~1 500-character `R -q -e` argument.
Shard arithmetic and resume logic do not belong in a YAML string, and none of it is testable there.
The loop moves to `local/rebuild-missing.R`, invoked as `Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX`, mirroring `local/build-all.R`.
Its body is unchanged in substance: read `/tmp/rebuild_pkgs.txt`, subtract `local/excluded-packages.json`, loop with `tryCatch` around `bincraft::build_binary_package()`.

The slice is **interleaved**, not contiguous:

```r
# the list is alphabetical and build cost clusters by name (Rcpp*, Bioc*,
# rstan*), so contiguous thirds would be badly unbalanced
mine <- pkgs[seq(split_index, length(pkgs), by = split_into)]
```

`local/build-all.R:64` uses contiguous chunks via `cut()`.
That is fine there because its list is every CRAN package and version, so the chunks average out.
Here the list is a filtered backlog in which expensive families sit adjacent, so interleaving is the better default.
Interleaving also makes each shard's `[i/n]` progress representative of the slot as a whole.

### 3. Resume by re-deriving state from the bucket

Before the loop, the shard performs one `s3fs::s3_dir_info()` on `devxy-rpkgs-binaries/<arch>/<codename>/latest/src/contrib` and reads the `etag` column.
It fetches CRAN's `PACKAGES` once for the latest version and published `MD5sum` of every package.
A package is still outstanding if and only if the object at `<pkg>_<version>.tar.gz` has an ETag equal to CRAN's `MD5sum` for that version, which is the definition `check_s3_root_package()` already applies one package at a time.

```r
# one paginated listing instead of ~2900 sequential HEAD requests per shard
info <- s3fs::s3_dir_info(slot_dir)
etag <- setNames(gsub('^"|"$', "", info$etag), basename(info$uri))

key <- sprintf("%s_%s.tar.gz", mine, cran_version[mine])
# keep a package when no object exists yet, or when the object is still
# byte-identical to CRAN's source; drop it once a real binary is published
mine <- mine[is.na(etag[key]) | etag[key] == cran_md5[key]]
```

This is the whole resume mechanism.
There is no progress file, no volume, and no database cursor.
A restarted shard recomputes ground truth and continues where it stopped, and it is correct even when a sibling shard, a `process-updates` cron, or a manual `just rebuild` completed something in the meantime.

Three properties make this the right source of truth:

- **It is what the build itself checks.** Any other store can disagree with the bucket; this one cannot.
- **It is agent-independent.** `.crow/weekly-rebuild-missing.yaml` mounts no `volumes:`, unlike `.crow/build-all-versions.yaml:132-133`, so `/mnt/cache` is per-job and cannot carry progress anyway.
- **It costs one listing.** `cranlike`'s `s3` fork already does exactly this call against this bucket at ~24 000 objects, so the approach is proven at the required scale.

It must read ETags rather than the slot index's `Built` field, which is how `local/packages-to-build.R:104-130` answers the same question.
Under this design the index is not rewritten until the dependent re-index pipeline runs (section 5), so mid-run it cannot reflect the current run's progress.

Packages that genuinely fail to build re-publish their CRAN source, so they stay outstanding and would be retried on every restart.
That is already handled upstream: `bincraft::filter_packages_with_errors()` (`R/build_binaries.R:1018`, `:1143`) drops anything with `error_occurred = TRUE`, and `store_build_metadata = TRUE` is passed on every call.
No additional poison-pill filter is needed here.

Only the flat `src/contrib` path is considered.
The rebuild call passes no `is_r_minor_sensitive`, so it defaults to `FALSE` and only ever targets the flat path; the resume filter matches that scope deliberately.

### 4. Give each shard a wall-clock budget

`local/rebuild-missing.R` takes a budget, defaulting to 20 hours, and breaks out of the loop once it is exceeded:

```r
# exit cleanly rather than being killed, so the dependent re-index still runs
if (difftime(Sys.time(), started, units = "hours") > budget_hours) {
  cat(sprintf("Budget of %sh reached after %d/%d packages; stopping cleanly\n", budget_hours, i, n))
  break
}
```

It exits 0 and reports how much of the slice it covered.
Every run then has a terminating condition, the re-index and purge always fire, and the remainder is picked up by the next run with no bookkeeping, because section 3 recomputes the outstanding set from scratch.

### 5. Move the re-index and purge into `.crow/weekly-rebuild-reindex.yaml`

Three shards per slot means three concurrent `upload_package_index()` calls on the same S3 prefix.
`cranlike::update_PACKAGES()` lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work.
The re-index steps (`.crow/weekly-rebuild-missing.yaml:171-176`) and the purge step (`:187-207`) therefore leave that file entirely.

The new file carries:

```yaml
depends_on:
  - weekly-rebuild-missing
runs_on: [success, failure]
```

`runs_on: [success, failure]` validates as a workflow-level key under `crow lint`, so a failing shard no longer withholds the re-index.
The file uses the same 18-row matrix and the same `when:` gating as `weekly-rebuild-missing`, so it only re-indexes slots that actually ran.
Each row re-indexes the flat slot and every per-minor slot.
`scripts/purge_cdn_zone.sh` runs once on a single row, because all hostnames share pull zone `3857050` and 18 identical zone purges would be waste.

## Failure behaviour

| case                       | today                               | after                                                 |
| -------------------------- | ----------------------------------- | ----------------------------------------------------- |
| one package errors         | `tryCatch` logs, loop continues     | unchanged                                             |
| a shard fails outright     | purge runs, re-index does not       | re-index and purge run via `runs_on`                  |
| a shard exceeds its budget | cannot happen, runs until killed    | exits 0, re-index and purge run                       |
| a shard is killed          | nothing runs                        | still nothing; trigger the re-index pipeline alone    |
| a shard restarts           | re-walks the list, HEAD per package | one listing, resumes at the first outstanding package |

The known cost of `depends_on` being file-level rather than row-level: on the weekly cron no slot is re-indexed until the slowest of all 54 jobs finishes.
The 20-hour budget bounds that at roughly one day.

## Out of scope

- `build-all-versions` still cannot rebuild source fallbacks, because `local/build-all.R:113-122` drops every version with any `single_builds` row for the platform and arch, which is precisely the source-fallback set. That is a separate change.
- Bunny Perma-Cache eviction. `scripts/purge_cdn_zone.sh` purges the regular edge cache only; see the note in `CLAUDE.md` and issue history.
- The audit that produces the rebuild list is unchanged.

## Verification

- `crow lint .crow/` passes for both pipeline files.
- `local/rebuild-missing.R` gets unit coverage in `local/tests/` for the two pure pieces: the interleaved slice (disjoint, covering, deterministic) and the outstanding-set filter (source-served ETag kept, binary ETag dropped, absent object kept).
- A single-slot manual run of `alpine-324-amd64` shard 1 confirms the listing shortcut against the live bucket, and that the reported outstanding count is close to the 8 917 measured above divided by three.
- Restarting that shard mid-run confirms it resumes rather than replaying, by comparing the outstanding count it reports on the second start.