perf: seed pure-R binaries into a new slot, interleave shards, and stop builds before the timeout #242

Merged

pat-s merged 1 commit from perf/full-rebuild-seed-and-shards into main

2026-10-24 02:04:06 +00:00

pat-s commented

2026-10-24 01:58:41 +00:00

Owner

Speeds up the full rebuild for a new slot, e.g. the upcoming Alpine version.

Changes

Seed pure-R binaries (local/seed-pure-r.R, local/seed-helpers.R), opt-in via the new SEED_FROM variable on build-all-versions-install-deps.
About 74% of the ~84k candidate versions have no compiled code and cost ~20s each to build; their R objects are identical across slots.
Before the build list is computed, each donor candidate is downloaded and verified, then copied server-side:
- donor must be the same OS family and arch;
- DESCRIPTION has Built (not a CRAN source fallback) with an empty platform and no libs/;
- built by the target's primary R minor;
- bincraft's tarball_abi_verdict() calls it portable across every installed minor (unknown means no seed).
Each copy gets the single_builds and abi_flat_safety rows an upload writes, so build-all skips it and the per-minor indexes keep it.
Re-runs skip existing objects and rewrite rows idempotently.
Interleaved shards in local/build-all.R via shard_slice(); contiguous alphabetical quarters ranged from 0.1h to 28h.
Shared build deadline: build-all-versions sets BUILD_DEADLINE from BUILD_BUDGET_HOURS (44h, repo timeout 50h); build-all.R and build-newer-r.R stop cleanly when it passes, so archiving still runs.
codename_platform() replaces the hard-coded codename-to-platform list.

Testing

New unit tests for the seed verdict, candidate filter, platform names and deadline; all local/tests pass.
Verdicts checked on real amd64/alpine324 tarballs: R6 2.6.1/2.5.1 and jsonstat seeded, cli refused as compiled.
crow lint .crow/ passes.
The seed has not run against B2/Postgres yet; the first bootstrap run is its end-to-end test.

Usage

Trigger build-all-versions with target_arch, OS, OS_VERSION and SEED_FROM="alpine324 alpine323".
The new Alpine version still needs adding to the OS_VERSION options once its image exists.

Speeds up the full rebuild for a new slot, e.g. the upcoming Alpine version. ## Changes - **Seed pure-R binaries** (`local/seed-pure-r.R`, `local/seed-helpers.R`), opt-in via the new `SEED_FROM` variable on `build-all-versions-install-deps`. About 74% of the ~84k candidate versions have no compiled code and cost ~20s each to build; their R objects are identical across slots. Before the build list is computed, each donor candidate is downloaded and verified, then copied server-side: - donor must be the same OS family and arch; - DESCRIPTION has `Built` (not a CRAN source fallback) with an empty platform and no `libs/`; - built by the target's primary R minor; - bincraft's `tarball_abi_verdict()` calls it portable across every installed minor (unknown means no seed). Each copy gets the `single_builds` and `abi_flat_safety` rows an upload writes, so build-all skips it and the per-minor indexes keep it. Re-runs skip existing objects and rewrite rows idempotently. - **Interleaved shards** in `local/build-all.R` via `shard_slice()`; contiguous alphabetical quarters ranged from 0.1h to 28h. - **Shared build deadline**: `build-all-versions` sets `BUILD_DEADLINE` from `BUILD_BUDGET_HOURS` (44h, repo timeout 50h); `build-all.R` and `build-newer-r.R` stop cleanly when it passes, so archiving still runs. - `codename_platform()` replaces the hard-coded codename-to-platform list. ## Testing - New unit tests for the seed verdict, candidate filter, platform names and deadline; all `local/tests` pass. - Verdicts checked on real amd64/alpine324 tarballs: R6 2.6.1/2.5.1 and jsonstat seeded, cli refused as compiled. - `crow lint .crow/` passes. - The seed has not run against B2/Postgres yet; the first bootstrap run is its end-to-end test. ## Usage Trigger `build-all-versions` with `target_arch`, `OS`, `OS_VERSION` and `SEED_FROM="alpine324 alpine323"`. The new Alpine version still needs adding to the `OS_VERSION` options once its image exists.

pat-s added 1 commit

2026-10-24 01:58:41 +00:00

perf: seed pure-R binaries into a new slot, interleave shards, and stop builds before the timeout cdfdcd035d

- Add local/seed-pure-r.R, opt-in via SEED_FROM on install-deps: copy pure-R
  binaries server-side from a slot of the same OS family and arch, after
  verifying each is a binary, has no compiled code, was built by the primary
  R minor, and passes bincraft's cross-minor ABI check; write the
  single_builds and abi_flat_safety rows an upload would.
- Interleave build-all shards with shard_slice() instead of contiguous
  alphabetical blocks, which ranged from 0.1h to 28h per shard.
- Share one BUILD_DEADLINE (BUILD_BUDGET_HOURS, default 44h) across every
  build pass of a build-all-versions step so long runs stop cleanly and
  still archive and index what they built.
- Derive the single_builds platform with codename_platform() instead of a
  hard-coded list.

pat-s merged commit e23bd3e539 into main

2026-10-24 02:04:06 +00:00

pat-s deleted branch perf/full-rebuild-seed-and-shards

2026-10-24 02:04:07 +00:00

pat-s referenced this pull request from a commit

2026-10-24 02:04:07 +00:00

perf: seed pure-R binaries into a new slot, interleave shards, and stop builds before the timeout (#242)