devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 11 Pull requests Releases Activity

Enable and verify per-R-minor index routing #176

New issue
Open
opened 2026-08-30 15:25:31 +00:00 by pat-s · 1 comment
pat-s commented 2026-08-30 15:25:31 +00:00
Owner

Tracking issue for enabling per-R-minor index routing (UNION_SLOTS) on cran.rpkgs.com, and for verifying the claims we make about R version support once it is on.

Context and the full verification run are in #175. This issue is the checklist that has to be green before we enable routing in production or change the support wording on the website and in the docs.

State today

  • UNION_SLOTS is empty. Every client is served the flat, single-R-version index.
  • Per-minor union indexes are published for 4.4, 4.5, 4.6 on all 16 <arch>/<os> slots and are supersets of flat.
  • 4.3 and 4.2 are 404 on all 16 slots, while the website and docs/configuration.mdoc still advertise R 4.3 as fully supported.
  • build-env-images pins R_VERSION_LATEST=4.6.0, PREV1=4.5.3, PREV2=4.4.3. R 4.3 is not installed.

Before enabling

  • Merge #175 (KNOWN_MINORS fallback, EXTRA_PUBLIC_HOSTS, staging zone, verification script).
  • tofu apply the cran-rpkgs-test pull zone.
  • BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live passes on all 16 slots.
  • Confirm on staging that an R 4.3 User-Agent is served the flat index and still resolves packages, rather than a 404.
  • Confirm on staging that a non-R User-Agent is unaffected.
  • Confirm on staging that tarball requests are never rewritten into a per-minor directory.
  • Decide the cache_vary_headers = ["User-Agent"] question. The comment in cdn.tf says dropping it is a real win but should be its own change once routing is confirmed live, not a side effect of enabling it.

Enabling

  • Set production UNION_SLOTS, ideally one slot first rather than all 16.
  • Re-run scripts/verify-r-minor-routing.sh --live against cran.rpkgs.com.
  • Watch for 404s on contrib/<minor>/ paths in the pull zone logs.

Before claiming it

Coverage is currently uneven, so "full coverage for ABI-sensitive packages" is not yet accurate:

slot flat 4.4 4.5 4.6
amd64/resolute 24305 24402 24748 24395
amd64/alpine324 24397 24457 24744 24448
arm64/resolute 24345 24449 24733 24449
arm64/alpine324 24382 24449 24742 24438
amd64/noble 24780 24805 24805 24805

On the R 4.5-built distros a 4.4 or 4.6 client sees roughly 300 fewer packages. scripts/verify-r-minor-routing.sh fails these slots via PARITY_TOLERANCE.

  • Close the parity gap on resolute and alpine324 (both arches), or state the limitation explicitly rather than claiming full coverage.
  • Decide whether R 4.3 is still supported. It is advertised but not built.
  • Update docs/configuration.mdoc: it says the ABI heuristic "must first be developed", but bincraft's classifier is built and running. The supported set is 4.4/4.5/4.6, not 4.4/4.3.
  • Update the website support notes (rpkgs/website), which carry the same stale 4.4/4.3 wording.

Verify my work

Everything above came from an agent session. Worth an independent check:

  • The 404 claim for R 4.3: curl -sI https://cran.rpkgs.com/amd64/noble/latest/src/contrib/4.3/PACKAGES.gz
  • That routing is genuinely off today: the same PACKAGES.gz request under R/4.4.3 and R/4.5.3 User-Agents returns byte-identical bodies.
  • The parity numbers, by re-running the script.
  • That the two new edge tests fail without the KNOWN_MINORS gate. I checked this by reverting the gate and re-running; worth reproducing.
Tracking issue for enabling per-R-minor index routing (`UNION_SLOTS`) on `cran.rpkgs.com`, and for verifying the claims we make about R version support once it is on. Context and the full verification run are in #175. This issue is the checklist that has to be green before we enable routing in production or change the support wording on the website and in the docs. ## State today - `UNION_SLOTS` is empty. Every client is served the flat, single-R-version index. - Per-minor union indexes are published for `4.4`, `4.5`, `4.6` on all 16 `<arch>/<os>` slots and are supersets of flat. - `4.3` and `4.2` are 404 on all 16 slots, while the website and `docs/configuration.mdoc` still advertise R 4.3 as fully supported. - `build-env-images` pins `R_VERSION_LATEST=4.6.0`, `PREV1=4.5.3`, `PREV2=4.4.3`. R 4.3 is not installed. ## Before enabling - [ ] Merge #175 (`KNOWN_MINORS` fallback, `EXTRA_PUBLIC_HOSTS`, staging zone, verification script). - [ ] `tofu apply` the `cran-rpkgs-test` pull zone. - [ ] `BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live` passes on all 16 slots. - [ ] Confirm on staging that an R 4.3 User-Agent is served the flat index and still resolves packages, rather than a 404. - [ ] Confirm on staging that a non-R User-Agent is unaffected. - [ ] Confirm on staging that tarball requests are never rewritten into a per-minor directory. - [ ] Decide the `cache_vary_headers = ["User-Agent"]` question. The comment in `cdn.tf` says dropping it is a real win but should be its own change once routing is confirmed live, not a side effect of enabling it. ## Enabling - [ ] Set production `UNION_SLOTS`, ideally one slot first rather than all 16. - [ ] Re-run `scripts/verify-r-minor-routing.sh --live` against `cran.rpkgs.com`. - [ ] Watch for 404s on `contrib/<minor>/` paths in the pull zone logs. ## Before claiming it Coverage is currently uneven, so "full coverage for ABI-sensitive packages" is not yet accurate: | slot | flat | 4.4 | 4.5 | 4.6 | |---|---|---|---|---| | amd64/resolute | 24305 | 24402 | 24748 | 24395 | | amd64/alpine324 | 24397 | 24457 | 24744 | 24448 | | arm64/resolute | 24345 | 24449 | 24733 | 24449 | | arm64/alpine324 | 24382 | 24449 | 24742 | 24438 | | amd64/noble | 24780 | 24805 | 24805 | 24805 | On the R 4.5-built distros a 4.4 or 4.6 client sees roughly 300 fewer packages. `scripts/verify-r-minor-routing.sh` fails these slots via `PARITY_TOLERANCE`. - [ ] Close the parity gap on `resolute` and `alpine324` (both arches), or state the limitation explicitly rather than claiming full coverage. - [ ] Decide whether R 4.3 is still supported. It is advertised but not built. - [ ] Update `docs/configuration.mdoc`: it says the ABI heuristic "must first be developed", but `bincraft`'s classifier is built and running. The supported set is 4.4/4.5/4.6, not 4.4/4.3. - [ ] Update the website support notes (rpkgs/website), which carry the same stale 4.4/4.3 wording. ## Verify my work Everything above came from an agent session. Worth an independent check: - The 404 claim for R 4.3: `curl -sI https://cran.rpkgs.com/amd64/noble/latest/src/contrib/4.3/PACKAGES.gz` - That routing is genuinely off today: the same `PACKAGES.gz` request under `R/4.4.3` and `R/4.5.3` User-Agents returns byte-identical bodies. - The parity numbers, by re-running the script. - That the two new edge tests fail without the `KNOWN_MINORS` gate. I checked this by reverting the gate and re-running; worth reproducing.
pat-s commented 2026-08-30 15:46:46 +00:00
Author
Owner

Updated after the decision to send unsupported minors to CRAN and to publish a supported window. Supersedes the checklist items about the flat fallback.

Supported window

Current R minor plus the two previous — today 4.4, 4.5, 4.6. This is what build-env-images installs (R_VERSION_LATEST/PREV1/PREV2) and what is published on all 16 slots. It is declared once as local.rpkgs_supported_minors in cdn.tf and pushed to KNOWN_MINORS on both zones, so the router and the build images cannot drift apart silently.

Excluded minors are sent to CRAN for sources (index and tarballs), matching what parseSlot() already does for an unidentifiable distro.

Current state: 12 of 16 slots ready

Latest full run: 140 passed, 4 failed. Every failure is the coverage gap on the R 4.5-built distros:

slot best R 4.4 R 4.6
amd64/resolute 24748 -346 -353
amd64/alpine324 24753 -282 -292
arm64/resolute 24733 -284 -284
arm64/alpine324 24742 -293 -304

RHEL 10 has closed its gap since the website warning was written; Ubuntu 26.04 and Alpine 3.24 have not.

This suggests enabling UNION_SLOTS for the 12 passing slots first and leaving the other 4 until the gap closes, rather than all-or-nothing.

Revised checklist

  • Merge #175.
  • Read the pull-zone logs for the R-minor distribution of real traffic. This gates how aggressively 4.3 can be moved to CRAN, and it is the one input nobody has yet. Not readable without CDN credentials.
  • tofu apply the cran-rpkgs-test zone.
  • BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live — expect the same 140/4, plus the live routing and CRAN-redirect assertions.
  • Confirm on staging that a 4.3 client ends up installing from source successfully, end to end in a real R session. The unit tests assert the redirect; they do not prove install.packages() recovers.
  • Enable production UNION_SLOTS for the 12 passing slots.
  • Re-run --live against cran.rpkgs.com.
  • Close the parity gap on resolute/alpine324 (both arches), then add those 4 slots.
  • Revisit cache_vary_headers = ["User-Agent"] as its own change.
  • Update docs/configuration.mdoc and the website (rpkgs/website PR #47) to the 4.4-4.6 window. Deliberately held until routing is live, so the published claim matches what is served.

Verify my work

  • R 4.3 has no index: curl -sI https://cran.rpkgs.com/amd64/noble/latest/src/contrib/4.3/PACKAGES.gz
  • Routing is genuinely off today: the same PACKAGES.gz under R/4.4.3 and R/4.5.3 User-Agents returns byte-identical bodies.
  • Parity numbers: re-run the script.
  • The CRAN fallthrough tests fail if the KNOWN_MINORS gate is removed. I verified this by reverting the gate; worth reproducing.
  • The claim that 4.3 is compatible with 4.4 holds for the ~24,300 non-risky packages but is least certain for the ~529 abi_classify() marks risky, which is the subset the per-minor slots exist for. That reasoning drove the CRAN decision and deserves a second opinion.
Updated after the decision to send unsupported minors to CRAN and to publish a supported window. Supersedes the checklist items about the flat fallback. ## Supported window **Current R minor plus the two previous** — today 4.4, 4.5, 4.6. This is what `build-env-images` installs (`R_VERSION_LATEST`/`PREV1`/`PREV2`) and what is published on all 16 slots. It is declared once as `local.rpkgs_supported_minors` in `cdn.tf` and pushed to `KNOWN_MINORS` on both zones, so the router and the build images cannot drift apart silently. Excluded minors are sent to CRAN for sources (index *and* tarballs), matching what `parseSlot()` already does for an unidentifiable distro. ## Current state: 12 of 16 slots ready Latest full run: **140 passed, 4 failed**. Every failure is the coverage gap on the R 4.5-built distros: | slot | best | R 4.4 | R 4.6 | |---|---|---|---| | amd64/resolute | 24748 | -346 | -353 | | amd64/alpine324 | 24753 | -282 | -292 | | arm64/resolute | 24733 | -284 | -284 | | arm64/alpine324 | 24742 | -293 | -304 | RHEL 10 has closed its gap since the website warning was written; Ubuntu 26.04 and Alpine 3.24 have not. This suggests enabling `UNION_SLOTS` for the 12 passing slots first and leaving the other 4 until the gap closes, rather than all-or-nothing. ## Revised checklist - [ ] Merge #175. - [ ] **Read the pull-zone logs for the R-minor distribution of real traffic.** This gates how aggressively 4.3 can be moved to CRAN, and it is the one input nobody has yet. Not readable without CDN credentials. - [ ] `tofu apply` the `cran-rpkgs-test` zone. - [ ] `BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live` — expect the same 140/4, plus the live routing and CRAN-redirect assertions. - [ ] Confirm on staging that a 4.3 client ends up installing from source successfully, end to end in a real R session. The unit tests assert the redirect; they do not prove `install.packages()` recovers. - [ ] Enable production `UNION_SLOTS` for the 12 passing slots. - [ ] Re-run `--live` against `cran.rpkgs.com`. - [ ] Close the parity gap on resolute/alpine324 (both arches), then add those 4 slots. - [ ] Revisit `cache_vary_headers = ["User-Agent"]` as its own change. - [ ] Update `docs/configuration.mdoc` and the website (rpkgs/website PR #47) to the 4.4-4.6 window. Deliberately held until routing is live, so the published claim matches what is served. ## Verify my work - R 4.3 has no index: `curl -sI https://cran.rpkgs.com/amd64/noble/latest/src/contrib/4.3/PACKAGES.gz` - Routing is genuinely off today: the same `PACKAGES.gz` under `R/4.4.3` and `R/4.5.3` User-Agents returns byte-identical bodies. - Parity numbers: re-run the script. - The CRAN fallthrough tests fail if the `KNOWN_MINORS` gate is removed. I verified this by reverting the gate; worth reproducing. - The claim that 4.3 is compatible with 4.4 holds for the ~24,300 non-risky packages but is least certain for the ~529 `abi_classify()` marks risky, which is the subset the per-minor slots exist for. That reasoning drove the CRAN decision and deserves a second opinion.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
t3code/shard-weekly-rebuild
t3code/auto-apply-patches
t3code/propose-patches-and-tracking
pr/cdn-tf-for-each
feat/build-precondition-optimization
renovate/binwiederhier-ntfy-2.x
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries#176
No description provided.