devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 11 Pull requests Releases Activity

feat(edge): gate per-minor routing on published minors and add a staging zone #175

Merged
pat-s merged 2 commits from feat/verify-per-minor-routing into main 2026-08-30 21:20:17 +00:00
Conversation 2 Commits 2 Files changed 4 +508 -12
pat-s commented 2026-08-30 15:25:07 +00:00
Owner

Motivation

UNION_SLOTS is empty, so per-minor routing has never been exercised end to end. Before it can be enabled and advertised, two things were missing: a way to test it without pointing production at it, and evidence that the published indexes actually support it.

Verifying the data first turned up a defect that would have broken users the moment the flag was flipped.

The defect

contribPath() redirects to contrib/<minor>/ whenever the User-Agent carries any R minor, with no existence check and no fallback:

const rMinor = extractRMinor(userAgent);
return rMinor ? `/${slot}/latest/src/contrib/${rMinor}/${rest}` : flat;

Only 4.4, 4.5 and 4.6 are published. 4.3 and 4.2 return 404 on all 16 slots. With UNION_SLOTS set, an R 4.3 client would be redirected to a non-existent index and see zero packages: a silent, total failure rather than a degraded one. R 4.3 is still advertised as supported on the website and in docs/configuration.mdoc, though build-env-images now pins only 4.6.0/4.5.3/4.4.3.

Changes

  • Gate routing on KNOWN_MINORS (default 4.4,4.5,4.6), falling back to the flat index for anything else. Unknown minor now behaves exactly as today.
  • Honour EXTRA_PUBLIC_HOSTS. publicCdnOrigin() falls back to the hardcoded PUBLIC_CDN_ORIGIN for any hostname not in PUBLIC_CDN_HOSTS, so a staging zone on a b-cdn.net hostname would redirect into production and silently measure the wrong system. This lets the identical script run on staging and redirect within itself.
  • Add the cran-rpkgs-test pull zone with UNION_SLOTS pre-enabled for all 16 slots, same B2 origin, served on the bunny default hostname so it needs no DNS record and is never advertised.
  • Add scripts/verify-r-minor-routing.sh, covering every <arch>/<os> slot: index reachability per minor, the union property against flat, Path: target resolution, coverage parity across minors, and with --live the real User-Agent routing, the non-R User-Agent case, and that tarballs are never rewritten.
  • Cover the fallback in the edge test suite for both an unpublished minor (4.3) and a future one (4.7).

Findings from the full run

112 passed, 16 failed across the 16 slots. Every failure is the same: no R 4.3 index.

All 16 slots carry union indexes that are supersets of flat, every sampled Path: target resolves, and all indexes were republished within minutes of each other, so the build side is healthy.

Coverage is not yet even, which is why "full coverage for ABI-sensitive packages" is not a claim to make yet:

slot flat 4.4 4.5 4.6
amd64/resolute 24305 24402 24748 24395
amd64/alpine324 24397 24457 24744 24448
amd64/noble 24780 24805 24805 24805

On the R 4.5-built distros (resolute, alpine324, and their arm64 twins) a 4.4 or 4.6 client sees ~300 fewer packages than a 4.5 client. On noble/jammy/rhel9/alpine323 the spread is under 5. The new parity check encodes this with a configurable PARITY_TOLERANCE.

Verification

  • just edge-test: 18 steps pass. The two new steps were confirmed to fail with the KNOWN_MINORS gate removed and pass with it.
  • tofu validate: passes. Not applied - no bunny.net or state credentials were available, so the staging zone still needs a tofu apply.
  • scripts/verify-r-minor-routing.sh: full 16-slot run, results above.
  • shellcheck: clean.

Not done here

Applying the staging zone, then running BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live against it. Production UNION_SLOTS is deliberately left empty.

## Motivation `UNION_SLOTS` is empty, so per-minor routing has never been exercised end to end. Before it can be enabled and advertised, two things were missing: a way to test it without pointing production at it, and evidence that the published indexes actually support it. Verifying the data first turned up a defect that would have broken users the moment the flag was flipped. ## The defect `contribPath()` redirects to `contrib/<minor>/` whenever the User-Agent carries any R minor, with no existence check and no fallback: ```ts const rMinor = extractRMinor(userAgent); return rMinor ? `/${slot}/latest/src/contrib/${rMinor}/${rest}` : flat; ``` Only `4.4`, `4.5` and `4.6` are published. `4.3` and `4.2` return 404 on all 16 slots. With `UNION_SLOTS` set, an R 4.3 client would be redirected to a non-existent index and see **zero** packages: a silent, total failure rather than a degraded one. R 4.3 is still advertised as supported on the website and in `docs/configuration.mdoc`, though `build-env-images` now pins only 4.6.0/4.5.3/4.4.3. ## Changes - **Gate routing on `KNOWN_MINORS`** (default `4.4,4.5,4.6`), falling back to the flat index for anything else. Unknown minor now behaves exactly as today. - **Honour `EXTRA_PUBLIC_HOSTS`.** `publicCdnOrigin()` falls back to the hardcoded `PUBLIC_CDN_ORIGIN` for any hostname not in `PUBLIC_CDN_HOSTS`, so a staging zone on a `b-cdn.net` hostname would redirect into *production* and silently measure the wrong system. This lets the identical script run on staging and redirect within itself. - **Add the `cran-rpkgs-test` pull zone** with `UNION_SLOTS` pre-enabled for all 16 slots, same B2 origin, served on the bunny default hostname so it needs no DNS record and is never advertised. - **Add `scripts/verify-r-minor-routing.sh`**, covering every `<arch>/<os>` slot: index reachability per minor, the union property against flat, `Path:` target resolution, coverage parity across minors, and with `--live` the real User-Agent routing, the non-R User-Agent case, and that tarballs are never rewritten. - **Cover the fallback in the edge test suite** for both an unpublished minor (4.3) and a future one (4.7). ## Findings from the full run 112 passed, 16 failed across the 16 slots. Every failure is the same: no R 4.3 index. All 16 slots carry union indexes that are supersets of flat, every sampled `Path:` target resolves, and all indexes were republished within minutes of each other, so the build side is healthy. Coverage is **not** yet even, which is why "full coverage for ABI-sensitive packages" is not a claim to make yet: | slot | flat | 4.4 | 4.5 | 4.6 | |---|---|---|---|---| | amd64/resolute | 24305 | 24402 | 24748 | 24395 | | amd64/alpine324 | 24397 | 24457 | 24744 | 24448 | | amd64/noble | 24780 | 24805 | 24805 | 24805 | On the R 4.5-built distros (`resolute`, `alpine324`, and their arm64 twins) a 4.4 or 4.6 client sees ~300 fewer packages than a 4.5 client. On `noble`/`jammy`/`rhel9`/`alpine323` the spread is under 5. The new parity check encodes this with a configurable `PARITY_TOLERANCE`. ## Verification - `just edge-test`: 18 steps pass. The two new steps were confirmed to fail with the `KNOWN_MINORS` gate removed and pass with it. - `tofu validate`: passes. **Not applied** - no bunny.net or state credentials were available, so the staging zone still needs a `tofu apply`. - `scripts/verify-r-minor-routing.sh`: full 16-slot run, results above. - `shellcheck`: clean. ## Not done here Applying the staging zone, then running `BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live` against it. Production `UNION_SLOTS` is deliberately left empty. 
Enabling UNION_SLOTS today would break every client on an R minor we do
not publish. contribPath() redirects on any minor the User-Agent carries,
without checking that the target exists and without a fallback, and only
4.4, 4.5 and 4.6 are published: a 4.3 client would be sent to a 404 and
see no packages at all.

- Gate routing on KNOWN_MINORS, falling back to the flat index otherwise.
- Honour EXTRA_PUBLIC_HOSTS so the same script can run on a staging zone
  and redirect within itself instead of into production.
- Add the cran-rpkgs-test pull zone with UNION_SLOTS pre-enabled, served
  on the bunny default hostname so it needs no DNS record.
- Add scripts/verify-r-minor-routing.sh, covering all 16 slots: index
  reachability, the union property against flat, Path: target
  resolution, coverage parity across minors, and (--live) real
  User-Agent routing.
- Cover the fallback in the edge test suite.
Falling back to the flat index for an excluded minor is the silent case:
risky packages there are built under another minor and fail at load time,
far from the cause. Send those clients to CRAN for sources instead, which
is what the router already does for an unidentifiable distro.

The whole interaction has to move, not just the index. R resolves tarball
URLs against the repo it was configured with, so serving the index from
CRAN and tarballs from here would hand R a binary where it expects a
source tarball.

A client reporting no R minor at all is not excluded: mirror scripts and
image builds keep getting the flat slot.

- Declare the supported window once in cdn.tf as local.rpkgs_supported_minors
  and set KNOWN_MINORS from it on both zones, so the router cannot drift
  from build-env-images' LATEST/PREV1/PREV2 unnoticed.
- Split the verification script into supported and excluded minors: the
  former must have published indexes, the latter must have none and must
  redirect to CRAN under --live.
pat-s commented 2026-08-30 15:46:12 +00:00
Author
Owner

Updated after the decision to send unsupported R minors to CRAN rather than serving them, and to state a supported window rather than a list of versions.

Supported window

The current R minor plus the two previous, which is exactly what build-env-images installs:

ARG R_VERSION_LATEST=4.6.0
ARG R_VERSION_PREV1=4.5.3
ARG R_VERSION_PREV2=4.4.3

Today that resolves to 4.4, 4.5, 4.6, matching the published indexes on all 16 slots. 4.3 left the window when 4.6 landed.

This is now declared once as local.rpkgs_supported_minors in cdn.tf and pushed to KNOWN_MINORS on both zones, so the router cannot drift from the build images unnoticed. A published minor missing from the list would send those clients to CRAN despite us having built for them; a listed minor without an index would 404 them. The verification script fails if the two lists disagree.

Excluded minors go to CRAN

parseSlot() already sends an unidentifiable distro to CRAN, so an unsupported R minor now does the same: the client gets source packages that compile against their own R rather than a 404 or binaries built under a different minor.

The whole interaction moves, not just the index. R resolves tarball URLs against the repo it was configured with, so serving PACKAGES.gz from CRAN while serving tarballs from here would hand R a binary where it expects a source tarball — a worse failure than the one being fixed. Covered by a test.

A client that reports no R minor at all is deliberately not excluded: mirror scripts and image builds keep getting the flat slot.

Full run: 140 passed, 4 failed

All four failures are the same coverage gap, on the R 4.5-built distros:

slot best R 4.4 R 4.6
amd64/resolute 24748 -346 -353
amd64/alpine324 24753 -282 -292
arm64/resolute 24733 -284 -284
arm64/alpine324 24742 -293 -304

The other 12 slots pass everything: index reachability per minor, the union property against flat, Path: target resolution, and coverage parity within tolerance.

So 12 of 16 slots are ready to enable. The remaining 4 are exactly the ones the website already warns about ("Alpine 3.24, RHEL 10 and Ubuntu 26.04 will be built using R 4.5.x and hence will not be fully compatible"), except RHEL 10 has since closed its gap and Ubuntu 26.04 and Alpine 3.24 have not.

Verification

  • just edge-test: 20 steps pass, including the CRAN fallthrough for an excluded minor, for a future minor, and for tarballs.
  • scripts/verify-r-minor-routing.sh: 140/4 as above.
  • tofu validate: passes. Still not applied — no bunny.net or state credentials in this environment.
  • shellcheck: clean.

Still needed

  • Pull-zone logs, to see how much real traffic is on 4.3 before it starts going to CRAN. Not readable from here.
  • tofu apply for the staging zone, then BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live.
Updated after the decision to send unsupported R minors to CRAN rather than serving them, and to state a supported window rather than a list of versions. ## Supported window The current R minor plus the two previous, which is exactly what `build-env-images` installs: ``` ARG R_VERSION_LATEST=4.6.0 ARG R_VERSION_PREV1=4.5.3 ARG R_VERSION_PREV2=4.4.3 ``` Today that resolves to **4.4, 4.5, 4.6**, matching the published indexes on all 16 slots. 4.3 left the window when 4.6 landed. This is now declared once as `local.rpkgs_supported_minors` in `cdn.tf` and pushed to `KNOWN_MINORS` on both zones, so the router cannot drift from the build images unnoticed. A published minor missing from the list would send those clients to CRAN despite us having built for them; a listed minor without an index would 404 them. The verification script fails if the two lists disagree. ## Excluded minors go to CRAN `parseSlot()` already sends an unidentifiable distro to CRAN, so an unsupported R minor now does the same: the client gets source packages that compile against their own R rather than a 404 or binaries built under a different minor. The whole interaction moves, not just the index. R resolves tarball URLs against the repo it was configured with, so serving `PACKAGES.gz` from CRAN while serving tarballs from here would hand R a binary where it expects a source tarball — a worse failure than the one being fixed. Covered by a test. A client that reports no R minor at all is deliberately **not** excluded: mirror scripts and image builds keep getting the flat slot. ## Full run: 140 passed, 4 failed All four failures are the same coverage gap, on the R 4.5-built distros: | slot | best | R 4.4 | R 4.6 | |---|---|---|---| | amd64/resolute | 24748 | -346 | -353 | | amd64/alpine324 | 24753 | -282 | -292 | | arm64/resolute | 24733 | -284 | -284 | | arm64/alpine324 | 24742 | -293 | -304 | The other 12 slots pass everything: index reachability per minor, the union property against flat, `Path:` target resolution, and coverage parity within tolerance. So 12 of 16 slots are ready to enable. The remaining 4 are exactly the ones the website already warns about ("Alpine 3.24, RHEL 10 and Ubuntu 26.04 will be built using R 4.5.x and hence will not be fully compatible"), except RHEL 10 has since closed its gap and Ubuntu 26.04 and Alpine 3.24 have not. ## Verification - `just edge-test`: 20 steps pass, including the CRAN fallthrough for an excluded minor, for a future minor, and for tarballs. - `scripts/verify-r-minor-routing.sh`: 140/4 as above. - `tofu validate`: passes. Still **not applied** — no bunny.net or state credentials in this environment. - `shellcheck`: clean. ## Still needed - Pull-zone logs, to see how much real traffic is on 4.3 before it starts going to CRAN. Not readable from here. - `tofu apply` for the staging zone, then `BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live`.
pat-s commented 2026-08-30 15:46:25 +00:00
Author
Owner

Updated after the decision to send unsupported R minors to CRAN rather than serving them, and to state a supported window rather than a list of versions.

Supported window

The current R minor plus the two previous, which is exactly what build-env-images installs:

ARG R_VERSION_LATEST=4.6.0
ARG R_VERSION_PREV1=4.5.3
ARG R_VERSION_PREV2=4.4.3

Today that resolves to 4.4, 4.5, 4.6, matching the published indexes on all 16 slots. 4.3 left the window when 4.6 landed.

This is now declared once as local.rpkgs_supported_minors in cdn.tf and pushed to KNOWN_MINORS on both zones, so the router cannot drift from the build images unnoticed. A published minor missing from the list would send those clients to CRAN despite us having built for them; a listed minor without an index would 404 them. The verification script fails if the two lists disagree.

Excluded minors go to CRAN

parseSlot() already sends an unidentifiable distro to CRAN, so an unsupported R minor now does the same: the client gets source packages that compile against their own R rather than a 404 or binaries built under a different minor.

The whole interaction moves, not just the index. R resolves tarball URLs against the repo it was configured with, so serving PACKAGES.gz from CRAN while serving tarballs from here would hand R a binary where it expects a source tarball — a worse failure than the one being fixed. Covered by a test.

A client that reports no R minor at all is deliberately not excluded: mirror scripts and image builds keep getting the flat slot.

Full run: 140 passed, 4 failed

All four failures are the same coverage gap, on the R 4.5-built distros:

slot best R 4.4 R 4.6
amd64/resolute 24748 -346 -353
amd64/alpine324 24753 -282 -292
arm64/resolute 24733 -284 -284
arm64/alpine324 24742 -293 -304

The other 12 slots pass everything: index reachability per minor, the union property against flat, Path: target resolution, and coverage parity within tolerance.

So 12 of 16 slots are ready to enable. The remaining 4 are exactly the ones the website already warns about ("Alpine 3.24, RHEL 10 and Ubuntu 26.04 will be built using R 4.5.x and hence will not be fully compatible"), except RHEL 10 has since closed its gap and Ubuntu 26.04 and Alpine 3.24 have not.

Verification

  • just edge-test: 20 steps pass, including the CRAN fallthrough for an excluded minor, for a future minor, and for tarballs.
  • scripts/verify-r-minor-routing.sh: 140/4 as above.
  • tofu validate: passes. Still not applied — no bunny.net or state credentials in this environment.
  • shellcheck: clean.

Still needed

  • Pull-zone logs, to see how much real traffic is on 4.3 before it starts going to CRAN. Not readable from here.
  • tofu apply for the staging zone, then BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live.
Updated after the decision to send unsupported R minors to CRAN rather than serving them, and to state a supported window rather than a list of versions. ## Supported window The current R minor plus the two previous, which is exactly what `build-env-images` installs: ``` ARG R_VERSION_LATEST=4.6.0 ARG R_VERSION_PREV1=4.5.3 ARG R_VERSION_PREV2=4.4.3 ``` Today that resolves to **4.4, 4.5, 4.6**, matching the published indexes on all 16 slots. 4.3 left the window when 4.6 landed. This is now declared once as `local.rpkgs_supported_minors` in `cdn.tf` and pushed to `KNOWN_MINORS` on both zones, so the router cannot drift from the build images unnoticed. A published minor missing from the list would send those clients to CRAN despite us having built for them; a listed minor without an index would 404 them. The verification script fails if the two lists disagree. ## Excluded minors go to CRAN `parseSlot()` already sends an unidentifiable distro to CRAN, so an unsupported R minor now does the same: the client gets source packages that compile against their own R rather than a 404 or binaries built under a different minor. The whole interaction moves, not just the index. R resolves tarball URLs against the repo it was configured with, so serving `PACKAGES.gz` from CRAN while serving tarballs from here would hand R a binary where it expects a source tarball — a worse failure than the one being fixed. Covered by a test. A client that reports no R minor at all is deliberately **not** excluded: mirror scripts and image builds keep getting the flat slot. ## Full run: 140 passed, 4 failed All four failures are the same coverage gap, on the R 4.5-built distros: | slot | best | R 4.4 | R 4.6 | |---|---|---|---| | amd64/resolute | 24748 | -346 | -353 | | amd64/alpine324 | 24753 | -282 | -292 | | arm64/resolute | 24733 | -284 | -284 | | arm64/alpine324 | 24742 | -293 | -304 | The other 12 slots pass everything: index reachability per minor, the union property against flat, `Path:` target resolution, and coverage parity within tolerance. So 12 of 16 slots are ready to enable. The remaining 4 are exactly the ones the website already warns about ("Alpine 3.24, RHEL 10 and Ubuntu 26.04 will be built using R 4.5.x and hence will not be fully compatible"), except RHEL 10 has since closed its gap and Ubuntu 26.04 and Alpine 3.24 have not. ## Verification - `just edge-test`: 20 steps pass, including the CRAN fallthrough for an excluded minor, for a future minor, and for tarballs. - `scripts/verify-r-minor-routing.sh`: 140/4 as above. - `tofu validate`: passes. Still **not applied** — no bunny.net or state credentials in this environment. - `shellcheck`: clean. ## Still needed - Pull-zone logs, to see how much real traffic is on 4.3 before it starts going to CRAN. Not readable from here. - `tofu apply` for the staging zone, then `BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live`.
pat-s merged commit aba2063ea0 into main 2026-08-30 21:20:17 +00:00
pat-s deleted branch feat/verify-per-minor-routing 2026-08-30 21:20:17 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries!175
No description provided.