feat(edge): gate per-minor routing on published minors and add a staging zone (#175)
All checks were successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
All checks were successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
## Motivation
`UNION_SLOTS` is empty, so per-minor routing has never been exercised end to end. Before it can be enabled and advertised, two things were missing: a way to test it without pointing production at it, and evidence that the published indexes actually support it.
Verifying the data first turned up a defect that would have broken users the moment the flag was flipped.
## The defect
`contribPath()` redirects to `contrib/<minor>/` whenever the User-Agent carries any R minor, with no existence check and no fallback:
```ts
const rMinor = extractRMinor(userAgent);
return rMinor ? `/${slot}/latest/src/contrib/${rMinor}/${rest}` : flat;
```
Only `4.4`, `4.5` and `4.6` are published. `4.3` and `4.2` return 404 on all 16 slots. With `UNION_SLOTS` set, an R 4.3 client would be redirected to a non-existent index and see **zero** packages: a silent, total failure rather than a degraded one. R 4.3 is still advertised as supported on the website and in `docs/configuration.mdoc`, though `build-env-images` now pins only 4.6.0/4.5.3/4.4.3.
## Changes
- **Gate routing on `KNOWN_MINORS`** (default `4.4,4.5,4.6`), falling back to the flat index for anything else. Unknown minor now behaves exactly as today.
- **Honour `EXTRA_PUBLIC_HOSTS`.** `publicCdnOrigin()` falls back to the hardcoded `PUBLIC_CDN_ORIGIN` for any hostname not in `PUBLIC_CDN_HOSTS`, so a staging zone on a `b-cdn.net` hostname would redirect into *production* and silently measure the wrong system. This lets the identical script run on staging and redirect within itself.
- **Add the `cran-rpkgs-test` pull zone** with `UNION_SLOTS` pre-enabled for all 16 slots, same B2 origin, served on the bunny default hostname so it needs no DNS record and is never advertised.
- **Add `scripts/verify-r-minor-routing.sh`**, covering every `<arch>/<os>` slot: index reachability per minor, the union property against flat, `Path:` target resolution, coverage parity across minors, and with `--live` the real User-Agent routing, the non-R User-Agent case, and that tarballs are never rewritten.
- **Cover the fallback in the edge test suite** for both an unpublished minor (4.3) and a future one (4.7).
## Findings from the full run
112 passed, 16 failed across the 16 slots. Every failure is the same: no R 4.3 index.
All 16 slots carry union indexes that are supersets of flat, every sampled `Path:` target resolves, and all indexes were republished within minutes of each other, so the build side is healthy.
Coverage is **not** yet even, which is why "full coverage for ABI-sensitive packages" is not a claim to make yet:
| slot | flat | 4.4 | 4.5 | 4.6 |
|---|---|---|---|---|
| amd64/resolute | 24305 | 24402 | 24748 | 24395 |
| amd64/alpine324 | 24397 | 24457 | 24744 | 24448 |
| amd64/noble | 24780 | 24805 | 24805 | 24805 |
On the R 4.5-built distros (`resolute`, `alpine324`, and their arm64 twins) a 4.4 or 4.6 client sees ~300 fewer packages than a 4.5 client. On `noble`/`jammy`/`rhel9`/`alpine323` the spread is under 5. The new parity check encodes this with a configurable `PARITY_TOLERANCE`.
## Verification
- `just edge-test`: 18 steps pass. The two new steps were confirmed to fail with the `KNOWN_MINORS` gate removed and pass with it.
- `tofu validate`: passes. **Not applied** - no bunny.net or state credentials were available, so the staging zone still needs a `tofu apply`.
- `scripts/verify-r-minor-routing.sh`: full 16-slot run, results above.
- `shellcheck`: clean.
## Not done here
Applying the staging zone, then running `BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live` against it. Production `UNION_SLOTS` is deliberately left empty.
Reviewed-on: #175
This commit is contained in:
parent
eeebef8edb
commit
aba2063ea0
1 changed files with 508 additions and 12 deletions
|
|
@ -17,6 +17,8 @@ const UNION_SLOTS = 'amd64/alpine324';
|
|||
|
||||
const UA_R45_MUSL = 'R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl)';
|
||||
const UA_R46_MUSL = 'R (4.6.0 x86_64-pc-linux-musl x86_64 linux-musl)';
|
||||
const UA_R43_MUSL = 'R (4.3.3 x86_64-pc-linux-musl x86_64 linux-musl)';
|
||||
const UA_R47_MUSL = 'R (4.7.0 x86_64-pc-linux-musl x86_64 linux-musl)';
|
||||
const UA_R45_ALPINE = 'R/4.5.3 R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl) Alpine Linux 3.24';
|
||||
const UA_R45_RESOLUTE = 'R/4.5.3 (Ubuntu 26.04) (aarch64-unknown-linux-gnu aarch64 linux-gnu)';
|
||||
const UA_R45_FUTURE_UBUNTU =
|
||||
|
|
@ -96,6 +98,33 @@ Deno.test('rpkgs-router', async (t) => {
|
|||
assertEquals(res.location, `https://cran.rpkgs.com${SLOT}/4.6/PACKAGES.gz`);
|
||||
});
|
||||
|
||||
// We publish binaries only for the supported window. An excluded minor has
|
||||
// no slot we can serve safely, so it goes to CRAN for sources rather than
|
||||
// to a 404 or to binaries built under another minor.
|
||||
await t.step('sends an excluded R minor to CRAN for the index', async () => {
|
||||
const res = await probe(`${SLOT}/PACKAGES.gz`, UA_R43_MUSL);
|
||||
assertEquals(res.location, 'https://cran.r-project.org/src/contrib/PACKAGES.gz');
|
||||
});
|
||||
|
||||
await t.step('sends a future R minor to CRAN too', async () => {
|
||||
const res = await probe(`${SLOT}/PACKAGES.gz`, UA_R47_MUSL);
|
||||
assertEquals(res.location, 'https://cran.r-project.org/src/contrib/PACKAGES.gz');
|
||||
});
|
||||
|
||||
// The index and the tarballs R resolves against it have to come from the
|
||||
// same place. Serving one from CRAN and the other from here would hand R a
|
||||
// binary where it expects a source tarball.
|
||||
await t.step('sends an excluded minor to CRAN for tarballs as well', async () => {
|
||||
const res = await probe(`${SLOT}/foo_1.0.tar.gz`, UA_R43_MUSL);
|
||||
assertEquals(res.location, 'https://cran.r-project.org/src/contrib/foo_1.0.tar.gz');
|
||||
});
|
||||
|
||||
await t.step('leaves an excluded minor alone on a slot outside UNION_SLOTS', async () => {
|
||||
const res = await probe(`${OTHER_SLOT}/PACKAGES.gz`, UA_R43_MUSL);
|
||||
assertEquals(res.location, null);
|
||||
assertEquals(res.status, 200);
|
||||
});
|
||||
|
||||
await t.step('routes PACKAGES and PACKAGES.rds too', async () => {
|
||||
for (const file of ['PACKAGES', 'PACKAGES.rds']) {
|
||||
const res = await probe(`${SLOT}/${file}`, UA_R45_MUSL);
|
||||
|
|
@ -146,18 +175,12 @@ Deno.test('rpkgs-router', async (t) => {
|
|||
|
||||
await t.step('resolves Ubuntu 26.04 to the resolute slot', async () => {
|
||||
const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_RESOLUTE);
|
||||
assertEquals(
|
||||
res.location,
|
||||
'https://cran.rpkgs.com/arm64/resolute/latest/src/contrib/PACKAGES.gz',
|
||||
);
|
||||
assertEquals(res.location, 'https://cran.rpkgs.com/arm64/resolute/latest/src/contrib/PACKAGES.gz');
|
||||
});
|
||||
|
||||
await t.step('resolves a future Ubuntu release from its codename', async () => {
|
||||
const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_FUTURE_UBUNTU);
|
||||
assertEquals(
|
||||
res.location,
|
||||
'https://cran.rpkgs.com/arm64/dynamic-dugong/latest/src/contrib/PACKAGES.gz',
|
||||
);
|
||||
assertEquals(res.location, 'https://cran.rpkgs.com/arm64/dynamic-dugong/latest/src/contrib/PACKAGES.gz');
|
||||
});
|
||||
|
||||
await t.step('sends an unidentifiable distro to CRAN', async () => {
|
||||
|
|
|
|||
Loading…
Reference in a new issue