build-cran-binaries/edge/rpkgs-router.test.ts
pat-s 2f732457d2
Some checks failed
ci/crow/manual/trial-build-registry/1 Pipeline is pending
ci/crow/manual/trial-build-registry/3 Pipeline is pending
ci/crow/manual/trial-build-registry/5 Pipeline is pending
ci/crow/manual/trial-build-registry/7 Pipeline is pending
ci/crow/manual/trial-build-registry/9 Pipeline is pending
ci/crow/manual/trial-build-registry/11 Pipeline is pending
ci/crow/manual/trial-build-registry/13 Pipeline is pending
ci/crow/manual/trial-build-registry/15 Pipeline is pending
ci/crow/manual/trial-build-registry/17 Pipeline is pending
ci/crow/manual/repair-built-stamp/2 Pipeline failed
ci/crow/manual/weekly-audit-missing/17 Pipeline was successful
ci/crow/manual/weekly-audit-missing/9 Pipeline was successful
ci/crow/manual/weekly-audit-missing/15 Pipeline was successful
ci/crow/manual/weekly-audit-missing/11 Pipeline was successful
ci/crow/manual/weekly-audit-missing/13 Pipeline was successful
ci/crow/manual/weekly-audit-missing/7 Pipeline was successful
ci/crow/manual/trial-build-registry/4 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/trial-build-registry/6 Pipeline was successful
ci/crow/manual/trial-build-registry/2 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/trial-build-registry/8 Pipeline was successful
ci/crow/manual/trial-build-registry/14 Pipeline was successful
ci/crow/manual/trial-build-registry/12 Pipeline was successful
ci/crow/manual/trial-build-registry/10 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/trial-build-registry/16 Pipeline was successful
ci/crow/manual/trial-build-registry/18 Pipeline was successful
ci/crow/manual/weekly-audit-missing/5 Pipeline was successful
ci/crow/manual/weekly-audit-missing/1 Pipeline was successful
ci/crow/manual/auto-apply-patches Pipeline was successful
ci/crow/manual/weekly-audit-missing/3 Pipeline was successful
ci/crow/manual/weekly-audit-missing/2 Pipeline was successful
ci/crow/manual/weekly-audit-missing/4 Pipeline was successful
ci/crow/manual/weekly-audit-missing/6 Pipeline was successful
ci/crow/manual/weekly-audit-missing/8 Pipeline was successful
ci/crow/manual/weekly-audit-missing/10 Pipeline was successful
ci/crow/manual/weekly-audit-missing/12 Pipeline was successful
ci/crow/manual/weekly-audit-missing/14 Pipeline was successful
ci/crow/manual/weekly-audit-missing/16 Pipeline was successful
ci/crow/manual/weekly-audit-missing/18 Pipeline was successful
ci/crow/manual/weekly-patch-proposals Pipeline was successful
ci/crow/manual/process-updates/6 Pipeline was successful
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/7 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/9 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/11 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/8 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/10 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/12 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/14 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/17 Pipeline failed
ci/crow/manual/weekly-rebuild-missing/6 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/5 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/18 Pipeline was successful
feat(edge): route PACKAGES requests to the per-R-minor slot (#152)
## Problem

`install.packages("curl")` fails in `reg.devxy.io/r/r-alpine:4.5-3.24` with "package 'curl' is not available for this version of R", on both arches.

`curl` is not missing from the repo: it is in `…/latest/src/contrib/4.5/` and `…/4.6/`, the per-minor slots that base R cannot address. The image's repo URL resolves to `…/latest/src/contrib`, whose index does not list it. On `amd64/alpine324` that is 2 886 packages invisible to `install.packages()` (23 on `amd64/noble`) — what issue #63 records as "missing binaries".

Two further findings while investigating:

- The middleware only ever rewrote the bare `cran.rpkgs.com/src/contrib/…` form, and that form was broken for every Linux client on a stock R user agent: `ALPINE_REGEX`/`UBUNTU_REGEX`/`RHEL_REGEX` need a Posit-style UA that carries the distro, so stock R fell through to `extractOs()` and got redirected to `/amd64/linux-musl/latest/…`, a slot that does not exist.
- `PACKAGES*` is served `cdn-cache: BYPASS` (bincraft uploads it `no-store`), so the middleware sees every index request and no purge is needed for routing changes to take effect.

## What this changes

**`edge/rpkgs-router.ts`** — the middleware, now a reviewed file in this repo rather than dashboard state. It routes `PACKAGES`, `PACKAGES.gz` and `PACKAGES.rds` into `…/src/contrib/<x.y>/` for slots listed in `UNION_SLOTS`, and nothing else.

Tarballs are deliberately left alone. R keeps the `contriburl` it *asked for*, not the one the redirect served it, so every tarball URL is resolved against the flat directory and the union index steers the per-minor ones with a `Path: <x.y>` field. Rewriting a tarball request here would send flat-slot packages into a directory that does not hold them.

Also in the script: the phantom `linux-gnu`/`linux-musl` fallback is gone (an unidentifiable distro goes to CRAN, as an unparseable UA already did), and every redirect carries `Cache-Control: no-store` since its target depends on the User-Agent. The macOS branches are unchanged.

**`cdn.tf`** — `bunnynet_compute_script.rpkgs_router` with `content = file("edge/rpkgs-router.ts")`, the `UNION_SLOTS` variable, and `middleware_script` pointing at the resource instead of the literal `29277`.

`UNION_SLOTS` is empty, so merging and applying this changes no client's behaviour. A slot is added only once bincraft has republished its per-minor index as a union (rpkgs/bincraft#97); routing to a raw per-minor index would hide every package it does not carry. Rolling back is a variable edit, not a deploy.

**`specs/`, `plans/`** — the design and the implementation plan, including the two approaches that were rejected (edge-side merge, moving the minor up the path) and why.

## Verification

`just edge-test` runs 13 routing cases against the SDK's local server, so what is tested is the artifact that gets deployed; pass-through cases proxy to the real origin. All pass.

End to end, with the middleware in front of a locally built union index for `amd64/alpine324` (31 507 records), inside the runtime image:

```
curl:     7.1.0  -> …/latest/src/contrib/4.5  -> curl_7.1.0.tar.gz      717 725 B
jsonlite: 2.0.0  -> …/latest/src/contrib      -> jsonlite_2.0.0.tar.gz  1 055 849 B
```

`tofu validate` passes. `tofu plan` has not been run: no `BUNNYNET_API_KEY` available in this environment.

## Before applying

The script pre-dates this configuration, so it must be adopted, not created:

```sh
tofu import bunnynet_compute_script.rpkgs_router 29277
tofu plan
```

The plan should show an in-place `content` update and no replacement of the pull zone. Without the import, tofu creates a second script and repoints the zone at it. Note that `name = "rpkgs-router"` will rename the existing script on apply.

## Not fixed here

`install.packages("curl")` on `alpine324` will now *resolve*, then fail to build: that slot's tarballs are byte-identical CRAN **source** tarballs (no `Meta/`, no `Built:` in DESCRIPTION) which the index nevertheless stamps `Built: R 4.5.3; …-linux-musl`. Sampled: `amd64/alpine324` 3/12 binary, `arm64/alpine324` 13/30, `amd64/noble` 12/12, `amd64/alpine323` 17/20. That slot needs a rebuild, tracked separately.

Reviewed-on: #152
2026-08-07 14:12:22 +00:00

160 lines
5.7 KiB
TypeScript

/**
* Routing matrix for `edge/rpkgs-router.ts`.
*
* The script is exercised through the SDK's local server rather than by
* importing its internals, so what is tested is the artifact that gets
* deployed. Requests that the script passes through are proxied to the real
* origin, which keeps the "no redirect" cases honest: they assert that the
* client reached the flat slot, not merely that no `Location` was set.
*
* Run with `just edge-test`.
*/
import { assertEquals } from 'jsr:@std/assert@1';
const SCRIPT = new URL('./rpkgs-router.ts', import.meta.url).pathname;
const BASE = 'http://127.0.0.1:8080';
const UNION_SLOTS = 'amd64/alpine324';
const UA_R45_MUSL = 'R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl)';
const UA_R46_MUSL = 'R (4.6.0 x86_64-pc-linux-musl x86_64 linux-musl)';
const UA_R45_ALPINE = 'R/4.5.3 R (4.5.3 x86_64-pc-linux-musl x86_64 linux-musl) Alpine Linux 3.24';
const UA_R45_DARWIN = 'R (4.5.1 aarch64-apple-darwin20 aarch64 darwin20)';
const UA_CURL = 'curl/8.0.1';
const SLOT = '/amd64/alpine324/latest/src/contrib';
const OTHER_SLOT = '/amd64/noble/latest/src/contrib';
interface Probe {
status: number;
location: string | null;
cacheControl: string | null;
}
async function probe(path: string, userAgent: string): Promise<Probe> {
const res = await fetch(BASE + path, {
headers: { 'User-Agent': userAgent },
redirect: 'manual',
});
await res.body?.cancel();
return {
status: res.status,
location: res.headers.get('location'),
cacheControl: res.headers.get('cache-control'),
};
}
/** Kill tolerantly: the child has already exited if the script failed to load. */
async function stopServer(child: Deno.ChildProcess): Promise<void> {
try {
child.kill();
} catch {
// already gone
}
await child.status;
}
async function startServer(): Promise<Deno.ChildProcess> {
const child = new Deno.Command(Deno.execPath(), {
args: ['run', '-A', SCRIPT],
env: { UNION_SLOTS },
stdout: 'null',
stderr: 'inherit',
}).spawn();
for (let attempt = 0; attempt < 150; attempt++) {
try {
const res = await fetch(`${BASE}/`, {
headers: { 'User-Agent': UA_CURL },
redirect: 'manual',
});
await res.body?.cancel();
return child;
} catch {
await new Promise((resolve) => setTimeout(resolve, 200));
}
}
await stopServer(child);
throw new Error('edge script did not start listening on ' + BASE);
}
Deno.test('rpkgs-router', async (t) => {
const server = await startServer();
try {
await t.step("routes an index request to the client's R minor", async () => {
const res = await probe(`${SLOT}/PACKAGES.gz`, UA_R45_MUSL);
assertEquals(res.status, 302);
assertEquals(res.location, `https://cran.rpkgs.com${SLOT}/4.5/PACKAGES.gz`);
});
await t.step('routes R 4.6 to its own slot', async () => {
const res = await probe(`${SLOT}/PACKAGES.gz`, UA_R46_MUSL);
assertEquals(res.location, `https://cran.rpkgs.com${SLOT}/4.6/PACKAGES.gz`);
});
await t.step('routes PACKAGES and PACKAGES.rds too', async () => {
for (const file of ['PACKAGES', 'PACKAGES.rds']) {
const res = await probe(`${SLOT}/${file}`, UA_R45_MUSL);
assertEquals(res.location, `https://cran.rpkgs.com${SLOT}/4.5/${file}`, `expected ${file} to be routed`);
}
});
await t.step('marks the redirect uncacheable', async () => {
const res = await probe(`${SLOT}/PACKAGES.gz`, UA_R45_MUSL);
assertEquals(res.cacheControl, 'no-store');
});
await t.step('leaves a slot outside UNION_SLOTS alone', async () => {
const res = await probe(`${OTHER_SLOT}/PACKAGES.gz`, UA_R45_MUSL);
assertEquals(res.location, null);
assertEquals(res.status, 200);
});
await t.step('never routes a tarball', async () => {
const res = await probe(`${SLOT}/jsonlite_2.0.0.tar.gz`, UA_R45_MUSL);
assertEquals(res.location, null);
assertEquals(res.status, 200);
});
await t.step('does not redirect a path already under a minor', async () => {
const res = await probe(`${SLOT}/4.5/PACKAGES.gz`, UA_R45_MUSL);
assertEquals(res.location, null);
assertEquals(res.status, 200);
});
await t.step('leaves a client without an R version alone', async () => {
const res = await probe(`${SLOT}/PACKAGES.gz`, UA_CURL);
assertEquals(res.location, null);
assertEquals(res.status, 200);
});
await t.step('resolves the bare root to slot and minor', async () => {
const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_ALPINE);
assertEquals(res.location, `https://cran.rpkgs.com${SLOT}/4.5/PACKAGES.gz`);
});
await t.step('sends an unidentifiable distro to CRAN', async () => {
const res = await probe('/src/contrib/PACKAGES.gz', UA_R45_MUSL);
assertEquals(res.location, 'https://cran.r-project.org/src/contrib/PACKAGES.gz');
});
await t.step('keeps the macOS rewrite', async () => {
const res = await probe('/src/contrib/foo_1.0.tar.gz', UA_R45_DARWIN);
assertEquals(res.location, 'https://cran.rpkgs.com/bin/macosx/big-sur-arm64/contrib/4.5/foo_1.0.tar.gz');
});
await t.step('keeps the macOS binary passthrough to CRAN', async () => {
const path = '/bin/macosx/big-sur-arm64/contrib/4.5/foo_1.0.tar.gz';
const res = await probe(path, UA_R45_DARWIN);
assertEquals(res.location, `https://cran.r-project.org${path}`);
});
await t.step('collapses duplicate slashes before matching', async () => {
const res = await probe(`/amd64/alpine324//latest/src/contrib//PACKAGES.gz`, UA_R45_MUSL);
assertEquals(res.location, `https://cran.rpkgs.com${SLOT}/4.5/PACKAGES.gz`);
});
} finally {
await stopServer(server);
}
});