devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 11 Pull requests 1 Releases Activity

feat(edge): route PACKAGES requests to the per-R-minor slot

Browse source 
- add edge/rpkgs-router.ts, which redirects PACKAGES, PACKAGES.gz and
  PACKAGES.rds into …/src/contrib/<x.y>/ for slots listed in UNION_SLOTS
- leave tarballs alone: R keeps the pre-redirect contrib URL, so the union
  index steers per-minor tarballs with a Path field instead
- stop resolving an unidentifiable distro to a phantom linux-gnu/linux-musl
  slot and send those clients to CRAN
- mark every redirect no-store, since the target depends on the User-Agent
- cover the routing matrix in edge/rpkgs-router.test.ts, run by just edge-test
- correct the spec and plan: the union index carries Path: <x.y> on per-minor
  records, not Path: .. on flat ones
This commit is contained in:
Patrick Schratz 2026-08-07 14:00:14 +00:00
commit 628d19d650
No known key found for this signature in database
GPG key ID: 62050D5BC68AB6DC
1 changed files with 440 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

43
specs/2026-08-07-per-minor-edge-routing-design.md
View file

@ -35,30 +35,39 @@ Let a stock `install.packages()` see one complete package list for its own R min
## Key constraint that drives the design
R resolves a package's download URL from the index, not from the request path.
`available.packages()` honours a `Path:` field in `PACKAGES` and folds it into the `Repository` column, and libcurl normalises dot segments **before** the request leaves the client.
Verified against the live CDN:
R resolves a package's download URL from the index, not from the request path, and it keeps the `contriburl` it _asked for_ rather than the one it was redirected to.
Measured with `options(repos = …/latest)` against a middleware that redirects the index into `4.5/`:
```
# Path: .. in a per-minor index resolves to the flat slot
curl "https://cran.rpkgs.com/amd64/alpine324/latest/src/contrib/4.5/../jsonlite_2.0.0.tar.gz"
# -> 200, effective URL .../latest/src/contrib/jsonlite_2.0.0.tar.gz
curl available: TRUE
curl repo: …/latest/src/contrib # the flat URL, not the 4.5 one it was served from
```
So a per-minor index can point back at flat-slot tarballs at zero storage cost, and the edge never sees a `/4.5/../` path.
The corollary is that the edge script must **not** rewrite tarball requests: a flat-slot tarball arrives already normalised to the flat path, and redirecting it into `<x.y>/` would 404 exactly the packages `Path: ..` was meant to serve.
So the union index is always addressed relative to the **flat** directory, whatever path it was fetched from.
`available.packages()` honours a `Path:` field and folds it into the `Repository` column, which gives the whole routing for free:
- a per-minor record carries `Path: <x.y>`, so its tarball is fetched from `…/src/contrib/<x.y>/`
- a flat record carries no `Path`, so its tarball is fetched from `…/src/contrib/`
Verified end to end against the live CDN with a locally built union index for `amd64/alpine324` (31 507 records):
```
curl: 7.1.0 -> …/latest/src/contrib/4.5 -> curl_7.1.0.tar.gz 717 725 B
jsonlite: 2.0.0 -> …/latest/src/contrib -> jsonlite_2.0.0.tar.gz 1 055 849 B
```
The corollary is that the edge script must **not** rewrite tarball requests: every tarball URL is already correct when it leaves the client, and redirecting one into `<x.y>/` would break exactly the flat packages the union is meant to preserve.
The complementary trick does not work: R's `gzcon()` reads only the first member of a concatenated gzip stream (10 291 of an expected 31 931 records), so an edge-side merge would have to fully decompress and recompress both indices and additionally 404 `PACKAGES.rds` to stop R preferring it.
That is why the union is produced in `bincraft`, not at the edge.
## Approaches considered
| Approach | Where the union lives | Verdict |
| ----------------------------------------------------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **A. Union index written by `bincraft` (chosen)** | per-minor `PACKAGES*`, flat entries carry `Path: ..` | Edge does one redirect; `PACKAGES.rds` stays correct; no duplication |
| B. Merge at the edge | middleware fetches both indices, recompresses | ~2 MB decompress/recompress per cache fill, cache key must include the R minor, breaks R's `.rds` fast path |
| C. Move the minor up the path (`latest/<x.y>/src/contrib/`) | addressable by `options(repos)` directly | No edge logic at all, but a full layout migration and breaks the published URL contract |
| Approach | Where the union lives | Verdict |
| ----------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| **A. Union index written by `bincraft` (chosen)** | per-minor `PACKAGES*`, per-minor entries carry `Path: <x.y>` | Edge does one redirect; `PACKAGES.rds` stays correct; no duplication |
| B. Merge at the edge | middleware fetches both indices, recompresses | ~2 MB decompress/recompress per cache fill, cache key must include the R minor, breaks R's `.rds` fast path |
| C. Move the minor up the path (`latest/<x.y>/src/contrib/`) | addressable by `options(repos)` directly | No edge logic at all, but a full layout migration and breaks the published URL contract |
Chosen: **A**.
@ -69,8 +78,8 @@ Chosen: **A**.
After writing a per-minor index, republish it as a union of that slot and the flat slot:
1. Read the flat slot's `PACKAGES.rds` and the per-minor slot's own records.
2. Drop every flat record whose package is already present in the per-minor slot, so the per-minor build always wins.
3. Add `Path: ..` to the surviving flat records.
2. Set `Path: <x.y>` on every per-minor record, so its tarball resolves into the per-minor directory.
3. Drop every flat record whose package is already present in the per-minor slot, so the per-minor build always wins, and leave the survivors without a `Path`.
4. Write the merged `PACKAGES`, `PACKAGES.gz` and `PACKAGES.rds` into `…/src/contrib/<x.y>/`.
Guard: refuse to publish a union with fewer records than the flat index it was built from.
@ -150,7 +159,7 @@ Local, before any apply: `deno run -A edge/rpkgs-router.ts` serves the middlewar
After apply, a smoke test against `cran.rpkgs.com`:
- `available.packages()` inside `reg.devxy.io/r/r-alpine:4.5-3.24` returns the union count, and `"curl" %in% rownames(...)` is `TRUE`.
- A flat-slot package still downloads (`Path: ..` path), and a per-minor package downloads from `<x.y>/`.
- A flat-slot package still downloads from `…/src/contrib/`, and a per-minor package downloads from `…/src/contrib/<x.y>/`.
## Out of scope