build-cran-binaries/specs/2026-08-07-per-minor-edge-routing-design.md

Design: Routing clients to per-R-minor binary slots

Date: 2026-08-07 Status: Approved (pending spec review)

Problem

bincraft routes ABI-"risky" packages to a per-minor slot …/latest/src/contrib/<x.y>/ and indexes every directory independently (upload_package_index() calls cranlike::update_PACKAGES() on one prefix at a time). Nothing unions those indices, and contrib.url() only ever yields <repos>/src/contrib, so no value of options(repos) can address a per-minor slot. Only uvr resolves per-minor URLs, which means the per-minor slots are invisible to install.packages() by construction.

Measured on 2026-08-07:

slot	flat src/contrib	src/contrib/4.5	unique packages only in the per-minor slot
amd64/alpine324	21 640	3 310	2 886
amd64/noble	24 495	398	23

This is what issue #63 records as "missing binaries" on alpine324. The packages are not missing; they are in a directory base R cannot reach. The user-visible symptom in reg.devxy.io/r/r-alpine:4.5-3.24 is:

> install.packages("curl")
Warning message:
package 'curl' is not available for this version of R

A second, unrelated defect exists on the same slot and is out of scope here: many alpine324 tarballs are byte-identical CRAN source tarballs that the index nevertheless stamps Built: R 4.5.3; …-linux-musl. Routing exposes curl; only a rebuild of that slot makes it install.

Goal

Let a stock install.packages() see one complete package list for its own R minor, without duplicating tarballs and without an R-version-varying cache key anywhere in the CDN.

Key constraint that drives the design

R resolves a package's download URL from the index, not from the request path, and it keeps the contriburl it asked for rather than the one it was redirected to. Measured with options(repos = …/latest) against a middleware that redirects the index into 4.5/:

curl available: TRUE
curl repo: …/latest/src/contrib          # the flat URL, not the 4.5 one it was served from

So the union index is always addressed relative to the flat directory, whatever path it was fetched from. available.packages() honours a Path: field and folds it into the Repository column, which gives the whole routing for free:

• a per-minor record carries Path: <x.y>, so its tarball is fetched from …/src/contrib/<x.y>/
• a flat record carries no Path, so its tarball is fetched from …/src/contrib/

Verified end to end against the live CDN with a locally built union index for amd64/alpine324 (31 507 records):

curl:     7.1.0  -> …/latest/src/contrib/4.5   -> curl_7.1.0.tar.gz     717 725 B
jsonlite: 2.0.0  -> …/latest/src/contrib       -> jsonlite_2.0.0.tar.gz 1 055 849 B

The corollary is that the edge script must not rewrite tarball requests: every tarball URL is already correct when it leaves the client, and redirecting one into <x.y>/ would break exactly the flat packages the union is meant to preserve.

The complementary trick does not work: R's gzcon() reads only the first member of a concatenated gzip stream (10 291 of an expected 31 931 records), so an edge-side merge would have to fully decompress and recompress both indices and additionally 404 PACKAGES.rds to stop R preferring it. That is why the union is produced in bincraft, not at the edge.

Approaches considered

Approach	Where the union lives	Verdict
A. Union index written by bincraft (chosen)	per-minor PACKAGES*, per-minor entries carry Path: <x.y>	Edge does one redirect; PACKAGES.rds stays correct; no duplication
B. Merge at the edge	middleware fetches both indices, recompresses	~2 MB decompress/recompress per cache fill, cache key must include the R minor, breaks R's .rds fast path
C. Move the minor up the path (latest/<x.y>/src/contrib/)	addressable by options(repos) directly	No edge logic at all, but a full layout migration and breaks the published URL contract

Chosen: A.

Architecture

bincraft: union index (separate PR)

After writing a per-minor index, republish it as a union of that slot and the flat slot:

1. Read the flat slot's PACKAGES.rds and the per-minor slot's own records.
2. Set Path: <x.y> on every per-minor record, so its tarball resolves into the per-minor directory.
3. Drop every flat record whose package is already present in the per-minor slot, so the per-minor build always wins, and leave the survivors without a Path.
4. Write the merged PACKAGES, PACKAGES.gz and PACKAGES.rds into …/src/contrib/<x.y>/.

Guard: refuse to publish a union with fewer records than the flat index it was built from. A truncated union is worse than no union, because it silently removes packages from every client on that minor.

Edge script (this repo)

The script routes PACKAGES, PACKAGES.gz and PACKAGES.rds requests, and nothing else.

normalize path
parseClient(UA) -> { rMinor, arch, os }      # rMinor from "R (4.5.3 …)" or "R/4.5.3"
darwin branches                               # unchanged
if path is /{arch}/{os}/latest/src/contrib/PACKAGES*
    already under /<x.y>/ ? pass through      # loop guard
    rMinor known && slot in UNION_SLOTS ? 302 -> …/src/contrib/<rMinor>/PACKAGES*
    else pass through                         # flat slot, today's behaviour
if path is /src/contrib/…                     # bare root
    resolve arch + os; unknown -> 302 to CRAN
    then apply the same PACKAGES* rule
else pass through

Redirects carry Cache-Control: no-store. Every cacheable URL is therefore UA-independent, and no cache key has to vary by R version.

Repaired bare-root detection

The bare https://cran.rpkgs.com form is currently broken for every Linux client that uses a stock R user agent. ALPINE_REGEX, UBUNTU_REGEX and RHEL_REGEX only match a Posit-style user agent the user has to set by hand; stock R never carries the distro, so the script falls through to extractOs() and redirects to a slot that does not exist:

UA: R (4.5.3 x86_64-pc-linux-musl …)  ->  302 /amd64/linux-musl/latest/…   (404)
UA: R (4.5.3 x86_64-pc-linux-gnu  …)  ->  302 /amd64/linux-gnu/latest/…    (404)

The fallback to a phantom linux-musl / linux-gnu slot is removed. An unidentifiable distro redirects to CRAN, which is the existing behaviour for an unparseable user agent. The R minor is always present in a stock user agent, so per-minor routing itself does not depend on distro detection.

Rollout gate

UNION_SLOTS is a bunnynet_compute_script_variable listing the slots whose per-minor index is already a union. It is empty by default, so deploying the script changes nothing until bincraft has backfilled a slot, and a rollback is a variable edit rather than a code deploy. All slots currently carry 4.4, 4.5 and 4.6; a client on any other minor falls through to the flat slot.

Deployment from this repo

The script is a file in the repo, applied by the existing OpenTofu configuration:

edge/rpkgs-router.ts          # the script
edge/rpkgs-router.test.ts     # UA x path -> expected Location matrix
cdn.tf                        # bunnynet_compute_script + _variable

Provider BunnyWay/bunnynet v0.17.0 (already pinned) ships bunnynet_compute_script with content loadable via file(), plus bunnynet_compute_script_variable. middleware_script = bunnynet_compute_script.rpkgs_router.id replaces the hard-coded 29277, after a one-time tofu import of the existing script.

Error handling

• Unknown R minor, or a slot not listed in UNION_SLOTS: pass through to the flat slot. The client sees exactly today's behaviour.
• Unparseable distro on the bare-root form: redirect to CRAN.
• A request already under …/src/contrib/<x.y>/: pass through, so a redirect can never loop.
• A per-minor slot that does not exist for a listed minor: the client gets the origin's 404. UNION_SLOTS is the operator's assertion that the slot is ready, so this is a configuration error, not a runtime condition to paper over.

Testing

Local, before any apply: deno run -A edge/rpkgs-router.ts serves the middleware against the real origin, so edge/rpkgs-router.test.ts drives the whole matrix against that local server.

• User agent matrix: R 4.4 / 4.5 / 4.6 on musl and gnu, both arches, Posit-style and stock forms, plus a darwin UA and a non-R UA.
• Path matrix: PACKAGES, PACKAGES.gz, PACKAGES.rds, a tarball, a path already under 4.5/, and /src/contrib/… on the bare root.
• Assertion is the Location header (or its absence), not the body.

After apply, a smoke test against cran.rpkgs.com:

• available.packages() inside reg.devxy.io/r/r-alpine:4.5-3.24 returns the union count, and "curl" %in% rownames(...) is TRUE.
• A flat-slot package still downloads from …/src/contrib/, and a per-minor package downloads from …/src/contrib/<x.y>/.

Out of scope

• Meta/archive.rds stays flat-only, so remotes::install_version() does not see per-minor archives.
• The alpine324 source-tarball defect: that slot serves CRAN sources stamped as binaries, and needs a rebuild independent of this work.
• Any change to how uvr resolves per-minor URLs; it already addresses the slots directly.

Split of work

1. bincraft: union index writer plus its guard, and a re-index of one slot to validate.
2. This repo: edge/rpkgs-router.ts, its test matrix, and the cdn.tf resources with UNION_SLOTS empty.
3. Enable UNION_SLOTS slot by slot as bincraft backfills them.