build-cran-binaries/plans/2026-08-07-per-minor-edge-routing.md
pat-s 2f732457d2
Some checks failed
ci/crow/manual/trial-build-registry/1 Pipeline is pending
ci/crow/manual/trial-build-registry/3 Pipeline is pending
ci/crow/manual/trial-build-registry/5 Pipeline is pending
ci/crow/manual/trial-build-registry/7 Pipeline is pending
ci/crow/manual/trial-build-registry/9 Pipeline is pending
ci/crow/manual/trial-build-registry/11 Pipeline is pending
ci/crow/manual/trial-build-registry/13 Pipeline is pending
ci/crow/manual/trial-build-registry/15 Pipeline is pending
ci/crow/manual/trial-build-registry/17 Pipeline is pending
ci/crow/manual/repair-built-stamp/2 Pipeline failed
ci/crow/manual/weekly-audit-missing/17 Pipeline was successful
ci/crow/manual/weekly-audit-missing/9 Pipeline was successful
ci/crow/manual/weekly-audit-missing/15 Pipeline was successful
ci/crow/manual/weekly-audit-missing/11 Pipeline was successful
ci/crow/manual/weekly-audit-missing/13 Pipeline was successful
ci/crow/manual/weekly-audit-missing/7 Pipeline was successful
ci/crow/manual/trial-build-registry/4 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/trial-build-registry/6 Pipeline was successful
ci/crow/manual/trial-build-registry/2 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/trial-build-registry/8 Pipeline was successful
ci/crow/manual/trial-build-registry/14 Pipeline was successful
ci/crow/manual/trial-build-registry/12 Pipeline was successful
ci/crow/manual/trial-build-registry/10 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/trial-build-registry/16 Pipeline was successful
ci/crow/manual/trial-build-registry/18 Pipeline was successful
ci/crow/manual/weekly-audit-missing/5 Pipeline was successful
ci/crow/manual/weekly-audit-missing/1 Pipeline was successful
ci/crow/manual/auto-apply-patches Pipeline was successful
ci/crow/manual/weekly-audit-missing/3 Pipeline was successful
ci/crow/manual/weekly-audit-missing/2 Pipeline was successful
ci/crow/manual/weekly-audit-missing/4 Pipeline was successful
ci/crow/manual/weekly-audit-missing/6 Pipeline was successful
ci/crow/manual/weekly-audit-missing/8 Pipeline was successful
ci/crow/manual/weekly-audit-missing/10 Pipeline was successful
ci/crow/manual/weekly-audit-missing/12 Pipeline was successful
ci/crow/manual/weekly-audit-missing/14 Pipeline was successful
ci/crow/manual/weekly-audit-missing/16 Pipeline was successful
ci/crow/manual/weekly-audit-missing/18 Pipeline was successful
ci/crow/manual/weekly-patch-proposals Pipeline was successful
ci/crow/manual/process-updates/6 Pipeline was successful
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/7 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/9 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/11 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/8 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/10 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/12 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/14 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/17 Pipeline failed
ci/crow/manual/weekly-rebuild-missing/6 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/5 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/18 Pipeline was successful
feat(edge): route PACKAGES requests to the per-R-minor slot (#152)
## Problem

`install.packages("curl")` fails in `reg.devxy.io/r/r-alpine:4.5-3.24` with "package 'curl' is not available for this version of R", on both arches.

`curl` is not missing from the repo: it is in `…/latest/src/contrib/4.5/` and `…/4.6/`, the per-minor slots that base R cannot address. The image's repo URL resolves to `…/latest/src/contrib`, whose index does not list it. On `amd64/alpine324` that is 2 886 packages invisible to `install.packages()` (23 on `amd64/noble`) — what issue #63 records as "missing binaries".

Two further findings while investigating:

- The middleware only ever rewrote the bare `cran.rpkgs.com/src/contrib/…` form, and that form was broken for every Linux client on a stock R user agent: `ALPINE_REGEX`/`UBUNTU_REGEX`/`RHEL_REGEX` need a Posit-style UA that carries the distro, so stock R fell through to `extractOs()` and got redirected to `/amd64/linux-musl/latest/…`, a slot that does not exist.
- `PACKAGES*` is served `cdn-cache: BYPASS` (bincraft uploads it `no-store`), so the middleware sees every index request and no purge is needed for routing changes to take effect.

## What this changes

**`edge/rpkgs-router.ts`** — the middleware, now a reviewed file in this repo rather than dashboard state. It routes `PACKAGES`, `PACKAGES.gz` and `PACKAGES.rds` into `…/src/contrib/<x.y>/` for slots listed in `UNION_SLOTS`, and nothing else.

Tarballs are deliberately left alone. R keeps the `contriburl` it *asked for*, not the one the redirect served it, so every tarball URL is resolved against the flat directory and the union index steers the per-minor ones with a `Path: <x.y>` field. Rewriting a tarball request here would send flat-slot packages into a directory that does not hold them.

Also in the script: the phantom `linux-gnu`/`linux-musl` fallback is gone (an unidentifiable distro goes to CRAN, as an unparseable UA already did), and every redirect carries `Cache-Control: no-store` since its target depends on the User-Agent. The macOS branches are unchanged.

**`cdn.tf`** — `bunnynet_compute_script.rpkgs_router` with `content = file("edge/rpkgs-router.ts")`, the `UNION_SLOTS` variable, and `middleware_script` pointing at the resource instead of the literal `29277`.

`UNION_SLOTS` is empty, so merging and applying this changes no client's behaviour. A slot is added only once bincraft has republished its per-minor index as a union (rpkgs/bincraft#97); routing to a raw per-minor index would hide every package it does not carry. Rolling back is a variable edit, not a deploy.

**`specs/`, `plans/`** — the design and the implementation plan, including the two approaches that were rejected (edge-side merge, moving the minor up the path) and why.

## Verification

`just edge-test` runs 13 routing cases against the SDK's local server, so what is tested is the artifact that gets deployed; pass-through cases proxy to the real origin. All pass.

End to end, with the middleware in front of a locally built union index for `amd64/alpine324` (31 507 records), inside the runtime image:

```
curl:     7.1.0  -> …/latest/src/contrib/4.5  -> curl_7.1.0.tar.gz      717 725 B
jsonlite: 2.0.0  -> …/latest/src/contrib      -> jsonlite_2.0.0.tar.gz  1 055 849 B
```

`tofu validate` passes. `tofu plan` has not been run: no `BUNNYNET_API_KEY` available in this environment.

## Before applying

The script pre-dates this configuration, so it must be adopted, not created:

```sh
tofu import bunnynet_compute_script.rpkgs_router 29277
tofu plan
```

The plan should show an in-place `content` update and no replacement of the pull zone. Without the import, tofu creates a second script and repoints the zone at it. Note that `name = "rpkgs-router"` will rename the existing script on apply.

## Not fixed here

`install.packages("curl")` on `alpine324` will now *resolve*, then fail to build: that slot's tarballs are byte-identical CRAN **source** tarballs (no `Meta/`, no `Built:` in DESCRIPTION) which the index nevertheless stamps `Built: R 4.5.3; …-linux-musl`. Sampled: `amd64/alpine324` 3/12 binary, `arm64/alpine324` 13/30, `amd64/noble` 12/12, `amd64/alpine323` 17/20. That slot needs a rebuild, tracked separately.

Reviewed-on: #152
2026-08-07 14:12:22 +00:00

8.2 KiB

Per-R-minor edge routing implementation plan

Spec: specs/2026-08-07-per-minor-edge-routing-design.md

Goal: let a stock install.packages() see the per-minor packages by routing PACKAGES* requests to …/src/contrib/<x.y>/, where bincraft publishes a union index.

Architecture: the union is built in bincraft; the edge script only redirects index requests, gated on a UNION_SLOTS script variable; the script lives in this repo and is applied by OpenTofu.

Tech stack: Deno / TypeScript (Bunny Edge Scripting, SDK 0.12), OpenTofu with BunnyWay/bunnynet 0.17, R (bincraft).

Global constraints

  • Redirect only PACKAGES, PACKAGES.gz and PACKAGES.rds; never a tarball, because the union index already carries the correct tarball URL for both classes of package.
  • Every redirect carries Cache-Control: no-store; redirect targets stay UA-independent.
  • UNION_SLOTS is empty by default, so deploying the script is a no-op until a slot is backfilled.
  • A slot is <arch>/<os>, e.g. amd64/alpine324.
  • Verified prerequisites: PACKAGES* is served cdn-cache: BYPASS, so the script sees every index request; Deno.env.get() reads script variables; the SDK local server listens on 127.0.0.1:8080.

Task 1: Edge script and its test matrix

Files:

  • Create: edge/rpkgs-router.ts
  • Create: edge/rpkgs-router.test.ts
  • Modify: justfile (add edge-test)

Produces: a single-file script deployable as bunnynet_compute_script.content, reading UNION_SLOTS from the environment.

  • Step 1: write the test matrix first

edge/rpkgs-router.test.ts spawns deno run -A edge/rpkgs-router.ts with UNION_SLOTS=amd64/alpine324, waits for 127.0.0.1:8080, and issues requests with redirect: "manual".

Cases, asserted on the location header (or its absence):

# path User-Agent expectation
1 /amd64/alpine324/latest/src/contrib/PACKAGES.gz R (4.5.3 x86_64-pc-linux-musl …) 302 → …/src/contrib/4.5/PACKAGES.gz
2 same R (4.6.0 …) 302 → …/src/contrib/4.6/PACKAGES.gz
3 same, but slot amd64/noble R (4.5.3 …) no redirect (slot not in UNION_SLOTS)
4 …/src/contrib/curl_7.1.0.tar.gz R (4.5.3 …) no redirect
5 …/src/contrib/4.5/PACKAGES.gz R (4.5.3 …) no redirect (loop guard)
6 …/src/contrib/PACKAGES.gz curl/8.0 no redirect (no R minor)
7 /src/contrib/PACKAGES.gz alpine UA with Alpine Linux … 3.24 302 → /amd64/alpine324/latest/src/contrib/4.5/PACKAGES.gz
8 /src/contrib/PACKAGES.gz R (4.5.3 x86_64-pc-linux-musl …), no distro 302 → cran.r-project.org, not a linux-musl slot
9 /src/contrib/foo_1.0.tar.gz R (4.5.1 aarch64-apple-darwin20 …) 302 → /bin/macosx/big-sur-arm64/contrib/4.5/foo_1.0.tar.gz
10 /bin/macosx/big-sur-arm64/contrib/4.5/foo_1.0.tar.gz any 302 → cran.r-project.org
11 any redirect above cache-control: no-store
  • Step 2: run the tests and watch them fail

just edge-test → every case fails, because edge/rpkgs-router.ts does not exist.

  • Step 3: write edge/rpkgs-router.ts

Order of evaluation in onOriginRequest:

  1. normalise // runs in the path
  2. darwin /src/contrib/*/bin/macosx/<flavour>/contrib/<x.y>/
  3. /bin/macosx/** → CRAN
  4. /{arch}/{os}/latest/src/contrib/<rest>: pass through if rest already starts with <x.y>/, or is not an index file, or the slot is not in UNION_SLOTS, or the UA has no R minor; otherwise redirect into <x.y>/
  5. /, /src/contrib, /src/contrib/**: resolve arch+os from the UA, redirect to CRAN when the distro is unidentifiable, otherwise redirect to the qualified path, adding <x.y>/ under the same index-file rule
  6. anything else: pass through

The R minor comes from either R/4.5.3 or R (4.5.3 …), so a stock UA is enough. The linux-gnu / linux-musl fallback in parseUserAgent is deleted: those are not slot names.

  • Step 4: run the tests until they pass

just edge-test

  • Step 5: commit
git add edge/rpkgs-router.ts edge/rpkgs-router.test.ts justfile
git commit -m "feat(edge): route PACKAGES requests to the per-R-minor slot"

Task 2: Manage the script from OpenTofu

Files:

  • Modify: cdn.tf

Consumes: edge/rpkgs-router.ts from Task 1.

  • Step 1: add the resources
resource "bunnynet_compute_script" "rpkgs_router" {
  type    = "middleware"
  name    = "rpkgs-router"
  content = file("${path.module}/edge/rpkgs-router.ts")
}

resource "bunnynet_compute_script_variable" "rpkgs_router_union_slots" {
  script        = bunnynet_compute_script.rpkgs_router.id
  name          = "UNION_SLOTS"
  default_value = ""
  required      = false
}

and replace middleware_script = 29277 with middleware_script = bunnynet_compute_script.rpkgs_router.id.

  • Step 2: validate

tofu init -backend=false && tofu validate

  • Step 3: import the existing script (needs BUNNYNET_API_KEY)
tofu import bunnynet_compute_script.rpkgs_router 29277
tofu plan

The plan must show an in-place content update and no replacement of the pull zone. A replacement means the import did not take.

  • Step 4: commit
git add cdn.tf
git commit -m "feat(cdn): manage the edge middleware script from this repo"

Task 3: Union index writer in bincraft

Files (repo codefloe.com/rpkgs/bincraft):

  • Modify: R/package_index.R
  • Test: tests/testthat/test-package_index.R

Produces: write_union_index(flat_records, minor_records) returning the merged records, called from upload_package_index() when r_minor is set.

  • Step 1: write the failing tests

  • a package present in both slots keeps the per-minor record, with Path = "4.5"

  • a package only in the flat slot survives with no Path

  • a package only in the per-minor slot survives with Path = "4.5"

  • a union smaller than the flat input raises an error rather than returning

  • Step 2: run them and watch them fail

Rscript -e 'testthat::test_file("tests/testthat/test-package_index.R")'

  • Step 3: implement write_union_index() and call it from upload_package_index()

After update_PACKAGES() has written the per-minor index, read the flat slot's PACKAGES.rds, set Path = <r_minor> on the per-minor records, drop the flat records for packages the per-minor slot already has, and rewrite PACKAGES, PACKAGES.gz and PACKAGES.rds in the per-minor slot.

  • Step 4: run the tests until they pass

  • Step 5: commit and open the PR against bincraft


Task 4: Roll out slot by slot

  • Re-index one slot (amd64/alpine324, R 4.5) and confirm the union index lists both curl (per-minor, Path: 4.5) and jsonlite (flat, no Path).
  • Set UNION_SLOTS = "amd64/alpine324" and confirm in reg.devxy.io/r/r-alpine:4.5-3.24 that available.packages() returns the union count and "curl" %in% rownames(...).
  • Add arm64/alpine324, then the remaining slots.

install.packages("curl") will still fail to build on alpine324 until that slot's source tarballs are replaced with real binaries. That is tracked separately.