devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 10 Pull requests Releases Activity
build-cran-binaries/plans/2026-08-07-per-minor-edge-routing.md

8.2 KiB
Raw Blame History

Per-R-minor edge routing implementation plan

Spec: specs/2026-08-07-per-minor-edge-routing-design.md

Goal: let a stock install.packages() see the per-minor packages by routing PACKAGES* requests to …/src/contrib/<x.y>/, where bincraft publishes a union index.

Architecture: the union is built in bincraft; the edge script only redirects index requests, gated on a UNION_SLOTS script variable; the script lives in this repo and is applied by OpenTofu.

Tech stack: Deno / TypeScript (Bunny Edge Scripting, SDK 0.12), OpenTofu with BunnyWay/bunnynet 0.17, R (bincraft).

Global constraints

  • Redirect only PACKAGES, PACKAGES.gz and PACKAGES.rds; never a tarball, because Path: .. entries arrive already normalised to the flat path.
  • Every redirect carries Cache-Control: no-store; redirect targets stay UA-independent.
  • UNION_SLOTS is empty by default, so deploying the script is a no-op until a slot is backfilled.
  • A slot is <arch>/<os>, e.g. amd64/alpine324.
  • Verified prerequisites: PACKAGES* is served cdn-cache: BYPASS, so the script sees every index request; Deno.env.get() reads script variables; the SDK local server listens on 127.0.0.1:8080.

Task 1: Edge script and its test matrix

Files:

  • Create: edge/rpkgs-router.ts
  • Create: edge/rpkgs-router.test.ts
  • Modify: justfile (add edge-test)

Produces: a single-file script deployable as bunnynet_compute_script.content, reading UNION_SLOTS from the environment.

  • Step 1: write the test matrix first

edge/rpkgs-router.test.ts spawns deno run -A edge/rpkgs-router.ts with UNION_SLOTS=amd64/alpine324, waits for 127.0.0.1:8080, and issues requests with redirect: "manual".

Cases, asserted on the location header (or its absence):

# path User-Agent expectation
1 /amd64/alpine324/latest/src/contrib/PACKAGES.gz R (4.5.3 x86_64-pc-linux-musl …) 302 → …/src/contrib/4.5/PACKAGES.gz
2 same R (4.6.0 …) 302 → …/src/contrib/4.6/PACKAGES.gz
3 same, but slot amd64/noble R (4.5.3 …) no redirect (slot not in UNION_SLOTS)
4 …/src/contrib/curl_7.1.0.tar.gz R (4.5.3 …) no redirect
5 …/src/contrib/4.5/PACKAGES.gz R (4.5.3 …) no redirect (loop guard)
6 …/src/contrib/PACKAGES.gz curl/8.0 no redirect (no R minor)
7 /src/contrib/PACKAGES.gz alpine UA with Alpine Linux … 3.24 302 → /amd64/alpine324/latest/src/contrib/4.5/PACKAGES.gz
8 /src/contrib/PACKAGES.gz R (4.5.3 x86_64-pc-linux-musl …), no distro 302 → cran.r-project.org, not a linux-musl slot
9 /src/contrib/foo_1.0.tar.gz R (4.5.1 aarch64-apple-darwin20 …) 302 → /bin/macosx/big-sur-arm64/contrib/4.5/foo_1.0.tar.gz
10 /bin/macosx/big-sur-arm64/contrib/4.5/foo_1.0.tar.gz any 302 → cran.r-project.org
11 any redirect above — cache-control: no-store
  • Step 2: run the tests and watch them fail

just edge-test → every case fails, because edge/rpkgs-router.ts does not exist.

  • Step 3: write edge/rpkgs-router.ts

Order of evaluation in onOriginRequest:

  1. normalise // runs in the path
  2. darwin /src/contrib/* → /bin/macosx/<flavour>/contrib/<x.y>/
  3. /bin/macosx/** → CRAN
  4. /{arch}/{os}/latest/src/contrib/<rest>: pass through if rest already starts with <x.y>/, or is not an index file, or the slot is not in UNION_SLOTS, or the UA has no R minor; otherwise redirect into <x.y>/
  5. /, /src/contrib, /src/contrib/**: resolve arch+os from the UA, redirect to CRAN when the distro is unidentifiable, otherwise redirect to the qualified path, adding <x.y>/ under the same index-file rule
  6. anything else: pass through

The R minor comes from either R/4.5.3 or R (4.5.3 …), so a stock UA is enough. The linux-gnu / linux-musl fallback in parseUserAgent is deleted: those are not slot names.

  • Step 4: run the tests until they pass

just edge-test

  • Step 5: commit
git add edge/rpkgs-router.ts edge/rpkgs-router.test.ts justfile
git commit -m "feat(edge): route PACKAGES requests to the per-R-minor slot"

Task 2: Manage the script from OpenTofu

Files:

  • Modify: cdn.tf

Consumes: edge/rpkgs-router.ts from Task 1.

  • Step 1: add the resources
resource "bunnynet_compute_script" "rpkgs_router" {
  type    = "middleware"
  name    = "rpkgs-router"
  content = file("${path.module}/edge/rpkgs-router.ts")
}

resource "bunnynet_compute_script_variable" "rpkgs_router_union_slots" {
  script        = bunnynet_compute_script.rpkgs_router.id
  name          = "UNION_SLOTS"
  default_value = ""
  required      = false
}

and replace middleware_script = 29277 with middleware_script = bunnynet_compute_script.rpkgs_router.id.

  • Step 2: validate

tofu init -backend=false && tofu validate

  • Step 3: import the existing script (needs BUNNYNET_API_KEY)
tofu import bunnynet_compute_script.rpkgs_router 29277
tofu plan

The plan must show an in-place content update and no replacement of the pull zone. A replacement means the import did not take.

  • Step 4: commit
git add cdn.tf
git commit -m "feat(cdn): manage the edge middleware script from this repo"

Task 3: Union index writer in bincraft

Files (repo codefloe.com/rpkgs/bincraft):

  • Modify: R/package_index.R
  • Test: tests/testthat/test-package_index.R

Produces: write_union_index(flat_records, minor_records) returning the merged records, called from upload_package_index() when r_minor is set.

  • Step 1: write the failing tests

  • a package present in both slots keeps the per-minor record and gains no Path

  • a package only in the flat slot survives with Path = ".."

  • a package only in the per-minor slot survives unchanged

  • a union smaller than the flat input raises an error rather than returning

  • Step 2: run them and watch them fail

Rscript -e 'testthat::test_file("tests/testthat/test-package_index.R")'

  • Step 3: implement write_union_index() and call it from upload_package_index()

After update_PACKAGES() has written the per-minor index, read the flat slot's PACKAGES.rds, drop packages already in the per-minor index, set Path = ".." on the rest, and rewrite PACKAGES, PACKAGES.gz and PACKAGES.rds in the per-minor slot.

  • Step 4: run the tests until they pass

  • Step 5: commit and open the PR against bincraft

Task 4: Roll out slot by slot

  • Re-index one slot (amd64/alpine324, R 4.5) and confirm the union index lists both curl (per-minor) and jsonlite (flat, Path: ..).
  • Set UNION_SLOTS = "amd64/alpine324" and confirm in reg.devxy.io/r/r-alpine:4.5-3.24 that available.packages() returns the union count and "curl" %in% rownames(...).
  • Add arm64/alpine324, then the remaining slots.

install.packages("curl") will still fail to build on alpine324 until that slot's source tarballs are replaced with real binaries. That is tracked separately.