feat(edge): route PACKAGES requests to the per-R-minor slot #152
1 changed files with 166 additions and 0 deletions
docs(plans): add the implementation plan for per-R-minor edge routing
commit
5a0a4fa200
166
plans/2026-08-07-per-minor-edge-routing.md
Normal file
166
plans/2026-08-07-per-minor-edge-routing.md
Normal file
|
|
@ -0,0 +1,166 @@
|
|||
# Per-R-minor edge routing implementation plan
|
||||
|
||||
Spec: `specs/2026-08-07-per-minor-edge-routing-design.md`
|
||||
|
||||
**Goal:** let a stock `install.packages()` see the per-minor packages by routing `PACKAGES*` requests to `…/src/contrib/<x.y>/`, where `bincraft` publishes a union index.
|
||||
|
||||
**Architecture:** the union is built in `bincraft`; the edge script only redirects index requests, gated on a `UNION_SLOTS` script variable; the script lives in this repo and is applied by OpenTofu.
|
||||
|
||||
**Tech stack:** Deno / TypeScript (Bunny Edge Scripting, SDK 0.12), OpenTofu with `BunnyWay/bunnynet` 0.17, R (bincraft).
|
||||
|
||||
## Global constraints
|
||||
|
||||
- Redirect only `PACKAGES`, `PACKAGES.gz` and `PACKAGES.rds`; never a tarball, because `Path: ..` entries arrive already normalised to the flat path.
|
||||
- Every redirect carries `Cache-Control: no-store`; redirect targets stay UA-independent.
|
||||
- `UNION_SLOTS` is empty by default, so deploying the script is a no-op until a slot is backfilled.
|
||||
- A slot is `<arch>/<os>`, e.g. `amd64/alpine324`.
|
||||
- Verified prerequisites: `PACKAGES*` is served `cdn-cache: BYPASS`, so the script sees every index request; `Deno.env.get()` reads script variables; the SDK local server listens on `127.0.0.1:8080`.
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Edge script and its test matrix
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `edge/rpkgs-router.ts`
|
||||
- Create: `edge/rpkgs-router.test.ts`
|
||||
- Modify: `justfile` (add `edge-test`)
|
||||
|
||||
**Produces:** a single-file script deployable as `bunnynet_compute_script.content`, reading `UNION_SLOTS` from the environment.
|
||||
|
||||
- [ ] **Step 1: write the test matrix first**
|
||||
|
||||
`edge/rpkgs-router.test.ts` spawns `deno run -A edge/rpkgs-router.ts` with `UNION_SLOTS=amd64/alpine324`, waits for `127.0.0.1:8080`, and issues requests with `redirect: "manual"`.
|
||||
|
||||
Cases, asserted on the `location` header (or its absence):
|
||||
|
||||
| # | path | User-Agent | expectation |
|
||||
| --- | ------------------------------------------------------ | --------------------------------------------- | ------------------------------------------------------------ |
|
||||
| 1 | `/amd64/alpine324/latest/src/contrib/PACKAGES.gz` | `R (4.5.3 x86_64-pc-linux-musl …)` | 302 → `…/src/contrib/4.5/PACKAGES.gz` |
|
||||
| 2 | same | `R (4.6.0 …)` | 302 → `…/src/contrib/4.6/PACKAGES.gz` |
|
||||
| 3 | same, but slot `amd64/noble` | `R (4.5.3 …)` | no redirect (slot not in `UNION_SLOTS`) |
|
||||
| 4 | `…/src/contrib/curl_7.1.0.tar.gz` | `R (4.5.3 …)` | no redirect |
|
||||
| 5 | `…/src/contrib/4.5/PACKAGES.gz` | `R (4.5.3 …)` | no redirect (loop guard) |
|
||||
| 6 | `…/src/contrib/PACKAGES.gz` | `curl/8.0` | no redirect (no R minor) |
|
||||
| 7 | `/src/contrib/PACKAGES.gz` | alpine UA with `Alpine Linux … 3.24` | 302 → `/amd64/alpine324/latest/src/contrib/4.5/PACKAGES.gz` |
|
||||
| 8 | `/src/contrib/PACKAGES.gz` | `R (4.5.3 x86_64-pc-linux-musl …)`, no distro | 302 → `cran.r-project.org`, **not** a `linux-musl` slot |
|
||||
| 9 | `/src/contrib/foo_1.0.tar.gz` | `R (4.5.1 aarch64-apple-darwin20 …)` | 302 → `/bin/macosx/big-sur-arm64/contrib/4.5/foo_1.0.tar.gz` |
|
||||
| 10 | `/bin/macosx/big-sur-arm64/contrib/4.5/foo_1.0.tar.gz` | any | 302 → `cran.r-project.org` |
|
||||
| 11 | any redirect above | — | `cache-control: no-store` |
|
||||
|
||||
- [ ] **Step 2: run the tests and watch them fail**
|
||||
|
||||
`just edge-test` → every case fails, because `edge/rpkgs-router.ts` does not exist.
|
||||
|
||||
- [ ] **Step 3: write `edge/rpkgs-router.ts`**
|
||||
|
||||
Order of evaluation in `onOriginRequest`:
|
||||
|
||||
1. normalise `//` runs in the path
|
||||
2. darwin `/src/contrib/*` → `/bin/macosx/<flavour>/contrib/<x.y>/`
|
||||
3. `/bin/macosx/**` → CRAN
|
||||
4. `/{arch}/{os}/latest/src/contrib/<rest>`: pass through if `rest` already starts with `<x.y>/`, or is not an index file, or the slot is not in `UNION_SLOTS`, or the UA has no R minor; otherwise redirect into `<x.y>/`
|
||||
5. `/`, `/src/contrib`, `/src/contrib/**`: resolve arch+os from the UA, redirect to CRAN when the distro is unidentifiable, otherwise redirect to the qualified path, adding `<x.y>/` under the same index-file rule
|
||||
6. anything else: pass through
|
||||
|
||||
The R minor comes from either `R/4.5.3` or `R (4.5.3 …)`, so a stock UA is enough. The `linux-gnu` / `linux-musl` fallback in `parseUserAgent` is deleted: those are not slot names.
|
||||
|
||||
- [ ] **Step 4: run the tests until they pass**
|
||||
|
||||
`just edge-test`
|
||||
|
||||
- [ ] **Step 5: commit**
|
||||
|
||||
```bash
|
||||
git add edge/rpkgs-router.ts edge/rpkgs-router.test.ts justfile
|
||||
git commit -m "feat(edge): route PACKAGES requests to the per-R-minor slot"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Manage the script from OpenTofu
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `cdn.tf`
|
||||
|
||||
**Consumes:** `edge/rpkgs-router.ts` from Task 1.
|
||||
|
||||
- [ ] **Step 1: add the resources**
|
||||
|
||||
```terraform
|
||||
resource "bunnynet_compute_script" "rpkgs_router" {
|
||||
type = "middleware"
|
||||
name = "rpkgs-router"
|
||||
content = file("${path.module}/edge/rpkgs-router.ts")
|
||||
}
|
||||
|
||||
resource "bunnynet_compute_script_variable" "rpkgs_router_union_slots" {
|
||||
script = bunnynet_compute_script.rpkgs_router.id
|
||||
name = "UNION_SLOTS"
|
||||
default_value = ""
|
||||
required = false
|
||||
}
|
||||
```
|
||||
|
||||
and replace `middleware_script = 29277` with `middleware_script = bunnynet_compute_script.rpkgs_router.id`.
|
||||
|
||||
- [ ] **Step 2: validate**
|
||||
|
||||
`tofu init -backend=false && tofu validate`
|
||||
|
||||
- [ ] **Step 3: import the existing script (needs `BUNNYNET_API_KEY`)**
|
||||
|
||||
```bash
|
||||
tofu import bunnynet_compute_script.rpkgs_router 29277
|
||||
tofu plan
|
||||
```
|
||||
|
||||
The plan must show an in-place `content` update and **no** replacement of the pull zone. A replacement means the import did not take.
|
||||
|
||||
- [ ] **Step 4: commit**
|
||||
|
||||
```bash
|
||||
git add cdn.tf
|
||||
git commit -m "feat(cdn): manage the edge middleware script from this repo"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Union index writer in bincraft
|
||||
|
||||
**Files (repo `codefloe.com/rpkgs/bincraft`):**
|
||||
|
||||
- Modify: `R/package_index.R`
|
||||
- Test: `tests/testthat/test-package_index.R`
|
||||
|
||||
**Produces:** `write_union_index(flat_records, minor_records)` returning the merged records, called from `upload_package_index()` when `r_minor` is set.
|
||||
|
||||
- [ ] **Step 1: write the failing tests**
|
||||
|
||||
- a package present in both slots keeps the per-minor record and gains no `Path`
|
||||
- a package only in the flat slot survives with `Path = ".."`
|
||||
- a package only in the per-minor slot survives unchanged
|
||||
- a union smaller than the flat input raises an error rather than returning
|
||||
|
||||
- [ ] **Step 2: run them and watch them fail**
|
||||
|
||||
`Rscript -e 'testthat::test_file("tests/testthat/test-package_index.R")'`
|
||||
|
||||
- [ ] **Step 3: implement `write_union_index()` and call it from `upload_package_index()`**
|
||||
|
||||
After `update_PACKAGES()` has written the per-minor index, read the flat slot's `PACKAGES.rds`, drop packages already in the per-minor index, set `Path = ".."` on the rest, and rewrite `PACKAGES`, `PACKAGES.gz` and `PACKAGES.rds` in the per-minor slot.
|
||||
|
||||
- [ ] **Step 4: run the tests until they pass**
|
||||
|
||||
- [ ] **Step 5: commit and open the PR against bincraft**
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Roll out slot by slot
|
||||
|
||||
- [ ] Re-index one slot (`amd64/alpine324`, R 4.5) and confirm the union index lists both `curl` (per-minor) and `jsonlite` (flat, `Path: ..`).
|
||||
- [ ] Set `UNION_SLOTS = "amd64/alpine324"` and confirm in `reg.devxy.io/r/r-alpine:4.5-3.24` that `available.packages()` returns the union count and `"curl" %in% rownames(...)`.
|
||||
- [ ] Add `arm64/alpine324`, then the remaining slots.
|
||||
|
||||
`install.packages("curl")` will still fail to build on `alpine324` until that slot's source tarballs are replaced with real binaries. That is tracked separately.
|
||||
Loading…
Reference in a new issue