feat(edge): gate per-minor routing on published minors and add a staging zone (#175)

## Motivation

`UNION_SLOTS` is empty, so per-minor routing has never been exercised end to end. Before it can be enabled and advertised, two things were missing: a way to test it without pointing production at it, and evidence that the published indexes actually support it.

Verifying the data first turned up a defect that would have broken users the moment the flag was flipped.

## The defect

`contribPath()` redirects to `contrib/<minor>/` whenever the User-Agent carries any R minor, with no existence check and no fallback:

```ts
const rMinor = extractRMinor(userAgent);
return rMinor ? `/${slot}/latest/src/contrib/${rMinor}/${rest}` : flat;
```

Only `4.4`, `4.5` and `4.6` are published. `4.3` and `4.2` return 404 on all 16 slots. With `UNION_SLOTS` set, an R 4.3 client would be redirected to a non-existent index and see **zero** packages: a silent, total failure rather than a degraded one. R 4.3 is still advertised as supported on the website and in `docs/configuration.mdoc`, though `build-env-images` now pins only 4.6.0/4.5.3/4.4.3.

## Changes

- **Gate routing on `KNOWN_MINORS`** (default `4.4,4.5,4.6`), falling back to the flat index for anything else. Unknown minor now behaves exactly as today.
- **Honour `EXTRA_PUBLIC_HOSTS`.** `publicCdnOrigin()` falls back to the hardcoded `PUBLIC_CDN_ORIGIN` for any hostname not in `PUBLIC_CDN_HOSTS`, so a staging zone on a `b-cdn.net` hostname would redirect into *production* and silently measure the wrong system. This lets the identical script run on staging and redirect within itself.
- **Add the `cran-rpkgs-test` pull zone** with `UNION_SLOTS` pre-enabled for all 16 slots, same B2 origin, served on the bunny default hostname so it needs no DNS record and is never advertised.
- **Add `scripts/verify-r-minor-routing.sh`**, covering every `<arch>/<os>` slot: index reachability per minor, the union property against flat, `Path:` target resolution, coverage parity across minors, and with `--live` the real User-Agent routing, the non-R User-Agent case, and that tarballs are never rewritten.
- **Cover the fallback in the edge test suite** for both an unpublished minor (4.3) and a future one (4.7).

## Findings from the full run

112 passed, 16 failed across the 16 slots. Every failure is the same: no R 4.3 index.

All 16 slots carry union indexes that are supersets of flat, every sampled `Path:` target resolves, and all indexes were republished within minutes of each other, so the build side is healthy.

Coverage is **not** yet even, which is why "full coverage for ABI-sensitive packages" is not a claim to make yet:

| slot | flat | 4.4 | 4.5 | 4.6 |
|---|---|---|---|---|
| amd64/resolute | 24305 | 24402 | 24748 | 24395 |
| amd64/alpine324 | 24397 | 24457 | 24744 | 24448 |
| amd64/noble | 24780 | 24805 | 24805 | 24805 |

On the R 4.5-built distros (`resolute`, `alpine324`, and their arm64 twins) a 4.4 or 4.6 client sees ~300 fewer packages than a 4.5 client. On `noble`/`jammy`/`rhel9`/`alpine323` the spread is under 5. The new parity check encodes this with a configurable `PARITY_TOLERANCE`.

## Verification

- `just edge-test`: 18 steps pass. The two new steps were confirmed to fail with the `KNOWN_MINORS` gate removed and pass with it.
- `tofu validate`: passes. **Not applied** - no bunny.net or state credentials were available, so the staging zone still needs a `tofu apply`.
- `scripts/verify-r-minor-routing.sh`: full 16-slot run, results above.
- `shellcheck`: clean.

## Not done here

Applying the staging zone, then running `BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live` against it. Production `UNION_SLOTS` is deliberately left empty.

Reviewed-on: #175

scripts/verify-r-minor-routing.sh

@@ -0,0 +1,309 @@
+#!/usr/bin/env bash
+#
+# Verify per-R-minor index routing for cran.rpkgs.com across every published
+# <arch>/<os> slot.
+#
+# The edge router (edge/rpkgs-router.ts) rewrites PACKAGES* requests to
+# `contrib/<x.y>/` when the slot is listed in UNION_SLOTS and the client's
+# User-Agent carries an R minor. Two properties have to hold before a slot may
+# be added to UNION_SLOTS:
+#
+#   1. the per-minor index is a UNION of the per-minor and flat slots, so
+#      routing to it hides nothing the flat index carries; and
+#   2. every R minor a client might report resolves to an index that exists,
+#      because contribPath() does not check existence and has no fallback.
+#
+# Modes:
+#   (default)  Resolve routing decisions without depending on UNION_SLOTS being
+#              set. Safe to run before enabling: it reads the per-minor indexes
+#              directly and reproduces the router's target path.
+#   --live     Additionally drive the real CDN with R User-Agents and assert the
+#              bytes served match the expected index. Only meaningful once the
+#              slot is in UNION_SLOTS.
+#
+# Usage:
+#   scripts/verify-r-minor-routing.sh
+#   scripts/verify-r-minor-routing.sh --live
+#   MINORS="4.4 4.5" SAMPLE=10 scripts/verify-r-minor-routing.sh
+#
+# Exits non-zero if any check fails.
+
+set -uo pipefail
+
+BASE=${BASE:-https://cran.rpkgs.com}
+ARCHES=${ARCHES:-"amd64 arm64"}
+DISTROS=${DISTROS:-"resolute noble jammy rhel8 rhel9 rhel10 alpine323 alpine324"}
+# The supported window: the current R minor plus the two previous, matching
+# build-env-images' R_VERSION_LATEST/PREV1/PREV2 and cdn.tf's
+# local.rpkgs_supported_minors. Each of these must have a published index.
+MINORS=${MINORS:-"4.4 4.5 4.6"}
+# Minors we deliberately do not serve. These must have NO published index and,
+# once routing is live, must be sent to CRAN for sources rather than 404ing or
+# being handed binaries built under another minor.
+EXCLUDED_MINORS=${EXCLUDED_MINORS:-"4.3"}
+# How many Path: targets to HEAD-check per slot/minor. 0 disables.
+SAMPLE=${SAMPLE:-5}
+# Largest package-count shortfall a non-primary minor may have against the best
+# minor on the same slot before coverage counts as uneven. A slot built under
+# one R minor carries fewer per-minor binaries for the others; until that gap
+# closes, "full coverage for ABI-sensitive packages" is not a claim we can make.
+PARITY_TOLERANCE=${PARITY_TOLERANCE:-25}
+LIVE=0
+
+for arg in "$@"; do
+  case "$arg" in
+    --live) LIVE=1 ;;
+    -h | --help)
+      sed -n '2,32p' "$0"
+      exit 0
+      ;;
+    *)
+      echo "unknown argument: $arg" >&2
+      exit 2
+      ;;
+  esac
+done
+
+WORK=$(mktemp -d)
+trap 'rm -rf "$WORK"' EXIT
+
+PASS=0
+FAIL=0
+FAILURES=""
+
+ok() {
+  PASS=$((PASS + 1))
+  printf '  ok    %s\n' "$1"
+}
+
+bad() {
+  FAIL=$((FAIL + 1))
+  FAILURES="${FAILURES}\n  - $1"
+  printf '  FAIL  %s\n' "$1"
+}
+
+# Fetch a URL into a file, echoing the HTTP status. Cached per URL.
+fetch() {
+  local url=$1 dest=$2 ua=${3:-}
+  if [ -s "$dest" ]; then
+    cat "$dest.status"
+    return 0
+  fi
+  local status
+  if [ -n "$ua" ]; then
+    status=$(curl -sS -A "$ua" -o "$dest" -w '%{http_code}' --max-time 120 "$url" 2>/dev/null)
+  else
+    status=$(curl -sS -o "$dest" -w '%{http_code}' --max-time 120 "$url" 2>/dev/null)
+  fi
+  echo "$status" > "$dest.status"
+  echo "$status"
+}
+
+head_status() {
+  curl -sS -o /dev/null -w '%{http_code}' -I --max-time 60 "$1" 2>/dev/null
+}
+
+# Package names from a gzipped PACKAGES index, sorted.
+pkg_names() {
+  gunzip -c "$1" 2>/dev/null | awk '/^Package:/ {print $2}' | sort -u
+}
+
+# "<Package> <Path>" pairs for entries that carry a Path: field.
+path_entries() {
+  gunzip -c "$1" 2>/dev/null | awk '
+    /^Package:/ { pkg = $2; ver = ""; path = "" }
+    /^Version:/ { ver = $2 }
+    /^Path:/    { path = $2 }
+    /^$/        { if (pkg != "" && path != "") print pkg, ver, path; pkg = "" }
+    END         { if (pkg != "" && path != "") print pkg, ver, path }
+  '
+}
+
+# An R User-Agent of the shape R actually sends.
+r_user_agent() {
+  printf 'R/%s.0 (Ubuntu 24.04; codename=noble) (x86_64-pc-linux-gnu x86_64 linux-gnu)' "$1"
+}
+
+echo "verify-r-minor-routing: $BASE"
+echo "  slots:  $(echo "$ARCHES" | wc -w) arch x $(echo "$DISTROS" | wc -w) os"
+echo "  minors: $MINORS  (excluded: $EXCLUDED_MINORS)"
+echo "  live:   $LIVE"
+echo
+
+for arch in $ARCHES; do
+  for distro in $DISTROS; do
+    slot="$arch/$distro"
+    echo "$slot"
+
+    flat_url="$BASE/$slot/latest/src/contrib/PACKAGES.gz"
+    flat_file="$WORK/${arch}-${distro}-flat.gz"
+    flat_status=$(fetch "$flat_url" "$flat_file")
+
+    if [ "$flat_status" != "200" ]; then
+      bad "$slot flat index unreachable (HTTP $flat_status)"
+      continue
+    fi
+
+    pkg_names "$flat_file" > "$flat_file.names"
+    flat_count=$(wc -l < "$flat_file.names")
+    if [ "$flat_count" -lt 1000 ]; then
+      bad "$slot flat index has only $flat_count packages"
+      continue
+    fi
+    ok "$slot flat index: $flat_count packages"
+
+    for minor in $MINORS; do
+      minor_url="$BASE/$slot/latest/src/contrib/$minor/PACKAGES.gz"
+      minor_file="$WORK/${arch}-${distro}-${minor}.gz"
+      minor_status=$(fetch "$minor_url" "$minor_file")
+
+      # A minor the router would route to must exist, or clients on that R
+      # version get a 404 and see no packages at all.
+      if [ "$minor_status" != "200" ]; then
+        bad "$slot R $minor index missing (HTTP $minor_status) - routing would 404 for R $minor clients"
+        continue
+      fi
+
+      pkg_names "$minor_file" > "$minor_file.names"
+      minor_count=$(wc -l < "$minor_file.names")
+
+      # Union property: nothing the flat index carries may be missing here.
+      missing=$(comm -23 "$flat_file.names" "$minor_file.names" | head -5)
+      missing_count=$(comm -23 "$flat_file.names" "$minor_file.names" | wc -l)
+      if [ "$missing_count" -ne 0 ]; then
+        bad "$slot R $minor index is not a union: $missing_count flat packages absent (e.g. $(echo "$missing" | tr '\n' ' '))"
+      else
+        ok "$slot R $minor index: $minor_count packages, union holds"
+      fi
+
+      # Path: entries steer to per-minor binaries; they must resolve.
+      if [ "$SAMPLE" -gt 0 ]; then
+        path_entries "$minor_file" > "$minor_file.paths"
+        total_paths=$(wc -l < "$minor_file.paths")
+        broken=0
+        checked=0
+        while read -r pkg ver path; do
+          [ -z "${pkg:-}" ] && continue
+          tarball="$BASE/$slot/latest/src/contrib/$path/${pkg}_${ver}.tar.gz"
+          status=$(head_status "$tarball")
+          checked=$((checked + 1))
+          if [ "$status" != "200" ]; then
+            broken=$((broken + 1))
+            [ "$broken" -le 2 ] && printf '        broken target: %s (HTTP %s)\n' "$tarball" "$status"
+          fi
+        done < <(shuf -n "$SAMPLE" "$minor_file.paths" 2>/dev/null || head -n "$SAMPLE" "$minor_file.paths")
+
+        if [ "$broken" -ne 0 ]; then
+          bad "$slot R $minor: $broken/$checked sampled Path: targets do not resolve (of $total_paths total)"
+        elif [ "$checked" -gt 0 ]; then
+          ok "$slot R $minor: $checked/$checked sampled Path: targets resolve (of $total_paths total)"
+        fi
+      fi
+
+      # Live routing: what a real R client on this minor actually receives.
+      if [ "$LIVE" -eq 1 ]; then
+        ua=$(r_user_agent "$minor")
+        live_file="$WORK/${arch}-${distro}-${minor}-live.gz"
+        live_status=$(fetch "$flat_url" "$live_file" "$ua")
+        if [ "$live_status" != "200" ]; then
+          bad "$slot R $minor live request failed (HTTP $live_status)"
+        elif cmp -s "$live_file" "$minor_file"; then
+          ok "$slot R $minor live request served the per-minor index"
+        elif cmp -s "$live_file" "$flat_file"; then
+          bad "$slot R $minor live request served the FLAT index - slot not in UNION_SLOTS?"
+        else
+          bad "$slot R $minor live request served neither the per-minor nor the flat index"
+        fi
+      fi
+    done
+
+    # Coverage parity across minors. The union property only guarantees no
+    # client loses packages relative to the flat index; it says nothing about a
+    # 4.4 client seeing fewer packages than a 4.5 client on the same slot.
+    best=0
+    for minor in $MINORS; do
+      f="$WORK/${arch}-${distro}-${minor}.gz.names"
+      [ -s "$f" ] || continue
+      c=$(wc -l < "$f")
+      [ "$c" -gt "$best" ] && best=$c
+    done
+    if [ "$best" -gt 0 ]; then
+      uneven=""
+      for minor in $MINORS; do
+        f="$WORK/${arch}-${distro}-${minor}.gz.names"
+        [ -s "$f" ] || continue
+        c=$(wc -l < "$f")
+        gap=$((best - c))
+        [ "$gap" -gt "$PARITY_TOLERANCE" ] && uneven="$uneven R$minor:-$gap"
+      done
+      if [ -n "$uneven" ]; then
+        bad "$slot coverage uneven across minors (vs best $best):$uneven"
+      else
+        ok "$slot coverage parity across minors (best $best, all within $PARITY_TOLERANCE)"
+      fi
+    fi
+
+    # Excluded minors: no published index, and under --live a redirect to CRAN.
+    for minor in $EXCLUDED_MINORS; do
+      ex_url="$BASE/$slot/latest/src/contrib/$minor/PACKAGES.gz"
+      ex_status=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 60 "$ex_url" 2>/dev/null)
+      if [ "$ex_status" = "200" ]; then
+        bad "$slot R $minor is excluded but an index is published - the two lists disagree"
+      else
+        ok "$slot R $minor correctly has no published index"
+      fi
+
+      if [ "$LIVE" -eq 1 ]; then
+        loc=$(curl -sS -o /dev/null -w '%{redirect_url}' -A "$(r_user_agent "$minor")" \
+          --max-time 60 "$flat_url" 2>/dev/null)
+        case "$loc" in
+          https://cran.r-project.org/*)
+            ok "$slot R $minor is sent to CRAN ($loc)"
+            ;;
+          "")
+            bad "$slot R $minor was served directly instead of being sent to CRAN"
+            ;;
+          *)
+            bad "$slot R $minor redirected somewhere unexpected: $loc"
+            ;;
+        esac
+      fi
+    done
+
+    # A client whose User-Agent carries no R version must keep getting the flat
+    # index, never a per-minor one.
+    if [ "$LIVE" -eq 1 ]; then
+      plain_file="$WORK/${arch}-${distro}-plain.gz"
+      plain_status=$(fetch "$flat_url" "$plain_file" "curl/8.0.0")
+      if [ "$plain_status" != "200" ]; then
+        bad "$slot non-R User-Agent request failed (HTTP $plain_status)"
+      elif cmp -s "$plain_file" "$flat_file"; then
+        ok "$slot non-R User-Agent still served the flat index"
+      else
+        bad "$slot non-R User-Agent was routed away from the flat index"
+      fi
+
+      # Tarball requests must never be rewritten into a per-minor directory:
+      # flat-slot packages do not live there.
+      sample_pkg=$(gunzip -c "$flat_file" | awk '/^Package:/ {p=$2} /^Version:/ {print p, $2; exit}')
+      if [ -n "$sample_pkg" ]; then
+        # shellcheck disable=SC2086 # deliberate split into $1 (package) and $2 (version)
+        set -- $sample_pkg
+        tb="$BASE/$slot/latest/src/contrib/${1}_${2}.tar.gz"
+        tb_status=$(curl -sS -o /dev/null -w '%{http_code}' -A "$(r_user_agent 4.5)" --max-time 60 "$tb" 2>/dev/null)
+        if [ "$tb_status" = "200" ]; then
+          ok "$slot tarball request under an R User-Agent still resolves"
+        else
+          bad "$slot tarball ${1}_${2}.tar.gz broke under an R User-Agent (HTTP $tb_status)"
+        fi
+      fi
+    fi
+  done
+done
+
+echo
+echo "passed: $PASS   failed: $FAIL"
+if [ "$FAIL" -ne 0 ]; then
+  printf 'failures:%b\n' "$FAILURES"
+  exit 1
+fi