All checks were successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
## Motivation
`UNION_SLOTS` is empty, so per-minor routing has never been exercised end to end. Before it can be enabled and advertised, two things were missing: a way to test it without pointing production at it, and evidence that the published indexes actually support it.
Verifying the data first turned up a defect that would have broken users the moment the flag was flipped.
## The defect
`contribPath()` redirects to `contrib/<minor>/` whenever the User-Agent carries any R minor, with no existence check and no fallback:
```ts
const rMinor = extractRMinor(userAgent);
return rMinor ? `/${slot}/latest/src/contrib/${rMinor}/${rest}` : flat;
```
Only `4.4`, `4.5` and `4.6` are published. `4.3` and `4.2` return 404 on all 16 slots. With `UNION_SLOTS` set, an R 4.3 client would be redirected to a non-existent index and see **zero** packages: a silent, total failure rather than a degraded one. R 4.3 is still advertised as supported on the website and in `docs/configuration.mdoc`, though `build-env-images` now pins only 4.6.0/4.5.3/4.4.3.
## Changes
- **Gate routing on `KNOWN_MINORS`** (default `4.4,4.5,4.6`), falling back to the flat index for anything else. Unknown minor now behaves exactly as today.
- **Honour `EXTRA_PUBLIC_HOSTS`.** `publicCdnOrigin()` falls back to the hardcoded `PUBLIC_CDN_ORIGIN` for any hostname not in `PUBLIC_CDN_HOSTS`, so a staging zone on a `b-cdn.net` hostname would redirect into *production* and silently measure the wrong system. This lets the identical script run on staging and redirect within itself.
- **Add the `cran-rpkgs-test` pull zone** with `UNION_SLOTS` pre-enabled for all 16 slots, same B2 origin, served on the bunny default hostname so it needs no DNS record and is never advertised.
- **Add `scripts/verify-r-minor-routing.sh`**, covering every `<arch>/<os>` slot: index reachability per minor, the union property against flat, `Path:` target resolution, coverage parity across minors, and with `--live` the real User-Agent routing, the non-R User-Agent case, and that tarballs are never rewritten.
- **Cover the fallback in the edge test suite** for both an unpublished minor (4.3) and a future one (4.7).
## Findings from the full run
112 passed, 16 failed across the 16 slots. Every failure is the same: no R 4.3 index.
All 16 slots carry union indexes that are supersets of flat, every sampled `Path:` target resolves, and all indexes were republished within minutes of each other, so the build side is healthy.
Coverage is **not** yet even, which is why "full coverage for ABI-sensitive packages" is not a claim to make yet:
| slot | flat | 4.4 | 4.5 | 4.6 |
|---|---|---|---|---|
| amd64/resolute | 24305 | 24402 | 24748 | 24395 |
| amd64/alpine324 | 24397 | 24457 | 24744 | 24448 |
| amd64/noble | 24780 | 24805 | 24805 | 24805 |
On the R 4.5-built distros (`resolute`, `alpine324`, and their arm64 twins) a 4.4 or 4.6 client sees ~300 fewer packages than a 4.5 client. On `noble`/`jammy`/`rhel9`/`alpine323` the spread is under 5. The new parity check encodes this with a configurable `PARITY_TOLERANCE`.
## Verification
- `just edge-test`: 18 steps pass. The two new steps were confirmed to fail with the `KNOWN_MINORS` gate removed and pass with it.
- `tofu validate`: passes. **Not applied** - no bunny.net or state credentials were available, so the staging zone still needs a `tofu apply`.
- `scripts/verify-r-minor-routing.sh`: full 16-slot run, results above.
- `shellcheck`: clean.
## Not done here
Applying the staging zone, then running `BASE=https://cran-rpkgs-test.b-cdn.net scripts/verify-r-minor-routing.sh --live` against it. Production `UNION_SLOTS` is deliberately left empty.
Reviewed-on: #175
328 lines
9.9 KiB
HCL
328 lines
9.9 KiB
HCL
# https://registry.terraform.io/providers/BunnyWay/bunnynet/latest/docs/resources/pullzone
|
|
# terraform import bunnynet_pullzone.devxy-r-binaries cran
|
|
# resource "bunnynet_pullzone" "devxy-r-binaries" {
|
|
# name = "cran"
|
|
|
|
# origin {
|
|
# type = "OriginUrl"
|
|
# url = "https://devxy-rpkgs-binaries.s3.eu-central-003.backblazeb2.com"
|
|
# }
|
|
|
|
# routing {
|
|
# tier = "Standard"
|
|
# }
|
|
|
|
# s3_auth_enabled = true
|
|
# s3_auth_key = var.B2_S3_ACCESS_KEY
|
|
# s3_auth_secret = var.B2_S3_SECRET_KEY
|
|
# s3_auth_region = "eu-central-003"
|
|
|
|
# cache_enabled = true
|
|
# cache_errors = true
|
|
# request_coalescing_enabled = true
|
|
# block_post_requests = true
|
|
|
|
# limit_requests = 500
|
|
# limit_connections = 50
|
|
|
|
# safehop_enabled = true
|
|
|
|
# add_canonical_header = true
|
|
|
|
# cache_stale = ["offline", "updating"]
|
|
# use_background_update = true
|
|
|
|
# block_ips = var.cdn_block_ips
|
|
|
|
# # 50 TB
|
|
# limit_bandwidth = 50000000000000
|
|
|
|
# permacache_storagezone = bunnynet_storage_zone.devxy-r-binaries.id
|
|
|
|
# # rec from docs: https://www.backblaze.com/docs/cloud-storage-integrate-bunnynet-with-backblaze-b2
|
|
# block_root_path = true
|
|
# }
|
|
|
|
# resource "bunnynet_pullzone_hostname" "devxy-r-binaries" {
|
|
# pullzone = bunnynet_pullzone.devxy-r-binaries.id
|
|
# name = "cran.devxy.io"
|
|
# force_ssl = true
|
|
# tls_enabled = true
|
|
# }
|
|
|
|
### cran.rpkgs.com
|
|
|
|
locals {
|
|
rpkgs_slots = [
|
|
for pair in setproduct(
|
|
["amd64", "arm64"],
|
|
["resolute", "noble", "jammy", "rhel8", "rhel9", "rhel10", "alpine323", "alpine324"]
|
|
) : "${pair[0]}/${pair[1]}"
|
|
]
|
|
|
|
# The supported R minors: the current one plus the two previous, which is
|
|
# exactly what build-env-images installs as R_VERSION_LATEST / PREV1 / PREV2.
|
|
# These must stay in step. A minor listed here without a published index
|
|
# sends those clients to a 404; a published minor missing from this list
|
|
# sends them to CRAN for sources instead of serving the binaries we built.
|
|
rpkgs_supported_minors = ["4.4", "4.5", "4.6"]
|
|
|
|
# bunny.net serves every pull zone on <name>.b-cdn.net, so staging needs no
|
|
# DNS record and is never advertised.
|
|
rpkgs_test_hostname = "cran-rpkgs-test.b-cdn.net"
|
|
}
|
|
|
|
# The edge middleware that resolves the bare cran.rpkgs.com form to an
|
|
# <arch>/<os> slot and routes PACKAGES* to the per-R-minor slot. The source of
|
|
# truth is edge/rpkgs-router.ts; `tofu apply` publishes a new release.
|
|
#
|
|
# The script pre-dates this configuration, so it is adopted rather than created:
|
|
# tofu import bunnynet_compute_script.rpkgs_router 29277
|
|
resource "bunnynet_compute_script" "rpkgs_router" {
|
|
type = "middleware"
|
|
name = "rpkgs-router"
|
|
content = file("${path.module}/edge/rpkgs-router.ts")
|
|
}
|
|
|
|
# Slots ("<arch>/<os>", comma separated) whose per-minor index bincraft has
|
|
# already republished as a union of the per-minor and flat slots. Routing to a
|
|
# slot that is not listed here would hide every package the per-minor index does
|
|
# not carry, so this stays empty until a slot has been backfilled.
|
|
resource "bunnynet_compute_script_variable" "rpkgs_router_union_slots" {
|
|
script = bunnynet_compute_script.rpkgs_router.id
|
|
name = "UNION_SLOTS"
|
|
default_value = ""
|
|
required = false
|
|
}
|
|
|
|
resource "bunnynet_compute_script_variable" "rpkgs_router_known_minors" {
|
|
script = bunnynet_compute_script.rpkgs_router.id
|
|
name = "KNOWN_MINORS"
|
|
default_value = join(",", local.rpkgs_supported_minors)
|
|
required = false
|
|
}
|
|
|
|
resource "bunnynet_pullzone" "cran_rpkgs_com" {
|
|
name = "cran-rpkgs"
|
|
|
|
cache_errors = false
|
|
|
|
cache_expiration_time = 31919000
|
|
websockets_enabled = false
|
|
errorpage_whitelabel = true
|
|
|
|
origin {
|
|
type = "OriginUrl"
|
|
url = "https://devxy-rpkgs-binaries.s3.eu-central-003.backblazeb2.com"
|
|
middleware_script = bunnynet_compute_script.rpkgs_router.id
|
|
}
|
|
|
|
routing {
|
|
filters = [
|
|
"scripting",
|
|
]
|
|
}
|
|
|
|
s3_auth_enabled = true
|
|
s3_auth_key = var.B2_S3_ACCESS_KEY
|
|
s3_auth_secret = var.B2_S3_SECRET_KEY
|
|
s3_auth_region = "eu-central-003"
|
|
|
|
cache_enabled = true
|
|
request_coalescing_enabled = true
|
|
block_post_requests = true
|
|
|
|
# Set on the zone since before this configuration existed; declared here so
|
|
# `tofu apply` stops silently removing it.
|
|
#
|
|
# The router makes it redundant on paper: the only UA-dependent responses it
|
|
# produces are redirects, and those carry `Cache-Control: no-store`, while
|
|
# their targets are concrete per-slot, per-minor URLs whose content depends
|
|
# only on the path. Dropping it would also be a real win, because otherwise
|
|
# every distinct R version string keys its own copy of every tarball.
|
|
#
|
|
# It stays for now anyway: it is the second line of defence against the one
|
|
# failure that would be quiet and confusing (an R 4.6 client served the 4.5
|
|
# index), and removing it is worth doing on its own once per-minor routing is
|
|
# confirmed live, not as a side effect of enabling that routing.
|
|
cache_vary_headers = ["User-Agent"]
|
|
|
|
limit_requests = 5000
|
|
limit_connections = 1000
|
|
|
|
safehop_enabled = true
|
|
|
|
add_canonical_header = true
|
|
|
|
cache_stale = ["offline", "updating"]
|
|
|
|
block_ips = var.cdn_block_ips
|
|
|
|
# 50 TB
|
|
limit_bandwidth = 50000000000000
|
|
|
|
# permacache_storagezone = bunnynet_storage_zone.devxy-r-binaries.id
|
|
|
|
# rec from docs: https://www.backblaze.com/docs/cloud-storage-integrate-bunnynet-with-backblaze-b2
|
|
block_root_path = true
|
|
}
|
|
|
|
resource "bunnynet_pullzone_hostname" "cran_rpkgs_com" {
|
|
pullzone = bunnynet_pullzone.cran_rpkgs_com.id
|
|
name = "cran.rpkgs.com"
|
|
force_ssl = true
|
|
tls_enabled = true
|
|
}
|
|
|
|
### Staging zone for edge-router changes
|
|
|
|
# Every published <arch>/<os> slot. The staging zone enables per-minor routing
|
|
# for all of them at once; production adopts the same list only after
|
|
# `scripts/verify-r-minor-routing.sh --live` passes against staging.
|
|
|
|
# A second copy of the same router, bound to the same B2 origin, so UNION_SLOTS
|
|
# can be exercised end to end before production is touched.
|
|
resource "bunnynet_compute_script" "rpkgs_router_test" {
|
|
type = "middleware"
|
|
name = "rpkgs-router-test"
|
|
content = file("${path.module}/edge/rpkgs-router.ts")
|
|
}
|
|
|
|
resource "bunnynet_compute_script_variable" "rpkgs_router_test_union_slots" {
|
|
script = bunnynet_compute_script.rpkgs_router_test.id
|
|
name = "UNION_SLOTS"
|
|
default_value = join(",", local.rpkgs_slots)
|
|
required = false
|
|
}
|
|
|
|
# Without this the staging zone rewrites to PUBLIC_CDN_ORIGIN, so its redirects
|
|
# land on production and the test silently measures the wrong system.
|
|
resource "bunnynet_compute_script_variable" "rpkgs_router_test_extra_hosts" {
|
|
script = bunnynet_compute_script.rpkgs_router_test.id
|
|
name = "EXTRA_PUBLIC_HOSTS"
|
|
default_value = local.rpkgs_test_hostname
|
|
required = false
|
|
}
|
|
|
|
resource "bunnynet_compute_script_variable" "rpkgs_router_test_known_minors" {
|
|
script = bunnynet_compute_script.rpkgs_router_test.id
|
|
name = "KNOWN_MINORS"
|
|
default_value = join(",", local.rpkgs_supported_minors)
|
|
required = false
|
|
}
|
|
|
|
resource "bunnynet_pullzone" "cran_rpkgs_test" {
|
|
name = "cran-rpkgs-test"
|
|
|
|
cache_errors = false
|
|
|
|
cache_expiration_time = 31919000
|
|
websockets_enabled = false
|
|
errorpage_whitelabel = true
|
|
|
|
origin {
|
|
type = "OriginUrl"
|
|
url = "https://devxy-rpkgs-binaries.s3.eu-central-003.backblazeb2.com"
|
|
middleware_script = bunnynet_compute_script.rpkgs_router_test.id
|
|
}
|
|
|
|
routing {
|
|
filters = [
|
|
"scripting",
|
|
]
|
|
}
|
|
|
|
s3_auth_enabled = true
|
|
s3_auth_key = var.B2_S3_ACCESS_KEY
|
|
s3_auth_secret = var.B2_S3_SECRET_KEY
|
|
s3_auth_region = "eu-central-003"
|
|
|
|
cache_enabled = true
|
|
request_coalescing_enabled = true
|
|
block_post_requests = true
|
|
|
|
cache_vary_headers = ["User-Agent"]
|
|
|
|
# Staging carries only synthetic verification traffic, so the production
|
|
# ceilings would be pure headroom.
|
|
limit_requests = 500
|
|
limit_connections = 100
|
|
|
|
safehop_enabled = true
|
|
add_canonical_header = true
|
|
cache_stale = ["offline", "updating"]
|
|
block_ips = var.cdn_block_ips
|
|
|
|
# 1 TB
|
|
limit_bandwidth = 1000000000000
|
|
|
|
block_root_path = true
|
|
}
|
|
|
|
|
|
# Alliance SwissPass historically used a separate, manually configured pull
|
|
# zone. Adopt it so both public repositories use the same B2 origin, middleware
|
|
# release and cache behavior.
|
|
import {
|
|
to = bunnynet_pullzone.cran_allianceswisspass
|
|
id = "3265648"
|
|
}
|
|
|
|
resource "bunnynet_pullzone" "cran_allianceswisspass" {
|
|
name = "cran-allianceswisspass"
|
|
|
|
cache_errors = false
|
|
cache_expiration_time = 31919000
|
|
websockets_enabled = false
|
|
errorpage_whitelabel = true
|
|
|
|
origin {
|
|
type = "OriginUrl"
|
|
url = "https://devxy-rpkgs-binaries.s3.eu-central-003.backblazeb2.com"
|
|
middleware_script = bunnynet_compute_script.rpkgs_router.id
|
|
}
|
|
|
|
routing {
|
|
filters = [
|
|
"scripting",
|
|
]
|
|
}
|
|
|
|
s3_auth_enabled = true
|
|
s3_auth_key = var.B2_S3_ACCESS_KEY
|
|
s3_auth_secret = var.B2_S3_SECRET_KEY
|
|
s3_auth_region = "eu-central-003"
|
|
|
|
cache_enabled = true
|
|
request_coalescing_enabled = true
|
|
block_post_requests = true
|
|
cache_vary_headers = ["User-Agent"]
|
|
|
|
limit_requests = 5000
|
|
limit_connections = 1000
|
|
|
|
safehop_enabled = true
|
|
add_canonical_header = true
|
|
cache_stale = ["offline", "updating"]
|
|
block_ips = var.cdn_block_ips
|
|
|
|
# 50 TB
|
|
limit_bandwidth = 50000000000000
|
|
|
|
block_root_path = true
|
|
}
|
|
|
|
resource "bunnynet_pullzone_hostname" "cran_allianceswisspass" {
|
|
pullzone = bunnynet_pullzone.cran_allianceswisspass.id
|
|
name = "cran.allianceswisspass.devxy.io"
|
|
force_ssl = true
|
|
tls_enabled = true
|
|
}
|
|
|
|
# resource "bunnynet_storage_zone" "devxy-r-binaries" {
|
|
# name = "devxy-r-binaries-storage"
|
|
# region = "DE"
|
|
# zone_tier = "Standard"
|
|
# # Los Angeles and Singapore
|
|
# replication_regions = ["LA", "SG"]
|
|
# }
|