Some checks failed
ci/crow/manual/trial-build-registry/1 Pipeline is pending
ci/crow/manual/trial-build-registry/3 Pipeline is pending
ci/crow/manual/trial-build-registry/5 Pipeline is pending
ci/crow/manual/trial-build-registry/7 Pipeline is pending
ci/crow/manual/trial-build-registry/9 Pipeline is pending
ci/crow/manual/trial-build-registry/11 Pipeline is pending
ci/crow/manual/trial-build-registry/13 Pipeline is pending
ci/crow/manual/trial-build-registry/15 Pipeline is pending
ci/crow/manual/trial-build-registry/17 Pipeline is pending
ci/crow/manual/repair-built-stamp/2 Pipeline failed
ci/crow/manual/weekly-audit-missing/17 Pipeline was successful
ci/crow/manual/weekly-audit-missing/9 Pipeline was successful
ci/crow/manual/weekly-audit-missing/15 Pipeline was successful
ci/crow/manual/weekly-audit-missing/11 Pipeline was successful
ci/crow/manual/weekly-audit-missing/13 Pipeline was successful
ci/crow/manual/weekly-audit-missing/7 Pipeline was successful
ci/crow/manual/trial-build-registry/4 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/trial-build-registry/6 Pipeline was successful
ci/crow/manual/trial-build-registry/2 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/trial-build-registry/8 Pipeline was successful
ci/crow/manual/trial-build-registry/14 Pipeline was successful
ci/crow/manual/trial-build-registry/12 Pipeline was successful
ci/crow/manual/trial-build-registry/10 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/trial-build-registry/16 Pipeline was successful
ci/crow/manual/trial-build-registry/18 Pipeline was successful
ci/crow/manual/weekly-audit-missing/5 Pipeline was successful
ci/crow/manual/weekly-audit-missing/1 Pipeline was successful
ci/crow/manual/auto-apply-patches Pipeline was successful
ci/crow/manual/weekly-audit-missing/3 Pipeline was successful
ci/crow/manual/weekly-audit-missing/2 Pipeline was successful
ci/crow/manual/weekly-audit-missing/4 Pipeline was successful
ci/crow/manual/weekly-audit-missing/6 Pipeline was successful
ci/crow/manual/weekly-audit-missing/8 Pipeline was successful
ci/crow/manual/weekly-audit-missing/10 Pipeline was successful
ci/crow/manual/weekly-audit-missing/12 Pipeline was successful
ci/crow/manual/weekly-audit-missing/14 Pipeline was successful
ci/crow/manual/weekly-audit-missing/16 Pipeline was successful
ci/crow/manual/weekly-audit-missing/18 Pipeline was successful
ci/crow/manual/weekly-patch-proposals Pipeline was successful
ci/crow/manual/process-updates/6 Pipeline was successful
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/7 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/9 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/11 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/8 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/10 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/12 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/14 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/17 Pipeline failed
ci/crow/manual/weekly-rebuild-missing/6 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/5 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/18 Pipeline was successful
## Problem
`install.packages("curl")` fails in `reg.devxy.io/r/r-alpine:4.5-3.24` with "package 'curl' is not available for this version of R", on both arches.
`curl` is not missing from the repo: it is in `…/latest/src/contrib/4.5/` and `…/4.6/`, the per-minor slots that base R cannot address. The image's repo URL resolves to `…/latest/src/contrib`, whose index does not list it. On `amd64/alpine324` that is 2 886 packages invisible to `install.packages()` (23 on `amd64/noble`) — what issue #63 records as "missing binaries".
Two further findings while investigating:
- The middleware only ever rewrote the bare `cran.rpkgs.com/src/contrib/…` form, and that form was broken for every Linux client on a stock R user agent: `ALPINE_REGEX`/`UBUNTU_REGEX`/`RHEL_REGEX` need a Posit-style UA that carries the distro, so stock R fell through to `extractOs()` and got redirected to `/amd64/linux-musl/latest/…`, a slot that does not exist.
- `PACKAGES*` is served `cdn-cache: BYPASS` (bincraft uploads it `no-store`), so the middleware sees every index request and no purge is needed for routing changes to take effect.
## What this changes
**`edge/rpkgs-router.ts`** — the middleware, now a reviewed file in this repo rather than dashboard state. It routes `PACKAGES`, `PACKAGES.gz` and `PACKAGES.rds` into `…/src/contrib/<x.y>/` for slots listed in `UNION_SLOTS`, and nothing else.
Tarballs are deliberately left alone. R keeps the `contriburl` it *asked for*, not the one the redirect served it, so every tarball URL is resolved against the flat directory and the union index steers the per-minor ones with a `Path: <x.y>` field. Rewriting a tarball request here would send flat-slot packages into a directory that does not hold them.
Also in the script: the phantom `linux-gnu`/`linux-musl` fallback is gone (an unidentifiable distro goes to CRAN, as an unparseable UA already did), and every redirect carries `Cache-Control: no-store` since its target depends on the User-Agent. The macOS branches are unchanged.
**`cdn.tf`** — `bunnynet_compute_script.rpkgs_router` with `content = file("edge/rpkgs-router.ts")`, the `UNION_SLOTS` variable, and `middleware_script` pointing at the resource instead of the literal `29277`.
`UNION_SLOTS` is empty, so merging and applying this changes no client's behaviour. A slot is added only once bincraft has republished its per-minor index as a union (rpkgs/bincraft#97); routing to a raw per-minor index would hide every package it does not carry. Rolling back is a variable edit, not a deploy.
**`specs/`, `plans/`** — the design and the implementation plan, including the two approaches that were rejected (edge-side merge, moving the minor up the path) and why.
## Verification
`just edge-test` runs 13 routing cases against the SDK's local server, so what is tested is the artifact that gets deployed; pass-through cases proxy to the real origin. All pass.
End to end, with the middleware in front of a locally built union index for `amd64/alpine324` (31 507 records), inside the runtime image:
```
curl: 7.1.0 -> …/latest/src/contrib/4.5 -> curl_7.1.0.tar.gz 717 725 B
jsonlite: 2.0.0 -> …/latest/src/contrib -> jsonlite_2.0.0.tar.gz 1 055 849 B
```
`tofu validate` passes. `tofu plan` has not been run: no `BUNNYNET_API_KEY` available in this environment.
## Before applying
The script pre-dates this configuration, so it must be adopted, not created:
```sh
tofu import bunnynet_compute_script.rpkgs_router 29277
tofu plan
```
The plan should show an in-place `content` update and no replacement of the pull zone. Without the import, tofu creates a second script and repoints the zone at it. Note that `name = "rpkgs-router"` will rename the existing script on apply.
## Not fixed here
`install.packages("curl")` on `alpine324` will now *resolve*, then fail to build: that slot's tarballs are byte-identical CRAN **source** tarballs (no `Meta/`, no `Built:` in DESCRIPTION) which the index nevertheless stamps `Built: R 4.5.3; …-linux-musl`. Sampled: `amd64/alpine324` 3/12 binary, `arm64/alpine324` 13/30, `amd64/noble` 12/12, `amd64/alpine323` 17/20. That slot needs a rebuild, tracked separately.
Reviewed-on: #152
89 lines
4 KiB
Makefile
89 lines
4 KiB
Makefile
# Local helpers for build-cran-binaries.
|
|
#
|
|
# `rebuild` (re)builds specific versions of a single package on a given OS/arch
|
|
# by dispatching to a remote buildx builder (the build runs there, not locally).
|
|
# Sensitivity is auto-detected: risky packages land in the per-minor slot
|
|
# (contrib/<x.y>/), everything else in the generic slot; the touched index is
|
|
# refreshed so the result is immediately servable.
|
|
#
|
|
# Prerequisites:
|
|
# - buildx builders named `artemis` (amd64) and `gaia` (arm64), created with the
|
|
# docker-container driver (runs BuildKit on the remote host's docker daemon over
|
|
# SSH). The default `remote` driver does NOT work with an ssh:// docker host.
|
|
# Pass --config docker/buildkitd.toml so BuildKit's GC keeps a usable cache
|
|
# (the default caps the cache-mount tier at 512 MB, forcing re-downloads).
|
|
# docker buildx create --name artemis --driver docker-container --config docker/buildkitd.toml ssh://<user@host-amd64>
|
|
# docker buildx create --name gaia --driver docker-container --config docker/buildkitd.toml ssh://<user@host-arm64>
|
|
# - exported secrets: B2_S3_ACCESS_KEY, B2_S3_SECRET_KEY, PGPASS (GITHUB_PAT optional)
|
|
#
|
|
# Overridable (env or `just VAR=… rebuild …`):
|
|
# R_VERSION (default 4.5.3) — selects the R minor → the per-minor slot
|
|
# AMD64_BUILDER / ARM64_BUILDER — buildx builder names
|
|
#
|
|
# Examples:
|
|
# just rebuild alpine 3.23 amd64 rlang 1.1.4 1.1.3
|
|
# just rebuild redhat 10 arm64 data.table 1.15.4
|
|
# R_VERSION=4.4.3 just rebuild ubuntu noble amd64 Rcpp 1.0.12
|
|
|
|
r_version := env_var_or_default("R_VERSION", "4.5.3")
|
|
amd64_builder := env_var_or_default("AMD64_BUILDER", "artemis")
|
|
arm64_builder := env_var_or_default("ARM64_BUILDER", "gaia")
|
|
|
|
# (re)build PACKAGE at one or more VERSIONS on OS/TAG/ARCH via a remote builder
|
|
rebuild os tag arch package *versions:
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
if [ -z "{{ versions }}" ]; then
|
|
echo "error: provide at least one version, e.g. just rebuild alpine 3.23 amd64 rlang 1.1.4" >&2
|
|
exit 1
|
|
fi
|
|
case "{{ arch }}" in
|
|
amd64) builder="{{ amd64_builder }}" ;;
|
|
arm64) builder="{{ arm64_builder }}" ;;
|
|
*) echo "error: arch must be 'amd64' or 'arm64'" >&2; exit 1 ;;
|
|
esac
|
|
# Accept TF_VAR_-prefixed names (direnv) or plain names.
|
|
export B2_S3_ACCESS_KEY="${B2_S3_ACCESS_KEY:-${TF_VAR_B2_S3_ACCESS_KEY:-}}"
|
|
export B2_S3_SECRET_KEY="${B2_S3_SECRET_KEY:-${TF_VAR_B2_S3_SECRET_KEY:-}}"
|
|
: "${B2_S3_ACCESS_KEY:?set B2_S3_ACCESS_KEY or TF_VAR_B2_S3_ACCESS_KEY}"
|
|
: "${B2_S3_SECRET_KEY:?set B2_S3_SECRET_KEY or TF_VAR_B2_S3_SECRET_KEY}"
|
|
: "${PGPASS:?set PGPASS}"
|
|
secret_args=(
|
|
--secret id=b2_access,env=B2_S3_ACCESS_KEY
|
|
--secret id=b2_secret,env=B2_S3_SECRET_KEY
|
|
--secret id=pgpass,env=PGPASS
|
|
)
|
|
# GITHUB_PAT is optional (raises the GitHub API rate limit); pass only if set.
|
|
if [ -n "${GITHUB_PAT:-}" ]; then
|
|
secret_args+=(--secret id=github_pat,env=GITHUB_PAT)
|
|
fi
|
|
echo "Dispatching build of {{ package }} ({{ versions }}) on {{ os }}:{{ tag }}/{{ arch }} (R {{ r_version }}) to builder '$builder'"
|
|
docker buildx build \
|
|
--builder "$builder" \
|
|
--platform "linux/{{ arch }}" \
|
|
--no-cache \
|
|
--progress=plain \
|
|
--output type=cacheonly \
|
|
"${secret_args[@]}" \
|
|
--build-arg OS="{{ os }}" \
|
|
--build-arg OS_VERSION="{{ tag }}" \
|
|
--build-arg R_VERSION="{{ r_version }}" \
|
|
--build-arg PACKAGE="{{ package }}" \
|
|
--build-arg VERSIONS="{{ versions }}" \
|
|
--build-arg CACHEBUST="$(date +%s)" \
|
|
-f docker/build-one.Dockerfile \
|
|
local
|
|
|
|
# run the edge middleware routing matrix (uses a local deno, else the deno image)
|
|
edge-test:
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
if command -v deno >/dev/null 2>&1; then
|
|
deno test -A edge/rpkgs-router.test.ts
|
|
else
|
|
docker run --rm \
|
|
-v "$PWD:/w" -w /w \
|
|
-v deno-cache:/deno-dir \
|
|
denoland/deno:latest \
|
|
deno test -A edge/rpkgs-router.test.ts
|
|
fi
|