fix(rebuild): re-index and purge the CDN after a rebuild (#160)
Some checks failed
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline is running
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/weekly-audit-missing/6 Pipeline was successful
ci/crow/manual/weekly-audit-missing/5 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/5 Pipeline was canceled

## Problem

The rebuild now works — `AATtools 0.0.3` was detected as a source fallback, built, and published:

```
ℹ `upload_single_binary()`: Replacing the CRAN source published for AATtools 0.0.3 … with the binary.
✔ Successfully uploaded package AATtools with tag 0.0.3.
```

But clients still get the source, and will for about a year:

```
$ curl -sI .../src/contrib/AATtools_0.0.3.tar.gz
etag: "ea8127d953ca6a2f118ea49441772af6"   # CRAN's source MD5
cdn-cache: HIT
cdn-cachedat: 08/09/2026 16:43:32          # predates the 18:12 upload
```

Two causes, both specific to a rebuild:

1. **The slot is never re-indexed.** `weekly-rebuild-missing` has no `upload_package_index` step, so the index keeps the old MD5 and — for anything that had been served from source — no `Built` stamp. This one self-heals at the next `process-updates` run.
2. **The tarball URL is never purged.** A normal update publishes new packages at *new* URLs, so `purge_cdn_cache.sh` only needs the five index files. A rebuild replaces an object *in place*, and the zone caches tarballs for `cache_expiration_time = 31919000` (~370 days). This does not self-heal.

Nothing about a stale package looks wrong from the outside, which is what makes it worth fixing rather than documenting.

## What this changes

**Re-index at the end of a rebuild**, flat and per-minor, mirroring the tail of `process-updates`. The codename is detected from the image's `/etc/os-release` (as `local/packages-to-build.R` already does) rather than adding `OS_ID` to all 18 matrix rows.

**Purge the zone afterwards**, via a new `scripts/purge_cdn_zone.sh`. One call to `POST /pullzone/{id}/purgeCache` covers every replaced object, and all three hostnames — `cran.devxy.io`, `cran.allianceswisspass.devxy.io`, `cran.rpkgs.com` — share pull zone `3857050`, confirmed from the `cdn-pullzone` response header.

Purging per URL was the alternative and is worse here: ~13.5k rate-limited calls per arch, where a single missed call leaves a package silently stale. The cost of the zone purge is a cold cache for everything else, which is why it stays out of the daily update path — `purge_cdn_cache.sh` is untouched.

The purge runs on failure too (`when: status: [success, failure]`): a rebuild that died part-way still replaced objects, and those are exactly the ones a stale edge keeps hiding.

## Verification

`crow lint .crow/` reports all ten configs valid; `bash -n` on the new script passes; prek hooks pass.

Not yet exercised against Bunny — it needs `BUNNYNET_API_KEY`, which is a CI secret. The failure mode is explicit rather than silent: any status other than 200/204 prints the response body and exits non-zero.

Reviewed-on: #160
This commit is contained in:
Patrick Schratz 2026-08-10 06:30:35 +00:00 committed by Patrick Schratz
commit 01b8ab43df

52
scripts/purge_cdn_zone.sh Executable file
View file

@ -0,0 +1,52 @@
#!/usr/bin/env bash
#
# Purge the entire BunnyCDN pull zone.
#
# `purge_cdn_cache.sh` purges the five index files by URL, which is right after
# a normal update: new packages arrive at new URLs, so only the index is stale.
#
# A rebuild is different. It replaces an object *in place*: a package whose
# build failed was published as its CRAN source, and the rebuilt binary takes
# exactly the same URL. The zone caches tarballs for ~370 days
# (`cache_expiration_time` in cdn.tf), so without a purge every client keeps
# receiving the source tarball for up to a year, and nothing about it looks
# wrong from the outside.
#
# Purging per URL would mean one API call per replaced package -- ~13.5k per
# arch against a rate-limited endpoint, where a single missed call leaves a
# silently stale package. One zone purge is a single call regardless of how many
# objects were replaced. The cost is a cold cache for everything else, which is
# why this is not used by the daily update path.
#
# All hostnames on the zone (cran.devxy.io, cran.allianceswisspass.devxy.io,
# cran.rpkgs.com) share pull zone 3857050, so one purge covers all of them.
#
# Usage:
# purge_cdn_zone.sh <BUNNYNET_API_KEY> <pull_zone_id>
#
set -euo pipefail
if (($# < 2)); then
echo "usage: $0 <api_key> <pull_zone_id>" >&2
exit 2
fi
api_key="$1"
zone_id="$2"
echo "Purging BunnyCDN pull zone ${zone_id}"
status=$(
curl -sS -o /tmp/purge_zone_response.txt -w '%{http_code}' -X POST \
-H "AccessKey: ${api_key}" \
-H "Content-Length: 0" \
"https://api.bunny.net/pullzone/${zone_id}/purgeCache"
)
if [[ "${status}" != "200" && "${status}" != "204" ]]; then
echo "Purge of pull zone ${zone_id} failed with HTTP ${status}:" >&2
cat /tmp/purge_zone_response.txt >&2
exit 1
fi
echo "Purged pull zone ${zone_id} (HTTP ${status})"