fix(abi): enumerate the flat slot from S3, not through the edge #203

Merged

pat-s merged 1 commit from fix/flat-safety-enumeration into main

2026-09-05 09:41:58 +00:00

pat-s commented

2026-09-05 09:40:51 +00:00

Owner

Why

The first backfill run reported:

amd64/resolute: 27105 flat binaries, 8187 compiled and worth inspecting
Done: 300 inspected, 7 safe for every minor, 1 unsafe, 292 skipped

292 of 300 skipped, and 27105 is larger than the flat index (~24k). Both have the same cause: the script read PACKAGES.gz over HTTPS from inside R, which sends an R User-Agent, so the edge routed it to a per-minor index. It then enumerated union records carrying Path: <minor>, whose objects do not exist at the flat path, and every download 404'd.

An ironic failure given what this whole change is about, and one the trial run existed to catch.

What changed

List the flat slot with s3fs::s3_dir_ls(), keeping only objects with no <minor>/ prefix and nothing under Archive/. The bucket cannot be routed and is authoritative about what is actually in the flat slot.
Fetch each object from S3 too. The edge does not rewrite tarball requests, but it does cache them, and the verdict is about the object.
NeedsCompilation still comes from CRAN's own index, which is not behind our edge.

Verification

The next trial run is the check: flat objects should now match the flat index rather than exceed it, and skipped should be near zero.

## Why The first backfill run reported: ``` amd64/resolute: 27105 flat binaries, 8187 compiled and worth inspecting Done: 300 inspected, 7 safe for every minor, 1 unsafe, 292 skipped ``` 292 of 300 skipped, and `27105` is larger than the flat index (~24k). Both have the same cause: the script read `PACKAGES.gz` over HTTPS from inside R, which sends an R User-Agent, so **the edge routed it to a per-minor index**. It then enumerated union records carrying `Path: <minor>`, whose objects do not exist at the flat path, and every download 404'd. An ironic failure given what this whole change is about, and one the trial run existed to catch. ## What changed - List the flat slot with `s3fs::s3_dir_ls()`, keeping only objects with no `<minor>/` prefix and nothing under `Archive/`. The bucket cannot be routed and is authoritative about what is actually in the flat slot. - Fetch each object from S3 too. The edge does not rewrite tarball requests, but it does cache them, and the verdict is about the object. - `NeedsCompilation` still comes from CRAN's own index, which is not behind our edge. ## Verification The next trial run is the check: `flat objects` should now match the flat index rather than exceed it, and `skipped` should be near zero.

pat-s added 1 commit

2026-09-05 09:40:51 +00:00

fix(abi): enumerate the flat slot from S3, not through the edge bd0003dfdb

Reading PACKAGES.gz over HTTPS from inside R sends an R User-Agent, which the
edge routes to a per-minor index. The script then enumerated union records
carrying `Path: <minor>`, whose objects do not exist at the flat path, so every
download 404'd: the first run skipped 292 of 300.

The bucket cannot be routed and is authoritative about which objects are
actually in the flat slot, so list and fetch from there.

pat-s force-pushed fix/flat-safety-enumeration from bd0003dfdb to 1c1b242dc6

2026-09-05 09:41:00 +00:00

Compare

pat-s merged commit 844ad9749b into main

2026-09-05 09:41:58 +00:00

pat-s deleted branch fix/flat-safety-enumeration

2026-09-05 09:41:58 +00:00

pat-s referenced this pull request from a commit

2026-09-05 09:41:59 +00:00

fix(abi): enumerate the flat slot from S3, not through the edge (#203)

pat-s referenced this pull request

2026-09-09 18:27:17 +00:00