devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 11 Pull requests Releases Activity

fix(build): strip the s3 scheme so per-minor objects reach their pass #192

Merged
pat-s merged 2 commits from fix/strip-s3-scheme-from-cache-paths into main 2026-09-02 07:11:11 +00:00
Conversation 0 Commits 2 Files changed 3 +50 -9
pat-s commented 2026-09-01 22:47:58 +00:00
Owner

Why

#191 gave the existence cache a per-minor view of the slot, but the paths it filters never carried a minor prefix, so the fix could not take effect.

s3fs::s3_dir_ls() returns keys with the s3:// scheme attached:

s3://devxy-rpkgs-binaries/amd64/resolute/latest/src/contrib/4.4/oeli_0.7.6.tar.gz

The contrib prefix was stripped as a fixed substring, so it was removed from the middle of the key and the scheme survived:

s3://4.4/oeli_0.7.6.tar.gz

That leading s3:// defeats both ^<minor>/ and ^[0-9]+\.[0-9]+/, so all 21212 per-minor objects on amd64/resolute were classified as flat-slot objects. Pipeline 12011 shows it exactly:

sensitive pass:      0 files (0      of 119053 objects apply to this pass)
primary pass:   119053 files (119053 of 119053 objects apply to this pass)

Each sensitive pass therefore ran with an empty cache and recompiled all 10248 sensitive packages it already had.

What changed

  • local/packages-to-build.R: anchor the contrib prefix and swallow an optional s3:// with it.
  • local/packages-to-build.R: abort when the strip leaves fewer per-minor paths than the listing held. The failure mode is silent and only surfaces as a multi-hour rebuild, and both counts derive from the same listing so they must agree exactly.
  • local/packages-to-build.R / local/build-all.R: mark the cache with a slot_relative attribute and read that, instead of sniffing for a /. A slot-relative cache for a slot with no per-minor or Archive object holds bare names too, and would have been misread as legacy and used unfiltered, which makes a per-minor pass believe the flat slot's binaries are its own and build nothing.
  • scripts/purge_cdn_zone.sh: indent the jq continuation lines by a multiple of two. This is unrelated, but it fails editorconfig-checker on main and blocks prek run -a for everyone. jq ignores the whitespace, and both response shapes still resolve.

Verification

Replaying the real key shapes through the old and new code:

            sensitive(4.6)  primary
OLD paths:  0               8        <- reproduces production
NEW paths:  3               3

Invariant NEW: listing=4 stripped=4 -> PASS
Invariant OLD: listing=4 stripped=0 -> ABORT (would have caught this)

Per-minor Archive/ objects are attributed to their minor, and the flat bucket keeps its own Archive/.

prek run -a passes.

Pipelines 12011-12015 were stopped rather than left to spend hours recompiling what they already had. Their uploads are not lost, so a fresh run inherits them.

## Why #191 gave the existence cache a per-minor view of the slot, but the paths it filters never carried a minor prefix, so the fix could not take effect. `s3fs::s3_dir_ls()` returns keys with the `s3://` scheme attached: ``` s3://devxy-rpkgs-binaries/amd64/resolute/latest/src/contrib/4.4/oeli_0.7.6.tar.gz ``` The contrib prefix was stripped as a *fixed substring*, so it was removed from the middle of the key and the scheme survived: ``` s3://4.4/oeli_0.7.6.tar.gz ``` That leading `s3://` defeats both `^<minor>/` and `^[0-9]+\.[0-9]+/`, so all 21212 per-minor objects on amd64/resolute were classified as flat-slot objects. Pipeline 12011 shows it exactly: ``` sensitive pass: 0 files (0 of 119053 objects apply to this pass) primary pass: 119053 files (119053 of 119053 objects apply to this pass) ``` Each sensitive pass therefore ran with an empty cache and recompiled all 10248 sensitive packages it already had. ## What changed - `local/packages-to-build.R`: anchor the contrib prefix and swallow an optional `s3://` with it. - `local/packages-to-build.R`: abort when the strip leaves fewer per-minor paths than the listing held. The failure mode is silent and only surfaces as a multi-hour rebuild, and both counts derive from the same listing so they must agree exactly. - `local/packages-to-build.R` / `local/build-all.R`: mark the cache with a `slot_relative` attribute and read that, instead of sniffing for a `/`. A slot-relative cache for a slot with no per-minor or Archive object holds bare names too, and would have been misread as legacy and used unfiltered, which makes a per-minor pass believe the flat slot's binaries are its own and build nothing. - `scripts/purge_cdn_zone.sh`: indent the jq continuation lines by a multiple of two. This is unrelated, but it fails editorconfig-checker on `main` and blocks `prek run -a` for everyone. jq ignores the whitespace, and both response shapes still resolve. ## Verification Replaying the real key shapes through the old and new code: ``` sensitive(4.6) primary OLD paths: 0 8 <- reproduces production NEW paths: 3 3 Invariant NEW: listing=4 stripped=4 -> PASS Invariant OLD: listing=4 stripped=0 -> ABORT (would have caught this) ``` Per-minor `Archive/` objects are attributed to their minor, and the flat bucket keeps its own `Archive/`. `prek run -a` passes. Pipelines 12011-12015 were stopped rather than left to spend hours recompiling what they already had. Their uploads are not lost, so a fresh run inherits them. 
`s3_dir_ls()` returns keys with the `s3://` scheme attached, but the contrib
prefix was removed as a fixed substring. That took the prefix out of the middle
of the key and left `s3://4.4/curl_1.0.tar.gz`, so the leading scheme defeated
the `^<minor>/` test in `select_cache_for_pass()`.

Every one of the 21212 per-minor objects on amd64/resolute was therefore read as
a flat-slot object: the primary pass reported "119053 of 119053 objects apply to
this pass" and each sensitive pass got an empty cache and recompiled all 10248
sensitive packages it already had.

This change will:

- anchor the contrib prefix and swallow an optional `s3://` with it
- abort when the strip leaves fewer per-minor paths than the listing held, since
  the failure is otherwise silent and only shows up as a full rebuild
- mark the cache format with a `slot_relative` attribute and read that in
  `build-all.R`, rather than sniffing for a `/`, which misreads a slot-relative
  cache holding no per-minor or Archive object as a legacy one
The continuation lines of the zone-lookup filter sat at nine spaces, which fails
editorconfig-checker and blocks `prek run -a` for everyone. jq ignores the
leading whitespace, so the filter is unchanged: it still resolves the zone id
from both the object and the bare array response shape.
pat-s merged commit d4f093923a into main 2026-09-02 07:11:11 +00:00
pat-s deleted branch fix/strip-s3-scheme-from-cache-paths 2026-09-02 07:11:11 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries!192
No description provided.