fix(build): give the existence cache a per-minor view of the slot #191

Merged

pat-s merged 1 commit from fix/s3-cache-per-minor into main

2026-09-01 21:54:13 +00:00

pat-s commented

2026-09-01 21:47:34 +00:00

Owner

Motivation

A restarted run recompiled everything it had already built.

attempted:                  8683
already exists in S3:          3
actually compiled:          8683

amd64/resolute had uploaded roughly 3000 binaries before being restarted, and rebuilt all of them.

Cause

This is a regression from #189, which I introduced.

s3_cache.rds is derived from the same file_names that #189 filtered:

binary_cache <- setdiff(file_names, source_served)
saveRDS(binary_cache, "/mnt/cache/packages/s3_cache.rds")

Removing per-minor objects from file_names was right for the candidate list and wrong for the existence cache. build_binary_package() was handed a cache containing no per-minor object at all, so its "not present in the remote bucket" check answered not-present for every one of them.

The two consumers need opposite views of one listing:

the candidate list must ignore per-minor objects, or a package present under one minor prunes itself from every other minor's work;
the existence cache must not ignore them, or the work is redone.

Conflating them was wrong in both directions: before #189 a per-minor pass believed the flat slot's binaries were its own and built nothing; after #189 it believed it had nothing and rebuilt the lot.

Change

The cache keeps paths relative to the slot, and build-all.R selects the part matching the pass it is running. build_binary_package() compares basenames and cannot distinguish 4.4/curl_1.0.tar.gz from curl_1.0.tar.gz, so the choice has to be made where the running R minor is known.

Verified:

pass	selects
primary (writes flat)	`curl_1.0`, `jsonlite_2.0`, `Archive/curl_0.9`, `PACKAGES.gz`
`--sensitive-only` under 4.4	`curl_1.0`, `rlang_1.3.0` (from `4.4/` only)
`--sensitive-only` under 4.6	`rlang_1.3.0` (from `4.6/` only)
legacy basename cache	passthrough, unfiltered

Archive/ counts as present for the primary pass: those are versions built and later superseded.

Backwards compatibility

A cache written before this change holds bare basenames. Filtering those by path would select nothing and trigger the very rebuild this prevents, so they are detected and used unfiltered; #190's staleness check replaces them on the next pipeline.

Note on the running builds

The five pipelines currently running are recompiling packages they already have. Their output is still correct — the binaries carry the right Built stamp for their slot — but the work is wasted. They should be restarted once this is merged.

## Motivation A restarted run recompiled everything it had already built. ``` attempted: 8683 already exists in S3: 3 actually compiled: 8683 ``` `amd64/resolute` had uploaded roughly 3000 binaries before being restarted, and rebuilt all of them. ## Cause This is a regression from #189, which I introduced. `s3_cache.rds` is derived from the same `file_names` that #189 filtered: ```r binary_cache <- setdiff(file_names, source_served) saveRDS(binary_cache, "/mnt/cache/packages/s3_cache.rds") ``` Removing per-minor objects from `file_names` was right for the candidate list and wrong for the existence cache. `build_binary_package()` was handed a cache containing no per-minor object at all, so its "not present in the remote bucket" check answered not-present for every one of them. The two consumers need opposite views of one listing: - the **candidate list** must ignore per-minor objects, or a package present under one minor prunes itself from every other minor's work; - the **existence cache** must not ignore them, or the work is redone. Conflating them was wrong in both directions: before #189 a per-minor pass believed the flat slot's binaries were its own and built nothing; after #189 it believed it had nothing and rebuilt the lot. ## Change The cache keeps paths relative to the slot, and `build-all.R` selects the part matching the pass it is running. `build_binary_package()` compares basenames and cannot distinguish `4.4/curl_1.0.tar.gz` from `curl_1.0.tar.gz`, so the choice has to be made where the running R minor is known. Verified: | pass | selects | |---|---| | primary (writes flat) | `curl_1.0`, `jsonlite_2.0`, `Archive/curl_0.9`, `PACKAGES.gz` | | `--sensitive-only` under 4.4 | `curl_1.0`, `rlang_1.3.0` (from `4.4/` only) | | `--sensitive-only` under 4.6 | `rlang_1.3.0` (from `4.6/` only) | | legacy basename cache | passthrough, unfiltered | `Archive/` counts as present for the primary pass: those are versions built and later superseded. ## Backwards compatibility A cache written before this change holds bare basenames. Filtering those by path would select nothing and trigger the very rebuild this prevents, so they are detected and used unfiltered; #190's staleness check replaces them on the next pipeline. ## Note on the running builds The five pipelines currently running are recompiling packages they already have. Their output is still correct — the binaries carry the right `Built` stamp for their slot — but the work is wasted. They should be restarted once this is merged.

pat-s added 1 commit

2026-09-01 21:47:35 +00:00

fix(build): give the existence cache a per-minor view of the slot c442e8d35c

#189 stopped per-minor objects from pruning the candidate list, which was
right, but s3_cache.rds is derived from the same file_names. The cache
handed to build_binary_package() therefore contained no per-minor object
at all, its 'not present in the remote bucket' check answered not-present
for every one, and a restarted run recompiled everything it had already
built: amd64/resolute attempted 8683 packages, compiled 8683, and
reported 'already exists in S3' three times.

The two consumers need opposite views of one listing, and conflating them
was wrong in both directions. The candidate list must ignore per-minor
objects or a package present under one minor prunes itself from every
other minor's work. The existence cache must not ignore them or the work
is redone.

The cache now keeps paths relative to the slot and build-all.R selects
the part matching its pass: <minor>/ for a per-minor pass, the flat slot
plus Archive/ for the primary. build_binary_package() compares basenames
and cannot tell 4.4/curl from curl, so the choice has to be made where
the running R minor is known.

A cache written before this change holds bare basenames; filtering those
by path would select nothing and trigger the very rebuild this prevents,
so they are used unfiltered and #190's staleness check replaces them on
the next pipeline.

pat-s merged commit 8f97cf99ef into main

2026-09-01 21:54:13 +00:00

pat-s deleted branch fix/s3-cache-per-minor

2026-09-01 21:54:14 +00:00

pat-s referenced this pull request from a commit

2026-09-01 21:54:15 +00:00

fix(build): give the existence cache a per-minor view of the slot (#191)

pat-s referenced this pull request

2026-09-01 22:47:58 +00:00

fix(build): strip the s3 scheme so per-minor objects reach their pass #192