devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 11 Pull requests Releases Activity

fix(build): give the existence cache a per-minor view of the slot

Browse source 
#189 stopped per-minor objects from pruning the candidate list, which was
right, but s3_cache.rds is derived from the same file_names. The cache
handed to build_binary_package() therefore contained no per-minor object
at all, its 'not present in the remote bucket' check answered not-present
for every one, and a restarted run recompiled everything it had already
built: amd64/resolute attempted 8683 packages, compiled 8683, and
reported 'already exists in S3' three times.

The two consumers need opposite views of one listing, and conflating them
was wrong in both directions. The candidate list must ignore per-minor
objects or a package present under one minor prunes itself from every
other minor's work. The existence cache must not ignore them or the work
is redone.

The cache now keeps paths relative to the slot and build-all.R selects
the part matching its pass: <minor>/ for a per-minor pass, the flat slot
plus Archive/ for the primary. build_binary_package() compares basenames
and cannot tell 4.4/curl from curl, so the choice has to be made where
the running R minor is known.

A cache written before this change holds bare basenames; filtering those
by path would select nothing and trigger the very rebuild this prevents,
so they are used unfiltered and #190's staleness check replaces them on
the next pipeline.
This commit is contained in:
Patrick Schratz 2026-09-01 21:47:17 +00:00
commit c442e8d35c
No known key found for this signature in database
GPG key ID: 62050D5BC68AB6DC
1 changed files with 59 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

38
local/build-all.R
View file

@ -189,8 +189,42 @@ sprintf(
# Read pre-computed S3 listing from install-deps step
# This avoids loading s3fs/reticulate/Python in the build container,
# saving significant memory for the dependency-installer subprocesses
s3_cache <- readRDS("/mnt/cache/packages/s3_cache.rds")
sprintf("S3 cache: %s files", length(s3_cache))
s3_cache_paths <- readRDS("/mnt/cache/packages/s3_cache.rds")
# The cache is stored as paths relative to the slot, so a pass can select the
# objects that belong to it. `build_binary_package()` compares basenames, which
# cannot distinguish `4.4/curl_1.0.tar.gz` from `curl_1.0.tar.gz`, so the choice
# has to be made here where the running R minor is known.
#
# Getting this wrong is expensive in both directions: hand it everything and a
# per-minor pass believes the flat slot's binaries are its own and builds
# nothing; hand it nothing and it rebuilds what it already has. amd64/resolute
# recompiled 8683 packages that way.
select_cache_for_pass <- function(paths, sensitive_only, r_minor) {
# A cache written before this change holds bare basenames. Filtering those by
# path would select nothing and trigger a full rebuild, so use them as they
# are; #190's staleness check replaces it on the next pipeline anyway.
if (!any(grepl("/", paths, fixed = TRUE))) {
message("S3 cache is in the legacy basename format; using it unfiltered.")
return(paths)
}
in_minor <- grepl(sprintf("^%s/", r_minor), paths)
if (sensitive_only) {
basename(paths[in_minor])
} else {
# The primary pass writes the flat slot. Archive/ counts as present there:
# those are versions built and later superseded.
basename(paths[!grepl("^[0-9]+\\.[0-9]+/", paths)])
}
}
s3_cache <- select_cache_for_pass(s3_cache_paths, sensitive_only, r_minor)
sprintf(
"S3 cache: %s files (%s of %s objects apply to this pass)",
length(s3_cache),
length(s3_cache),
length(s3_cache_paths)
)
n <- nrow(chunk)
mapply(