devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 11 Pull requests Releases Activity

fix(build): give the existence cache a per-minor view of the slot

Browse source 
#189 stopped per-minor objects from pruning the candidate list, which was
right, but s3_cache.rds is derived from the same file_names. The cache
handed to build_binary_package() therefore contained no per-minor object
at all, its 'not present in the remote bucket' check answered not-present
for every one, and a restarted run recompiled everything it had already
built: amd64/resolute attempted 8683 packages, compiled 8683, and
reported 'already exists in S3' three times.

The two consumers need opposite views of one listing, and conflating them
was wrong in both directions. The candidate list must ignore per-minor
objects or a package present under one minor prunes itself from every
other minor's work. The existence cache must not ignore them or the work
is redone.

The cache now keeps paths relative to the slot and build-all.R selects
the part matching its pass: <minor>/ for a per-minor pass, the flat slot
plus Archive/ for the primary. build_binary_package() compares basenames
and cannot tell 4.4/curl from curl, so the choice has to be made where
the running R minor is known.

A cache written before this change holds bare basenames; filtering those
by path would select nothing and trigger the very rebuild this prevents,
so they are used unfiltered and #190's staleness check replaces them on
the next pipeline.
This commit is contained in:
Patrick Schratz 2026-09-01 21:47:17 +00:00
commit c442e8d35c
No known key found for this signature in database
GPG key ID: 62050D5BC68AB6DC
1 changed files with 59 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

24
local/packages-to-build.R
View file

@ -176,6 +176,28 @@ source_served <- tryCatch(
)
binary_cache <- setdiff(file_names, source_served)
# The existence cache and the candidate list need different views of the same
# listing, and conflating them is what made this wrong in both directions.
#
# The candidate list must ignore per-minor objects, or a package present under
# one minor prunes itself from every other minor's work (#189). The existence
# cache must NOT ignore them, or `build_binary_package()` is told nothing is
# present under any minor and recompiles the lot: amd64/resolute recompiled
# 8683 packages it had already built, reporting "already exists in S3" three
# times.
#
# So the cache keeps the path relative to the slot, and `build-all.R` selects
# the part that matches the pass it is running: the flat slot for the primary,
# `<minor>/` for a per-minor pass.
contrib_prefix <- sprintf(
"devxy-rpkgs-binaries/%s/%s/latest/src/contrib/",
arch,
codename
)
relative_paths <- sub(contrib_prefix, "", s3_pkgs, fixed = TRUE)
source_basenames <- source_served
existence_cache <- relative_paths[!basename(relative_paths) %in% source_basenames]
cat(sprintf(
"S3 cache: %d objects, %d served as CRAN source, %d usable binaries\n",
length(file_names),
@ -186,7 +208,7 @@ cat(sprintf(
# Save the S3 file listing for the build step to use as s3_package_cache.
# This avoids loading s3fs/reticulate in the build container, saving memory for
# the dependency-installer subprocesses
saveRDS(binary_cache, "/mnt/cache/packages/s3_cache.rds")
saveRDS(existence_cache, "/mnt/cache/packages/s3_cache.rds")
# Built from the filtered listing, not the raw one: `s3_dt` is subtracted from
# the build list below, so a source fallback left in here would exclude the very
# package that needs building.