fix(build): give the existence cache a per-minor view of the slot (#191)
Some checks failed
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline was canceled
ci/crow/manual/build-all-versions/7 Pipeline was canceled
ci/crow/manual/build-all-versions/4 Pipeline was canceled
ci/crow/manual/build-all-versions/5 Pipeline was canceled
ci/crow/manual/build-all-versions/8 Pipeline was canceled
ci/crow/manual/build-all-versions/6 Pipeline was canceled
ci/crow/manual/build-all-versions/3 Pipeline was canceled
ci/crow/manual/build-all-versions/2 Pipeline was canceled
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline failed
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful

## Motivation

A restarted run recompiled everything it had already built.

```
attempted:                  8683
already exists in S3:          3
actually compiled:          8683
```

`amd64/resolute` had uploaded roughly 3000 binaries before being restarted, and rebuilt all of them.

## Cause

This is a regression from #189, which I introduced.

`s3_cache.rds` is derived from the same `file_names` that #189 filtered:

```r
binary_cache <- setdiff(file_names, source_served)
saveRDS(binary_cache, "/mnt/cache/packages/s3_cache.rds")
```

Removing per-minor objects from `file_names` was right for the candidate list and wrong for the existence cache. `build_binary_package()` was handed a cache containing no per-minor object at all, so its "not present in the remote bucket" check answered not-present for every one of them.

The two consumers need opposite views of one listing:

- the **candidate list** must ignore per-minor objects, or a package present under one minor prunes itself from every other minor's work;
- the **existence cache** must not ignore them, or the work is redone.

Conflating them was wrong in both directions: before #189 a per-minor pass believed the flat slot's binaries were its own and built nothing; after #189 it believed it had nothing and rebuilt the lot.

## Change

The cache keeps paths relative to the slot, and `build-all.R` selects the part matching the pass it is running. `build_binary_package()` compares basenames and cannot distinguish `4.4/curl_1.0.tar.gz` from `curl_1.0.tar.gz`, so the choice has to be made where the running R minor is known.

Verified:

| pass | selects |
|---|---|
| primary (writes flat) | `curl_1.0`, `jsonlite_2.0`, `Archive/curl_0.9`, `PACKAGES.gz` |
| `--sensitive-only` under 4.4 | `curl_1.0`, `rlang_1.3.0` (from `4.4/` only) |
| `--sensitive-only` under 4.6 | `rlang_1.3.0` (from `4.6/` only) |
| legacy basename cache | passthrough, unfiltered |

`Archive/` counts as present for the primary pass: those are versions built and later superseded.

## Backwards compatibility

A cache written before this change holds bare basenames. Filtering those by path would select nothing and trigger the very rebuild this prevents, so they are detected and used unfiltered; #190's staleness check replaces them on the next pipeline.

## Note on the running builds

The five pipelines currently running are recompiling packages they already have. Their output is still correct — the binaries carry the right `Built` stamp for their slot — but the work is wasted. They should be restarted once this is merged.

Reviewed-on: #191
This commit is contained in:
Patrick Schratz 2026-09-01 21:54:13 +00:00 committed by Patrick Schratz
commit 8f97cf99ef

View file

@ -176,6 +176,28 @@ source_served <- tryCatch(
)
binary_cache <- setdiff(file_names, source_served)
# The existence cache and the candidate list need different views of the same
# listing, and conflating them is what made this wrong in both directions.
#
# The candidate list must ignore per-minor objects, or a package present under
# one minor prunes itself from every other minor's work (#189). The existence
# cache must NOT ignore them, or `build_binary_package()` is told nothing is
# present under any minor and recompiles the lot: amd64/resolute recompiled
# 8683 packages it had already built, reporting "already exists in S3" three
# times.
#
# So the cache keeps the path relative to the slot, and `build-all.R` selects
# the part that matches the pass it is running: the flat slot for the primary,
# `<minor>/` for a per-minor pass.
contrib_prefix <- sprintf(
"devxy-rpkgs-binaries/%s/%s/latest/src/contrib/",
arch,
codename
)
relative_paths <- sub(contrib_prefix, "", s3_pkgs, fixed = TRUE)
source_basenames <- source_served
existence_cache <- relative_paths[!basename(relative_paths) %in% source_basenames]
cat(sprintf(
"S3 cache: %d objects, %d served as CRAN source, %d usable binaries\n",
length(file_names),
@ -186,7 +208,7 @@ cat(sprintf(
# Save the S3 file listing for the build step to use as s3_package_cache.
# This avoids loading s3fs/reticulate in the build container, saving memory for
# the dependency-installer subprocesses
saveRDS(binary_cache, "/mnt/cache/packages/s3_cache.rds")
saveRDS(existence_cache, "/mnt/cache/packages/s3_cache.rds")
# Built from the filtered listing, not the raw one: `s3_dt` is subtracted from
# the build list below, so a source fallback left in here would exclude the very
# package that needs building.