fix(build): keep source fallbacks out of the S3 package cache #159

Merged

pat-s merged 1 commit from fix/cache-excludes-source-fallback into main

2026-08-09 16:27:51 +00:00

pat-s commented

2026-08-09 16:25:32 +00:00

Owner

Problem

This is the gap flagged in rpkgs/bincraft#106. build_binary_package() has a fast path that compares against s3_package_cache instead of querying S3 per package, and that cache is produced here:

s3_pkgs <- s3fs::s3_dir_ls(".../latest/src/contrib", recurse = TRUE)
saveRDS(basename(s3_pkgs), "/mnt/cache/packages/s3_cache.rds")

A raw bucket listing cannot tell a binary from a package whose build failed and was published as its CRAN source — the two occupy the same key. So every source fallback reads as "already built" and is skipped for good. That is how alpine324 accumulated ~13.5k of them.

The same listing feeds s3_dt, which is subtracted from the build list at line 194 (pkgs <- pkgs_no_error[!s3_dt]). That one matters more: it excludes the very packages that need building, before build_binary_package() is even called.

What this changes

Drops from the listing every object the slot's own index reports as served from source. bincraft leaves the Built stamp off exactly those records (rpkgs/bincraft#105), so the index already carries the answer and no credentials, downloads or extra API calls are needed.

Both consumers are fixed: the saved cache and s3_dt.

The cache stays a plain filename vector, so the build container still needs no s3fs/reticulate — that was the point of saving it in the first place.

Two deliberately conservative edges:

archived objects have no index record, so they are kept. Unknown means binary, never "rebuild it".
if the index cannot be read, the full listing is kept and a warning is printed, so a CDN blip cannot mass-schedule a rebuild.

Verification

The script parses, and the new block run against the live indices:

amd64/alpine324: index=24235 source-served=13542  e.g. AATtools_0.0.3.tar.gz, ABCDscores_7.0.0.tar.gz
amd64/noble:     index=24681 source-served=0

alpine324 is re-indexed by bincraft 5.1.1, so 13 542 objects drop out and those packages become buildable. noble has not been re-indexed yet, so every record still carries Built, nothing is dropped, and its behaviour is exactly what it is today — the safe failure mode this relies on.

The new log line makes it visible per run:

S3 cache: N objects, M served as CRAN source, K usable binaries

Sequencing

Needs rpkgs/bincraft#106 (and a release) before a rebuild actually builds: this fixes the bulk build path's list, #106 fixes the per-package pre-build skip.

## Problem This is the gap flagged in rpkgs/bincraft#106. `build_binary_package()` has a fast path that compares against `s3_package_cache` instead of querying S3 per package, and that cache is produced here: ```r s3_pkgs <- s3fs::s3_dir_ls(".../latest/src/contrib", recurse = TRUE) saveRDS(basename(s3_pkgs), "/mnt/cache/packages/s3_cache.rds") ``` A raw bucket listing cannot tell a binary from a package whose build failed and was published as its CRAN source — the two occupy the same key. So every source fallback reads as "already built" and is skipped for good. That is how `alpine324` accumulated ~13.5k of them. The same listing feeds `s3_dt`, which is subtracted from the build list at line 194 (`pkgs <- pkgs_no_error[!s3_dt]`). That one matters more: it excludes the very packages that need building, before `build_binary_package()` is even called. ## What this changes Drops from the listing every object the slot's own index reports as served from source. bincraft leaves the `Built` stamp off exactly those records (rpkgs/bincraft#105), so the index already carries the answer and no credentials, downloads or extra API calls are needed. Both consumers are fixed: the saved cache and `s3_dt`. The cache stays a plain filename vector, so the build container still needs no `s3fs`/reticulate — that was the point of saving it in the first place. Two deliberately conservative edges: - archived objects have no index record, so they are kept. Unknown means binary, never "rebuild it". - if the index cannot be read, the full listing is kept and a warning is printed, so a CDN blip cannot mass-schedule a rebuild. ## Verification The script parses, and the new block run against the live indices: ``` amd64/alpine324: index=24235 source-served=13542 e.g. AATtools_0.0.3.tar.gz, ABCDscores_7.0.0.tar.gz amd64/noble: index=24681 source-served=0 ``` `alpine324` is re-indexed by bincraft 5.1.1, so 13 542 objects drop out and those packages become buildable. `noble` has not been re-indexed yet, so every record still carries `Built`, nothing is dropped, and its behaviour is exactly what it is today — the safe failure mode this relies on. The new log line makes it visible per run: ``` S3 cache: N objects, M served as CRAN source, K usable binaries ``` ## Sequencing Needs rpkgs/bincraft#106 (and a release) before a rebuild actually builds: this fixes the bulk build path's list, #106 fixes the per-package pre-build skip.

pat-s added 1 commit

2026-08-09 16:25:32 +00:00

fix(build): keep source fallbacks out of the S3 package cache 6b0a987a6a

The cache handed to build_binary_package() as s3_package_cache was the raw
bucket listing, and the build list subtracted the same listing. Neither could
tell a binary from a package whose build failed and was published as its CRAN
source, so every source fallback read as "already built" and was skipped for
good - which is how alpine324 accumulated ~13.5k of them.

- drop objects the slot's index reports as served from source, which bincraft
  marks by leaving the Built stamp off
- build s3_dt from the filtered listing too, since it is subtracted from the
  build list and would otherwise exclude the packages that need building
- log how many objects were dropped

Archived objects have no index record and are kept: unknown means binary, never
"rebuild it". A slot last indexed by a bincraft that predates the Built change
stamps everything, so its cache is unchanged from before.

pat-s merged commit a126d74cd3 into main

2026-08-09 16:27:51 +00:00

pat-s deleted branch fix/cache-excludes-source-fallback

2026-08-09 16:27:52 +00:00

pat-s referenced this pull request from a commit

2026-08-09 16:27:53 +00:00

fix(build): keep source fallbacks out of the S3 package cache (#159)

pat-s referenced this pull request

2026-08-12 08:28:19 +00:00

feat(rebuild): shard the weekly rebuild and make each shard resumable #163

pat-s referenced this pull request from a commit

2026-08-12 08:30:30 +00:00