devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 11 Pull requests Releases Activity

fix(build): stop per-minor objects masking the per-minor candidate list #189

Merged
pat-s merged 1 commit from fix/candidate-list-per-minor into main 2026-08-31 21:36:12 +00:00
Conversation 0 Commits 1 Files changed 1 +35 -3
pat-s commented 2026-08-31 21:30:18 +00:00
Owner

Motivation

arm64/alpine324 (pipeline 11953) finished in minutes having uploaded 57 packages, and reported:

Skipped 0 package versions already attempted under R 4.4; 0 remaining
Skipped 0 package versions already attempted under R 4.6; 3 remaining

The same run's index step dropped 2407 packages as missing for 4.4 and 2436 for 4.6. Nothing to build, and thousands missing — the candidate list is wrong.

Two omissions

1. Per-minor objects mask the per-minor candidates.

s3_pkgs <- s3fs::s3_dir_ls(".../latest/src/contrib", recurse = TRUE)
file_names <- basename(s3_pkgs)

recurse = TRUE walks 4.4/, 4.5/, 4.6/; basename() throws the directory away. 4.5/curl_1.0.tar.gz and curl_1.0.tar.gz collapse to one name, so a package present under any R minor counts as built for all of them — pruning exactly the packages a per-minor pass exists to build.

Per-minor objects are now excluded, and presence in a specific minor is decided downstream where the running R version is known: build-all.R filters on it, and build_binary_package() checks the per-minor path per package and skips what is already there.

Archive/ is kept. Those are versions built and later superseded; dropping them would make every archived version look unbuilt.

Validated against real path shapes:

path
curl_1.0.tar.gz keep
4.4/curl_1.0.tar.gz exclude
4.6/rlang_1.3.0.tar.gz exclude
Archive/curl/curl_0.9.tar.gz keep
PACKAGES.gz keep

2. The error query ignores r_version.

SELECT error_occurred FROM single_builds
 WHERE name = $1 AND tag = $2 AND platform = $3 AND arch = $4

A failure under the primary minor drops the package from every other minor's candidate list. This is the same omission fixed in local/build-all.R (#187) and in bincraft's check_package_error() (rpkgs/bincraft#119). This is the third and last consumer of that table — I have grepped the rest; bincraft::R/cran-internal.R also reads it, but to list packages present rather than to skip, where the R minor does not apply.

Expected effect

The per-minor passes get real candidate lists. Expect slots that reported "0 remaining" to report thousands, and correspondingly long runs.

There is a cost: the list is no longer pruned by per-minor presence, so each pass asks build_binary_package() about packages that may already exist, and it answers already exists in S3 ... Skipping build per package. Slower per pass, and correct — the pruning it replaces was removing the wrong things.

## Motivation `arm64/alpine324` (pipeline 11953) finished in minutes having uploaded 57 packages, and reported: ``` Skipped 0 package versions already attempted under R 4.4; 0 remaining Skipped 0 package versions already attempted under R 4.6; 3 remaining ``` The same run's index step dropped **2407** packages as missing for 4.4 and **2436** for 4.6. Nothing to build, and thousands missing — the candidate list is wrong. ## Two omissions **1. Per-minor objects mask the per-minor candidates.** ```r s3_pkgs <- s3fs::s3_dir_ls(".../latest/src/contrib", recurse = TRUE) file_names <- basename(s3_pkgs) ``` `recurse = TRUE` walks `4.4/`, `4.5/`, `4.6/`; `basename()` throws the directory away. `4.5/curl_1.0.tar.gz` and `curl_1.0.tar.gz` collapse to one name, so a package present under **any** R minor counts as built for **all** of them — pruning exactly the packages a per-minor pass exists to build. Per-minor objects are now excluded, and presence in a specific minor is decided downstream where the running R version is known: `build-all.R` filters on it, and `build_binary_package()` checks the per-minor path per package and skips what is already there. `Archive/` is kept. Those are versions built and later superseded; dropping them would make every archived version look unbuilt. Validated against real path shapes: | path | | |---|---| | `curl_1.0.tar.gz` | keep | | `4.4/curl_1.0.tar.gz` | exclude | | `4.6/rlang_1.3.0.tar.gz` | exclude | | `Archive/curl/curl_0.9.tar.gz` | keep | | `PACKAGES.gz` | keep | **2. The error query ignores `r_version`.** ```sql SELECT error_occurred FROM single_builds WHERE name = $1 AND tag = $2 AND platform = $3 AND arch = $4 ``` A failure under the primary minor drops the package from every other minor's candidate list. This is the same omission fixed in `local/build-all.R` (#187) and in bincraft's `check_package_error()` (rpkgs/bincraft#119). This is the third and last consumer of that table — I have grepped the rest; `bincraft::R/cran-internal.R` also reads it, but to list packages present rather than to skip, where the R minor does not apply. ## Expected effect The per-minor passes get real candidate lists. Expect slots that reported "0 remaining" to report thousands, and correspondingly long runs. There is a cost: the list is no longer pruned by per-minor presence, so each pass asks `build_binary_package()` about packages that may already exist, and it answers `already exists in S3 ... Skipping build` per package. Slower per pass, and correct — the pruning it replaces was removing the wrong things. 
Two omissions in the snapshot, both the same shape as the ones already
fixed downstream.

s3_dir_ls(recurse = TRUE) walks the per-minor slots, and basename()
throws the directory away, so 4.5/curl_1.0.tar.gz and curl_1.0.tar.gz
collapse to one name. A package present under any R minor then counts as
built for all of them, and the candidate list prunes exactly the packages
a per-minor pass exists to build. arm64/alpine324 reported '0 remaining'
for both 4.4 and 4.6 while the same run's indexes dropped 2407 and 2436
packages as missing for those minors.

Per-minor objects are excluded here and presence is decided downstream,
where the running R version is known: build-all.R filters on it and
build_binary_package() checks the per-minor path per package. Archive/ is
kept, or every superseded version would look unbuilt.

The error query also matched on name, tag, platform and arch only, so a
failure under the primary minor dropped the package from every other
minor's candidate list. Same omission as local/build-all.R and bincraft's
check_package_error(); this is the third and last consumer of that
table.
pat-s merged commit 94e6c697cf into main 2026-08-31 21:36:12 +00:00
pat-s deleted branch fix/candidate-list-per-minor 2026-08-31 21:36:12 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries!189
No description provided.