devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 12 Pull requests Releases Activity

feat(ci): add a targeted per-R-minor rebuild workflow #198

Merged
pat-s merged 1 commit from feat/rebuild-abi-gap into main 2026-09-04 10:24:48 +00:00
Conversation 0 Commits 1 Files changed 1 +218
pat-s commented 2026-09-04 10:23:17 +00:00
Owner

Why

A user hit this on amd64/resolute under R 4.6.1:

base64enc.so: undefined symbol: SETLENGTH

R 4.6 no longer exports SETLENGTH:

R 4.4: missing -> R_ClosureEnv R_ClosureFormals R_MakeMissingBinding R_PromiseEnv R_PromiseValue R_mkClosure
R 4.5: missing -> R_MakeMissingBinding R_PromiseEnv R_PromiseValue
R 4.6: missing -> R_PromiseEnv R_PromiseValue SETLENGTH

base64enc calls SETLENGTH (base64.c:206, 253), its flat binary was built under R 4.4.3, and it has no per-minor copy at all, so the union index hands that binary to a 4.6 client.

The safety net missed it because the risky set is defined circularly:

risky_packages_across_minors <- function(...) {
  # a package is "risky" iff it already has a tarball in some per-minor slot

A package counts as ABI-risky only if it has already been built per-minor. base64enc never was, and having no new CRAN version it never enters the build list to be classified. So the per-minor protection only covers packages that passed through the pipeline after the scheme existed.

Scope

Scanning all 4,585 compiled binaries in amd64/resolute's flat index for symbols R 4.6 does not export found 10 referencing SETLENGTH. Most already have per-minor builds and are served correctly. Two do not:

base64enc_0.1-6   flat=200  4.4=404  4.5=404  4.6=404
lobstr_1.2.1      flat=200  4.4=404  4.5=200  4.6=404

(iotools lacks a 4.5 copy but 4.5 still exports SETLENGTH, so it does not break.)

What this adds

Rebuilding one package into one minor was only possible with just rebuild from a workstation holding the S3 and Postgres credentials. This adds the CI path:

crow pipeline create devxy/build-cran-binaries \
  --var rebuild_abi_gap=ubuntu-2604-amd64 \
  --var ABI_PACKAGES="base64enc@0.1-6 lobstr@1.2.1" \
  --var ABI_R_MINOR=4.6
  • reuses reindex.yaml's slot matrix verbatim, so the slot-to-image mapping cannot drift
  • runs build-one.R --sensitive-only, which reclassifies each version and skips anything not ABI-risky rather than writing it to the wrong slot
  • mounts no shared cache volume, so it cannot disturb the build pipeline's per-agent caches
  • attempts every package, then exits non-zero if any failed

crow lint passes.

Not in this PR

This is the remediation path, not the fix. The circular risky_packages_across_minors() definition is still in place, so new instances of this class can still appear. That is the follow-up.

## Why A user hit this on amd64/resolute under R 4.6.1: ``` base64enc.so: undefined symbol: SETLENGTH ``` R 4.6 no longer exports `SETLENGTH`: ``` R 4.4: missing -> R_ClosureEnv R_ClosureFormals R_MakeMissingBinding R_PromiseEnv R_PromiseValue R_mkClosure R 4.5: missing -> R_MakeMissingBinding R_PromiseEnv R_PromiseValue R 4.6: missing -> R_PromiseEnv R_PromiseValue SETLENGTH ``` `base64enc` calls `SETLENGTH` (`base64.c:206`, `253`), its flat binary was built under R 4.4.3, and it has no per-minor copy at all, so the union index hands that binary to a 4.6 client. The safety net missed it because the risky set is defined circularly: ```r risky_packages_across_minors <- function(...) { # a package is "risky" iff it already has a tarball in some per-minor slot ``` A package counts as ABI-risky only if it has *already* been built per-minor. `base64enc` never was, and having no new CRAN version it never enters the build list to be classified. So the per-minor protection only covers packages that passed through the pipeline after the scheme existed. ## Scope Scanning all 4,585 compiled binaries in amd64/resolute's flat index for symbols R 4.6 does not export found 10 referencing `SETLENGTH`. Most already have per-minor builds and are served correctly. Two do not: ``` base64enc_0.1-6 flat=200 4.4=404 4.5=404 4.6=404 lobstr_1.2.1 flat=200 4.4=404 4.5=200 4.6=404 ``` (`iotools` lacks a 4.5 copy but 4.5 still exports `SETLENGTH`, so it does not break.) ## What this adds Rebuilding one package into one minor was only possible with `just rebuild` from a workstation holding the S3 and Postgres credentials. This adds the CI path: ```sh crow pipeline create devxy/build-cran-binaries \ --var rebuild_abi_gap=ubuntu-2604-amd64 \ --var ABI_PACKAGES="base64enc@0.1-6 lobstr@1.2.1" \ --var ABI_R_MINOR=4.6 ``` - reuses `reindex.yaml`'s slot matrix verbatim, so the slot-to-image mapping cannot drift - runs `build-one.R --sensitive-only`, which reclassifies each version and skips anything not ABI-risky rather than writing it to the wrong slot - mounts no shared cache volume, so it cannot disturb the build pipeline's per-agent caches - attempts every package, then exits non-zero if any failed `crow lint` passes. ## Not in this PR This is the remediation path, not the fix. The circular `risky_packages_across_minors()` definition is still in place, so new instances of this class can still appear. That is the follow-up. 
The per-minor scheme only ever covered packages that passed through the build
pipeline after it existed. A package built into the flat slot before that, and
not updated on CRAN since, is never reclassified and never gets a per-minor
copy, so the union index serves its flat binary to every minor.

When that binary references a symbol the client's R no longer exports, the
install fails at dyn.load():

  base64enc.so: undefined symbol: SETLENGTH

base64enc calls SETLENGTH, its flat binary was built under R 4.4.3, and R 4.6
does not export SETLENGTH.

Rebuilding one package into one minor was only possible via `just rebuild` from
a workstation holding the S3 and Postgres credentials. This adds a CI path.

This change will:

- take a package@version list and a target R minor, over the same slot matrix
  as reindex.yaml so the slot-to-image mapping cannot drift
- run build-one.R --sensitive-only, which reclassifies each version and skips
  anything that is not ABI-risky rather than writing it to the wrong slot
- mount no shared cache volume, so it cannot disturb the build pipeline's
  per-agent caches
- report a non-zero exit if any package fails, after attempting them all
pat-s merged commit 26406a3e86 into main 2026-09-04 10:24:48 +00:00
pat-s deleted branch feat/rebuild-abi-gap 2026-09-04 10:24:48 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries!198
No description provided.