devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 12 Pull requests Releases Activity

fix(abi): prune flat-slot binaries of ABI-risky packages #200

Merged
pat-s merged 1 commit from fix/prune-stale-flat-binaries into main 2026-09-04 10:54:10 +00:00
Conversation 0 Commits 1 Files changed 2 +381
pat-s commented 2026-09-04 10:51:49 +00:00
Owner

Why

base64enc still fails to install under R 4.6 even after it was built into the 4.6 slot:

base64enc.so: undefined symbol: SETLENGTH

The per-minor binary is correct and serves correctly to any client the router can identify. But the flat slot still holds a binary built under R 4.4.3, and that is what every client receives when it cannot be routed by R minor.

That object should not exist. upload_single_binary() writes an ABI-risky package only to the per-minor path:

if (is_r_minor_sensitive) {
  remote_tarball_path <- file.path(remote_bin_path, minor_version, tarball_name)
} else {
  remote_tarball_path <- file.path(remote_bin_path, tarball_name)
}

base64enc's flat binary is a legacy artifact from before it was classified risky. Removing it makes the outcome correct regardless of routing, user agent, downloader or renv configuration, which is the part I could not otherwise guarantee.

What it does

local/prune-stale-flat-binaries.R lists a slot's flat objects, intersects them with the risky set, and deletes them. Deleting is safe here: the source fallback republishes the CRAN source under the same key, so the package stays installable everywhere and compiles against the client's own R, while routed clients keep getting the per-minor binary.

Safety posture, given this deletes production objects:

  • dry run by default; nothing is removed without --apply
  • takes an explicit package list, so the first runs need not trust the classifier cache at all
  • refuses to proceed on an empty risky set, which is indistinguishable from an unreachable cache and would otherwise report success having pruned nothing
  • keeps objects whose package CRAN no longer ships, since there would be no source to fall back to and the package would vanish
  • re-indexes and purges in the same run. This is not optional: a published index naming deleted files serves 404s, which is worse than the binary it replaced.

Use

# dry run first
crow pipeline create devxy/build-cran-binaries \
  --var prune_flat=ubuntu-2604-amd64 --var PRUNE_PACKAGES="base64enc lobstr"

# then apply
crow pipeline create devxy/build-cran-binaries \
  --var prune_flat=ubuntu-2604-amd64 --var PRUNE_PACKAGES="base64enc lobstr" \
  --var PRUNE_APPLY=yes

crow lint passes.

Sequencing

Run with an explicit list first, on one slot, and confirm the flat URL then serves source while the per-minor URLs still serve binaries. Only widen to the cache-derived set once the backfill has finished and its output looks sane: the first 250 classifications came back 232 r-minor-sensitive, and a rate that high suggests the source-grep classifier is broader than the set that actually breaks, which is worth checking before pruning thousands of flat binaries.

## Why `base64enc` still fails to install under R 4.6 even after it was built into the 4.6 slot: ``` base64enc.so: undefined symbol: SETLENGTH ``` The per-minor binary is correct and serves correctly to any client the router can identify. But the **flat slot still holds a binary built under R 4.4.3**, and that is what every client receives when it cannot be routed by R minor. That object should not exist. `upload_single_binary()` writes an ABI-risky package only to the per-minor path: ```r if (is_r_minor_sensitive) { remote_tarball_path <- file.path(remote_bin_path, minor_version, tarball_name) } else { remote_tarball_path <- file.path(remote_bin_path, tarball_name) } ``` `base64enc`'s flat binary is a legacy artifact from before it was classified risky. Removing it makes the outcome correct regardless of routing, user agent, downloader or renv configuration, which is the part I could not otherwise guarantee. ## What it does `local/prune-stale-flat-binaries.R` lists a slot's flat objects, intersects them with the risky set, and deletes them. Deleting is safe here: the source fallback republishes the CRAN source under the same key, so the package stays installable everywhere and compiles against the client's own R, while routed clients keep getting the per-minor binary. Safety posture, given this deletes production objects: - **dry run by default**; nothing is removed without `--apply` - takes an **explicit package list**, so the first runs need not trust the classifier cache at all - **refuses to proceed on an empty risky set**, which is indistinguishable from an unreachable cache and would otherwise report success having pruned nothing - **keeps objects whose package CRAN no longer ships**, since there would be no source to fall back to and the package would vanish - **re-indexes and purges in the same run**. This is not optional: a published index naming deleted files serves 404s, which is worse than the binary it replaced. ## Use ```sh # dry run first crow pipeline create devxy/build-cran-binaries \ --var prune_flat=ubuntu-2604-amd64 --var PRUNE_PACKAGES="base64enc lobstr" # then apply crow pipeline create devxy/build-cran-binaries \ --var prune_flat=ubuntu-2604-amd64 --var PRUNE_PACKAGES="base64enc lobstr" \ --var PRUNE_APPLY=yes ``` `crow lint` passes. ## Sequencing Run with an explicit list first, on one slot, and confirm the flat URL then serves source while the per-minor URLs still serve binaries. Only widen to the cache-derived set once the backfill has finished and its output looks sane: the first 250 classifications came back 232 r-minor-sensitive, and a rate that high suggests the source-grep classifier is broader than the set that actually breaks, which is worth checking before pruning thousands of flat binaries. 
upload_single_binary() writes an ABI-risky package only to the per-minor slot,
so a risky package should have no flat binary at all. Packages built before they
were classified risky still do, and the flat slot is what every client that
cannot be routed by R minor receives.

Such a binary was built under whichever R was primary at the time, so it fails
at load under any minor that has since dropped a symbol it uses. base64enc was
built under R 4.4.3, calls SETLENGTH, and R 4.6 no longer exports it:

  base64enc.so: undefined symbol: SETLENGTH

Deleting the flat object does not create a hole. The source fallback republishes
the CRAN source under the same key, so the package stays installable everywhere
and compiles against the client's own R, while routed clients keep getting the
per-minor binary.

This change will:

- add local/prune-stale-flat-binaries.R, dry run unless --apply, taking either
  an explicit package list or every package the classifier cache calls risky
- refuse to run on an empty risky set, which is indistinguishable from an
  unreachable cache and would report success having pruned nothing
- keep any object whose package CRAN no longer ships, since there would be no
  source to fall back to
- re-index and purge in the same run, because a published index naming deleted
  files serves 404s rather than the source fallback
pat-s merged commit f9ebb3dba4 into main 2026-09-04 10:54:10 +00:00
pat-s deleted branch fix/prune-stale-flat-binaries 2026-09-04 10:54:10 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries!200
No description provided.