pat-s
c216e44578
All checks were successful
ci/crow/manual/backfill-flat-safety/17 Pipeline was successful
## Why bincraft #128 lets `union_index_records()` keep a carried flat record when that specific object is recorded safe. On its own it changes nothing for the records already being dropped, because they are legacy objects with no verdict: ``` 4.4: 16398 dropped across 9 indexes (~1822 each) 4.5: 20076 dropped across 9 indexes (~2230 each) 4.6: 15532 dropped across 9 indexes (~1726 each) ``` Every one of those resolves to CRAN source for clients the routing serves correctly. This backfill inspects the objects already in the bucket and writes their verdicts, which is what actually restores the coverage. ## How the verdict is made By symbols, not heuristics. A binary fails at `dyn.load()` under minor M exactly when it references a symbol M does not export, and the volatile set is derived by diffing the `libR.so` exports of the R builds installed in the image — so a new R release updates it with no curated list to maintain. That matters here because the source-level classifier calls **93%** of compiled packages risky (`LinkingTo: Rcpp` alone is enough), while a symbol scan of the same slot found **17** packages that break under 4.6 and **72** under 4.4. Recording the real answer per object is what makes the difference between ~2000 dropped records and the few dozen that genuinely need dropping. ## Safety - **Refuses to run where fewer than two R minors are installed.** With nothing to compare, every verdict would read "safe" — writing a bucket full of false clearances is far worse than writing none. - Skips packages with no compiled code: they cannot fail this way and are never dropped, so downloading them to prove it is wasted. - Resumable; already-recorded objects cost one query. `crow lint` passes. ## Sequencing Merge and release bincraft #128 first, then run this per slot, then re-index. Verifying the drop counts fall is the point of the exercise, so that re-index is where the fix is confirmed rather than assumed. Reviewed-on: #202
148 lines
4 KiB
R
148 lines
4 KiB
R
### Record, per flat-slot object, whether it loads under every supported R minor.
|
|
#
|
|
# Usage: Rscript local/backfill-flat-safety.R <arch> <codename> [limit]
|
|
#
|
|
# `union_index_records()` keeps a carried flat record when that object is
|
|
# recorded safe, and drops it otherwise. Objects uploaded before that record
|
|
# existed have no verdict, so they are dropped: a full re-index removed ~2000
|
|
# records per per-minor index, all resolving to CRAN source for clients the
|
|
# routing serves correctly.
|
|
#
|
|
# This inspects the objects already in the bucket and writes the verdict, which
|
|
# is what restores that coverage. The question is decided by symbols, not
|
|
# heuristics: a binary fails at dyn.load() under minor M exactly when it
|
|
# references a symbol M does not export, and the volatile set is derived by
|
|
# diffing the R builds installed in this image.
|
|
#
|
|
# Only compiled packages are inspected. A package with no shared object cannot
|
|
# fail this way and is never dropped, so proving it would be wasted downloads.
|
|
#
|
|
# Resumable: objects already recorded are skipped, so a re-run costs one query.
|
|
|
|
options(error = function() {
|
|
cat("ERROR:", geterrmessage(), "\n", file = stdout())
|
|
traceback(2)
|
|
q(status = 1)
|
|
})
|
|
|
|
library(bincraft, quietly = TRUE)
|
|
|
|
args <- commandArgs(trailingOnly = TRUE)
|
|
if (length(args) < 2L) {
|
|
stop("usage: backfill-flat-safety.R <arch> <codename> [limit]", call. = FALSE)
|
|
}
|
|
arch <- args[[1L]]
|
|
codename <- args[[2L]]
|
|
limit <- if (length(args) >= 3L) as.integer(args[[3L]]) else NA_integer_
|
|
|
|
sets <- bincraft:::installed_r_symbol_sets()
|
|
if (is.null(sets)) {
|
|
stop(
|
|
"Fewer than two R minors are installed here, so nothing can be compared. Refusing to record verdicts that would all read 'safe'.", # nolint
|
|
call. = FALSE
|
|
)
|
|
}
|
|
cat(sprintf(
|
|
"Comparing R minors: %s (%d symbols vary across them)\n",
|
|
paste(names(sets), collapse = ", "),
|
|
length(bincraft:::cross_minor_volatile_symbols(sets))
|
|
))
|
|
|
|
read_index <- function(index_url) {
|
|
con <- gzcon(url(index_url, open = "rb"))
|
|
on.exit(close(con), add = TRUE)
|
|
read.dcf(con, fields = c("Package", "Version", "Built", "NeedsCompilation"))
|
|
}
|
|
|
|
flat <- read_index(sprintf(
|
|
"https://cran.rpkgs.com/%s/%s/latest/src/contrib/PACKAGES.gz",
|
|
arch,
|
|
codename
|
|
))
|
|
# Source fallbacks carry no binary, so there is nothing to inspect.
|
|
flat <- flat[!is.na(flat[, "Built"]), , drop = FALSE]
|
|
compiled <- flat[
|
|
!is.na(flat[, "NeedsCompilation"]) & flat[, "NeedsCompilation"] == "yes",
|
|
,
|
|
drop = FALSE
|
|
]
|
|
cat(sprintf(
|
|
"%s/%s: %d flat binaries, %d compiled and worth inspecting\n",
|
|
arch,
|
|
codename,
|
|
nrow(flat),
|
|
nrow(compiled)
|
|
))
|
|
if (!is.na(limit) && limit > 0L && limit < nrow(compiled)) {
|
|
compiled <- compiled[seq_len(limit), , drop = FALSE]
|
|
cat(sprintf("Limited to the first %d\n", limit))
|
|
}
|
|
|
|
base <- sprintf(
|
|
"https://cran.rpkgs.com/%s/%s/latest/src/contrib",
|
|
arch,
|
|
codename
|
|
)
|
|
safe <- 0L
|
|
unsafe <- 0L
|
|
skipped <- 0L
|
|
for (i in seq_len(nrow(compiled))) {
|
|
pkg <- compiled[i, "Package"]
|
|
ver <- compiled[i, "Version"]
|
|
tarball <- file.path(tempdir(), sprintf("%s_%s.tar.gz", pkg, ver))
|
|
ok <- tryCatch(
|
|
{
|
|
utils::download.file(
|
|
sprintf("%s/%s_%s.tar.gz", base, pkg, ver),
|
|
tarball,
|
|
quiet = TRUE,
|
|
mode = "wb"
|
|
)
|
|
TRUE
|
|
},
|
|
error = function(e) FALSE,
|
|
warning = function(w) FALSE
|
|
)
|
|
if (!ok) {
|
|
skipped <- skipped + 1L
|
|
next
|
|
}
|
|
verdict <- bincraft:::tarball_abi_verdict(tarball, sets)
|
|
unlink(tarball)
|
|
if (!isTRUE(verdict$inspected)) {
|
|
skipped <- skipped + 1L
|
|
next
|
|
}
|
|
bincraft:::flat_safety_store(
|
|
pkg,
|
|
ver,
|
|
codename,
|
|
arch,
|
|
safe = !isTRUE(verdict$sensitive),
|
|
unsupported = verdict$unsupported
|
|
)
|
|
if (isTRUE(verdict$sensitive)) {
|
|
unsafe <- unsafe + 1L
|
|
} else {
|
|
safe <- safe + 1L
|
|
}
|
|
if (i %% 250L == 0L) {
|
|
cat(sprintf(
|
|
" inspected %d/%d (%d safe, %d unsafe, %d skipped)\n",
|
|
i,
|
|
nrow(compiled),
|
|
safe,
|
|
unsafe,
|
|
skipped
|
|
))
|
|
utils::flush.console()
|
|
}
|
|
}
|
|
|
|
cat(sprintf(
|
|
"Done: %d inspected, %d safe for every minor, %d unsafe, %d skipped\n",
|
|
nrow(compiled),
|
|
safe,
|
|
unsafe,
|
|
skipped
|
|
))
|