feat(abi): backfill classifier verdicts for pre-per-minor packages
bincraft now unions the ABI classifier cache into its risky-package set, but the cache only holds verdicts for packages that passed through the build pipeline after the per-minor scheme existed. Everything older has no verdict, and no verdict means the flat binary is carried into every per-minor index. That is the population the fix has to reach: a base64enc built under R 4.4.3 was served to R 4.6 clients, which have no SETLENGTH to bind. The verdict is computed from the package source, so it is independent of OS, arch and R minor: one run populates the cache for every slot. Only NeedsCompilation=yes packages are classified. A package with no compiled code has no symbols to bind and cannot fail this way, which cuts the work from 23218 packages to 4585 on amd64/resolute. This change will: - add local/backfill-abi-classification.R, enumerating a slot's flat index and classifying each compiled package through bincraft's cached classifier - add a manual .crow workflow to run it - stay resumable, since cached verdicts return without touching the network, so a re-run only downloads what is still unclassified
This commit is contained in:
parent
26406a3e86
commit
1d450c7e5d
1 changed files with 219 additions and 0 deletions
90
.crow/backfill-abi-classification.yaml
Normal file
90
.crow/backfill-abi-classification.yaml
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
# Populate the ABI classifier cache for packages that predate the per-minor scheme.
|
||||
#
|
||||
# bincraft decides whether a flat binary may be carried into a per-minor index
|
||||
# from its risky-package set. That set is now the union of the per-minor tarball
|
||||
# scan and the `abi_classification` cache, but the cache only holds verdicts for
|
||||
# packages that passed through the build pipeline after the scheme existed.
|
||||
# Everything older has no verdict, and no verdict means "carry it" - which is how
|
||||
# a base64enc built under R 4.4.3 reached R 4.6 clients and died at dyn.load()
|
||||
# on SETLENGTH.
|
||||
#
|
||||
# The verdict is computed from the package source, so it does not depend on OS,
|
||||
# arch or R minor: one run populates the cache for every slot. The arch/codename
|
||||
# below only choose which flat index to enumerate.
|
||||
#
|
||||
# Re-running is cheap. Cached verdicts are returned without touching the network,
|
||||
# so a second run costs one query per package and downloads only what is still
|
||||
# unclassified.
|
||||
#
|
||||
# crow pipeline create devxy/build-cran-binaries --var backfill_abi=run
|
||||
|
||||
variables:
|
||||
# A manual pipeline creation instantiates every file in .crow/, so the default
|
||||
# must be a value the `when` clause rejects.
|
||||
backfill_abi:
|
||||
description: "Set to 'run' to classify the flat index into the ABI cache."
|
||||
options:
|
||||
- none
|
||||
- run
|
||||
default: none
|
||||
BACKFILL_ARCH:
|
||||
description: 'Arch of the flat index to enumerate.'
|
||||
options:
|
||||
- amd64
|
||||
- arm64
|
||||
default: amd64
|
||||
BACKFILL_CODENAME:
|
||||
description: 'Codename of the flat index to enumerate.'
|
||||
default: resolute
|
||||
BACKFILL_LIMIT:
|
||||
description: 'Classify at most this many packages (0 = no limit).'
|
||||
default: '0'
|
||||
|
||||
when:
|
||||
- event: manual
|
||||
evaluate: 'backfill_abi == "run"'
|
||||
|
||||
skip_clone: true
|
||||
|
||||
labels:
|
||||
group: rpkgs-amd64
|
||||
|
||||
steps:
|
||||
- name: 'Classify the flat index into the ABI cache'
|
||||
image: reg.devxy.io/rpkgs/build-env-ubuntu:resolute
|
||||
pull: true
|
||||
environment:
|
||||
OTEL_R_TRACES_EXPORTER: none
|
||||
OTEL_R_LOGS_EXPORTER: none
|
||||
OTEL_R_METRICS_EXPORTER: none
|
||||
PGPASS:
|
||||
from_secret: PGPASS
|
||||
REPO_RO_TOKEN:
|
||||
from_secret: REPO_RO_TOKEN
|
||||
GITHUB_PAT:
|
||||
from_secret: GITHUB_PAT
|
||||
GIT_USER: pat-s
|
||||
# Pipeline variables are substituted into the YAML, not exported, so they
|
||||
# have to be bound here to be readable as env vars in the script below.
|
||||
BACKFILL_ARCH: ${BACKFILL_ARCH}
|
||||
BACKFILL_CODENAME: ${BACKFILL_CODENAME}
|
||||
BACKFILL_LIMIT: ${BACKFILL_LIMIT}
|
||||
R_LIBS_USER: /tmp/R-pkgs
|
||||
commands:
|
||||
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
||||
- mkdir -p /tmp/R-pkgs
|
||||
- /opt/R/4.5.3/bin/Rscript local/install-bincraft.R
|
||||
- |
|
||||
set -eu
|
||||
LIMIT=""
|
||||
if [ "$$BACKFILL_LIMIT" != "0" ]; then
|
||||
LIMIT="$$BACKFILL_LIMIT"
|
||||
fi
|
||||
/opt/R/4.5.3/bin/Rscript local/backfill-abi-classification.R \
|
||||
"$$BACKFILL_ARCH" "$$BACKFILL_CODENAME" $$LIMIT
|
||||
backend_options:
|
||||
docker:
|
||||
resources:
|
||||
requests:
|
||||
memory: 4Gi
|
||||
cpu: 1000m
|
||||
Loading…
Reference in a new issue