pat-s
e77686affc
Some checks failed
ci/crow/cron/process-updates/5 Pipeline is running
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline failed
ci/crow/cron/process-updates/8 Pipeline is running
ci/crow/cron/process-updates/12 Pipeline is running
ci/crow/cron/process-updates/13 Pipeline is running
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline is running
ci/crow/cron/process-updates/9 Pipeline is running
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline is running
ci/crow/cron/process-updates/10 Pipeline is pending
ci/crow/cron/process-updates/15 Pipeline is running
ci/crow/cron/process-updates/4 Pipeline is pending
ci/crow/cron/process-updates/11 Pipeline was successful
Every `process-cran-updates-redhat-9-arm64` run since bincraft v5.5.1 fails with `there is no package called 'progressr'` (e.g. pipeline 14179). ## Cause bincraft v5.5.1 dropped `lubridate`, `dplyr`, `purrr`, `data.table`, `gert`, `gh` and `cranlike` from its imports. The pipelines call some of these directly, and so far got them from the build-env image's site library or the shared `R_LIBS_USER` cache. The arm64 `redhat:9` image has no `progressr` (and no `bincraft`) under R 4.4.3, and no `RPostgres`, `dplyr` or `purrr` under R 4.5.3. The amd64 image has all of them, which is why only this slot broke. ## Changes - `process-updates.yaml`: install `progressr` and `lubridate`, which the inline R calls use (`progressr::handlers`, `$INTERVAL`). - `build-all-versions*.yaml`: install `purrr` for `packages-to-build.R`. - `weekly-audit-missing.yaml`: install `data.table` for the audit script. - `local/packages-to-build.R`: drop the unused `dplyr` and `progressr` `library()` calls. - `reindex`, `weekly-rebuild-reindex`, `weekly-rebuild-missing`, `backfill-*`, `rebuild-abi-gap`, `repair-unsafe-flat-objects`, `prune-stale-flat-binaries`: install `RPostgres`, because bincraft quietly skips the metadata DB without it and the per-minor index build aborts. - `prune-stale-flat-binaries.yaml`: reinstall bincraft in the re-index step, since its `/tmp/R-pkgs` library doesn't survive from the prune step. ## Verification - `crow lint .crow/` passes; `prek` passes on the changed files. - The image contents were checked by exporting the arm64 and amd64 `build-env-redhat:9` images. - Not yet run in CI. Reviewed-on: #238
91 lines
3.1 KiB
YAML
91 lines
3.1 KiB
YAML
# Populate the ABI classifier cache for packages that predate the per-minor scheme.
|
|
#
|
|
# bincraft decides whether a flat binary may be carried into a per-minor index
|
|
# from its risky-package set. That set is now the union of the per-minor tarball
|
|
# scan and the `abi_classification` cache, but the cache only holds verdicts for
|
|
# packages that passed through the build pipeline after the scheme existed.
|
|
# Everything older has no verdict, and no verdict means "carry it" - which is how
|
|
# a base64enc built under R 4.4.3 reached R 4.6 clients and died at dyn.load()
|
|
# on SETLENGTH.
|
|
#
|
|
# The verdict is computed from the package source, so it does not depend on OS,
|
|
# arch or R minor: one run populates the cache for every slot. The arch/codename
|
|
# below only choose which flat index to enumerate.
|
|
#
|
|
# Re-running is cheap. Cached verdicts are returned without touching the network,
|
|
# so a second run costs one query per package and downloads only what is still
|
|
# unclassified.
|
|
#
|
|
# crow pipeline create devxy/build-cran-binaries --var backfill_abi=run
|
|
|
|
variables:
|
|
# A manual pipeline creation instantiates every file in .crow/, so the default
|
|
# must be a value the `when` clause rejects.
|
|
backfill_abi:
|
|
description: "Set to 'run' to classify the flat index into the ABI cache."
|
|
options:
|
|
- none
|
|
- run
|
|
default: none
|
|
BACKFILL_ARCH:
|
|
description: 'Arch of the flat index to enumerate.'
|
|
options:
|
|
- amd64
|
|
- arm64
|
|
default: amd64
|
|
BACKFILL_CODENAME:
|
|
description: 'Codename of the flat index to enumerate.'
|
|
default: resolute
|
|
BACKFILL_LIMIT:
|
|
description: 'Classify at most this many packages (0 = no limit).'
|
|
default: '0'
|
|
|
|
when:
|
|
- event: manual
|
|
evaluate: 'backfill_abi == "run"'
|
|
|
|
skip_clone: true
|
|
|
|
labels:
|
|
group: rpkgs-amd64
|
|
|
|
steps:
|
|
- name: 'Classify the flat index into the ABI cache'
|
|
image: reg.devxy.io/rpkgs/build-env-ubuntu:resolute
|
|
pull: true
|
|
environment:
|
|
OTEL_R_TRACES_EXPORTER: none
|
|
OTEL_R_LOGS_EXPORTER: none
|
|
OTEL_R_METRICS_EXPORTER: none
|
|
PGPASS:
|
|
from_secret: PGPASS
|
|
REPO_RO_TOKEN:
|
|
from_secret: REPO_RO_TOKEN
|
|
GITHUB_PAT:
|
|
from_secret: GITHUB_PAT
|
|
GIT_USER: pat-s
|
|
# Pipeline variables are substituted into the YAML, not exported, so they
|
|
# have to be bound here to be readable as env vars in the script below.
|
|
BACKFILL_ARCH: ${BACKFILL_ARCH}
|
|
BACKFILL_CODENAME: ${BACKFILL_CODENAME}
|
|
BACKFILL_LIMIT: ${BACKFILL_LIMIT}
|
|
R_LIBS_USER: /tmp/R-pkgs
|
|
commands:
|
|
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
|
- mkdir -p /tmp/R-pkgs
|
|
- /opt/R/4.5.3/bin/Rscript local/install-bincraft.R
|
|
- UVR_R_BIN=/opt/R/4.5.3/bin/R local/uvr-install.sh RPostgres
|
|
- |
|
|
set -eu
|
|
LIMIT=""
|
|
if [ "$$BACKFILL_LIMIT" != "0" ]; then
|
|
LIMIT="$$BACKFILL_LIMIT"
|
|
fi
|
|
/opt/R/4.5.3/bin/Rscript local/backfill-abi-classification.R \
|
|
"$$BACKFILL_ARCH" "$$BACKFILL_CODENAME" $$LIMIT
|
|
backend_options:
|
|
docker:
|
|
resources:
|
|
requests:
|
|
memory: 4Gi
|
|
cpu: 1000m
|