devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 12 Pull requests Releases Activity
build-cran-binaries/.crow/weekly-rebuild-missing.yaml
pat-s e77686affc
Some checks failed
ci/crow/cron/process-updates/5 Pipeline is running
Details
ci/crow/cron/process-updates/3 Pipeline was successful
Details
ci/crow/cron/process-updates/7 Pipeline failed
Details
ci/crow/cron/process-updates/8 Pipeline is running
Details
ci/crow/cron/process-updates/12 Pipeline is running
Details
ci/crow/cron/process-updates/13 Pipeline is running
Details
ci/crow/cron/process-updates/1 Pipeline was successful
Details
ci/crow/cron/process-updates/2 Pipeline was successful
Details
ci/crow/cron/process-updates/14 Pipeline is running
Details
ci/crow/cron/process-updates/9 Pipeline is running
Details
ci/crow/cron/process-updates/6 Pipeline was successful
Details
ci/crow/cron/process-updates/16 Pipeline is running
Details
ci/crow/cron/process-updates/10 Pipeline is pending
Details
ci/crow/cron/process-updates/15 Pipeline is running
Details
ci/crow/cron/process-updates/4 Pipeline is pending
Details
ci/crow/cron/process-updates/11 Pipeline was successful
Details
fix(ci): install the R packages pipelines use instead of relying on the image (#238) 
Every `process-cran-updates-redhat-9-arm64` run since bincraft v5.5.1 fails with `there is no package called 'progressr'` (e.g. pipeline 14179).

## Cause

bincraft v5.5.1 dropped `lubridate`, `dplyr`, `purrr`, `data.table`, `gert`, `gh` and `cranlike` from its imports.
The pipelines call some of these directly, and so far got them from the build-env image's site library or the shared `R_LIBS_USER` cache.
The arm64 `redhat:9` image has no `progressr` (and no `bincraft`) under R 4.4.3, and no `RPostgres`, `dplyr` or `purrr` under R 4.5.3.
The amd64 image has all of them, which is why only this slot broke.

## Changes

- `process-updates.yaml`: install `progressr` and `lubridate`, which the inline R calls use (`progressr::handlers`, `$INTERVAL`).
- `build-all-versions*.yaml`: install `purrr` for `packages-to-build.R`.
- `weekly-audit-missing.yaml`: install `data.table` for the audit script.
- `local/packages-to-build.R`: drop the unused `dplyr` and `progressr` `library()` calls.
- `reindex`, `weekly-rebuild-reindex`, `weekly-rebuild-missing`, `backfill-*`, `rebuild-abi-gap`, `repair-unsafe-flat-objects`, `prune-stale-flat-binaries`: install `RPostgres`, because bincraft quietly skips the metadata DB without it and the per-minor index build aborts.
- `prune-stale-flat-binaries.yaml`: reinstall bincraft in the re-index step, since its `/tmp/R-pkgs` library doesn't survive from the prune step.

## Verification

- `crow lint .crow/` passes; `prek` passes on the changed files.
- The image contents were checked by exporting the arm64 and amd64 `build-env-redhat:9` images.
- Not yet run in CI.

Reviewed-on: #238
2026-10-20 08:51:06 +00:00

403 lines
10 KiB
YAML
Raw Blame History

# Consolidated weekly-rebuild-missing pipeline (all platforms, both arches).
# Three matrix rows per OS/arch, one per shard of that slot's rebuild list.
# Routing is preserved 1:1:
# - cron: each existing `weekly-rebuild-missing-<os>-<arch>` cron fires only
# its matching matrix rows (via the per-row `cron:` name filter),
# which is now all three shards of that slot.
# - manual: pick a target from the `weekly_rebuild_missing` dropdown;
# "all" fans out every os/arch and shard as parallel matrix
# workflows, while a single <os>-<arch> runs its three shards.
# Arch placement is handled by the group label (rpkgs-amd64, rpkgs-arm64).
#
# The shard picks up its own slice and re-derives what is still outstanding
# from the bucket, so a restart resumes rather than replaying; see
# local/rebuild-missing.R.
#
# Re-indexing and the CDN purge deliberately do NOT live here. Three shards
# writing one slot's PACKAGES concurrently would race, so they moved to
# .crow/weekly-rebuild-reindex.yaml, which depends on this pipeline.
variables:
# Gates this pipeline. A manual pipeline creation instantiates every file in
# .crow/, and a declared default is applied even when the run never passed
# this variable, so the default must be a value that matches no matrix row.
weekly_rebuild_missing:
description: "Manual run target: a specific <os>-<arch>, 'all' to run every os/arch in parallel, or 'none' to run nothing."
options:
- none
- all
- alpine-323-amd64
- alpine-323-arm64
- alpine-324-amd64
- alpine-324-arm64
- redhat-8-amd64
- redhat-8-arm64
- redhat-9-amd64
- redhat-9-arm64
- redhat-10-amd64
- redhat-10-arm64
- ubuntu-2204-amd64
- ubuntu-2204-arm64
- ubuntu-2404-amd64
- ubuntu-2404-arm64
- ubuntu-2604-amd64
- ubuntu-2604-arm64
default: none
when:
- event: cron
cron: weekly-rebuild-missing-${OS}-${ARCH}
- event: manual
evaluate: 'weekly_rebuild_missing == "all" || weekly_rebuild_missing == "${OS}-${ARCH}"'
skip_clone: true
labels:
group: rpkgs-${ARCH}
matrix:
include:
- OS: alpine-323
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-323
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-323
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-323
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-323
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-323
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-324
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-324
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-324
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-324
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-324
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-324
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-8
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-8
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-8
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-8
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-8
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-8
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-9
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-9
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-9
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-9
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-9
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-9
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-10
ARCH: amd64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-10
ARCH: amd64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-10
ARCH: amd64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-10
ARCH: arm64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-10
ARCH: arm64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-10
ARCH: arm64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2204
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2204
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2204
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2204
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2204
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2204
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2404
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2404
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2404
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2404
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2404
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2404
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2604
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2604
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2604
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2604
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2604
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2604
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 3
steps:
- name: 'Rebuild missing binaries'
image: reg.devxy.io/rpkgs/build-env-${IMG}
pull: true
environment:
OTEL_R_TRACES_EXPORTER: none
OTEL_R_LOGS_EXPORTER: none
OTEL_R_METRICS_EXPORTER: none
RED_HAT_DEV_PW:
from_secret: RED_HAT_DEV_PW
B2_S3_ACCESS_KEY:
from_secret: B2_S3_ACCESS_KEY
B2_S3_SECRET_KEY:
from_secret: B2_S3_SECRET_KEY
PGPASS:
from_secret: PGPASS
REPO_RO_TOKEN:
from_secret: REPO_RO_TOKEN
GITHUB_PAT:
from_secret: GITHUB_PAT
FORGEJO_TOKEN:
from_secret: FORGEJO_TOKEN
GIT_USER: pat-s
UVR_CACHE_DIR: /mnt/cache/uvr/cache
UVR_PACKAGES_DIR: /mnt/cache/uvr/packages
R_LIBS_USER: /mnt/cache/R-pkgs
R_VERSION: ${R_VERSION}
CCACHE_DIR: /mnt/cache/ccache
PLATFORM: ${OS}
ARCH: ${ARCH}
NCPUS: 2
SPLIT_INTO: ${SPLIT_INTO}
SPLIT_INDEX: ${SPLIT_INDEX}
# Wall clock after which the shard stops cleanly instead of having to be
# killed. A kill matches neither `success` nor `failure`, so it would skip
# the dependent re-index and leave rebuilt binaries behind a stale edge.
REBUILD_BUDGET_HOURS: 20
commands:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- mkdir -p /mnt/cache/uvr/cache /mnt/cache/uvr/packages /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
- /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R
- /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")'
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS="-a"; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
- UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh RPostgres httr2
- /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")'
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX $REBUILD_BUDGET_HOURS 2>&1
backend_options:
docker:
resources:
requests:
memory: 5Gi
cpu: 3000m
limits:
memory: 18Gi
cpu: 3000m
Reference in a new issue View git blame Copy permalink