devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 10 Pull requests Releases Activity
build-cran-binaries/.crow/weekly-rebuild-missing.yaml
pat-s c072b780b2
 feat(rebuild): shard the weekly rebuild and make each shard resumable 
Split every OS/arch row of weekly-rebuild-missing into three shards and move
the build loop out of the inline R one-liner into local/rebuild-missing.R.

A shard re-derives its outstanding set from the bucket on every start: an
object whose ETag equals CRAN's published MD5sum is still a source fallback and
needs building. That is bincraft's check_s3_root_package() evaluated in bulk,
so a restart resumes rather than replaying thousands of per-package HEADs, and
it stays correct when a sibling shard or a process-updates run finishes work in
the meantime. No progress file, no volume, no database cursor.

Give each shard a 20h wall-clock budget so it exits cleanly instead of having
to be killed. A kill matches neither success nor failure, which is how pipeline
10910 skipped its CDN purge and left ~4600 rebuilt binaries behind stale edge
copies.

Move re-indexing and the purge into weekly-rebuild-reindex.yaml, which depends
on the rebuild and runs on success or failure. Three shards writing one slot's
PACKAGES concurrently would race: update_PACKAGES lists the live bucket, so an
early lister that uploads last publishes an index missing its siblings' work.
2026-08-12 08:27:45 +00:00

441 lines
11 KiB
YAML
Raw Blame History

# Consolidated weekly-rebuild-missing pipeline (all platforms, both arches).
# Three matrix rows per OS/arch, one per shard of that slot's rebuild list.
# Routing is preserved 1:1:
# - cron: each existing `weekly-rebuild-missing-<os>-<arch>` cron fires only
# its matching matrix rows (via the per-row `cron:` name filter),
# which is now all three shards of that slot.
# - manual: `weekly_rebuild_missing` dropdown, default "all" (matches the
# previous bare manual trigger that ran every os/arch); pick a
# single <os>-<arch> to run just one.
# Arch placement is handled by the group label (rpkgs-amd64, rpkgs-arm64).
#
# The shard picks up its own slice and re-derives what is still outstanding
# from the bucket, so a restart resumes rather than replaying; see
# local/rebuild-missing.R.
#
# Re-indexing and the CDN purge deliberately do NOT live here. Three shards
# writing one slot's PACKAGES concurrently would race, so they moved to
# .crow/weekly-rebuild-reindex.yaml, which depends on this pipeline.
variables:
# Gates this pipeline. A manual pipeline creation instantiates every file in
# .crow/, and a declared default is applied even when the run never passed
# this variable, so the default must be a value that matches no matrix row.
weekly_rebuild_missing:
description: "Manual run target: a specific <os>-<arch>, 'all' for every os/arch, or 'none' to run nothing."
options:
- none
- all
- alpine-322-amd64
- alpine-322-arm64
- alpine-323-amd64
- alpine-323-arm64
- alpine-324-amd64
- alpine-324-arm64
- redhat-8-amd64
- redhat-8-arm64
- redhat-9-amd64
- redhat-9-arm64
- redhat-10-amd64
- redhat-10-arm64
- ubuntu-2204-amd64
- ubuntu-2204-arm64
- ubuntu-2404-amd64
- ubuntu-2404-arm64
- ubuntu-2604-amd64
- ubuntu-2604-arm64
default: none
when:
- event: cron
cron: weekly-rebuild-missing-${OS}-${ARCH}
- event: manual
evaluate: 'weekly_rebuild_missing == "all" || weekly_rebuild_missing == "${OS}-${ARCH}"'
skip_clone: true
labels:
group: rpkgs-${ARCH}
matrix:
include:
- OS: alpine-322
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.22
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-322
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.22
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-322
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.22
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-322
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.22
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-322
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.22
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-322
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.22
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-323
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-323
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-323
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-323
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-323
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-323
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-324
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-324
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-324
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-324
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-324
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-324
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-8
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-8
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-8
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-8
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-8
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-8
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-9
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-9
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-9
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-9
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-9
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-9
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-10
ARCH: amd64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-10
ARCH: amd64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-10
ARCH: amd64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-10
ARCH: arm64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-10
ARCH: arm64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-10
ARCH: arm64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2204
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2204
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2204
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2204
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2204
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2204
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2404
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2404
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2404
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2404
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2404
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2404
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2604
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2604
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2604
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2604
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2604
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2604
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 3
steps:
- name: 'Rebuild missing binaries'
image: reg.devxy.io/rpkgs/build-env-${IMG}
pull: true
environment:
OTEL_R_TRACES_EXPORTER: none
OTEL_R_LOGS_EXPORTER: none
OTEL_R_METRICS_EXPORTER: none
RED_HAT_DEV_PW:
from_secret: RED_HAT_DEV_PW
B2_S3_ACCESS_KEY:
from_secret: B2_S3_ACCESS_KEY
B2_S3_SECRET_KEY:
from_secret: B2_S3_SECRET_KEY
PGPASS:
from_secret: PGPASS
REPO_RO_TOKEN:
from_secret: REPO_RO_TOKEN
GITHUB_PAT:
from_secret: GITHUB_PAT
FORGEJO_TOKEN:
from_secret: FORGEJO_TOKEN
GIT_USER: pat-s
UVR_CACHE_DIR: /mnt/cache/uvr/cache
UVR_PACKAGES_DIR: /mnt/cache/uvr/packages
R_LIBS_USER: /mnt/cache/R-pkgs
R_VERSION: ${R_VERSION}
CCACHE_DIR: /mnt/cache/ccache
PLATFORM: ${OS}
ARCH: ${ARCH}
NCPUS: 2
SPLIT_INTO: ${SPLIT_INTO}
SPLIT_INDEX: ${SPLIT_INDEX}
# Wall clock after which the shard stops cleanly instead of having to be
# killed. A kill matches neither `success` nor `failure`, so it would skip
# the dependent re-index and leave rebuilt binaries behind a stale edge.
REBUILD_BUDGET_HOURS: 20
commands:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- mkdir -p /mnt/cache/uvr/cache /mnt/cache/uvr/packages /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
- /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R
- /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")'
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
- UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh httr2
- /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")'
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX $REBUILD_BUDGET_HOURS 2>&1
backend_options:
docker:
resources:
requests:
memory: 5Gi
cpu: 3000m
limits:
memory: 18Gi
cpu: 3000m
Reference in a new issue View git blame Copy permalink