devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 12 Pull requests Releases Activity
build-cran-binaries/.crow/weekly-rebuild-missing.yaml
pat-s d992a2b636
All checks were successful
ci/crow/cron/process-updates/2 Pipeline was successful
Details
ci/crow/cron/process-updates/11 Pipeline was successful
Details
ci/crow/cron/process-updates/8 Pipeline was successful
Details
ci/crow/cron/process-updates/9 Pipeline was successful
Details
ci/crow/cron/process-updates/16 Pipeline was successful
Details
ci/crow/cron/process-updates/13 Pipeline was successful
Details
ci/crow/cron/process-updates/15 Pipeline was successful
Details
ci/crow/cron/process-updates/10 Pipeline was successful
Details
ci/crow/cron/process-updates/12 Pipeline was successful
Details
ci/crow/cron/process-updates/14 Pipeline was successful
Details
ci/crow/cron/process-updates/4 Pipeline was successful
Details
ci/crow/cron/process-updates/3 Pipeline was successful
Details
ci/crow/cron/process-updates/5 Pipeline was successful
Details
ci/crow/cron/process-updates/7 Pipeline was successful
Details
ci/crow/cron/process-updates/1 Pipeline was successful
Details
ci/crow/cron/process-updates/6 Pipeline was successful
Details
fix(ci): give each xvfb-run its own display (#230) 
Every `process-cran-updates-ubuntu-2204-arm64` cron run since pipeline 13744 has failed with `xvfb-run: error: Xvfb failed to start` (e.g. 13829).

## Cause

The newer-R loop added in #216 runs two `xvfb-run` calls back to back: the R-minor-sensitive `process_cran_updates` pass, then `local/build-newer-r.R`.
Without `-a`, `xvfb-run` always uses display `:99`, and on exit it kills its Xvfb without waiting for it.
The second call can therefore start while the old server still holds the `:99` lock, so the new Xvfb refuses to start.
Only ubuntu-2204-arm64 hits this because it is the only image with a second R minor installed, so the loop body only runs there.

## Change

- Default `XVFB_ARGS` to `-a` (`--auto-servernum`) in `process-updates.yaml`, `build-all-versions.yaml` and `weekly-rebuild-missing.yaml`, so each call picks a free display.
- The RHEL 10 `xwfb-run` branch still overrides `XVFB_ARGS` with `-c weston`, so it is unchanged.

## Verification

`crow lint .crow/` passes.
A local repro was not possible (`docker run` hangs on the dev host), so the next ubuntu-2204-arm64 cron run is the real check.

Reviewed-on: #230
2026-10-11 12:12:28 +00:00

403 lines
10 KiB
YAML
Raw Blame History

# Consolidated weekly-rebuild-missing pipeline (all platforms, both arches).
# Three matrix rows per OS/arch, one per shard of that slot's rebuild list.
# Routing is preserved 1:1:
# - cron: each existing `weekly-rebuild-missing-<os>-<arch>` cron fires only
# its matching matrix rows (via the per-row `cron:` name filter),
# which is now all three shards of that slot.
# - manual: pick a target from the `weekly_rebuild_missing` dropdown;
# "all" fans out every os/arch and shard as parallel matrix
# workflows, while a single <os>-<arch> runs its three shards.
# Arch placement is handled by the group label (rpkgs-amd64, rpkgs-arm64).
#
# The shard picks up its own slice and re-derives what is still outstanding
# from the bucket, so a restart resumes rather than replaying; see
# local/rebuild-missing.R.
#
# Re-indexing and the CDN purge deliberately do NOT live here. Three shards
# writing one slot's PACKAGES concurrently would race, so they moved to
# .crow/weekly-rebuild-reindex.yaml, which depends on this pipeline.
variables:
# Gates this pipeline. A manual pipeline creation instantiates every file in
# .crow/, and a declared default is applied even when the run never passed
# this variable, so the default must be a value that matches no matrix row.
weekly_rebuild_missing:
description: "Manual run target: a specific <os>-<arch>, 'all' to run every os/arch in parallel, or 'none' to run nothing."
options:
- none
- all
- alpine-323-amd64
- alpine-323-arm64
- alpine-324-amd64
- alpine-324-arm64
- redhat-8-amd64
- redhat-8-arm64
- redhat-9-amd64
- redhat-9-arm64
- redhat-10-amd64
- redhat-10-arm64
- ubuntu-2204-amd64
- ubuntu-2204-arm64
- ubuntu-2404-amd64
- ubuntu-2404-arm64
- ubuntu-2604-amd64
- ubuntu-2604-arm64
default: none
when:
- event: cron
cron: weekly-rebuild-missing-${OS}-${ARCH}
- event: manual
evaluate: 'weekly_rebuild_missing == "all" || weekly_rebuild_missing == "${OS}-${ARCH}"'
skip_clone: true
labels:
group: rpkgs-${ARCH}
matrix:
include:
- OS: alpine-323
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-323
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-323
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-323
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-323
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-323
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.23
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-324
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-324
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-324
ARCH: amd64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: alpine-324
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: alpine-324
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: alpine-324
ARCH: arm64
R_VERSION: 4.5.3
IMG: alpine:3.24
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-8
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-8
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-8
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-8
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-8
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-8
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:8
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-9
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-9
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-9
ARCH: amd64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-9
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-9
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-9
ARCH: arm64
R_VERSION: 4.4.3
IMG: redhat:9
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-10
ARCH: amd64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-10
ARCH: amd64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-10
ARCH: amd64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: redhat-10
ARCH: arm64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: redhat-10
ARCH: arm64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: redhat-10
ARCH: arm64
R_VERSION: 4.5.3
IMG: redhat:10
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2204
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2204
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2204
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2204
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2204
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2204
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:jammy
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2404
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2404
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2404
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2404
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2404
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2404
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:noble
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2604
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2604
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2604
ARCH: amd64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 3
- OS: ubuntu-2604
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 1
- OS: ubuntu-2604
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 2
- OS: ubuntu-2604
ARCH: arm64
R_VERSION: 4.4.3
IMG: ubuntu:resolute
SPLIT_INTO: 3
SPLIT_INDEX: 3
steps:
- name: 'Rebuild missing binaries'
image: reg.devxy.io/rpkgs/build-env-${IMG}
pull: true
environment:
OTEL_R_TRACES_EXPORTER: none
OTEL_R_LOGS_EXPORTER: none
OTEL_R_METRICS_EXPORTER: none
RED_HAT_DEV_PW:
from_secret: RED_HAT_DEV_PW
B2_S3_ACCESS_KEY:
from_secret: B2_S3_ACCESS_KEY
B2_S3_SECRET_KEY:
from_secret: B2_S3_SECRET_KEY
PGPASS:
from_secret: PGPASS
REPO_RO_TOKEN:
from_secret: REPO_RO_TOKEN
GITHUB_PAT:
from_secret: GITHUB_PAT
FORGEJO_TOKEN:
from_secret: FORGEJO_TOKEN
GIT_USER: pat-s
UVR_CACHE_DIR: /mnt/cache/uvr/cache
UVR_PACKAGES_DIR: /mnt/cache/uvr/packages
R_LIBS_USER: /mnt/cache/R-pkgs
R_VERSION: ${R_VERSION}
CCACHE_DIR: /mnt/cache/ccache
PLATFORM: ${OS}
ARCH: ${ARCH}
NCPUS: 2
SPLIT_INTO: ${SPLIT_INTO}
SPLIT_INDEX: ${SPLIT_INDEX}
# Wall clock after which the shard stops cleanly instead of having to be
# killed. A kill matches neither `success` nor `failure`, so it would skip
# the dependent re-index and leave rebuilt binaries behind a stale edge.
REBUILD_BUDGET_HOURS: 20
commands:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- mkdir -p /mnt/cache/uvr/cache /mnt/cache/uvr/packages /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
- /opt/R/$R_VERSION/bin/Rscript local/install-bincraft.R
- /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")'
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS="-a"; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
- UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh httr2
- /opt/R/$R_VERSION/bin/R -q -e 'source("local/fetch-rebuild-packages-from-issue.R")'
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/rebuild-missing.R $SPLIT_INTO $SPLIT_INDEX $REBUILD_BUDGET_HOURS 2>&1
backend_options:
docker:
resources:
requests:
memory: 5Gi
cpu: 3000m
limits:
memory: 18Gi
cpu: 3000m
Reference in a new issue View git blame Copy permalink