devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 10 Pull requests Releases Activity
build-cran-binaries/.crow/weekly-audit-missing.yaml
pat-s 4406a8735b refactor(ci): consolidate per-platform crow workflows into 3 matrix files (#92) 
## Summary

Collapse **44** per-platform crow pipeline files into **3** matrix-driven files (one per family), using `matrix.include` + the crow #1165 declarative manual `variables:` block:

| Family | Before | After |
|---|---|---|
| process-updates | 14 | `process-updates.yaml` |
| weekly-audit-missing | 16 | `weekly-audit-missing.yaml` |
| weekly-rebuild-missing | 14 | `weekly-rebuild-missing.yaml` |

- **One file per family** (not split by arch). Arch placement is via the agent label as a matrix var (`artemis`=amd64, `gaia`=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-`nodeSelector` placement (no agent label).
- **Matrix axis is named `OS`** (e.g. `redhat-9`) per request, with per-row `R_VERSION`, image tag, codename, and `process_new` (FALSE for alpine).
- **Runs preserved 1:1:**
  - *Cron*: each existing `<family>-<os>-<arch>` cron still fires only its matching matrix row via `CI_PIPELINE_CRON`. Server-side cron entries unchanged.
  - *Manual*: a #1165 dropdown variable (`process_cran_updates` / `weekly_audit_missing` / `weekly_rebuild_missing`) selects a single `<os>-<arch>` or `all`. weekly-rebuild defaults to `all`, matching its former bare `event: manual` trigger.

## Behavior deltas (intentional, flagged)

To fit one file per family the backend was unified:
- The amd64-only `node.kubernetes.io/instance-type: AX42` nodeSelector pin is **dropped**; placement now relies on the `artemis`/`gaia` agents.
- **process-updates arm64** jobs now share the same `backend_options` as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped.

## Validation required before merge

Relies on crow interpolating matrix variables inside `when.evaluate`, `labels.agent`, and `commands`. Interpolation in image/env/commands is standard crow; the novel bit is `when.evaluate`. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch:
1. Trigger manually with the dropdown = `redhat-9-amd64` -> confirm only that one job runs (image `build-env-redhat:9`, R 4.4.3, agent artemis).
2. Trigger with `all` -> confirm all os/arch jobs schedule.

Crons only fire on the default branch, so no collision while unmerged. Fallback if `${...}` doesn't interpolate in `when`: a runtime `case "$CI_PIPELINE_CRON" in ...` guard, same files otherwise.

Reviewed-on: #92
2026-06-16 09:33:06 +00:00

161 lines
4 KiB
YAML
Raw Blame History

# Consolidated weekly-audit-missing pipeline (all platforms, both arches).
# One matrix row per OS/arch replaces the former per-platform files.
# Routing is preserved 1:1:
# - cron: each existing `weekly-audit-missing-<os>-<arch>` cron fires only
# its matching matrix row (via CI_PIPELINE_CRON).
# - manual: pick a target from the `weekly_audit_missing` dropdown
# ("all" = every os/arch).
# Arch placement is via the agent label (artemis=amd64, gaia=arm64).
variables:
weekly_audit_missing:
description: "Manual run target: a specific <os>-<arch>, or 'all' for every os/arch."
options:
- all
- alpine-321-amd64
- alpine-321-arm64
- alpine-322-amd64
- alpine-322-arm64
- alpine-323-amd64
- alpine-323-arm64
- redhat-8-amd64
- redhat-8-arm64
- redhat-9-amd64
- redhat-9-arm64
- redhat-10-amd64
- redhat-10-arm64
- ubuntu-2204-amd64
- ubuntu-2204-arm64
- ubuntu-2404-amd64
- ubuntu-2404-arm64
default: all
when:
- event: cron
evaluate: 'CI_PIPELINE_CRON == "weekly-audit-missing-${OS}-${ARCH}"'
- event: manual
evaluate: 'weekly_audit_missing == "all" || weekly_audit_missing == "${OS}-${ARCH}"'
skip_clone: true
labels:
agent: ${AGENT}
matrix:
include:
- OS: alpine-321
ARCH: amd64
AGENT: artemis
R_VERSION: 4.5.3
IMG: alpine:3.24
- OS: alpine-321
ARCH: arm64
AGENT: gaia
R_VERSION: 4.5.3
IMG: alpine:3.24
- OS: alpine-322
ARCH: amd64
AGENT: artemis
R_VERSION: 4.5.3
IMG: alpine:3.24
- OS: alpine-322
ARCH: arm64
AGENT: gaia
R_VERSION: 4.5.3
IMG: alpine:3.24
- OS: alpine-323
ARCH: amd64
AGENT: artemis
R_VERSION: 4.5.3
IMG: alpine:3.24
- OS: alpine-323
ARCH: arm64
AGENT: gaia
R_VERSION: 4.5.3
IMG: alpine:3.24
- OS: redhat-8
ARCH: amd64
AGENT: artemis
R_VERSION: 4.4.3
IMG: redhat:8
- OS: redhat-8
ARCH: arm64
AGENT: gaia
R_VERSION: 4.4.3
IMG: redhat:8
- OS: redhat-9
ARCH: amd64
AGENT: artemis
R_VERSION: 4.4.3
IMG: redhat:9
- OS: redhat-9
ARCH: arm64
AGENT: gaia
R_VERSION: 4.4.3
IMG: redhat:9
- OS: redhat-10
ARCH: amd64
AGENT: artemis
R_VERSION: 4.5.3
IMG: redhat:10
- OS: redhat-10
ARCH: arm64
AGENT: gaia
R_VERSION: 4.5.3
IMG: redhat:10
- OS: ubuntu-2204
ARCH: amd64
AGENT: artemis
R_VERSION: 4.4.3
IMG: ubuntu:jammy
- OS: ubuntu-2204
ARCH: arm64
AGENT: gaia
R_VERSION: 4.4.3
IMG: ubuntu:jammy
- OS: ubuntu-2404
ARCH: amd64
AGENT: artemis
R_VERSION: 4.4.3
IMG: ubuntu:noble
- OS: ubuntu-2404
ARCH: arm64
AGENT: gaia
R_VERSION: 4.4.3
IMG: ubuntu:noble
steps:
- name: 'Audit missing binaries'
image: reg.devxy.io/rpkgs/build-env-${IMG}
pull: true
environment:
B2_S3_ACCESS_KEY:
from_secret: B2_S3_ACCESS_KEY
B2_S3_SECRET_KEY:
from_secret: B2_S3_SECRET_KEY
PGPASS:
from_secret: PGPASS
REPO_RO_TOKEN:
from_secret: REPO_RO_TOKEN
GITHUB_PAT:
from_secret: GITHUB_PAT
FORGEJO_TOKEN:
from_secret: FORGEJO_TOKEN
PLATFORM: ${OS}
ARCH: ${ARCH}
R_LIBS_USER: /mnt/cache/R-pkgs
R_VERSION: ${R_VERSION}
commands:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
- /opt/R/$R_VERSION/bin/R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))'
- /opt/R/$R_VERSION/bin/R -q -e 'source("local/weekly-missing-binaries-audit.R")'
backend_options:
docker:
resources:
requests:
memory: 2Gi
cpu: 2000m
limits:
memory: 4Gi
cpu: 2000m
Reference in a new issue View git blame Copy permalink