devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 10 Pull requests Releases Activity

refactor(ci): consolidate per-platform crow workflows into 3 matrix files #92

Merged
pat-s merged 2 commits from refactor/consolidate-process-updates into main 2026-06-16 09:33:07 +00:00
Conversation 0 Commits 2 Files changed 46 +498 -3056
pat-s commented 2026-06-16 07:53:39 +00:00
Owner

Summary

Collapse 44 per-platform crow pipeline files into 3 matrix-driven files (one per family), using matrix.include + the crow #1165 declarative manual variables: block:

Family Before After
process-updates 14 process-updates.yaml
weekly-audit-missing 16 weekly-audit-missing.yaml
weekly-rebuild-missing 14 weekly-rebuild-missing.yaml
  • One file per family (not split by arch). Arch placement is via the agent label as a matrix var (artemis=amd64, gaia=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-nodeSelector placement (no agent label).
  • Matrix axis is named OS (e.g. redhat-9) per request, with per-row R_VERSION, image tag, codename, and process_new (FALSE for alpine).
  • Runs preserved 1:1:
    • Cron: each existing <family>-<os>-<arch> cron still fires only its matching matrix row via CI_PIPELINE_CRON. Server-side cron entries unchanged.
    • Manual: a #1165 dropdown variable (process_cran_updates / weekly_audit_missing / weekly_rebuild_missing) selects a single <os>-<arch> or all. weekly-rebuild defaults to all, matching its former bare event: manual trigger.

Behavior deltas (intentional, flagged)

To fit one file per family the backend was unified:

  • The amd64-only node.kubernetes.io/instance-type: AX42 nodeSelector pin is dropped; placement now relies on the artemis/gaia agents.
  • process-updates arm64 jobs now share the same backend_options as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped.

Validation required before merge

Relies on crow interpolating matrix variables inside when.evaluate, labels.agent, and commands. Interpolation in image/env/commands is standard crow; the novel bit is when.evaluate. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch:

  1. Trigger manually with the dropdown = redhat-9-amd64 -> confirm only that one job runs (image build-env-redhat:9, R 4.4.3, agent artemis).
  2. Trigger with all -> confirm all os/arch jobs schedule.

Crons only fire on the default branch, so no collision while unmerged. Fallback if ${...} doesn't interpolate in when: a runtime case "$CI_PIPELINE_CRON" in ... guard, same files otherwise.

## Summary Collapse **44** per-platform crow pipeline files into **3** matrix-driven files (one per family), using `matrix.include` + the crow #1165 declarative manual `variables:` block: | Family | Before | After | |---|---|---| | process-updates | 14 | `process-updates.yaml` | | weekly-audit-missing | 16 | `weekly-audit-missing.yaml` | | weekly-rebuild-missing | 14 | `weekly-rebuild-missing.yaml` | - **One file per family** (not split by arch). Arch placement is via the agent label as a matrix var (`artemis`=amd64, `gaia`=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-`nodeSelector` placement (no agent label). - **Matrix axis is named `OS`** (e.g. `redhat-9`) per request, with per-row `R_VERSION`, image tag, codename, and `process_new` (FALSE for alpine). - **Runs preserved 1:1:** - *Cron*: each existing `<family>-<os>-<arch>` cron still fires only its matching matrix row via `CI_PIPELINE_CRON`. Server-side cron entries unchanged. - *Manual*: a #1165 dropdown variable (`process_cran_updates` / `weekly_audit_missing` / `weekly_rebuild_missing`) selects a single `<os>-<arch>` or `all`. weekly-rebuild defaults to `all`, matching its former bare `event: manual` trigger. ## Behavior deltas (intentional, flagged) To fit one file per family the backend was unified: - The amd64-only `node.kubernetes.io/instance-type: AX42` nodeSelector pin is **dropped**; placement now relies on the `artemis`/`gaia` agents. - **process-updates arm64** jobs now share the same `backend_options` as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped. ## Validation required before merge Relies on crow interpolating matrix variables inside `when.evaluate`, `labels.agent`, and `commands`. Interpolation in image/env/commands is standard crow; the novel bit is `when.evaluate`. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch: 1. Trigger manually with the dropdown = `redhat-9-amd64` -> confirm only that one job runs (image `build-env-redhat:9`, R 4.4.3, agent artemis). 2. Trigger with `all` -> confirm all os/arch jobs schedule. Crons only fire on the default branch, so no collision while unmerged. Fallback if `${...}` doesn't interpolate in `when`: a runtime `case "$CI_PIPELINE_CRON" in ...` guard, same files otherwise.
build-all reads a static pkgs_to_build.rds produced once by the install-deps
step, so a restarted build job re-cycles every package an interrupted run
already produced. Query the build-metadata DB at job start and drop the
(Package, Version) pairs already built successfully for this platform/arch,
so a restart only processes what is genuinely left.

Relies on bincraft writing the success row only after a confirmed S3 upload
(rpkgs/bincraft#56), so a DB success guarantees the binary is published.
Moves the process-updates-*, weekly-rebuild-missing-* and archive-missed
workflows onto bincraft 4.2.2, which records a build's success metadata only
after the S3 upload is confirmed (rpkgs/bincraft#56). Keeps these pipelines
consistent with the metadata-after-upload behavior the restart-skip filter
relies on.
Resolve .crow conflicts by taking main's content (which moved bincraft pins
to 4.2.1 and added per-minor build/install loops) and re-bumping every
bincraft pin to 4.2.2, so all pinned pipelines use the metadata-after-upload
release. build-all.R auto-merged: the restart skip-filter coexists with
main's r-minor-sensitive build path.
Replace the per-platform process-updates-<platform>-<arch>.yaml files with
process-updates-amd64.yaml and process-updates-arm64.yaml, each driving all
7 platforms via matrix.include. Split by arch because amd64 pins a nodeSelector
instance-type and runs on agent 'artemis' with backend_options, while arm64
runs on agent 'gaia' with none -- a single template can't cleanly express both.

Per-platform variance (R version, image tag, codename, and the process_new
flag which is FALSE for alpine) lives in matrix rows. Current runs are preserved
1:1: each existing per-platform cron still fires only its matching row via
CI_PIPELINE_CRON, and the manual trigger uses a PR-1165 dropdown
(process_cran_updates) for 'all' or a specific <platform>-<arch>. Server-side
cron entries are unchanged.
pat-s force-pushed refactor/consolidate-process-updates from 866957bdd5 to ed9442476f 2026-06-16 07:54:30 +00:00 Compare
pat-s force-pushed refactor/consolidate-process-updates from ed9442476f to d3bf7bc3a8
Some checks failed
ci/crow/manual/process-updates/13 Pipeline failed
Details
2026-06-16 08:08:44 +00:00 Compare
pat-s changed title from refactor(ci): consolidate process-updates workflows into 2 matrix files to refactor(ci): consolidate per-platform crow workflows into 3 matrix files 2026-06-16 08:09:18 +00:00 
Match the build-all cleanup: only the docker backend is used and arch placement
is via the agent label, so the kubernetes backend_options blocks (with arch
nodeSelector and tolerations) are dead. Convert them to docker resource blocks.
weekly-audit had no agent label (it relied on the nodeSelector), so add
agent: ${AGENT} with an AGENT column (artemis=amd64, gaia=arm64) per matrix row.
pat-s merged commit 4406a8735b into main 2026-06-16 09:33:07 +00:00
pat-s deleted branch refactor/consolidate-process-updates 2026-06-16 09:33:08 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries!92
No description provided.