887 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
pat-s	13fcad6dc4	fix(crow): make the trial-build gate no-op when the auto-patch branch is absent (#132) ... ## Problem The trial pipeline errored: ``` git clone -q --branch auto/registry-patch-proposals ... fatal: Remote branch auto/registry-patch-proposals not found in upstream origin ``` I deleted the stale `auto/registry-patch-proposals` branch during cleanup, and — now that cascade detection (#128) makes the proposer correctly find **no candidates** — the proposer exits before ever recreating it. So the branch is legitimately absent, and the gate's hard `git clone --branch` fails. ## Fix "Branch absent" = "no pending auto-patch proposals" = **nothing to verify**, which should be a clean no-op, not a failure. Clone `main`, then fetch + checkout the patch branch only if it exists; otherwise log and `exit 0`. A present branch is verified exactly as before (checkout its tip, diff registry vs `main`). This also means the gate now runs `main`'s `trial-build-registry.R` (with the #130 fix) rather than a possibly-stale copy on the branch. Reviewed-on: #132	2026-07-17 10:27:58 +00:00
pat-s	6c03f278ec	feat(local): aggregate blocked-on-dependency reporting by dependency (#131) ... ## Why With cascade detection (#128) live, the latest `auto-apply-patches` run did exactly the right thing — **proposed nothing** (`No auto-proposable candidates`) because every failure is a dependency cascade, and it surfaced the ~30 root-cause dependencies to fix. But the "Blocked on a dependency" list printed **one line per fingerprint group**, so the same dependency repeated (rstan ×4, lpsymphony ×4, salso ×2, BH ×2, GO.db ×2, RcppCWB ×2, …), burying the priority. ## What Aggregate blocked packages across all groups **by the dependency they wait on**: - Expose `blocked_map` (package → dependency) from `build_triage_report()`. - Add `blocked_by_dependency()` — dedupes dependents (a package in two groups counts once) and ranks dependencies by how many distinct dependents they block. - Proposer and tracker (log + issue) now print one line per dependency, sorted by impact. Replaces the per-group `blocked_summary`. ## Result (same data, aggregated) ``` Blocked on a dependency (3 dependencies block 6 dependents; fix the dependency, not each dependent): RcppParallel 3 dependent(s) rstan 2 dependent(s) sf 1 dependent(s) ``` So the real run becomes a crisp, ranked worklist: RcppParallel (894), sf (128), rstan (~96), Rfast (33), clarabel/DescTools (26), Rglpk (22), xgboost (18), … ## Verified New test covers cross-group aggregation, dedup (a dependent in two groups counted once), the example cap, and ranking. 112 tests pass; hooks pass. Reviewed-on: #131	2026-07-16 21:24:03 +00:00
pat-s	118a92889f	fix(local): make the trial-build gate detect non-throwing build failures (#130) ... ## Critical: the gate was false-green The latest run printed **`3/3 passed on alpine-324`** while all three builds actually **failed** (their `rstan` dependency won't compile). A false-green gate would let broken registry entries merge — worse than no gate. Root cause: `bincraft::build_binary_package()` catches build failures internally and **returns `"error"`** for the failed tag rather than throwing. The gate's `tryCatch` only treated a *thrown* exception as failure, so every non-throwing failure looked like a pass. Fix: inspect the return value. A tag passes only if the flattened result is non-empty and contains no `"error"` sentinel; a thrown error still counts as failure. Verified the verdict against `error`/`skipped`/`TRUE`/`list(success=TRUE)`/`NULL`/mixed inputs. With this, the current rstan-blocked entries will correctly show **0/3 (red)** — which is the right answer until rstan builds. Reviewed-on: #130	2026-07-16 08:24:03 +00:00
pat-s	f9d399fac0	feat(local): detect dependency-cascade failures generally, not just RcppParallel (#128) ... ## Why (from the #127 trial-build gate) The gate did its job: 0/3 passed, merge blocked. The log showed *why* -- BFpack, BayesERtools, GMLTM all fail while building their shared dependency **`rstan`**, not in their own code: ``` Failed to build source package rstan. .../StanHeaders/include/stan/math/prim/core/init_threadpool_tbb.hpp:9:10: fatal error: tbb/tbb_stddef.h: No such file or directory ``` So the per-package `-DTBB_INTERFACE_NEW` makevars entries the classifier proposed are useless for these packages -- they're blocked on `rstan` (which already has a registry entry). This is the **same dependency cascade** the RcppParallel `applies_to` guard catches, but `tbb-stddef-removed` is a generic signature with no such pin, so ~73 Stan packages kept getting proposed. ## What Generalise cascade detection beyond the RcppParallel special case: - `failing_dependency(error_text, package)` -- when the log names a **different** package as the one that failed to compile (`Failed to build source package X`, `compilation failed for package 'X'`, `dependency 'X' ... not available`), that package is the real cause. - `build_triage_report()` now blocks any package whose **every** failing build is such a cascade: reported as `blocked_on` that dependency, never proposed a bogus per-package entry. A package that fails in its **own** compilation is still proposed. - The `applies_to` (RcppParallel) and data-driven (rstan) cases are unified into one `blocked_packages` / `blocked_on` model; the report, proposer, and `blocked_summary` count the actually-blocked packages, and the blocked note shows even when a group also has genuine proposals. ## Effect Next auto-apply run will stop proposing the rstan-blocked Stan packages (and any future dependency cascade) and surface them as "blocked on rstan" instead. Fixing `rstan` once clears the whole cluster. ## Verification - New tests: `failing_dependency` (cascade vs own-compile vs none), and an end-to-end split where BFpack/GMLTM (blocked on rstan) are not proposed while an own-compile package still is. - Full suite: 112 tests pass; all pre-commit hooks pass. Refs #120, #127. (Separate follow-ups: fixing rstan's build itself, and quieting the gate's metadata-DB retry storm -- both root-caused to bincraft.) Reviewed-on: #128	2026-07-16 08:23:54 +00:00
pat-s	875b086122	ci(crow): enable verbose patched builds in the trial-build gate (#129) ... Companion to bincraft codefloe #65. Sets `BINCRAFT_VERBOSE_PATCH_BUILD=TRUE` in the trial-build gate so a failed isolated patched build prints the real compiler error instead of `System command 'R' failed`. Harmless on bincraft versions without the flag (unknown env var is ignored). Takes effect once bincraft #65 is released and the build-env images pick it up. Reviewed-on: #129	2026-07-16 08:23:44 +00:00
pat-s	ff9f5f5177	fix(local): shell-quote git args in the auto-patch push and trial-build gate (#126) ... ## Problem The first real `auto-apply-patches` run classified, limited to the top-10, and validated the candidate registry cleanly, then died at the push step: ``` Validating candidate registry: Patch registry OK (13 entries). sh: syntax error: unexpected "(" Error in git("commit", "-m", ...): git commit -m feat(patches): auto-propose 10 registry entries ... failed ``` Root cause: R's `system2()` with captured output (`stdout=TRUE`) runs the command through `/bin/sh`, and the arguments were passed **unquoted**. The commit message `feat(patches): ...` contains `()`, which the shell tried to interpret. The same class of bug affects the `^{commit}` and `ref:path` git refs in the trial-build gate. ## Fix `shQuote()` every git argument: - `propose-patches.R` -- the `git()` helper used by `--open-pr` (commit, push, checkout). - `trial-build-registry.R` -- the base-ref reads (`rev-parse ... ^{commit}`, `ls-tree`, `show ref:path`). ## Verification - Reproduced against a real git repo: the unquoted call fails with status 2 (the same `unexpected "("`); the `shQuote`d call commits successfully, and `rev-parse HEAD^{commit}` resolves. - 105 tests pass; all pre-commit hooks pass. Everything else in that run was correct: 842,659 failing builds classified, RcppParallel's 895 dependents correctly reported as blocked (not proposed), top-10 tbb-stddef candidates selected, 69 deferred, registry validated. Only the shell quoting was broken. Reviewed-on: #126	2026-07-15 15:43:02 +00:00
pat-s	ed1485c542	ci: remove evaluate conditions	2026-07-15 16:51:15 +02:00
pat-s	1bca6004fd	chore(local): reuse FORGEJO_TOKEN for the auto-patch push (#125) ... Follow-up to #124 (merged): the auto-patch pipeline required a separate write-scoped `REPO_RW_TOKEN` to push the branch. Reuse the existing `FORGEJO_TOKEN` instead. - `propose-patches.R --open-pr` now pushes `auto/registry-patch-proposals` over HTTPS with `FORGEJO_TOKEN` (the same token used for the PR API); the read-only `origin` clone URL can't push, so it builds an authenticated URL explicitly. - Drop `REPO_RW_TOKEN` from `.crow/auto-apply-patches.yaml` and the docs. No new secret needed: the pipeline's secrets are now `PGPASS`, `REPO_RO_TOKEN`, and `FORGEJO_TOKEN` (all existing). `FORGEJO_TOKEN` must have repository write scope for the push to succeed. 105 tests pass; all pre-commit hooks pass. Reviewed-on: #125	2026-07-15 14:50:19 +00:00
pat-s	4bca17e4ac	feat(local): auto-apply registry patches with a build-env trial-build gate (#124) ... Closes the classifier loop (issue #115, step 3): turns the auto-proposable candidates into an actual PR, gated by a real trial build **in our own build-env images**. Chosen model (from the design discussion): **autonomous PR**, **PR-first with a CI trial-build gate**, **bounded top-N batch** per run. ## Creating the patch PR - `propose-patches.R` gains: - `--limit N` -- act on the top-N candidates by failure volume; the rest defer to the next run (logged). - `--open-pr` -- write the entries onto the reused `auto/registry-patch-proposals` branch, push (with `REPO_RW_TOKEN`), and open/update **one** PR via the Forgejo API (so re-runs update the same PR instead of piling up). - `.crow/auto-apply-patches.yaml` -- a single job that runs `--open-pr --limit` on a cron/manual trigger. Needs `FORGEJO_TOKEN` + a write-scoped `REPO_RW_TOKEN`. Novel source diffs and unknown signatures are still never proposed; nothing merges. ## The merge gate (our build-env images) - `.crow/trial-build-registry.yaml` -- matrixed over the real `OS/IMG` build-env matrix (alpine:3.24, redhat:8/9/10, ubuntu:jammy/noble/**resolute** for ubuntu-2604). Each platform runs `local/trial-build-registry.R`, which diffs the branch registry against `main` and trial-builds **only the entries the branch adds** that apply to that platform, inside `reg.devxy.io/rpkgs/build-env-*`. Green only if every new entry builds; a platform with no new entries is a fast no-op. Nothing is uploaded/archived/recorded. - The base-registry read **fails loud** if it can't read `registry.json` at `main`, rather than silently treating the base as empty and trial-building the whole registry. ## Notes / follow-up - The repo uses **no `pull_request` triggers**, so the gate runs manually or on a cron against the branch (`--var patch_branch=...`). Wiring it to fire automatically on the PR needs `event: pull_request` enabled on the Forgejo webhook -- a one-line addition once that's on. - Two new crons to register in the crow UI: `auto-apply-patches` and `trial-build-registry`. New secret needed: `REPO_RW_TOKEN` (write scope) for the push. ## Verification - New pure helpers `entry_applies_to_os()` / `new_registry_packages()` covered by tests (platform codename/family/wildcard matching; added-vs-unchanged entry detection; per-platform filtering). - `--limit` smoke (stubbed DB): top-2 by volume proposed, 3 deferred, candidate registry validates. - Full suite: 105 tests pass; all pre-commit hooks pass (air, prettier, markdownlint, yamllint, validate-patches). Reviewed-on: #124	2026-07-15 08:28:15 +00:00
pat-s	21a2fe9e6c	feat(local): report unclassified and dependency-blocked failures for discovery (#122) ... ## Why Issue #120 (the auto-proposed-patches issue) only lists **auto-proposable** fixes -- currently just the TBB signatures. So a reasonable read of it was "TBB is our only build failure", when in fact three whole categories are simply not shown there: - **Unclassified failures** -- anything that doesn't match a seeded signature is routed to human triage and never appears (we've only seeded TBB and libuv signatures). - **Dependency-blocked failures** -- the ~800 RcppParallel dependents (post #121) are still failing; they only show as a log line. - Human-only signatures (libuv). These blind spots are exactly where the *next* signatures should come from, so they deserve the same visibility as the proposals. ## What Extend the feedback-loop tracker to surface the classifier's blind spots: - **`unclassified_summary()`** -- groups every unknown-signature failure by normalised fingerprint, ranked by build count, capped with an explicit `dropped_groups` count (no silent truncation), each with example packages + platforms. These are the candidates for new `build_signatures()` rules. - **`blocked_summary()`** -- lists each dependency (e.g. RcppParallel) and how many dependents wait on it. - `proposal-tracking.R` prints both sections, and a new **`--open-issue`** mode posts/updates a *"Unclassified build failures (needs signatures) (#115)"* Forgejo issue. - The weekly crow pipeline now runs the tracker with `--open-issue`, so it maintains a second tracking issue alongside the proposals one. Read-only on the DB; the only writes are the two issues. ## Verification - New tests cover `unclassified_summary` (ranking + both caps) and `blocked_summary`. - Tracker smoke with a stubbed DB (proposable + blocked + unclassified mix) prints the hit rate, `Blocked on a dependency: RcppParallel: 2 dependent(s)`, and `Unclassified failures ... [2 builds | 2 pkgs] ld: undefined reference ...`. - Full suite: 95 tests pass; all pre-commit hooks pass (air, prettier, markdownlint, yamllint, validate-patches). Reviewed-on: #122	2026-07-15 07:46:29 +00:00
pat-s	9bc4973159	fix(local): stop proposing per-dependent patches for dependency-cascade failures (#121) ... ## Problem The scheduled `--open-issue` run (issue #120) posted **800+** proposed entries, all `rcppparallel-bundled-tbb` on `ubuntu-2604`, each pre-filling `RcppParallel/disable-tbb.patch` for an unrelated package (`ACEsimFit`, `AovBay`, `AdaptGauss`, ...). Root cause: those packages fail on ubuntu-2604 only because their **RcppParallel dependency** does not build there. bincraft records RcppParallel's TBB error text against each dependent, so every one of them matches the `rcppparallel-bundled-tbb` signature. Proposing a RcppParallel source patch for `ACEsimFit` is not just noise -- it is **broken**: the diff targets RcppParallel's `Makevars.in`, so it cannot apply to a dependent's source. ## Fix Add an optional `applies_to` field that pins a package-specific fix to the package it targets: - A signature whose fix is a curated per-package source patch (`rcppparallel-bundled-tbb` -> `RcppParallel`) is only ever proposed **for that package**. - Other packages matching the signature are downstream failures, reported as **"blocked on `<dependency>`"** and never proposed a bogus entry. Fix the dependency once and the whole cascade clears. Generic levers (the `tbb-stddef-removed` makevars fix) are unaffected -- they carry no `applies_to`, so they still apply to any matching package. ## Changes - gate proposal generation in `build_triage_report()` on `applies_to`; add a `blocked_on` field to group records - surface blocked groups in both `failing-builds-report.R` and `propose-patches.R` instead of the misleading "already registered" note - document `applies_to` in the signature-table header ## Verification - New tests: RcppParallel dependents are blocked (no `proposed_entries`, `blocked_on = "RcppParallel"`), while RcppParallel *itself* is still proposed when it is the failing package. - Reproduced the avalanche end-to-end with a stubbed DB: 5 dependents now report "Blocked on RcppParallel: 5 package(s) fail because RcppParallel does not build" and **zero** candidates are emitted; a genuine `tbb-stddef` proposal in the same run is unaffected. - Full suite: 84 tests pass; all pre-commit hooks pass. Refs #120. Note: the underlying build problem (RcppParallel failing on ubuntu-2604 despite its registry entry) is real and separate -- this PR stops the classifier from spamming broken per-dependent proposals about it. Reviewed-on: #121	2026-07-14 18:40:41 +00:00
pat-s	f1dd661213	fix(local): route multi-signature packages to human triage in the proposer (#119) ... ## Problem A live `propose-patches.R --open-issue` run aborted with: ``` Patch registry validation FAILED: - ambiguous duplicate entries: hmmTMB|ubuntu-2604|*, imt|ubuntu-2604|*, refundBayes|ubuntu-2604|* Error: Candidate registry failed validation; not writing or proposing. ``` Root cause: a package whose failing builds match **more than one** auto-proposable signature (e.g. some logs hit `tbb-stddef-removed`, others `rcppparallel-bundled-tbb`) landed in two separate signature groups, so the proposer emitted two candidate entries with the same `package|platforms|versions` key. The validator's ambiguous-duplicate check then failed the whole candidate set, aborting the run — so even the unambiguous candidates (e.g. `vacalibration`) never got proposed. ## Fix Add a pure, tested `dedupe_candidates()` that: - keeps a package's candidate only when it maps to a **single** signature (collapsing an accidental repeat under the same signature), and - routes a package that maps to **multiple** signatures (conflicting fix tiers -- makevars vs source patch) to **human triage** instead of guessing between them. The proposer prints the skipped ambiguous packages and proceeds with the clean candidates, so one ambiguous package no longer blocks the rest. This matches the issue #115 guardrail: don't auto-pick when the fix is ambiguous. ## Verification - New unit tests in `test-proposal-tracking-lib.R` cover the single-signature, multi-signature (conflict), duplicate-under-one-signature, and empty cases. - Reproduced the original failure end-to-end with a stubbed DB (`hmmTMB` ambiguous + `vacalibration` clean): `hmmTMB` is now listed under "Ambiguous -> human triage", `vacalibration` is proposed, and the candidate registry validates (`Patch registry OK`). - Full suite: 78 tests pass; all pre-commit hooks pass. Reviewed-on: #119	2026-07-14 17:11:54 +00:00
pat-s	77508f8fc5	ci: clean weekly-patch-proposals trigger	2026-07-14 18:17:28 +02:00
pat-s	af481f5129	ci(crow): add weekly patch-proposal + feedback-loop pipeline (#118) ... Adds the scheduled pipeline for the failure-triage proposer/tracker landed in #117 (issue #115, steps 3 + 4). ## `.crow/weekly-patch-proposals.yaml` A single (non-matrix) job -- the classifier groups over all of `single_builds`, so unlike the per-platform `weekly-audit-missing` it runs once globally. Weekly it: 1. `propose-patches.R --open-issue` -- posts/updates a Forgejo tracking issue with the auto-proposable registry entries (human reviews, trial-builds, opens the PR; nothing merges). 2. `proposal-tracking.R` -- logs the feedback-loop metrics (signature hit rate, proposed-vs-merged, retirement candidates). Low blast radius: clones read-only (`REPO_RO_TOKEN`); the only write is the tracking issue via `FORGEJO_TOKEN`. It builds nothing (no B2 keys, no bincraft). Triggers: `cron: weekly-patch-proposals` (register the cron in the crow UI) or manual with `task=weekly-patch-proposals`, matching the `archive-missed-packages` convention. Documented under "Scheduled run" in `local/patches/README.md`. `yamllint` and the other pre-commit hooks pass. Reviewed-on: #118	2026-07-14 15:17:16 +00:00
pat-s	1c297c01bf	feat(local): auto-propose registry patches and track the feedback loop (#117) ... Implements steps 3 + 4 of #115, building on the classifier merged in #116. Now that bincraft **v4.4.3** applies registry `patch`/`makevars`/`configure_args` to the *target* package build (previously deps-only), a trial patched build is a meaningful acceptance gate, so the "propose" half is viable. ## Step 3 — propose, do not apply - **`local/propose-patches.R`** — for each classified, safe fix affecting a package with no current registry entry, emits a pre-filled `registry.json` entry and validates the candidate set against a *temporary* merged registry (the real one is never touched unless asked). - default: print candidates + validation, **take no action** - `--write`: append entries to `registry.json` + the proposals ledger (you commit + open the PR) - `--open-issue`: post/update a Forgejo tracking issue (reuses the weekly-audit `httr2` + `FORGEJO_TOKEN` pattern) - **`local/trial-build-patch.R`** — isolated bincraft build of one package with the registry applied (no upload/archive/metadata; `patchhash` keeps it out of the real cache). Exit 0/1, so it gates a CI step or manual pre-merge check. The human gate stays: nothing merges. Acceptance = `validate-patches.R` passes (checked automatically) **and** the trial build succeeds. Novel source diffs and unknown signatures are never proposed (they carry `auto = FALSE`). ## Step 4 — feedback loop - **`local/proposal-tracking.R`** (read-only) — signature hit rate (builds/pkgs/addressed/open per signature), proposed-vs-merged (a proposal counts merged once its package is in the registry), and retirement candidates (registry entries whose package no longer fails, i.e. likely fixed upstream). - **`local/proposal-tracking-lib.R`** — the pure metric/ledger helpers. ## Supporting changes - Refactored the classify helpers to expose a pure `build_triage_report()` + a list-returning entry builder; `failing-builds-report.R` now renders from the shared function (no behaviour change). - `validate-patches.R` gains optional `PATCH_DIR`/`REGISTRY_FILE` overrides (backward-compatible) so a candidate registry can be validated in isolation. - Documented the propose/trial-build/tracking workflow in `local/patches/README.md`. ## Verification - 71 unit tests pass (incl. new `test-proposal-tracking-lib.R`) under the Dockerized R 4.5.3 build env. - All pre-commit hooks pass (`air-format`, `validate-patches`, prettier, etc.). - Smoke-tested all three entrypoints end-to-end with a stubbed DB: dry-run, `--write` (produces a registry that passes the canonical validator + a valid ledger, then reverted), and the tracker. Closes #115 Reviewed-on: #117	2026-07-14 15:03:24 +00:00
pat-s	f11ba7172f	feat(local): classify failing binary builds and pre-fill registry suggestions (#116) ... Implements steps 1 + 2 of #115: turn recorded build failures into triaged patch suggestions instead of hand-scraping Crow logs. ## What this adds A **read-only** reporting pipeline over the `single_builds` metadata table. It never writes to the DB or the registry. - `local/failing-builds-classify.R` — pure, DB-free helpers: - `normalise_error()` strips temp paths, version numbers, hex addresses, and the package name so the same root cause collapses to one fingerprint. - `fingerprint_error()` extracts the salient error line and normalises it. - `classify_error()` matches against a seed signature set; unmatched errors are never guessed at. - `propose_registry_entry()` renders a schema-valid `registry.json` entry. - `local/failing-builds-report.R` — entrypoint: queries `single_builds WHERE error_occurred = TRUE AND removed = FALSE`, groups by root cause (signature when classified, fingerprint otherwise), classifies each group, and prints a triaged report. Flags: `--platform`, `--arch`, `--min`, `--json`; `PLATFORM`/`ARCH` env fallbacks. - `local/tests/test-failing-builds-classify.R` — unit tests for the helpers. - `local/patches/README.md` — documents the workflow. ## Seed signatures Each rule carries a fix tier, confidence, and an auto/human-only flag: | Signature | Fix | Disposition | | --- | --- | --- | | `tbb/tbb_stddef.h: No such file` | makevars `-DTBB_INTERFACE_NEW` | auto-proposable | | RcppParallel bundled TBB (musl / new g++) | curated `disable-tbb.patch` | auto-proposable | | system `libuv.so` link leak | force vendored/static lib | **human triage** (novel source diff) | | unmatched | none | **human triage** | ## Guardrails honored - No autonomous novel source diffs: only known env/makevars levers and already-curated package patches are auto-proposable; anything needing a brand-new diff, and any unknown signature, is routed to human triage. - No DB or registry writes; no change to the public `src/contrib` index. - Reuses `single_builds.error_text`; no new failure-capture pipeline. ## Verification - All helper unit tests pass under the Dockerized R 4.5.3 build env. - Pre-commit hooks pass (`air-format`, `validate-patches`, prettier, etc.). - Smoke-tested the full report path with a stubbed DB; generated proposals pass the real `local/validate-patches.R`. Steps 3 (auto-open PRs) and 4 (feedback loop) are intentionally deferred, per the issue's suggestion to validate the signature set first. Closes #115 Reviewed-on: #116	2026-07-14 14:45:25 +00:00
pat-s	04059f20e3	fix(patches): add rstan entry to skip removed tbb_stddef.h (#114) ... ## Motivation `rstan` fails to compile when it is pulled in as a transitive dependency (e.g. of `ctsem`), aborting the dependency install: ``` /mnt/cache/R-pkgs/StanHeaders/include/stan/math/prim/core/init_threadpool_tbb.hpp:9:10: fatal error: tbb/tbb_stddef.h: No such file or directory 9 | #include <tbb/tbb_stddef.h> ``` StanHeaders' `init_threadpool_tbb.hpp` unconditionally includes the legacy `<tbb/tbb_stddef.h>` to detect the TBB version, but that header was removed in oneTBB 2021+, which is the TBB that StanHeaders now bundles. ## Fix Add an `rstan` entry to `local/patches/registry.json` that pre-defines `TBB_INTERFACE_NEW` via `CPPFLAGS`. The header's own guard skips the removed include when that macro is already defined and selects the modern `tbb/global_control.h` + `tbb/task_arena.h` path that the bundled oneTBB actually provides: ```cpp #ifndef TBB_INTERFACE_NEW #include <tbb/tbb_stddef.h> // removed in oneTBB 2021+ #if TBB_VERSION_MAJOR >= 2020 #define TBB_INTERFACE_NEW #endif #endif #ifdef TBB_INTERFACE_NEW #include <tbb/global_control.h> // taken when we pre-define the macro #include <tbb/task_arena.h> ``` This is a makevars-tier override (no source diff), applied through bincraft's `R_MAKEVARS_USER`, so `CPPFLAGS` stays additive and does not clobber rstan's own `PKG_CPPFLAGS`. `-I/usr/local/include` is kept because the override replaces the default `CPPFLAGS`. ## Files changed - `local/patches/registry.json`: add the `rstan` entry (`versions: "*"`, `platforms: ["*"]`, `makevars.CPPFLAGS = "-DTBB_INTERFACE_NEW -I/usr/local/include"`). ## Behaviour change When bincraft resolves `rstan` (direct or transitive) it builds a patched binary with `TBB_INTERFACE_NEW` defined, so `rstan` (and dependents such as `ctsem`) compile against the bundled oneTBB. The scope is `["*"]` because the failure is bound to StanHeaders' bundled TBB version, not the OS or toolchain. Reviewed-on: #114	2026-07-14 13:52:17 +00:00
pat-s	a4b274f281	fix(ci): bound pkgcache _metadata growth to stop macmini disk-fill (#110) ... ## Problem The arm64 `build-all` pipeline fills the macmini (gaia) host disk despite an 8h prune. Root cause is not images or job volumes: it is the persistent dep-cache volume, specifically `pkgcache/R/pkgcache/_metadata`, which grew to ~165 GB. `{pkgcache}` mints a new content hash for the "patched" binaries repo on every PACKAGES change, so each per-package build writes a fresh ~70 MB `pkgs-<hash>.rds` (+ `patched-<hash>/`) that is never evicted (2407 snapshots observed). When the disk hits 100% OrbStack stops and the on-host prune can no longer connect to the daemon, so it never self-heals. ## Change (Workstream A of the disk-fill fix) - Add `trim_pkgcache_metadata()` to `local/r-minor-helpers.R`: keeps the newest `keep` (default 20) `patched-*`/`pkgs-*.rds` entries under `_metadata`, deleting only entries older than `min_age_secs` (default 600s) so it never races the up-to-4 concurrent split-jobs sharing the volume. Preserves `pkg/` downloads and the stable CRAN/BioC/INLA repo dirs. No-op when `R_PKG_CACHE_DIR` is empty (amd64) or `_metadata` is absent (first run). - Call it every 25 packages inside the build loop in `local/build-all.R`. - Add a defensive start-of-run cleanup of `_metadata/patched-*` + `pkgs-*.rds` to the two workflows that mount the persistent volume (`build-all-versions.yaml`, `build-all-versions-install-deps.yaml`). Only these paths are touched; `process-updates.yaml`/`weekly-rebuild-missing.yaml` (no persistent volume) are unchanged. Follow-ups (separate workstreams): on-host self-healing prune watcher + OrbStack disk cap (ansible), and Prometheus/Grafana alerting (k8s-talos). Upstream: bincraft patched-repo hash churn is the true source fix. New unit tests (6) for the helper; full suite 24/24 green. Reviewed-on: #110	2026-07-13 09:31:29 +00:00
pat-s	ffc2319558	fix(build-all): exclude previously-errored versions in prefilter (#113) ... ## Motivation Build jobs were cycling through hundreds of packages that were only ever printed as `Skipping … due to previous build error recorded in metadata DB`, wasting wall-clock on per-package preparation before dropping each one. ## Cause The prefilter query in `local/build-all.R` selected only successfully-built versions (`error_occurred = FALSE`) into `built`, so line 112 removed only those from the chunk. Every previously-errored version stayed in the work list and was walked one-by-one, each hitting the internal skip in `build_binary_package()`. This also explains the misleading `Skipped 0 already-built package versions` line for alphabetical chunks whose leading packages only have error records. ## Changes - `local/build-all.R`: drop the `AND error_occurred = FALSE` clause so `built` holds every version already attempted (built or errored) for this platform/arch; the existing filter then removes all of them up front. - Rename the log line to `already-attempted` so the reported count reflects successes and errors. - Update the surrounding comment to explain why errored versions are excluded. ## Behaviour change Previously-errored versions are now dropped before the build loop instead of being iterated and individually skipped. No package that would otherwise build is affected, `build_binary_package()` already skipped these internally. Retrying errored versions is out of scope and would need a separate opt-in flag on both the prefilter and the in-loop skip. Reviewed-on: #113	2026-07-13 09:28:45 +00:00
pat-s	09536cddd6	feat(build): raise BuildKit cache-mount budget for remote builders (#112) ... BuildKit's default GC caps the ephemeral cache tier — `RUN --mount=type=cache` mounts, local build context, git checkouts — at a hardcoded **512 MB** (shown as `488.3 MiB` in `buildx inspect`). Across our 7-distro build matrix that fills instantly and forces re-downloads of system + R packages on every rebuild. ## Changes - **`docker/buildkitd.toml`** (new) — GC config passed via `--config` when creating the `docker-container` builders `artemis` (amd64) and `gaia` (arm64): - cache-mount tier: **512 MB → 8 GB**, retained 7 days - total cache bounded at **40 GB** with **20 GB min-free** - **`justfile`** — document the `--config docker/buildkitd.toml` flag on the builder-create commands so a recreate does not silently revert to the 512 MB default. ## Notes - Limits are absolute (not `%`) because the two hosts differ ~6× in free space (Hetzner ~42 GiB free vs Mac mini ~279 GiB). The 20 GB min-free is the safety valve on the disk-tight Hetzner host; the old default wanted 64 GiB free, which does not exist there. - Both live builders were already recreated with this config and verified running. Reviewed-on: #112	2026-07-08 08:22:03 +00:00
pat-s	36d3bc8604	fix(patches): force fs to build vendored static libuv (#111) ... ## Problem The `fs` 2.1.0 binary links **system libuv** (`readelf -d fs.so` shows `NEEDED libuv.so.1`). fs's `configure` prefers system libuv whenever `pkg-config` resolves it, and our build images ship `libuv-devel` (installed as a pak build-time system requirement), so the resulting binary is dynamically linked against `libuv.so.1`. That binary fails to load on any consumer machine without runtime libuv: ```text unable to load shared object '.../fs/libs/fs.so': libuv.so.1: cannot open shared object file: No such file or directory ``` `install.packages()`/renv do **not** install `SystemRequirements` (only `pak` does, and only inside the build container), so most consumers hit this. Older fs 1.6.x always vendored libuv, so only the 2.x binaries regressed. Reproduced in a clean `reg.devxy.io/r/r-alma:4.5-9`. ## Fix Add `local/patches/fs/force-vendored-libuv.patch`, registered for all platforms. It short-circuits `configure` to `cp -f src/Makevars.vendor src/Makevars; exit 0` before the pkg-config detection, forcing the bundled static libuv build (`tools/libuv-v1.52.0.tar.gz`, built via cmake). An env/pkg-config override (`PKG_CONFIG_LIBDIR`) was tried first but the rebuilt binary still linked `libuv.so.1` (the registry `env` tier does not reach fs's configure step), so a source patch is used instead. ## Verification Built end-to-end inside the real `build-env-redhat:9` image (system libuv present): - patch fires (`Building static libuv (bincraft: forced vendored)`), - cmake compiles the vendored libuv, - resulting `fs.so` has **no `libuv.so.1`** in `NEEDED` (only libR, libstdc++, libm, libgcc_s, libc). `Rscript local/validate-patches.R` passes (2 entries). cmake confirmed present in the build-env images. ## Follow-up (not in this PR) - Rebuild `fs 2.1.0` on every affected platform (rhel8/9/10, ubuntu jammy/noble, alpine 3.22/3.23; amd64 + arm64) and purge the CDN binary paths. - CDN delivery gap: `purge_cdn_cache.sh` only purges `PACKAGES*`, never package binaries, so rebuilt binaries stay masked until their `.tar.gz` path is purged. Reviewed-on: #111	2026-07-07 18:12:36 +00:00
pat-s	1e06576084	docs: add CLAUDE.md with B2 storage gotchas and build pipeline conventions for agents	2026-07-02 08:43:37 +02:00
automation-bot	5f10863a9c	chore(deps): update pre-commit hook davidanson/markdownlint-cli2 to v0.23.0	2026-07-02 00:32:11 +00:00
pat-s	1ad7a6bfe9	chore: resolve latest bincraft release dynamically (no hardcoded pins) (#107) ... ## Summary Stop hardcoding the bincraft version. Every `.crow` workflow and the build-one image pinned `@vX.Y.Z` (and a `packageVersion() != "X.Y.Z"` guard), so each bincraft release meant editing the version in ~8 places — and it was easy to miss one (the Dockerfile lagged at v4.2.1; v4.4.1 shipped without the empty-env fix because of exactly this churn). ## Change New `local/install-bincraft.R` resolves the **latest release tag dynamically**: - `git ls-remote --tags` on the public repo (no token), - keep `vX.Y.Z` tags, pick the highest version (filtered/sorted in R for portability, not via git `--sort`/refspec which behaved inconsistently under `system2()`), - `pak::pak("git::…@<latest>")` — idempotent on the git ref, so re-runs keep the package unless a newer tag exists. All call sites now invoke the helper instead of a pinned version: - `.crow/build-all-versions.yaml` (primary + per-minor pass) - `.crow/build-all-versions-install-deps.yaml` - `.crow/process-updates.yaml` (primary + per-minor pass) - `.crow/weekly-rebuild-missing.yaml` - `.crow/archive-missed-packages.yaml` - `docker/build-one.Dockerfile` (ships the helper into the image; `ensure_bincraft` sources it) ## Effect Tag a new bincraft release → the next CI run / `just rebuild` picks it up automatically. No more pin edits, and no more "forgot to bump the Dockerfile" drift. ## Verified - Resolver returns the current latest tag (`v4.4.2`) via `git ls-remote` + R-side version sort. - All five workflow YAMLs parse; helper R parses; air/editorconfig clean. Note: this tracks the latest **tag**, so cutting a release is still the deliberate gate — CI won't pick up un-tagged main. Reviewed-on: #107	2026-07-01 08:10:02 +00:00
automation-bot	2cf5714ad9	chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.9.4	2026-07-01 00:32:49 +00:00
pat-s	b4ec6291a9	chore: bump bincraft to 4.4.2	2026-06-30 16:09:44 +02:00
pat-s	d496029a79	chore: bump bincraft to 4.4.1	2026-06-30 15:51:07 +02:00
pat-s	e59590e2d6	fix: RcppParallel disable-TBB source patch (env var was a no-op) (#106) ... ## Summary Fixes the RcppParallel patch, which was a **no-op** and left the build hanging. The previous registry entry set `env: { RCPP_PARALLEL_USE_TBB: "0" }`. But `RCPP_PARALLEL_USE_TBB` is a **compile-time `-D` flag** in RcppParallel's Makevars — it is never read from the environment. So the override did nothing: `USE_TBB=Linux` (hardcoded from `uname`) still triggered the **bundled Intel TBB build**, which hangs/fails on musl (Alpine) and newer toolchains (g++ 15 on ubuntu-2604). The `Applying patch …` log only meant the env was set, not that it had any effect. ## Fix Replace the env entry with a **source patch** (`local/patches/RcppParallel/disable-tbb.patch`) on `src/Makevars.in` that, on Linux: - leaves `USE_TBB` unset → the whole bundled-TBB build/link path is skipped (no hang), and - forces `PKG_CXXFLAGS += -DRCPP_PARALLEL_USE_TBB=0` → the sources compile the **TinyThread** backend (needed because `RcppParallel.h` otherwise auto-defaults TBB on for glibc Linux). ## Verification In a Linux container, applying the patch and running `R CMD INSTALL RcppParallel`: ``` bundled_TBB_build=0 # bundled TBB build never runs * DONE (RcppParallel) # installs via TinyThread ``` ## Note bincraft's `apply_source_patch` shells out to `patch`. If a build-env image lacks the `patch` tool (common on Alpine), the patch will report "did not apply cleanly" and fall back to an unpatched (hanging) build. If that happens, the follow-up is to switch bincraft's patch application to `git apply` (git is always present) — happy to do that if needed. Reviewed-on: #106	2026-06-30 13:27:01 +00:00
pat-s	b93d1ef6bd	chore: silence otelsdk warnings across build paths (#105) ... ## Summary Silences the recurring `OpenTelemetry error: there is no package called 'otelsdk'` warnings during builds. ## Root cause The `build-env-*` images configure an OTel exporter (traces/logs/metrics), but `otelsdk` (the R OTel SDK backend) isn't installed. pak's `otel` instrumentation therefore tries to load `otelsdk` on every run and logs the error, falling back to a no-op. `OTEL_SDK_DISABLED=true` (already set in `build-one.Dockerfile`) does **not** help — the R `otel` package ignores it and gates purely on `OTEL_R_<SIGNAL>_EXPORTER` (then the standard `OTEL_<SIGNAL>_EXPORTER`). When that resolves to a real exporter (`otlp`/`http`/…) with no SDK present, you get the error. ## Fix Set `OTEL_R_TRACES_EXPORTER`, `OTEL_R_LOGS_EXPORTER`, and `OTEL_R_METRICS_EXPORTER` to `none` so the R otel providers are clean no-ops: - `docker/build-one.Dockerfile` — exported alongside the existing OTel var. - `.crow/process-updates.yaml`, `weekly-rebuild-missing.yaml`, `build-all-versions.yaml`, `build-all-versions-install-deps.yaml` — added to each step's `environment` block. Using the R-specific variables (not the standard `OTEL_*_EXPORTER`) keeps OTel intact for any non-R tooling in the images. Reviewed-on: #105	2026-06-30 12:41:32 +00:00
pat-s	b744ae1bb1	fix: build-one image bincraft v4.4.0 + ship patch registry (#104) ... ## Summary Fixes the **targeted rebuild** path (`just rebuild` → `docker/build-one.Dockerfile`) so it actually applies patches. Two problems, both of which would make `build-one.R`'s `patches = "local/patches"` a silent no-op or a hard error: 1. **Stale bincraft pin.** The Dockerfile pinned bincraft `v4.2.1`, which predates the `patches` argument — so `build_binary_package(patches = ...)` would fail with `unused argument`. Bumped to **v4.4.0** to match the `.crow` workflows. 2. **Registry not shipped into the image.** Only `build-one.R` was copied in; `local/patches/registry.json` was absent, so `patches = "local/patches"` resolved to a nonexistent `/work/local/patches` and silently applied nothing. Added `COPY patches /work/local/patches` (build context is `local/`, CWD is `/work`). ## Why This is the path used to verify the patching end-to-end, e.g.: ```bash just rebuild alpine 3.23 amd64 rts2 1.0.3 ``` rts2 depends on RcppParallel; with this fix the container installs bincraft v4.4.0, ships the registry, and the patched RcppParallel binary (`RCPP_PARALLEL_USE_TBB=0`) is served to pak during dependency install. Expect `Applying patch to RcppParallel …` in the log, RcppParallel installed as a binary (no recompile), then rts2 building and uploading. Reviewed-on: #104	2026-06-30 12:15:39 +00:00
pat-s	00db47398d	chore: 4.4.0 instead of 4.3.1	2026-06-30 14:07:22 +02:00
pat-s	55a18fd87d	feat: patch registry + wiring for per-package patching (#103) ... ## Summary Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`). Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading. ## What's included - `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs. - `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace). - `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes). - `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`. - `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`. ## ⚠️ Merge ordering (blocker) This PR adds a `patches = ...` argument to `build_binary_package()` calls. The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until: 1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and 2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`. The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then. Reviewed-on: #103	2026-06-30 08:50:19 +00:00
automation-bot	1e910bc703	chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.9.1	2026-06-29 00:31:40 +00:00
automation-bot	597f759bb2	chore(deps): update pre-commit hook editorconfig-checker/editorconfig-checker to v3.8.0	2026-06-29 00:31:27 +00:00
automation-bot	3ffa25ddac	chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.8.5	2026-06-28 00:32:21 +00:00
automation-bot	81ca78edb6	chore(deps): update terraform bunnynet to ~> 0.15	2026-06-23 00:31:42 +00:00
pat-s	a1d7064fb8	feat(ci): add process-updates rows for alpine-324 and ubuntu resolute ... Add matrix rows and manual dropdown options for alpine 3.24 (alpine-324, PROCESS_NEW=FALSE) and ubuntu resolute / 26.04 (ubuntu-2604, OS_ID resolute, R 4.5.3, PROCESS_NEW=TRUE).	2026-06-21 11:04:24 +02:00
pat-s	b46472b59e	fix(ci): pass Backblaze B2 credentials to process_unarchived_pkgs ... The function defaults to Hetzner S3 with HETZNER_S3_*_K3S env vars that are not set in these workflows, so paws failed with "No compatible credentials provided". Pass the B2 endpoint/region/bucket and B2_S3_* secrets explicitly, matching the other S3 calls. Also use set_codename(NULL) for the build-all-versions codename so it matches the S3 repo path.	2026-06-21 10:55:22 +02:00
automation-bot	a6788f9044	chore(deps): update pre-commit hook posit-dev/air-pre-commit to v0.10.0	2026-06-20 00:31:34 +00:00
pat-s	3420523849	chore: add all options for OS_VERSION	2026-06-18 13:40:13 +02:00
pat-s	83c761b736	feat(ci): OS/OS_VERSION manual dropdowns + restore lost crow fix (#97) ... ## Summary Two changes: 1. **`OS`/`OS_VERSION` manual-run dropdowns** — give these form variables explicit `options:` lists (like `target_arch` and `R_VERSION`), so the manual-run form shows dropdowns instead of free-text, in both `build-all-versions.yaml` and `build-all-versions-install-deps.yaml`. Crow form variables are independent (no cascading), so the operator still has to pick a coherent `OS` + `OS_VERSION` combination (e.g. `redhat` + `9`, not `alpine` + `jammy`). 2. **Repairs `main`** — the crow fix from PR #96 (`bc2f6f1`) was lost when that PR was squashed (only the first commit was captured). As a result `main` currently carries the `OS: ${OS}` env vars that break Crow parsing (`unable to parse variable name`) and the unfixed `build-all.R`. This PR re-applies that fix: drop the env additions and derive `platform`/`arch` inside `build-all.R` from the container (bincraft codename → platform mapping + `Sys.info()` arch). ## Notes - `OS_VERSION` options are quoted strings so tags like `8`/`9`/`10` aren't parsed as integers. - Validated: both YAMLs parse, `build-all.R` parses. Reviewed-on: #97	2026-06-18 11:39:11 +00:00
pat-s	0f4330c05c	fix(ci): make build-all self-sufficient on agents without install-deps (#96) ... ## Summary Fixes the recurring `build-all-*` failure on agents where `build-all-versions-install-deps` did **not** run. Both symptoms in the logs — `Parameter 3 does not have length 1` (repeated, in the metadata DB queries) and `argument is of length zero` (system-dependency install) — were the same bug: inside bincraft, `platform` was zero-length. `local/build-all.R` calls `build_binary_package()` without passing `platform`, so bincraft resolves it from the container codename. The primary build step never (re)installed/pinned bincraft and relied on whatever sat in the **per-agent** cache volume; `depends_on` only orders steps, it does not co-locate them on the same agent, so a job landing where install-deps never ran got a stale bincraft that left `platform` empty. ## Changes - **`build-all-versions.yaml`**: pin bincraft `@v4.2.3` in the primary build step (mirroring the R-minor pass and `process-updates.yaml`), so every agent uses a known-good bincraft regardless of where install-deps ran. - Align the R-minor pass `v4.2.2 → v4.2.3`. - Export `OS`/`OS_VERSION`/`ARCH` as runtime env vars — previously only available for `${...}` interpolation, so `build-all.R`'s already-built dedup query matched platform `"-"` and skipped nothing. - Fix the unarchive call: pass the codename via `bincraft::set_codename(NULL)` instead of the malformed `paste(OS, OS_VERSION)` (`"alpine 3.24"`), matching `archive-missed-packages.yaml`. - **`build-all-versions-install-deps.yaml`**: pin install-deps to `@v4.2.3` (was installing HEAD), so the precomputed snapshot and per-agent library stay consistent pipeline-wide. Reviewed-on: #96	2026-06-18 11:27:02 +00:00
pat-s	f9bc72d5ab	chore: bump bincraft to 4.2.3	2026-06-18 09:43:05 +02:00
pat-s	fa955c10bf	fix: recompute package snapshot when missing from cache (#95) ... ## Summary `build-all.R` reads three snapshot files from `/mnt/cache/packages/` that the `build-all-versions-install-deps` step precomputes: `pkgs_to_build.rds`, `r_minor_sensitive_pkgs.rds`, and `s3_cache.rds`. That cache volume is **per-agent**, so a build job scheduled on a different (fresh) agent than the one that ran install-deps finds the snapshot absent and dies at `readRDS` (`cannot open compressed file '/mnt/cache/packages/pkgs_to_build.rds'`). This adds a conditional guard at the top of `build-all.R`: when any of the three files is missing, it sources `local/packages-to-build.R` (which has all needed creds via `PGPASS` / `B2_S3_*` env, already present in the build step) and writes the derived `.rds` files — exactly mirroring the install-deps command. - The first build job on a fresh agent repopulates the shared cache, so subsequent jobs on that agent reuse it. - Concurrent jobs that also miss simply redo the work (accepted tradeoff vs. slow shared storage like NFS). - Saves use a temp-file + atomic `file.rename`, so a concurrent reader never sees a half-written `.rds`. Reviewed-on: #95	2026-06-17 17:49:04 +00:00
pat-s	4c67703f52	refactor(ci): route workflows by agent group label instead of named agent (#94) ... ## Summary - Switch crow workflow placement from a per-agent label (`agent: ${AGENT}`) to a group label (`group: rpkgs-${ARCH}`), so jobs target the `rpkgs-amd64` / `rpkgs-arm64` agent pools instead of a single named agent (`artemis`/`gaia`). - Remove the now-unused `AGENT` matrix variable from all build/process/weekly workflows. - Update header comments to reflect group-based placement. Agents opt into a pool via `CROW_AGENT_LABELS="group=rpkgs-<arch>"`. The group name derives from each matrix row's `ARCH`, so adding or replacing agents no longer requires touching the workflow files. Reviewed-on: #94	2026-06-17 16:26:04 +00:00
pat-s	3875612237	fix(ci): restore xwfb-run/xvfb-run detection for rhel-10 process-updates ... The matrix consolidation hardcoded `xvfb-run`, which is absent on the rhel-10 image (it ships Xwayland's xwfb-run), causing `xvfb-run: command not found`. Re-add the per-platform launcher detection the old rhel-10 files used: prefer xwfb-run + weston, fall back to xvfb-run.	2026-06-17 12:47:04 +02:00
pat-s	76f677ba5b	fix(ci): quote matrix PROCESS_NEW so R receives uppercase FALSE/TRUE ... Unquoted `PROCESS_NEW: FALSE` is parsed as a YAML boolean and interpolated into the R command as lowercase `false`, causing `process_new = false` and `Error: object 'false' not found`. Quoting keeps the literal FALSE/TRUE that R expects.	2026-06-17 11:35:44 +02:00
pat-s	dc92bd6a2c	fix(ci): restore cron triggers via per-row cron name filter ... The matrix consolidation routed cron events through `evaluate: 'CI_PIPELINE_CRON == "..."'`, but CI_PIPELINE_CRON is not exposed to the when.evaluate constraint context, so no matrix row ever matched and the scheduled process-updates / weekly runs stopped firing. Switch back to the first-class `cron:` name filter (still parameterized per matrix row) which the server matches directly against the fired cron.	2026-06-17 11:03:12 +02:00
pat-s	b297f0fbc0	refactor(ci): consolidate build-all-versions workflows into 2 files (#93) ... ## Summary Consolidate the four `build-all-versions*` files into two, mirroring the matrix approach used for the process-updates / weekly families: | Before | After | |---|---| | `build-all-versions-amd64.yaml` + `build-all-versions-arm64.yaml` | `build-all-versions.yaml` | | `build-all-versions-install-deps-amd64.yaml` + `-arm64.yaml` | `build-all-versions-install-deps.yaml` | `ARCH` becomes a matrix axis carrying the per-arch differences: `agent` (artemis/gaia), `BACKEND` (kubernetes/docker), and `R_PKG_CACHE_DIR`. Build-all keeps its `SPLIT_INDEX` parallelism, so its matrix is ARCH x SPLIT = 8 rows; only the rows matching the selected arch run. **Invocation unchanged.** Routing uses the existing task selector `task == "build-all-${ARCH}"`, so `--var task=build-all-amd64 ...` still runs exactly the amd64 rows. `OS` / `OS_VERSION` / `R_VERSION` / `K8S_INSTANCE_TYPE` are still passed via `--var`. (You can drop `--var ARCH=...` now — arch comes from the matrix/task.) **No cron.** These pipelines are manual-only, so there are no scheduled runs to affect. ## Validation before merge Same matrix-interpolation-in-`when`/`labels` mechanism validated for #92, plus it now relies on per-row `agent`/`BACKEND` selecting the right execution backend (kubernetes for amd64, docker for arm64). Before merge, on this branch trigger a small `task=build-all-arm64` run and confirm it lands on the gaia/docker backend (and amd64 on artemis/kubernetes). Originals are in git history for rollback. Note: `archive-missed-packages.yaml` was already a single consolidated matrix file (cron + manual) and is left untouched. Reviewed-on: #93	2026-06-16 09:33:13 +00:00
pat-s	4406a8735b	refactor(ci): consolidate per-platform crow workflows into 3 matrix files (#92) ... ## Summary Collapse **44** per-platform crow pipeline files into **3** matrix-driven files (one per family), using `matrix.include` + the crow #1165 declarative manual `variables:` block: | Family | Before | After | |---|---|---| | process-updates | 14 | `process-updates.yaml` | | weekly-audit-missing | 16 | `weekly-audit-missing.yaml` | | weekly-rebuild-missing | 14 | `weekly-rebuild-missing.yaml` | - **One file per family** (not split by arch). Arch placement is via the agent label as a matrix var (`artemis`=amd64, `gaia`=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-`nodeSelector` placement (no agent label). - **Matrix axis is named `OS`** (e.g. `redhat-9`) per request, with per-row `R_VERSION`, image tag, codename, and `process_new` (FALSE for alpine). - **Runs preserved 1:1:** - *Cron*: each existing `<family>-<os>-<arch>` cron still fires only its matching matrix row via `CI_PIPELINE_CRON`. Server-side cron entries unchanged. - *Manual*: a #1165 dropdown variable (`process_cran_updates` / `weekly_audit_missing` / `weekly_rebuild_missing`) selects a single `<os>-<arch>` or `all`. weekly-rebuild defaults to `all`, matching its former bare `event: manual` trigger. ## Behavior deltas (intentional, flagged) To fit one file per family the backend was unified: - The amd64-only `node.kubernetes.io/instance-type: AX42` nodeSelector pin is **dropped**; placement now relies on the `artemis`/`gaia` agents. - **process-updates arm64** jobs now share the same `backend_options` as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped. ## Validation required before merge Relies on crow interpolating matrix variables inside `when.evaluate`, `labels.agent`, and `commands`. Interpolation in image/env/commands is standard crow; the novel bit is `when.evaluate`. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch: 1. Trigger manually with the dropdown = `redhat-9-amd64` -> confirm only that one job runs (image `build-env-redhat:9`, R 4.4.3, agent artemis). 2. Trigger with `all` -> confirm all os/arch jobs schedule. Crons only fire on the default branch, so no collision while unmerged. Fallback if `${...}` doesn't interpolate in `when`: a runtime `case "$CI_PIPELINE_CRON" in ...` guard, same files otherwise. Reviewed-on: #92	2026-06-16 09:33:06 +00:00