Author	SHA1	Message	Date
pat-s	930a4acd12	fix(ci): install the R packages pipelines use instead of relying on the image ... bincraft v5.5.1 dropped lubridate, dplyr, purrr, data.table and others from its imports, so the pipelines' own calls now depend on whatever the build-env image's site library or the shared R_LIBS_USER cache happens to hold. The arm64 redhat:9 image has no progressr under R 4.4.3, which fails every process-cran-updates-redhat-9-arm64 run since v5.5.1. - Install progressr and lubridate for process-updates, which call them inline. - Install purrr for packages-to-build.R and data.table for the weekly audit. - Drop the unused dplyr and progressr library() calls from packages-to-build.R. - Install RPostgres in every pipeline that talks to the metadata DB, since bincraft silently skips the DB (and the per-minor index aborts) without it. - Reinstall bincraft in the prune re-index step, whose /tmp library does not survive from the prune step.	2026-10-20 08:46:00 +00:00
pat-s	416a2d64d9	chore(alpine): drop the deprecated Alpine 3.22 slot (#213) ... ## Why Alpine 3.22 is deprecated and should be dropped. It was never a published slot to begin with. `local.rpkgs_slots` in `cdn.tf` enumerates eight distros and alpine322 is not one of them, so `UNION_SLOTS` never gated per-minor rewriting for it. The website's `DistroId` union lists the same eight. Every request to the slot therefore fell through to the flat index, which holds R 4.5 builds. ## Evidence The fleet-wide routing verification (pipeline 12591, `verify_routing=all`) measured this end to end. Across all 16 published slots, both architectures, three R minors and both contractual URL forms: **1056 of 1056 install-and-load checks passed.** Every failure in the run was alpine322, identical on amd64 and arm64: ``` amd64/alpine322 R 4.4 via arch|generic|flat: 8 of 11 amd64/alpine322 R 4.6 via arch|generic|flat: 10 of 11 arm64/alpine322 R 4.4 via arch|generic|flat: 8 of 11 arm64/alpine322 R 4.6 via arch|generic|flat: 10 of 11 ``` The tell is that all three forms agree. On a routed slot `arch` and `generic` diverge from `flat`, because that divergence *is* the routing. On alpine322 there is nothing to diverge from. R 4.5 clients were 11 of 11 on every form, because flat happens to be what they need. The data itself is not broken, only unreachable. `alpine322/contrib/4.4/data.table_1.18.6.1.tar.gz` is genuinely `Built: R 4.4.3` and the 4.6 object is `Built: R 4.6.0`. Correct objects, correct per-minor indexes, never served. ## What changed - alpine-322 matrix rows and dropdown options removed from the 12 pipelines that carried them, so `=all` no longer fans out to a slot nobody can reach. - `3.22` removed from the `OS_VERSION` dropdowns in `build-all-versions.yaml` and `repair-built-stamp.yaml`; the latter's default moves to `3.23`. - Stale `3.22` examples updated in `local/packages-to-build.R`. `crow lint .crow/` reports all 18 configs valid. ## Also done outside this diff The six alpine-322 cron jobs are deleted from the Crow server, since a cron whose matrix row is gone keeps firing against nothing: ``` 5 process-cran-updates-alpine-322-amd64 0 5/8 * * * 6 process-cran-updates-alpine-322-arm64 30 5/8 * * * 68 weekly-audit-missing-alpine-322-amd64 20 0 * * 0 69 weekly-audit-missing-alpine-322-arm64 30 0 * * 0 82 weekly-rebuild-missing-alpine-322-amd64 0 4 * * 0 83 weekly-rebuild-missing-alpine-322-arm64 10 4 * * 0 ``` They are listed here so they can be recreated if this is ever reversed. ## Not done The published objects and indexes under `amd64/alpine322` and `arm64/alpine322` are left in the bucket. Deleting them is a separate decision, and nothing routes to them in the meantime. Reviewed-on: #213	2026-09-11 07:27:03 +00:00
pat-s	77a2f1f04a	fix(ci): install RPostgres for audit workflows (#172) ... ## Summary - Install RPostgres before running the missing-binaries audit. - Install RPostgres before running weekly patch proposal and automatic patch workflows. ## Validation - `prek run --files .crow/auto-apply-patches.yaml .crow/weekly-audit-missing.yaml .crow/weekly-patch-proposals.yaml` - `crow lint .crow/` - `git diff --check` Reviewed-on: #172	2026-08-30 07:31:28 +00:00
pat-s	132d1d2d3c	docs(ci): clarify parallel manual matrix runs	2026-08-14 06:44:10 +00:00
pat-s	f8e31af75b	fix(ci): make every manual gate default to a value that matches nothing (#158) ... ## Problem A manual `crow pipeline create` instantiates **every** file in `.crow/`, and a declared variable default is applied even when the run never passed that variable. A gate is therefore only a gate if its default matches nothing. #155 fixed the three pipelines that had no manual gate at all. It missed that a *permissive default* leaves a pipeline just as exposed. Demonstrated the expensive way: creating a pipeline with only ``` --var weekly_audit_missing=alpine-324-amd64 ``` also started `build-all-versions` — because its gate `target_arch` defaults to `amd64`, which matches its own amd64 matrix rows — and `process-updates` across every row, because that gate defaults to `all`. The run was killed before any `Upload package indexes` step produced output and both alpine324 indices were verified unchanged, but `build-all-versions` uploads binaries and rewrites indexes, so the next one might not be caught in time. Before: | pipeline | gate | default | fired on an unrelated manual run | | --- | --- | --- | --- | | `build-all-versions` | `target_arch` | `amd64` | amd64 rows — builds and uploads | | `build-all-versions-install-deps` | `target_arch` | `amd64` | amd64 rows | | `weekly-rebuild-missing` | `weekly_rebuild_missing` | `all` | every row | | `weekly-audit-missing` | `weekly_audit_missing` | `all` | every row | | `process-updates` | `process_cran_updates` | `all` | every row | | `repair-built-stamp` | `repair_built_stamp` | `arm64` | arm64 rows | `archive-missed-packages` was the one that behaved, because its gate variable is never declared and so matches nothing. That is the property this restores everywhere. ## What this changes Each of the six gets a `none` option on its gate variable and defaults to it, so a manual run has to name its target explicitly. The reason is recorded next to the default, where someone would go to change it. `none` is used rather than dropping the default so the expression always has a defined value to compare, instead of relying on undefined-variable semantics. Cron triggers are untouched — they match on the `cron:` name, not the variable. ## Verification `crow lint .crow/` reports all ten configs valid. Auditing every pipeline that accepts a manual event: ``` archive-missed-packages.yaml: gate=task default=<none> auto-apply-patches.yaml: gate=auto_apply_patches default='false' build-all-versions-install-deps.yaml gate=target_arch default=none build-all-versions.yaml: gate=target_arch default=none process-updates.yaml: gate=process_cran_updates default=none repair-built-stamp.yaml: gate=repair_built_stamp default=none trial-build-registry.yaml: gate=trial_build_registry default='false' weekly-audit-missing.yaml: gate=weekly_audit_missing default=none weekly-patch-proposals.yaml: gate=weekly_patch_proposals default='false' weekly-rebuild-missing.yaml: gate=weekly_rebuild_missing default=none ``` Every gate now defaults to something that matches no matrix row. Reviewed-on: #158	2026-08-09 15:31:14 +00:00
pat-s	ee15c50f53	refactor: migrate package installation from pak to uvr (#147) ... `bincraft` dropped pak in favour of uvr (5.0.x, "Dependencies and their system requirements are now installed with `uvr` instead of pak during `build_binary_package()`"), so the pipelines, helper scripts and images in this repo move with it. ## Approach uvr is project-scoped in a way pak is not: `uvr add` refuses to run outside a project and always installs into `.uvr/library/`, and only `uvr sync` honours `--library`. So there is no one-line `pak::pak(...)` equivalent. `local/uvr-install.sh` encapsulates the dance — bootstrap a pinned uvr, mint a throwaway project under `TMPDIR`, `uvr add --no-install`, then `uvr sync --library <target>`. Keeping the project outside the checkout also keeps `uvr init`'s `.Rprofile` from hijacking `.libPaths()` for every other R call in the pipeline. This matches what bincraft itself does (`uvr sync --install-system-deps --library <lib>`), and bincraft requires `uvr` on `PATH`, which the bootstrap provides: every pipeline that calls `bincraft::` runs `install-bincraft.R` (and therefore the bootstrap) first. ## Changes | File | Change | | --- | --- | | `local/uvr-install.sh` | **New.** The single replacement for `pak::pak(...)`. Bootstraps uvr `v0.4.4`, resolves the R interpreter from `UVR_R_BIN`/`R_VERSION`/`PATH`, pins the manifest to that R's exact version, and syncs into `UVR_TARGET_LIB`/`R_LIBS_USER`. | | `local/install-bincraft.R` | Installs `forgejo::codefloe.com/rpkgs/bincraft@<tag>` instead of a `git::` URL; keeps the `git ls-remote` tag resolution. Exports `UVR_R_BIN`/`UVR_TARGET_LIB` from `R.home()`/`.libPaths()[1]` so the per-R-minor passes target their own R and library. | | `.crow/auto-apply-patches.yaml`, `.crow/weekly-patch-proposals.yaml`, `.crow/weekly-audit-missing.yaml`, `.crow/weekly-rebuild-missing.yaml`, `.crow/build-all-versions-install-deps.yaml` | `pak::pak(...)` → `UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh ...`. The explicit `UVR_R_BIN` matters in `build-all-versions-install-deps.yaml`, which has no `R_VERSION` in its step environment. | | `.crow/build-all-versions.yaml`, `.crow/process-updates.yaml`, `.crow/weekly-rebuild-missing.yaml` | `R_PKG_CACHE_DIR` → `UVR_CACHE_DIR` + `UVR_PACKAGES_DIR` on the same `/mnt/cache` volume, preserving the amd64-off/arm64-on split. Drops the `rm -rf .../pkgcache/_metadata/...` cleanup. | | `local/r-minor-helpers.R`, `local/build-all.R`, `local/tests/test-trim-pkgcache.R` | Removes `trim_pkgcache_metadata()`, its every-25-packages call and its tests. uvr's cache does not mint a fresh ~70 MB snapshot per `PACKAGES` change. | | `.crow/build-all-versions-install-deps.yaml` | Drops `pak::sysreqs_db_update()`; uvr resolves sysreqs from its vendored `r-system-requirements` rules via `--install-system-deps`. | | `docker/Containerfile-shiny-app` | Bootstraps uvr and drives both dependency installs through one uvr project with `UVR_LIBRARY` pointed at the image's R library. | | `docker/build-one.Dockerfile` | Ships `uvr-install.sh` at `/work/local/` so `install-bincraft.R` finds it. | | `docker/reprex/alpine.sh` | Replaces `pak::local_install_deps()` with DESCRIPTION parsing + `uvr add`. | | `local/test-package-loading.R` | Installs via the helper instead of `pak::pkg_install()`. | | `README.md` | Documents uvr for sysreq inference, archived-version installs and cache clearing. | | `renovate.json` | Tracks the `UVR_PIN` in `uvr-install.sh` via `github-releases`. | ## Behaviour notes - **`weekly-audit-missing` still takes bincraft from the default branch**, not the latest release tag, matching what the `git::` pak call did. Called out in a comment rather than silently changed. - **The uvr pin is repo-wide.** bincraft resolves `uvr` from `PATH` and pins no version of its own, so `UVR_PIN` in `uvr-install.sh` governs the whole pipeline. - **Persistent caches now also benefit bincraft**, which reads `UVR_CACHE_DIR`/`UVR_PACKAGES_DIR` from the inherited pipeline environment. - **`uvr sync` will not prune the shared library.** Pruning is disabled whenever `--library` is passed (`do_prune = prune && library_override.is_none()`), so `/mnt/cache/R-pkgs` keeps bincraft and its dependencies. The wipe-on-ABI-mismatch path is *not* similarly guarded, which is why the helper pins the manifest to the active R's exact version. - **`plans/` and `specs/` are untouched** — they are dated records of decisions made in June 2026 and describe bincraft's then-pak-based internals; rewriting them would misstate history. ## Verification `shellcheck`, all pre-commit hooks (on this commit's file range) and the `local/tests/` suite (100 assertions) pass. Not yet exercised in CI: the build-env images do not ship `uvr`, so the per-step `curl install.sh` bootstrap is untested against a real image. Worth a manual `build-all-versions-install-deps` run before merging. Reviewed-on: #147	2026-07-31 12:19:26 +00:00
pat-s	61e1c0188d	chore: adjust weekly audit runs	2026-07-20 19:09:50 +02:00
pat-s	4c67703f52	refactor(ci): route workflows by agent group label instead of named agent (#94) ... ## Summary - Switch crow workflow placement from a per-agent label (`agent: ${AGENT}`) to a group label (`group: rpkgs-${ARCH}`), so jobs target the `rpkgs-amd64` / `rpkgs-arm64` agent pools instead of a single named agent (`artemis`/`gaia`). - Remove the now-unused `AGENT` matrix variable from all build/process/weekly workflows. - Update header comments to reflect group-based placement. Agents opt into a pool via `CROW_AGENT_LABELS="group=rpkgs-<arch>"`. The group name derives from each matrix row's `ARCH`, so adding or replacing agents no longer requires touching the workflow files. Reviewed-on: #94	2026-06-17 16:26:04 +00:00
pat-s	dc92bd6a2c	fix(ci): restore cron triggers via per-row cron name filter ... The matrix consolidation routed cron events through `evaluate: 'CI_PIPELINE_CRON == "..."'`, but CI_PIPELINE_CRON is not exposed to the when.evaluate constraint context, so no matrix row ever matched and the scheduled process-updates / weekly runs stopped firing. Switch back to the first-class `cron:` name filter (still parameterized per matrix row) which the server matches directly against the fired cron.	2026-06-17 11:03:12 +02:00
pat-s	4406a8735b	refactor(ci): consolidate per-platform crow workflows into 3 matrix files (#92) ... ## Summary Collapse **44** per-platform crow pipeline files into **3** matrix-driven files (one per family), using `matrix.include` + the crow #1165 declarative manual `variables:` block: | Family | Before | After | |---|---|---| | process-updates | 14 | `process-updates.yaml` | | weekly-audit-missing | 16 | `weekly-audit-missing.yaml` | | weekly-rebuild-missing | 14 | `weekly-rebuild-missing.yaml` | - **One file per family** (not split by arch). Arch placement is via the agent label as a matrix var (`artemis`=amd64, `gaia`=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-`nodeSelector` placement (no agent label). - **Matrix axis is named `OS`** (e.g. `redhat-9`) per request, with per-row `R_VERSION`, image tag, codename, and `process_new` (FALSE for alpine). - **Runs preserved 1:1:** - *Cron*: each existing `<family>-<os>-<arch>` cron still fires only its matching matrix row via `CI_PIPELINE_CRON`. Server-side cron entries unchanged. - *Manual*: a #1165 dropdown variable (`process_cran_updates` / `weekly_audit_missing` / `weekly_rebuild_missing`) selects a single `<os>-<arch>` or `all`. weekly-rebuild defaults to `all`, matching its former bare `event: manual` trigger. ## Behavior deltas (intentional, flagged) To fit one file per family the backend was unified: - The amd64-only `node.kubernetes.io/instance-type: AX42` nodeSelector pin is **dropped**; placement now relies on the `artemis`/`gaia` agents. - **process-updates arm64** jobs now share the same `backend_options` as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped. ## Validation required before merge Relies on crow interpolating matrix variables inside `when.evaluate`, `labels.agent`, and `commands`. Interpolation in image/env/commands is standard crow; the novel bit is `when.evaluate`. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch: 1. Trigger manually with the dropdown = `redhat-9-amd64` -> confirm only that one job runs (image `build-env-redhat:9`, R 4.4.3, agent artemis). 2. Trigger with `all` -> confirm all os/arch jobs schedule. Crons only fire on the default branch, so no collision while unmerged. Fallback if `${...}` doesn't interpolate in `when`: a runtime `case "$CI_PIPELINE_CRON" in ...` guard, same files otherwise. Reviewed-on: #92	2026-06-16 09:33:06 +00:00