Commit graph build-cran-binaries/.crow/weekly-rebuild-missing.yaml
Author SHA1 Message Date
4b7dc28cc8 feat(rebuild): shard the weekly rebuild and make each shard resumable (#163)
Some checks failed
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/weekly-audit-missing/6 Pipeline was successful
ci/crow/cron/weekly-audit-missing/5 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/18 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/17 Pipeline was successful
ci/crow/manual/weekly-rebuild-reindex/6 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/51 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/14 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/manual/weekly-rebuild-reindex/5 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/49 Pipeline was successful
ci/crow/manual/weekly-rebuild-reindex/17 Pipeline failed
ci/crow/manual/weekly-rebuild-missing/50 Pipeline was successful
## Problem

`weekly-rebuild-missing` runs one job per `<os>-<arch>` and walks that slot's list serially in a single `R -q -e` argument.
That was cheap while every source fallback was skipped as "already built".
Since bincraft #105/#106/#107 and #159 the gate works, and the lists are large: 8 917 source-served records on `amd64/alpine324`, 15 023 on `amd64/resolute`.

Pipeline 10910 (`weekly_rebuild_missing:alpine-324-amd64`) ran for two days, reached `[8692/23885] cholera`, and was killed there.

Two failures follow from that shape:

- **No parallelism.** The work is embarrassingly parallel across packages; one job does all of it.
- **No resumability and no clean stopping point.** The loop ends only by exhausting the list, so the only way to stop it is a kill. A restart re-walks from the first entry, paying a CRAN version resolution and an S3 `HEAD` per package before reaching new work. And a kill matches neither `success` nor `failure`, so the `Purge CDN cache` step never ran: the ~4 600 binaries 10910 did publish stayed hidden behind stale edge copies.

## What this changes

**Three shards per slot.** Each of the 18 `OS`/`ARCH` rows gains `SPLIT_INTO`/`SPLIT_INDEX`, mirroring `build-all-versions.yaml`. Cron and manual routing are unchanged: both filters already match on `${OS}-${ARCH}`, so they now match all three shards of a slot.

**`local/rebuild-missing.R`** replaces the ~1 500-character inline one-liner. The slice is interleaved rather than contiguous, because the list is alphabetical and cost clusters by name (`Rcpp*`, `Bioc*`, `rstan*`).

**Resume by re-deriving state from the bucket.** One `s3_dir_info()` listing gives ETags for the slot; a package is outstanding iff its object's ETag equals CRAN's published `MD5sum`, i.e. it is still byte-identical to CRAN's source. That is `check_s3_root_package()` evaluated in bulk. No progress file, no volume, no DB cursor, and correct when a sibling shard or a `process-updates` run completes something concurrently.

It reads ETags rather than the index's `Built` field the way `packages-to-build.R` does, because the index is no longer rewritten until the dependent pipeline runs and so cannot reflect the current run's progress.

Unknown always means "already a binary", never "rebuild it": a multipart ETag, an unreadable CRAN index or an empty listing can never mass-schedule work.

**A 20 h wall-clock budget** per shard. It exits 0, so the re-index and purge always fire and the remainder is picked up next run with no bookkeeping.

**`.crow/weekly-rebuild-reindex.yaml`** takes over re-indexing and the purge, with `depends_on: [weekly-rebuild-missing]` and `runs_on: [success, failure]`. Three shards writing one slot's `PACKAGES` concurrently would race: `update_PACKAGES()` lists the live bucket, so an early lister that uploads last publishes an index missing its siblings' work.

## Verification

`crow lint .crow/` passes on all 11 pipelines. `prek run` passes.

19 assertions in `local/tests/test-rebuild-missing.R`, 0 failures, covering the partition (disjoint, covering, deterministic, short lists, out-of-range index) and the outstanding filter (source ETag kept, binary ETag dropped, absent object kept, multipart and missing-from-CRAN treated as built).

One of those tests caught a real bug before it shipped: an empty ETag table indexed to zero length rather than to `NA`, which recycled the result away and reported "nothing to build" — the dangerous direction. Fixed with an explicit `lookup()`.

The filter run against the live `amd64/alpine324` index, using its `MD5sum` column as the ETag (established to match the objects):

```
index packages:               24343
outstanding (filter):          8950
no Built stamp:                8917
filter vs no-Built agreement:  8917 of 8917
outstanding but stamped Built:   33 (version drift vs CRAN)
shard sizes: 2984/2983/2983 (sum 8950, unique 8950)
```

It reproduces the source-served set exactly. The extra 33 are packages whose slot version differs from CRAN's current one, so no object exists at the CRAN version key: correctly outstanding.

## Notes for review

- The 20 h budget is a chosen default, exposed as `REBUILD_BUDGET_HOURS` in the pipeline.
- `depends_on` is file-level, not row-level, so on a full cron run no slot is re-indexed until the slowest of all 54 jobs finishes. The budget bounds that at roughly a day.
- An explicit cancel still skips the re-index. Recovery is to trigger `weekly-rebuild-reindex` on its own.
- The purge runs on every re-index row rather than one designated slot: a cron fires only its own slot's row, so gating on a named slot would leave every other slot unpurged.
- Out of scope: `build-all-versions` still cannot rebuild source fallbacks, because `local/build-all.R:113-122` drops every version with any `single_builds` row, which is precisely the source-fallback set.

Design: `specs/2026-08-12-shard-weekly-rebuild-design.md`
Reviewed-on: #163
2026-08-12 08:30:29 +00:00
01b8ab43df fix(rebuild): re-index and purge the CDN after a rebuild (#160)
Some checks failed
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline is running
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/weekly-audit-missing/6 Pipeline was successful
ci/crow/manual/weekly-audit-missing/5 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/5 Pipeline was canceled
## Problem

The rebuild now works — `AATtools 0.0.3` was detected as a source fallback, built, and published:

```
ℹ `upload_single_binary()`: Replacing the CRAN source published for AATtools 0.0.3 … with the binary.
✔ Successfully uploaded package AATtools with tag 0.0.3.
```

But clients still get the source, and will for about a year:

```
$ curl -sI .../src/contrib/AATtools_0.0.3.tar.gz
etag: "ea8127d953ca6a2f118ea49441772af6"   # CRAN's source MD5
cdn-cache: HIT
cdn-cachedat: 08/09/2026 16:43:32          # predates the 18:12 upload
```

Two causes, both specific to a rebuild:

1. **The slot is never re-indexed.** `weekly-rebuild-missing` has no `upload_package_index` step, so the index keeps the old MD5 and — for anything that had been served from source — no `Built` stamp. This one self-heals at the next `process-updates` run.
2. **The tarball URL is never purged.** A normal update publishes new packages at *new* URLs, so `purge_cdn_cache.sh` only needs the five index files. A rebuild replaces an object *in place*, and the zone caches tarballs for `cache_expiration_time = 31919000` (~370 days). This does not self-heal.

Nothing about a stale package looks wrong from the outside, which is what makes it worth fixing rather than documenting.

## What this changes

**Re-index at the end of a rebuild**, flat and per-minor, mirroring the tail of `process-updates`. The codename is detected from the image's `/etc/os-release` (as `local/packages-to-build.R` already does) rather than adding `OS_ID` to all 18 matrix rows.

**Purge the zone afterwards**, via a new `scripts/purge_cdn_zone.sh`. One call to `POST /pullzone/{id}/purgeCache` covers every replaced object, and all three hostnames — `cran.devxy.io`, `cran.allianceswisspass.devxy.io`, `cran.rpkgs.com` — share pull zone `3857050`, confirmed from the `cdn-pullzone` response header.

Purging per URL was the alternative and is worse here: ~13.5k rate-limited calls per arch, where a single missed call leaves a package silently stale. The cost of the zone purge is a cold cache for everything else, which is why it stays out of the daily update path — `purge_cdn_cache.sh` is untouched.

The purge runs on failure too (`when: status: [success, failure]`): a rebuild that died part-way still replaced objects, and those are exactly the ones a stale edge keeps hiding.

## Verification

`crow lint .crow/` reports all ten configs valid; `bash -n` on the new script passes; prek hooks pass.

Not yet exercised against Bunny — it needs `BUNNYNET_API_KEY`, which is a CI secret. The failure mode is explicit rather than silent: any status other than 200/204 prints the response body and exits non-zero.

Reviewed-on: #160
2026-08-10 06:30:35 +00:00
f8e31af75b fix(ci): make every manual gate default to a value that matches nothing (#158)
Some checks failed
ci/crow/manual/weekly-audit-missing/6 Pipeline was successful
ci/crow/manual/weekly-audit-missing/5 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/5 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/6 Pipeline was canceled
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
## Problem

A manual `crow pipeline create` instantiates **every** file in `.crow/`, and a declared variable default is applied even when the run never passed that variable. A gate is therefore only a gate if its default matches nothing.

#155 fixed the three pipelines that had no manual gate at all. It missed that a *permissive default* leaves a pipeline just as exposed. Demonstrated the expensive way: creating a pipeline with only

```
--var weekly_audit_missing=alpine-324-amd64
```

also started `build-all-versions` — because its gate `target_arch` defaults to `amd64`, which matches its own amd64 matrix rows — and `process-updates` across every row, because that gate defaults to `all`. The run was killed before any `Upload package indexes` step produced output and both alpine324 indices were verified unchanged, but `build-all-versions` uploads binaries and rewrites indexes, so the next one might not be caught in time.

Before:

| pipeline | gate | default | fired on an unrelated manual run |
| --- | --- | --- | --- |
| `build-all-versions` | `target_arch` | `amd64` | amd64 rows — builds and uploads |
| `build-all-versions-install-deps` | `target_arch` | `amd64` | amd64 rows |
| `weekly-rebuild-missing` | `weekly_rebuild_missing` | `all` | every row |
| `weekly-audit-missing` | `weekly_audit_missing` | `all` | every row |
| `process-updates` | `process_cran_updates` | `all` | every row |
| `repair-built-stamp` | `repair_built_stamp` | `arm64` | arm64 rows |

`archive-missed-packages` was the one that behaved, because its gate variable is never declared and so matches nothing. That is the property this restores everywhere.

## What this changes

Each of the six gets a `none` option on its gate variable and defaults to it, so a manual run has to name its target explicitly. The reason is recorded next to the default, where someone would go to change it.

`none` is used rather than dropping the default so the expression always has a defined value to compare, instead of relying on undefined-variable semantics.

Cron triggers are untouched — they match on the `cron:` name, not the variable.

## Verification

`crow lint .crow/` reports all ten configs valid. Auditing every pipeline that accepts a manual event:

```
archive-missed-packages.yaml:        gate=task                    default=<none>
auto-apply-patches.yaml:             gate=auto_apply_patches      default='false'
build-all-versions-install-deps.yaml gate=target_arch             default=none
build-all-versions.yaml:             gate=target_arch             default=none
process-updates.yaml:                gate=process_cran_updates    default=none
repair-built-stamp.yaml:             gate=repair_built_stamp      default=none
trial-build-registry.yaml:           gate=trial_build_registry    default='false'
weekly-audit-missing.yaml:           gate=weekly_audit_missing    default=none
weekly-patch-proposals.yaml:         gate=weekly_patch_proposals  default='false'
weekly-rebuild-missing.yaml:         gate=weekly_rebuild_missing  default=none
```

Every gate now defaults to something that matches no matrix row.

Reviewed-on: #158
2026-08-09 15:31:14 +00:00
ee15c50f53 refactor: migrate package installation from pak to uvr (#147)
Some checks failed
ci/crow/cron/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/14 Pipeline failed
ci/crow/cron/weekly-rebuild-missing/9 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/10 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/11 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/12 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
`bincraft` dropped pak in favour of uvr (5.0.x, "Dependencies and their system requirements are now installed with `uvr` instead of pak during `build_binary_package()`"), so the pipelines, helper scripts and images in this repo move with it.

## Approach

uvr is project-scoped in a way pak is not: `uvr add` refuses to run outside a project and always installs into `.uvr/library/`, and only `uvr sync` honours `--library`. So there is no one-line `pak::pak(...)` equivalent. `local/uvr-install.sh` encapsulates the dance — bootstrap a pinned uvr, mint a throwaway project under `TMPDIR`, `uvr add --no-install`, then `uvr sync --library <target>`. Keeping the project outside the checkout also keeps `uvr init`'s `.Rprofile` from hijacking `.libPaths()` for every other R call in the pipeline.

This matches what bincraft itself does (`uvr sync --install-system-deps --library <lib>`), and bincraft requires `uvr` on `PATH`, which the bootstrap provides: every pipeline that calls `bincraft::` runs `install-bincraft.R` (and therefore the bootstrap) first.

## Changes

| File | Change |
| --- | --- |
| `local/uvr-install.sh` | **New.** The single replacement for `pak::pak(...)`. Bootstraps uvr `v0.4.4`, resolves the R interpreter from `UVR_R_BIN`/`R_VERSION`/`PATH`, pins the manifest to that R's exact version, and syncs into `UVR_TARGET_LIB`/`R_LIBS_USER`. |
| `local/install-bincraft.R` | Installs `forgejo::codefloe.com/rpkgs/bincraft@<tag>` instead of a `git::` URL; keeps the `git ls-remote` tag resolution. Exports `UVR_R_BIN`/`UVR_TARGET_LIB` from `R.home()`/`.libPaths()[1]` so the per-R-minor passes target their own R and library. |
| `.crow/auto-apply-patches.yaml`, `.crow/weekly-patch-proposals.yaml`, `.crow/weekly-audit-missing.yaml`, `.crow/weekly-rebuild-missing.yaml`, `.crow/build-all-versions-install-deps.yaml` | `pak::pak(...)` → `UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh ...`. The explicit `UVR_R_BIN` matters in `build-all-versions-install-deps.yaml`, which has no `R_VERSION` in its step environment. |
| `.crow/build-all-versions.yaml`, `.crow/process-updates.yaml`, `.crow/weekly-rebuild-missing.yaml` | `R_PKG_CACHE_DIR` → `UVR_CACHE_DIR` + `UVR_PACKAGES_DIR` on the same `/mnt/cache` volume, preserving the amd64-off/arm64-on split. Drops the `rm -rf .../pkgcache/_metadata/...` cleanup. |
| `local/r-minor-helpers.R`, `local/build-all.R`, `local/tests/test-trim-pkgcache.R` | Removes `trim_pkgcache_metadata()`, its every-25-packages call and its tests. uvr's cache does not mint a fresh ~70 MB snapshot per `PACKAGES` change. |
| `.crow/build-all-versions-install-deps.yaml` | Drops `pak::sysreqs_db_update()`; uvr resolves sysreqs from its vendored `r-system-requirements` rules via `--install-system-deps`. |
| `docker/Containerfile-shiny-app` | Bootstraps uvr and drives both dependency installs through one uvr project with `UVR_LIBRARY` pointed at the image's R library. |
| `docker/build-one.Dockerfile` | Ships `uvr-install.sh` at `/work/local/` so `install-bincraft.R` finds it. |
| `docker/reprex/alpine.sh` | Replaces `pak::local_install_deps()` with DESCRIPTION parsing + `uvr add`. |
| `local/test-package-loading.R` | Installs via the helper instead of `pak::pkg_install()`. |
| `README.md` | Documents uvr for sysreq inference, archived-version installs and cache clearing. |
| `renovate.json` | Tracks the `UVR_PIN` in `uvr-install.sh` via `github-releases`. |

## Behaviour notes

- **`weekly-audit-missing` still takes bincraft from the default branch**, not the latest release tag, matching what the `git::` pak call did. Called out in a comment rather than silently changed.
- **The uvr pin is repo-wide.** bincraft resolves `uvr` from `PATH` and pins no version of its own, so `UVR_PIN` in `uvr-install.sh` governs the whole pipeline.
- **Persistent caches now also benefit bincraft**, which reads `UVR_CACHE_DIR`/`UVR_PACKAGES_DIR` from the inherited pipeline environment.
- **`uvr sync` will not prune the shared library.** Pruning is disabled whenever `--library` is passed (`do_prune = prune && library_override.is_none()`), so `/mnt/cache/R-pkgs` keeps bincraft and its dependencies. The wipe-on-ABI-mismatch path is *not* similarly guarded, which is why the helper pins the manifest to the active R's exact version.
- **`plans/` and `specs/` are untouched** — they are dated records of decisions made in June 2026 and describe bincraft's then-pak-based internals; rewriting them would misstate history.

## Verification

`shellcheck`, all pre-commit hooks (on this commit's file range) and the `local/tests/` suite (100 assertions) pass.

Not yet exercised in CI: the build-env images do not ship `uvr`, so the per-step `curl install.sh` bootstrap is untested against a real image. Worth a manual `build-all-versions-install-deps` run before merging.

Reviewed-on: #147
2026-07-31 12:19:26 +00:00
61e1c0188d
chore: adjust weekly audit runs
Some checks failed
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/12 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/14 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/16 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/18 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/2 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/4 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/6 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/10 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/8 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/1 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/13 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/5 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/15 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/7 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/9 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/3 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/17 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/11 Pipeline was canceled
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
2026-07-20 19:09:50 +02:00
1ad7a6bfe9 chore: resolve latest bincraft release dynamically (no hardcoded pins) (#107)
Some checks failed
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline failed
ci/crow/cron/process-updates/9 Pipeline failed
ci/crow/cron/process-updates/7 Pipeline failed
ci/crow/cron/process-updates/8 Pipeline failed
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions/5 Pipeline failed
ci/crow/manual/build-all-versions/7 Pipeline failed
ci/crow/manual/build-all-versions/6 Pipeline failed
ci/crow/manual/build-all-versions/8 Pipeline was canceled
## Summary

Stop hardcoding the bincraft version. Every `.crow` workflow and the build-one image pinned `@vX.Y.Z` (and a `packageVersion() != "X.Y.Z"` guard), so each bincraft release meant editing the version in ~8 places — and it was easy to miss one (the Dockerfile lagged at v4.2.1; v4.4.1 shipped without the empty-env fix because of exactly this churn).

## Change

New `local/install-bincraft.R` resolves the **latest release tag dynamically**:

- `git ls-remote --tags` on the public repo (no token),
- keep `vX.Y.Z` tags, pick the highest version (filtered/sorted in R for portability, not via git `--sort`/refspec which behaved inconsistently under `system2()`),
- `pak::pak("git::…@<latest>")` — idempotent on the git ref, so re-runs keep the package unless a newer tag exists.

All call sites now invoke the helper instead of a pinned version:

- `.crow/build-all-versions.yaml` (primary + per-minor pass)
- `.crow/build-all-versions-install-deps.yaml`
- `.crow/process-updates.yaml` (primary + per-minor pass)
- `.crow/weekly-rebuild-missing.yaml`
- `.crow/archive-missed-packages.yaml`
- `docker/build-one.Dockerfile` (ships the helper into the image; `ensure_bincraft` sources it)

## Effect

Tag a new bincraft release → the next CI run / `just rebuild` picks it up automatically. No more pin edits, and no more "forgot to bump the Dockerfile" drift.

## Verified

- Resolver returns the current latest tag (`v4.4.2`) via `git ls-remote` + R-side version sort.
- All five workflow YAMLs parse; helper R parses; air/editorconfig clean.

Note: this tracks the latest **tag**, so cutting a release is still the deliberate gate — CI won't pick up un-tagged main.
Reviewed-on: #107
2026-07-01 08:10:02 +00:00
b4ec6291a9
chore: bump bincraft to 4.4.2
All checks were successful
ci/crow/cron/weekly-audit-missing/15 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-audit-missing/5 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
2026-06-30 16:09:44 +02:00
d496029a79
chore: bump bincraft to 4.4.1
All checks were successful
ci/crow/cron/process-updates/7 Pipeline was successful
2026-06-30 15:51:07 +02:00
b93d1ef6bd chore: silence otelsdk warnings across build paths (#105)
All checks were successful
ci/crow/cron/process-updates/1 Pipeline was successful
## Summary

Silences the recurring `OpenTelemetry error: there is no package called 'otelsdk'` warnings during builds.

## Root cause

The `build-env-*` images configure an OTel exporter (traces/logs/metrics), but `otelsdk` (the R OTel SDK backend) isn't installed. pak's `otel` instrumentation therefore tries to load `otelsdk` on every run and logs the error, falling back to a no-op.

`OTEL_SDK_DISABLED=true` (already set in `build-one.Dockerfile`) does **not** help — the R `otel` package ignores it and gates purely on `OTEL_R_<SIGNAL>_EXPORTER` (then the standard `OTEL_<SIGNAL>_EXPORTER`). When that resolves to a real exporter (`otlp`/`http`/…) with no SDK present, you get the error.

## Fix

Set `OTEL_R_TRACES_EXPORTER`, `OTEL_R_LOGS_EXPORTER`, and `OTEL_R_METRICS_EXPORTER` to `none` so the R otel providers are clean no-ops:

- `docker/build-one.Dockerfile` — exported alongside the existing OTel var.
- `.crow/process-updates.yaml`, `weekly-rebuild-missing.yaml`, `build-all-versions.yaml`, `build-all-versions-install-deps.yaml` — added to each step's `environment` block.

Using the R-specific variables (not the standard `OTEL_*_EXPORTER`) keeps OTel intact for any non-R tooling in the images.

Reviewed-on: #105
2026-06-30 12:41:32 +00:00
00db47398d
chore: 4.4.0 instead of 4.3.1
All checks were successful
ci/crow/cron/process-updates/6 Pipeline was successful
2026-06-30 14:07:22 +02:00
55a18fd87d feat: patch registry + wiring for per-package patching (#103)
Some checks failed
ci/crow/cron/process-updates/15 Pipeline failed
ci/crow/cron/process-updates/14 Pipeline failed
ci/crow/cron/process-updates/16 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/5 Pipeline failed
## Summary

Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`).
Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading.

## What's included

- `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs.
- `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace).
- `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes).
- `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`.
- `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`.

## ⚠️ Merge ordering (blocker)

This PR adds a `patches = ...` argument to `build_binary_package()` calls.
The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until:

1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and
2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`.

The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then.

Reviewed-on: #103
2026-06-30 08:50:19 +00:00
f9bc72d5ab
chore: bump bincraft to 4.2.3
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
2026-06-18 09:43:05 +02:00
4c67703f52 refactor(ci): route workflows by agent group label instead of named agent (#94)
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline failed
## Summary

- Switch crow workflow placement from a per-agent label (`agent: ${AGENT}`) to a group label (`group: rpkgs-${ARCH}`), so jobs target the `rpkgs-amd64` / `rpkgs-arm64` agent pools instead of a single named agent (`artemis`/`gaia`).
- Remove the now-unused `AGENT` matrix variable from all build/process/weekly workflows.
- Update header comments to reflect group-based placement.

Agents opt into a pool via `CROW_AGENT_LABELS="group=rpkgs-<arch>"`. The group name derives from each matrix row's `ARCH`, so adding or replacing agents no longer requires touching the workflow files.

Reviewed-on: #94
2026-06-17 16:26:04 +00:00
dc92bd6a2c
fix(ci): restore cron triggers via per-row cron name filter
Some checks failed
ci/crow/cron/process-updates/3 Pipeline failed
ci/crow/cron/process-updates/13 Pipeline failed
The matrix consolidation routed cron events through
`evaluate: 'CI_PIPELINE_CRON == "..."'`, but CI_PIPELINE_CRON is not
exposed to the when.evaluate constraint context, so no matrix row ever
matched and the scheduled process-updates / weekly runs stopped firing.
Switch back to the first-class `cron:` name filter (still parameterized
per matrix row) which the server matches directly against the fired cron.
2026-06-17 11:03:12 +02:00
4406a8735b refactor(ci): consolidate per-platform crow workflows into 3 matrix files (#92)
## Summary

Collapse **44** per-platform crow pipeline files into **3** matrix-driven files (one per family), using `matrix.include` + the crow #1165 declarative manual `variables:` block:

| Family | Before | After |
|---|---|---|
| process-updates | 14 | `process-updates.yaml` |
| weekly-audit-missing | 16 | `weekly-audit-missing.yaml` |
| weekly-rebuild-missing | 14 | `weekly-rebuild-missing.yaml` |

- **One file per family** (not split by arch). Arch placement is via the agent label as a matrix var (`artemis`=amd64, `gaia`=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-`nodeSelector` placement (no agent label).
- **Matrix axis is named `OS`** (e.g. `redhat-9`) per request, with per-row `R_VERSION`, image tag, codename, and `process_new` (FALSE for alpine).
- **Runs preserved 1:1:**
  - *Cron*: each existing `<family>-<os>-<arch>` cron still fires only its matching matrix row via `CI_PIPELINE_CRON`. Server-side cron entries unchanged.
  - *Manual*: a #1165 dropdown variable (`process_cran_updates` / `weekly_audit_missing` / `weekly_rebuild_missing`) selects a single `<os>-<arch>` or `all`. weekly-rebuild defaults to `all`, matching its former bare `event: manual` trigger.

## Behavior deltas (intentional, flagged)

To fit one file per family the backend was unified:
- The amd64-only `node.kubernetes.io/instance-type: AX42` nodeSelector pin is **dropped**; placement now relies on the `artemis`/`gaia` agents.
- **process-updates arm64** jobs now share the same `backend_options` as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped.

## Validation required before merge

Relies on crow interpolating matrix variables inside `when.evaluate`, `labels.agent`, and `commands`. Interpolation in image/env/commands is standard crow; the novel bit is `when.evaluate`. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch:
1. Trigger manually with the dropdown = `redhat-9-amd64` -> confirm only that one job runs (image `build-env-redhat:9`, R 4.4.3, agent artemis).
2. Trigger with `all` -> confirm all os/arch jobs schedule.

Crons only fire on the default branch, so no collision while unmerged. Fallback if `${...}` doesn't interpolate in `when`: a runtime `case "$CI_PIPELINE_CRON" in ...` guard, same files otherwise.

Reviewed-on: #92
2026-06-16 09:33:06 +00:00
Renamed from .crow/weekly-rebuild-missing-alpine-322-arm64.yaml (Previous history)