| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 77ce1d212d |
fix(ci): gate the Built-stamp repair on its own variable (#151)
Some checks failed
ci/crow/manual/build-all-versions/4 Pipeline was canceled
ci/crow/manual/process-updates/12 Pipeline was canceled
ci/crow/manual/build-all-versions/2 Pipeline was canceled
ci/crow/manual/build-all-versions/1 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/1 Pipeline was canceled
ci/crow/manual/process-updates/6 Pipeline was canceled
ci/crow/manual/process-updates/18 Pipeline was canceled
ci/crow/manual/process-updates/16 Pipeline was canceled
ci/crow/manual/process-updates/17 Pipeline was canceled
ci/crow/manual/process-updates/11 Pipeline was canceled
ci/crow/manual/process-updates/1 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/3 Pipeline was canceled
ci/crow/manual/process-updates/5 Pipeline was canceled
ci/crow/manual/process-updates/15 Pipeline was canceled
ci/crow/manual/build-all-versions/3 Pipeline was canceled
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline was successful
ci/crow/manual/build-all-versions/5 Pipeline failed
ci/crow/manual/build-all-versions/7 Pipeline failed
ci/crow/manual/build-all-versions/8 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
## Problem
`repair-built-stamp.yaml` gated on `target_arch`:
```yaml
when:
- event: manual
evaluate: 'target_arch == "${ARCH}"'
```
That is the same variable `build-all-versions.yaml` and `build-all-versions-install-deps.yaml` gate on. A manual run passing `target_arch=arm64` to reach the repair therefore matched all three, so triggering a dry-run repair also queued a full arm64 rebuild.
I hit this triggering the alpine 3.22 dry run (pipeline 10706), which I killed. `crow pipeline ps` renders empty states on this version, so I could not confirm from the CLI whether the rebuild workflows started before the kill or only sat queued behind the running cron jobs; no output was attributable to them.
## Change
Gate on `repair_built_stamp` instead. Every other pipeline here already gates on a variable named after itself (`process_cran_updates`, `weekly_audit_missing`, `weekly_rebuild_missing`), and `target_arch` was the odd one out being shared by two.
The header now records the collision and the exact invocation, so the next pipeline added here does not repeat it.
## Verification
`crow lint` passes; `prek` clean.
Grep of every trigger condition in `.crow/` confirms `repair_built_stamp` is unique and that no other pipeline gates on `OS`, `OS_VERSION`, `R_VERSION` or `dry_run` alone:
```
archive-missed-packages.yaml: task == "archive-missed-packages"
build-all-versions.yaml: target_arch == "${ARCH}"
build-all-versions-install-deps.yaml: target_arch == "${ARCH}"
weekly-audit-missing.yaml: weekly_audit_missing == ...
repair-built-stamp.yaml: repair_built_stamp == "${ARCH}"
process-updates.yaml: process_cran_updates == ...
weekly-rebuild-missing.yaml: weekly_rebuild_missing == ...
```
## Note for whoever runs it
`crow pipeline create` against this instance returned HTTP 504 while still creating the pipeline. Retrying on that error duplicates the run: I created four before noticing. Verify with `pipeline ls` rather than trusting the exit status.
Reviewed-on: #151
|
|||
| 0da29ab3f6 |
fix(index): add a slot repair for a broken Built stamp and retire alpine 3.21 (#150)
Some checks failed
ci/crow/manual/trial-build-registry/3 Pipeline was successful
ci/crow/manual/trial-build-registry/5 Pipeline was successful
ci/crow/manual/process-updates/7 Pipeline was successful
ci/crow/manual/trial-build-registry/7 Pipeline was successful
ci/crow/manual/trial-build-registry/9 Pipeline was successful
ci/crow/manual/trial-build-registry/11 Pipeline was successful
ci/crow/manual/trial-build-registry/13 Pipeline was successful
ci/crow/manual/auto-apply-patches Pipeline was successful
ci/crow/manual/trial-build-registry/15 Pipeline was successful
ci/crow/manual/trial-build-registry/17 Pipeline was successful
ci/crow/manual/weekly-audit-missing/1 Pipeline was successful
ci/crow/manual/weekly-audit-missing/5 Pipeline was successful
ci/crow/manual/weekly-audit-missing/3 Pipeline was successful
ci/crow/manual/weekly-audit-missing/7 Pipeline was successful
ci/crow/manual/weekly-audit-missing/9 Pipeline was successful
ci/crow/manual/weekly-audit-missing/11 Pipeline was successful
ci/crow/manual/weekly-audit-missing/13 Pipeline was successful
ci/crow/manual/weekly-audit-missing/17 Pipeline was successful
ci/crow/manual/weekly-audit-missing/15 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/1 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/3 Pipeline was canceled
ci/crow/manual/process-updates/3 Pipeline was canceled
ci/crow/manual/process-updates/15 Pipeline was canceled
ci/crow/manual/process-updates/11 Pipeline was canceled
ci/crow/manual/weekly-patch-proposals Pipeline was canceled
ci/crow/manual/process-updates/13 Pipeline was canceled
ci/crow/manual/process-updates/1 Pipeline was canceled
ci/crow/manual/process-updates/5 Pipeline was canceled
ci/crow/manual/process-updates/17 Pipeline was canceled
ci/crow/cron/process-updates/15 Pipeline failed
## Problem `arm64/alpine321` and `arm64/alpine322` advertise a broken stamp: ``` arm64/alpine321 :: 22930 Built: R 4.4.0; NA; 2026-07-31 13:35:52 UTC; unix arm64/alpine322 :: 24696 Built: R 4.5.0; NA; 2026-07-31 13:51:54 UTC; unix ``` The per-minor sub-slots (`contrib/4.4`, `4.5`, `4.6`) are affected too. All 18 other slots are correct. uvr matches the stamp's platform triple plus R minor to pick binary over source, so nothing matches `NA` and both slots silently serve as source-only, which is exactly the regression bincraft#85 added the stamp to prevent. `install.packages()` is unaffected, since it reads `Built:` from each tarball's own `DESCRIPTION`. The tarballs are fine (`arm64/alpine322/.../dress.graph_0.8.3.tar.gz` carries `aarch64-unknown-linux-musl`), and so is the R that built them (`r-4.5.0_1_aarch64.apk` ships `R_PLATFORM='aarch64-unknown-linux-musl'`). Only the index is wrong. bincraft#96 stops a stamp like this being written again, but it cannot repair what is already there. ## Why not just re-run the index update `upload_package_index()` reuses the slot's remote `PACKAGES.db`, and cranlike's `update_db()` only reparses files whose md5 changed, so entries already in the database keep the stamp they were written with. Dropping `PACKAGES.db` to force a full reparse does work, and it is what bincraft#85's rollout note suggested, but for an S3 repo cranlike reads each package's metadata from the CRAN *source* mirror on GitHub. A 25k-entry slot is then 25k requests to raw.githubusercontent.com, with a real risk of being rate-limited part-way through and leaving the slot half-written. Only the `Built` column is wrong, so this corrects it in place instead: patch the column in `PACKAGES.db`, put the database back, and let `upload_package_index()` re-emit `PACKAGES*` from it. `update_db()` always rewrites the index files even when nothing was reparsed, so no tarball is re-read and nothing is fetched from GitHub. ## Change - `local/repair-built-stamp.R` — repairs the generic slot plus every per-minor sub-slot. Dry-run by default; `--apply` writes. The replacement comes from `bincraft::built_stamp()` under the R running the script, so it is exactly what a healthy run would have written, and bincraft#96's guard makes a broken build image fail rather than write a second bad stamp. - `.crow/repair-built-stamp.yaml` — manual pipeline, routed by `target_arch` to the matching agent group and platform image, with `dry_run` defaulting to `true`. - `.crow/archive-missed-packages.yaml` — drop the two `alpine321` matrix entries. Alpine 3.21 is EOL: the website advertises only 3.23/3.24 and `process-updates.yaml` already dropped it, so that slot is retired rather than repaired. ## Verification `crow lint .crow/` passes on all nine pipelines. `air format` and `jarl check` are clean; the script parses, and the `/opt/R` minor-version derivation was checked against `4.4.3 / 4.5.3 / 4.6.0 / current` → `4.4 4.5 4.6`. The repair itself is unrun by design — it needs B2 credentials and an arm64 agent. ## Rollout 1. Cut a bincraft release so `local/install-bincraft.R` picks up #96 (it resolves the latest `vX.Y.Z` tag, and #96 is only on `main`). 2. Run this pipeline with `target_arch=arm64`, `OS=alpine`, `OS_VERSION=3.22`, `R_VERSION=4.5.3`, `dry_run=true` and check the reported counts. 3. Re-run with `dry_run=false`. 4. Confirm: `curl -sS https://cran.devxy.io/arm64/alpine322/latest/src/contrib/PACKAGES | grep '^Built:' | sort | uniq -c` `arm64/alpine321` is deliberately left alone. Reviewed-on: #150 |
|||
| ee15c50f53 |
refactor: migrate package installation from pak to uvr (#147)
Some checks failed
ci/crow/cron/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/14 Pipeline failed
ci/crow/cron/weekly-rebuild-missing/9 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/10 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/11 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/12 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
`bincraft` dropped pak in favour of uvr (5.0.x, "Dependencies and their system requirements are now installed with `uvr` instead of pak during `build_binary_package()`"), so the pipelines, helper scripts and images in this repo move with it. ## Approach uvr is project-scoped in a way pak is not: `uvr add` refuses to run outside a project and always installs into `.uvr/library/`, and only `uvr sync` honours `--library`. So there is no one-line `pak::pak(...)` equivalent. `local/uvr-install.sh` encapsulates the dance — bootstrap a pinned uvr, mint a throwaway project under `TMPDIR`, `uvr add --no-install`, then `uvr sync --library <target>`. Keeping the project outside the checkout also keeps `uvr init`'s `.Rprofile` from hijacking `.libPaths()` for every other R call in the pipeline. This matches what bincraft itself does (`uvr sync --install-system-deps --library <lib>`), and bincraft requires `uvr` on `PATH`, which the bootstrap provides: every pipeline that calls `bincraft::` runs `install-bincraft.R` (and therefore the bootstrap) first. ## Changes | File | Change | | --- | --- | | `local/uvr-install.sh` | **New.** The single replacement for `pak::pak(...)`. Bootstraps uvr `v0.4.4`, resolves the R interpreter from `UVR_R_BIN`/`R_VERSION`/`PATH`, pins the manifest to that R's exact version, and syncs into `UVR_TARGET_LIB`/`R_LIBS_USER`. | | `local/install-bincraft.R` | Installs `forgejo::codefloe.com/rpkgs/bincraft@<tag>` instead of a `git::` URL; keeps the `git ls-remote` tag resolution. Exports `UVR_R_BIN`/`UVR_TARGET_LIB` from `R.home()`/`.libPaths()[1]` so the per-R-minor passes target their own R and library. | | `.crow/auto-apply-patches.yaml`, `.crow/weekly-patch-proposals.yaml`, `.crow/weekly-audit-missing.yaml`, `.crow/weekly-rebuild-missing.yaml`, `.crow/build-all-versions-install-deps.yaml` | `pak::pak(...)` → `UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh ...`. The explicit `UVR_R_BIN` matters in `build-all-versions-install-deps.yaml`, which has no `R_VERSION` in its step environment. | | `.crow/build-all-versions.yaml`, `.crow/process-updates.yaml`, `.crow/weekly-rebuild-missing.yaml` | `R_PKG_CACHE_DIR` → `UVR_CACHE_DIR` + `UVR_PACKAGES_DIR` on the same `/mnt/cache` volume, preserving the amd64-off/arm64-on split. Drops the `rm -rf .../pkgcache/_metadata/...` cleanup. | | `local/r-minor-helpers.R`, `local/build-all.R`, `local/tests/test-trim-pkgcache.R` | Removes `trim_pkgcache_metadata()`, its every-25-packages call and its tests. uvr's cache does not mint a fresh ~70 MB snapshot per `PACKAGES` change. | | `.crow/build-all-versions-install-deps.yaml` | Drops `pak::sysreqs_db_update()`; uvr resolves sysreqs from its vendored `r-system-requirements` rules via `--install-system-deps`. | | `docker/Containerfile-shiny-app` | Bootstraps uvr and drives both dependency installs through one uvr project with `UVR_LIBRARY` pointed at the image's R library. | | `docker/build-one.Dockerfile` | Ships `uvr-install.sh` at `/work/local/` so `install-bincraft.R` finds it. | | `docker/reprex/alpine.sh` | Replaces `pak::local_install_deps()` with DESCRIPTION parsing + `uvr add`. | | `local/test-package-loading.R` | Installs via the helper instead of `pak::pkg_install()`. | | `README.md` | Documents uvr for sysreq inference, archived-version installs and cache clearing. | | `renovate.json` | Tracks the `UVR_PIN` in `uvr-install.sh` via `github-releases`. | ## Behaviour notes - **`weekly-audit-missing` still takes bincraft from the default branch**, not the latest release tag, matching what the `git::` pak call did. Called out in a comment rather than silently changed. - **The uvr pin is repo-wide.** bincraft resolves `uvr` from `PATH` and pins no version of its own, so `UVR_PIN` in `uvr-install.sh` governs the whole pipeline. - **Persistent caches now also benefit bincraft**, which reads `UVR_CACHE_DIR`/`UVR_PACKAGES_DIR` from the inherited pipeline environment. - **`uvr sync` will not prune the shared library.** Pruning is disabled whenever `--library` is passed (`do_prune = prune && library_override.is_none()`), so `/mnt/cache/R-pkgs` keeps bincraft and its dependencies. The wipe-on-ABI-mismatch path is *not* similarly guarded, which is why the helper pins the manifest to the active R's exact version. - **`plans/` and `specs/` are untouched** — they are dated records of decisions made in June 2026 and describe bincraft's then-pak-based internals; rewriting them would misstate history. ## Verification `shellcheck`, all pre-commit hooks (on this commit's file range) and the `local/tests/` suite (100 assertions) pass. Not yet exercised in CI: the build-env images do not ship `uvr`, so the per-step `curl install.sh` bootstrap is untested against a real image. Worth a manual `build-all-versions-install-deps` run before merging. Reviewed-on: #147 |
|||
|
736564a4ff |
fix: prevent renovate bumps for fixed matrix assignments
All checks were successful
ci/crow/cron/weekly-rebuild-missing/9 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/11 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/10 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/12 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/14 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
|
|||
|
61e1c0188d |
chore: adjust weekly audit runs
Some checks failed
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/12 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/14 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/16 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/18 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/2 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/4 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/6 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/10 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/8 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/1 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/13 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/5 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/15 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/7 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/9 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/3 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/17 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/11 Pipeline was canceled
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
|
|||
| a30c6f9532 |
fix(crow): probe the auto-patch branch with git ls-remote to avoid a scary fatal (#133)
All checks were successful
ci/crow/manual/trial-build-registry/3 Pipeline was successful
ci/crow/manual/trial-build-registry/11 Pipeline was successful
ci/crow/manual/trial-build-registry/7 Pipeline was successful
ci/crow/manual/trial-build-registry/8 Pipeline was successful
ci/crow/manual/trial-build-registry/10 Pipeline was successful
ci/crow/manual/trial-build-registry/2 Pipeline was successful
ci/crow/manual/trial-build-registry/6 Pipeline was successful
ci/crow/manual/trial-build-registry/4 Pipeline was successful
ci/crow/manual/trial-build-registry/14 Pipeline was successful
ci/crow/manual/trial-build-registry/18 Pipeline was successful
ci/crow/manual/trial-build-registry/16 Pipeline was successful
ci/crow/manual/trial-build-registry/12 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
Follow-up to #132. The no-op path already worked (the `else` branch runs and `exit 0`s), but using `git fetch` as the *existence check* printed `fatal: couldn't find remote ref auto/registry-patch-proposals` to stderr — which looks like a failure even though the step succeeds. Probe with `git ls-remote --exit-code --heads origin ${patch_branch}` instead: verified against the live repo it exits 0 for an existing branch and 2 for a missing one, **silently** (no `fatal`). Only fetch + checkout when the branch actually exists. Net: same behavior, clean log — a run with no pending auto-patch branch prints just `No ... branch; no pending auto-patch proposals to verify.` and exits 0. Reviewed-on: #133 |
|||
| 13fcad6dc4 |
fix(crow): make the trial-build gate no-op when the auto-patch branch is absent (#132)
All checks were successful
ci/crow/manual/trial-build-registry/5 Pipeline was successful
ci/crow/manual/trial-build-registry/1 Pipeline was successful
ci/crow/manual/trial-build-registry/3 Pipeline was successful
ci/crow/manual/trial-build-registry/17 Pipeline was successful
ci/crow/manual/trial-build-registry/15 Pipeline was successful
ci/crow/manual/trial-build-registry/6 Pipeline was successful
ci/crow/manual/trial-build-registry/2 Pipeline was successful
ci/crow/manual/trial-build-registry/4 Pipeline was successful
ci/crow/manual/trial-build-registry/8 Pipeline was successful
ci/crow/manual/trial-build-registry/12 Pipeline was successful
ci/crow/manual/trial-build-registry/10 Pipeline was successful
ci/crow/manual/trial-build-registry/16 Pipeline was successful
ci/crow/manual/trial-build-registry/18 Pipeline was successful
ci/crow/manual/trial-build-registry/14 Pipeline was successful
ci/crow/manual/trial-build-registry/9 Pipeline was successful
ci/crow/manual/trial-build-registry/13 Pipeline was successful
ci/crow/manual/trial-build-registry/7 Pipeline was successful
ci/crow/manual/trial-build-registry/11 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
## Problem The trial pipeline errored: ``` git clone -q --branch auto/registry-patch-proposals ... fatal: Remote branch auto/registry-patch-proposals not found in upstream origin ``` I deleted the stale `auto/registry-patch-proposals` branch during cleanup, and — now that cascade detection (#128) makes the proposer correctly find **no candidates** — the proposer exits before ever recreating it. So the branch is legitimately absent, and the gate's hard `git clone --branch` fails. ## Fix "Branch absent" = "no pending auto-patch proposals" = **nothing to verify**, which should be a clean no-op, not a failure. Clone `main`, then fetch + checkout the patch branch only if it exists; otherwise log and `exit 0`. A present branch is verified exactly as before (checkout its tip, diff registry vs `main`). This also means the gate now runs `main`'s `trial-build-registry.R` (with the #130 fix) rather than a possibly-stale copy on the branch. Reviewed-on: #132 |
|||
| 875b086122 |
ci(crow): enable verbose patched builds in the trial-build gate (#129)
Companion to bincraft codefloe #65. Sets `BINCRAFT_VERBOSE_PATCH_BUILD=TRUE` in the trial-build gate so a failed isolated patched build prints the real compiler error instead of `System command 'R' failed`. Harmless on bincraft versions without the flag (unknown env var is ignored). Takes effect once bincraft #65 is released and the build-env images pick it up. Reviewed-on: #129 |
|||
|
ed1485c542 |
ci: remove evaluate conditions | |||
| 1bca6004fd |
chore(local): reuse FORGEJO_TOKEN for the auto-patch push (#125)
Follow-up to #124 (merged): the auto-patch pipeline required a separate write-scoped `REPO_RW_TOKEN` to push the branch. Reuse the existing `FORGEJO_TOKEN` instead. - `propose-patches.R --open-pr` now pushes `auto/registry-patch-proposals` over HTTPS with `FORGEJO_TOKEN` (the same token used for the PR API); the read-only `origin` clone URL can't push, so it builds an authenticated URL explicitly. - Drop `REPO_RW_TOKEN` from `.crow/auto-apply-patches.yaml` and the docs. No new secret needed: the pipeline's secrets are now `PGPASS`, `REPO_RO_TOKEN`, and `FORGEJO_TOKEN` (all existing). `FORGEJO_TOKEN` must have repository write scope for the push to succeed. 105 tests pass; all pre-commit hooks pass. Reviewed-on: #125 |
|||
| 4bca17e4ac |
feat(local): auto-apply registry patches with a build-env trial-build gate (#124)
All checks were successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
Closes the classifier loop (issue #115, step 3): turns the auto-proposable candidates into an actual PR, gated by a real trial build **in our own build-env images**. Chosen model (from the design discussion): **autonomous PR**, **PR-first with a CI trial-build gate**, **bounded top-N batch** per run. ## Creating the patch PR - `propose-patches.R` gains: - `--limit N` -- act on the top-N candidates by failure volume; the rest defer to the next run (logged). - `--open-pr` -- write the entries onto the reused `auto/registry-patch-proposals` branch, push (with `REPO_RW_TOKEN`), and open/update **one** PR via the Forgejo API (so re-runs update the same PR instead of piling up). - `.crow/auto-apply-patches.yaml` -- a single job that runs `--open-pr --limit` on a cron/manual trigger. Needs `FORGEJO_TOKEN` + a write-scoped `REPO_RW_TOKEN`. Novel source diffs and unknown signatures are still never proposed; nothing merges. ## The merge gate (our build-env images) - `.crow/trial-build-registry.yaml` -- matrixed over the real `OS/IMG` build-env matrix (alpine:3.24, redhat:8/9/10, ubuntu:jammy/noble/**resolute** for ubuntu-2604). Each platform runs `local/trial-build-registry.R`, which diffs the branch registry against `main` and trial-builds **only the entries the branch adds** that apply to that platform, inside `reg.devxy.io/rpkgs/build-env-*`. Green only if every new entry builds; a platform with no new entries is a fast no-op. Nothing is uploaded/archived/recorded. - The base-registry read **fails loud** if it can't read `registry.json` at `main`, rather than silently treating the base as empty and trial-building the whole registry. ## Notes / follow-up - The repo uses **no `pull_request` triggers**, so the gate runs manually or on a cron against the branch (`--var patch_branch=...`). Wiring it to fire automatically on the PR needs `event: pull_request` enabled on the Forgejo webhook -- a one-line addition once that's on. - Two new crons to register in the crow UI: `auto-apply-patches` and `trial-build-registry`. New secret needed: `REPO_RW_TOKEN` (write scope) for the push. ## Verification - New pure helpers `entry_applies_to_os()` / `new_registry_packages()` covered by tests (platform codename/family/wildcard matching; added-vs-unchanged entry detection; per-platform filtering). - `--limit` smoke (stubbed DB): top-2 by volume proposed, 3 deferred, candidate registry validates. - Full suite: 105 tests pass; all pre-commit hooks pass (air, prettier, markdownlint, yamllint, validate-patches). Reviewed-on: #124 |
|||
| 21a2fe9e6c |
feat(local): report unclassified and dependency-blocked failures for discovery (#122)
## Why Issue #120 (the auto-proposed-patches issue) only lists **auto-proposable** fixes -- currently just the TBB signatures. So a reasonable read of it was "TBB is our only build failure", when in fact three whole categories are simply not shown there: - **Unclassified failures** -- anything that doesn't match a seeded signature is routed to human triage and never appears (we've only seeded TBB and libuv signatures). - **Dependency-blocked failures** -- the ~800 RcppParallel dependents (post #121) are still failing; they only show as a log line. - Human-only signatures (libuv). These blind spots are exactly where the *next* signatures should come from, so they deserve the same visibility as the proposals. ## What Extend the feedback-loop tracker to surface the classifier's blind spots: - **`unclassified_summary()`** -- groups every unknown-signature failure by normalised fingerprint, ranked by build count, capped with an explicit `dropped_groups` count (no silent truncation), each with example packages + platforms. These are the candidates for new `build_signatures()` rules. - **`blocked_summary()`** -- lists each dependency (e.g. RcppParallel) and how many dependents wait on it. - `proposal-tracking.R` prints both sections, and a new **`--open-issue`** mode posts/updates a *"Unclassified build failures (needs signatures) (#115)"* Forgejo issue. - The weekly crow pipeline now runs the tracker with `--open-issue`, so it maintains a second tracking issue alongside the proposals one. Read-only on the DB; the only writes are the two issues. ## Verification - New tests cover `unclassified_summary` (ranking + both caps) and `blocked_summary`. - Tracker smoke with a stubbed DB (proposable + blocked + unclassified mix) prints the hit rate, `Blocked on a dependency: RcppParallel: 2 dependent(s)`, and `Unclassified failures ... [2 builds | 2 pkgs] ld: undefined reference ...`. - Full suite: 95 tests pass; all pre-commit hooks pass (air, prettier, markdownlint, yamllint, validate-patches). Reviewed-on: #122 |
|||
|
77508f8fc5 |
ci: clean weekly-patch-proposals trigger | |||
| af481f5129 |
ci(crow): add weekly patch-proposal + feedback-loop pipeline (#118)
Adds the scheduled pipeline for the failure-triage proposer/tracker landed in #117 (issue #115, steps 3 + 4). ## `.crow/weekly-patch-proposals.yaml` A single (non-matrix) job -- the classifier groups over all of `single_builds`, so unlike the per-platform `weekly-audit-missing` it runs once globally. Weekly it: 1. `propose-patches.R --open-issue` -- posts/updates a Forgejo tracking issue with the auto-proposable registry entries (human reviews, trial-builds, opens the PR; nothing merges). 2. `proposal-tracking.R` -- logs the feedback-loop metrics (signature hit rate, proposed-vs-merged, retirement candidates). Low blast radius: clones read-only (`REPO_RO_TOKEN`); the only write is the tracking issue via `FORGEJO_TOKEN`. It builds nothing (no B2 keys, no bincraft). Triggers: `cron: weekly-patch-proposals` (register the cron in the crow UI) or manual with `task=weekly-patch-proposals`, matching the `archive-missed-packages` convention. Documented under "Scheduled run" in `local/patches/README.md`. `yamllint` and the other pre-commit hooks pass. Reviewed-on: #118 |
|||
| a4b274f281 |
fix(ci): bound pkgcache _metadata growth to stop macmini disk-fill (#110)
Some checks failed
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was canceled
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was canceled
ci/crow/cron/process-updates/8 Pipeline failed
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was canceled
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline failed
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline failed
ci/crow/manual/build-all-versions/5 Pipeline was canceled
ci/crow/manual/build-all-versions/8 Pipeline was canceled
ci/crow/cron/process-updates/12 Pipeline was canceled
ci/crow/manual/build-all-versions/7 Pipeline was canceled
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/2 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline was canceled
## Problem
The arm64 `build-all` pipeline fills the macmini (gaia) host disk despite an 8h prune.
Root cause is not images or job volumes: it is the persistent dep-cache volume, specifically `pkgcache/R/pkgcache/_metadata`, which grew to ~165 GB.
`{pkgcache}` mints a new content hash for the "patched" binaries repo on every PACKAGES change, so each per-package build writes a fresh ~70 MB `pkgs-<hash>.rds` (+ `patched-<hash>/`) that is never evicted (2407 snapshots observed).
When the disk hits 100% OrbStack stops and the on-host prune can no longer connect to the daemon, so it never self-heals.
## Change (Workstream A of the disk-fill fix)
- Add `trim_pkgcache_metadata()` to `local/r-minor-helpers.R`: keeps the newest `keep` (default 20) `patched-*`/`pkgs-*.rds` entries under `_metadata`, deleting only entries older than `min_age_secs` (default 600s) so it never races the up-to-4 concurrent split-jobs sharing the volume.
Preserves `pkg/` downloads and the stable CRAN/BioC/INLA repo dirs.
No-op when `R_PKG_CACHE_DIR` is empty (amd64) or `_metadata` is absent (first run).
- Call it every 25 packages inside the build loop in `local/build-all.R`.
- Add a defensive start-of-run cleanup of `_metadata/patched-*` + `pkgs-*.rds` to the two workflows that mount the persistent volume (`build-all-versions.yaml`, `build-all-versions-install-deps.yaml`).
Only these paths are touched; `process-updates.yaml`/`weekly-rebuild-missing.yaml` (no persistent volume) are unchanged.
Follow-ups (separate workstreams): on-host self-healing prune watcher + OrbStack disk cap (ansible), and Prometheus/Grafana alerting (k8s-talos).
Upstream: bincraft patched-repo hash churn is the true source fix.
New unit tests (6) for the helper; full suite 24/24 green.
Reviewed-on: #110
|
|||
| 1ad7a6bfe9 |
chore: resolve latest bincraft release dynamically (no hardcoded pins) (#107)
Some checks failed
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline failed
ci/crow/cron/process-updates/9 Pipeline failed
ci/crow/cron/process-updates/7 Pipeline failed
ci/crow/cron/process-updates/8 Pipeline failed
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions/5 Pipeline failed
ci/crow/manual/build-all-versions/7 Pipeline failed
ci/crow/manual/build-all-versions/6 Pipeline failed
ci/crow/manual/build-all-versions/8 Pipeline was canceled
## Summary
Stop hardcoding the bincraft version. Every `.crow` workflow and the build-one image pinned `@vX.Y.Z` (and a `packageVersion() != "X.Y.Z"` guard), so each bincraft release meant editing the version in ~8 places — and it was easy to miss one (the Dockerfile lagged at v4.2.1; v4.4.1 shipped without the empty-env fix because of exactly this churn).
## Change
New `local/install-bincraft.R` resolves the **latest release tag dynamically**:
- `git ls-remote --tags` on the public repo (no token),
- keep `vX.Y.Z` tags, pick the highest version (filtered/sorted in R for portability, not via git `--sort`/refspec which behaved inconsistently under `system2()`),
- `pak::pak("git::…@<latest>")` — idempotent on the git ref, so re-runs keep the package unless a newer tag exists.
All call sites now invoke the helper instead of a pinned version:
- `.crow/build-all-versions.yaml` (primary + per-minor pass)
- `.crow/build-all-versions-install-deps.yaml`
- `.crow/process-updates.yaml` (primary + per-minor pass)
- `.crow/weekly-rebuild-missing.yaml`
- `.crow/archive-missed-packages.yaml`
- `docker/build-one.Dockerfile` (ships the helper into the image; `ensure_bincraft` sources it)
## Effect
Tag a new bincraft release → the next CI run / `just rebuild` picks it up automatically. No more pin edits, and no more "forgot to bump the Dockerfile" drift.
## Verified
- Resolver returns the current latest tag (`v4.4.2`) via `git ls-remote` + R-side version sort.
- All five workflow YAMLs parse; helper R parses; air/editorconfig clean.
Note: this tracks the latest **tag**, so cutting a release is still the deliberate gate — CI won't pick up un-tagged main.
Reviewed-on: #107
|
|||
|
b4ec6291a9 |
chore: bump bincraft to 4.4.2
All checks were successful
ci/crow/cron/weekly-audit-missing/15 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-audit-missing/5 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
|
|||
|
d496029a79 |
chore: bump bincraft to 4.4.1
All checks were successful
ci/crow/cron/process-updates/7 Pipeline was successful
|
|||
| b93d1ef6bd |
chore: silence otelsdk warnings across build paths (#105)
All checks were successful
ci/crow/cron/process-updates/1 Pipeline was successful
## Summary Silences the recurring `OpenTelemetry error: there is no package called 'otelsdk'` warnings during builds. ## Root cause The `build-env-*` images configure an OTel exporter (traces/logs/metrics), but `otelsdk` (the R OTel SDK backend) isn't installed. pak's `otel` instrumentation therefore tries to load `otelsdk` on every run and logs the error, falling back to a no-op. `OTEL_SDK_DISABLED=true` (already set in `build-one.Dockerfile`) does **not** help — the R `otel` package ignores it and gates purely on `OTEL_R_<SIGNAL>_EXPORTER` (then the standard `OTEL_<SIGNAL>_EXPORTER`). When that resolves to a real exporter (`otlp`/`http`/…) with no SDK present, you get the error. ## Fix Set `OTEL_R_TRACES_EXPORTER`, `OTEL_R_LOGS_EXPORTER`, and `OTEL_R_METRICS_EXPORTER` to `none` so the R otel providers are clean no-ops: - `docker/build-one.Dockerfile` — exported alongside the existing OTel var. - `.crow/process-updates.yaml`, `weekly-rebuild-missing.yaml`, `build-all-versions.yaml`, `build-all-versions-install-deps.yaml` — added to each step's `environment` block. Using the R-specific variables (not the standard `OTEL_*_EXPORTER`) keeps OTel intact for any non-R tooling in the images. Reviewed-on: #105 |
|||
|
00db47398d |
chore: 4.4.0 instead of 4.3.1
All checks were successful
ci/crow/cron/process-updates/6 Pipeline was successful
|
|||
| 55a18fd87d |
feat: patch registry + wiring for per-package patching (#103)
Some checks failed
ci/crow/cron/process-updates/15 Pipeline failed
ci/crow/cron/process-updates/14 Pipeline failed
ci/crow/cron/process-updates/16 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/5 Pipeline failed
## Summary Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`). Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading. ## What's included - `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs. - `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace). - `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes). - `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`. - `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`. ## ⚠️ Merge ordering (blocker) This PR adds a `patches = ...` argument to `build_binary_package()` calls. The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until: 1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and 2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`. The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then. Reviewed-on: #103 |
|||
|
a1d7064fb8 |
feat(ci): add process-updates rows for alpine-324 and ubuntu resolute
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline was successful
ci/crow/manual/build-all-versions/5 Pipeline was successful
ci/crow/manual/build-all-versions/7 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/build-all-versions/8 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/build-all-versions/2 Pipeline failed
Add matrix rows and manual dropdown options for alpine 3.24 (alpine-324, PROCESS_NEW=FALSE) and ubuntu resolute / 26.04 (ubuntu-2604, OS_ID resolute, R 4.5.3, PROCESS_NEW=TRUE). |
|||
|
b46472b59e |
fix(ci): pass Backblaze B2 credentials to process_unarchived_pkgs
All checks were successful
ci/crow/cron/process-updates/12 Pipeline was successful
The function defaults to Hetzner S3 with HETZNER_S3_*_K3S env vars that are not set in these workflows, so paws failed with "No compatible credentials provided". Pass the B2 endpoint/region/bucket and B2_S3_* secrets explicitly, matching the other S3 calls. Also use set_codename(NULL) for the build-all-versions codename so it matches the S3 repo path. |
|||
|
3420523849 |
chore: add all options for OS_VERSION
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
|
|||
| 83c761b736 |
feat(ci): OS/OS_VERSION manual dropdowns + restore lost crow fix (#97)
## Summary
Two changes:
1. **`OS`/`OS_VERSION` manual-run dropdowns** — give these form variables explicit `options:` lists (like `target_arch` and `R_VERSION`), so the manual-run form shows dropdowns instead of free-text, in both `build-all-versions.yaml` and `build-all-versions-install-deps.yaml`. Crow form variables are independent (no cascading), so the operator still has to pick a coherent `OS` + `OS_VERSION` combination (e.g. `redhat` + `9`, not `alpine` + `jammy`).
2. **Repairs `main`** — the crow fix from PR #96 (`
|
|||
| 0f4330c05c |
fix(ci): make build-all self-sufficient on agents without install-deps (#96)
## Summary
Fixes the recurring `build-all-*` failure on agents where `build-all-versions-install-deps` did **not** run.
Both symptoms in the logs — `Parameter 3 does not have length 1` (repeated, in the metadata DB queries) and `argument is of length zero` (system-dependency install) — were the same bug: inside bincraft, `platform` was zero-length. `local/build-all.R` calls `build_binary_package()` without passing `platform`, so bincraft resolves it from the container codename. The primary build step never (re)installed/pinned bincraft and relied on whatever sat in the **per-agent** cache volume; `depends_on` only orders steps, it does not co-locate them on the same agent, so a job landing where install-deps never ran got a stale bincraft that left `platform` empty.
## Changes
- **`build-all-versions.yaml`**: pin bincraft `@v4.2.3` in the primary build step (mirroring the R-minor pass and `process-updates.yaml`), so every agent uses a known-good bincraft regardless of where install-deps ran.
- Align the R-minor pass `v4.2.2 → v4.2.3`.
- Export `OS`/`OS_VERSION`/`ARCH` as runtime env vars — previously only available for `${...}` interpolation, so `build-all.R`'s already-built dedup query matched platform `"-"` and skipped nothing.
- Fix the unarchive call: pass the codename via `bincraft::set_codename(NULL)` instead of the malformed `paste(OS, OS_VERSION)` (`"alpine 3.24"`), matching `archive-missed-packages.yaml`.
- **`build-all-versions-install-deps.yaml`**: pin install-deps to `@v4.2.3` (was installing HEAD), so the precomputed snapshot and per-agent library stay consistent pipeline-wide.
Reviewed-on: #96
|
|||
|
f9bc72d5ab |
chore: bump bincraft to 4.2.3
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
|
|||
| 4c67703f52 |
refactor(ci): route workflows by agent group label instead of named agent (#94)
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline failed
## Summary
- Switch crow workflow placement from a per-agent label (`agent: ${AGENT}`) to a group label (`group: rpkgs-${ARCH}`), so jobs target the `rpkgs-amd64` / `rpkgs-arm64` agent pools instead of a single named agent (`artemis`/`gaia`).
- Remove the now-unused `AGENT` matrix variable from all build/process/weekly workflows.
- Update header comments to reflect group-based placement.
Agents opt into a pool via `CROW_AGENT_LABELS="group=rpkgs-<arch>"`. The group name derives from each matrix row's `ARCH`, so adding or replacing agents no longer requires touching the workflow files.
Reviewed-on: #94
|
|||
|
3875612237 |
fix(ci): restore xwfb-run/xvfb-run detection for rhel-10 process-updates
Some checks failed
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline failed
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline failed
ci/crow/cron/process-updates/3 Pipeline failed
ci/crow/cron/process-updates/8 Pipeline was successful
The matrix consolidation hardcoded `xvfb-run`, which is absent on the rhel-10 image (it ships Xwayland's xwfb-run), causing `xvfb-run: command not found`. Re-add the per-platform launcher detection the old rhel-10 files used: prefer xwfb-run + weston, fall back to xvfb-run. |
|||
|
76f677ba5b |
fix(ci): quote matrix PROCESS_NEW so R receives uppercase FALSE/TRUE
Unquoted `PROCESS_NEW: FALSE` is parsed as a YAML boolean and interpolated into the R command as lowercase `false`, causing `process_new = false` and `Error: object 'false' not found`. Quoting keeps the literal FALSE/TRUE that R expects. |
|||
|
dc92bd6a2c |
fix(ci): restore cron triggers via per-row cron name filter
The matrix consolidation routed cron events through `evaluate: 'CI_PIPELINE_CRON == "..."'`, but CI_PIPELINE_CRON is not exposed to the when.evaluate constraint context, so no matrix row ever matched and the scheduled process-updates / weekly runs stopped firing. Switch back to the first-class `cron:` name filter (still parameterized per matrix row) which the server matches directly against the fired cron. |
|||
| b297f0fbc0 |
refactor(ci): consolidate build-all-versions workflows into 2 files (#93)
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
## Summary
Consolidate the four `build-all-versions*` files into two, mirroring the matrix approach used for the process-updates / weekly families:
| Before | After |
|---|---|
| `build-all-versions-amd64.yaml` + `build-all-versions-arm64.yaml` | `build-all-versions.yaml` |
| `build-all-versions-install-deps-amd64.yaml` + `-arm64.yaml` | `build-all-versions-install-deps.yaml` |
`ARCH` becomes a matrix axis carrying the per-arch differences: `agent` (artemis/gaia), `BACKEND` (kubernetes/docker), and `R_PKG_CACHE_DIR`. Build-all keeps its `SPLIT_INDEX` parallelism, so its matrix is ARCH x SPLIT = 8 rows; only the rows matching the selected arch run.
**Invocation unchanged.** Routing uses the existing task selector `task == "build-all-${ARCH}"`, so `--var task=build-all-amd64 ...` still runs exactly the amd64 rows. `OS` / `OS_VERSION` / `R_VERSION` / `K8S_INSTANCE_TYPE` are still passed via `--var`. (You can drop `--var ARCH=...` now — arch comes from the matrix/task.)
**No cron.** These pipelines are manual-only, so there are no scheduled runs to affect.
## Validation before merge
Same matrix-interpolation-in-`when`/`labels` mechanism validated for #92, plus it now relies on per-row `agent`/`BACKEND` selecting the right execution backend (kubernetes for amd64, docker for arm64). Before merge, on this branch trigger a small `task=build-all-arm64` run and confirm it lands on the gaia/docker backend (and amd64 on artemis/kubernetes). Originals are in git history for rollback.
Note: `archive-missed-packages.yaml` was already a single consolidated matrix file (cron + manual) and is left untouched.
Reviewed-on: #93
|
|||
| 4406a8735b |
refactor(ci): consolidate per-platform crow workflows into 3 matrix files (#92)
## Summary Collapse **44** per-platform crow pipeline files into **3** matrix-driven files (one per family), using `matrix.include` + the crow #1165 declarative manual `variables:` block: | Family | Before | After | |---|---|---| | process-updates | 14 | `process-updates.yaml` | | weekly-audit-missing | 16 | `weekly-audit-missing.yaml` | | weekly-rebuild-missing | 14 | `weekly-rebuild-missing.yaml` | - **One file per family** (not split by arch). Arch placement is via the agent label as a matrix var (`artemis`=amd64, `gaia`=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-`nodeSelector` placement (no agent label). - **Matrix axis is named `OS`** (e.g. `redhat-9`) per request, with per-row `R_VERSION`, image tag, codename, and `process_new` (FALSE for alpine). - **Runs preserved 1:1:** - *Cron*: each existing `<family>-<os>-<arch>` cron still fires only its matching matrix row via `CI_PIPELINE_CRON`. Server-side cron entries unchanged. - *Manual*: a #1165 dropdown variable (`process_cran_updates` / `weekly_audit_missing` / `weekly_rebuild_missing`) selects a single `<os>-<arch>` or `all`. weekly-rebuild defaults to `all`, matching its former bare `event: manual` trigger. ## Behavior deltas (intentional, flagged) To fit one file per family the backend was unified: - The amd64-only `node.kubernetes.io/instance-type: AX42` nodeSelector pin is **dropped**; placement now relies on the `artemis`/`gaia` agents. - **process-updates arm64** jobs now share the same `backend_options` as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped. ## Validation required before merge Relies on crow interpolating matrix variables inside `when.evaluate`, `labels.agent`, and `commands`. Interpolation in image/env/commands is standard crow; the novel bit is `when.evaluate`. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch: 1. Trigger manually with the dropdown = `redhat-9-amd64` -> confirm only that one job runs (image `build-env-redhat:9`, R 4.4.3, agent artemis). 2. Trigger with `all` -> confirm all os/arch jobs schedule. Crons only fire on the default branch, so no collision while unmerged. Fallback if `${...}` doesn't interpolate in `when`: a runtime `case "$CI_PIPELINE_CRON" in ...` guard, same files otherwise. Reviewed-on: #92 |
|||
| 1cd86b65e1 |
feat(build): skip already-built package versions on workflow restart (#91)
All checks were successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
## Summary When a `build-all-*` workflow is restarted, the build job re-reads the static `pkgs_to_build.rds` that the install-deps step produced once, so it cycles over every package an interrupted run already built. This adds a DB-based skip filter so a restart only processes what is genuinely left. - At job start, `build-all.R` queries the `single_builds` metadata table for `(name, tag)` already built successfully (`error_occurred = FALSE`) on this `platform`/`arch`, and drops those pairs from the chunk before the build loop. It logs how many it skipped. - One indexed query, one round trip, run before the pak forks — no extra S3 listing and no new Python/s3fs memory pressure (`RPostgres`/`DBI` are already used in the container). - Errored versions are intentionally **not** skipped, so transient failures still get retried on restart. ## Dependency Correctness depends on a `error_occurred = FALSE` row meaning the binary is actually published. That guarantee is added in rpkgs/bincraft#56 (success row written only after a confirmed S3 upload). This PR should land together with / after a bincraft release including that fix. Reviewed-on: #91 |
|||
|
8cf8e33bf5 |
fix(ci): build per-minor binaries with a per-minor R library
All checks were successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
The sensitive-only multi-R build passes ran under each minor's R but loaded bincraft from the shared R_LIBS_USER=/mnt/cache/R-pkgs cache, which is compiled under the primary R and fails to load under a newer minor (e.g. 4.6: "undefined symbol: SETLENGTH") — so those minor builds silently no-op'd. Give each non-primary minor its own R_LIBS_USER=/mnt/cache/R-pkgs-<minor> and install bincraft 4.2.1 there (built for that minor); the rest of bincraft's deps load from the per-version /opt/R/<minor>/lib. Applied to the build-all-versions and process-updates sensitive-only passes. |
|||
|
61a9ac5609 |
fix(ci): run per-minor index under the primary R in process-updates
The per-minor upload_package_index() loop invoked each minor's own R, which
loads bincraft from the shared /mnt/cache/R-pkgs cache (compiled under the
primary R). Under a newer minor (e.g. 4.6) that fails to load
("undefined symbol: SETLENGTH") and the index step dies. Indexing is pure S3
work and needs no specific R, so run it under /opt/R/$R_VERSION like the generic
index and build-all-versions already do.
|
|||
|
95bcf6b377 |
chore: build alpine 3.24
Some checks failed
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps-amd64 Pipeline was successful
ci/crow/manual/build-all-versions-amd64/1 Pipeline failed
ci/crow/manual/build-all-versions-amd64/4 Pipeline failed
ci/crow/manual/build-all-versions-amd64/3 Pipeline failed
ci/crow/manual/build-all-versions-amd64/2 Pipeline failed
ci/crow/manual/build-all-versions-install-deps-arm64 Pipeline was successful
ci/crow/manual/build-all-versions-arm64/4 Pipeline failed
ci/crow/manual/build-all-versions-arm64/1 Pipeline failed
ci/crow/manual/build-all-versions-arm64/2 Pipeline failed
ci/crow/manual/build-all-versions-arm64/3 Pipeline failed
|
|||
|
f51617018d |
ci: bump bincraft to 4.2.1 and scope multi-R loops to versioned R dirs
- Pin bincraft v4.2.0 -> v4.2.1 (per-minor archive.rds/index fix) across the process-updates, weekly-rebuild and archive workflows. - Restrict the per-R-minor build and index loops to /opt/R/[0-9]* so the `current` symlink dir is not treated as a distinct minor (avoids double- building a minor and writing a bogus contrib/current/ index slot). |
|||
| d558e27c11 |
feat: R-minor-sensitive binary builds (full + iterative) (#84)
## Summary
Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered.
## What's in here
**Detection / precompute**
- `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests.
- `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions.
**Full build**
- `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset.
- `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor.
**Iterative build**
- All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added.
**Tooling / housekeeping**
- Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`.
- Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts.
- Design + implementation docs under `docs/superpowers/`.
## Requires before merge
- A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected.
Reviewed-on: #84
|
|||
|
|
f28ccaf008 |
chore(deps): update reg.devxy.io/rpkgs/build-env-alpine docker tag to v3.24
Some checks failed
ci/crow/cron/weekly-audit-missing-redhat-9-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-redhat-10-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-redhat-10-arm64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline failed
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline is running
ci/crow/cron/weekly-rebuild-missing-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline is running
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-322-arm64 Pipeline failed
ci/crow/cron/weekly-rebuild-missing-alpine-322-amd64 Pipeline was successful
|
||
|
|
d018f31376 | chore(deps): update dependency bincraft to v4.2.0 | ||
| 4242e63f30 |
refactor(ci): simplify process-updates cache cleanup (#76)
All checks were successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-alpine-322-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
## Summary
The `rm -rf /mnt/cache/...` block in 14 `process-updates-*` files had two real issues:
1. `/mnt/cache/pkgcache/R/pkgcache` listed twice — harmless duplicate.
2. `/mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres` — missing a space between two paths, so `rm` walks a nested nonsense path that never exists. The `00LOCK-RPostgres` lock cleanup never actually ran.
Collapses to the same shape `weekly-rebuild-missing-*` already uses:
```
rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/{bincraft,pkgcache} /mnt/cache/pkgcache/R/pkgcache
```
`00LOCK-*` covers every stale lock (pak, bincraft, RPostgres, r-pkg-binaries, …) without enumerating them, so adding a new locked package later doesn't need another YAML touch.
14 files, 1 line changed each.
Reviewed-on: #76
|
|||
|
|
81f585f7f5 |
chore(deps): update reg.devxy.io/docker.io/library/alpine docker tag to v3.24
Some checks failed
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
|
||
| 4425904eeb |
feat(ci): archive missed packages for redhat-10 and alpine323 (#75)
## Summary
`archive-missed-packages.yaml`'s matrix was last touched when redhat-9 and alpine322 were the newest targets. redhat-10 and alpine323 builds are now wired up across `process-updates-*` and `weekly-rebuild-missing-*` workflows, but the daily archive sweep wasn't catching them — so missed-archive packages on those platforms accumulate untouched.
Adds the 4 missing entries:
- `redhat-10` × `{amd64, arm64}`
- `alpine323` × `{amd64, arm64}`
Reviewed-on: #75
|
|||
| 53da568749 |
refactor: single source of truth for excluded packages (#79)
All checks were successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
## Summary
`local/build-all.R` had a 75-name hardcoded `exclude <- c(...)` vector that had drifted from `local/excluded-packages.json` — `RcmdrPlugin.ROC` was in the R vector but missing from the JSON.
`.crow/weekly-rebuild-missing-*.yaml` already reads the JSON via `jsonlite::fromJSON(...)[["package"]]`. This brings `build-all.R` in line with that pattern.
Changes:
- **`local/excluded-packages.json`**: add the missing `RcmdrPlugin.ROC` entry (reason `"hang"`, matching siblings).
- **`local/build-all.R`**: replace the 16-line hardcoded vector with one `jsonlite::fromJSON(...)` call.
- **`.crow/build-all-versions-install-deps-{amd,arm}64.yaml`**: add `jsonlite` to the install-deps `pak::pak()` list so it's available in `/mnt/cache/R-pkgs` for the build step.
- **`.crow/build-all-versions-arm64.yaml`**: drop the dead base64-encoded `SKIP_PKGS` docs comment that nobody was passing as a `--var` anyway; replace with a one-line pointer to the JSON.
Reviewed-on: #79
|
|||
|
2ba6ab28f7 |
chore: bump to bincraft 4.1.1
Some checks failed
ci/crow/cron/weekly-audit-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-322-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-322-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline failed
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
|
|||
| b1bed68898 |
refactor(ci): skip bincraft reinstall when image already has it (#72)
Some checks failed
ci/crow/cron/weekly-audit-missing-redhat-8-amd64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-8-arm64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-9-amd64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-9-arm64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-10-amd64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-10-arm64 Pipeline failed
ci/crow/cron/weekly-audit-missing-ubuntu-2204-amd64 Pipeline is running
ci/crow/cron/weekly-audit-missing-ubuntu-2204-arm64 Pipeline failed
ci/crow/cron/weekly-audit-missing-ubuntu-2404-amd64 Pipeline failed
ci/crow/cron/weekly-audit-missing-ubuntu-2404-arm64 Pipeline failed
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-amd64 Pipeline failed
ci/crow/cron/weekly-rebuild-missing-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-arm64 Pipeline failed
ci/crow/cron/weekly-rebuild-missing-alpine-322-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-322-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
## Summary
bincraft is baked into the build-env images, but every workflow still unconditionally calls `pak::pak("git::.../bincraft.git")` (~30–60s per job).
Pin bincraft to a specific tag (currently `v4.0.2`) and guard the install on a version check.
```
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.0.2") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.0.2")'
```
- Image matches the pin → skip install entirely.
- Image is missing bincraft or has a different version → install the pinned tag (covers both broken bakes and intentional version bumps).
- Renovate customManager keeps the pin and the version-check string in sync whenever bincraft cuts a new tag (uses `git-tags` datasource against the bincraft repo).
Touches 43 `.crow` YAMLs + `renovate.json`.
Reviewed-on: #72
|
|||
| 9cbb43dada |
refactor(ci): drop dedicated update-package-index-* chain (#74)
## Summary `process-updates-*` already calls `bincraft::upload_package_index()` at the end of every daily delta run, and with #71 the CDN purge happens there too. The dedicated `update-package-index-*` chain re-ran the same `upload_package_index()` against the same S3 prefixes — once per platform per day — and was serialized via `depends_on:` purely to dodge S3 rate limits. Net effect: ~40 min cold rebuild × 14 platforms = ~9 h/day of duplicated work, with no consumer waiting on it (process-updates had already published the fresh index). This deletes the 14 chain files (~1.1k lines). The matching `update-package-index` / `update-package-index-*` cron entries on the Crow side become inert with the YAMLs gone and can be removed in a follow-up cron-config change. Best reviewed alongside (or after): - #71 — purge CDN after process-updates - #72 — guard bincraft reinstall - #73 — drop redundant cranlike/desc installs ## Risks - If `process-updates-*` ever fails for a given platform, the chain used to act as a daily safety net for the PACKAGES index. Failures will now surface only via the missing daily push instead of being silently masked. The existing `weekly-audit-missing-*` workflow continues to flag stale state. - `local/manual-package-index-update.R` still exists for one-off manual rebuilds. Reviewed-on: #74 |
|||
| e620bb1c33 |
refactor(ci): drop redundant cranlike/desc installs (#73)
## Summary
`bincraft`'s DESCRIPTION already pulls both forks via `Remotes:`:
```
Imports:
...
cranlike,
...
Remotes:
pat-s/cranlike@s3,
pat-s/desc@description-from-remote
```
So `pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")` resolves the forks and installs them transitively. The explicit follow-up `pak::pak("pat-s/cranlike@s3")` and `pak::pak("pat-s/desc@description-from-remote")` calls in `update-package-index-*.yaml` are dead weight.
Drops the two lines from all 14 `update-package-index-*` files.
`process-updates-*.yaml` already only installs `bincraft` — this brings the two workflows in sync.
Reviewed-on: #73
|
|||
| 494730be5d |
feat(ci): purge CDN cache after process-updates runs (#71)
## Summary
- `process_cran_updates()` ends with `upload_package_index()`, refreshing `src/contrib/PACKAGES{,.gz,.rds,.db}` in S3 — but no CDN purge follows.
- The CDN was only purged by the daily `update-package-index-*` chain, so between cron runs the edge kept serving stale indexes. A binary that landed in S3 was effectively invisible to `install.packages()` until the index-update chain caught up.
Adds the same `Purge CDN cache` step that `update-package-index-*` already uses, parameterized per platform/arch. The script and image are unchanged.
Reviewed-on: #71
|
| Author | SHA1 | Date | |
|---|---|---|---|
| 77ce1d212d | |||
| 0da29ab3f6 | |||
| ee15c50f53 | |||
|
736564a4ff |
|||
|
61e1c0188d |
|||
| a30c6f9532 | |||
| 13fcad6dc4 | |||
| 875b086122 | |||
|
ed1485c542 |
|||
| 1bca6004fd | |||
| 4bca17e4ac | |||
| 21a2fe9e6c | |||
|
77508f8fc5 |
|||
| af481f5129 | |||
| a4b274f281 | |||
| 1ad7a6bfe9 | |||
|
b4ec6291a9 |
|||
|
d496029a79 |
|||
| b93d1ef6bd | |||
|
00db47398d |
|||
| 55a18fd87d | |||
|
a1d7064fb8 |
|||
|
b46472b59e |
|||
|
3420523849 |
|||
| 83c761b736 | |||
| 0f4330c05c | |||
|
f9bc72d5ab |
|||
| 4c67703f52 | |||
|
3875612237 |
|||
|
76f677ba5b |
|||
|
dc92bd6a2c |
|||
| b297f0fbc0 | |||
| 4406a8735b | |||
| 1cd86b65e1 | |||
|
8cf8e33bf5 |
|||
|
61a9ac5609 |
|||
|
95bcf6b377 |
|||
|
f51617018d |
|||
| d558e27c11 | |||
|
|
f28ccaf008 | ||
|
|
d018f31376 | ||
| 4242e63f30 | |||
|
|
81f585f7f5 | ||
| 4425904eeb | |||
| 53da568749 | |||
|
2ba6ab28f7 |
|||
| b1bed68898 | |||
| 9cbb43dada | |||
| e620bb1c33 | |||
| 494730be5d |