908 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| b744ae1bb1 |
fix: build-one image bincraft v4.4.0 + ship patch registry (#104)
## Summary Fixes the **targeted rebuild** path (`just rebuild` → `docker/build-one.Dockerfile`) so it actually applies patches. Two problems, both of which would make `build-one.R`'s `patches = "local/patches"` a silent no-op or a hard error: 1. **Stale bincraft pin.** The Dockerfile pinned bincraft `v4.2.1`, which predates the `patches` argument — so `build_binary_package(patches = ...)` would fail with `unused argument`. Bumped to **v4.4.0** to match the `.crow` workflows. 2. **Registry not shipped into the image.** Only `build-one.R` was copied in; `local/patches/registry.json` was absent, so `patches = "local/patches"` resolved to a nonexistent `/work/local/patches` and silently applied nothing. Added `COPY patches /work/local/patches` (build context is `local/`, CWD is `/work`). ## Why This is the path used to verify the patching end-to-end, e.g.: ```bash just rebuild alpine 3.23 amd64 rts2 1.0.3 ``` rts2 depends on RcppParallel; with this fix the container installs bincraft v4.4.0, ships the registry, and the patched RcppParallel binary (`RCPP_PARALLEL_USE_TBB=0`) is served to pak during dependency install. Expect `Applying patch to RcppParallel …` in the log, RcppParallel installed as a binary (no recompile), then rts2 building and uploading. Reviewed-on: #104 |
|||
|
00db47398d |
chore: 4.4.0 instead of 4.3.1
All checks were successful
ci/crow/cron/process-updates/6 Pipeline was successful
|
|||
| 55a18fd87d |
feat: patch registry + wiring for per-package patching (#103)
Some checks failed
ci/crow/cron/process-updates/15 Pipeline failed
ci/crow/cron/process-updates/14 Pipeline failed
ci/crow/cron/process-updates/16 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/5 Pipeline failed
## Summary Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`). Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading. ## What's included - `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs. - `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace). - `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes). - `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`. - `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`. ## ⚠️ Merge ordering (blocker) This PR adds a `patches = ...` argument to `build_binary_package()` calls. The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until: 1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and 2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`. The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then. Reviewed-on: #103 |
|||
|
|
1e910bc703 |
chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.9.1
Some checks failed
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline failed
ci/crow/manual/build-all-versions/5 Pipeline failed
ci/crow/manual/build-all-versions/8 Pipeline failed
ci/crow/manual/build-all-versions/7 Pipeline failed
|
||
|
|
597f759bb2 | chore(deps): update pre-commit hook editorconfig-checker/editorconfig-checker to v3.8.0 | ||
|
|
3ffa25ddac |
chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.8.5
Some checks failed
ci/crow/cron/weekly-audit-missing/9 Pipeline was successful
ci/crow/cron/weekly-audit-missing/10 Pipeline failed
ci/crow/cron/weekly-audit-missing/11 Pipeline was successful
ci/crow/cron/weekly-audit-missing/13 Pipeline is running
ci/crow/cron/weekly-rebuild-missing/11 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-audit-missing/12 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/12 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/14 Pipeline was successful
ci/crow/manual/build-all-versions/7 Pipeline is pending
ci/crow/manual/build-all-versions/8 Pipeline is pending
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline was canceled
ci/crow/manual/build-all-versions/5 Pipeline was canceled
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was canceled
ci/crow/cron/process-updates/18 Pipeline was canceled
ci/crow/cron/process-updates/14 Pipeline was canceled
ci/crow/cron/process-updates/2 Pipeline was canceled
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline failed
|
||
|
|
81ca78edb6 |
chore(deps): update terraform bunnynet to ~> 0.15
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/2 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/manual/build-all-versions/5 Pipeline failed
ci/crow/manual/build-all-versions/6 Pipeline failed
ci/crow/manual/build-all-versions/8 Pipeline failed
ci/crow/manual/build-all-versions/7 Pipeline failed
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline failed
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline is running
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline failed
ci/crow/cron/process-updates/9 Pipeline failed
ci/crow/cron/process-updates/2 Pipeline failed
ci/crow/cron/process-updates/8 Pipeline failed
ci/crow/cron/weekly-audit-missing/3 Pipeline was successful
ci/crow/cron/weekly-audit-missing/4 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline failed
|
||
|
a1d7064fb8 |
feat(ci): add process-updates rows for alpine-324 and ubuntu resolute
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline was successful
ci/crow/manual/build-all-versions/5 Pipeline was successful
ci/crow/manual/build-all-versions/7 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/build-all-versions/8 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/build-all-versions/2 Pipeline failed
Add matrix rows and manual dropdown options for alpine 3.24 (alpine-324, PROCESS_NEW=FALSE) and ubuntu resolute / 26.04 (ubuntu-2604, OS_ID resolute, R 4.5.3, PROCESS_NEW=TRUE). |
|||
|
b46472b59e |
fix(ci): pass Backblaze B2 credentials to process_unarchived_pkgs
All checks were successful
ci/crow/cron/process-updates/12 Pipeline was successful
The function defaults to Hetzner S3 with HETZNER_S3_*_K3S env vars that are not set in these workflows, so paws failed with "No compatible credentials provided". Pass the B2 endpoint/region/bucket and B2_S3_* secrets explicitly, matching the other S3 calls. Also use set_codename(NULL) for the build-all-versions codename so it matches the S3 repo path. |
|||
|
|
a6788f9044 |
chore(deps): update pre-commit hook posit-dev/air-pre-commit to v0.10.0
Some checks failed
ci/crow/cron/weekly-audit-missing/15 Pipeline was successful
ci/crow/cron/weekly-audit-missing/16 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/7 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/8 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/5 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/6 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/11 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/12 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/10 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline failed
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/14 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/9 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
|
||
|
3420523849 |
chore: add all options for OS_VERSION
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
|
|||
| 83c761b736 |
feat(ci): OS/OS_VERSION manual dropdowns + restore lost crow fix (#97)
## Summary
Two changes:
1. **`OS`/`OS_VERSION` manual-run dropdowns** — give these form variables explicit `options:` lists (like `target_arch` and `R_VERSION`), so the manual-run form shows dropdowns instead of free-text, in both `build-all-versions.yaml` and `build-all-versions-install-deps.yaml`. Crow form variables are independent (no cascading), so the operator still has to pick a coherent `OS` + `OS_VERSION` combination (e.g. `redhat` + `9`, not `alpine` + `jammy`).
2. **Repairs `main`** — the crow fix from PR #96 (`
|
|||
| 0f4330c05c |
fix(ci): make build-all self-sufficient on agents without install-deps (#96)
## Summary
Fixes the recurring `build-all-*` failure on agents where `build-all-versions-install-deps` did **not** run.
Both symptoms in the logs — `Parameter 3 does not have length 1` (repeated, in the metadata DB queries) and `argument is of length zero` (system-dependency install) — were the same bug: inside bincraft, `platform` was zero-length. `local/build-all.R` calls `build_binary_package()` without passing `platform`, so bincraft resolves it from the container codename. The primary build step never (re)installed/pinned bincraft and relied on whatever sat in the **per-agent** cache volume; `depends_on` only orders steps, it does not co-locate them on the same agent, so a job landing where install-deps never ran got a stale bincraft that left `platform` empty.
## Changes
- **`build-all-versions.yaml`**: pin bincraft `@v4.2.3` in the primary build step (mirroring the R-minor pass and `process-updates.yaml`), so every agent uses a known-good bincraft regardless of where install-deps ran.
- Align the R-minor pass `v4.2.2 → v4.2.3`.
- Export `OS`/`OS_VERSION`/`ARCH` as runtime env vars — previously only available for `${...}` interpolation, so `build-all.R`'s already-built dedup query matched platform `"-"` and skipped nothing.
- Fix the unarchive call: pass the codename via `bincraft::set_codename(NULL)` instead of the malformed `paste(OS, OS_VERSION)` (`"alpine 3.24"`), matching `archive-missed-packages.yaml`.
- **`build-all-versions-install-deps.yaml`**: pin install-deps to `@v4.2.3` (was installing HEAD), so the precomputed snapshot and per-agent library stay consistent pipeline-wide.
Reviewed-on: #96
|
|||
|
f9bc72d5ab |
chore: bump bincraft to 4.2.3
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
|
|||
| fa955c10bf |
fix: recompute package snapshot when missing from cache (#95)
Some checks failed
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline failed
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions/5 Pipeline was canceled
ci/crow/manual/build-all-versions/6 Pipeline was canceled
ci/crow/manual/build-all-versions/7 Pipeline was canceled
ci/crow/manual/build-all-versions/8 Pipeline was canceled
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/2 Pipeline failed
## Summary `build-all.R` reads three snapshot files from `/mnt/cache/packages/` that the `build-all-versions-install-deps` step precomputes: `pkgs_to_build.rds`, `r_minor_sensitive_pkgs.rds`, and `s3_cache.rds`. That cache volume is **per-agent**, so a build job scheduled on a different (fresh) agent than the one that ran install-deps finds the snapshot absent and dies at `readRDS` (`cannot open compressed file '/mnt/cache/packages/pkgs_to_build.rds'`). This adds a conditional guard at the top of `build-all.R`: when any of the three files is missing, it sources `local/packages-to-build.R` (which has all needed creds via `PGPASS` / `B2_S3_*` env, already present in the build step) and writes the derived `.rds` files — exactly mirroring the install-deps command. - The first build job on a fresh agent repopulates the shared cache, so subsequent jobs on that agent reuse it. - Concurrent jobs that also miss simply redo the work (accepted tradeoff vs. slow shared storage like NFS). - Saves use a temp-file + atomic `file.rename`, so a concurrent reader never sees a half-written `.rds`. Reviewed-on: #95 |
|||
| 4c67703f52 |
refactor(ci): route workflows by agent group label instead of named agent (#94)
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline failed
## Summary
- Switch crow workflow placement from a per-agent label (`agent: ${AGENT}`) to a group label (`group: rpkgs-${ARCH}`), so jobs target the `rpkgs-amd64` / `rpkgs-arm64` agent pools instead of a single named agent (`artemis`/`gaia`).
- Remove the now-unused `AGENT` matrix variable from all build/process/weekly workflows.
- Update header comments to reflect group-based placement.
Agents opt into a pool via `CROW_AGENT_LABELS="group=rpkgs-<arch>"`. The group name derives from each matrix row's `ARCH`, so adding or replacing agents no longer requires touching the workflow files.
Reviewed-on: #94
|
|||
|
3875612237 |
fix(ci): restore xwfb-run/xvfb-run detection for rhel-10 process-updates
Some checks failed
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline failed
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline failed
ci/crow/cron/process-updates/3 Pipeline failed
ci/crow/cron/process-updates/8 Pipeline was successful
The matrix consolidation hardcoded `xvfb-run`, which is absent on the rhel-10 image (it ships Xwayland's xwfb-run), causing `xvfb-run: command not found`. Re-add the per-platform launcher detection the old rhel-10 files used: prefer xwfb-run + weston, fall back to xvfb-run. |
|||
|
76f677ba5b |
fix(ci): quote matrix PROCESS_NEW so R receives uppercase FALSE/TRUE
Unquoted `PROCESS_NEW: FALSE` is parsed as a YAML boolean and interpolated into the R command as lowercase `false`, causing `process_new = false` and `Error: object 'false' not found`. Quoting keeps the literal FALSE/TRUE that R expects. |
|||
|
dc92bd6a2c |
fix(ci): restore cron triggers via per-row cron name filter
The matrix consolidation routed cron events through `evaluate: 'CI_PIPELINE_CRON == "..."'`, but CI_PIPELINE_CRON is not exposed to the when.evaluate constraint context, so no matrix row ever matched and the scheduled process-updates / weekly runs stopped firing. Switch back to the first-class `cron:` name filter (still parameterized per matrix row) which the server matches directly against the fired cron. |
|||
| b297f0fbc0 |
refactor(ci): consolidate build-all-versions workflows into 2 files (#93)
Some checks failed
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
## Summary
Consolidate the four `build-all-versions*` files into two, mirroring the matrix approach used for the process-updates / weekly families:
| Before | After |
|---|---|
| `build-all-versions-amd64.yaml` + `build-all-versions-arm64.yaml` | `build-all-versions.yaml` |
| `build-all-versions-install-deps-amd64.yaml` + `-arm64.yaml` | `build-all-versions-install-deps.yaml` |
`ARCH` becomes a matrix axis carrying the per-arch differences: `agent` (artemis/gaia), `BACKEND` (kubernetes/docker), and `R_PKG_CACHE_DIR`. Build-all keeps its `SPLIT_INDEX` parallelism, so its matrix is ARCH x SPLIT = 8 rows; only the rows matching the selected arch run.
**Invocation unchanged.** Routing uses the existing task selector `task == "build-all-${ARCH}"`, so `--var task=build-all-amd64 ...` still runs exactly the amd64 rows. `OS` / `OS_VERSION` / `R_VERSION` / `K8S_INSTANCE_TYPE` are still passed via `--var`. (You can drop `--var ARCH=...` now — arch comes from the matrix/task.)
**No cron.** These pipelines are manual-only, so there are no scheduled runs to affect.
## Validation before merge
Same matrix-interpolation-in-`when`/`labels` mechanism validated for #92, plus it now relies on per-row `agent`/`BACKEND` selecting the right execution backend (kubernetes for amd64, docker for arm64). Before merge, on this branch trigger a small `task=build-all-arm64` run and confirm it lands on the gaia/docker backend (and amd64 on artemis/kubernetes). Originals are in git history for rollback.
Note: `archive-missed-packages.yaml` was already a single consolidated matrix file (cron + manual) and is left untouched.
Reviewed-on: #93
|
|||
| 4406a8735b |
refactor(ci): consolidate per-platform crow workflows into 3 matrix files (#92)
## Summary Collapse **44** per-platform crow pipeline files into **3** matrix-driven files (one per family), using `matrix.include` + the crow #1165 declarative manual `variables:` block: | Family | Before | After | |---|---|---| | process-updates | 14 | `process-updates.yaml` | | weekly-audit-missing | 16 | `weekly-audit-missing.yaml` | | weekly-rebuild-missing | 14 | `weekly-rebuild-missing.yaml` | - **One file per family** (not split by arch). Arch placement is via the agent label as a matrix var (`artemis`=amd64, `gaia`=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-`nodeSelector` placement (no agent label). - **Matrix axis is named `OS`** (e.g. `redhat-9`) per request, with per-row `R_VERSION`, image tag, codename, and `process_new` (FALSE for alpine). - **Runs preserved 1:1:** - *Cron*: each existing `<family>-<os>-<arch>` cron still fires only its matching matrix row via `CI_PIPELINE_CRON`. Server-side cron entries unchanged. - *Manual*: a #1165 dropdown variable (`process_cran_updates` / `weekly_audit_missing` / `weekly_rebuild_missing`) selects a single `<os>-<arch>` or `all`. weekly-rebuild defaults to `all`, matching its former bare `event: manual` trigger. ## Behavior deltas (intentional, flagged) To fit one file per family the backend was unified: - The amd64-only `node.kubernetes.io/instance-type: AX42` nodeSelector pin is **dropped**; placement now relies on the `artemis`/`gaia` agents. - **process-updates arm64** jobs now share the same `backend_options` as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped. ## Validation required before merge Relies on crow interpolating matrix variables inside `when.evaluate`, `labels.agent`, and `commands`. Interpolation in image/env/commands is standard crow; the novel bit is `when.evaluate`. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch: 1. Trigger manually with the dropdown = `redhat-9-amd64` -> confirm only that one job runs (image `build-env-redhat:9`, R 4.4.3, agent artemis). 2. Trigger with `all` -> confirm all os/arch jobs schedule. Crons only fire on the default branch, so no collision while unmerged. Fallback if `${...}` doesn't interpolate in `when`: a runtime `case "$CI_PIPELINE_CRON" in ...` guard, same files otherwise. Reviewed-on: #92 |
|||
| 1cd86b65e1 |
feat(build): skip already-built package versions on workflow restart (#91)
All checks were successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
## Summary When a `build-all-*` workflow is restarted, the build job re-reads the static `pkgs_to_build.rds` that the install-deps step produced once, so it cycles over every package an interrupted run already built. This adds a DB-based skip filter so a restart only processes what is genuinely left. - At job start, `build-all.R` queries the `single_builds` metadata table for `(name, tag)` already built successfully (`error_occurred = FALSE`) on this `platform`/`arch`, and drops those pairs from the chunk before the build loop. It logs how many it skipped. - One indexed query, one round trip, run before the pak forks — no extra S3 listing and no new Python/s3fs memory pressure (`RPostgres`/`DBI` are already used in the container). - Errored versions are intentionally **not** skipped, so transient failures still get retried on restart. ## Dependency Correctness depends on a `error_occurred = FALSE` row meaning the binary is actually published. That guarantee is added in rpkgs/bincraft#56 (success row written only after a confirmed S3 upload). This PR should land together with / after a bincraft release including that fix. Reviewed-on: #91 |
|||
|
|
2dddc5fe43 |
chore(deps): update terraform bunnynet to v0.14.3
All checks were successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
|
||
|
|
6c97f34cf5 |
chore(deps): update pre-commit hook posit-dev/air-pre-commit to v0.9.0
All checks were successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
|
||
|
8cf8e33bf5 |
fix(ci): build per-minor binaries with a per-minor R library
All checks were successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
The sensitive-only multi-R build passes ran under each minor's R but loaded bincraft from the shared R_LIBS_USER=/mnt/cache/R-pkgs cache, which is compiled under the primary R and fails to load under a newer minor (e.g. 4.6: "undefined symbol: SETLENGTH") — so those minor builds silently no-op'd. Give each non-primary minor its own R_LIBS_USER=/mnt/cache/R-pkgs-<minor> and install bincraft 4.2.1 there (built for that minor); the rest of bincraft's deps load from the per-version /opt/R/<minor>/lib. Applied to the build-all-versions and process-updates sensitive-only passes. |
|||
|
61a9ac5609 |
fix(ci): run per-minor index under the primary R in process-updates
The per-minor upload_package_index() loop invoked each minor's own R, which
loads bincraft from the shared /mnt/cache/R-pkgs cache (compiled under the
primary R). Under a newer minor (e.g. 4.6) that fails to load
("undefined symbol: SETLENGTH") and the index step dies. Indexing is pure S3
work and needs no specific R, so run it under /opt/R/$R_VERSION like the generic
index and build-all-versions already do.
|
|||
|
7e01f481c0 |
perf(local): skip per-minor passes for non-r-minor-sensitive packages
A package's ABI sensitivity is the same across R minors, so for a non-sensitive package the per-minor loop only ran `ensure_bincraft` (a ~14s bincraft/cranlike install per minor) before build-one.R classified and skipped it. The primary pass now writes a .r_minor_sensitive sentinel when it builds a sensitive package, and the wrapper gates the whole per-minor loop (installs included) on that file — non-sensitive rebuilds no longer touch other minors' libraries. |
|||
|
95bcf6b377 |
chore: build alpine 3.24
Some checks failed
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps-amd64 Pipeline was successful
ci/crow/manual/build-all-versions-amd64/1 Pipeline failed
ci/crow/manual/build-all-versions-amd64/4 Pipeline failed
ci/crow/manual/build-all-versions-amd64/3 Pipeline failed
ci/crow/manual/build-all-versions-amd64/2 Pipeline failed
ci/crow/manual/build-all-versions-install-deps-arm64 Pipeline was successful
ci/crow/manual/build-all-versions-arm64/4 Pipeline failed
ci/crow/manual/build-all-versions-arm64/1 Pipeline failed
ci/crow/manual/build-all-versions-arm64/2 Pipeline failed
ci/crow/manual/build-all-versions-arm64/3 Pipeline failed
|
|||
|
a9ad907f9c |
chore(local): add S3 migration and CRAN-source helper scripts
- migrate-s3-hetzner-to-backblaze.sh: rclone-based bucket migration helper. - find-R-api-packages.sh: scan CRAN package sources for a C-API usage pattern. - query-pkgs-without-old-versions.R, test-package-loading.R: ad-hoc helpers. |
|||
|
3ae4672d66 |
feat(local): just rebuild builds across all installed R minors
- Dockerfile: install bincraft into each R minor, run a primary pass plus a sensitive-only pass under every other /opt/R/[0-9]* minor (deduped by minor), probe/skip xvfb, set GIT_TERMINAL_PROMPT=0; extra-minor failures are non-fatal. - build-one.R: add --sensitive-only mode, log + shallow-clone the ABI classify step, and clear cranlike's stale ./PACKAGES.db before each index refresh (workaround for the "table packages already exists" bug; pending cranlike fix). |
|||
|
e5398af5f5 |
feat(local): just rebuild — TF_VAR secrets, live progress, builder docs
- Accept TF_VAR_-prefixed B2 keys (from direnv); make GITHUB_PAT optional. - Stream the remote build with --progress=plain. - Document creating the artemis/gaia buildx builders with the docker-container driver (the default remote driver does not work with an ssh:// docker host). |
|||
|
f51617018d |
ci: bump bincraft to 4.2.1 and scope multi-R loops to versioned R dirs
- Pin bincraft v4.2.0 -> v4.2.1 (per-minor archive.rds/index fix) across the process-updates, weekly-rebuild and archive workflows. - Restrict the per-R-minor build and index loops to /opt/R/[0-9]* so the `current` symlink dir is not treated as a distinct minor (avoids double- building a minor and writing a bogus contrib/current/ index slot). |
|||
| 7f5674ffb3 |
fix(local): accept TF_VAR_-prefixed B2 keys, optional GITHUB_PAT (#88)
Some checks failed
## Summary Follow-up fix to the `just rebuild` recipe (#87): the local `.envrc` exports the B2 credentials as `TF_VAR_B2_S3_ACCESS_KEY` / `TF_VAR_B2_S3_SECRET_KEY` (and has no `GITHUB_PAT`), but the recipe required the plain `B2_S3_ACCESS_KEY` / `B2_S3_SECRET_KEY` names and always passed a `github_pat` secret. - Accept the `TF_VAR_`-prefixed B2 names (falling back to the plain names if set directly). - Pass the `github_pat` buildx secret only when `GITHUB_PAT` is set, so the build works without it. Reviewed-on: #88 |
|||
| 1224bd74fe |
feat(local): just rebuild recipe for targeted builds via remote buildx (#87)
## Summary Adds a local `just rebuild` recipe to (re)build specific versions of a single package on a given OS/arch, dispatching to a remote buildx builder (the build runs there, not locally). - `just rebuild <os> <tag> <arch> <package> <version>...` → `docker buildx build --builder <artemis|gaia> --platform linux/<arch> …` (amd64→artemis, arm64→gaia; names + `R_VERSION` env-overridable). - `docker/build-one.Dockerfile` runs `build-one.R` as a secret-mounted `RUN`, built `--no-cache --output type=cacheonly` (pure side-effect: the S3 upload; no image kept). - `local/build-one.R` auto-classifies each version via the ABI classifier (risky → per-minor slot `contrib/<x.y>/`, else generic), force-rebuilds + uploads + stores metadata, then refreshes the touched slot's `PACKAGES` index. ## Prerequisites - buildx builders named `artemis` (amd64) and `gaia` (arm64) registered (`docker buildx create --name artemis ssh://…`). - Exported secrets: `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY`, `PGPASS` (`GITHUB_PAT` optional). - bincraft `v4.2.0` tag must exist (the build installs `@v4.2.0` and uses its classifier + per-minor index API). Reviewed-on: #87 |
|||
| d558e27c11 |
feat: R-minor-sensitive binary builds (full + iterative) (#84)
## Summary
Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered.
## What's in here
**Detection / precompute**
- `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests.
- `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions.
**Full build**
- `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset.
- `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor.
**Iterative build**
- All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added.
**Tooling / housekeeping**
- Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`.
- Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts.
- Design + implementation docs under `docs/superpowers/`.
## Requires before merge
- A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected.
Reviewed-on: #84
|
|||
|
|
f28ccaf008 |
chore(deps): update reg.devxy.io/rpkgs/build-env-alpine docker tag to v3.24
Some checks failed
ci/crow/cron/weekly-audit-missing-redhat-9-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-redhat-10-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-redhat-10-arm64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline failed
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline is running
ci/crow/cron/weekly-rebuild-missing-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline is running
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-322-arm64 Pipeline failed
ci/crow/cron/weekly-rebuild-missing-alpine-322-amd64 Pipeline was successful
|
||
|
|
d018f31376 | chore(deps): update dependency bincraft to v4.2.0 | ||
| 4242e63f30 |
refactor(ci): simplify process-updates cache cleanup (#76)
All checks were successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-alpine-322-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
## Summary
The `rm -rf /mnt/cache/...` block in 14 `process-updates-*` files had two real issues:
1. `/mnt/cache/pkgcache/R/pkgcache` listed twice — harmless duplicate.
2. `/mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres` — missing a space between two paths, so `rm` walks a nested nonsense path that never exists. The `00LOCK-RPostgres` lock cleanup never actually ran.
Collapses to the same shape `weekly-rebuild-missing-*` already uses:
```
rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/{bincraft,pkgcache} /mnt/cache/pkgcache/R/pkgcache
```
`00LOCK-*` covers every stale lock (pak, bincraft, RPostgres, r-pkg-binaries, …) without enumerating them, so adding a new locked package later doesn't need another YAML touch.
14 files, 1 line changed each.
Reviewed-on: #76
|
|||
|
a02bd5c4ff |
chore: adjust cdn limits
Some checks are pending
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline is running
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
|
|||
|
44482b18f9 |
chore: cleanup and adjust cdn definitions
All checks were successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
|
|||
|
|
81f585f7f5 |
chore(deps): update reg.devxy.io/docker.io/library/alpine docker tag to v3.24
Some checks failed
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
|
||
| 228459c0ba |
fix(ci): actually purge BunnyCDN cache in purge_cdn_cache.sh (#78)
All checks were successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
## Summary While cleaning this up I noticed the script **defined** `purge_cdn_cache()` but never **called** it. Every CI run only declared the helper and exited cleanly without issuing a single curl. The PACKAGES freshness on cran.devxy.io / cran.rpkgs.com has been carried entirely by R's `Cache-Control: no-cache` header on the index files. Fixes in one shot: 1. **Actually run the purge.** The shifted args (`api_key`, `arch`, `os_id`, `domain…`) are now consumed inline and a curl POST is issued per (domain × resource). 2. **Drop `set -x`** — leaked the `AccessKey:` header into job logs. 3. **Drop the Python URL-encoder.** `curl -G --data-urlencode "url=…" --data "async=false" https://api.bunny.net/purge` does the same with no Python. The workflow purge steps can later switch from `alpine:3.23 + apk add bash curl` to a slimmer curl-only image. 4. **Add `src/contrib/Meta/archive.rds`** to the purged resource list. It's rewritten on every `process_cran_updates` run (see README) and was being served stale. ## Risks - This is the **first time** the script actually purges anything. If anything else (e.g. a downstream service) relied on the no-op behavior, this PR is the moment it stops being silent. I don't see any such caller. - Edge cache miss right after a purge means an origin S3 fetch — minor latency uptick on the first request per region per resource. Reviewed-on: #78 |
|||
| 4425904eeb |
feat(ci): archive missed packages for redhat-10 and alpine323 (#75)
## Summary
`archive-missed-packages.yaml`'s matrix was last touched when redhat-9 and alpine322 were the newest targets. redhat-10 and alpine323 builds are now wired up across `process-updates-*` and `weekly-rebuild-missing-*` workflows, but the daily archive sweep wasn't catching them — so missed-archive packages on those platforms accumulate untouched.
Adds the 4 missing entries:
- `redhat-10` × `{amd64, arm64}`
- `alpine323` × `{amd64, arm64}`
Reviewed-on: #75
|
|||
| 53da568749 |
refactor: single source of truth for excluded packages (#79)
All checks were successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
## Summary
`local/build-all.R` had a 75-name hardcoded `exclude <- c(...)` vector that had drifted from `local/excluded-packages.json` — `RcmdrPlugin.ROC` was in the R vector but missing from the JSON.
`.crow/weekly-rebuild-missing-*.yaml` already reads the JSON via `jsonlite::fromJSON(...)[["package"]]`. This brings `build-all.R` in line with that pattern.
Changes:
- **`local/excluded-packages.json`**: add the missing `RcmdrPlugin.ROC` entry (reason `"hang"`, matching siblings).
- **`local/build-all.R`**: replace the 16-line hardcoded vector with one `jsonlite::fromJSON(...)` call.
- **`.crow/build-all-versions-install-deps-{amd,arm}64.yaml`**: add `jsonlite` to the install-deps `pak::pak()` list so it's available in `/mnt/cache/R-pkgs` for the build step.
- **`.crow/build-all-versions-arm64.yaml`**: drop the dead base64-encoded `SKIP_PKGS` docs comment that nobody was passing as a `--var` anyway; replace with a one-line pointer to the JSON.
Reviewed-on: #79
|
|||
| fb95042d16 |
chore: drop stale Hetzner-pointing helpers (#77)
## Summary Both helpers still point at the old Hetzner storage: - `Justfile` — all 3 recipes (`build-all`, `build-single`, `process-updates`) hit `hel1.your-objectstorage.com` / bucket `devxy-r-package-binaries-hel1` using `HETZNER_S3_*_K3S` env vars. - `local/manual-package-index-update.R` — same endpoint + bucket, hardcoded into `s3fs::s3_dir_ls()` / `s3_file_delete()` calls. Storage moved to Backblaze (`s3.eu-central-003.backblazeb2.com` / `devxy-rpkgs-binaries`) a while back, so running either of these today would write to the wrong bucket or fail outright. No callers reference them in-tree, deleting outright. A one-off rebuild can just call `bincraft::upload_package_index()` directly with current Backblaze settings. Reviewed-on: #77 |
|||
| 35dafab737 |
refactor: hoist CDN block_ips to a variable (#81)
## Summary `block_ips = ["185.172.53.0"]` was duplicated in both `bunnynet_pullzone` resources in `cdn.tf`. Move it to a typed `cdn_block_ips` variable in `vars.tf` with the existing IP as the default. - No plan diff on apply — same value, just sourced from `var.cdn_block_ips` instead of a literal. - Adding/removing IPs is now a one-line var override (or a default change) instead of two edits in the resource bodies. ## Interaction with #80 (for_each refactor) `#80` collapses the two pullzones to `bunnynet_pullzone.this[for_each]`. Either order works; whichever lands second is a trivial one-line rebase on the surviving `block_ips =` line. Reviewed-on: #81 |
|||
|
2ba6ab28f7 |
chore: bump to bincraft 4.1.1
Some checks failed
ci/crow/cron/weekly-audit-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-audit-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-322-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-322-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline failed
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
|
|||
| b1bed68898 |
refactor(ci): skip bincraft reinstall when image already has it (#72)
Some checks failed
ci/crow/cron/weekly-audit-missing-redhat-8-amd64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-8-arm64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-9-amd64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-9-arm64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-10-amd64 Pipeline failed
ci/crow/cron/weekly-audit-missing-redhat-10-arm64 Pipeline failed
ci/crow/cron/weekly-audit-missing-ubuntu-2204-amd64 Pipeline is running
ci/crow/cron/weekly-audit-missing-ubuntu-2204-arm64 Pipeline failed
ci/crow/cron/weekly-audit-missing-ubuntu-2404-amd64 Pipeline failed
ci/crow/cron/weekly-audit-missing-ubuntu-2404-arm64 Pipeline failed
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-323-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-8-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-9-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-amd64 Pipeline failed
ci/crow/cron/weekly-rebuild-missing-redhat-9-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-redhat-10-arm64 Pipeline failed
ci/crow/cron/weekly-rebuild-missing-alpine-322-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-alpine-322-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
## Summary
bincraft is baked into the build-env images, but every workflow still unconditionally calls `pak::pak("git::.../bincraft.git")` (~30–60s per job).
Pin bincraft to a specific tag (currently `v4.0.2`) and guard the install on a version check.
```
- /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.0.2") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.0.2")'
```
- Image matches the pin → skip install entirely.
- Image is missing bincraft or has a different version → install the pinned tag (covers both broken bakes and intentional version bumps).
- Renovate customManager keeps the pin and the version-check string in sync whenever bincraft cuts a new tag (uses `git-tags` datasource against the bincraft repo).
Touches 43 `.crow` YAMLs + `renovate.json`.
Reviewed-on: #72
|
|||
| 9cbb43dada |
refactor(ci): drop dedicated update-package-index-* chain (#74)
## Summary `process-updates-*` already calls `bincraft::upload_package_index()` at the end of every daily delta run, and with #71 the CDN purge happens there too. The dedicated `update-package-index-*` chain re-ran the same `upload_package_index()` against the same S3 prefixes — once per platform per day — and was serialized via `depends_on:` purely to dodge S3 rate limits. Net effect: ~40 min cold rebuild × 14 platforms = ~9 h/day of duplicated work, with no consumer waiting on it (process-updates had already published the fresh index). This deletes the 14 chain files (~1.1k lines). The matching `update-package-index` / `update-package-index-*` cron entries on the Crow side become inert with the YAMLs gone and can be removed in a follow-up cron-config change. Best reviewed alongside (or after): - #71 — purge CDN after process-updates - #72 — guard bincraft reinstall - #73 — drop redundant cranlike/desc installs ## Risks - If `process-updates-*` ever fails for a given platform, the chain used to act as a daily safety net for the PACKAGES index. Failures will now surface only via the missing daily push instead of being silently masked. The existing `weekly-audit-missing-*` workflow continues to flag stale state. - `local/manual-package-index-update.R` still exists for one-off manual rebuilds. Reviewed-on: #74 |
|||
| e620bb1c33 |
refactor(ci): drop redundant cranlike/desc installs (#73)
## Summary
`bincraft`'s DESCRIPTION already pulls both forks via `Remotes:`:
```
Imports:
...
cranlike,
...
Remotes:
pat-s/cranlike@s3,
pat-s/desc@description-from-remote
```
So `pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")` resolves the forks and installs them transitively. The explicit follow-up `pak::pak("pat-s/cranlike@s3")` and `pak::pak("pat-s/desc@description-from-remote")` calls in `update-package-index-*.yaml` are dead weight.
Drops the two lines from all 14 `update-package-index-*` files.
`process-updates-*.yaml` already only installs `bincraft` — this brings the two workflows in sync.
Reviewed-on: #73
|