903 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
automation-bot	3ffa25ddac	chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.8.5	2026-06-28 00:32:21 +00:00
automation-bot	81ca78edb6	chore(deps): update terraform bunnynet to ~> 0.15	2026-06-23 00:31:42 +00:00
pat-s	a1d7064fb8	feat(ci): add process-updates rows for alpine-324 and ubuntu resolute ... Add matrix rows and manual dropdown options for alpine 3.24 (alpine-324, PROCESS_NEW=FALSE) and ubuntu resolute / 26.04 (ubuntu-2604, OS_ID resolute, R 4.5.3, PROCESS_NEW=TRUE).	2026-06-21 11:04:24 +02:00
pat-s	b46472b59e	fix(ci): pass Backblaze B2 credentials to process_unarchived_pkgs ... The function defaults to Hetzner S3 with HETZNER_S3_*_K3S env vars that are not set in these workflows, so paws failed with "No compatible credentials provided". Pass the B2 endpoint/region/bucket and B2_S3_* secrets explicitly, matching the other S3 calls. Also use set_codename(NULL) for the build-all-versions codename so it matches the S3 repo path.	2026-06-21 10:55:22 +02:00
automation-bot	a6788f9044	chore(deps): update pre-commit hook posit-dev/air-pre-commit to v0.10.0	2026-06-20 00:31:34 +00:00
pat-s	3420523849	chore: add all options for OS_VERSION	2026-06-18 13:40:13 +02:00
pat-s	83c761b736	feat(ci): OS/OS_VERSION manual dropdowns + restore lost crow fix (#97) ... ## Summary Two changes: 1. **`OS`/`OS_VERSION` manual-run dropdowns** — give these form variables explicit `options:` lists (like `target_arch` and `R_VERSION`), so the manual-run form shows dropdowns instead of free-text, in both `build-all-versions.yaml` and `build-all-versions-install-deps.yaml`. Crow form variables are independent (no cascading), so the operator still has to pick a coherent `OS` + `OS_VERSION` combination (e.g. `redhat` + `9`, not `alpine` + `jammy`). 2. **Repairs `main`** — the crow fix from PR #96 (`bc2f6f1`) was lost when that PR was squashed (only the first commit was captured). As a result `main` currently carries the `OS: ${OS}` env vars that break Crow parsing (`unable to parse variable name`) and the unfixed `build-all.R`. This PR re-applies that fix: drop the env additions and derive `platform`/`arch` inside `build-all.R` from the container (bincraft codename → platform mapping + `Sys.info()` arch). ## Notes - `OS_VERSION` options are quoted strings so tags like `8`/`9`/`10` aren't parsed as integers. - Validated: both YAMLs parse, `build-all.R` parses. Reviewed-on: #97	2026-06-18 11:39:11 +00:00
pat-s	0f4330c05c	fix(ci): make build-all self-sufficient on agents without install-deps (#96) ... ## Summary Fixes the recurring `build-all-*` failure on agents where `build-all-versions-install-deps` did **not** run. Both symptoms in the logs — `Parameter 3 does not have length 1` (repeated, in the metadata DB queries) and `argument is of length zero` (system-dependency install) — were the same bug: inside bincraft, `platform` was zero-length. `local/build-all.R` calls `build_binary_package()` without passing `platform`, so bincraft resolves it from the container codename. The primary build step never (re)installed/pinned bincraft and relied on whatever sat in the **per-agent** cache volume; `depends_on` only orders steps, it does not co-locate them on the same agent, so a job landing where install-deps never ran got a stale bincraft that left `platform` empty. ## Changes - **`build-all-versions.yaml`**: pin bincraft `@v4.2.3` in the primary build step (mirroring the R-minor pass and `process-updates.yaml`), so every agent uses a known-good bincraft regardless of where install-deps ran. - Align the R-minor pass `v4.2.2 → v4.2.3`. - Export `OS`/`OS_VERSION`/`ARCH` as runtime env vars — previously only available for `${...}` interpolation, so `build-all.R`'s already-built dedup query matched platform `"-"` and skipped nothing. - Fix the unarchive call: pass the codename via `bincraft::set_codename(NULL)` instead of the malformed `paste(OS, OS_VERSION)` (`"alpine 3.24"`), matching `archive-missed-packages.yaml`. - **`build-all-versions-install-deps.yaml`**: pin install-deps to `@v4.2.3` (was installing HEAD), so the precomputed snapshot and per-agent library stay consistent pipeline-wide. Reviewed-on: #96	2026-06-18 11:27:02 +00:00
pat-s	f9bc72d5ab	chore: bump bincraft to 4.2.3	2026-06-18 09:43:05 +02:00
pat-s	fa955c10bf	fix: recompute package snapshot when missing from cache (#95) ... ## Summary `build-all.R` reads three snapshot files from `/mnt/cache/packages/` that the `build-all-versions-install-deps` step precomputes: `pkgs_to_build.rds`, `r_minor_sensitive_pkgs.rds`, and `s3_cache.rds`. That cache volume is **per-agent**, so a build job scheduled on a different (fresh) agent than the one that ran install-deps finds the snapshot absent and dies at `readRDS` (`cannot open compressed file '/mnt/cache/packages/pkgs_to_build.rds'`). This adds a conditional guard at the top of `build-all.R`: when any of the three files is missing, it sources `local/packages-to-build.R` (which has all needed creds via `PGPASS` / `B2_S3_*` env, already present in the build step) and writes the derived `.rds` files — exactly mirroring the install-deps command. - The first build job on a fresh agent repopulates the shared cache, so subsequent jobs on that agent reuse it. - Concurrent jobs that also miss simply redo the work (accepted tradeoff vs. slow shared storage like NFS). - Saves use a temp-file + atomic `file.rename`, so a concurrent reader never sees a half-written `.rds`. Reviewed-on: #95	2026-06-17 17:49:04 +00:00
pat-s	4c67703f52	refactor(ci): route workflows by agent group label instead of named agent (#94) ... ## Summary - Switch crow workflow placement from a per-agent label (`agent: ${AGENT}`) to a group label (`group: rpkgs-${ARCH}`), so jobs target the `rpkgs-amd64` / `rpkgs-arm64` agent pools instead of a single named agent (`artemis`/`gaia`). - Remove the now-unused `AGENT` matrix variable from all build/process/weekly workflows. - Update header comments to reflect group-based placement. Agents opt into a pool via `CROW_AGENT_LABELS="group=rpkgs-<arch>"`. The group name derives from each matrix row's `ARCH`, so adding or replacing agents no longer requires touching the workflow files. Reviewed-on: #94	2026-06-17 16:26:04 +00:00
pat-s	3875612237	fix(ci): restore xwfb-run/xvfb-run detection for rhel-10 process-updates ... The matrix consolidation hardcoded `xvfb-run`, which is absent on the rhel-10 image (it ships Xwayland's xwfb-run), causing `xvfb-run: command not found`. Re-add the per-platform launcher detection the old rhel-10 files used: prefer xwfb-run + weston, fall back to xvfb-run.	2026-06-17 12:47:04 +02:00
pat-s	76f677ba5b	fix(ci): quote matrix PROCESS_NEW so R receives uppercase FALSE/TRUE ... Unquoted `PROCESS_NEW: FALSE` is parsed as a YAML boolean and interpolated into the R command as lowercase `false`, causing `process_new = false` and `Error: object 'false' not found`. Quoting keeps the literal FALSE/TRUE that R expects.	2026-06-17 11:35:44 +02:00
pat-s	dc92bd6a2c	fix(ci): restore cron triggers via per-row cron name filter ... The matrix consolidation routed cron events through `evaluate: 'CI_PIPELINE_CRON == "..."'`, but CI_PIPELINE_CRON is not exposed to the when.evaluate constraint context, so no matrix row ever matched and the scheduled process-updates / weekly runs stopped firing. Switch back to the first-class `cron:` name filter (still parameterized per matrix row) which the server matches directly against the fired cron.	2026-06-17 11:03:12 +02:00
pat-s	b297f0fbc0	refactor(ci): consolidate build-all-versions workflows into 2 files (#93) ... ## Summary Consolidate the four `build-all-versions*` files into two, mirroring the matrix approach used for the process-updates / weekly families: | Before | After | |---|---| | `build-all-versions-amd64.yaml` + `build-all-versions-arm64.yaml` | `build-all-versions.yaml` | | `build-all-versions-install-deps-amd64.yaml` + `-arm64.yaml` | `build-all-versions-install-deps.yaml` | `ARCH` becomes a matrix axis carrying the per-arch differences: `agent` (artemis/gaia), `BACKEND` (kubernetes/docker), and `R_PKG_CACHE_DIR`. Build-all keeps its `SPLIT_INDEX` parallelism, so its matrix is ARCH x SPLIT = 8 rows; only the rows matching the selected arch run. **Invocation unchanged.** Routing uses the existing task selector `task == "build-all-${ARCH}"`, so `--var task=build-all-amd64 ...` still runs exactly the amd64 rows. `OS` / `OS_VERSION` / `R_VERSION` / `K8S_INSTANCE_TYPE` are still passed via `--var`. (You can drop `--var ARCH=...` now — arch comes from the matrix/task.) **No cron.** These pipelines are manual-only, so there are no scheduled runs to affect. ## Validation before merge Same matrix-interpolation-in-`when`/`labels` mechanism validated for #92, plus it now relies on per-row `agent`/`BACKEND` selecting the right execution backend (kubernetes for amd64, docker for arm64). Before merge, on this branch trigger a small `task=build-all-arm64` run and confirm it lands on the gaia/docker backend (and amd64 on artemis/kubernetes). Originals are in git history for rollback. Note: `archive-missed-packages.yaml` was already a single consolidated matrix file (cron + manual) and is left untouched. Reviewed-on: #93	2026-06-16 09:33:13 +00:00
pat-s	4406a8735b	refactor(ci): consolidate per-platform crow workflows into 3 matrix files (#92) ... ## Summary Collapse **44** per-platform crow pipeline files into **3** matrix-driven files (one per family), using `matrix.include` + the crow #1165 declarative manual `variables:` block: | Family | Before | After | |---|---|---| | process-updates | 14 | `process-updates.yaml` | | weekly-audit-missing | 16 | `weekly-audit-missing.yaml` | | weekly-rebuild-missing | 14 | `weekly-rebuild-missing.yaml` | - **One file per family** (not split by arch). Arch placement is via the agent label as a matrix var (`artemis`=amd64, `gaia`=arm64) for process-updates and weekly-rebuild; weekly-audit keeps its arch-`nodeSelector` placement (no agent label). - **Matrix axis is named `OS`** (e.g. `redhat-9`) per request, with per-row `R_VERSION`, image tag, codename, and `process_new` (FALSE for alpine). - **Runs preserved 1:1:** - *Cron*: each existing `<family>-<os>-<arch>` cron still fires only its matching matrix row via `CI_PIPELINE_CRON`. Server-side cron entries unchanged. - *Manual*: a #1165 dropdown variable (`process_cran_updates` / `weekly_audit_missing` / `weekly_rebuild_missing`) selects a single `<os>-<arch>` or `all`. weekly-rebuild defaults to `all`, matching its former bare `event: manual` trigger. ## Behavior deltas (intentional, flagged) To fit one file per family the backend was unified: - The amd64-only `node.kubernetes.io/instance-type: AX42` nodeSelector pin is **dropped**; placement now relies on the `artemis`/`gaia` agents. - **process-updates arm64** jobs now share the same `backend_options` as amd64 (resource requests/limits + tolerations) — they previously had none, so they gain an 18Gi memory limit. Tell me if arm64 should stay uncapped. ## Validation required before merge Relies on crow interpolating matrix variables inside `when.evaluate`, `labels.agent`, and `commands`. Interpolation in image/env/commands is standard crow; the novel bit is `when.evaluate`. The manual dropdown routing exercises the same interpolation as the cron routing, so on this branch: 1. Trigger manually with the dropdown = `redhat-9-amd64` -> confirm only that one job runs (image `build-env-redhat:9`, R 4.4.3, agent artemis). 2. Trigger with `all` -> confirm all os/arch jobs schedule. Crons only fire on the default branch, so no collision while unmerged. Fallback if `${...}` doesn't interpolate in `when`: a runtime `case "$CI_PIPELINE_CRON" in ...` guard, same files otherwise. Reviewed-on: #92	2026-06-16 09:33:06 +00:00
pat-s	1cd86b65e1	feat(build): skip already-built package versions on workflow restart (#91) ... ## Summary When a `build-all-*` workflow is restarted, the build job re-reads the static `pkgs_to_build.rds` that the install-deps step produced once, so it cycles over every package an interrupted run already built. This adds a DB-based skip filter so a restart only processes what is genuinely left. - At job start, `build-all.R` queries the `single_builds` metadata table for `(name, tag)` already built successfully (`error_occurred = FALSE`) on this `platform`/`arch`, and drops those pairs from the chunk before the build loop. It logs how many it skipped. - One indexed query, one round trip, run before the pak forks — no extra S3 listing and no new Python/s3fs memory pressure (`RPostgres`/`DBI` are already used in the container). - Errored versions are intentionally **not** skipped, so transient failures still get retried on restart. ## Dependency Correctness depends on a `error_occurred = FALSE` row meaning the binary is actually published. That guarantee is added in rpkgs/bincraft#56 (success row written only after a confirmed S3 upload). This PR should land together with / after a bincraft release including that fix. Reviewed-on: #91	2026-06-16 07:27:20 +00:00
automation-bot	2dddc5fe43	chore(deps): update terraform bunnynet to v0.14.3	2026-06-16 00:31:56 +00:00
automation-bot	6c97f34cf5	chore(deps): update pre-commit hook posit-dev/air-pre-commit to v0.9.0	2026-06-15 00:31:42 +00:00
pat-s	8cf8e33bf5	fix(ci): build per-minor binaries with a per-minor R library ... The sensitive-only multi-R build passes ran under each minor's R but loaded bincraft from the shared R_LIBS_USER=/mnt/cache/R-pkgs cache, which is compiled under the primary R and fails to load under a newer minor (e.g. 4.6: "undefined symbol: SETLENGTH") — so those minor builds silently no-op'd. Give each non-primary minor its own R_LIBS_USER=/mnt/cache/R-pkgs-<minor> and install bincraft 4.2.1 there (built for that minor); the rest of bincraft's deps load from the per-version /opt/R/<minor>/lib. Applied to the build-all-versions and process-updates sensitive-only passes.	2026-06-14 22:22:56 +02:00
pat-s	61a9ac5609	fix(ci): run per-minor index under the primary R in process-updates ... The per-minor upload_package_index() loop invoked each minor's own R, which loads bincraft from the shared /mnt/cache/R-pkgs cache (compiled under the primary R). Under a newer minor (e.g. 4.6) that fails to load ("undefined symbol: SETLENGTH") and the index step dies. Indexing is pure S3 work and needs no specific R, so run it under /opt/R/$R_VERSION like the generic index and build-all-versions already do.	2026-06-14 22:18:19 +02:00
pat-s	7e01f481c0	perf(local): skip per-minor passes for non-r-minor-sensitive packages ... A package's ABI sensitivity is the same across R minors, so for a non-sensitive package the per-minor loop only ran `ensure_bincraft` (a ~14s bincraft/cranlike install per minor) before build-one.R classified and skipped it. The primary pass now writes a .r_minor_sensitive sentinel when it builds a sensitive package, and the wrapper gates the whole per-minor loop (installs included) on that file — non-sensitive rebuilds no longer touch other minors' libraries.	2026-06-14 14:50:40 +02:00
pat-s	95bcf6b377	chore: build alpine 3.24	2026-06-14 14:12:05 +02:00
pat-s	a9ad907f9c	chore(local): add S3 migration and CRAN-source helper scripts ... - migrate-s3-hetzner-to-backblaze.sh: rclone-based bucket migration helper. - find-R-api-packages.sh: scan CRAN package sources for a C-API usage pattern. - query-pkgs-without-old-versions.R, test-package-loading.R: ad-hoc helpers.	2026-06-14 13:02:48 +02:00
pat-s	3ae4672d66	feat(local): just rebuild builds across all installed R minors ... - Dockerfile: install bincraft into each R minor, run a primary pass plus a sensitive-only pass under every other /opt/R/[0-9]* minor (deduped by minor), probe/skip xvfb, set GIT_TERMINAL_PROMPT=0; extra-minor failures are non-fatal. - build-one.R: add --sensitive-only mode, log + shallow-clone the ABI classify step, and clear cranlike's stale ./PACKAGES.db before each index refresh (workaround for the "table packages already exists" bug; pending cranlike fix).	2026-06-14 13:02:48 +02:00
pat-s	e5398af5f5	feat(local): just rebuild — TF_VAR secrets, live progress, builder docs ... - Accept TF_VAR_-prefixed B2 keys (from direnv); make GITHUB_PAT optional. - Stream the remote build with --progress=plain. - Document creating the artemis/gaia buildx builders with the docker-container driver (the default remote driver does not work with an ssh:// docker host).	2026-06-14 13:02:47 +02:00
pat-s	f51617018d	ci: bump bincraft to 4.2.1 and scope multi-R loops to versioned R dirs ... - Pin bincraft v4.2.0 -> v4.2.1 (per-minor archive.rds/index fix) across the process-updates, weekly-rebuild and archive workflows. - Restrict the per-R-minor build and index loops to /opt/R/[0-9]* so the `current` symlink dir is not treated as a distinct minor (avoids double- building a minor and writing a bogus contrib/current/ index slot).	2026-06-14 13:02:47 +02:00
pat-s	7f5674ffb3	fix(local): accept TF_VAR_-prefixed B2 keys, optional GITHUB_PAT (#88) ... ## Summary Follow-up fix to the `just rebuild` recipe (#87): the local `.envrc` exports the B2 credentials as `TF_VAR_B2_S3_ACCESS_KEY` / `TF_VAR_B2_S3_SECRET_KEY` (and has no `GITHUB_PAT`), but the recipe required the plain `B2_S3_ACCESS_KEY` / `B2_S3_SECRET_KEY` names and always passed a `github_pat` secret. - Accept the `TF_VAR_`-prefixed B2 names (falling back to the plain names if set directly). - Pass the `github_pat` buildx secret only when `GITHUB_PAT` is set, so the build works without it. Reviewed-on: #88	2026-06-14 09:08:48 +00:00
pat-s	1224bd74fe	feat(local): just rebuild recipe for targeted builds via remote buildx (#87) ... ## Summary Adds a local `just rebuild` recipe to (re)build specific versions of a single package on a given OS/arch, dispatching to a remote buildx builder (the build runs there, not locally). - `just rebuild <os> <tag> <arch> <package> <version>...` → `docker buildx build --builder <artemis|gaia> --platform linux/<arch> …` (amd64→artemis, arm64→gaia; names + `R_VERSION` env-overridable). - `docker/build-one.Dockerfile` runs `build-one.R` as a secret-mounted `RUN`, built `--no-cache --output type=cacheonly` (pure side-effect: the S3 upload; no image kept). - `local/build-one.R` auto-classifies each version via the ABI classifier (risky → per-minor slot `contrib/<x.y>/`, else generic), force-rebuilds + uploads + stores metadata, then refreshes the touched slot's `PACKAGES` index. ## Prerequisites - buildx builders named `artemis` (amd64) and `gaia` (arm64) registered (`docker buildx create --name artemis ssh://…`). - Exported secrets: `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY`, `PGPASS` (`GITHUB_PAT` optional). - bincraft `v4.2.0` tag must exist (the build installs `@v4.2.0` and uses its classifier + per-minor index API). Reviewed-on: #87	2026-06-14 09:03:45 +00:00
pat-s	d558e27c11	feat: R-minor-sensitive binary builds (full + iterative) (#84) ... ## Summary Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered. ## What's in here **Detection / precompute** - `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests. - `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions. **Full build** - `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset. - `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor. **Iterative build** - All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added. **Tooling / housekeeping** - Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`. - Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts. - Design + implementation docs under `docs/superpowers/`. ## Requires before merge - A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected. Reviewed-on: #84	2026-06-14 08:04:40 +00:00
automation-bot	f28ccaf008	chore(deps): update reg.devxy.io/rpkgs/build-env-alpine docker tag to v3.24	2026-06-14 00:33:22 +00:00
automation-bot	d018f31376	chore(deps): update dependency bincraft to v4.2.0	2026-06-14 00:32:46 +00:00
pat-s	4242e63f30	refactor(ci): simplify process-updates cache cleanup (#76) ... ## Summary The `rm -rf /mnt/cache/...` block in 14 `process-updates-*` files had two real issues: 1. `/mnt/cache/pkgcache/R/pkgcache` listed twice — harmless duplicate. 2. `/mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres` — missing a space between two paths, so `rm` walks a nested nonsense path that never exists. The `00LOCK-RPostgres` lock cleanup never actually ran. Collapses to the same shape `weekly-rebuild-missing-*` already uses: ``` rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/{bincraft,pkgcache} /mnt/cache/pkgcache/R/pkgcache ``` `00LOCK-*` covers every stale lock (pak, bincraft, RPostgres, r-pkg-binaries, …) without enumerating them, so adding a new locked package later doesn't need another YAML touch. 14 files, 1 line changed each. Reviewed-on: #76	2026-06-12 13:41:53 +00:00
pat-s	a02bd5c4ff	chore: adjust cdn limits	2026-06-11 14:42:51 +02:00
pat-s	44482b18f9	chore: cleanup and adjust cdn definitions	2026-06-11 11:19:26 +02:00
automation-bot	81f585f7f5	chore(deps): update reg.devxy.io/docker.io/library/alpine docker tag to v3.24	2026-06-10 00:31:30 +00:00
pat-s	228459c0ba	fix(ci): actually purge BunnyCDN cache in purge_cdn_cache.sh (#78) ... ## Summary While cleaning this up I noticed the script **defined** `purge_cdn_cache()` but never **called** it. Every CI run only declared the helper and exited cleanly without issuing a single curl. The PACKAGES freshness on cran.devxy.io / cran.rpkgs.com has been carried entirely by R's `Cache-Control: no-cache` header on the index files. Fixes in one shot: 1. **Actually run the purge.** The shifted args (`api_key`, `arch`, `os_id`, `domain…`) are now consumed inline and a curl POST is issued per (domain × resource). 2. **Drop `set -x`** — leaked the `AccessKey:` header into job logs. 3. **Drop the Python URL-encoder.** `curl -G --data-urlencode "url=…" --data "async=false" https://api.bunny.net/purge` does the same with no Python. The workflow purge steps can later switch from `alpine:3.23 + apk add bash curl` to a slimmer curl-only image. 4. **Add `src/contrib/Meta/archive.rds`** to the purged resource list. It's rewritten on every `process_cran_updates` run (see README) and was being served stale. ## Risks - This is the **first time** the script actually purges anything. If anything else (e.g. a downstream service) relied on the no-op behavior, this PR is the moment it stops being silent. I don't see any such caller. - Edge cache miss right after a purge means an origin S3 fetch — minor latency uptick on the first request per region per resource. Reviewed-on: #78	2026-06-08 08:31:03 +00:00
pat-s	4425904eeb	feat(ci): archive missed packages for redhat-10 and alpine323 (#75) ... ## Summary `archive-missed-packages.yaml`'s matrix was last touched when redhat-9 and alpine322 were the newest targets. redhat-10 and alpine323 builds are now wired up across `process-updates-*` and `weekly-rebuild-missing-*` workflows, but the daily archive sweep wasn't catching them — so missed-archive packages on those platforms accumulate untouched. Adds the 4 missing entries: - `redhat-10` × `{amd64, arm64}` - `alpine323` × `{amd64, arm64}` Reviewed-on: #75	2026-06-08 08:30:38 +00:00
pat-s	53da568749	refactor: single source of truth for excluded packages (#79) ... ## Summary `local/build-all.R` had a 75-name hardcoded `exclude <- c(...)` vector that had drifted from `local/excluded-packages.json` — `RcmdrPlugin.ROC` was in the R vector but missing from the JSON. `.crow/weekly-rebuild-missing-*.yaml` already reads the JSON via `jsonlite::fromJSON(...)[["package"]]`. This brings `build-all.R` in line with that pattern. Changes: - **`local/excluded-packages.json`**: add the missing `RcmdrPlugin.ROC` entry (reason `"hang"`, matching siblings). - **`local/build-all.R`**: replace the 16-line hardcoded vector with one `jsonlite::fromJSON(...)` call. - **`.crow/build-all-versions-install-deps-{amd,arm}64.yaml`**: add `jsonlite` to the install-deps `pak::pak()` list so it's available in `/mnt/cache/R-pkgs` for the build step. - **`.crow/build-all-versions-arm64.yaml`**: drop the dead base64-encoded `SKIP_PKGS` docs comment that nobody was passing as a `--var` anyway; replace with a one-line pointer to the JSON. Reviewed-on: #79	2026-06-08 08:30:19 +00:00
pat-s	fb95042d16	chore: drop stale Hetzner-pointing helpers (#77) ... ## Summary Both helpers still point at the old Hetzner storage: - `Justfile` — all 3 recipes (`build-all`, `build-single`, `process-updates`) hit `hel1.your-objectstorage.com` / bucket `devxy-r-package-binaries-hel1` using `HETZNER_S3_*_K3S` env vars. - `local/manual-package-index-update.R` — same endpoint + bucket, hardcoded into `s3fs::s3_dir_ls()` / `s3_file_delete()` calls. Storage moved to Backblaze (`s3.eu-central-003.backblazeb2.com` / `devxy-rpkgs-binaries`) a while back, so running either of these today would write to the wrong bucket or fail outright. No callers reference them in-tree, deleting outright. A one-off rebuild can just call `bincraft::upload_package_index()` directly with current Backblaze settings. Reviewed-on: #77	2026-06-08 08:29:40 +00:00
pat-s	35dafab737	refactor: hoist CDN block_ips to a variable (#81) ... ## Summary `block_ips = ["185.172.53.0"]` was duplicated in both `bunnynet_pullzone` resources in `cdn.tf`. Move it to a typed `cdn_block_ips` variable in `vars.tf` with the existing IP as the default. - No plan diff on apply — same value, just sourced from `var.cdn_block_ips` instead of a literal. - Adding/removing IPs is now a one-line var override (or a default change) instead of two edits in the resource bodies. ## Interaction with #80 (for_each refactor) `#80` collapses the two pullzones to `bunnynet_pullzone.this[for_each]`. Either order works; whichever lands second is a trivial one-line rebase on the surviving `block_ips =` line. Reviewed-on: #81	2026-06-08 08:29:06 +00:00
pat-s	2ba6ab28f7	chore: bump to bincraft 4.1.1	2026-05-31 09:30:02 +02:00
pat-s	b1bed68898	refactor(ci): skip bincraft reinstall when image already has it (#72) ... ## Summary bincraft is baked into the build-env images, but every workflow still unconditionally calls `pak::pak("git::.../bincraft.git")` (~30–60s per job). Pin bincraft to a specific tag (currently `v4.0.2`) and guard the install on a version check. ``` - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.0.2") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.0.2")' ``` - Image matches the pin → skip install entirely. - Image is missing bincraft or has a different version → install the pinned tag (covers both broken bakes and intentional version bumps). - Renovate customManager keeps the pin and the version-check string in sync whenever bincraft cuts a new tag (uses `git-tags` datasource against the bincraft repo). Touches 43 `.crow` YAMLs + `renovate.json`. Reviewed-on: #72	2026-05-28 12:37:58 +00:00
pat-s	9cbb43dada	refactor(ci): drop dedicated update-package-index-* chain (#74) ... ## Summary `process-updates-*` already calls `bincraft::upload_package_index()` at the end of every daily delta run, and with #71 the CDN purge happens there too. The dedicated `update-package-index-*` chain re-ran the same `upload_package_index()` against the same S3 prefixes — once per platform per day — and was serialized via `depends_on:` purely to dodge S3 rate limits. Net effect: ~40 min cold rebuild × 14 platforms = ~9 h/day of duplicated work, with no consumer waiting on it (process-updates had already published the fresh index). This deletes the 14 chain files (~1.1k lines). The matching `update-package-index` / `update-package-index-*` cron entries on the Crow side become inert with the YAMLs gone and can be removed in a follow-up cron-config change. Best reviewed alongside (or after): - #71 — purge CDN after process-updates - #72 — guard bincraft reinstall - #73 — drop redundant cranlike/desc installs ## Risks - If `process-updates-*` ever fails for a given platform, the chain used to act as a daily safety net for the PACKAGES index. Failures will now surface only via the missing daily push instead of being silently masked. The existing `weekly-audit-missing-*` workflow continues to flag stale state. - `local/manual-package-index-update.R` still exists for one-off manual rebuilds. Reviewed-on: #74	2026-05-28 12:00:36 +00:00
pat-s	e620bb1c33	refactor(ci): drop redundant cranlike/desc installs (#73) ... ## Summary `bincraft`'s DESCRIPTION already pulls both forks via `Remotes:`: ``` Imports: ... cranlike, ... Remotes: pat-s/cranlike@s3, pat-s/desc@description-from-remote ``` So `pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")` resolves the forks and installs them transitively. The explicit follow-up `pak::pak("pat-s/cranlike@s3")` and `pak::pak("pat-s/desc@description-from-remote")` calls in `update-package-index-*.yaml` are dead weight. Drops the two lines from all 14 `update-package-index-*` files. `process-updates-*.yaml` already only installs `bincraft` — this brings the two workflows in sync. Reviewed-on: #73	2026-05-28 11:51:16 +00:00
pat-s	494730be5d	feat(ci): purge CDN cache after process-updates runs (#71) ... ## Summary - `process_cran_updates()` ends with `upload_package_index()`, refreshing `src/contrib/PACKAGES{,.gz,.rds,.db}` in S3 — but no CDN purge follows. - The CDN was only purged by the daily `update-package-index-*` chain, so between cron runs the edge kept serving stale indexes. A binary that landed in S3 was effectively invisible to `install.packages()` until the index-update chain caught up. Adds the same `Purge CDN cache` step that `update-package-index-*` already uses, parameterized per platform/arch. The script and image are unchanged. Reviewed-on: #71	2026-05-28 11:50:04 +00:00
pat-s	8c5362b7ab	refactor: R_VERSION syntax	2026-05-27 22:36:09 +02:00
pat-s	6faa3772d1	refactor(ci): use multi-R-version images across workflows (#70) ... ## Summary - Switches all 64 `.crow/*.yaml` workflows, the `Justfile`, and the commented-out `build-all-versions-install-deps.yaml` template to the new multi-R-version container images. - Each image tag drops `-${R_VERSION}` (e.g. `build-env-alpine:3.22-4.5` → `build-env-alpine:3.22`); workflows pick an R version by calling `/opt/R/${R_VERSION}/bin/R` explicitly. - Pattern-1 workflows (hard-coded image tag) gain a single `R_VERSION:` env var per file. - Pattern-2 workflows (`build-all-versions-*`, matrix-driven) continue to receive `R_VERSION` via `crow pipeline create --var`. ## Platform → R_VERSION mapping | Platform | Image | R_VERSION | | --- | --- | --- | | alpine 3.22 / 3.23 | `build-env-alpine:3.{22,23}` | 4.5.3 | | ubuntu jammy / noble | `build-env-ubuntu:{jammy,noble}` | 4.4.3 | | redhat 8 / 9 | `build-env-redhat:{8,9}` | 4.4.3 | | redhat 10 | `build-env-redhat:10` | 4.5.3 | The `alpine-321` audit-only workflows fall back to `build-env-alpine:3.23` since no 3.21 image exists in the new scheme; the audits only query S3/CRAN, so the container OS does not affect correctness. ## Bug fixes folded in 1. All six `weekly-audit-missing-alpine-*.yaml` files were pinned to `alpine:3.23-4.5` regardless of platform. Now each one points at its own alpine image. 2. All fourteen `update-package-index-*.yaml` files were pinned to `build-env-ubuntu:noble-4.4` (or `noble-4.5` for `redhat-10-amd64`) regardless of platform. Now each one uses its own platform's image. 3. `process-updates-ubuntu-2404-{amd64,arm64}.yaml` had `R_VERSION` of `4.4` (no patch) baked into the tag. Aligned to `4.4.3`, matching the audit and rebuild counterparts. ## Smoke targets (post-merge) One workflow per shape, ready to dry-run: - `process-updates-alpine-322-amd64.yaml` - `weekly-rebuild-missing-redhat-9-amd64.yaml` - `weekly-audit-missing-ubuntu-2204-amd64.yaml` - `update-package-index-redhat-10-amd64.yaml` - `archive-missed-packages.yaml` - `build-all-versions-amd64.yaml` (with its install-deps predecessor) - `build-r-minor-sensitive-packages.yaml` ## Follow-ups (out of scope, surfaced during review) - `process-updates-redhat-10-{amd64,arm64}.yaml` set `codename = "rhel9"` in their `upload_package_index(...)` calls. Pre-existing bug — should be `"rhel10"`. The `update-package-index-redhat-10-*` counterparts already use `"rhel10"` correctly. - `update-package-index-redhat-8-amd64.yaml` is missing the `packageVersion("bincraft")` diagnostic line that every sibling file has. Pre-existing asymmetry between amd64 and arm64. ## Spec & plan - Spec: `docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md` - Plan: `docs/superpowers/plans/2026-05-25-multi-r-version-images.md` Reviewed-on: #70	2026-05-26 08:51:27 +00:00
automation-bot	8e453a56ce	chore(deps): update terraform bunnynet to v0.14.2	2026-05-20 00:31:33 +00:00
automation-bot	c9b7e1080a	chore(deps): lock file maintenance	2026-05-18 00:31:48 +00:00