923 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
automation-bot	f28ccaf008	chore(deps): update reg.devxy.io/rpkgs/build-env-alpine docker tag to v3.24	2026-06-14 00:33:22 +00:00
automation-bot	d018f31376	chore(deps): update dependency bincraft to v4.2.0	2026-06-14 00:32:46 +00:00
pat-s	4242e63f30	refactor(ci): simplify process-updates cache cleanup (#76) ... ## Summary The `rm -rf /mnt/cache/...` block in 14 `process-updates-*` files had two real issues: 1. `/mnt/cache/pkgcache/R/pkgcache` listed twice — harmless duplicate. 2. `/mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres` — missing a space between two paths, so `rm` walks a nested nonsense path that never exists. The `00LOCK-RPostgres` lock cleanup never actually ran. Collapses to the same shape `weekly-rebuild-missing-*` already uses: ``` rm -rf /mnt/cache/R-pkgs/00LOCK-* /mnt/cache/R-pkgs/{bincraft,pkgcache} /mnt/cache/pkgcache/R/pkgcache ``` `00LOCK-*` covers every stale lock (pak, bincraft, RPostgres, r-pkg-binaries, …) without enumerating them, so adding a new locked package later doesn't need another YAML touch. 14 files, 1 line changed each. Reviewed-on: #76	2026-06-12 13:41:53 +00:00
pat-s	a02bd5c4ff	chore: adjust cdn limits	2026-06-11 14:42:51 +02:00
pat-s	44482b18f9	chore: cleanup and adjust cdn definitions	2026-06-11 11:19:26 +02:00
automation-bot	81f585f7f5	chore(deps): update reg.devxy.io/docker.io/library/alpine docker tag to v3.24	2026-06-10 00:31:30 +00:00
pat-s	228459c0ba	fix(ci): actually purge BunnyCDN cache in purge_cdn_cache.sh (#78) ... ## Summary While cleaning this up I noticed the script **defined** `purge_cdn_cache()` but never **called** it. Every CI run only declared the helper and exited cleanly without issuing a single curl. The PACKAGES freshness on cran.devxy.io / cran.rpkgs.com has been carried entirely by R's `Cache-Control: no-cache` header on the index files. Fixes in one shot: 1. **Actually run the purge.** The shifted args (`api_key`, `arch`, `os_id`, `domain…`) are now consumed inline and a curl POST is issued per (domain × resource). 2. **Drop `set -x`** — leaked the `AccessKey:` header into job logs. 3. **Drop the Python URL-encoder.** `curl -G --data-urlencode "url=…" --data "async=false" https://api.bunny.net/purge` does the same with no Python. The workflow purge steps can later switch from `alpine:3.23 + apk add bash curl` to a slimmer curl-only image. 4. **Add `src/contrib/Meta/archive.rds`** to the purged resource list. It's rewritten on every `process_cran_updates` run (see README) and was being served stale. ## Risks - This is the **first time** the script actually purges anything. If anything else (e.g. a downstream service) relied on the no-op behavior, this PR is the moment it stops being silent. I don't see any such caller. - Edge cache miss right after a purge means an origin S3 fetch — minor latency uptick on the first request per region per resource. Reviewed-on: #78	2026-06-08 08:31:03 +00:00
pat-s	4425904eeb	feat(ci): archive missed packages for redhat-10 and alpine323 (#75) ... ## Summary `archive-missed-packages.yaml`'s matrix was last touched when redhat-9 and alpine322 were the newest targets. redhat-10 and alpine323 builds are now wired up across `process-updates-*` and `weekly-rebuild-missing-*` workflows, but the daily archive sweep wasn't catching them — so missed-archive packages on those platforms accumulate untouched. Adds the 4 missing entries: - `redhat-10` × `{amd64, arm64}` - `alpine323` × `{amd64, arm64}` Reviewed-on: #75	2026-06-08 08:30:38 +00:00
pat-s	53da568749	refactor: single source of truth for excluded packages (#79) ... ## Summary `local/build-all.R` had a 75-name hardcoded `exclude <- c(...)` vector that had drifted from `local/excluded-packages.json` — `RcmdrPlugin.ROC` was in the R vector but missing from the JSON. `.crow/weekly-rebuild-missing-*.yaml` already reads the JSON via `jsonlite::fromJSON(...)[["package"]]`. This brings `build-all.R` in line with that pattern. Changes: - **`local/excluded-packages.json`**: add the missing `RcmdrPlugin.ROC` entry (reason `"hang"`, matching siblings). - **`local/build-all.R`**: replace the 16-line hardcoded vector with one `jsonlite::fromJSON(...)` call. - **`.crow/build-all-versions-install-deps-{amd,arm}64.yaml`**: add `jsonlite` to the install-deps `pak::pak()` list so it's available in `/mnt/cache/R-pkgs` for the build step. - **`.crow/build-all-versions-arm64.yaml`**: drop the dead base64-encoded `SKIP_PKGS` docs comment that nobody was passing as a `--var` anyway; replace with a one-line pointer to the JSON. Reviewed-on: #79	2026-06-08 08:30:19 +00:00
pat-s	fb95042d16	chore: drop stale Hetzner-pointing helpers (#77) ... ## Summary Both helpers still point at the old Hetzner storage: - `Justfile` — all 3 recipes (`build-all`, `build-single`, `process-updates`) hit `hel1.your-objectstorage.com` / bucket `devxy-r-package-binaries-hel1` using `HETZNER_S3_*_K3S` env vars. - `local/manual-package-index-update.R` — same endpoint + bucket, hardcoded into `s3fs::s3_dir_ls()` / `s3_file_delete()` calls. Storage moved to Backblaze (`s3.eu-central-003.backblazeb2.com` / `devxy-rpkgs-binaries`) a while back, so running either of these today would write to the wrong bucket or fail outright. No callers reference them in-tree, deleting outright. A one-off rebuild can just call `bincraft::upload_package_index()` directly with current Backblaze settings. Reviewed-on: #77	2026-06-08 08:29:40 +00:00
pat-s	35dafab737	refactor: hoist CDN block_ips to a variable (#81) ... ## Summary `block_ips = ["185.172.53.0"]` was duplicated in both `bunnynet_pullzone` resources in `cdn.tf`. Move it to a typed `cdn_block_ips` variable in `vars.tf` with the existing IP as the default. - No plan diff on apply — same value, just sourced from `var.cdn_block_ips` instead of a literal. - Adding/removing IPs is now a one-line var override (or a default change) instead of two edits in the resource bodies. ## Interaction with #80 (for_each refactor) `#80` collapses the two pullzones to `bunnynet_pullzone.this[for_each]`. Either order works; whichever lands second is a trivial one-line rebase on the surviving `block_ips =` line. Reviewed-on: #81	2026-06-08 08:29:06 +00:00
pat-s	2ba6ab28f7	chore: bump to bincraft 4.1.1	2026-05-31 09:30:02 +02:00
pat-s	b1bed68898	refactor(ci): skip bincraft reinstall when image already has it (#72) ... ## Summary bincraft is baked into the build-env images, but every workflow still unconditionally calls `pak::pak("git::.../bincraft.git")` (~30–60s per job). Pin bincraft to a specific tag (currently `v4.0.2`) and guard the install on a version check. ``` - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.0.2") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.0.2")' ``` - Image matches the pin → skip install entirely. - Image is missing bincraft or has a different version → install the pinned tag (covers both broken bakes and intentional version bumps). - Renovate customManager keeps the pin and the version-check string in sync whenever bincraft cuts a new tag (uses `git-tags` datasource against the bincraft repo). Touches 43 `.crow` YAMLs + `renovate.json`. Reviewed-on: #72	2026-05-28 12:37:58 +00:00
pat-s	9cbb43dada	refactor(ci): drop dedicated update-package-index-* chain (#74) ... ## Summary `process-updates-*` already calls `bincraft::upload_package_index()` at the end of every daily delta run, and with #71 the CDN purge happens there too. The dedicated `update-package-index-*` chain re-ran the same `upload_package_index()` against the same S3 prefixes — once per platform per day — and was serialized via `depends_on:` purely to dodge S3 rate limits. Net effect: ~40 min cold rebuild × 14 platforms = ~9 h/day of duplicated work, with no consumer waiting on it (process-updates had already published the fresh index). This deletes the 14 chain files (~1.1k lines). The matching `update-package-index` / `update-package-index-*` cron entries on the Crow side become inert with the YAMLs gone and can be removed in a follow-up cron-config change. Best reviewed alongside (or after): - #71 — purge CDN after process-updates - #72 — guard bincraft reinstall - #73 — drop redundant cranlike/desc installs ## Risks - If `process-updates-*` ever fails for a given platform, the chain used to act as a daily safety net for the PACKAGES index. Failures will now surface only via the missing daily push instead of being silently masked. The existing `weekly-audit-missing-*` workflow continues to flag stale state. - `local/manual-package-index-update.R` still exists for one-off manual rebuilds. Reviewed-on: #74	2026-05-28 12:00:36 +00:00
pat-s	e620bb1c33	refactor(ci): drop redundant cranlike/desc installs (#73) ... ## Summary `bincraft`'s DESCRIPTION already pulls both forks via `Remotes:`: ``` Imports: ... cranlike, ... Remotes: pat-s/cranlike@s3, pat-s/desc@description-from-remote ``` So `pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")` resolves the forks and installs them transitively. The explicit follow-up `pak::pak("pat-s/cranlike@s3")` and `pak::pak("pat-s/desc@description-from-remote")` calls in `update-package-index-*.yaml` are dead weight. Drops the two lines from all 14 `update-package-index-*` files. `process-updates-*.yaml` already only installs `bincraft` — this brings the two workflows in sync. Reviewed-on: #73	2026-05-28 11:51:16 +00:00
pat-s	494730be5d	feat(ci): purge CDN cache after process-updates runs (#71) ... ## Summary - `process_cran_updates()` ends with `upload_package_index()`, refreshing `src/contrib/PACKAGES{,.gz,.rds,.db}` in S3 — but no CDN purge follows. - The CDN was only purged by the daily `update-package-index-*` chain, so between cron runs the edge kept serving stale indexes. A binary that landed in S3 was effectively invisible to `install.packages()` until the index-update chain caught up. Adds the same `Purge CDN cache` step that `update-package-index-*` already uses, parameterized per platform/arch. The script and image are unchanged. Reviewed-on: #71	2026-05-28 11:50:04 +00:00
pat-s	8c5362b7ab	refactor: R_VERSION syntax	2026-05-27 22:36:09 +02:00
pat-s	6faa3772d1	refactor(ci): use multi-R-version images across workflows (#70) ... ## Summary - Switches all 64 `.crow/*.yaml` workflows, the `Justfile`, and the commented-out `build-all-versions-install-deps.yaml` template to the new multi-R-version container images. - Each image tag drops `-${R_VERSION}` (e.g. `build-env-alpine:3.22-4.5` → `build-env-alpine:3.22`); workflows pick an R version by calling `/opt/R/${R_VERSION}/bin/R` explicitly. - Pattern-1 workflows (hard-coded image tag) gain a single `R_VERSION:` env var per file. - Pattern-2 workflows (`build-all-versions-*`, matrix-driven) continue to receive `R_VERSION` via `crow pipeline create --var`. ## Platform → R_VERSION mapping | Platform | Image | R_VERSION | | --- | --- | --- | | alpine 3.22 / 3.23 | `build-env-alpine:3.{22,23}` | 4.5.3 | | ubuntu jammy / noble | `build-env-ubuntu:{jammy,noble}` | 4.4.3 | | redhat 8 / 9 | `build-env-redhat:{8,9}` | 4.4.3 | | redhat 10 | `build-env-redhat:10` | 4.5.3 | The `alpine-321` audit-only workflows fall back to `build-env-alpine:3.23` since no 3.21 image exists in the new scheme; the audits only query S3/CRAN, so the container OS does not affect correctness. ## Bug fixes folded in 1. All six `weekly-audit-missing-alpine-*.yaml` files were pinned to `alpine:3.23-4.5` regardless of platform. Now each one points at its own alpine image. 2. All fourteen `update-package-index-*.yaml` files were pinned to `build-env-ubuntu:noble-4.4` (or `noble-4.5` for `redhat-10-amd64`) regardless of platform. Now each one uses its own platform's image. 3. `process-updates-ubuntu-2404-{amd64,arm64}.yaml` had `R_VERSION` of `4.4` (no patch) baked into the tag. Aligned to `4.4.3`, matching the audit and rebuild counterparts. ## Smoke targets (post-merge) One workflow per shape, ready to dry-run: - `process-updates-alpine-322-amd64.yaml` - `weekly-rebuild-missing-redhat-9-amd64.yaml` - `weekly-audit-missing-ubuntu-2204-amd64.yaml` - `update-package-index-redhat-10-amd64.yaml` - `archive-missed-packages.yaml` - `build-all-versions-amd64.yaml` (with its install-deps predecessor) - `build-r-minor-sensitive-packages.yaml` ## Follow-ups (out of scope, surfaced during review) - `process-updates-redhat-10-{amd64,arm64}.yaml` set `codename = "rhel9"` in their `upload_package_index(...)` calls. Pre-existing bug — should be `"rhel10"`. The `update-package-index-redhat-10-*` counterparts already use `"rhel10"` correctly. - `update-package-index-redhat-8-amd64.yaml` is missing the `packageVersion("bincraft")` diagnostic line that every sibling file has. Pre-existing asymmetry between amd64 and arm64. ## Spec & plan - Spec: `docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md` - Plan: `docs/superpowers/plans/2026-05-25-multi-r-version-images.md` Reviewed-on: #70	2026-05-26 08:51:27 +00:00
automation-bot	8e453a56ce	chore(deps): update terraform bunnynet to v0.14.2	2026-05-20 00:31:33 +00:00
automation-bot	c9b7e1080a	chore(deps): lock file maintenance	2026-05-18 00:31:48 +00:00
automation-bot	73b6677258	chore(deps): update terraform bunnynet to v0.14.1	2026-05-12 00:31:40 +00:00
pat-s	84c59d1984	fix: audit workflows	2026-04-21 21:54:48 +02:00
pat-s	dfad39ebf1	chore: remove unused FORGEJO_TOKEN	2026-04-21 20:24:48 +02:00
pat-s	39bd6cd690	chore: remove manual condition trigger form weekly audit jobs	2026-04-21 20:20:54 +02:00
pat-s	3b5454aa99	chore: purge package indexes for alpine	2026-04-21 19:57:37 +02:00
pat-s	0d6c1c8312	fix: correct platform param in alpine 322/323 arm64 CI pipelines ... Both pipelines had platform = 'alpine-321' instead of their actual target, causing builds to upload to the wrong S3 path and the PACKAGES index to not reflect newly built packages.	2026-04-21 19:45:06 +02:00
pat-s	7892eb0423	chore: remove unused cdns	2026-04-21 19:31:01 +02:00
pat-s	6440e3c7a1	ci: make rhel10 update runs compatible with xwayland	2026-04-21 11:05:13 +02:00
pat-s	8974bc4c92	feat: add rhel10 package update workflows	2026-04-20 11:51:27 +02:00
pat-s	8a98cf96a6	fix: save S3 listing in precompute step, avoid loading s3fs in build step ... s3fs uses reticulate/Python which significantly increases R process memory. When pak tries to fork R to install package dependencies, the enlarged process can't be duplicated within the container memory limit. Now the install-deps step saves the S3 file listing as s3_cache.rds, and the build step reads it with readRDS() — no s3fs loading needed.	2026-04-17 20:05:26 +02:00
pat-s	bc3dbd6126	fix: map x86_64/aarch64 to amd64/arm64 for S3 cache path	2026-04-17 19:04:18 +02:00
pat-s	9b27b3e650	refactor: extract build-all R one-liner into local/build-all.R ... - Load bincraft eagerly via library() to avoid lazy-load memory spike - Do bulk S3 listing upfront and pass as s3_package_cache to avoid per-package S3 calls that accumulate memory and trigger fork failures - Consolidate amd64/arm64 exclude lists into single script - Much easier to read and maintain than a YAML-embedded one-liner	2026-04-17 19:01:08 +02:00
pat-s	87d3f54b21	fix: pass NULL to set_codename() which requires an explicit argument	2026-04-17 18:55:40 +02:00
pat-s	92084e51ed	fix: use bincraft::set_codename() for S3 path in precompute step ... The precompute script was constructing S3 paths as "redhat10" while bincraft uses "rhel10" as the codename. This mismatch caused the precompute to find no existing packages in S3, producing ~77K false positive package versions that all get skipped at build time. Now uses bincraft::set_codename() to ensure path alignment. Also fixes nrow() vs length() in install-deps summary message.	2026-04-17 18:54:29 +02:00
pat-s	e47949f546	fix: remove force=TRUE to preserve error metadata tracking ... force=TRUE bypasses the error-DB filter and overwrites build metadata, causing failed packages to be retried indefinitely. Since the precompute step already filters to missing versions, force is unnecessary.	2026-04-17 18:35:59 +02:00
pat-s	a732993d2b	feat: iterate over (package, version) pairs in build-all pipeline ... Build step now reads precomputed package+version pairs and passes specific tags with force=TRUE, eliminating all per-package precondition I/O.	2026-04-17 18:22:36 +02:00
pat-s	21ec8a67dd	feat: save package+version pairs instead of just package names ... packages-to-build.R now saves the full (Package, Version) data.table so the build step can pass specific tags to build_binary_package(), eliminating redundant per-package tag discovery and S3 checks. Also reduces archive versions from 9 to 4 (+ 1 release = 5 total).	2026-04-17 18:20:00 +02:00
pat-s	9330bef97c	chore: adjust matrix split counts for amd64 and arm64 builds	2026-04-15 10:13:16 +02:00
pat-s	5293967137	feat: run incremental package index update after each process-updates workflow ... Each process-updates workflow now calls upload_package_index at the end to update the PACKAGES files for its platform/arch immediately after builds complete. The full update-package-index cron should be rescheduled to weekly.	2026-04-15 10:13:16 +02:00
automation-bot	477d21ce54	chore(deps): update reg.devxy.io/rpkgs/build-env-alpine docker tag to v3.23	2026-04-13 00:01:26 +00:00
pat-s	a27011d049	chore: 4-way split for amd	2026-04-12 19:07:32 +02:00
pat-s	360aa45e90	fix: replace colon in error message to avoid YAML parsing issue in rebuild workflows	2026-04-11 21:53:22 +02:00
pat-s	5d9bdbb3a9	feat: add 14 weekly rebuild workflows for all platform/arch combos	2026-04-11 21:46:38 +02:00
pat-s	10aeffd9fa	feat: add 14 weekly audit workflows for all platform/arch combos	2026-04-11 21:45:29 +02:00
pat-s	ffb8c42511	feat: add weekly missing binaries audit R script	2026-04-11 21:41:59 +02:00
pat-s	84f59682eb	feat: add excluded-packages.json for weekly missing binaries workflow	2026-04-11 21:41:55 +02:00
pat-s	82dd2af5b2	docs: add implementation plan for weekly missing binaries audit & rebuild	2026-04-11 21:23:59 +02:00
pat-s	154a517cfc	docs: add design spec for weekly missing binaries audit & rebuild workflow	2026-04-11 21:16:30 +02:00
pat-s	14950a6bfc	refactor: chain update-package-index workflows sequentially to avoid overwhelming S3 endpoint ... All 12 workflows were triggered simultaneously by the update-package-index cron. Now they run one after another via depends_on. Individual cron triggers still allow running one at a time.	2026-04-11 10:18:35 +02:00
pat-s	a64dea6b23	refactor: separate install-deps workflows for arch	2026-04-11 09:49:44 +02:00