808 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
pat-s	494730be5d	feat(ci): purge CDN cache after process-updates runs (#71) ... ## Summary - `process_cran_updates()` ends with `upload_package_index()`, refreshing `src/contrib/PACKAGES{,.gz,.rds,.db}` in S3 — but no CDN purge follows. - The CDN was only purged by the daily `update-package-index-*` chain, so between cron runs the edge kept serving stale indexes. A binary that landed in S3 was effectively invisible to `install.packages()` until the index-update chain caught up. Adds the same `Purge CDN cache` step that `update-package-index-*` already uses, parameterized per platform/arch. The script and image are unchanged. Reviewed-on: #71	2026-05-28 11:50:04 +00:00
pat-s	8c5362b7ab	refactor: R_VERSION syntax	2026-05-27 22:36:09 +02:00
pat-s	6faa3772d1	refactor(ci): use multi-R-version images across workflows (#70) ... ## Summary - Switches all 64 `.crow/*.yaml` workflows, the `Justfile`, and the commented-out `build-all-versions-install-deps.yaml` template to the new multi-R-version container images. - Each image tag drops `-${R_VERSION}` (e.g. `build-env-alpine:3.22-4.5` → `build-env-alpine:3.22`); workflows pick an R version by calling `/opt/R/${R_VERSION}/bin/R` explicitly. - Pattern-1 workflows (hard-coded image tag) gain a single `R_VERSION:` env var per file. - Pattern-2 workflows (`build-all-versions-*`, matrix-driven) continue to receive `R_VERSION` via `crow pipeline create --var`. ## Platform → R_VERSION mapping | Platform | Image | R_VERSION | | --- | --- | --- | | alpine 3.22 / 3.23 | `build-env-alpine:3.{22,23}` | 4.5.3 | | ubuntu jammy / noble | `build-env-ubuntu:{jammy,noble}` | 4.4.3 | | redhat 8 / 9 | `build-env-redhat:{8,9}` | 4.4.3 | | redhat 10 | `build-env-redhat:10` | 4.5.3 | The `alpine-321` audit-only workflows fall back to `build-env-alpine:3.23` since no 3.21 image exists in the new scheme; the audits only query S3/CRAN, so the container OS does not affect correctness. ## Bug fixes folded in 1. All six `weekly-audit-missing-alpine-*.yaml` files were pinned to `alpine:3.23-4.5` regardless of platform. Now each one points at its own alpine image. 2. All fourteen `update-package-index-*.yaml` files were pinned to `build-env-ubuntu:noble-4.4` (or `noble-4.5` for `redhat-10-amd64`) regardless of platform. Now each one uses its own platform's image. 3. `process-updates-ubuntu-2404-{amd64,arm64}.yaml` had `R_VERSION` of `4.4` (no patch) baked into the tag. Aligned to `4.4.3`, matching the audit and rebuild counterparts. ## Smoke targets (post-merge) One workflow per shape, ready to dry-run: - `process-updates-alpine-322-amd64.yaml` - `weekly-rebuild-missing-redhat-9-amd64.yaml` - `weekly-audit-missing-ubuntu-2204-amd64.yaml` - `update-package-index-redhat-10-amd64.yaml` - `archive-missed-packages.yaml` - `build-all-versions-amd64.yaml` (with its install-deps predecessor) - `build-r-minor-sensitive-packages.yaml` ## Follow-ups (out of scope, surfaced during review) - `process-updates-redhat-10-{amd64,arm64}.yaml` set `codename = "rhel9"` in their `upload_package_index(...)` calls. Pre-existing bug — should be `"rhel10"`. The `update-package-index-redhat-10-*` counterparts already use `"rhel10"` correctly. - `update-package-index-redhat-8-amd64.yaml` is missing the `packageVersion("bincraft")` diagnostic line that every sibling file has. Pre-existing asymmetry between amd64 and arm64. ## Spec & plan - Spec: `docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md` - Plan: `docs/superpowers/plans/2026-05-25-multi-r-version-images.md` Reviewed-on: #70	2026-05-26 08:51:27 +00:00
automation-bot	8e453a56ce	chore(deps): update terraform bunnynet to v0.14.2	2026-05-20 00:31:33 +00:00
automation-bot	c9b7e1080a	chore(deps): lock file maintenance	2026-05-18 00:31:48 +00:00
automation-bot	73b6677258	chore(deps): update terraform bunnynet to v0.14.1	2026-05-12 00:31:40 +00:00
pat-s	84c59d1984	fix: audit workflows	2026-04-21 21:54:48 +02:00
pat-s	dfad39ebf1	chore: remove unused FORGEJO_TOKEN	2026-04-21 20:24:48 +02:00
pat-s	39bd6cd690	chore: remove manual condition trigger form weekly audit jobs	2026-04-21 20:20:54 +02:00
pat-s	3b5454aa99	chore: purge package indexes for alpine	2026-04-21 19:57:37 +02:00
pat-s	0d6c1c8312	fix: correct platform param in alpine 322/323 arm64 CI pipelines ... Both pipelines had platform = 'alpine-321' instead of their actual target, causing builds to upload to the wrong S3 path and the PACKAGES index to not reflect newly built packages.	2026-04-21 19:45:06 +02:00
pat-s	7892eb0423	chore: remove unused cdns	2026-04-21 19:31:01 +02:00
pat-s	6440e3c7a1	ci: make rhel10 update runs compatible with xwayland	2026-04-21 11:05:13 +02:00
pat-s	8974bc4c92	feat: add rhel10 package update workflows	2026-04-20 11:51:27 +02:00
pat-s	8a98cf96a6	fix: save S3 listing in precompute step, avoid loading s3fs in build step ... s3fs uses reticulate/Python which significantly increases R process memory. When pak tries to fork R to install package dependencies, the enlarged process can't be duplicated within the container memory limit. Now the install-deps step saves the S3 file listing as s3_cache.rds, and the build step reads it with readRDS() — no s3fs loading needed.	2026-04-17 20:05:26 +02:00
pat-s	bc3dbd6126	fix: map x86_64/aarch64 to amd64/arm64 for S3 cache path	2026-04-17 19:04:18 +02:00
pat-s	9b27b3e650	refactor: extract build-all R one-liner into local/build-all.R ... - Load bincraft eagerly via library() to avoid lazy-load memory spike - Do bulk S3 listing upfront and pass as s3_package_cache to avoid per-package S3 calls that accumulate memory and trigger fork failures - Consolidate amd64/arm64 exclude lists into single script - Much easier to read and maintain than a YAML-embedded one-liner	2026-04-17 19:01:08 +02:00
pat-s	87d3f54b21	fix: pass NULL to set_codename() which requires an explicit argument	2026-04-17 18:55:40 +02:00
pat-s	92084e51ed	fix: use bincraft::set_codename() for S3 path in precompute step ... The precompute script was constructing S3 paths as "redhat10" while bincraft uses "rhel10" as the codename. This mismatch caused the precompute to find no existing packages in S3, producing ~77K false positive package versions that all get skipped at build time. Now uses bincraft::set_codename() to ensure path alignment. Also fixes nrow() vs length() in install-deps summary message.	2026-04-17 18:54:29 +02:00
pat-s	e47949f546	fix: remove force=TRUE to preserve error metadata tracking ... force=TRUE bypasses the error-DB filter and overwrites build metadata, causing failed packages to be retried indefinitely. Since the precompute step already filters to missing versions, force is unnecessary.	2026-04-17 18:35:59 +02:00
pat-s	a732993d2b	feat: iterate over (package, version) pairs in build-all pipeline ... Build step now reads precomputed package+version pairs and passes specific tags with force=TRUE, eliminating all per-package precondition I/O.	2026-04-17 18:22:36 +02:00
pat-s	21ec8a67dd	feat: save package+version pairs instead of just package names ... packages-to-build.R now saves the full (Package, Version) data.table so the build step can pass specific tags to build_binary_package(), eliminating redundant per-package tag discovery and S3 checks. Also reduces archive versions from 9 to 4 (+ 1 release = 5 total).	2026-04-17 18:20:00 +02:00
pat-s	9330bef97c	chore: adjust matrix split counts for amd64 and arm64 builds	2026-04-15 10:13:16 +02:00
pat-s	5293967137	feat: run incremental package index update after each process-updates workflow ... Each process-updates workflow now calls upload_package_index at the end to update the PACKAGES files for its platform/arch immediately after builds complete. The full update-package-index cron should be rescheduled to weekly.	2026-04-15 10:13:16 +02:00
automation-bot	477d21ce54	chore(deps): update reg.devxy.io/rpkgs/build-env-alpine docker tag to v3.23	2026-04-13 00:01:26 +00:00
pat-s	a27011d049	chore: 4-way split for amd	2026-04-12 19:07:32 +02:00
pat-s	360aa45e90	fix: replace colon in error message to avoid YAML parsing issue in rebuild workflows	2026-04-11 21:53:22 +02:00
pat-s	5d9bdbb3a9	feat: add 14 weekly rebuild workflows for all platform/arch combos	2026-04-11 21:46:38 +02:00
pat-s	10aeffd9fa	feat: add 14 weekly audit workflows for all platform/arch combos	2026-04-11 21:45:29 +02:00
pat-s	ffb8c42511	feat: add weekly missing binaries audit R script	2026-04-11 21:41:59 +02:00
pat-s	84f59682eb	feat: add excluded-packages.json for weekly missing binaries workflow	2026-04-11 21:41:55 +02:00
pat-s	82dd2af5b2	docs: add implementation plan for weekly missing binaries audit & rebuild	2026-04-11 21:23:59 +02:00
pat-s	154a517cfc	docs: add design spec for weekly missing binaries audit & rebuild workflow	2026-04-11 21:16:30 +02:00
pat-s	14950a6bfc	refactor: chain update-package-index workflows sequentially to avoid overwhelming S3 endpoint ... All 12 workflows were triggered simultaneously by the update-package-index cron. Now they run one after another via depends_on. Individual cron triggers still allow running one at a time.	2026-04-11 10:18:35 +02:00
pat-s	a64dea6b23	refactor: separate install-deps workflows for arch	2026-04-11 09:49:44 +02:00
pat-s	1c1bfb5f6d	chore: align arm64 workflow	2026-04-11 09:47:19 +02:00
pat-s	ce8d74d0eb	refactor: apply same build improvements to arm64 workflow ... Uses install-deps step, precomputed package list, weston compositor, stderr redirect, sequential plan with progress output, and unique X server numbers.	2026-04-11 09:47:04 +02:00
pat-s	9ff676a5cb	chore: don't clean cache	2026-04-10 20:16:05 +02:00
pat-s	4b27430270	fix: use weston instead of mutter as Wayland compositor to reduce memory usage ... Mutter (default xwfb-run compositor) consumes ~200-500MB per instance, causing fork failures with 5 parallel matrix jobs. Weston is much lighter.	2026-04-10 20:15:42 +02:00
pat-s	4892497c76	fix: redirect R message stream to stdout to bypass xwfb-run stderr swallowing	2026-04-10 17:48:29 +02:00
pat-s	112bc60ec2	debug: add per-package progress output to stdout to diagnose hang vs buffering	2026-04-10 17:30:20 +02:00
pat-s	270843eaf9	refactor: precompute package list in install-deps step to reduce matrix job memory usage ... Move packages-to-build.R execution to the single install-deps step and save results as RDS. Matrix jobs read the precomputed list instead of loading the massive CRAN archive/release databases, freeing memory for fork/system calls.	2026-04-10 16:21:33 +02:00
pat-s	e53f11a896	fix: one-time full R-pkgs cache wipe to fix corrupted .so files from previous failed builds	2026-04-10 16:04:49 +02:00
pat-s	858f13140b	fix: clear pak cache before installing bincraft to ensure new version is picked up	2026-04-10 15:02:10 +02:00
pat-s	6da70e5074	fix: use unique X server number per matrix job to prevent Xwayland collisions	2026-04-10 12:43:11 +02:00
pat-s	3f985b3bc7	fix: use plan('sequential') instead of removing future, bincraft needs it internally for future_mapply	2026-04-10 12:41:09 +02:00
pat-s	f831db042c	fix: use plan('multisession') instead of plan('sequential') for future_mapply compatibility	2026-04-10 12:37:44 +02:00
pat-s	1ff6a0c32a	fix: use plan('sequential') instead of removing future, bincraft needs it internally for future_mapply	2026-04-10 08:18:53 +02:00
pat-s	7ca1b6e5c6	fix: remove pak::sysreqs_db_update() from matrix step, already cached by install-deps	2026-04-10 00:28:34 +02:00
pat-s	7bd77a2873	fix: stop wiping entire R-pkgs cache, only clean locks and bincraft now that cache is rebuilt	2026-04-10 00:17:11 +02:00