diff --git a/docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md b/docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md new file mode 100644 index 0000000..482b89a --- /dev/null +++ b/docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md @@ -0,0 +1,242 @@ +# Refactor build workflows to multi-R-version images + +## Goal + +The container images at `reg.devxy.io/rpkgs/build-env-*` are moving from a one-R-version-per-tag model to a multi-R-version-per-tag model. +The image tag now encodes only the OS version (e.g. `build-env-alpine:3.23`), and each image ships several R installs under `/opt/R//`. +Workflows and recipes must select an R version explicitly by calling `/opt/R/${R_VERSION}/bin/R` instead of relying on `R`/`Rscript` from `PATH`. + +## Scope + +In scope: + +- Every `.crow/*.yaml` workflow that references a `build-env-*` image (64 files). +- `Justfile` recipes that run `docker run` against a `build-env-*` image (3 recipes). +- The commented-out `build-all-versions-install-deps.yaml` in the repo root (kept consistent so the example doesn't go stale). +- Three correctness bug fixes that the user asked to roll into the same change: + - The six `weekly-audit-missing-alpine-{321,322,323}-{amd64,arm64}.yaml` files all incorrectly use `alpine:3.23-4.5`; each should use its own alpine image. + - The fourteen `update-package-index-*.yaml` files all use `build-env-ubuntu:noble-4.4` regardless of the platform they index; each should use its own platform's image. + - `process-updates-ubuntu-2404-{amd64,arm64}.yaml` use `noble-4.4` while the audit and rebuild counterparts use `noble-4.4.3`; align to 4.4.3. + +Out of scope: + +- `local/build-all.R` and other R scripts run *inside* a container with `Rscript`. Once R is launched, child processes inherit `R.home()`; the scripts themselves need no change. +- `docker/`, `benchmark/`. +- Historical docs in `docs/superpowers/plans/` and `docs/superpowers/specs/` that reference old image tags. +- Any workflow restructuring beyond image and R-path changes plus the three bug fixes above. + +## Image and R-path scheme + +New image tag: + +``` +reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION} +``` + +`R_VERSION` is no longer encoded in the tag. +Each image contains R installs under `/opt/R//`, accessed via: + +- `/opt/R/${R_VERSION}/bin/R` +- `/opt/R/${R_VERSION}/bin/Rscript` + +`R_VERSION` is always a full patch string (e.g. `4.5.3`, `4.4.3`), never a minor (`4.5`). + +## Platform → image + R_VERSION mapping + +| Platform | New image | `R_VERSION` | +|--------------|-------------------------------------------------|-------------| +| alpine-322 | `reg.devxy.io/rpkgs/build-env-alpine:3.22` | 4.5.3 | +| alpine-323 | `reg.devxy.io/rpkgs/build-env-alpine:3.23` | 4.5.3 | +| ubuntu-2204 | `reg.devxy.io/rpkgs/build-env-ubuntu:jammy` | 4.4.3 | +| ubuntu-2404 | `reg.devxy.io/rpkgs/build-env-ubuntu:noble` | 4.4.3 | +| redhat-8 | `reg.devxy.io/rpkgs/build-env-redhat:8` | 4.4.3 | +| redhat-9 | `reg.devxy.io/rpkgs/build-env-redhat:9` | 4.4.3 | +| redhat-10 | `reg.devxy.io/rpkgs/build-env-redhat:10` | 4.5.3 | + +The `alpine-321` platform has no matching new image; its two audit-only workflows fall back to `build-env-alpine:3.23` with `R_VERSION=4.5.3` (rationale in the "Edge cases" section). + +## How workflows reference R + +Two patterns appear in the repo today: + +1. **Hard-coded image, no `R_VERSION` env var.** The R version is implicit in the image tag. +2. **Parameterised image via matrix/`--var`.** `R_VERSION` is already an environment variable; the image tag interpolates `${R_VERSION}`. + +After the refactor: + +- Pattern (1) workflows gain a single `R_VERSION:` entry in their `environment:` block. All `R …` and `Rscript …` invocations in the `commands:` block become `/opt/R/${R_VERSION}/bin/R …` / `/opt/R/${R_VERSION}/bin/Rscript …`. +- Pattern (2) workflows keep their existing `R_VERSION` value source (caller-supplied `--var`); only the image tag and the R invocations change. + +No `PATH` munging, no wrapper script, no shell aliasing. +Every R call site is explicit about which R is invoked. + +### Example: pattern (1) before → after + +Before (excerpt from `process-updates-alpine-322-amd64.yaml`): + +```yaml +- name: 'Processing Updates' + image: reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 + environment: + PLATFORM: alpine-322 + ARCH: amd64 + # ... + commands: + - R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")' + - R -q -e 'packageVersion("bincraft")' + - xvfb-run R -q -e "..." +``` + +After: + +```yaml +- name: 'Processing Updates' + image: reg.devxy.io/rpkgs/build-env-alpine:3.22 + environment: + PLATFORM: alpine-322 + ARCH: amd64 + R_VERSION: 4.5.3 + # ... + commands: + - /opt/R/${R_VERSION}/bin/R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")' + - /opt/R/${R_VERSION}/bin/R -q -e 'packageVersion("bincraft")' + - xvfb-run /opt/R/${R_VERSION}/bin/R -q -e "..." +``` + +### Example: pattern (2) before → after + +Before (excerpt from `build-all-versions-amd64.yaml`): + +```yaml +- name: 'Build binaries' + image: reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}-${R_VERSION} + commands: + - $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 + - R -q -e "bincraft::process_unarchived_pkgs(...)" +``` + +After: + +```yaml +- name: 'Build binaries' + image: reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION} + commands: + - $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/${R_VERSION}/bin/Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 + - /opt/R/${R_VERSION}/bin/R -q -e "bincraft::process_unarchived_pkgs(...)" +``` + +`R_VERSION` (e.g. `4.5.3`) is already supplied by the `crow pipeline create --var` invocations documented in the file header. + +## Files touched + +### A. `.crow/build-all-versions-*.yaml` (4 files, pattern 2) + +- `.crow/build-all-versions-amd64.yaml` +- `.crow/build-all-versions-arm64.yaml` +- `.crow/build-all-versions-install-deps-amd64.yaml` +- `.crow/build-all-versions-install-deps-arm64.yaml` + +Change: drop `-${R_VERSION}` from the image tag; substitute the explicit R path in every `R`/`Rscript` invocation. +`R_VERSION` already arrives via `--var`. + +### B. `.crow/process-updates-*.yaml` (14 files, pattern 1) + +- `process-updates-alpine-322-{amd64,arm64}.yaml` +- `process-updates-alpine-323-{amd64,arm64}.yaml` +- `process-updates-ubuntu-2204-{amd64,arm64}.yaml` +- `process-updates-ubuntu-2404-{amd64,arm64}.yaml` +- `process-updates-redhat-8-{amd64,arm64}.yaml` +- `process-updates-redhat-9-{amd64,arm64}.yaml` +- `process-updates-redhat-10-{amd64,arm64}.yaml` + +Change: image swap per mapping table; add `R_VERSION:` env var; substitute R path in every `R`/`Rscript`/`xvfb-run R` invocation. +The two `ubuntu-2404` files also bump from `4.4` to `4.4.3` (bug fix; see "Edge cases"). + +### C. `.crow/weekly-rebuild-missing-*.yaml` (14 files, pattern 1) + +One per platform/arch listed in the mapping table. Same treatment as B. + +### D. `.crow/weekly-audit-missing-*.yaml` (16 files, pattern 1) + +Same treatment as B, *plus* repointing each alpine audit file to its own alpine image: + +| File | New image | `R_VERSION` | +|-------------------------------------------------|--------------------------------------------|-------------| +| `weekly-audit-missing-alpine-321-amd64.yaml` | `build-env-alpine:3.23` (no 3.21 image) | 4.5.3 | +| `weekly-audit-missing-alpine-321-arm64.yaml` | `build-env-alpine:3.23` (no 3.21 image) | 4.5.3 | +| `weekly-audit-missing-alpine-322-amd64.yaml` | `build-env-alpine:3.22` | 4.5.3 | +| `weekly-audit-missing-alpine-322-arm64.yaml` | `build-env-alpine:3.22` | 4.5.3 | +| `weekly-audit-missing-alpine-323-amd64.yaml` | `build-env-alpine:3.23` | 4.5.3 | +| `weekly-audit-missing-alpine-323-arm64.yaml` | `build-env-alpine:3.23` | 4.5.3 | + +The non-alpine audit files follow the mapping table directly. + +### E. `.crow/update-package-index-*.yaml` (14 files, pattern 1) + +Each currently uses `build-env-ubuntu:noble-4.4` regardless of which platform's package index it uploads. Repoint each to its own platform's image and R_VERSION per the mapping table. + +Files: + +- `update-package-index-alpine-322-{amd64,arm64}.yaml` +- `update-package-index-alpine-323-{amd64,arm64}.yaml` +- `update-package-index-ubuntu-2204-{amd64,arm64}.yaml` +- `update-package-index-ubuntu-2404-{amd64,arm64}.yaml` +- `update-package-index-redhat-8-{amd64,arm64}.yaml` +- `update-package-index-redhat-9-{amd64,arm64}.yaml` +- `update-package-index-redhat-10-{amd64,arm64}.yaml` + +The second step in each (`Purge CDN cache`) runs on `alpine:3.23` and does not invoke R; it is unchanged. + +### F. `.crow/archive-missed-packages.yaml` (1 file) + +Currently uses `build-env-alpine:3.23-4.5`. The image OS doesn't matter for this workflow (it only writes to S3 + Postgres). New: `build-env-alpine:3.23` + `R_VERSION: 4.5.3`. Same R-path substitution as elsewhere. + +### G. `.crow/build-r-minor-sensitive-packages.yaml` (1 file) + +Special case: uses `docker.io/devxygmbh/rpkgs-build-env-${os}:${os_version}-${r_version}` (lowercase matrix vars; different registry). + +Decision (user-confirmed): keep the `docker.io/devxygmbh/` registry. Drop the `-${r_version}` suffix from the image tag, leaving `docker.io/devxygmbh/rpkgs-build-env-${os}:${os_version}`. Substitute every `R`/`Rscript` for `/opt/R/${r_version}/bin/R` / `/opt/R/${r_version}/bin/Rscript`. + +The workflow's matrix continues to use `r_version: 4.5` / `4.4`. To remain consistent with the rest of the refactor's "always full patch" rule, the matrix values should be updated to `4.5.3` and `4.4.3` respectively (matching alpine-321's R 4.5.3 and the historical 4.4.3 patch). + +### H. `Justfile` (3 recipes) + +- `build-all OS OS_VERSION ARCH R_VERSION PACKAGE NCPUS` +- `build-single OS OS_VERSION ARCH R_VERSION PACKAGE TAG NCPUS` +- `process-updates OS OS_VERSION ARCH R_VERSION interval` + +Change in each: drop `-{{R_VERSION}}` from the image tag, and replace every `R `/`R -q -e` inside the `bash -c '…'` string with `/opt/R/{{R_VERSION}}/bin/R `/`/opt/R/{{R_VERSION}}/bin/R -q -e`. +The example comments above each recipe (`# just build-all alpine 3.21 arm64 4.5.0 …`) should be updated to use a current platform/R combination (e.g. `alpine 3.22 amd64 4.5.3`). + +### I. `build-all-versions-install-deps.yaml` (commented-out, repo root) + +Apply the same edits as the active `.crow/build-all-versions-install-deps-*.yaml` files so the commented-out example remains a faithful template. + +## Edge cases and bug fixes folded in + +1. **Alpine audit images.** All six `weekly-audit-missing-alpine-{321,322,323}-{amd64,arm64}.yaml` files currently point at `alpine:3.23-4.5`. After the refactor, each one points at the image that matches its own alpine version. `alpine-321` has no matching image in the new scheme, so its two files use `build-env-alpine:3.23` (the audit workflow reads `PLATFORM` from env and queries S3/CRAN; the container's own OS does not affect correctness). +2. **Package-index workflows.** All fourteen `update-package-index-*.yaml` files are repointed to their own platform's image, matching the rest of the per-platform workflows. +3. **Ubuntu-2404 R version.** `process-updates-ubuntu-2404-{amd64,arm64}.yaml` move from `noble-4.4` to `build-env-ubuntu:noble` + `R_VERSION: 4.4.3`, matching the audit and rebuild counterparts. + +## Validation + +There is no automated test suite for workflow files in this repo. Validation is: + +1. **Static checks per file**: after edit, grep each touched workflow for leftover bare `R `, `Rscript `, `R -q`, `R -e`, `R CMD` invocations. Any hit that is not part of a longer path (`/opt/R/…/bin/R`) is a regression. +2. **Image tag check**: grep for `build-env-` lines and confirm no tag still contains `-${R_VERSION}`, `-4.4`, `-4.4.3`, `-4.5`, or `-4.5.3`. +3. **Smoke runs**: trigger one workflow per shape on a feature branch and confirm green: + - `process-updates-alpine-322-amd64.yaml` + - `weekly-rebuild-missing-redhat-9-amd64.yaml` + - `weekly-audit-missing-ubuntu-2204-amd64.yaml` + - `update-package-index-redhat-10-amd64.yaml` + - `archive-missed-packages.yaml` + - `build-all-versions-amd64.yaml` (with its `install-deps` predecessor) + - `build-r-minor-sensitive-packages.yaml` + +The `Justfile` recipes are exercised by running each once locally against a current platform. + +## Risks + +- **Wrong `R_VERSION` in a file**: typo in the platform→version mapping causes `/opt/R//bin/R: not found`. Mitigated by the static grep in validation and by smoke-running one workflow per shape. +- **`build-r-minor-sensitive-packages.yaml` assumes new images exist at `docker.io/devxygmbh/`**: if the multi-R image is only published to `reg.devxy.io/rpkgs/`, this workflow will fail on the first pull. If that turns out to be the case, switch to option (a) — repoint to `reg.devxy.io/rpkgs/` — as a follow-up. +- **R subprocesses inside scripts**: `pak`, `future`, and similar libraries spawn child R processes via `R.home()`, which is set to the parent's install. No additional action needed.