devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 10 Pull requests Releases Activity

docs: add spec for multi-R-version image refactor

Browse source 
Image tag drops the R version (e.g., build-env-alpine:3.23 instead of
:3.23-4.5); workflows pick an R version by calling
/opt/R/${R_VERSION}/bin/R explicitly.
Includes three folded-in bug fixes (alpine audit images, package-index
images, ubuntu-2404 R patch).
This commit is contained in:
Patrick Schratz 2026-05-25 20:08:31 +02:00
commit 2432f3ff94
Signed by: pat-s
GPG key ID: 3C6318841EF78925
1 changed files with 242 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

242
docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md Normal file
View file

@ -0,0 +1,242 @@
# Refactor build workflows to multi-R-version images
## Goal
The container images at `reg.devxy.io/rpkgs/build-env-*` are moving from a one-R-version-per-tag model to a multi-R-version-per-tag model.
The image tag now encodes only the OS version (e.g. `build-env-alpine:3.23`), and each image ships several R installs under `/opt/R/<patch>/`.
Workflows and recipes must select an R version explicitly by calling `/opt/R/${R_VERSION}/bin/R` instead of relying on `R`/`Rscript` from `PATH`.
## Scope
In scope:
- Every `.crow/*.yaml` workflow that references a `build-env-*` image (64 files).
- `Justfile` recipes that run `docker run` against a `build-env-*` image (3 recipes).
- The commented-out `build-all-versions-install-deps.yaml` in the repo root (kept consistent so the example doesn't go stale).
- Three correctness bug fixes that the user asked to roll into the same change:
- The six `weekly-audit-missing-alpine-{321,322,323}-{amd64,arm64}.yaml` files all incorrectly use `alpine:3.23-4.5`; each should use its own alpine image.
- The fourteen `update-package-index-*.yaml` files all use `build-env-ubuntu:noble-4.4` regardless of the platform they index; each should use its own platform's image.
- `process-updates-ubuntu-2404-{amd64,arm64}.yaml` use `noble-4.4` while the audit and rebuild counterparts use `noble-4.4.3`; align to 4.4.3.
Out of scope:
- `local/build-all.R` and other R scripts run *inside* a container with `Rscript`. Once R is launched, child processes inherit `R.home()`; the scripts themselves need no change.
- `docker/`, `benchmark/`.
- Historical docs in `docs/superpowers/plans/` and `docs/superpowers/specs/` that reference old image tags.
- Any workflow restructuring beyond image and R-path changes plus the three bug fixes above.
## Image and R-path scheme
New image tag:
```
reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}
```
`R_VERSION` is no longer encoded in the tag.
Each image contains R installs under `/opt/R/<patch>/`, accessed via:
- `/opt/R/${R_VERSION}/bin/R`
- `/opt/R/${R_VERSION}/bin/Rscript`
`R_VERSION` is always a full patch string (e.g. `4.5.3`, `4.4.3`), never a minor (`4.5`).
## Platform → image + R_VERSION mapping
| Platform | New image | `R_VERSION` |
|--------------|-------------------------------------------------|-------------|
| alpine-322 | `reg.devxy.io/rpkgs/build-env-alpine:3.22` | 4.5.3 |
| alpine-323 | `reg.devxy.io/rpkgs/build-env-alpine:3.23` | 4.5.3 |
| ubuntu-2204 | `reg.devxy.io/rpkgs/build-env-ubuntu:jammy` | 4.4.3 |
| ubuntu-2404 | `reg.devxy.io/rpkgs/build-env-ubuntu:noble` | 4.4.3 |
| redhat-8 | `reg.devxy.io/rpkgs/build-env-redhat:8` | 4.4.3 |
| redhat-9 | `reg.devxy.io/rpkgs/build-env-redhat:9` | 4.4.3 |
| redhat-10 | `reg.devxy.io/rpkgs/build-env-redhat:10` | 4.5.3 |
The `alpine-321` platform has no matching new image; its two audit-only workflows fall back to `build-env-alpine:3.23` with `R_VERSION=4.5.3` (rationale in the "Edge cases" section).
## How workflows reference R
Two patterns appear in the repo today:
1. **Hard-coded image, no `R_VERSION` env var.** The R version is implicit in the image tag.
2. **Parameterised image via matrix/`--var`.** `R_VERSION` is already an environment variable; the image tag interpolates `${R_VERSION}`.
After the refactor:
- Pattern (1) workflows gain a single `R_VERSION:` entry in their `environment:` block. All `R …` and `Rscript …` invocations in the `commands:` block become `/opt/R/${R_VERSION}/bin/R …` / `/opt/R/${R_VERSION}/bin/Rscript …`.
- Pattern (2) workflows keep their existing `R_VERSION` value source (caller-supplied `--var`); only the image tag and the R invocations change.
No `PATH` munging, no wrapper script, no shell aliasing.
Every R call site is explicit about which R is invoked.
### Example: pattern (1) before → after
Before (excerpt from `process-updates-alpine-322-amd64.yaml`):
```yaml
- name: 'Processing Updates'
image: reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5
environment:
PLATFORM: alpine-322
ARCH: amd64
# ...
commands:
- R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
- R -q -e 'packageVersion("bincraft")'
- xvfb-run R -q -e "..."
```
After:
```yaml
- name: 'Processing Updates'
image: reg.devxy.io/rpkgs/build-env-alpine:3.22
environment:
PLATFORM: alpine-322
ARCH: amd64
R_VERSION: 4.5.3
# ...
commands:
- /opt/R/${R_VERSION}/bin/R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
- /opt/R/${R_VERSION}/bin/R -q -e 'packageVersion("bincraft")'
- xvfb-run /opt/R/${R_VERSION}/bin/R -q -e "..."
```
### Example: pattern (2) before → after
Before (excerpt from `build-all-versions-amd64.yaml`):
```yaml
- name: 'Build binaries'
image: reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}-${R_VERSION}
commands:
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1
- R -q -e "bincraft::process_unarchived_pkgs(...)"
```
After:
```yaml
- name: 'Build binaries'
image: reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}
commands:
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/${R_VERSION}/bin/Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1
- /opt/R/${R_VERSION}/bin/R -q -e "bincraft::process_unarchived_pkgs(...)"
```
`R_VERSION` (e.g. `4.5.3`) is already supplied by the `crow pipeline create --var` invocations documented in the file header.
## Files touched
### A. `.crow/build-all-versions-*.yaml` (4 files, pattern 2)
- `.crow/build-all-versions-amd64.yaml`
- `.crow/build-all-versions-arm64.yaml`
- `.crow/build-all-versions-install-deps-amd64.yaml`
- `.crow/build-all-versions-install-deps-arm64.yaml`
Change: drop `-${R_VERSION}` from the image tag; substitute the explicit R path in every `R`/`Rscript` invocation.
`R_VERSION` already arrives via `--var`.
### B. `.crow/process-updates-*.yaml` (14 files, pattern 1)
- `process-updates-alpine-322-{amd64,arm64}.yaml`
- `process-updates-alpine-323-{amd64,arm64}.yaml`
- `process-updates-ubuntu-2204-{amd64,arm64}.yaml`
- `process-updates-ubuntu-2404-{amd64,arm64}.yaml`
- `process-updates-redhat-8-{amd64,arm64}.yaml`
- `process-updates-redhat-9-{amd64,arm64}.yaml`
- `process-updates-redhat-10-{amd64,arm64}.yaml`
Change: image swap per mapping table; add `R_VERSION:` env var; substitute R path in every `R`/`Rscript`/`xvfb-run R` invocation.
The two `ubuntu-2404` files also bump from `4.4` to `4.4.3` (bug fix; see "Edge cases").
### C. `.crow/weekly-rebuild-missing-*.yaml` (14 files, pattern 1)
One per platform/arch listed in the mapping table. Same treatment as B.
### D. `.crow/weekly-audit-missing-*.yaml` (16 files, pattern 1)
Same treatment as B, *plus* repointing each alpine audit file to its own alpine image:
| File | New image | `R_VERSION` |
|-------------------------------------------------|--------------------------------------------|-------------|
| `weekly-audit-missing-alpine-321-amd64.yaml` | `build-env-alpine:3.23` (no 3.21 image) | 4.5.3 |
| `weekly-audit-missing-alpine-321-arm64.yaml` | `build-env-alpine:3.23` (no 3.21 image) | 4.5.3 |
| `weekly-audit-missing-alpine-322-amd64.yaml` | `build-env-alpine:3.22` | 4.5.3 |
| `weekly-audit-missing-alpine-322-arm64.yaml` | `build-env-alpine:3.22` | 4.5.3 |
| `weekly-audit-missing-alpine-323-amd64.yaml` | `build-env-alpine:3.23` | 4.5.3 |
| `weekly-audit-missing-alpine-323-arm64.yaml` | `build-env-alpine:3.23` | 4.5.3 |
The non-alpine audit files follow the mapping table directly.
### E. `.crow/update-package-index-*.yaml` (14 files, pattern 1)
Each currently uses `build-env-ubuntu:noble-4.4` regardless of which platform's package index it uploads. Repoint each to its own platform's image and R_VERSION per the mapping table.
Files:
- `update-package-index-alpine-322-{amd64,arm64}.yaml`
- `update-package-index-alpine-323-{amd64,arm64}.yaml`
- `update-package-index-ubuntu-2204-{amd64,arm64}.yaml`
- `update-package-index-ubuntu-2404-{amd64,arm64}.yaml`
- `update-package-index-redhat-8-{amd64,arm64}.yaml`
- `update-package-index-redhat-9-{amd64,arm64}.yaml`
- `update-package-index-redhat-10-{amd64,arm64}.yaml`
The second step in each (`Purge CDN cache`) runs on `alpine:3.23` and does not invoke R; it is unchanged.
### F. `.crow/archive-missed-packages.yaml` (1 file)
Currently uses `build-env-alpine:3.23-4.5`. The image OS doesn't matter for this workflow (it only writes to S3 + Postgres). New: `build-env-alpine:3.23` + `R_VERSION: 4.5.3`. Same R-path substitution as elsewhere.
### G. `.crow/build-r-minor-sensitive-packages.yaml` (1 file)
Special case: uses `docker.io/devxygmbh/rpkgs-build-env-${os}:${os_version}-${r_version}` (lowercase matrix vars; different registry).
Decision (user-confirmed): keep the `docker.io/devxygmbh/` registry. Drop the `-${r_version}` suffix from the image tag, leaving `docker.io/devxygmbh/rpkgs-build-env-${os}:${os_version}`. Substitute every `R`/`Rscript` for `/opt/R/${r_version}/bin/R` / `/opt/R/${r_version}/bin/Rscript`.
The workflow's matrix continues to use `r_version: 4.5` / `4.4`. To remain consistent with the rest of the refactor's "always full patch" rule, the matrix values should be updated to `4.5.3` and `4.4.3` respectively (matching alpine-321's R 4.5.3 and the historical 4.4.3 patch).
### H. `Justfile` (3 recipes)
- `build-all OS OS_VERSION ARCH R_VERSION PACKAGE NCPUS`
- `build-single OS OS_VERSION ARCH R_VERSION PACKAGE TAG NCPUS`
- `process-updates OS OS_VERSION ARCH R_VERSION interval`
Change in each: drop `-{{R_VERSION}}` from the image tag, and replace every `R `/`R -q -e` inside the `bash -c '…'` string with `/opt/R/{{R_VERSION}}/bin/R `/`/opt/R/{{R_VERSION}}/bin/R -q -e`.
The example comments above each recipe (`# just build-all alpine 3.21 arm64 4.5.0 …`) should be updated to use a current platform/R combination (e.g. `alpine 3.22 amd64 4.5.3`).
### I. `build-all-versions-install-deps.yaml` (commented-out, repo root)
Apply the same edits as the active `.crow/build-all-versions-install-deps-*.yaml` files so the commented-out example remains a faithful template.
## Edge cases and bug fixes folded in
1. **Alpine audit images.** All six `weekly-audit-missing-alpine-{321,322,323}-{amd64,arm64}.yaml` files currently point at `alpine:3.23-4.5`. After the refactor, each one points at the image that matches its own alpine version. `alpine-321` has no matching image in the new scheme, so its two files use `build-env-alpine:3.23` (the audit workflow reads `PLATFORM` from env and queries S3/CRAN; the container's own OS does not affect correctness).
2. **Package-index workflows.** All fourteen `update-package-index-*.yaml` files are repointed to their own platform's image, matching the rest of the per-platform workflows.
3. **Ubuntu-2404 R version.** `process-updates-ubuntu-2404-{amd64,arm64}.yaml` move from `noble-4.4` to `build-env-ubuntu:noble` + `R_VERSION: 4.4.3`, matching the audit and rebuild counterparts.
## Validation
There is no automated test suite for workflow files in this repo. Validation is:
1. **Static checks per file**: after edit, grep each touched workflow for leftover bare `R `, `Rscript `, `R -q`, `R -e`, `R CMD` invocations. Any hit that is not part of a longer path (`/opt/R/…/bin/R`) is a regression.
2. **Image tag check**: grep for `build-env-` lines and confirm no tag still contains `-${R_VERSION}`, `-4.4`, `-4.4.3`, `-4.5`, or `-4.5.3`.
3. **Smoke runs**: trigger one workflow per shape on a feature branch and confirm green:
- `process-updates-alpine-322-amd64.yaml`
- `weekly-rebuild-missing-redhat-9-amd64.yaml`
- `weekly-audit-missing-ubuntu-2204-amd64.yaml`
- `update-package-index-redhat-10-amd64.yaml`
- `archive-missed-packages.yaml`
- `build-all-versions-amd64.yaml` (with its `install-deps` predecessor)
- `build-r-minor-sensitive-packages.yaml`
The `Justfile` recipes are exercised by running each once locally against a current platform.
## Risks
- **Wrong `R_VERSION` in a file**: typo in the platform→version mapping causes `/opt/R/<missing>/bin/R: not found`. Mitigated by the static grep in validation and by smoke-running one workflow per shape.
- **`build-r-minor-sensitive-packages.yaml` assumes new images exist at `docker.io/devxygmbh/`**: if the multi-R image is only published to `reg.devxy.io/rpkgs/`, this workflow will fail on the first pull. If that turns out to be the case, switch to option (a) — repoint to `reg.devxy.io/rpkgs/` — as a follow-up.
- **R subprocesses inside scripts**: `pak`, `future`, and similar libraries spawn child R processes via `R.home()`, which is set to the parent's install. No additional action needed.