docs: add spec for multi-R-version image refactor
Image tag drops the R version (e.g., build-env-alpine:3.23 instead of
:3.23-4.5); workflows pick an R version by calling
/opt/R/${R_VERSION}/bin/R explicitly.
Includes three folded-in bug fixes (alpine audit images, package-index
images, ubuntu-2404 R patch).
This commit is contained in:
parent
8e453a56ce
commit
2432f3ff94
GPG key ID:
3C6318841EF78925
1 changed files with 242 additions and 0 deletions
|
|
@ -0,0 +1,242 @@
|
|||
# Refactor build workflows to multi-R-version images
|
||||
|
||||
## Goal
|
||||
|
||||
The container images at `reg.devxy.io/rpkgs/build-env-*` are moving from a one-R-version-per-tag model to a multi-R-version-per-tag model.
|
||||
The image tag now encodes only the OS version (e.g. `build-env-alpine:3.23`), and each image ships several R installs under `/opt/R/<patch>/`.
|
||||
Workflows and recipes must select an R version explicitly by calling `/opt/R/${R_VERSION}/bin/R` instead of relying on `R`/`Rscript` from `PATH`.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- Every `.crow/*.yaml` workflow that references a `build-env-*` image (64 files).
|
||||
- `Justfile` recipes that run `docker run` against a `build-env-*` image (3 recipes).
|
||||
- The commented-out `build-all-versions-install-deps.yaml` in the repo root (kept consistent so the example doesn't go stale).
|
||||
- Three correctness bug fixes that the user asked to roll into the same change:
|
||||
- The six `weekly-audit-missing-alpine-{321,322,323}-{amd64,arm64}.yaml` files all incorrectly use `alpine:3.23-4.5`; each should use its own alpine image.
|
||||
- The fourteen `update-package-index-*.yaml` files all use `build-env-ubuntu:noble-4.4` regardless of the platform they index; each should use its own platform's image.
|
||||
- `process-updates-ubuntu-2404-{amd64,arm64}.yaml` use `noble-4.4` while the audit and rebuild counterparts use `noble-4.4.3`; align to 4.4.3.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- `local/build-all.R` and other R scripts run *inside* a container with `Rscript`. Once R is launched, child processes inherit `R.home()`; the scripts themselves need no change.
|
||||
- `docker/`, `benchmark/`.
|
||||
- Historical docs in `docs/superpowers/plans/` and `docs/superpowers/specs/` that reference old image tags.
|
||||
- Any workflow restructuring beyond image and R-path changes plus the three bug fixes above.
|
||||
|
||||
## Image and R-path scheme
|
||||
|
||||
New image tag:
|
||||
|
||||
```
|
||||
reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}
|
||||
```
|
||||
|
||||
`R_VERSION` is no longer encoded in the tag.
|
||||
Each image contains R installs under `/opt/R/<patch>/`, accessed via:
|
||||
|
||||
- `/opt/R/${R_VERSION}/bin/R`
|
||||
- `/opt/R/${R_VERSION}/bin/Rscript`
|
||||
|
||||
`R_VERSION` is always a full patch string (e.g. `4.5.3`, `4.4.3`), never a minor (`4.5`).
|
||||
|
||||
## Platform → image + R_VERSION mapping
|
||||
|
||||
| Platform | New image | `R_VERSION` |
|
||||
|--------------|-------------------------------------------------|-------------|
|
||||
| alpine-322 | `reg.devxy.io/rpkgs/build-env-alpine:3.22` | 4.5.3 |
|
||||
| alpine-323 | `reg.devxy.io/rpkgs/build-env-alpine:3.23` | 4.5.3 |
|
||||
| ubuntu-2204 | `reg.devxy.io/rpkgs/build-env-ubuntu:jammy` | 4.4.3 |
|
||||
| ubuntu-2404 | `reg.devxy.io/rpkgs/build-env-ubuntu:noble` | 4.4.3 |
|
||||
| redhat-8 | `reg.devxy.io/rpkgs/build-env-redhat:8` | 4.4.3 |
|
||||
| redhat-9 | `reg.devxy.io/rpkgs/build-env-redhat:9` | 4.4.3 |
|
||||
| redhat-10 | `reg.devxy.io/rpkgs/build-env-redhat:10` | 4.5.3 |
|
||||
|
||||
The `alpine-321` platform has no matching new image; its two audit-only workflows fall back to `build-env-alpine:3.23` with `R_VERSION=4.5.3` (rationale in the "Edge cases" section).
|
||||
|
||||
## How workflows reference R
|
||||
|
||||
Two patterns appear in the repo today:
|
||||
|
||||
1. **Hard-coded image, no `R_VERSION` env var.** The R version is implicit in the image tag.
|
||||
2. **Parameterised image via matrix/`--var`.** `R_VERSION` is already an environment variable; the image tag interpolates `${R_VERSION}`.
|
||||
|
||||
After the refactor:
|
||||
|
||||
- Pattern (1) workflows gain a single `R_VERSION:` entry in their `environment:` block. All `R …` and `Rscript …` invocations in the `commands:` block become `/opt/R/${R_VERSION}/bin/R …` / `/opt/R/${R_VERSION}/bin/Rscript …`.
|
||||
- Pattern (2) workflows keep their existing `R_VERSION` value source (caller-supplied `--var`); only the image tag and the R invocations change.
|
||||
|
||||
No `PATH` munging, no wrapper script, no shell aliasing.
|
||||
Every R call site is explicit about which R is invoked.
|
||||
|
||||
### Example: pattern (1) before → after
|
||||
|
||||
Before (excerpt from `process-updates-alpine-322-amd64.yaml`):
|
||||
|
||||
```yaml
|
||||
- name: 'Processing Updates'
|
||||
image: reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5
|
||||
environment:
|
||||
PLATFORM: alpine-322
|
||||
ARCH: amd64
|
||||
# ...
|
||||
commands:
|
||||
- R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
|
||||
- R -q -e 'packageVersion("bincraft")'
|
||||
- xvfb-run R -q -e "..."
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```yaml
|
||||
- name: 'Processing Updates'
|
||||
image: reg.devxy.io/rpkgs/build-env-alpine:3.22
|
||||
environment:
|
||||
PLATFORM: alpine-322
|
||||
ARCH: amd64
|
||||
R_VERSION: 4.5.3
|
||||
# ...
|
||||
commands:
|
||||
- /opt/R/${R_VERSION}/bin/R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
|
||||
- /opt/R/${R_VERSION}/bin/R -q -e 'packageVersion("bincraft")'
|
||||
- xvfb-run /opt/R/${R_VERSION}/bin/R -q -e "..."
|
||||
```
|
||||
|
||||
### Example: pattern (2) before → after
|
||||
|
||||
Before (excerpt from `build-all-versions-amd64.yaml`):
|
||||
|
||||
```yaml
|
||||
- name: 'Build binaries'
|
||||
image: reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}-${R_VERSION}
|
||||
commands:
|
||||
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1
|
||||
- R -q -e "bincraft::process_unarchived_pkgs(...)"
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```yaml
|
||||
- name: 'Build binaries'
|
||||
image: reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}
|
||||
commands:
|
||||
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/${R_VERSION}/bin/Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1
|
||||
- /opt/R/${R_VERSION}/bin/R -q -e "bincraft::process_unarchived_pkgs(...)"
|
||||
```
|
||||
|
||||
`R_VERSION` (e.g. `4.5.3`) is already supplied by the `crow pipeline create --var` invocations documented in the file header.
|
||||
|
||||
## Files touched
|
||||
|
||||
### A. `.crow/build-all-versions-*.yaml` (4 files, pattern 2)
|
||||
|
||||
- `.crow/build-all-versions-amd64.yaml`
|
||||
- `.crow/build-all-versions-arm64.yaml`
|
||||
- `.crow/build-all-versions-install-deps-amd64.yaml`
|
||||
- `.crow/build-all-versions-install-deps-arm64.yaml`
|
||||
|
||||
Change: drop `-${R_VERSION}` from the image tag; substitute the explicit R path in every `R`/`Rscript` invocation.
|
||||
`R_VERSION` already arrives via `--var`.
|
||||
|
||||
### B. `.crow/process-updates-*.yaml` (14 files, pattern 1)
|
||||
|
||||
- `process-updates-alpine-322-{amd64,arm64}.yaml`
|
||||
- `process-updates-alpine-323-{amd64,arm64}.yaml`
|
||||
- `process-updates-ubuntu-2204-{amd64,arm64}.yaml`
|
||||
- `process-updates-ubuntu-2404-{amd64,arm64}.yaml`
|
||||
- `process-updates-redhat-8-{amd64,arm64}.yaml`
|
||||
- `process-updates-redhat-9-{amd64,arm64}.yaml`
|
||||
- `process-updates-redhat-10-{amd64,arm64}.yaml`
|
||||
|
||||
Change: image swap per mapping table; add `R_VERSION:` env var; substitute R path in every `R`/`Rscript`/`xvfb-run R` invocation.
|
||||
The two `ubuntu-2404` files also bump from `4.4` to `4.4.3` (bug fix; see "Edge cases").
|
||||
|
||||
### C. `.crow/weekly-rebuild-missing-*.yaml` (14 files, pattern 1)
|
||||
|
||||
One per platform/arch listed in the mapping table. Same treatment as B.
|
||||
|
||||
### D. `.crow/weekly-audit-missing-*.yaml` (16 files, pattern 1)
|
||||
|
||||
Same treatment as B, *plus* repointing each alpine audit file to its own alpine image:
|
||||
|
||||
| File | New image | `R_VERSION` |
|
||||
|-------------------------------------------------|--------------------------------------------|-------------|
|
||||
| `weekly-audit-missing-alpine-321-amd64.yaml` | `build-env-alpine:3.23` (no 3.21 image) | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-321-arm64.yaml` | `build-env-alpine:3.23` (no 3.21 image) | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-322-amd64.yaml` | `build-env-alpine:3.22` | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-322-arm64.yaml` | `build-env-alpine:3.22` | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-323-amd64.yaml` | `build-env-alpine:3.23` | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-323-arm64.yaml` | `build-env-alpine:3.23` | 4.5.3 |
|
||||
|
||||
The non-alpine audit files follow the mapping table directly.
|
||||
|
||||
### E. `.crow/update-package-index-*.yaml` (14 files, pattern 1)
|
||||
|
||||
Each currently uses `build-env-ubuntu:noble-4.4` regardless of which platform's package index it uploads. Repoint each to its own platform's image and R_VERSION per the mapping table.
|
||||
|
||||
Files:
|
||||
|
||||
- `update-package-index-alpine-322-{amd64,arm64}.yaml`
|
||||
- `update-package-index-alpine-323-{amd64,arm64}.yaml`
|
||||
- `update-package-index-ubuntu-2204-{amd64,arm64}.yaml`
|
||||
- `update-package-index-ubuntu-2404-{amd64,arm64}.yaml`
|
||||
- `update-package-index-redhat-8-{amd64,arm64}.yaml`
|
||||
- `update-package-index-redhat-9-{amd64,arm64}.yaml`
|
||||
- `update-package-index-redhat-10-{amd64,arm64}.yaml`
|
||||
|
||||
The second step in each (`Purge CDN cache`) runs on `alpine:3.23` and does not invoke R; it is unchanged.
|
||||
|
||||
### F. `.crow/archive-missed-packages.yaml` (1 file)
|
||||
|
||||
Currently uses `build-env-alpine:3.23-4.5`. The image OS doesn't matter for this workflow (it only writes to S3 + Postgres). New: `build-env-alpine:3.23` + `R_VERSION: 4.5.3`. Same R-path substitution as elsewhere.
|
||||
|
||||
### G. `.crow/build-r-minor-sensitive-packages.yaml` (1 file)
|
||||
|
||||
Special case: uses `docker.io/devxygmbh/rpkgs-build-env-${os}:${os_version}-${r_version}` (lowercase matrix vars; different registry).
|
||||
|
||||
Decision (user-confirmed): keep the `docker.io/devxygmbh/` registry. Drop the `-${r_version}` suffix from the image tag, leaving `docker.io/devxygmbh/rpkgs-build-env-${os}:${os_version}`. Substitute every `R`/`Rscript` for `/opt/R/${r_version}/bin/R` / `/opt/R/${r_version}/bin/Rscript`.
|
||||
|
||||
The workflow's matrix continues to use `r_version: 4.5` / `4.4`. To remain consistent with the rest of the refactor's "always full patch" rule, the matrix values should be updated to `4.5.3` and `4.4.3` respectively (matching alpine-321's R 4.5.3 and the historical 4.4.3 patch).
|
||||
|
||||
### H. `Justfile` (3 recipes)
|
||||
|
||||
- `build-all OS OS_VERSION ARCH R_VERSION PACKAGE NCPUS`
|
||||
- `build-single OS OS_VERSION ARCH R_VERSION PACKAGE TAG NCPUS`
|
||||
- `process-updates OS OS_VERSION ARCH R_VERSION interval`
|
||||
|
||||
Change in each: drop `-{{R_VERSION}}` from the image tag, and replace every `R `/`R -q -e` inside the `bash -c '…'` string with `/opt/R/{{R_VERSION}}/bin/R `/`/opt/R/{{R_VERSION}}/bin/R -q -e`.
|
||||
The example comments above each recipe (`# just build-all alpine 3.21 arm64 4.5.0 …`) should be updated to use a current platform/R combination (e.g. `alpine 3.22 amd64 4.5.3`).
|
||||
|
||||
### I. `build-all-versions-install-deps.yaml` (commented-out, repo root)
|
||||
|
||||
Apply the same edits as the active `.crow/build-all-versions-install-deps-*.yaml` files so the commented-out example remains a faithful template.
|
||||
|
||||
## Edge cases and bug fixes folded in
|
||||
|
||||
1. **Alpine audit images.** All six `weekly-audit-missing-alpine-{321,322,323}-{amd64,arm64}.yaml` files currently point at `alpine:3.23-4.5`. After the refactor, each one points at the image that matches its own alpine version. `alpine-321` has no matching image in the new scheme, so its two files use `build-env-alpine:3.23` (the audit workflow reads `PLATFORM` from env and queries S3/CRAN; the container's own OS does not affect correctness).
|
||||
2. **Package-index workflows.** All fourteen `update-package-index-*.yaml` files are repointed to their own platform's image, matching the rest of the per-platform workflows.
|
||||
3. **Ubuntu-2404 R version.** `process-updates-ubuntu-2404-{amd64,arm64}.yaml` move from `noble-4.4` to `build-env-ubuntu:noble` + `R_VERSION: 4.4.3`, matching the audit and rebuild counterparts.
|
||||
|
||||
## Validation
|
||||
|
||||
There is no automated test suite for workflow files in this repo. Validation is:
|
||||
|
||||
1. **Static checks per file**: after edit, grep each touched workflow for leftover bare `R `, `Rscript `, `R -q`, `R -e`, `R CMD` invocations. Any hit that is not part of a longer path (`/opt/R/…/bin/R`) is a regression.
|
||||
2. **Image tag check**: grep for `build-env-` lines and confirm no tag still contains `-${R_VERSION}`, `-4.4`, `-4.4.3`, `-4.5`, or `-4.5.3`.
|
||||
3. **Smoke runs**: trigger one workflow per shape on a feature branch and confirm green:
|
||||
- `process-updates-alpine-322-amd64.yaml`
|
||||
- `weekly-rebuild-missing-redhat-9-amd64.yaml`
|
||||
- `weekly-audit-missing-ubuntu-2204-amd64.yaml`
|
||||
- `update-package-index-redhat-10-amd64.yaml`
|
||||
- `archive-missed-packages.yaml`
|
||||
- `build-all-versions-amd64.yaml` (with its `install-deps` predecessor)
|
||||
- `build-r-minor-sensitive-packages.yaml`
|
||||
|
||||
The `Justfile` recipes are exercised by running each once locally against a current platform.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Wrong `R_VERSION` in a file**: typo in the platform→version mapping causes `/opt/R/<missing>/bin/R: not found`. Mitigated by the static grep in validation and by smoke-running one workflow per shape.
|
||||
- **`build-r-minor-sensitive-packages.yaml` assumes new images exist at `docker.io/devxygmbh/`**: if the multi-R image is only published to `reg.devxy.io/rpkgs/`, this workflow will fail on the first pull. If that turns out to be the case, switch to option (a) — repoint to `reg.devxy.io/rpkgs/` — as a follow-up.
|
||||
- **R subprocesses inside scripts**: `pak`, `future`, and similar libraries spawn child R processes via `R.home()`, which is set to the parent's install. No additional action needed.
|
||||
Loading…
Reference in a new issue