refactor(ci): use multi-R-version images across workflows (#70)
Some checks failed
ci/crow/cron/update-package-index-alpine-322-arm64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-323-amd64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-323-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-10-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-10-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-8-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-8-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-9-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-9-arm64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2204-amd64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2204-arm64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2404-amd64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2404-arm64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-322-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline failed

## Summary

- Switches all 64 `.crow/*.yaml` workflows, the `Justfile`, and the commented-out `build-all-versions-install-deps.yaml` template to the new multi-R-version container images.
- Each image tag drops `-${R_VERSION}` (e.g. `build-env-alpine:3.22-4.5` → `build-env-alpine:3.22`); workflows pick an R version by calling `/opt/R/${R_VERSION}/bin/R` explicitly.
- Pattern-1 workflows (hard-coded image tag) gain a single `R_VERSION:` env var per file.
- Pattern-2 workflows (`build-all-versions-*`, matrix-driven) continue to receive `R_VERSION` via `crow pipeline create --var`.

## Platform → R_VERSION mapping

| Platform | Image | R_VERSION |
| --- | --- | --- |
| alpine 3.22 / 3.23 | `build-env-alpine:3.{22,23}` | 4.5.3 |
| ubuntu jammy / noble | `build-env-ubuntu:{jammy,noble}` | 4.4.3 |
| redhat 8 / 9 | `build-env-redhat:{8,9}` | 4.4.3 |
| redhat 10 | `build-env-redhat:10` | 4.5.3 |

The `alpine-321` audit-only workflows fall back to `build-env-alpine:3.23` since no 3.21 image exists in the new scheme; the audits only query S3/CRAN, so the container OS does not affect correctness.

## Bug fixes folded in

1. All six `weekly-audit-missing-alpine-*.yaml` files were pinned to `alpine:3.23-4.5` regardless of platform.
   Now each one points at its own alpine image.
2. All fourteen `update-package-index-*.yaml` files were pinned to `build-env-ubuntu:noble-4.4` (or `noble-4.5` for `redhat-10-amd64`) regardless of platform.
   Now each one uses its own platform's image.
3. `process-updates-ubuntu-2404-{amd64,arm64}.yaml` had `R_VERSION` of `4.4` (no patch) baked into the tag.
   Aligned to `4.4.3`, matching the audit and rebuild counterparts.

## Smoke targets (post-merge)

One workflow per shape, ready to dry-run:

- `process-updates-alpine-322-amd64.yaml`
- `weekly-rebuild-missing-redhat-9-amd64.yaml`
- `weekly-audit-missing-ubuntu-2204-amd64.yaml`
- `update-package-index-redhat-10-amd64.yaml`
- `archive-missed-packages.yaml`
- `build-all-versions-amd64.yaml` (with its install-deps predecessor)
- `build-r-minor-sensitive-packages.yaml`

## Follow-ups (out of scope, surfaced during review)

- `process-updates-redhat-10-{amd64,arm64}.yaml` set `codename = "rhel9"` in their `upload_package_index(...)` calls.
  Pre-existing bug — should be `"rhel10"`. The `update-package-index-redhat-10-*` counterparts already use `"rhel10"` correctly.
- `update-package-index-redhat-8-amd64.yaml` is missing the `packageVersion("bincraft")` diagnostic line that every sibling file has.
  Pre-existing asymmetry between amd64 and arm64.

## Spec & plan

- Spec: `docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md`
- Plan: `docs/superpowers/plans/2026-05-25-multi-r-version-images.md`

Reviewed-on: #70
This commit is contained in:
Patrick Schratz 2026-05-26 08:51:27 +00:00 committed by Patrick Schratz
commit 6faa3772d1

View file

@ -1,246 +0,0 @@
# Weekly Missing Binaries Audit & Rebuild
## Goal
A weekly CI workflow that identifies CRAN packages whose latest release version has no binary available, reports them in Forgejo issues grouped by OS family and arch, and rebuilds those that have no prior build failure recorded in the database.
## Architecture Overview
Two independent workflow sets, each with one file per platform/arch combo (14 files each), plus a shared R script and an excluded-packages config file.
```
weekly-audit-missing-*-*.yaml (14 files)
local/weekly-missing-binaries-audit.R
├── Updates Forgejo issues (3 issues, one per OS family)
└── Writes RDS files to /mnt/cache/packages/
weekly-rebuild-missing-*-*.yaml (14 files)
Reads RDS, builds missing packages via bincraft::build_binary_package()
```
The audit and build workflows are fully independent.
Either can be triggered on its own via cron or manually.
## Platforms
All current platform/arch combinations:
| Platform | Arch | Image | Codename |
|-------------|-------|-------------------------------------------------|------------|
| ubuntu-2204 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | jammy |
| ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | jammy |
| ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | noble |
| ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | noble |
| alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | alpine321 |
| alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | alpine321 |
| alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | alpine322 |
| alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | alpine322 |
| alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | alpine323 |
| alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | alpine323 |
| redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | rhel8 |
| redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | rhel8 |
| redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | rhel9 |
| redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | rhel9 |
## Component 1: Excluded Packages Config
**File:** `local/excluded-packages.json`
A JSON array of objects with `package` and `reason` fields:
```json
[
{"package": "RInno", "reason": "windows-only"},
{"package": "KeyboardSimulator", "reason": "windows-only"},
{"package": "doBy", "reason": "hang"},
{"package": "frailtypack", "reason": "hang"},
...
]
```
This file is the single source of truth for packages that should be skipped.
Both the audit script and rebuild workflows read from it.
The existing `build-all-versions-*.yaml` workflows retain their inline lists for now (migration is out of scope).
## Component 2: Audit R Script
**File:** `local/weekly-missing-binaries-audit.R`
**Environment variables consumed:**
- `PLATFORM` — e.g. `ubuntu-2204`, `alpine-321`, `redhat-9`
- `ARCH``amd64` or `arm64`
- `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY` — S3 credentials
- `PGPASS` — PostgreSQL password
- `FORGEJO_TOKEN` — API token for issue updates
The workflow commands (not the R script) also use:
- `REPO_RO_TOKEN` — for `git clone` in the workflow commands
**Logic:**
1. Parse `PLATFORM` to derive OS family (`Ubuntu`, `Alpine`, `Red Hat`) and S3 codename (e.g. `ubuntu-2204` -> `jammy`, `redhat-9` -> `rhel9`).
2. Fetch CRAN release packages via `tools::CRAN_package_db()` — extract `Package` and `Version`.
3. List S3 tarballs at `devxy-rpkgs-binaries/{arch}/{codename}/latest/src/contrib/` and parse `{name}_{version}.tar.gz`.
4. Find packages where the CRAN release version is missing from S3.
5. Read `local/excluded-packages.json` and remove those packages from the missing list.
6. Query the `single_builds` DB table: for each missing package+version+platform+arch, check if `error_occurred = TRUE`. Split into:
- **Rebuildable:** missing, not excluded, no prior failure for this version
- **Known failures:** missing, not excluded, but has a recorded failure for this version
7. Write the rebuildable package list (names only) to `/mnt/cache/packages/weekly_rebuild_{platform}_{arch}.rds`.
8. Update the Forgejo issue for this OS family.
**Issue update logic:**
- Issue title: `Missing package binaries for latest version (<OS family>)` where OS family is `Ubuntu`, `Alpine`, or `Red Hat`.
- Search for existing open issue via `GET /api/v1/repos/devxy/build-cran-binaries/issues?type=issues&state=open&q=<url-encoded title>`. Match by exact title.
- If found, read the existing body, replace the section for this platform/arch, and `PATCH` the issue.
- If not found, `POST` a new issue with just this platform/arch section.
**Issue body format:**
```markdown
_Last updated: 2026-04-11_
## ubuntu-2204
### amd64 (12 missing, 8 to rebuild)
- ggplot2 (3.5.2)
- dplyr (1.1.5)
- ...
#### Known build failures
- somepkg (1.0.0)
### arm64 (5 missing, 5 to rebuild)
- ...
## ubuntu-2404
### amd64 (3 missing, 3 to rebuild)
- ...
### arm64 (0 missing)
All binaries available.
---
## Excluded packages
doBy (hang), frailtypack (hang), RInno (windows-only), ...
```
Each audit workflow run updates only its own platform/arch section within the issue.
The "Excluded packages" section and "Last updated" timestamp are rewritten on every run.
**Section replacement strategy:**
The script parses the existing issue body as markdown, finds the `## {platform}` + `### {arch}` section, replaces it, and writes back the full body.
If the section doesn't exist yet, it's appended under the correct `## {platform}` header (or a new one is created).
## Component 3: Audit Workflows
**Files:** 14 files, named `weekly-audit-missing-{platform}-{arch}.yaml`
Example: `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`
**Trigger:**
```yaml
when:
- event: cron
cron: weekly-audit-missing-ubuntu-2204-amd64
- event: manual
evaluate: 'task == "weekly-audit-missing-ubuntu-2204-amd64"'
```
**Step:** Lightweight — clones repo, installs bincraft + dependencies, runs the audit R script.
**Container:** Uses the platform-appropriate build image (needed for correct platform identification), but with minimal resource requests since no building happens.
**Resources:** ~2Gi memory, 2 CPUs.
## Component 4: Rebuild Workflows
**Files:** 14 files, named `weekly-rebuild-missing-{platform}-{arch}.yaml`
Example: `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`
**Trigger:**
```yaml
when:
- event: cron
cron: weekly-rebuild-missing-ubuntu-2204-amd64
- event: manual
evaluate: 'task == "weekly-rebuild-missing-ubuntu-2204-amd64"'
```
**Step:**
1. Clone repo, install bincraft.
2. Read `local/excluded-packages.json` as a safety net.
3. Read `/mnt/cache/packages/weekly_rebuild_{platform}_{arch}.rds`. If missing or empty, exit 0.
4. Filter out excluded packages (double-check).
5. Iterate and call `bincraft::build_binary_package()` for each package with `tag_limit = 1L`.
Same S3/DB parameters as existing build workflows.
6. Uses `xvfb-run` / `xwfb-run` for graphical packages (same pattern as existing builds).
**Resources:** Same as `process-updates` workflows — 5Gi request, 18Gi limit, 3 CPUs.
**Cache volume:** Maps `${ARCH}-binaries-r-dep-cache-${PLATFORM}:/mnt/cache` (same volumes as existing builds, so the RDS files written by audit are visible).
## Secrets Required
All existing secrets are reused:
- `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY` — S3 access
- `PGPASS` — PostgreSQL
- `REPO_RO_TOKEN` — Git clone
- `GITHUB_PAT` — For bincraft GitHub mirror access
New secret needed:
- `FORGEJO_TOKEN` — API token for creating/updating issues on git.devxy.io
## Cron Schedule
The audit and rebuild workflows each get their own cron names.
The cron schedule itself is configured in the Crow/Woodpecker server, not in the YAML.
Intended cadence: once per week (e.g. Sunday morning).
## File Inventory
| File | Type | Description |
|------|------|-------------|
| `local/excluded-packages.json` | Config | Excluded packages with reasons |
| `local/weekly-missing-binaries-audit.R` | R script | Audit logic, parameterized by env vars |
| `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml` | Workflow | Audit for ubuntu-2204/amd64 |
| `.crow/weekly-audit-missing-ubuntu-2204-arm64.yaml` | Workflow | Audit for ubuntu-2204/arm64 |
| `.crow/weekly-audit-missing-ubuntu-2404-amd64.yaml` | Workflow | Audit for ubuntu-2404/amd64 |
| `.crow/weekly-audit-missing-ubuntu-2404-arm64.yaml` | Workflow | Audit for ubuntu-2404/arm64 |
| `.crow/weekly-audit-missing-alpine-321-amd64.yaml` | Workflow | Audit for alpine-321/amd64 |
| `.crow/weekly-audit-missing-alpine-321-arm64.yaml` | Workflow | Audit for alpine-321/arm64 |
| `.crow/weekly-audit-missing-alpine-322-amd64.yaml` | Workflow | Audit for alpine-322/amd64 |
| `.crow/weekly-audit-missing-alpine-322-arm64.yaml` | Workflow | Audit for alpine-322/arm64 |
| `.crow/weekly-audit-missing-alpine-323-amd64.yaml` | Workflow | Audit for alpine-323/amd64 |
| `.crow/weekly-audit-missing-alpine-323-arm64.yaml` | Workflow | Audit for alpine-323/arm64 |
| `.crow/weekly-audit-missing-redhat-8-amd64.yaml` | Workflow | Audit for redhat-8/amd64 |
| `.crow/weekly-audit-missing-redhat-8-arm64.yaml` | Workflow | Audit for redhat-8/arm64 |
| `.crow/weekly-audit-missing-redhat-9-amd64.yaml` | Workflow | Audit for redhat-9/amd64 |
| `.crow/weekly-audit-missing-redhat-9-arm64.yaml` | Workflow | Audit for redhat-9/arm64 |
| `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml` | Workflow | Rebuild for ubuntu-2204/amd64 |
| `.crow/weekly-rebuild-missing-ubuntu-2204-arm64.yaml` | Workflow | Rebuild for ubuntu-2204/arm64 |
| `.crow/weekly-rebuild-missing-ubuntu-2404-amd64.yaml` | Workflow | Rebuild for ubuntu-2404/amd64 |
| `.crow/weekly-rebuild-missing-ubuntu-2404-arm64.yaml` | Workflow | Rebuild for ubuntu-2404/arm64 |
| `.crow/weekly-rebuild-missing-alpine-321-amd64.yaml` | Workflow | Rebuild for alpine-321/amd64 |
| `.crow/weekly-rebuild-missing-alpine-321-arm64.yaml` | Workflow | Rebuild for alpine-321/arm64 |
| `.crow/weekly-rebuild-missing-alpine-322-amd64.yaml` | Workflow | Rebuild for alpine-322/amd64 |
| `.crow/weekly-rebuild-missing-alpine-322-arm64.yaml` | Workflow | Rebuild for alpine-322/arm64 |
| `.crow/weekly-rebuild-missing-alpine-323-amd64.yaml` | Workflow | Rebuild for alpine-323/amd64 |
| `.crow/weekly-rebuild-missing-alpine-323-arm64.yaml` | Workflow | Rebuild for alpine-323/arm64 |
| `.crow/weekly-rebuild-missing-redhat-8-amd64.yaml` | Workflow | Rebuild for redhat-8/amd64 |
| `.crow/weekly-rebuild-missing-redhat-8-arm64.yaml` | Workflow | Rebuild for redhat-8/arm64 |
| `.crow/weekly-rebuild-missing-redhat-9-amd64.yaml` | Workflow | Rebuild for redhat-9/amd64 |
| `.crow/weekly-rebuild-missing-redhat-9-arm64.yaml` | Workflow | Rebuild for redhat-9/arm64 |
**Total: 29 new files** (1 JSON config + 1 R script + 14 audit workflows + 14 rebuild workflows + the design doc itself)