refactor(ci): use multi-R-version images across workflows (#70)
Some checks failed
ci/crow/cron/update-package-index-alpine-322-arm64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-323-amd64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-323-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-10-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-10-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-8-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-8-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-9-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-9-arm64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2204-amd64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2204-arm64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2404-amd64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2404-arm64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-322-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline failed

## Summary

- Switches all 64 `.crow/*.yaml` workflows, the `Justfile`, and the commented-out `build-all-versions-install-deps.yaml` template to the new multi-R-version container images.
- Each image tag drops `-${R_VERSION}` (e.g. `build-env-alpine:3.22-4.5` → `build-env-alpine:3.22`); workflows pick an R version by calling `/opt/R/${R_VERSION}/bin/R` explicitly.
- Pattern-1 workflows (hard-coded image tag) gain a single `R_VERSION:` env var per file.
- Pattern-2 workflows (`build-all-versions-*`, matrix-driven) continue to receive `R_VERSION` via `crow pipeline create --var`.

## Platform → R_VERSION mapping

| Platform | Image | R_VERSION |
| --- | --- | --- |
| alpine 3.22 / 3.23 | `build-env-alpine:3.{22,23}` | 4.5.3 |
| ubuntu jammy / noble | `build-env-ubuntu:{jammy,noble}` | 4.4.3 |
| redhat 8 / 9 | `build-env-redhat:{8,9}` | 4.4.3 |
| redhat 10 | `build-env-redhat:10` | 4.5.3 |

The `alpine-321` audit-only workflows fall back to `build-env-alpine:3.23` since no 3.21 image exists in the new scheme; the audits only query S3/CRAN, so the container OS does not affect correctness.

## Bug fixes folded in

1. All six `weekly-audit-missing-alpine-*.yaml` files were pinned to `alpine:3.23-4.5` regardless of platform.
   Now each one points at its own alpine image.
2. All fourteen `update-package-index-*.yaml` files were pinned to `build-env-ubuntu:noble-4.4` (or `noble-4.5` for `redhat-10-amd64`) regardless of platform.
   Now each one uses its own platform's image.
3. `process-updates-ubuntu-2404-{amd64,arm64}.yaml` had `R_VERSION` of `4.4` (no patch) baked into the tag.
   Aligned to `4.4.3`, matching the audit and rebuild counterparts.

## Smoke targets (post-merge)

One workflow per shape, ready to dry-run:

- `process-updates-alpine-322-amd64.yaml`
- `weekly-rebuild-missing-redhat-9-amd64.yaml`
- `weekly-audit-missing-ubuntu-2204-amd64.yaml`
- `update-package-index-redhat-10-amd64.yaml`
- `archive-missed-packages.yaml`
- `build-all-versions-amd64.yaml` (with its install-deps predecessor)
- `build-r-minor-sensitive-packages.yaml`

## Follow-ups (out of scope, surfaced during review)

- `process-updates-redhat-10-{amd64,arm64}.yaml` set `codename = "rhel9"` in their `upload_package_index(...)` calls.
  Pre-existing bug — should be `"rhel10"`. The `update-package-index-redhat-10-*` counterparts already use `"rhel10"` correctly.
- `update-package-index-redhat-8-amd64.yaml` is missing the `packageVersion("bincraft")` diagnostic line that every sibling file has.
  Pre-existing asymmetry between amd64 and arm64.

## Spec & plan

- Spec: `docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md`
- Plan: `docs/superpowers/plans/2026-05-25-multi-r-version-images.md`

Reviewed-on: #70
This commit is contained in:
Patrick Schratz 2026-05-26 08:51:27 +00:00 committed by Patrick Schratz
commit 6faa3772d1
2 changed files with 382 additions and 1473 deletions

View file

@ -1,905 +0,0 @@
# Weekly Missing Binaries Audit & Rebuild — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add a weekly CI workflow that audits CRAN packages missing binaries for their latest release version, reports them in Forgejo issues by OS family, and rebuilds those without a prior build failure.
**Architecture:** Two independent workflow sets (audit + rebuild), each with one YAML per platform/arch (14 each). A shared R script does the audit logic. An excluded-packages JSON config is the shared blocklist. The audit writes per-platform RDS files consumed by rebuild workflows.
**Tech Stack:** R (bincraft, s3fs, DBI/RPostgres, jsonlite, httr2), Crow/Woodpecker CI YAML, Forgejo API, PostgreSQL, Backblaze S3.
---
## File Structure
| File | Responsibility |
|------|---------------|
| `local/excluded-packages.json` | Excluded packages with reasons (single source of truth) |
| `local/weekly-missing-binaries-audit.R` | Audit logic: S3 check, DB check, issue update, RDS output |
| `.crow/weekly-audit-missing-{platform}-{arch}.yaml` (14 files) | Audit workflow per platform/arch |
| `.crow/weekly-rebuild-missing-{platform}-{arch}.yaml` (14 files) | Rebuild workflow per platform/arch |
## Platform Reference
Used throughout the plan. Each row is one workflow file pair (audit + rebuild).
| PLATFORM env var | ARCH | S3 codename (sub("-","",PLATFORM)) | OS Family | Image |
|-----------------|------|-----------|-----------|-------|
| ubuntu-2204 | amd64 | ubuntu2204 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 |
| ubuntu-2204 | arm64 | ubuntu2204 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 |
| ubuntu-2404 | amd64 | ubuntu2404 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 |
| ubuntu-2404 | arm64 | ubuntu2404 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 |
| alpine-321 | amd64 | alpine321 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 |
| alpine-321 | arm64 | alpine321 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 |
| alpine-322 | amd64 | alpine322 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 |
| alpine-322 | arm64 | alpine322 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 |
| alpine-323 | amd64 | alpine323 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 |
| alpine-323 | arm64 | alpine323 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 |
| redhat-8 | amd64 | redhat8 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 |
| redhat-8 | arm64 | redhat8 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 |
| redhat-9 | amd64 | redhat9 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 |
| redhat-9 | arm64 | redhat9 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 |
---
## Task 1: Create excluded-packages.json
**Files:**
- Create: `local/excluded-packages.json`
- [ ] **Step 1: Create the JSON file**
Extract every package from the inline exclusion list in `.crow/build-all-versions-amd64.yaml` (line 72). The list has two categories: Windows-only packages (comment on line 69) and problematic packages (hang/OOM). Create `local/excluded-packages.json`:
```json
[
{"package": "RInno", "reason": "windows-only"},
{"package": "KeyboardSimulator", "reason": "windows-only"},
{"package": "R2PPT", "reason": "windows-only"},
{"package": "RWinEdt", "reason": "windows-only"},
{"package": "blatr", "reason": "windows-only"},
{"package": "excel.link", "reason": "windows-only"},
{"package": "spectrino", "reason": "windows-only"},
{"package": "taskscheduleR", "reason": "windows-only"},
{"package": "MDSGUI", "reason": "windows-only"},
{"package": "BiplotGUI", "reason": "windows-only"},
{"package": "R2wd", "reason": "windows-only"},
{"package": "rFUSION", "reason": "windows-only"},
{"package": "MediaNews", "reason": "windows-only"},
{"package": "doBy", "reason": "hang"},
{"package": "IDPmisc", "reason": "hang"},
{"package": "frailtypack", "reason": "hang"},
{"package": "afex", "reason": "hang"},
{"package": "FrF2", "reason": "hang"},
{"package": "DoE.base", "reason": "hang"},
{"package": "agricolae", "reason": "hang"},
{"package": "doFuture", "reason": "hang"},
{"package": "fscaret", "reason": "hang"},
{"package": "PHYLOGR", "reason": "hang"},
{"package": "seewave", "reason": "hang"},
{"package": "pls", "reason": "hang"},
{"package": "relaimpo", "reason": "hang"},
{"package": "geepack", "reason": "hang"},
{"package": "gggenes", "reason": "hang"},
{"package": "NPCirc", "reason": "hang"},
{"package": "repmis", "reason": "hang"},
{"package": "PNDSIBGE", "reason": "hang"},
{"package": "lidR", "reason": "hang"},
{"package": "poismf", "reason": "hang"},
{"package": "neonstore", "reason": "hang"},
{"package": "MachineShop", "reason": "hang"},
{"package": "mvst", "reason": "hang"},
{"package": "MacBehaviour", "reason": "hang"},
{"package": "mcmcderive", "reason": "hang"},
{"package": "RGIFT", "reason": "hang"},
{"package": "KnowBR", "reason": "hang"},
{"package": "netmeta", "reason": "hang"},
{"package": "spdep", "reason": "hang"},
{"package": "Rfast", "reason": "hang"},
{"package": "compareGroups", "reason": "hang"},
{"package": "ff", "reason": "hang"},
{"package": "GsymPoint", "reason": "hang"},
{"package": "RcppDynProg", "reason": "hang"},
{"package": "comtradr", "reason": "hang"},
{"package": "FD", "reason": "hang"},
{"package": "PearsonDS", "reason": "hang"},
{"package": "DCluster", "reason": "hang"},
{"package": "gRc", "reason": "hang"},
{"package": "mixlm", "reason": "hang"},
{"package": "geospt", "reason": "hang"},
{"package": "fdth", "reason": "hang"},
{"package": "ffmanova", "reason": "hang"},
{"package": "fiery", "reason": "hang"},
{"package": "ffscrapr", "reason": "hang"},
{"package": "cold", "reason": "hang"},
{"package": "RcmdrPlugin.DoE", "reason": "hang"},
{"package": "RcmdrPlugin.NMBU", "reason": "hang"},
{"package": "RcmdrPlugin.RiskDemo", "reason": "hang"},
{"package": "RcmdrPlugin.TeachStat", "reason": "hang"},
{"package": "RcmdrPlugin.TeachingDemos", "reason": "hang"},
{"package": "RcmdrPlugin.UCA", "reason": "hang"},
{"package": "RcmdrPlugin.WorldFlora", "reason": "hang"},
{"package": "RcmdrPlugin.aRnova", "reason": "hang"},
{"package": "RcmdrPlugin.depthTools", "reason": "hang"},
{"package": "RcmdrPlugin.orloca", "reason": "hang"},
{"package": "RcmdrPlugin.sos", "reason": "hang"},
{"package": "RcmdrPlugin.survival", "reason": "hang"},
{"package": "RcmdrPlugin.temis", "reason": "hang"},
{"package": "GWlasso", "reason": "hang"},
{"package": "GWmodelVis", "reason": "hang"}
]
```
- [ ] **Step 2: Commit**
```bash
git add local/excluded-packages.json
git commit -m "feat: add excluded-packages.json for weekly missing binaries workflow"
```
---
## Task 2: Create the audit R script
**Files:**
- Create: `local/weekly-missing-binaries-audit.R`
This is the core logic. It reads env vars `PLATFORM` and `ARCH`, checks S3 + DB, writes an RDS of rebuildable packages, and updates the Forgejo issue for the OS family.
- [ ] **Step 1: Create the R script**
Create `local/weekly-missing-binaries-audit.R` with the full content below:
```r
options(error = function() {
cat("ERROR:", geterrmessage(), "\n", file = stdout())
traceback(2)
q(status = 1)
})
library(DBI, quietly = TRUE)
suppressPackageStartupMessages(library(data.table))
platform <- Sys.getenv("PLATFORM")
arch <- Sys.getenv("ARCH")
stopifnot(nzchar(platform), nzchar(arch))
# S3 codename: remove hyphen from platform string (e.g. "ubuntu-2204" -> "ubuntu2204")
s3_codename <- gsub("-", "", platform)
# OS family for issue grouping
os_family <- if (grepl("^ubuntu", platform)) {
"Ubuntu"
} else if (grepl("^alpine", platform)) {
"Alpine"
} else if (grepl("^redhat", platform)) {
"Red Hat"
} else {
stop("Unknown platform: ", platform)
}
cat(sprintf("Audit: platform=%s, arch=%s, s3_codename=%s, os_family=%s\n",
platform, arch, s3_codename, os_family))
### 1. Get current CRAN release packages
cran_release <- as.data.table(tools::CRAN_package_db()[, c("Package", "Version")])
cran_release[, Version := as.character(Version)]
cat(sprintf("CRAN release packages: %d\n", nrow(cran_release)))
### 2. List S3 tarballs for this platform/arch
s3fs::s3_file_system(
aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"),
aws_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"),
endpoint = "https://s3.eu-central-003.backblazeb2.com",
region_name = "eu-central-003",
refresh = TRUE
)
s3_path <- sprintf("devxy-rpkgs-binaries/%s/%s/latest/src/contrib", arch, s3_codename)
s3_files <- tryCatch(
s3fs::s3_dir_ls(s3_path),
error = function(e) {
cat(sprintf("Warning: could not list S3 path %s: %s\n", s3_path, conditionMessage(e)))
character(0)
}
)
# Parse package name + version from tarball filenames
file_names <- basename(s3_files)
matches <- regexec("^([A-Za-z0-9.]+)_([0-9][^/]*)\\.tar\\.gz$", file_names)
parts <- regmatches(file_names, matches)
parts <- parts[lengths(parts) == 3]
s3_dt <- if (length(parts) > 0) {
data.table(
Package = vapply(parts, `[`, character(1), 2),
Version = vapply(parts, `[`, character(1), 3)
)
} else {
data.table(Package = character(0), Version = character(0))
}
cat(sprintf("S3 packages for %s/%s: %d\n", arch, s3_codename, nrow(s3_dt)))
### 3. Find CRAN release packages missing from S3
setkey(cran_release, Package, Version)
setkey(s3_dt, Package, Version)
missing <- cran_release[!s3_dt]
cat(sprintf("Missing binaries (latest CRAN version): %d\n", nrow(missing)))
### 4. Load excluded packages
excluded_json <- jsonlite::fromJSON("local/excluded-packages.json")
excluded_pkgs <- excluded_json$package
missing_not_excluded <- missing[!Package %in% excluded_pkgs]
missing_excluded <- missing[Package %in% excluded_pkgs]
cat(sprintf("Missing after excluding %d blocked packages: %d\n",
length(excluded_pkgs), nrow(missing_not_excluded)))
### 5. Check DB for prior build failures on these exact versions
con <- DBI::dbConnect(
RPostgres::Postgres(),
dbname = "build_metadata",
host = "r-binaries.devxy.io",
port = 15432,
user = "rpkgs",
password = Sys.getenv("PGPASS"),
sslmode = "require"
)
on.exit(DBI::dbDisconnect(con), add = TRUE)
errored_pkgs <- as.data.table(DBI::dbGetQuery(
con,
sprintf(
"SELECT name, tag FROM single_builds WHERE error_occurred = TRUE AND platform = '%s' AND arch = '%s'",
platform, arch
)
))
setnames(errored_pkgs, c("Package", "Version"))
setkey(errored_pkgs, Package, Version)
# Split: rebuildable vs known failures
known_failures <- missing_not_excluded[errored_pkgs, nomatch = 0, on = c("Package", "Version")]
rebuildable <- missing_not_excluded[!errored_pkgs, on = c("Package", "Version")]
cat(sprintf("Known build failures: %d\n", nrow(known_failures)))
cat(sprintf("Rebuildable (no prior failure): %d\n", nrow(rebuildable)))
### 6. Write RDS for rebuild workflow
cache_dir <- "/mnt/cache/packages"
if (dir.exists(cache_dir)) {
rds_path <- file.path(cache_dir, sprintf("weekly_rebuild_%s_%s.rds", platform, arch))
saveRDS(rebuildable$Package, rds_path)
cat(sprintf("Wrote %d packages to %s\n", nrow(rebuildable), rds_path))
} else {
cat(sprintf("Cache dir %s does not exist, skipping RDS write\n", cache_dir))
}
### 7. Update Forgejo issue
forgejo_token <- Sys.getenv("FORGEJO_TOKEN")
if (!nzchar(forgejo_token)) {
cat("FORGEJO_TOKEN not set, skipping issue update\n")
} else {
base_url <- "https://git.devxy.io/api/v1"
repo <- "devxy/build-cran-binaries"
issue_title <- sprintf("Missing package binaries for latest version (%s)", os_family)
# Helper: make API request
forgejo_get <- function(path, query = list()) {
url <- paste0(base_url, path)
resp <- httr2::request(url) |>
httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
httr2::req_url_query(!!!query) |>
httr2::req_perform()
httr2::resp_body_json(resp)
}
forgejo_patch <- function(path, body) {
url <- paste0(base_url, path)
httr2::request(url) |>
httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
httr2::req_method("PATCH") |>
httr2::req_body_json(body) |>
httr2::req_perform()
}
forgejo_post <- function(path, body) {
url <- paste0(base_url, path)
httr2::request(url) |>
httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
httr2::req_body_json(body) |>
httr2::req_perform()
}
# Build the markdown section for this platform/arch
build_section <- function() {
n_missing <- nrow(rebuildable) + nrow(known_failures)
n_rebuild <- nrow(rebuildable)
header <- sprintf("### %s (%d missing, %d to rebuild)", arch, n_missing, n_rebuild)
lines <- header
if (nrow(rebuildable) > 0) {
pkg_lines <- sprintf("- %s (%s)", rebuildable$Package, rebuildable$Version)
lines <- c(lines, "", pkg_lines)
} else if (nrow(known_failures) == 0) {
lines <- c(lines, "", "All binaries available.")
}
if (nrow(known_failures) > 0) {
lines <- c(lines, "", "#### Known build failures",
sprintf("- %s (%s)", known_failures$Package, known_failures$Version))
}
paste(lines, collapse = "\n")
}
# Build excluded packages footer
build_excluded_footer <- function() {
if (nrow(excluded_json) == 0) return("")
items <- sprintf("%s (%s)", excluded_json$package, excluded_json$reason)
paste0("\n---\n\n## Excluded packages\n", paste(items, collapse = ", "))
}
new_section <- build_section()
# Search for existing issue
issues <- forgejo_get(
sprintf("/repos/%s/issues", repo),
query = list(type = "issues", state = "open", q = issue_title, limit = 50)
)
# Find exact title match
existing <- Filter(function(i) i$title == issue_title, issues)
if (length(existing) > 0) {
issue <- existing[[1]]
body <- issue$body
# Replace or insert the platform section + arch subsection
platform_header <- sprintf("## %s", platform)
arch_header <- sprintf("### %s", arch)
# Split body into lines for manipulation
body_lines <- strsplit(body, "\n")[[1]]
# Find the platform section
platform_start <- which(body_lines == platform_header)
if (length(platform_start) > 0) {
platform_start <- platform_start[1]
# Find end of this platform section (next ## or --- or end)
remaining <- body_lines[(platform_start + 1):length(body_lines)]
platform_end_offset <- which(grepl("^## |^---$", remaining))
platform_end <- if (length(platform_end_offset) > 0) {
platform_start + platform_end_offset[1] - 1
} else {
length(body_lines)
}
# Within platform section, find the arch subsection
section_lines <- body_lines[platform_start:platform_end]
arch_start_offset <- which(grepl(sprintf("^### %s", arch), section_lines))
if (length(arch_start_offset) > 0) {
arch_start <- arch_start_offset[1]
# Find end of arch subsection (next ### or ## or --- or end of platform section)
arch_remaining <- section_lines[(arch_start + 1):length(section_lines)]
arch_end_offset <- which(grepl("^###|^## |^---$", arch_remaining))
arch_end <- if (length(arch_end_offset) > 0) {
arch_start + arch_end_offset[1] - 1
} else {
length(section_lines)
}
# Replace arch subsection within platform section
section_lines <- c(
section_lines[1:(arch_start - 1)],
strsplit(new_section, "\n")[[1]],
if (arch_end < length(section_lines)) section_lines[(arch_end + 1):length(section_lines)] else character(0)
)
} else {
# Append arch subsection to end of platform section
section_lines <- c(section_lines, "", strsplit(new_section, "\n")[[1]])
}
body_lines <- c(
body_lines[1:(platform_start - 1)],
section_lines,
if (platform_end < length(body_lines)) body_lines[(platform_end + 1):length(body_lines)] else character(0)
)
} else {
# Insert new platform section before "---" (excluded packages footer) or at end
footer_line <- which(body_lines == "---")
insert_at <- if (length(footer_line) > 0) footer_line[1] - 1 else length(body_lines)
body_lines <- c(
body_lines[1:insert_at],
"",
platform_header,
"",
strsplit(new_section, "\n")[[1]],
if (insert_at < length(body_lines)) body_lines[(insert_at + 1):length(body_lines)] else character(0)
)
}
# Update timestamp
timestamp_pattern <- "^_Last updated:.*_$"
ts_line <- which(grepl(timestamp_pattern, body_lines))
new_ts <- sprintf("_Last updated: %s_", Sys.Date())
if (length(ts_line) > 0) {
body_lines[ts_line[1]] <- new_ts
} else {
body_lines <- c(new_ts, "", body_lines)
}
# Rebuild excluded footer
footer_start <- which(body_lines == "---")
if (length(footer_start) > 0) {
body_lines <- c(body_lines[1:(footer_start[1] - 1)],
strsplit(build_excluded_footer(), "\n")[[1]])
} else {
body_lines <- c(body_lines, strsplit(build_excluded_footer(), "\n")[[1]])
}
new_body <- paste(body_lines, collapse = "\n")
forgejo_patch(
sprintf("/repos/%s/issues/%d", repo, issue$number),
list(body = new_body)
)
cat(sprintf("Updated issue #%d: %s\n", issue$number, issue_title))
} else {
# Create new issue
body_lines <- c(
sprintf("_Last updated: %s_", Sys.Date()),
"",
sprintf("## %s", platform),
"",
new_section,
build_excluded_footer()
)
new_body <- paste(body_lines, collapse = "\n")
forgejo_post(
sprintf("/repos/%s/issues", repo),
list(title = issue_title, body = new_body)
)
cat(sprintf("Created new issue: %s\n", issue_title))
}
}
cat("Audit complete.\n")
```
- [ ] **Step 2: Commit**
```bash
git add local/weekly-missing-binaries-audit.R
git commit -m "feat: add weekly missing binaries audit R script"
```
---
## Task 3: Create audit workflow for ubuntu-2204-amd64
**Files:**
- Create: `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`
This is the template. All other audit workflows follow the same structure with substituted values.
- [ ] **Step 1: Create the workflow YAML**
Create `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`:
```yaml
when:
- event: cron
cron: weekly-audit-missing-ubuntu-2204-amd64
- event: manual
evaluate: 'task == "weekly-audit-missing-ubuntu-2204-amd64"'
skip_clone: true
steps:
- name: 'Audit missing binaries'
image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
pull: true
environment:
B2_S3_ACCESS_KEY:
from_secret: B2_S3_ACCESS_KEY
B2_S3_SECRET_KEY:
from_secret: B2_S3_SECRET_KEY
PGPASS:
from_secret: PGPASS
REPO_RO_TOKEN:
from_secret: REPO_RO_TOKEN
FORGEJO_TOKEN:
from_secret: FORGEJO_TOKEN
GITHUB_PAT:
from_secret: GITHUB_PAT
PLATFORM: ubuntu-2204
ARCH: amd64
R_LIBS_USER: /mnt/cache/R-pkgs
volumes:
- amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache
commands:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
- R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))'
- R -q -e 'source("local/weekly-missing-binaries-audit.R")'
backend_options:
kubernetes:
resources:
requests:
memory: 2Gi
cpu: 2000m
limits:
memory: 4Gi
cpu: 2000m
nodeSelector:
kubernetes.io/arch: amd64
tolerations:
- key: 'CI'
operator: 'Equal'
value: 'true'
effect: 'NoSchedule'
```
- [ ] **Step 2: Commit**
```bash
git add .crow/weekly-audit-missing-ubuntu-2204-amd64.yaml
git commit -m "feat: add weekly audit workflow for ubuntu-2204-amd64"
```
---
## Task 4: Create remaining 13 audit workflows
**Files:**
- Create: 13 files in `.crow/` (see substitution table below)
Each file follows the exact same structure as Task 3 with these substitutions:
| File suffix | PLATFORM | ARCH | Image | Volume |
|-------------|----------|------|-------|--------|
| ubuntu-2204-arm64 | ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2204 |
| ubuntu-2404-amd64 | ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | amd64-binaries-r-dep-cache-ubuntu2404 |
| ubuntu-2404-arm64 | ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2404 |
| alpine-321-amd64 | alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | amd64-binaries-r-dep-cache-alpine321 |
| alpine-321-arm64 | alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | arm64-binaries-r-dep-cache-alpine321 |
| alpine-322-amd64 | alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | amd64-binaries-r-dep-cache-alpine322 |
| alpine-322-arm64 | alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | arm64-binaries-r-dep-cache-alpine322 |
| alpine-323-amd64 | alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | amd64-binaries-r-dep-cache-alpine323 |
| alpine-323-arm64 | alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | arm64-binaries-r-dep-cache-alpine323 |
| redhat-8-amd64 | redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | amd64-binaries-r-dep-cache-redhat8 |
| redhat-8-arm64 | redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | arm64-binaries-r-dep-cache-redhat8 |
| redhat-9-amd64 | redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | amd64-binaries-r-dep-cache-redhat9 |
| redhat-9-arm64 | redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | arm64-binaries-r-dep-cache-redhat9 |
**Key substitution points in each YAML (6 locations):**
1. `cron:` value — `weekly-audit-missing-{suffix}`
2. `evaluate:` value — `'task == "weekly-audit-missing-{suffix}"'`
3. `image:` — use the Image column
4. `PLATFORM:` env var — use the PLATFORM column
5. `ARCH:` env var — use the ARCH column
6. `volumes:``{ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache`
7. `nodeSelector: kubernetes.io/arch:` — use the ARCH column
- [ ] **Step 1: Create all 13 audit workflow files**
Copy the template from Task 3 and substitute the values per the table above. Example for alpine-321-arm64:
```yaml
when:
- event: cron
cron: weekly-audit-missing-alpine-321-arm64
- event: manual
evaluate: 'task == "weekly-audit-missing-alpine-321-arm64"'
skip_clone: true
steps:
- name: 'Audit missing binaries'
image: reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5
pull: true
environment:
B2_S3_ACCESS_KEY:
from_secret: B2_S3_ACCESS_KEY
B2_S3_SECRET_KEY:
from_secret: B2_S3_SECRET_KEY
PGPASS:
from_secret: PGPASS
REPO_RO_TOKEN:
from_secret: REPO_RO_TOKEN
FORGEJO_TOKEN:
from_secret: FORGEJO_TOKEN
GITHUB_PAT:
from_secret: GITHUB_PAT
PLATFORM: alpine-321
ARCH: arm64
R_LIBS_USER: /mnt/cache/R-pkgs
volumes:
- arm64-binaries-r-dep-cache-alpine321:/mnt/cache
commands:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
- R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))'
- R -q -e 'source("local/weekly-missing-binaries-audit.R")'
backend_options:
kubernetes:
resources:
requests:
memory: 2Gi
cpu: 2000m
limits:
memory: 4Gi
cpu: 2000m
nodeSelector:
kubernetes.io/arch: arm64
tolerations:
- key: 'CI'
operator: 'Equal'
value: 'true'
effect: 'NoSchedule'
```
Repeat for all 13 remaining suffixes from the table, substituting the 7 locations.
- [ ] **Step 2: Commit**
```bash
git add .crow/weekly-audit-missing-*.yaml
git commit -m "feat: add remaining 13 weekly audit workflows for all platform/arch combos"
```
---
## Task 5: Create rebuild workflow for ubuntu-2204-amd64
**Files:**
- Create: `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`
This is the template for rebuild workflows. It reads the RDS written by the audit and builds each package.
- [ ] **Step 1: Create the workflow YAML**
Create `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`:
```yaml
when:
- event: cron
cron: weekly-rebuild-missing-ubuntu-2204-amd64
- event: manual
evaluate: 'task == "weekly-rebuild-missing-ubuntu-2204-amd64"'
skip_clone: true
steps:
- name: 'Rebuild missing binaries'
image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
pull: true
environment:
RED_HAT_DEV_PW:
from_secret: RED_HAT_DEV_PW
B2_S3_ACCESS_KEY:
from_secret: B2_S3_ACCESS_KEY
B2_S3_SECRET_KEY:
from_secret: B2_S3_SECRET_KEY
PGPASS:
from_secret: PGPASS
REPO_RO_TOKEN:
from_secret: REPO_RO_TOKEN
GITHUB_PAT:
from_secret: GITHUB_PAT
GIT_USER: pat-s
R_PKG_CACHE_DIR: /mnt/cache/pkgcache
R_LIBS_USER: /mnt/cache/R-pkgs
CCACHE_DIR: /mnt/cache/ccache
PLATFORM: ubuntu-2204
ARCH: amd64
NCPUS: 2
volumes:
- amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache
commands:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
- R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
- R -q -e 'packageVersion("bincraft")'
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
- $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_ubuntu-2204_amd64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1
backend_options:
kubernetes:
resources:
requests:
memory: 5Gi
cpu: 3000m
limits:
memory: 18Gi
cpu: 3000m
nodeSelector:
kubernetes.io/arch: amd64
node.kubernetes.io/instance-type: AX42
tolerations:
- key: 'CI'
operator: 'Equal'
value: 'true'
effect: 'NoSchedule'
```
- [ ] **Step 2: Commit**
```bash
git add .crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml
git commit -m "feat: add weekly rebuild workflow for ubuntu-2204-amd64"
```
---
## Task 6: Create remaining 13 rebuild workflows
**Files:**
- Create: 13 files in `.crow/` (see substitution table below)
Each file follows the exact same structure as Task 5 with these substitutions:
| File suffix | PLATFORM | ARCH | Image | Volume | nodeSelector instance-type |
|-------------|----------|------|-------|--------|--------------------------|
| ubuntu-2204-arm64 | ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2204 | (omit) |
| ubuntu-2404-amd64 | ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | amd64-binaries-r-dep-cache-ubuntu2404 | AX42 |
| ubuntu-2404-arm64 | ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2404 | (omit) |
| alpine-321-amd64 | alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | amd64-binaries-r-dep-cache-alpine321 | AX42 |
| alpine-321-arm64 | alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | arm64-binaries-r-dep-cache-alpine321 | (omit) |
| alpine-322-amd64 | alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | amd64-binaries-r-dep-cache-alpine322 | AX42 |
| alpine-322-arm64 | alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | arm64-binaries-r-dep-cache-alpine322 | (omit) |
| alpine-323-amd64 | alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | amd64-binaries-r-dep-cache-alpine323 | AX42 |
| alpine-323-arm64 | alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | arm64-binaries-r-dep-cache-alpine323 | (omit) |
| redhat-8-amd64 | redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | amd64-binaries-r-dep-cache-redhat8 | AX42 |
| redhat-8-arm64 | redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | arm64-binaries-r-dep-cache-redhat8 | (omit) |
| redhat-9-amd64 | redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | amd64-binaries-r-dep-cache-redhat9 | AX42 |
| redhat-9-arm64 | redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | arm64-binaries-r-dep-cache-redhat9 | (omit) |
**Key substitution points (8 locations):**
1. `cron:` value — `weekly-rebuild-missing-{suffix}`
2. `evaluate:` value — `'task == "weekly-rebuild-missing-{suffix}"'`
3. `image:` — use the Image column
4. `PLATFORM:` env var — use the PLATFORM column
5. `ARCH:` env var — use the ARCH column
6. `volumes:``{ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache`
7. `rds_path` in the R command — `weekly_rebuild_{PLATFORM}_{ARCH}.rds`
8. `nodeSelector``kubernetes.io/arch: {ARCH}`, plus `node.kubernetes.io/instance-type: AX42` only for amd64 (omit the instance-type line entirely for arm64)
- [ ] **Step 1: Create all 13 rebuild workflow files**
Copy the template from Task 5 and substitute per the table. Example for redhat-9-arm64:
```yaml
when:
- event: cron
cron: weekly-rebuild-missing-redhat-9-arm64
- event: manual
evaluate: 'task == "weekly-rebuild-missing-redhat-9-arm64"'
skip_clone: true
steps:
- name: 'Rebuild missing binaries'
image: reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3
pull: true
environment:
RED_HAT_DEV_PW:
from_secret: RED_HAT_DEV_PW
B2_S3_ACCESS_KEY:
from_secret: B2_S3_ACCESS_KEY
B2_S3_SECRET_KEY:
from_secret: B2_S3_SECRET_KEY
PGPASS:
from_secret: PGPASS
REPO_RO_TOKEN:
from_secret: REPO_RO_TOKEN
GITHUB_PAT:
from_secret: GITHUB_PAT
GIT_USER: pat-s
R_PKG_CACHE_DIR: /mnt/cache/pkgcache
R_LIBS_USER: /mnt/cache/R-pkgs
CCACHE_DIR: /mnt/cache/ccache
PLATFORM: redhat-9
ARCH: arm64
NCPUS: 2
volumes:
- arm64-binaries-r-dep-cache-redhat9:/mnt/cache
commands:
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
- mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
- R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
- R -q -e 'packageVersion("bincraft")'
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
- $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_redhat-9_arm64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1
backend_options:
kubernetes:
resources:
requests:
memory: 5Gi
cpu: 3000m
limits:
memory: 18Gi
cpu: 3000m
nodeSelector:
kubernetes.io/arch: arm64
tolerations:
- key: 'CI'
operator: 'Equal'
value: 'true'
effect: 'NoSchedule'
```
Note: arm64 workflows do NOT include `node.kubernetes.io/instance-type: AX42` in nodeSelector.
Repeat for all 13 remaining suffixes from the table, substituting the 8 locations.
- [ ] **Step 2: Commit**
```bash
git add .crow/weekly-rebuild-missing-*.yaml
git commit -m "feat: add remaining 13 weekly rebuild workflows for all platform/arch combos"
```
---
## Task 7: Final verification
- [ ] **Step 1: Verify all files exist**
Run:
```bash
ls -1 local/excluded-packages.json local/weekly-missing-binaries-audit.R
ls -1 .crow/weekly-audit-missing-*.yaml | wc -l
ls -1 .crow/weekly-rebuild-missing-*.yaml | wc -l
```
Expected: both files exist, 14 audit workflows, 14 rebuild workflows.
- [ ] **Step 2: Validate JSON**
Run:
```bash
python3 -c "import json; json.load(open('local/excluded-packages.json')); print('JSON valid')"
```
Expected: `JSON valid`
- [ ] **Step 3: Validate YAML syntax**
Run:
```bash
python3 -c "
import yaml, glob
for f in sorted(glob.glob('.crow/weekly-*-missing-*.yaml') + glob.glob('.crow/weekly-audit-missing-*.yaml') + glob.glob('.crow/weekly-rebuild-missing-*.yaml')):
yaml.safe_load(open(f))
print(f'OK: {f}')
"
```
Expected: all files print `OK`.
- [ ] **Step 4: Verify unique cron names**
Run:
```bash
grep -h 'cron: weekly-' .crow/weekly-*.yaml | sort | uniq -c | sort -rn | head
```
Expected: all counts are 1 (no duplicates).
- [ ] **Step 5: Verify all 14 platform/arch combos covered**
Run:
```bash
for suffix in ubuntu-2204-amd64 ubuntu-2204-arm64 ubuntu-2404-amd64 ubuntu-2404-arm64 alpine-321-amd64 alpine-321-arm64 alpine-322-amd64 alpine-322-arm64 alpine-323-amd64 alpine-323-arm64 redhat-8-amd64 redhat-8-arm64 redhat-9-amd64 redhat-9-arm64; do
test -f ".crow/weekly-audit-missing-${suffix}.yaml" || echo "MISSING audit: ${suffix}"
test -f ".crow/weekly-rebuild-missing-${suffix}.yaml" || echo "MISSING rebuild: ${suffix}"
done
echo "All checks passed if no MISSING lines above"
```
Expected: no MISSING lines, just "All checks passed".

View file

@ -1,246 +0,0 @@
# Weekly Missing Binaries Audit & Rebuild
## Goal
A weekly CI workflow that identifies CRAN packages whose latest release version has no binary available, reports them in Forgejo issues grouped by OS family and arch, and rebuilds those that have no prior build failure recorded in the database.
## Architecture Overview
Two independent workflow sets, each with one file per platform/arch combo (14 files each), plus a shared R script and an excluded-packages config file.
```
weekly-audit-missing-*-*.yaml (14 files)
local/weekly-missing-binaries-audit.R
├── Updates Forgejo issues (3 issues, one per OS family)
└── Writes RDS files to /mnt/cache/packages/
weekly-rebuild-missing-*-*.yaml (14 files)
Reads RDS, builds missing packages via bincraft::build_binary_package()
```
The audit and build workflows are fully independent.
Either can be triggered on its own via cron or manually.
## Platforms
All current platform/arch combinations:
| Platform | Arch | Image | Codename |
|-------------|-------|-------------------------------------------------|------------|
| ubuntu-2204 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | jammy |
| ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | jammy |
| ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | noble |
| ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | noble |
| alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | alpine321 |
| alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | alpine321 |
| alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | alpine322 |
| alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | alpine322 |
| alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | alpine323 |
| alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | alpine323 |
| redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | rhel8 |
| redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | rhel8 |
| redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | rhel9 |
| redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | rhel9 |
## Component 1: Excluded Packages Config
**File:** `local/excluded-packages.json`
A JSON array of objects with `package` and `reason` fields:
```json
[
{"package": "RInno", "reason": "windows-only"},
{"package": "KeyboardSimulator", "reason": "windows-only"},
{"package": "doBy", "reason": "hang"},
{"package": "frailtypack", "reason": "hang"},
...
]
```
This file is the single source of truth for packages that should be skipped.
Both the audit script and rebuild workflows read from it.
The existing `build-all-versions-*.yaml` workflows retain their inline lists for now (migration is out of scope).
## Component 2: Audit R Script
**File:** `local/weekly-missing-binaries-audit.R`
**Environment variables consumed:**
- `PLATFORM` — e.g. `ubuntu-2204`, `alpine-321`, `redhat-9`
- `ARCH``amd64` or `arm64`
- `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY` — S3 credentials
- `PGPASS` — PostgreSQL password
- `FORGEJO_TOKEN` — API token for issue updates
The workflow commands (not the R script) also use:
- `REPO_RO_TOKEN` — for `git clone` in the workflow commands
**Logic:**
1. Parse `PLATFORM` to derive OS family (`Ubuntu`, `Alpine`, `Red Hat`) and S3 codename (e.g. `ubuntu-2204` -> `jammy`, `redhat-9` -> `rhel9`).
2. Fetch CRAN release packages via `tools::CRAN_package_db()` — extract `Package` and `Version`.
3. List S3 tarballs at `devxy-rpkgs-binaries/{arch}/{codename}/latest/src/contrib/` and parse `{name}_{version}.tar.gz`.
4. Find packages where the CRAN release version is missing from S3.
5. Read `local/excluded-packages.json` and remove those packages from the missing list.
6. Query the `single_builds` DB table: for each missing package+version+platform+arch, check if `error_occurred = TRUE`. Split into:
- **Rebuildable:** missing, not excluded, no prior failure for this version
- **Known failures:** missing, not excluded, but has a recorded failure for this version
7. Write the rebuildable package list (names only) to `/mnt/cache/packages/weekly_rebuild_{platform}_{arch}.rds`.
8. Update the Forgejo issue for this OS family.
**Issue update logic:**
- Issue title: `Missing package binaries for latest version (<OS family>)` where OS family is `Ubuntu`, `Alpine`, or `Red Hat`.
- Search for existing open issue via `GET /api/v1/repos/devxy/build-cran-binaries/issues?type=issues&state=open&q=<url-encoded title>`. Match by exact title.
- If found, read the existing body, replace the section for this platform/arch, and `PATCH` the issue.
- If not found, `POST` a new issue with just this platform/arch section.
**Issue body format:**
```markdown
_Last updated: 2026-04-11_
## ubuntu-2204
### amd64 (12 missing, 8 to rebuild)
- ggplot2 (3.5.2)
- dplyr (1.1.5)
- ...
#### Known build failures
- somepkg (1.0.0)
### arm64 (5 missing, 5 to rebuild)
- ...
## ubuntu-2404
### amd64 (3 missing, 3 to rebuild)
- ...
### arm64 (0 missing)
All binaries available.
---
## Excluded packages
doBy (hang), frailtypack (hang), RInno (windows-only), ...
```
Each audit workflow run updates only its own platform/arch section within the issue.
The "Excluded packages" section and "Last updated" timestamp are rewritten on every run.
**Section replacement strategy:**
The script parses the existing issue body as markdown, finds the `## {platform}` + `### {arch}` section, replaces it, and writes back the full body.
If the section doesn't exist yet, it's appended under the correct `## {platform}` header (or a new one is created).
## Component 3: Audit Workflows
**Files:** 14 files, named `weekly-audit-missing-{platform}-{arch}.yaml`
Example: `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`
**Trigger:**
```yaml
when:
- event: cron
cron: weekly-audit-missing-ubuntu-2204-amd64
- event: manual
evaluate: 'task == "weekly-audit-missing-ubuntu-2204-amd64"'
```
**Step:** Lightweight — clones repo, installs bincraft + dependencies, runs the audit R script.
**Container:** Uses the platform-appropriate build image (needed for correct platform identification), but with minimal resource requests since no building happens.
**Resources:** ~2Gi memory, 2 CPUs.
## Component 4: Rebuild Workflows
**Files:** 14 files, named `weekly-rebuild-missing-{platform}-{arch}.yaml`
Example: `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`
**Trigger:**
```yaml
when:
- event: cron
cron: weekly-rebuild-missing-ubuntu-2204-amd64
- event: manual
evaluate: 'task == "weekly-rebuild-missing-ubuntu-2204-amd64"'
```
**Step:**
1. Clone repo, install bincraft.
2. Read `local/excluded-packages.json` as a safety net.
3. Read `/mnt/cache/packages/weekly_rebuild_{platform}_{arch}.rds`. If missing or empty, exit 0.
4. Filter out excluded packages (double-check).
5. Iterate and call `bincraft::build_binary_package()` for each package with `tag_limit = 1L`.
Same S3/DB parameters as existing build workflows.
6. Uses `xvfb-run` / `xwfb-run` for graphical packages (same pattern as existing builds).
**Resources:** Same as `process-updates` workflows — 5Gi request, 18Gi limit, 3 CPUs.
**Cache volume:** Maps `${ARCH}-binaries-r-dep-cache-${PLATFORM}:/mnt/cache` (same volumes as existing builds, so the RDS files written by audit are visible).
## Secrets Required
All existing secrets are reused:
- `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY` — S3 access
- `PGPASS` — PostgreSQL
- `REPO_RO_TOKEN` — Git clone
- `GITHUB_PAT` — For bincraft GitHub mirror access
New secret needed:
- `FORGEJO_TOKEN` — API token for creating/updating issues on git.devxy.io
## Cron Schedule
The audit and rebuild workflows each get their own cron names.
The cron schedule itself is configured in the Crow/Woodpecker server, not in the YAML.
Intended cadence: once per week (e.g. Sunday morning).
## File Inventory
| File | Type | Description |
|------|------|-------------|
| `local/excluded-packages.json` | Config | Excluded packages with reasons |
| `local/weekly-missing-binaries-audit.R` | R script | Audit logic, parameterized by env vars |
| `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml` | Workflow | Audit for ubuntu-2204/amd64 |
| `.crow/weekly-audit-missing-ubuntu-2204-arm64.yaml` | Workflow | Audit for ubuntu-2204/arm64 |
| `.crow/weekly-audit-missing-ubuntu-2404-amd64.yaml` | Workflow | Audit for ubuntu-2404/amd64 |
| `.crow/weekly-audit-missing-ubuntu-2404-arm64.yaml` | Workflow | Audit for ubuntu-2404/arm64 |
| `.crow/weekly-audit-missing-alpine-321-amd64.yaml` | Workflow | Audit for alpine-321/amd64 |
| `.crow/weekly-audit-missing-alpine-321-arm64.yaml` | Workflow | Audit for alpine-321/arm64 |
| `.crow/weekly-audit-missing-alpine-322-amd64.yaml` | Workflow | Audit for alpine-322/amd64 |
| `.crow/weekly-audit-missing-alpine-322-arm64.yaml` | Workflow | Audit for alpine-322/arm64 |
| `.crow/weekly-audit-missing-alpine-323-amd64.yaml` | Workflow | Audit for alpine-323/amd64 |
| `.crow/weekly-audit-missing-alpine-323-arm64.yaml` | Workflow | Audit for alpine-323/arm64 |
| `.crow/weekly-audit-missing-redhat-8-amd64.yaml` | Workflow | Audit for redhat-8/amd64 |
| `.crow/weekly-audit-missing-redhat-8-arm64.yaml` | Workflow | Audit for redhat-8/arm64 |
| `.crow/weekly-audit-missing-redhat-9-amd64.yaml` | Workflow | Audit for redhat-9/amd64 |
| `.crow/weekly-audit-missing-redhat-9-arm64.yaml` | Workflow | Audit for redhat-9/arm64 |
| `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml` | Workflow | Rebuild for ubuntu-2204/amd64 |
| `.crow/weekly-rebuild-missing-ubuntu-2204-arm64.yaml` | Workflow | Rebuild for ubuntu-2204/arm64 |
| `.crow/weekly-rebuild-missing-ubuntu-2404-amd64.yaml` | Workflow | Rebuild for ubuntu-2404/amd64 |
| `.crow/weekly-rebuild-missing-ubuntu-2404-arm64.yaml` | Workflow | Rebuild for ubuntu-2404/arm64 |
| `.crow/weekly-rebuild-missing-alpine-321-amd64.yaml` | Workflow | Rebuild for alpine-321/amd64 |
| `.crow/weekly-rebuild-missing-alpine-321-arm64.yaml` | Workflow | Rebuild for alpine-321/arm64 |
| `.crow/weekly-rebuild-missing-alpine-322-amd64.yaml` | Workflow | Rebuild for alpine-322/amd64 |
| `.crow/weekly-rebuild-missing-alpine-322-arm64.yaml` | Workflow | Rebuild for alpine-322/arm64 |
| `.crow/weekly-rebuild-missing-alpine-323-amd64.yaml` | Workflow | Rebuild for alpine-323/amd64 |
| `.crow/weekly-rebuild-missing-alpine-323-arm64.yaml` | Workflow | Rebuild for alpine-323/arm64 |
| `.crow/weekly-rebuild-missing-redhat-8-amd64.yaml` | Workflow | Rebuild for redhat-8/amd64 |
| `.crow/weekly-rebuild-missing-redhat-8-arm64.yaml` | Workflow | Rebuild for redhat-8/arm64 |
| `.crow/weekly-rebuild-missing-redhat-9-amd64.yaml` | Workflow | Rebuild for redhat-9/amd64 |
| `.crow/weekly-rebuild-missing-redhat-9-arm64.yaml` | Workflow | Rebuild for redhat-9/arm64 |
**Total: 29 new files** (1 JSON config + 1 R script + 14 audit workflows + 14 rebuild workflows + the design doc itself)