chore: gitignore docs/ and untrack existing files
Stop tracking docs/ — the existing spec and plan files (both this PR's multi-R-version-images set and the prior weekly-missing-binaries set) were intermediate working artifacts that don't need to live in the repo. Files remain on disk locally but are no longer in git.
This commit is contained in:
parent
bd793868bf
commit
5c3383f9e1
GPG key ID:
3C6318841EF78925
5 changed files with 1 additions and 2425 deletions
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -95,6 +95,6 @@ terraform.rc
|
|||
.envrc
|
||||
exec.sh
|
||||
exec.R
|
||||
docs/_site
|
||||
docs/
|
||||
local/test.R
|
||||
.DS_Store
|
||||
|
|
|
|||
|
|
@ -1,905 +0,0 @@
|
|||
# Weekly Missing Binaries Audit & Rebuild — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add a weekly CI workflow that audits CRAN packages missing binaries for their latest release version, reports them in Forgejo issues by OS family, and rebuilds those without a prior build failure.
|
||||
|
||||
**Architecture:** Two independent workflow sets (audit + rebuild), each with one YAML per platform/arch (14 each). A shared R script does the audit logic. An excluded-packages JSON config is the shared blocklist. The audit writes per-platform RDS files consumed by rebuild workflows.
|
||||
|
||||
**Tech Stack:** R (bincraft, s3fs, DBI/RPostgres, jsonlite, httr2), Crow/Woodpecker CI YAML, Forgejo API, PostgreSQL, Backblaze S3.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Responsibility |
|
||||
|------|---------------|
|
||||
| `local/excluded-packages.json` | Excluded packages with reasons (single source of truth) |
|
||||
| `local/weekly-missing-binaries-audit.R` | Audit logic: S3 check, DB check, issue update, RDS output |
|
||||
| `.crow/weekly-audit-missing-{platform}-{arch}.yaml` (14 files) | Audit workflow per platform/arch |
|
||||
| `.crow/weekly-rebuild-missing-{platform}-{arch}.yaml` (14 files) | Rebuild workflow per platform/arch |
|
||||
|
||||
## Platform Reference
|
||||
|
||||
Used throughout the plan. Each row is one workflow file pair (audit + rebuild).
|
||||
|
||||
| PLATFORM env var | ARCH | S3 codename (sub("-","",PLATFORM)) | OS Family | Image |
|
||||
|-----------------|------|-----------|-----------|-------|
|
||||
| ubuntu-2204 | amd64 | ubuntu2204 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 |
|
||||
| ubuntu-2204 | arm64 | ubuntu2204 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 |
|
||||
| ubuntu-2404 | amd64 | ubuntu2404 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 |
|
||||
| ubuntu-2404 | arm64 | ubuntu2404 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 |
|
||||
| alpine-321 | amd64 | alpine321 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 |
|
||||
| alpine-321 | arm64 | alpine321 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 |
|
||||
| alpine-322 | amd64 | alpine322 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 |
|
||||
| alpine-322 | arm64 | alpine322 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 |
|
||||
| alpine-323 | amd64 | alpine323 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 |
|
||||
| alpine-323 | arm64 | alpine323 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 |
|
||||
| redhat-8 | amd64 | redhat8 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 |
|
||||
| redhat-8 | arm64 | redhat8 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 |
|
||||
| redhat-9 | amd64 | redhat9 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 |
|
||||
| redhat-9 | arm64 | redhat9 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Create excluded-packages.json
|
||||
|
||||
**Files:**
|
||||
- Create: `local/excluded-packages.json`
|
||||
|
||||
- [ ] **Step 1: Create the JSON file**
|
||||
|
||||
Extract every package from the inline exclusion list in `.crow/build-all-versions-amd64.yaml` (line 72). The list has two categories: Windows-only packages (comment on line 69) and problematic packages (hang/OOM). Create `local/excluded-packages.json`:
|
||||
|
||||
```json
|
||||
[
|
||||
{"package": "RInno", "reason": "windows-only"},
|
||||
{"package": "KeyboardSimulator", "reason": "windows-only"},
|
||||
{"package": "R2PPT", "reason": "windows-only"},
|
||||
{"package": "RWinEdt", "reason": "windows-only"},
|
||||
{"package": "blatr", "reason": "windows-only"},
|
||||
{"package": "excel.link", "reason": "windows-only"},
|
||||
{"package": "spectrino", "reason": "windows-only"},
|
||||
{"package": "taskscheduleR", "reason": "windows-only"},
|
||||
{"package": "MDSGUI", "reason": "windows-only"},
|
||||
{"package": "BiplotGUI", "reason": "windows-only"},
|
||||
{"package": "R2wd", "reason": "windows-only"},
|
||||
{"package": "rFUSION", "reason": "windows-only"},
|
||||
{"package": "MediaNews", "reason": "windows-only"},
|
||||
{"package": "doBy", "reason": "hang"},
|
||||
{"package": "IDPmisc", "reason": "hang"},
|
||||
{"package": "frailtypack", "reason": "hang"},
|
||||
{"package": "afex", "reason": "hang"},
|
||||
{"package": "FrF2", "reason": "hang"},
|
||||
{"package": "DoE.base", "reason": "hang"},
|
||||
{"package": "agricolae", "reason": "hang"},
|
||||
{"package": "doFuture", "reason": "hang"},
|
||||
{"package": "fscaret", "reason": "hang"},
|
||||
{"package": "PHYLOGR", "reason": "hang"},
|
||||
{"package": "seewave", "reason": "hang"},
|
||||
{"package": "pls", "reason": "hang"},
|
||||
{"package": "relaimpo", "reason": "hang"},
|
||||
{"package": "geepack", "reason": "hang"},
|
||||
{"package": "gggenes", "reason": "hang"},
|
||||
{"package": "NPCirc", "reason": "hang"},
|
||||
{"package": "repmis", "reason": "hang"},
|
||||
{"package": "PNDSIBGE", "reason": "hang"},
|
||||
{"package": "lidR", "reason": "hang"},
|
||||
{"package": "poismf", "reason": "hang"},
|
||||
{"package": "neonstore", "reason": "hang"},
|
||||
{"package": "MachineShop", "reason": "hang"},
|
||||
{"package": "mvst", "reason": "hang"},
|
||||
{"package": "MacBehaviour", "reason": "hang"},
|
||||
{"package": "mcmcderive", "reason": "hang"},
|
||||
{"package": "RGIFT", "reason": "hang"},
|
||||
{"package": "KnowBR", "reason": "hang"},
|
||||
{"package": "netmeta", "reason": "hang"},
|
||||
{"package": "spdep", "reason": "hang"},
|
||||
{"package": "Rfast", "reason": "hang"},
|
||||
{"package": "compareGroups", "reason": "hang"},
|
||||
{"package": "ff", "reason": "hang"},
|
||||
{"package": "GsymPoint", "reason": "hang"},
|
||||
{"package": "RcppDynProg", "reason": "hang"},
|
||||
{"package": "comtradr", "reason": "hang"},
|
||||
{"package": "FD", "reason": "hang"},
|
||||
{"package": "PearsonDS", "reason": "hang"},
|
||||
{"package": "DCluster", "reason": "hang"},
|
||||
{"package": "gRc", "reason": "hang"},
|
||||
{"package": "mixlm", "reason": "hang"},
|
||||
{"package": "geospt", "reason": "hang"},
|
||||
{"package": "fdth", "reason": "hang"},
|
||||
{"package": "ffmanova", "reason": "hang"},
|
||||
{"package": "fiery", "reason": "hang"},
|
||||
{"package": "ffscrapr", "reason": "hang"},
|
||||
{"package": "cold", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.DoE", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.NMBU", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.RiskDemo", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.TeachStat", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.TeachingDemos", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.UCA", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.WorldFlora", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.aRnova", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.depthTools", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.orloca", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.sos", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.survival", "reason": "hang"},
|
||||
{"package": "RcmdrPlugin.temis", "reason": "hang"},
|
||||
{"package": "GWlasso", "reason": "hang"},
|
||||
{"package": "GWmodelVis", "reason": "hang"}
|
||||
]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add local/excluded-packages.json
|
||||
git commit -m "feat: add excluded-packages.json for weekly missing binaries workflow"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Create the audit R script
|
||||
|
||||
**Files:**
|
||||
- Create: `local/weekly-missing-binaries-audit.R`
|
||||
|
||||
This is the core logic. It reads env vars `PLATFORM` and `ARCH`, checks S3 + DB, writes an RDS of rebuildable packages, and updates the Forgejo issue for the OS family.
|
||||
|
||||
- [ ] **Step 1: Create the R script**
|
||||
|
||||
Create `local/weekly-missing-binaries-audit.R` with the full content below:
|
||||
|
||||
```r
|
||||
options(error = function() {
|
||||
cat("ERROR:", geterrmessage(), "\n", file = stdout())
|
||||
traceback(2)
|
||||
q(status = 1)
|
||||
})
|
||||
|
||||
library(DBI, quietly = TRUE)
|
||||
suppressPackageStartupMessages(library(data.table))
|
||||
|
||||
platform <- Sys.getenv("PLATFORM")
|
||||
arch <- Sys.getenv("ARCH")
|
||||
stopifnot(nzchar(platform), nzchar(arch))
|
||||
|
||||
# S3 codename: remove hyphen from platform string (e.g. "ubuntu-2204" -> "ubuntu2204")
|
||||
s3_codename <- gsub("-", "", platform)
|
||||
|
||||
# OS family for issue grouping
|
||||
os_family <- if (grepl("^ubuntu", platform)) {
|
||||
"Ubuntu"
|
||||
} else if (grepl("^alpine", platform)) {
|
||||
"Alpine"
|
||||
} else if (grepl("^redhat", platform)) {
|
||||
"Red Hat"
|
||||
} else {
|
||||
stop("Unknown platform: ", platform)
|
||||
}
|
||||
|
||||
cat(sprintf("Audit: platform=%s, arch=%s, s3_codename=%s, os_family=%s\n",
|
||||
platform, arch, s3_codename, os_family))
|
||||
|
||||
### 1. Get current CRAN release packages
|
||||
cran_release <- as.data.table(tools::CRAN_package_db()[, c("Package", "Version")])
|
||||
cran_release[, Version := as.character(Version)]
|
||||
cat(sprintf("CRAN release packages: %d\n", nrow(cran_release)))
|
||||
|
||||
### 2. List S3 tarballs for this platform/arch
|
||||
s3fs::s3_file_system(
|
||||
aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"),
|
||||
aws_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"),
|
||||
endpoint = "https://s3.eu-central-003.backblazeb2.com",
|
||||
region_name = "eu-central-003",
|
||||
refresh = TRUE
|
||||
)
|
||||
|
||||
s3_path <- sprintf("devxy-rpkgs-binaries/%s/%s/latest/src/contrib", arch, s3_codename)
|
||||
s3_files <- tryCatch(
|
||||
s3fs::s3_dir_ls(s3_path),
|
||||
error = function(e) {
|
||||
cat(sprintf("Warning: could not list S3 path %s: %s\n", s3_path, conditionMessage(e)))
|
||||
character(0)
|
||||
}
|
||||
)
|
||||
|
||||
# Parse package name + version from tarball filenames
|
||||
file_names <- basename(s3_files)
|
||||
matches <- regexec("^([A-Za-z0-9.]+)_([0-9][^/]*)\\.tar\\.gz$", file_names)
|
||||
parts <- regmatches(file_names, matches)
|
||||
parts <- parts[lengths(parts) == 3]
|
||||
|
||||
s3_dt <- if (length(parts) > 0) {
|
||||
data.table(
|
||||
Package = vapply(parts, `[`, character(1), 2),
|
||||
Version = vapply(parts, `[`, character(1), 3)
|
||||
)
|
||||
} else {
|
||||
data.table(Package = character(0), Version = character(0))
|
||||
}
|
||||
cat(sprintf("S3 packages for %s/%s: %d\n", arch, s3_codename, nrow(s3_dt)))
|
||||
|
||||
### 3. Find CRAN release packages missing from S3
|
||||
setkey(cran_release, Package, Version)
|
||||
setkey(s3_dt, Package, Version)
|
||||
missing <- cran_release[!s3_dt]
|
||||
cat(sprintf("Missing binaries (latest CRAN version): %d\n", nrow(missing)))
|
||||
|
||||
### 4. Load excluded packages
|
||||
excluded_json <- jsonlite::fromJSON("local/excluded-packages.json")
|
||||
excluded_pkgs <- excluded_json$package
|
||||
missing_not_excluded <- missing[!Package %in% excluded_pkgs]
|
||||
missing_excluded <- missing[Package %in% excluded_pkgs]
|
||||
cat(sprintf("Missing after excluding %d blocked packages: %d\n",
|
||||
length(excluded_pkgs), nrow(missing_not_excluded)))
|
||||
|
||||
### 5. Check DB for prior build failures on these exact versions
|
||||
con <- DBI::dbConnect(
|
||||
RPostgres::Postgres(),
|
||||
dbname = "build_metadata",
|
||||
host = "r-binaries.devxy.io",
|
||||
port = 15432,
|
||||
user = "rpkgs",
|
||||
password = Sys.getenv("PGPASS"),
|
||||
sslmode = "require"
|
||||
)
|
||||
on.exit(DBI::dbDisconnect(con), add = TRUE)
|
||||
|
||||
errored_pkgs <- as.data.table(DBI::dbGetQuery(
|
||||
con,
|
||||
sprintf(
|
||||
"SELECT name, tag FROM single_builds WHERE error_occurred = TRUE AND platform = '%s' AND arch = '%s'",
|
||||
platform, arch
|
||||
)
|
||||
))
|
||||
setnames(errored_pkgs, c("Package", "Version"))
|
||||
setkey(errored_pkgs, Package, Version)
|
||||
|
||||
# Split: rebuildable vs known failures
|
||||
known_failures <- missing_not_excluded[errored_pkgs, nomatch = 0, on = c("Package", "Version")]
|
||||
rebuildable <- missing_not_excluded[!errored_pkgs, on = c("Package", "Version")]
|
||||
|
||||
cat(sprintf("Known build failures: %d\n", nrow(known_failures)))
|
||||
cat(sprintf("Rebuildable (no prior failure): %d\n", nrow(rebuildable)))
|
||||
|
||||
### 6. Write RDS for rebuild workflow
|
||||
cache_dir <- "/mnt/cache/packages"
|
||||
if (dir.exists(cache_dir)) {
|
||||
rds_path <- file.path(cache_dir, sprintf("weekly_rebuild_%s_%s.rds", platform, arch))
|
||||
saveRDS(rebuildable$Package, rds_path)
|
||||
cat(sprintf("Wrote %d packages to %s\n", nrow(rebuildable), rds_path))
|
||||
} else {
|
||||
cat(sprintf("Cache dir %s does not exist, skipping RDS write\n", cache_dir))
|
||||
}
|
||||
|
||||
### 7. Update Forgejo issue
|
||||
forgejo_token <- Sys.getenv("FORGEJO_TOKEN")
|
||||
if (!nzchar(forgejo_token)) {
|
||||
cat("FORGEJO_TOKEN not set, skipping issue update\n")
|
||||
} else {
|
||||
base_url <- "https://git.devxy.io/api/v1"
|
||||
repo <- "devxy/build-cran-binaries"
|
||||
issue_title <- sprintf("Missing package binaries for latest version (%s)", os_family)
|
||||
|
||||
# Helper: make API request
|
||||
forgejo_get <- function(path, query = list()) {
|
||||
url <- paste0(base_url, path)
|
||||
resp <- httr2::request(url) |>
|
||||
httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
|
||||
httr2::req_url_query(!!!query) |>
|
||||
httr2::req_perform()
|
||||
httr2::resp_body_json(resp)
|
||||
}
|
||||
|
||||
forgejo_patch <- function(path, body) {
|
||||
url <- paste0(base_url, path)
|
||||
httr2::request(url) |>
|
||||
httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
|
||||
httr2::req_method("PATCH") |>
|
||||
httr2::req_body_json(body) |>
|
||||
httr2::req_perform()
|
||||
}
|
||||
|
||||
forgejo_post <- function(path, body) {
|
||||
url <- paste0(base_url, path)
|
||||
httr2::request(url) |>
|
||||
httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
|
||||
httr2::req_body_json(body) |>
|
||||
httr2::req_perform()
|
||||
}
|
||||
|
||||
# Build the markdown section for this platform/arch
|
||||
build_section <- function() {
|
||||
n_missing <- nrow(rebuildable) + nrow(known_failures)
|
||||
n_rebuild <- nrow(rebuildable)
|
||||
header <- sprintf("### %s (%d missing, %d to rebuild)", arch, n_missing, n_rebuild)
|
||||
|
||||
lines <- header
|
||||
if (nrow(rebuildable) > 0) {
|
||||
pkg_lines <- sprintf("- %s (%s)", rebuildable$Package, rebuildable$Version)
|
||||
lines <- c(lines, "", pkg_lines)
|
||||
} else if (nrow(known_failures) == 0) {
|
||||
lines <- c(lines, "", "All binaries available.")
|
||||
}
|
||||
|
||||
if (nrow(known_failures) > 0) {
|
||||
lines <- c(lines, "", "#### Known build failures",
|
||||
sprintf("- %s (%s)", known_failures$Package, known_failures$Version))
|
||||
}
|
||||
|
||||
paste(lines, collapse = "\n")
|
||||
}
|
||||
|
||||
# Build excluded packages footer
|
||||
build_excluded_footer <- function() {
|
||||
if (nrow(excluded_json) == 0) return("")
|
||||
items <- sprintf("%s (%s)", excluded_json$package, excluded_json$reason)
|
||||
paste0("\n---\n\n## Excluded packages\n", paste(items, collapse = ", "))
|
||||
}
|
||||
|
||||
new_section <- build_section()
|
||||
|
||||
# Search for existing issue
|
||||
issues <- forgejo_get(
|
||||
sprintf("/repos/%s/issues", repo),
|
||||
query = list(type = "issues", state = "open", q = issue_title, limit = 50)
|
||||
)
|
||||
|
||||
# Find exact title match
|
||||
existing <- Filter(function(i) i$title == issue_title, issues)
|
||||
|
||||
if (length(existing) > 0) {
|
||||
issue <- existing[[1]]
|
||||
body <- issue$body
|
||||
|
||||
# Replace or insert the platform section + arch subsection
|
||||
platform_header <- sprintf("## %s", platform)
|
||||
arch_header <- sprintf("### %s", arch)
|
||||
|
||||
# Split body into lines for manipulation
|
||||
body_lines <- strsplit(body, "\n")[[1]]
|
||||
|
||||
# Find the platform section
|
||||
platform_start <- which(body_lines == platform_header)
|
||||
|
||||
if (length(platform_start) > 0) {
|
||||
platform_start <- platform_start[1]
|
||||
# Find end of this platform section (next ## or --- or end)
|
||||
remaining <- body_lines[(platform_start + 1):length(body_lines)]
|
||||
platform_end_offset <- which(grepl("^## |^---$", remaining))
|
||||
platform_end <- if (length(platform_end_offset) > 0) {
|
||||
platform_start + platform_end_offset[1] - 1
|
||||
} else {
|
||||
length(body_lines)
|
||||
}
|
||||
|
||||
# Within platform section, find the arch subsection
|
||||
section_lines <- body_lines[platform_start:platform_end]
|
||||
arch_start_offset <- which(grepl(sprintf("^### %s", arch), section_lines))
|
||||
|
||||
if (length(arch_start_offset) > 0) {
|
||||
arch_start <- arch_start_offset[1]
|
||||
# Find end of arch subsection (next ### or ## or --- or end of platform section)
|
||||
arch_remaining <- section_lines[(arch_start + 1):length(section_lines)]
|
||||
arch_end_offset <- which(grepl("^###|^## |^---$", arch_remaining))
|
||||
arch_end <- if (length(arch_end_offset) > 0) {
|
||||
arch_start + arch_end_offset[1] - 1
|
||||
} else {
|
||||
length(section_lines)
|
||||
}
|
||||
# Replace arch subsection within platform section
|
||||
section_lines <- c(
|
||||
section_lines[1:(arch_start - 1)],
|
||||
strsplit(new_section, "\n")[[1]],
|
||||
if (arch_end < length(section_lines)) section_lines[(arch_end + 1):length(section_lines)] else character(0)
|
||||
)
|
||||
} else {
|
||||
# Append arch subsection to end of platform section
|
||||
section_lines <- c(section_lines, "", strsplit(new_section, "\n")[[1]])
|
||||
}
|
||||
|
||||
body_lines <- c(
|
||||
body_lines[1:(platform_start - 1)],
|
||||
section_lines,
|
||||
if (platform_end < length(body_lines)) body_lines[(platform_end + 1):length(body_lines)] else character(0)
|
||||
)
|
||||
} else {
|
||||
# Insert new platform section before "---" (excluded packages footer) or at end
|
||||
footer_line <- which(body_lines == "---")
|
||||
insert_at <- if (length(footer_line) > 0) footer_line[1] - 1 else length(body_lines)
|
||||
body_lines <- c(
|
||||
body_lines[1:insert_at],
|
||||
"",
|
||||
platform_header,
|
||||
"",
|
||||
strsplit(new_section, "\n")[[1]],
|
||||
if (insert_at < length(body_lines)) body_lines[(insert_at + 1):length(body_lines)] else character(0)
|
||||
)
|
||||
}
|
||||
|
||||
# Update timestamp
|
||||
timestamp_pattern <- "^_Last updated:.*_$"
|
||||
ts_line <- which(grepl(timestamp_pattern, body_lines))
|
||||
new_ts <- sprintf("_Last updated: %s_", Sys.Date())
|
||||
if (length(ts_line) > 0) {
|
||||
body_lines[ts_line[1]] <- new_ts
|
||||
} else {
|
||||
body_lines <- c(new_ts, "", body_lines)
|
||||
}
|
||||
|
||||
# Rebuild excluded footer
|
||||
footer_start <- which(body_lines == "---")
|
||||
if (length(footer_start) > 0) {
|
||||
body_lines <- c(body_lines[1:(footer_start[1] - 1)],
|
||||
strsplit(build_excluded_footer(), "\n")[[1]])
|
||||
} else {
|
||||
body_lines <- c(body_lines, strsplit(build_excluded_footer(), "\n")[[1]])
|
||||
}
|
||||
|
||||
new_body <- paste(body_lines, collapse = "\n")
|
||||
forgejo_patch(
|
||||
sprintf("/repos/%s/issues/%d", repo, issue$number),
|
||||
list(body = new_body)
|
||||
)
|
||||
cat(sprintf("Updated issue #%d: %s\n", issue$number, issue_title))
|
||||
} else {
|
||||
# Create new issue
|
||||
body_lines <- c(
|
||||
sprintf("_Last updated: %s_", Sys.Date()),
|
||||
"",
|
||||
sprintf("## %s", platform),
|
||||
"",
|
||||
new_section,
|
||||
build_excluded_footer()
|
||||
)
|
||||
new_body <- paste(body_lines, collapse = "\n")
|
||||
forgejo_post(
|
||||
sprintf("/repos/%s/issues", repo),
|
||||
list(title = issue_title, body = new_body)
|
||||
)
|
||||
cat(sprintf("Created new issue: %s\n", issue_title))
|
||||
}
|
||||
}
|
||||
|
||||
cat("Audit complete.\n")
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add local/weekly-missing-binaries-audit.R
|
||||
git commit -m "feat: add weekly missing binaries audit R script"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Create audit workflow for ubuntu-2204-amd64
|
||||
|
||||
**Files:**
|
||||
- Create: `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`
|
||||
|
||||
This is the template. All other audit workflows follow the same structure with substituted values.
|
||||
|
||||
- [ ] **Step 1: Create the workflow YAML**
|
||||
|
||||
Create `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`:
|
||||
|
||||
```yaml
|
||||
when:
|
||||
- event: cron
|
||||
cron: weekly-audit-missing-ubuntu-2204-amd64
|
||||
- event: manual
|
||||
evaluate: 'task == "weekly-audit-missing-ubuntu-2204-amd64"'
|
||||
|
||||
skip_clone: true
|
||||
|
||||
steps:
|
||||
- name: 'Audit missing binaries'
|
||||
image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
|
||||
pull: true
|
||||
environment:
|
||||
B2_S3_ACCESS_KEY:
|
||||
from_secret: B2_S3_ACCESS_KEY
|
||||
B2_S3_SECRET_KEY:
|
||||
from_secret: B2_S3_SECRET_KEY
|
||||
PGPASS:
|
||||
from_secret: PGPASS
|
||||
REPO_RO_TOKEN:
|
||||
from_secret: REPO_RO_TOKEN
|
||||
FORGEJO_TOKEN:
|
||||
from_secret: FORGEJO_TOKEN
|
||||
GITHUB_PAT:
|
||||
from_secret: GITHUB_PAT
|
||||
PLATFORM: ubuntu-2204
|
||||
ARCH: amd64
|
||||
R_LIBS_USER: /mnt/cache/R-pkgs
|
||||
volumes:
|
||||
- amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache
|
||||
commands:
|
||||
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
||||
- mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs
|
||||
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
|
||||
- R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))'
|
||||
- R -q -e 'source("local/weekly-missing-binaries-audit.R")'
|
||||
backend_options:
|
||||
kubernetes:
|
||||
resources:
|
||||
requests:
|
||||
memory: 2Gi
|
||||
cpu: 2000m
|
||||
limits:
|
||||
memory: 4Gi
|
||||
cpu: 2000m
|
||||
nodeSelector:
|
||||
kubernetes.io/arch: amd64
|
||||
tolerations:
|
||||
- key: 'CI'
|
||||
operator: 'Equal'
|
||||
value: 'true'
|
||||
effect: 'NoSchedule'
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add .crow/weekly-audit-missing-ubuntu-2204-amd64.yaml
|
||||
git commit -m "feat: add weekly audit workflow for ubuntu-2204-amd64"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Create remaining 13 audit workflows
|
||||
|
||||
**Files:**
|
||||
- Create: 13 files in `.crow/` (see substitution table below)
|
||||
|
||||
Each file follows the exact same structure as Task 3 with these substitutions:
|
||||
|
||||
| File suffix | PLATFORM | ARCH | Image | Volume |
|
||||
|-------------|----------|------|-------|--------|
|
||||
| ubuntu-2204-arm64 | ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2204 |
|
||||
| ubuntu-2404-amd64 | ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | amd64-binaries-r-dep-cache-ubuntu2404 |
|
||||
| ubuntu-2404-arm64 | ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2404 |
|
||||
| alpine-321-amd64 | alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | amd64-binaries-r-dep-cache-alpine321 |
|
||||
| alpine-321-arm64 | alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | arm64-binaries-r-dep-cache-alpine321 |
|
||||
| alpine-322-amd64 | alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | amd64-binaries-r-dep-cache-alpine322 |
|
||||
| alpine-322-arm64 | alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | arm64-binaries-r-dep-cache-alpine322 |
|
||||
| alpine-323-amd64 | alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | amd64-binaries-r-dep-cache-alpine323 |
|
||||
| alpine-323-arm64 | alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | arm64-binaries-r-dep-cache-alpine323 |
|
||||
| redhat-8-amd64 | redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | amd64-binaries-r-dep-cache-redhat8 |
|
||||
| redhat-8-arm64 | redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | arm64-binaries-r-dep-cache-redhat8 |
|
||||
| redhat-9-amd64 | redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | amd64-binaries-r-dep-cache-redhat9 |
|
||||
| redhat-9-arm64 | redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | arm64-binaries-r-dep-cache-redhat9 |
|
||||
|
||||
**Key substitution points in each YAML (6 locations):**
|
||||
|
||||
1. `cron:` value — `weekly-audit-missing-{suffix}`
|
||||
2. `evaluate:` value — `'task == "weekly-audit-missing-{suffix}"'`
|
||||
3. `image:` — use the Image column
|
||||
4. `PLATFORM:` env var — use the PLATFORM column
|
||||
5. `ARCH:` env var — use the ARCH column
|
||||
6. `volumes:` — `{ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache`
|
||||
7. `nodeSelector: kubernetes.io/arch:` — use the ARCH column
|
||||
|
||||
- [ ] **Step 1: Create all 13 audit workflow files**
|
||||
|
||||
Copy the template from Task 3 and substitute the values per the table above. Example for alpine-321-arm64:
|
||||
|
||||
```yaml
|
||||
when:
|
||||
- event: cron
|
||||
cron: weekly-audit-missing-alpine-321-arm64
|
||||
- event: manual
|
||||
evaluate: 'task == "weekly-audit-missing-alpine-321-arm64"'
|
||||
|
||||
skip_clone: true
|
||||
|
||||
steps:
|
||||
- name: 'Audit missing binaries'
|
||||
image: reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5
|
||||
pull: true
|
||||
environment:
|
||||
B2_S3_ACCESS_KEY:
|
||||
from_secret: B2_S3_ACCESS_KEY
|
||||
B2_S3_SECRET_KEY:
|
||||
from_secret: B2_S3_SECRET_KEY
|
||||
PGPASS:
|
||||
from_secret: PGPASS
|
||||
REPO_RO_TOKEN:
|
||||
from_secret: REPO_RO_TOKEN
|
||||
FORGEJO_TOKEN:
|
||||
from_secret: FORGEJO_TOKEN
|
||||
GITHUB_PAT:
|
||||
from_secret: GITHUB_PAT
|
||||
PLATFORM: alpine-321
|
||||
ARCH: arm64
|
||||
R_LIBS_USER: /mnt/cache/R-pkgs
|
||||
volumes:
|
||||
- arm64-binaries-r-dep-cache-alpine321:/mnt/cache
|
||||
commands:
|
||||
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
||||
- mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs
|
||||
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
|
||||
- R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))'
|
||||
- R -q -e 'source("local/weekly-missing-binaries-audit.R")'
|
||||
backend_options:
|
||||
kubernetes:
|
||||
resources:
|
||||
requests:
|
||||
memory: 2Gi
|
||||
cpu: 2000m
|
||||
limits:
|
||||
memory: 4Gi
|
||||
cpu: 2000m
|
||||
nodeSelector:
|
||||
kubernetes.io/arch: arm64
|
||||
tolerations:
|
||||
- key: 'CI'
|
||||
operator: 'Equal'
|
||||
value: 'true'
|
||||
effect: 'NoSchedule'
|
||||
```
|
||||
|
||||
Repeat for all 13 remaining suffixes from the table, substituting the 7 locations.
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add .crow/weekly-audit-missing-*.yaml
|
||||
git commit -m "feat: add remaining 13 weekly audit workflows for all platform/arch combos"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Create rebuild workflow for ubuntu-2204-amd64
|
||||
|
||||
**Files:**
|
||||
- Create: `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`
|
||||
|
||||
This is the template for rebuild workflows. It reads the RDS written by the audit and builds each package.
|
||||
|
||||
- [ ] **Step 1: Create the workflow YAML**
|
||||
|
||||
Create `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`:
|
||||
|
||||
```yaml
|
||||
when:
|
||||
- event: cron
|
||||
cron: weekly-rebuild-missing-ubuntu-2204-amd64
|
||||
- event: manual
|
||||
evaluate: 'task == "weekly-rebuild-missing-ubuntu-2204-amd64"'
|
||||
|
||||
skip_clone: true
|
||||
|
||||
steps:
|
||||
- name: 'Rebuild missing binaries'
|
||||
image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
|
||||
pull: true
|
||||
environment:
|
||||
RED_HAT_DEV_PW:
|
||||
from_secret: RED_HAT_DEV_PW
|
||||
B2_S3_ACCESS_KEY:
|
||||
from_secret: B2_S3_ACCESS_KEY
|
||||
B2_S3_SECRET_KEY:
|
||||
from_secret: B2_S3_SECRET_KEY
|
||||
PGPASS:
|
||||
from_secret: PGPASS
|
||||
REPO_RO_TOKEN:
|
||||
from_secret: REPO_RO_TOKEN
|
||||
GITHUB_PAT:
|
||||
from_secret: GITHUB_PAT
|
||||
GIT_USER: pat-s
|
||||
R_PKG_CACHE_DIR: /mnt/cache/pkgcache
|
||||
R_LIBS_USER: /mnt/cache/R-pkgs
|
||||
CCACHE_DIR: /mnt/cache/ccache
|
||||
PLATFORM: ubuntu-2204
|
||||
ARCH: amd64
|
||||
NCPUS: 2
|
||||
volumes:
|
||||
- amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache
|
||||
commands:
|
||||
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
||||
- mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
|
||||
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
|
||||
- R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
|
||||
- R -q -e 'packageVersion("bincraft")'
|
||||
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
|
||||
- $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_ubuntu-2204_amd64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1
|
||||
backend_options:
|
||||
kubernetes:
|
||||
resources:
|
||||
requests:
|
||||
memory: 5Gi
|
||||
cpu: 3000m
|
||||
limits:
|
||||
memory: 18Gi
|
||||
cpu: 3000m
|
||||
nodeSelector:
|
||||
kubernetes.io/arch: amd64
|
||||
node.kubernetes.io/instance-type: AX42
|
||||
tolerations:
|
||||
- key: 'CI'
|
||||
operator: 'Equal'
|
||||
value: 'true'
|
||||
effect: 'NoSchedule'
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add .crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml
|
||||
git commit -m "feat: add weekly rebuild workflow for ubuntu-2204-amd64"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Create remaining 13 rebuild workflows
|
||||
|
||||
**Files:**
|
||||
- Create: 13 files in `.crow/` (see substitution table below)
|
||||
|
||||
Each file follows the exact same structure as Task 5 with these substitutions:
|
||||
|
||||
| File suffix | PLATFORM | ARCH | Image | Volume | nodeSelector instance-type |
|
||||
|-------------|----------|------|-------|--------|--------------------------|
|
||||
| ubuntu-2204-arm64 | ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2204 | (omit) |
|
||||
| ubuntu-2404-amd64 | ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | amd64-binaries-r-dep-cache-ubuntu2404 | AX42 |
|
||||
| ubuntu-2404-arm64 | ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2404 | (omit) |
|
||||
| alpine-321-amd64 | alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | amd64-binaries-r-dep-cache-alpine321 | AX42 |
|
||||
| alpine-321-arm64 | alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | arm64-binaries-r-dep-cache-alpine321 | (omit) |
|
||||
| alpine-322-amd64 | alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | amd64-binaries-r-dep-cache-alpine322 | AX42 |
|
||||
| alpine-322-arm64 | alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | arm64-binaries-r-dep-cache-alpine322 | (omit) |
|
||||
| alpine-323-amd64 | alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | amd64-binaries-r-dep-cache-alpine323 | AX42 |
|
||||
| alpine-323-arm64 | alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | arm64-binaries-r-dep-cache-alpine323 | (omit) |
|
||||
| redhat-8-amd64 | redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | amd64-binaries-r-dep-cache-redhat8 | AX42 |
|
||||
| redhat-8-arm64 | redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | arm64-binaries-r-dep-cache-redhat8 | (omit) |
|
||||
| redhat-9-amd64 | redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | amd64-binaries-r-dep-cache-redhat9 | AX42 |
|
||||
| redhat-9-arm64 | redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | arm64-binaries-r-dep-cache-redhat9 | (omit) |
|
||||
|
||||
**Key substitution points (8 locations):**
|
||||
|
||||
1. `cron:` value — `weekly-rebuild-missing-{suffix}`
|
||||
2. `evaluate:` value — `'task == "weekly-rebuild-missing-{suffix}"'`
|
||||
3. `image:` — use the Image column
|
||||
4. `PLATFORM:` env var — use the PLATFORM column
|
||||
5. `ARCH:` env var — use the ARCH column
|
||||
6. `volumes:` — `{ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache`
|
||||
7. `rds_path` in the R command — `weekly_rebuild_{PLATFORM}_{ARCH}.rds`
|
||||
8. `nodeSelector` — `kubernetes.io/arch: {ARCH}`, plus `node.kubernetes.io/instance-type: AX42` only for amd64 (omit the instance-type line entirely for arm64)
|
||||
|
||||
- [ ] **Step 1: Create all 13 rebuild workflow files**
|
||||
|
||||
Copy the template from Task 5 and substitute per the table. Example for redhat-9-arm64:
|
||||
|
||||
```yaml
|
||||
when:
|
||||
- event: cron
|
||||
cron: weekly-rebuild-missing-redhat-9-arm64
|
||||
- event: manual
|
||||
evaluate: 'task == "weekly-rebuild-missing-redhat-9-arm64"'
|
||||
|
||||
skip_clone: true
|
||||
|
||||
steps:
|
||||
- name: 'Rebuild missing binaries'
|
||||
image: reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3
|
||||
pull: true
|
||||
environment:
|
||||
RED_HAT_DEV_PW:
|
||||
from_secret: RED_HAT_DEV_PW
|
||||
B2_S3_ACCESS_KEY:
|
||||
from_secret: B2_S3_ACCESS_KEY
|
||||
B2_S3_SECRET_KEY:
|
||||
from_secret: B2_S3_SECRET_KEY
|
||||
PGPASS:
|
||||
from_secret: PGPASS
|
||||
REPO_RO_TOKEN:
|
||||
from_secret: REPO_RO_TOKEN
|
||||
GITHUB_PAT:
|
||||
from_secret: GITHUB_PAT
|
||||
GIT_USER: pat-s
|
||||
R_PKG_CACHE_DIR: /mnt/cache/pkgcache
|
||||
R_LIBS_USER: /mnt/cache/R-pkgs
|
||||
CCACHE_DIR: /mnt/cache/ccache
|
||||
PLATFORM: redhat-9
|
||||
ARCH: arm64
|
||||
NCPUS: 2
|
||||
volumes:
|
||||
- arm64-binaries-r-dep-cache-redhat9:/mnt/cache
|
||||
commands:
|
||||
- git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
||||
- mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
|
||||
- rm -rf /mnt/cache/R-pkgs/00LOCK-*
|
||||
- R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
|
||||
- R -q -e 'packageVersion("bincraft")'
|
||||
- XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
|
||||
- $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_redhat-9_arm64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1
|
||||
backend_options:
|
||||
kubernetes:
|
||||
resources:
|
||||
requests:
|
||||
memory: 5Gi
|
||||
cpu: 3000m
|
||||
limits:
|
||||
memory: 18Gi
|
||||
cpu: 3000m
|
||||
nodeSelector:
|
||||
kubernetes.io/arch: arm64
|
||||
tolerations:
|
||||
- key: 'CI'
|
||||
operator: 'Equal'
|
||||
value: 'true'
|
||||
effect: 'NoSchedule'
|
||||
```
|
||||
|
||||
Note: arm64 workflows do NOT include `node.kubernetes.io/instance-type: AX42` in nodeSelector.
|
||||
|
||||
Repeat for all 13 remaining suffixes from the table, substituting the 8 locations.
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add .crow/weekly-rebuild-missing-*.yaml
|
||||
git commit -m "feat: add remaining 13 weekly rebuild workflows for all platform/arch combos"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Final verification
|
||||
|
||||
- [ ] **Step 1: Verify all files exist**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
ls -1 local/excluded-packages.json local/weekly-missing-binaries-audit.R
|
||||
ls -1 .crow/weekly-audit-missing-*.yaml | wc -l
|
||||
ls -1 .crow/weekly-rebuild-missing-*.yaml | wc -l
|
||||
```
|
||||
|
||||
Expected: both files exist, 14 audit workflows, 14 rebuild workflows.
|
||||
|
||||
- [ ] **Step 2: Validate JSON**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python3 -c "import json; json.load(open('local/excluded-packages.json')); print('JSON valid')"
|
||||
```
|
||||
|
||||
Expected: `JSON valid`
|
||||
|
||||
- [ ] **Step 3: Validate YAML syntax**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python3 -c "
|
||||
import yaml, glob
|
||||
for f in sorted(glob.glob('.crow/weekly-*-missing-*.yaml') + glob.glob('.crow/weekly-audit-missing-*.yaml') + glob.glob('.crow/weekly-rebuild-missing-*.yaml')):
|
||||
yaml.safe_load(open(f))
|
||||
print(f'OK: {f}')
|
||||
"
|
||||
```
|
||||
|
||||
Expected: all files print `OK`.
|
||||
|
||||
- [ ] **Step 4: Verify unique cron names**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
grep -h 'cron: weekly-' .crow/weekly-*.yaml | sort | uniq -c | sort -rn | head
|
||||
```
|
||||
|
||||
Expected: all counts are 1 (no duplicates).
|
||||
|
||||
- [ ] **Step 5: Verify all 14 platform/arch combos covered**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
for suffix in ubuntu-2204-amd64 ubuntu-2204-arm64 ubuntu-2404-amd64 ubuntu-2404-arm64 alpine-321-amd64 alpine-321-arm64 alpine-322-amd64 alpine-322-arm64 alpine-323-amd64 alpine-323-arm64 redhat-8-amd64 redhat-8-arm64 redhat-9-amd64 redhat-9-arm64; do
|
||||
test -f ".crow/weekly-audit-missing-${suffix}.yaml" || echo "MISSING audit: ${suffix}"
|
||||
test -f ".crow/weekly-rebuild-missing-${suffix}.yaml" || echo "MISSING rebuild: ${suffix}"
|
||||
done
|
||||
echo "All checks passed if no MISSING lines above"
|
||||
```
|
||||
|
||||
Expected: no MISSING lines, just "All checks passed".
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,246 +0,0 @@
|
|||
# Weekly Missing Binaries Audit & Rebuild
|
||||
|
||||
## Goal
|
||||
|
||||
A weekly CI workflow that identifies CRAN packages whose latest release version has no binary available, reports them in Forgejo issues grouped by OS family and arch, and rebuilds those that have no prior build failure recorded in the database.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
Two independent workflow sets, each with one file per platform/arch combo (14 files each), plus a shared R script and an excluded-packages config file.
|
||||
|
||||
```
|
||||
weekly-audit-missing-*-*.yaml (14 files)
|
||||
│
|
||||
▼
|
||||
local/weekly-missing-binaries-audit.R
|
||||
│
|
||||
├── Updates Forgejo issues (3 issues, one per OS family)
|
||||
└── Writes RDS files to /mnt/cache/packages/
|
||||
|
||||
weekly-rebuild-missing-*-*.yaml (14 files)
|
||||
│
|
||||
▼
|
||||
Reads RDS, builds missing packages via bincraft::build_binary_package()
|
||||
```
|
||||
|
||||
The audit and build workflows are fully independent.
|
||||
Either can be triggered on its own via cron or manually.
|
||||
|
||||
## Platforms
|
||||
|
||||
All current platform/arch combinations:
|
||||
|
||||
| Platform | Arch | Image | Codename |
|
||||
|-------------|-------|-------------------------------------------------|------------|
|
||||
| ubuntu-2204 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | jammy |
|
||||
| ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | jammy |
|
||||
| ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | noble |
|
||||
| ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | noble |
|
||||
| alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | alpine321 |
|
||||
| alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | alpine321 |
|
||||
| alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | alpine322 |
|
||||
| alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | alpine322 |
|
||||
| alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | alpine323 |
|
||||
| alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | alpine323 |
|
||||
| redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | rhel8 |
|
||||
| redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | rhel8 |
|
||||
| redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | rhel9 |
|
||||
| redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | rhel9 |
|
||||
|
||||
## Component 1: Excluded Packages Config
|
||||
|
||||
**File:** `local/excluded-packages.json`
|
||||
|
||||
A JSON array of objects with `package` and `reason` fields:
|
||||
|
||||
```json
|
||||
[
|
||||
{"package": "RInno", "reason": "windows-only"},
|
||||
{"package": "KeyboardSimulator", "reason": "windows-only"},
|
||||
{"package": "doBy", "reason": "hang"},
|
||||
{"package": "frailtypack", "reason": "hang"},
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
This file is the single source of truth for packages that should be skipped.
|
||||
Both the audit script and rebuild workflows read from it.
|
||||
|
||||
The existing `build-all-versions-*.yaml` workflows retain their inline lists for now (migration is out of scope).
|
||||
|
||||
## Component 2: Audit R Script
|
||||
|
||||
**File:** `local/weekly-missing-binaries-audit.R`
|
||||
|
||||
**Environment variables consumed:**
|
||||
- `PLATFORM` — e.g. `ubuntu-2204`, `alpine-321`, `redhat-9`
|
||||
- `ARCH` — `amd64` or `arm64`
|
||||
- `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY` — S3 credentials
|
||||
- `PGPASS` — PostgreSQL password
|
||||
- `FORGEJO_TOKEN` — API token for issue updates
|
||||
|
||||
The workflow commands (not the R script) also use:
|
||||
- `REPO_RO_TOKEN` — for `git clone` in the workflow commands
|
||||
|
||||
**Logic:**
|
||||
|
||||
1. Parse `PLATFORM` to derive OS family (`Ubuntu`, `Alpine`, `Red Hat`) and S3 codename (e.g. `ubuntu-2204` -> `jammy`, `redhat-9` -> `rhel9`).
|
||||
2. Fetch CRAN release packages via `tools::CRAN_package_db()` — extract `Package` and `Version`.
|
||||
3. List S3 tarballs at `devxy-rpkgs-binaries/{arch}/{codename}/latest/src/contrib/` and parse `{name}_{version}.tar.gz`.
|
||||
4. Find packages where the CRAN release version is missing from S3.
|
||||
5. Read `local/excluded-packages.json` and remove those packages from the missing list.
|
||||
6. Query the `single_builds` DB table: for each missing package+version+platform+arch, check if `error_occurred = TRUE`. Split into:
|
||||
- **Rebuildable:** missing, not excluded, no prior failure for this version
|
||||
- **Known failures:** missing, not excluded, but has a recorded failure for this version
|
||||
7. Write the rebuildable package list (names only) to `/mnt/cache/packages/weekly_rebuild_{platform}_{arch}.rds`.
|
||||
8. Update the Forgejo issue for this OS family.
|
||||
|
||||
**Issue update logic:**
|
||||
|
||||
- Issue title: `Missing package binaries for latest version (<OS family>)` where OS family is `Ubuntu`, `Alpine`, or `Red Hat`.
|
||||
- Search for existing open issue via `GET /api/v1/repos/devxy/build-cran-binaries/issues?type=issues&state=open&q=<url-encoded title>`. Match by exact title.
|
||||
- If found, read the existing body, replace the section for this platform/arch, and `PATCH` the issue.
|
||||
- If not found, `POST` a new issue with just this platform/arch section.
|
||||
|
||||
**Issue body format:**
|
||||
|
||||
```markdown
|
||||
_Last updated: 2026-04-11_
|
||||
|
||||
## ubuntu-2204
|
||||
|
||||
### amd64 (12 missing, 8 to rebuild)
|
||||
- ggplot2 (3.5.2)
|
||||
- dplyr (1.1.5)
|
||||
- ...
|
||||
|
||||
#### Known build failures
|
||||
- somepkg (1.0.0)
|
||||
|
||||
### arm64 (5 missing, 5 to rebuild)
|
||||
- ...
|
||||
|
||||
## ubuntu-2404
|
||||
|
||||
### amd64 (3 missing, 3 to rebuild)
|
||||
- ...
|
||||
|
||||
### arm64 (0 missing)
|
||||
All binaries available.
|
||||
|
||||
---
|
||||
|
||||
## Excluded packages
|
||||
doBy (hang), frailtypack (hang), RInno (windows-only), ...
|
||||
```
|
||||
|
||||
Each audit workflow run updates only its own platform/arch section within the issue.
|
||||
The "Excluded packages" section and "Last updated" timestamp are rewritten on every run.
|
||||
|
||||
**Section replacement strategy:**
|
||||
The script parses the existing issue body as markdown, finds the `## {platform}` + `### {arch}` section, replaces it, and writes back the full body.
|
||||
If the section doesn't exist yet, it's appended under the correct `## {platform}` header (or a new one is created).
|
||||
|
||||
## Component 3: Audit Workflows
|
||||
|
||||
**Files:** 14 files, named `weekly-audit-missing-{platform}-{arch}.yaml`
|
||||
|
||||
Example: `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`
|
||||
|
||||
**Trigger:**
|
||||
```yaml
|
||||
when:
|
||||
- event: cron
|
||||
cron: weekly-audit-missing-ubuntu-2204-amd64
|
||||
- event: manual
|
||||
evaluate: 'task == "weekly-audit-missing-ubuntu-2204-amd64"'
|
||||
```
|
||||
|
||||
**Step:** Lightweight — clones repo, installs bincraft + dependencies, runs the audit R script.
|
||||
|
||||
**Container:** Uses the platform-appropriate build image (needed for correct platform identification), but with minimal resource requests since no building happens.
|
||||
|
||||
**Resources:** ~2Gi memory, 2 CPUs.
|
||||
|
||||
## Component 4: Rebuild Workflows
|
||||
|
||||
**Files:** 14 files, named `weekly-rebuild-missing-{platform}-{arch}.yaml`
|
||||
|
||||
Example: `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`
|
||||
|
||||
**Trigger:**
|
||||
```yaml
|
||||
when:
|
||||
- event: cron
|
||||
cron: weekly-rebuild-missing-ubuntu-2204-amd64
|
||||
- event: manual
|
||||
evaluate: 'task == "weekly-rebuild-missing-ubuntu-2204-amd64"'
|
||||
```
|
||||
|
||||
**Step:**
|
||||
|
||||
1. Clone repo, install bincraft.
|
||||
2. Read `local/excluded-packages.json` as a safety net.
|
||||
3. Read `/mnt/cache/packages/weekly_rebuild_{platform}_{arch}.rds`. If missing or empty, exit 0.
|
||||
4. Filter out excluded packages (double-check).
|
||||
5. Iterate and call `bincraft::build_binary_package()` for each package with `tag_limit = 1L`.
|
||||
Same S3/DB parameters as existing build workflows.
|
||||
6. Uses `xvfb-run` / `xwfb-run` for graphical packages (same pattern as existing builds).
|
||||
|
||||
**Resources:** Same as `process-updates` workflows — 5Gi request, 18Gi limit, 3 CPUs.
|
||||
|
||||
**Cache volume:** Maps `${ARCH}-binaries-r-dep-cache-${PLATFORM}:/mnt/cache` (same volumes as existing builds, so the RDS files written by audit are visible).
|
||||
|
||||
## Secrets Required
|
||||
|
||||
All existing secrets are reused:
|
||||
- `B2_S3_ACCESS_KEY`, `B2_S3_SECRET_KEY` — S3 access
|
||||
- `PGPASS` — PostgreSQL
|
||||
- `REPO_RO_TOKEN` — Git clone
|
||||
- `GITHUB_PAT` — For bincraft GitHub mirror access
|
||||
|
||||
New secret needed:
|
||||
- `FORGEJO_TOKEN` — API token for creating/updating issues on git.devxy.io
|
||||
|
||||
## Cron Schedule
|
||||
|
||||
The audit and rebuild workflows each get their own cron names.
|
||||
The cron schedule itself is configured in the Crow/Woodpecker server, not in the YAML.
|
||||
Intended cadence: once per week (e.g. Sunday morning).
|
||||
|
||||
## File Inventory
|
||||
|
||||
| File | Type | Description |
|
||||
|------|------|-------------|
|
||||
| `local/excluded-packages.json` | Config | Excluded packages with reasons |
|
||||
| `local/weekly-missing-binaries-audit.R` | R script | Audit logic, parameterized by env vars |
|
||||
| `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml` | Workflow | Audit for ubuntu-2204/amd64 |
|
||||
| `.crow/weekly-audit-missing-ubuntu-2204-arm64.yaml` | Workflow | Audit for ubuntu-2204/arm64 |
|
||||
| `.crow/weekly-audit-missing-ubuntu-2404-amd64.yaml` | Workflow | Audit for ubuntu-2404/amd64 |
|
||||
| `.crow/weekly-audit-missing-ubuntu-2404-arm64.yaml` | Workflow | Audit for ubuntu-2404/arm64 |
|
||||
| `.crow/weekly-audit-missing-alpine-321-amd64.yaml` | Workflow | Audit for alpine-321/amd64 |
|
||||
| `.crow/weekly-audit-missing-alpine-321-arm64.yaml` | Workflow | Audit for alpine-321/arm64 |
|
||||
| `.crow/weekly-audit-missing-alpine-322-amd64.yaml` | Workflow | Audit for alpine-322/amd64 |
|
||||
| `.crow/weekly-audit-missing-alpine-322-arm64.yaml` | Workflow | Audit for alpine-322/arm64 |
|
||||
| `.crow/weekly-audit-missing-alpine-323-amd64.yaml` | Workflow | Audit for alpine-323/amd64 |
|
||||
| `.crow/weekly-audit-missing-alpine-323-arm64.yaml` | Workflow | Audit for alpine-323/arm64 |
|
||||
| `.crow/weekly-audit-missing-redhat-8-amd64.yaml` | Workflow | Audit for redhat-8/amd64 |
|
||||
| `.crow/weekly-audit-missing-redhat-8-arm64.yaml` | Workflow | Audit for redhat-8/arm64 |
|
||||
| `.crow/weekly-audit-missing-redhat-9-amd64.yaml` | Workflow | Audit for redhat-9/amd64 |
|
||||
| `.crow/weekly-audit-missing-redhat-9-arm64.yaml` | Workflow | Audit for redhat-9/arm64 |
|
||||
| `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml` | Workflow | Rebuild for ubuntu-2204/amd64 |
|
||||
| `.crow/weekly-rebuild-missing-ubuntu-2204-arm64.yaml` | Workflow | Rebuild for ubuntu-2204/arm64 |
|
||||
| `.crow/weekly-rebuild-missing-ubuntu-2404-amd64.yaml` | Workflow | Rebuild for ubuntu-2404/amd64 |
|
||||
| `.crow/weekly-rebuild-missing-ubuntu-2404-arm64.yaml` | Workflow | Rebuild for ubuntu-2404/arm64 |
|
||||
| `.crow/weekly-rebuild-missing-alpine-321-amd64.yaml` | Workflow | Rebuild for alpine-321/amd64 |
|
||||
| `.crow/weekly-rebuild-missing-alpine-321-arm64.yaml` | Workflow | Rebuild for alpine-321/arm64 |
|
||||
| `.crow/weekly-rebuild-missing-alpine-322-amd64.yaml` | Workflow | Rebuild for alpine-322/amd64 |
|
||||
| `.crow/weekly-rebuild-missing-alpine-322-arm64.yaml` | Workflow | Rebuild for alpine-322/arm64 |
|
||||
| `.crow/weekly-rebuild-missing-alpine-323-amd64.yaml` | Workflow | Rebuild for alpine-323/amd64 |
|
||||
| `.crow/weekly-rebuild-missing-alpine-323-arm64.yaml` | Workflow | Rebuild for alpine-323/arm64 |
|
||||
| `.crow/weekly-rebuild-missing-redhat-8-amd64.yaml` | Workflow | Rebuild for redhat-8/amd64 |
|
||||
| `.crow/weekly-rebuild-missing-redhat-8-arm64.yaml` | Workflow | Rebuild for redhat-8/arm64 |
|
||||
| `.crow/weekly-rebuild-missing-redhat-9-amd64.yaml` | Workflow | Rebuild for redhat-9/amd64 |
|
||||
| `.crow/weekly-rebuild-missing-redhat-9-arm64.yaml` | Workflow | Rebuild for redhat-9/arm64 |
|
||||
|
||||
**Total: 29 new files** (1 JSON config + 1 R script + 14 audit workflows + 14 rebuild workflows + the design doc itself)
|
||||
|
|
@ -1,242 +0,0 @@
|
|||
# Refactor build workflows to multi-R-version images
|
||||
|
||||
## Goal
|
||||
|
||||
The container images at `reg.devxy.io/rpkgs/build-env-*` are moving from a one-R-version-per-tag model to a multi-R-version-per-tag model.
|
||||
The image tag now encodes only the OS version (e.g. `build-env-alpine:3.23`), and each image ships several R installs under `/opt/R/<patch>/`.
|
||||
Workflows and recipes must select an R version explicitly by calling `/opt/R/${R_VERSION}/bin/R` instead of relying on `R`/`Rscript` from `PATH`.
|
||||
|
||||
## Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- Every `.crow/*.yaml` workflow that references a `build-env-*` image (64 files).
|
||||
- `Justfile` recipes that run `docker run` against a `build-env-*` image (3 recipes).
|
||||
- The commented-out `build-all-versions-install-deps.yaml` in the repo root (kept consistent so the example doesn't go stale).
|
||||
- Three correctness bug fixes that the user asked to roll into the same change:
|
||||
- The six `weekly-audit-missing-alpine-{321,322,323}-{amd64,arm64}.yaml` files all incorrectly use `alpine:3.23-4.5`; each should use its own alpine image.
|
||||
- The fourteen `update-package-index-*.yaml` files all use `build-env-ubuntu:noble-4.4` regardless of the platform they index; each should use its own platform's image.
|
||||
- `process-updates-ubuntu-2404-{amd64,arm64}.yaml` use `noble-4.4` while the audit and rebuild counterparts use `noble-4.4.3`; align to 4.4.3.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- `local/build-all.R` and other R scripts run *inside* a container with `Rscript`. Once R is launched, child processes inherit `R.home()`; the scripts themselves need no change.
|
||||
- `docker/`, `benchmark/`.
|
||||
- Historical docs in `docs/superpowers/plans/` and `docs/superpowers/specs/` that reference old image tags.
|
||||
- Any workflow restructuring beyond image and R-path changes plus the three bug fixes above.
|
||||
|
||||
## Image and R-path scheme
|
||||
|
||||
New image tag:
|
||||
|
||||
```
|
||||
reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}
|
||||
```
|
||||
|
||||
`R_VERSION` is no longer encoded in the tag.
|
||||
Each image contains R installs under `/opt/R/<patch>/`, accessed via:
|
||||
|
||||
- `/opt/R/${R_VERSION}/bin/R`
|
||||
- `/opt/R/${R_VERSION}/bin/Rscript`
|
||||
|
||||
`R_VERSION` is always a full patch string (e.g. `4.5.3`, `4.4.3`), never a minor (`4.5`).
|
||||
|
||||
## Platform → image + R_VERSION mapping
|
||||
|
||||
| Platform | New image | `R_VERSION` |
|
||||
|--------------|-------------------------------------------------|-------------|
|
||||
| alpine-322 | `reg.devxy.io/rpkgs/build-env-alpine:3.22` | 4.5.3 |
|
||||
| alpine-323 | `reg.devxy.io/rpkgs/build-env-alpine:3.23` | 4.5.3 |
|
||||
| ubuntu-2204 | `reg.devxy.io/rpkgs/build-env-ubuntu:jammy` | 4.4.3 |
|
||||
| ubuntu-2404 | `reg.devxy.io/rpkgs/build-env-ubuntu:noble` | 4.4.3 |
|
||||
| redhat-8 | `reg.devxy.io/rpkgs/build-env-redhat:8` | 4.4.3 |
|
||||
| redhat-9 | `reg.devxy.io/rpkgs/build-env-redhat:9` | 4.4.3 |
|
||||
| redhat-10 | `reg.devxy.io/rpkgs/build-env-redhat:10` | 4.5.3 |
|
||||
|
||||
The `alpine-321` platform has no matching new image; its two audit-only workflows fall back to `build-env-alpine:3.23` with `R_VERSION=4.5.3` (rationale in the "Edge cases" section).
|
||||
|
||||
## How workflows reference R
|
||||
|
||||
Two patterns appear in the repo today:
|
||||
|
||||
1. **Hard-coded image, no `R_VERSION` env var.** The R version is implicit in the image tag.
|
||||
2. **Parameterised image via matrix/`--var`.** `R_VERSION` is already an environment variable; the image tag interpolates `${R_VERSION}`.
|
||||
|
||||
After the refactor:
|
||||
|
||||
- Pattern (1) workflows gain a single `R_VERSION:` entry in their `environment:` block. All `R …` and `Rscript …` invocations in the `commands:` block become `/opt/R/${R_VERSION}/bin/R …` / `/opt/R/${R_VERSION}/bin/Rscript …`.
|
||||
- Pattern (2) workflows keep their existing `R_VERSION` value source (caller-supplied `--var`); only the image tag and the R invocations change.
|
||||
|
||||
No `PATH` munging, no wrapper script, no shell aliasing.
|
||||
Every R call site is explicit about which R is invoked.
|
||||
|
||||
### Example: pattern (1) before → after
|
||||
|
||||
Before (excerpt from `process-updates-alpine-322-amd64.yaml`):
|
||||
|
||||
```yaml
|
||||
- name: 'Processing Updates'
|
||||
image: reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5
|
||||
environment:
|
||||
PLATFORM: alpine-322
|
||||
ARCH: amd64
|
||||
# ...
|
||||
commands:
|
||||
- R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
|
||||
- R -q -e 'packageVersion("bincraft")'
|
||||
- xvfb-run R -q -e "..."
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```yaml
|
||||
- name: 'Processing Updates'
|
||||
image: reg.devxy.io/rpkgs/build-env-alpine:3.22
|
||||
environment:
|
||||
PLATFORM: alpine-322
|
||||
ARCH: amd64
|
||||
R_VERSION: 4.5.3
|
||||
# ...
|
||||
commands:
|
||||
- /opt/R/${R_VERSION}/bin/R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
|
||||
- /opt/R/${R_VERSION}/bin/R -q -e 'packageVersion("bincraft")'
|
||||
- xvfb-run /opt/R/${R_VERSION}/bin/R -q -e "..."
|
||||
```
|
||||
|
||||
### Example: pattern (2) before → after
|
||||
|
||||
Before (excerpt from `build-all-versions-amd64.yaml`):
|
||||
|
||||
```yaml
|
||||
- name: 'Build binaries'
|
||||
image: reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}-${R_VERSION}
|
||||
commands:
|
||||
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1
|
||||
- R -q -e "bincraft::process_unarchived_pkgs(...)"
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```yaml
|
||||
- name: 'Build binaries'
|
||||
image: reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}
|
||||
commands:
|
||||
- $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/${R_VERSION}/bin/Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1
|
||||
- /opt/R/${R_VERSION}/bin/R -q -e "bincraft::process_unarchived_pkgs(...)"
|
||||
```
|
||||
|
||||
`R_VERSION` (e.g. `4.5.3`) is already supplied by the `crow pipeline create --var` invocations documented in the file header.
|
||||
|
||||
## Files touched
|
||||
|
||||
### A. `.crow/build-all-versions-*.yaml` (4 files, pattern 2)
|
||||
|
||||
- `.crow/build-all-versions-amd64.yaml`
|
||||
- `.crow/build-all-versions-arm64.yaml`
|
||||
- `.crow/build-all-versions-install-deps-amd64.yaml`
|
||||
- `.crow/build-all-versions-install-deps-arm64.yaml`
|
||||
|
||||
Change: drop `-${R_VERSION}` from the image tag; substitute the explicit R path in every `R`/`Rscript` invocation.
|
||||
`R_VERSION` already arrives via `--var`.
|
||||
|
||||
### B. `.crow/process-updates-*.yaml` (14 files, pattern 1)
|
||||
|
||||
- `process-updates-alpine-322-{amd64,arm64}.yaml`
|
||||
- `process-updates-alpine-323-{amd64,arm64}.yaml`
|
||||
- `process-updates-ubuntu-2204-{amd64,arm64}.yaml`
|
||||
- `process-updates-ubuntu-2404-{amd64,arm64}.yaml`
|
||||
- `process-updates-redhat-8-{amd64,arm64}.yaml`
|
||||
- `process-updates-redhat-9-{amd64,arm64}.yaml`
|
||||
- `process-updates-redhat-10-{amd64,arm64}.yaml`
|
||||
|
||||
Change: image swap per mapping table; add `R_VERSION:` env var; substitute R path in every `R`/`Rscript`/`xvfb-run R` invocation.
|
||||
The two `ubuntu-2404` files also bump from `4.4` to `4.4.3` (bug fix; see "Edge cases").
|
||||
|
||||
### C. `.crow/weekly-rebuild-missing-*.yaml` (14 files, pattern 1)
|
||||
|
||||
One per platform/arch listed in the mapping table. Same treatment as B.
|
||||
|
||||
### D. `.crow/weekly-audit-missing-*.yaml` (16 files, pattern 1)
|
||||
|
||||
Same treatment as B, *plus* repointing each alpine audit file to its own alpine image:
|
||||
|
||||
| File | New image | `R_VERSION` |
|
||||
|-------------------------------------------------|--------------------------------------------|-------------|
|
||||
| `weekly-audit-missing-alpine-321-amd64.yaml` | `build-env-alpine:3.23` (no 3.21 image) | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-321-arm64.yaml` | `build-env-alpine:3.23` (no 3.21 image) | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-322-amd64.yaml` | `build-env-alpine:3.22` | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-322-arm64.yaml` | `build-env-alpine:3.22` | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-323-amd64.yaml` | `build-env-alpine:3.23` | 4.5.3 |
|
||||
| `weekly-audit-missing-alpine-323-arm64.yaml` | `build-env-alpine:3.23` | 4.5.3 |
|
||||
|
||||
The non-alpine audit files follow the mapping table directly.
|
||||
|
||||
### E. `.crow/update-package-index-*.yaml` (14 files, pattern 1)
|
||||
|
||||
Each currently uses `build-env-ubuntu:noble-4.4` regardless of which platform's package index it uploads. Repoint each to its own platform's image and R_VERSION per the mapping table.
|
||||
|
||||
Files:
|
||||
|
||||
- `update-package-index-alpine-322-{amd64,arm64}.yaml`
|
||||
- `update-package-index-alpine-323-{amd64,arm64}.yaml`
|
||||
- `update-package-index-ubuntu-2204-{amd64,arm64}.yaml`
|
||||
- `update-package-index-ubuntu-2404-{amd64,arm64}.yaml`
|
||||
- `update-package-index-redhat-8-{amd64,arm64}.yaml`
|
||||
- `update-package-index-redhat-9-{amd64,arm64}.yaml`
|
||||
- `update-package-index-redhat-10-{amd64,arm64}.yaml`
|
||||
|
||||
The second step in each (`Purge CDN cache`) runs on `alpine:3.23` and does not invoke R; it is unchanged.
|
||||
|
||||
### F. `.crow/archive-missed-packages.yaml` (1 file)
|
||||
|
||||
Currently uses `build-env-alpine:3.23-4.5`. The image OS doesn't matter for this workflow (it only writes to S3 + Postgres). New: `build-env-alpine:3.23` + `R_VERSION: 4.5.3`. Same R-path substitution as elsewhere.
|
||||
|
||||
### G. `.crow/build-r-minor-sensitive-packages.yaml` (1 file)
|
||||
|
||||
Special case: uses `docker.io/devxygmbh/rpkgs-build-env-${os}:${os_version}-${r_version}` (lowercase matrix vars; different registry).
|
||||
|
||||
Decision (user-confirmed): keep the `docker.io/devxygmbh/` registry. Drop the `-${r_version}` suffix from the image tag, leaving `docker.io/devxygmbh/rpkgs-build-env-${os}:${os_version}`. Substitute every `R`/`Rscript` for `/opt/R/${r_version}/bin/R` / `/opt/R/${r_version}/bin/Rscript`.
|
||||
|
||||
The workflow's matrix continues to use `r_version: 4.5` / `4.4`. To remain consistent with the rest of the refactor's "always full patch" rule, the matrix values should be updated to `4.5.3` and `4.4.3` respectively (matching alpine-321's R 4.5.3 and the historical 4.4.3 patch).
|
||||
|
||||
### H. `Justfile` (3 recipes)
|
||||
|
||||
- `build-all OS OS_VERSION ARCH R_VERSION PACKAGE NCPUS`
|
||||
- `build-single OS OS_VERSION ARCH R_VERSION PACKAGE TAG NCPUS`
|
||||
- `process-updates OS OS_VERSION ARCH R_VERSION interval`
|
||||
|
||||
Change in each: drop `-{{R_VERSION}}` from the image tag, and replace every `R `/`R -q -e` inside the `bash -c '…'` string with `/opt/R/{{R_VERSION}}/bin/R `/`/opt/R/{{R_VERSION}}/bin/R -q -e`.
|
||||
The example comments above each recipe (`# just build-all alpine 3.21 arm64 4.5.0 …`) should be updated to use a current platform/R combination (e.g. `alpine 3.22 amd64 4.5.3`).
|
||||
|
||||
### I. `build-all-versions-install-deps.yaml` (commented-out, repo root)
|
||||
|
||||
Apply the same edits as the active `.crow/build-all-versions-install-deps-*.yaml` files so the commented-out example remains a faithful template.
|
||||
|
||||
## Edge cases and bug fixes folded in
|
||||
|
||||
1. **Alpine audit images.** All six `weekly-audit-missing-alpine-{321,322,323}-{amd64,arm64}.yaml` files currently point at `alpine:3.23-4.5`. After the refactor, each one points at the image that matches its own alpine version. `alpine-321` has no matching image in the new scheme, so its two files use `build-env-alpine:3.23` (the audit workflow reads `PLATFORM` from env and queries S3/CRAN; the container's own OS does not affect correctness).
|
||||
2. **Package-index workflows.** All fourteen `update-package-index-*.yaml` files are repointed to their own platform's image, matching the rest of the per-platform workflows.
|
||||
3. **Ubuntu-2404 R version.** `process-updates-ubuntu-2404-{amd64,arm64}.yaml` move from `noble-4.4` to `build-env-ubuntu:noble` + `R_VERSION: 4.4.3`, matching the audit and rebuild counterparts.
|
||||
|
||||
## Validation
|
||||
|
||||
There is no automated test suite for workflow files in this repo. Validation is:
|
||||
|
||||
1. **Static checks per file**: after edit, grep each touched workflow for leftover bare `R `, `Rscript `, `R -q`, `R -e`, `R CMD` invocations. Any hit that is not part of a longer path (`/opt/R/…/bin/R`) is a regression.
|
||||
2. **Image tag check**: grep for `build-env-` lines and confirm no tag still contains `-${R_VERSION}`, `-4.4`, `-4.4.3`, `-4.5`, or `-4.5.3`.
|
||||
3. **Smoke runs**: trigger one workflow per shape on a feature branch and confirm green:
|
||||
- `process-updates-alpine-322-amd64.yaml`
|
||||
- `weekly-rebuild-missing-redhat-9-amd64.yaml`
|
||||
- `weekly-audit-missing-ubuntu-2204-amd64.yaml`
|
||||
- `update-package-index-redhat-10-amd64.yaml`
|
||||
- `archive-missed-packages.yaml`
|
||||
- `build-all-versions-amd64.yaml` (with its `install-deps` predecessor)
|
||||
- `build-r-minor-sensitive-packages.yaml`
|
||||
|
||||
The `Justfile` recipes are exercised by running each once locally against a current platform.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Wrong `R_VERSION` in a file**: typo in the platform→version mapping causes `/opt/R/<missing>/bin/R: not found`. Mitigated by the static grep in validation and by smoke-running one workflow per shape.
|
||||
- **`build-r-minor-sensitive-packages.yaml` assumes new images exist at `docker.io/devxygmbh/`**: if the multi-R image is only published to `reg.devxy.io/rpkgs/`, this workflow will fail on the first pull. If that turns out to be the case, switch to option (a) — repoint to `reg.devxy.io/rpkgs/` — as a follow-up.
|
||||
- **R subprocesses inside scripts**: `pak`, `future`, and similar libraries spawn child R processes via `R.home()`, which is set to the parent's install. No additional action needed.
|
||||
Loading…
Reference in a new issue