diff --git a/docs/superpowers/plans/2026-04-11-weekly-missing-binaries.md b/docs/superpowers/plans/2026-04-11-weekly-missing-binaries.md new file mode 100644 index 0000000..d7a099c --- /dev/null +++ b/docs/superpowers/plans/2026-04-11-weekly-missing-binaries.md @@ -0,0 +1,905 @@ +# Weekly Missing Binaries Audit & Rebuild — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a weekly CI workflow that audits CRAN packages missing binaries for their latest release version, reports them in Forgejo issues by OS family, and rebuilds those without a prior build failure. + +**Architecture:** Two independent workflow sets (audit + rebuild), each with one YAML per platform/arch (14 each). A shared R script does the audit logic. An excluded-packages JSON config is the shared blocklist. The audit writes per-platform RDS files consumed by rebuild workflows. + +**Tech Stack:** R (bincraft, s3fs, DBI/RPostgres, jsonlite, httr2), Crow/Woodpecker CI YAML, Forgejo API, PostgreSQL, Backblaze S3. + +--- + +## File Structure + +| File | Responsibility | +|------|---------------| +| `local/excluded-packages.json` | Excluded packages with reasons (single source of truth) | +| `local/weekly-missing-binaries-audit.R` | Audit logic: S3 check, DB check, issue update, RDS output | +| `.crow/weekly-audit-missing-{platform}-{arch}.yaml` (14 files) | Audit workflow per platform/arch | +| `.crow/weekly-rebuild-missing-{platform}-{arch}.yaml` (14 files) | Rebuild workflow per platform/arch | + +## Platform Reference + +Used throughout the plan. Each row is one workflow file pair (audit + rebuild). + +| PLATFORM env var | ARCH | S3 codename (sub("-","",PLATFORM)) | OS Family | Image | +|-----------------|------|-----------|-----------|-------| +| ubuntu-2204 | amd64 | ubuntu2204 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | +| ubuntu-2204 | arm64 | ubuntu2204 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | +| ubuntu-2404 | amd64 | ubuntu2404 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | +| ubuntu-2404 | arm64 | ubuntu2404 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | +| alpine-321 | amd64 | alpine321 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | +| alpine-321 | arm64 | alpine321 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | +| alpine-322 | amd64 | alpine322 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | +| alpine-322 | arm64 | alpine322 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | +| alpine-323 | amd64 | alpine323 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | +| alpine-323 | arm64 | alpine323 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | +| redhat-8 | amd64 | redhat8 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | +| redhat-8 | arm64 | redhat8 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | +| redhat-9 | amd64 | redhat9 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | +| redhat-9 | arm64 | redhat9 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | + +--- + +## Task 1: Create excluded-packages.json + +**Files:** +- Create: `local/excluded-packages.json` + +- [ ] **Step 1: Create the JSON file** + +Extract every package from the inline exclusion list in `.crow/build-all-versions-amd64.yaml` (line 72). The list has two categories: Windows-only packages (comment on line 69) and problematic packages (hang/OOM). Create `local/excluded-packages.json`: + +```json +[ + {"package": "RInno", "reason": "windows-only"}, + {"package": "KeyboardSimulator", "reason": "windows-only"}, + {"package": "R2PPT", "reason": "windows-only"}, + {"package": "RWinEdt", "reason": "windows-only"}, + {"package": "blatr", "reason": "windows-only"}, + {"package": "excel.link", "reason": "windows-only"}, + {"package": "spectrino", "reason": "windows-only"}, + {"package": "taskscheduleR", "reason": "windows-only"}, + {"package": "MDSGUI", "reason": "windows-only"}, + {"package": "BiplotGUI", "reason": "windows-only"}, + {"package": "R2wd", "reason": "windows-only"}, + {"package": "rFUSION", "reason": "windows-only"}, + {"package": "MediaNews", "reason": "windows-only"}, + {"package": "doBy", "reason": "hang"}, + {"package": "IDPmisc", "reason": "hang"}, + {"package": "frailtypack", "reason": "hang"}, + {"package": "afex", "reason": "hang"}, + {"package": "FrF2", "reason": "hang"}, + {"package": "DoE.base", "reason": "hang"}, + {"package": "agricolae", "reason": "hang"}, + {"package": "doFuture", "reason": "hang"}, + {"package": "fscaret", "reason": "hang"}, + {"package": "PHYLOGR", "reason": "hang"}, + {"package": "seewave", "reason": "hang"}, + {"package": "pls", "reason": "hang"}, + {"package": "relaimpo", "reason": "hang"}, + {"package": "geepack", "reason": "hang"}, + {"package": "gggenes", "reason": "hang"}, + {"package": "NPCirc", "reason": "hang"}, + {"package": "repmis", "reason": "hang"}, + {"package": "PNDSIBGE", "reason": "hang"}, + {"package": "lidR", "reason": "hang"}, + {"package": "poismf", "reason": "hang"}, + {"package": "neonstore", "reason": "hang"}, + {"package": "MachineShop", "reason": "hang"}, + {"package": "mvst", "reason": "hang"}, + {"package": "MacBehaviour", "reason": "hang"}, + {"package": "mcmcderive", "reason": "hang"}, + {"package": "RGIFT", "reason": "hang"}, + {"package": "KnowBR", "reason": "hang"}, + {"package": "netmeta", "reason": "hang"}, + {"package": "spdep", "reason": "hang"}, + {"package": "Rfast", "reason": "hang"}, + {"package": "compareGroups", "reason": "hang"}, + {"package": "ff", "reason": "hang"}, + {"package": "GsymPoint", "reason": "hang"}, + {"package": "RcppDynProg", "reason": "hang"}, + {"package": "comtradr", "reason": "hang"}, + {"package": "FD", "reason": "hang"}, + {"package": "PearsonDS", "reason": "hang"}, + {"package": "DCluster", "reason": "hang"}, + {"package": "gRc", "reason": "hang"}, + {"package": "mixlm", "reason": "hang"}, + {"package": "geospt", "reason": "hang"}, + {"package": "fdth", "reason": "hang"}, + {"package": "ffmanova", "reason": "hang"}, + {"package": "fiery", "reason": "hang"}, + {"package": "ffscrapr", "reason": "hang"}, + {"package": "cold", "reason": "hang"}, + {"package": "RcmdrPlugin.DoE", "reason": "hang"}, + {"package": "RcmdrPlugin.NMBU", "reason": "hang"}, + {"package": "RcmdrPlugin.RiskDemo", "reason": "hang"}, + {"package": "RcmdrPlugin.TeachStat", "reason": "hang"}, + {"package": "RcmdrPlugin.TeachingDemos", "reason": "hang"}, + {"package": "RcmdrPlugin.UCA", "reason": "hang"}, + {"package": "RcmdrPlugin.WorldFlora", "reason": "hang"}, + {"package": "RcmdrPlugin.aRnova", "reason": "hang"}, + {"package": "RcmdrPlugin.depthTools", "reason": "hang"}, + {"package": "RcmdrPlugin.orloca", "reason": "hang"}, + {"package": "RcmdrPlugin.sos", "reason": "hang"}, + {"package": "RcmdrPlugin.survival", "reason": "hang"}, + {"package": "RcmdrPlugin.temis", "reason": "hang"}, + {"package": "GWlasso", "reason": "hang"}, + {"package": "GWmodelVis", "reason": "hang"} +] +``` + +- [ ] **Step 2: Commit** + +```bash +git add local/excluded-packages.json +git commit -m "feat: add excluded-packages.json for weekly missing binaries workflow" +``` + +--- + +## Task 2: Create the audit R script + +**Files:** +- Create: `local/weekly-missing-binaries-audit.R` + +This is the core logic. It reads env vars `PLATFORM` and `ARCH`, checks S3 + DB, writes an RDS of rebuildable packages, and updates the Forgejo issue for the OS family. + +- [ ] **Step 1: Create the R script** + +Create `local/weekly-missing-binaries-audit.R` with the full content below: + +```r +options(error = function() { + cat("ERROR:", geterrmessage(), "\n", file = stdout()) + traceback(2) + q(status = 1) +}) + +library(DBI, quietly = TRUE) +suppressPackageStartupMessages(library(data.table)) + +platform <- Sys.getenv("PLATFORM") +arch <- Sys.getenv("ARCH") +stopifnot(nzchar(platform), nzchar(arch)) + +# S3 codename: remove hyphen from platform string (e.g. "ubuntu-2204" -> "ubuntu2204") +s3_codename <- gsub("-", "", platform) + +# OS family for issue grouping +os_family <- if (grepl("^ubuntu", platform)) { + "Ubuntu" +} else if (grepl("^alpine", platform)) { + "Alpine" +} else if (grepl("^redhat", platform)) { + "Red Hat" +} else { + stop("Unknown platform: ", platform) +} + +cat(sprintf("Audit: platform=%s, arch=%s, s3_codename=%s, os_family=%s\n", + platform, arch, s3_codename, os_family)) + +### 1. Get current CRAN release packages +cran_release <- as.data.table(tools::CRAN_package_db()[, c("Package", "Version")]) +cran_release[, Version := as.character(Version)] +cat(sprintf("CRAN release packages: %d\n", nrow(cran_release))) + +### 2. List S3 tarballs for this platform/arch +s3fs::s3_file_system( + aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), + aws_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), + endpoint = "https://s3.eu-central-003.backblazeb2.com", + region_name = "eu-central-003", + refresh = TRUE +) + +s3_path <- sprintf("devxy-rpkgs-binaries/%s/%s/latest/src/contrib", arch, s3_codename) +s3_files <- tryCatch( + s3fs::s3_dir_ls(s3_path), + error = function(e) { + cat(sprintf("Warning: could not list S3 path %s: %s\n", s3_path, conditionMessage(e))) + character(0) + } +) + +# Parse package name + version from tarball filenames +file_names <- basename(s3_files) +matches <- regexec("^([A-Za-z0-9.]+)_([0-9][^/]*)\\.tar\\.gz$", file_names) +parts <- regmatches(file_names, matches) +parts <- parts[lengths(parts) == 3] + +s3_dt <- if (length(parts) > 0) { + data.table( + Package = vapply(parts, `[`, character(1), 2), + Version = vapply(parts, `[`, character(1), 3) + ) +} else { + data.table(Package = character(0), Version = character(0)) +} +cat(sprintf("S3 packages for %s/%s: %d\n", arch, s3_codename, nrow(s3_dt))) + +### 3. Find CRAN release packages missing from S3 +setkey(cran_release, Package, Version) +setkey(s3_dt, Package, Version) +missing <- cran_release[!s3_dt] +cat(sprintf("Missing binaries (latest CRAN version): %d\n", nrow(missing))) + +### 4. Load excluded packages +excluded_json <- jsonlite::fromJSON("local/excluded-packages.json") +excluded_pkgs <- excluded_json$package +missing_not_excluded <- missing[!Package %in% excluded_pkgs] +missing_excluded <- missing[Package %in% excluded_pkgs] +cat(sprintf("Missing after excluding %d blocked packages: %d\n", + length(excluded_pkgs), nrow(missing_not_excluded))) + +### 5. Check DB for prior build failures on these exact versions +con <- DBI::dbConnect( + RPostgres::Postgres(), + dbname = "build_metadata", + host = "r-binaries.devxy.io", + port = 15432, + user = "rpkgs", + password = Sys.getenv("PGPASS"), + sslmode = "require" +) +on.exit(DBI::dbDisconnect(con), add = TRUE) + +errored_pkgs <- as.data.table(DBI::dbGetQuery( + con, + sprintf( + "SELECT name, tag FROM single_builds WHERE error_occurred = TRUE AND platform = '%s' AND arch = '%s'", + platform, arch + ) +)) +setnames(errored_pkgs, c("Package", "Version")) +setkey(errored_pkgs, Package, Version) + +# Split: rebuildable vs known failures +known_failures <- missing_not_excluded[errored_pkgs, nomatch = 0, on = c("Package", "Version")] +rebuildable <- missing_not_excluded[!errored_pkgs, on = c("Package", "Version")] + +cat(sprintf("Known build failures: %d\n", nrow(known_failures))) +cat(sprintf("Rebuildable (no prior failure): %d\n", nrow(rebuildable))) + +### 6. Write RDS for rebuild workflow +cache_dir <- "/mnt/cache/packages" +if (dir.exists(cache_dir)) { + rds_path <- file.path(cache_dir, sprintf("weekly_rebuild_%s_%s.rds", platform, arch)) + saveRDS(rebuildable$Package, rds_path) + cat(sprintf("Wrote %d packages to %s\n", nrow(rebuildable), rds_path)) +} else { + cat(sprintf("Cache dir %s does not exist, skipping RDS write\n", cache_dir)) +} + +### 7. Update Forgejo issue +forgejo_token <- Sys.getenv("FORGEJO_TOKEN") +if (!nzchar(forgejo_token)) { + cat("FORGEJO_TOKEN not set, skipping issue update\n") +} else { + base_url <- "https://git.devxy.io/api/v1" + repo <- "devxy/build-cran-binaries" + issue_title <- sprintf("Missing package binaries for latest version (%s)", os_family) + + # Helper: make API request + forgejo_get <- function(path, query = list()) { + url <- paste0(base_url, path) + resp <- httr2::request(url) |> + httr2::req_headers(Authorization = paste("token", forgejo_token)) |> + httr2::req_url_query(!!!query) |> + httr2::req_perform() + httr2::resp_body_json(resp) + } + + forgejo_patch <- function(path, body) { + url <- paste0(base_url, path) + httr2::request(url) |> + httr2::req_headers(Authorization = paste("token", forgejo_token)) |> + httr2::req_method("PATCH") |> + httr2::req_body_json(body) |> + httr2::req_perform() + } + + forgejo_post <- function(path, body) { + url <- paste0(base_url, path) + httr2::request(url) |> + httr2::req_headers(Authorization = paste("token", forgejo_token)) |> + httr2::req_body_json(body) |> + httr2::req_perform() + } + + # Build the markdown section for this platform/arch + build_section <- function() { + n_missing <- nrow(rebuildable) + nrow(known_failures) + n_rebuild <- nrow(rebuildable) + header <- sprintf("### %s (%d missing, %d to rebuild)", arch, n_missing, n_rebuild) + + lines <- header + if (nrow(rebuildable) > 0) { + pkg_lines <- sprintf("- %s (%s)", rebuildable$Package, rebuildable$Version) + lines <- c(lines, "", pkg_lines) + } else if (nrow(known_failures) == 0) { + lines <- c(lines, "", "All binaries available.") + } + + if (nrow(known_failures) > 0) { + lines <- c(lines, "", "#### Known build failures", + sprintf("- %s (%s)", known_failures$Package, known_failures$Version)) + } + + paste(lines, collapse = "\n") + } + + # Build excluded packages footer + build_excluded_footer <- function() { + if (nrow(excluded_json) == 0) return("") + items <- sprintf("%s (%s)", excluded_json$package, excluded_json$reason) + paste0("\n---\n\n## Excluded packages\n", paste(items, collapse = ", ")) + } + + new_section <- build_section() + + # Search for existing issue + issues <- forgejo_get( + sprintf("/repos/%s/issues", repo), + query = list(type = "issues", state = "open", q = issue_title, limit = 50) + ) + + # Find exact title match + existing <- Filter(function(i) i$title == issue_title, issues) + + if (length(existing) > 0) { + issue <- existing[[1]] + body <- issue$body + + # Replace or insert the platform section + arch subsection + platform_header <- sprintf("## %s", platform) + arch_header <- sprintf("### %s", arch) + + # Split body into lines for manipulation + body_lines <- strsplit(body, "\n")[[1]] + + # Find the platform section + platform_start <- which(body_lines == platform_header) + + if (length(platform_start) > 0) { + platform_start <- platform_start[1] + # Find end of this platform section (next ## or --- or end) + remaining <- body_lines[(platform_start + 1):length(body_lines)] + platform_end_offset <- which(grepl("^## |^---$", remaining)) + platform_end <- if (length(platform_end_offset) > 0) { + platform_start + platform_end_offset[1] - 1 + } else { + length(body_lines) + } + + # Within platform section, find the arch subsection + section_lines <- body_lines[platform_start:platform_end] + arch_start_offset <- which(grepl(sprintf("^### %s", arch), section_lines)) + + if (length(arch_start_offset) > 0) { + arch_start <- arch_start_offset[1] + # Find end of arch subsection (next ### or ## or --- or end of platform section) + arch_remaining <- section_lines[(arch_start + 1):length(section_lines)] + arch_end_offset <- which(grepl("^###|^## |^---$", arch_remaining)) + arch_end <- if (length(arch_end_offset) > 0) { + arch_start + arch_end_offset[1] - 1 + } else { + length(section_lines) + } + # Replace arch subsection within platform section + section_lines <- c( + section_lines[1:(arch_start - 1)], + strsplit(new_section, "\n")[[1]], + if (arch_end < length(section_lines)) section_lines[(arch_end + 1):length(section_lines)] else character(0) + ) + } else { + # Append arch subsection to end of platform section + section_lines <- c(section_lines, "", strsplit(new_section, "\n")[[1]]) + } + + body_lines <- c( + body_lines[1:(platform_start - 1)], + section_lines, + if (platform_end < length(body_lines)) body_lines[(platform_end + 1):length(body_lines)] else character(0) + ) + } else { + # Insert new platform section before "---" (excluded packages footer) or at end + footer_line <- which(body_lines == "---") + insert_at <- if (length(footer_line) > 0) footer_line[1] - 1 else length(body_lines) + body_lines <- c( + body_lines[1:insert_at], + "", + platform_header, + "", + strsplit(new_section, "\n")[[1]], + if (insert_at < length(body_lines)) body_lines[(insert_at + 1):length(body_lines)] else character(0) + ) + } + + # Update timestamp + timestamp_pattern <- "^_Last updated:.*_$" + ts_line <- which(grepl(timestamp_pattern, body_lines)) + new_ts <- sprintf("_Last updated: %s_", Sys.Date()) + if (length(ts_line) > 0) { + body_lines[ts_line[1]] <- new_ts + } else { + body_lines <- c(new_ts, "", body_lines) + } + + # Rebuild excluded footer + footer_start <- which(body_lines == "---") + if (length(footer_start) > 0) { + body_lines <- c(body_lines[1:(footer_start[1] - 1)], + strsplit(build_excluded_footer(), "\n")[[1]]) + } else { + body_lines <- c(body_lines, strsplit(build_excluded_footer(), "\n")[[1]]) + } + + new_body <- paste(body_lines, collapse = "\n") + forgejo_patch( + sprintf("/repos/%s/issues/%d", repo, issue$number), + list(body = new_body) + ) + cat(sprintf("Updated issue #%d: %s\n", issue$number, issue_title)) + } else { + # Create new issue + body_lines <- c( + sprintf("_Last updated: %s_", Sys.Date()), + "", + sprintf("## %s", platform), + "", + new_section, + build_excluded_footer() + ) + new_body <- paste(body_lines, collapse = "\n") + forgejo_post( + sprintf("/repos/%s/issues", repo), + list(title = issue_title, body = new_body) + ) + cat(sprintf("Created new issue: %s\n", issue_title)) + } +} + +cat("Audit complete.\n") +``` + +- [ ] **Step 2: Commit** + +```bash +git add local/weekly-missing-binaries-audit.R +git commit -m "feat: add weekly missing binaries audit R script" +``` + +--- + +## Task 3: Create audit workflow for ubuntu-2204-amd64 + +**Files:** +- Create: `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml` + +This is the template. All other audit workflows follow the same structure with substituted values. + +- [ ] **Step 1: Create the workflow YAML** + +Create `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`: + +```yaml +when: + - event: cron + cron: weekly-audit-missing-ubuntu-2204-amd64 + - event: manual + evaluate: 'task == "weekly-audit-missing-ubuntu-2204-amd64"' + +skip_clone: true + +steps: + - name: 'Audit missing binaries' + image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 + pull: true + environment: + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + PGPASS: + from_secret: PGPASS + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + FORGEJO_TOKEN: + from_secret: FORGEJO_TOKEN + GITHUB_PAT: + from_secret: GITHUB_PAT + PLATFORM: ubuntu-2204 + ARCH: amd64 + R_LIBS_USER: /mnt/cache/R-pkgs + volumes: + - amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache + commands: + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs + - rm -rf /mnt/cache/R-pkgs/00LOCK-* + - R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))' + - R -q -e 'source("local/weekly-missing-binaries-audit.R")' + backend_options: + kubernetes: + resources: + requests: + memory: 2Gi + cpu: 2000m + limits: + memory: 4Gi + cpu: 2000m + nodeSelector: + kubernetes.io/arch: amd64 + tolerations: + - key: 'CI' + operator: 'Equal' + value: 'true' + effect: 'NoSchedule' +``` + +- [ ] **Step 2: Commit** + +```bash +git add .crow/weekly-audit-missing-ubuntu-2204-amd64.yaml +git commit -m "feat: add weekly audit workflow for ubuntu-2204-amd64" +``` + +--- + +## Task 4: Create remaining 13 audit workflows + +**Files:** +- Create: 13 files in `.crow/` (see substitution table below) + +Each file follows the exact same structure as Task 3 with these substitutions: + +| File suffix | PLATFORM | ARCH | Image | Volume | +|-------------|----------|------|-------|--------| +| ubuntu-2204-arm64 | ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2204 | +| ubuntu-2404-amd64 | ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | amd64-binaries-r-dep-cache-ubuntu2404 | +| ubuntu-2404-arm64 | ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2404 | +| alpine-321-amd64 | alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | amd64-binaries-r-dep-cache-alpine321 | +| alpine-321-arm64 | alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | arm64-binaries-r-dep-cache-alpine321 | +| alpine-322-amd64 | alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | amd64-binaries-r-dep-cache-alpine322 | +| alpine-322-arm64 | alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | arm64-binaries-r-dep-cache-alpine322 | +| alpine-323-amd64 | alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | amd64-binaries-r-dep-cache-alpine323 | +| alpine-323-arm64 | alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | arm64-binaries-r-dep-cache-alpine323 | +| redhat-8-amd64 | redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | amd64-binaries-r-dep-cache-redhat8 | +| redhat-8-arm64 | redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | arm64-binaries-r-dep-cache-redhat8 | +| redhat-9-amd64 | redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | amd64-binaries-r-dep-cache-redhat9 | +| redhat-9-arm64 | redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | arm64-binaries-r-dep-cache-redhat9 | + +**Key substitution points in each YAML (6 locations):** + +1. `cron:` value — `weekly-audit-missing-{suffix}` +2. `evaluate:` value — `'task == "weekly-audit-missing-{suffix}"'` +3. `image:` — use the Image column +4. `PLATFORM:` env var — use the PLATFORM column +5. `ARCH:` env var — use the ARCH column +6. `volumes:` — `{ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache` +7. `nodeSelector: kubernetes.io/arch:` — use the ARCH column + +- [ ] **Step 1: Create all 13 audit workflow files** + +Copy the template from Task 3 and substitute the values per the table above. Example for alpine-321-arm64: + +```yaml +when: + - event: cron + cron: weekly-audit-missing-alpine-321-arm64 + - event: manual + evaluate: 'task == "weekly-audit-missing-alpine-321-arm64"' + +skip_clone: true + +steps: + - name: 'Audit missing binaries' + image: reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 + pull: true + environment: + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + PGPASS: + from_secret: PGPASS + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + FORGEJO_TOKEN: + from_secret: FORGEJO_TOKEN + GITHUB_PAT: + from_secret: GITHUB_PAT + PLATFORM: alpine-321 + ARCH: arm64 + R_LIBS_USER: /mnt/cache/R-pkgs + volumes: + - arm64-binaries-r-dep-cache-alpine321:/mnt/cache + commands: + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs + - rm -rf /mnt/cache/R-pkgs/00LOCK-* + - R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))' + - R -q -e 'source("local/weekly-missing-binaries-audit.R")' + backend_options: + kubernetes: + resources: + requests: + memory: 2Gi + cpu: 2000m + limits: + memory: 4Gi + cpu: 2000m + nodeSelector: + kubernetes.io/arch: arm64 + tolerations: + - key: 'CI' + operator: 'Equal' + value: 'true' + effect: 'NoSchedule' +``` + +Repeat for all 13 remaining suffixes from the table, substituting the 7 locations. + +- [ ] **Step 2: Commit** + +```bash +git add .crow/weekly-audit-missing-*.yaml +git commit -m "feat: add remaining 13 weekly audit workflows for all platform/arch combos" +``` + +--- + +## Task 5: Create rebuild workflow for ubuntu-2204-amd64 + +**Files:** +- Create: `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml` + +This is the template for rebuild workflows. It reads the RDS written by the audit and builds each package. + +- [ ] **Step 1: Create the workflow YAML** + +Create `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`: + +```yaml +when: + - event: cron + cron: weekly-rebuild-missing-ubuntu-2204-amd64 + - event: manual + evaluate: 'task == "weekly-rebuild-missing-ubuntu-2204-amd64"' + +skip_clone: true + +steps: + - name: 'Rebuild missing binaries' + image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 + pull: true + environment: + RED_HAT_DEV_PW: + from_secret: RED_HAT_DEV_PW + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + PGPASS: + from_secret: PGPASS + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + GITHUB_PAT: + from_secret: GITHUB_PAT + GIT_USER: pat-s + R_PKG_CACHE_DIR: /mnt/cache/pkgcache + R_LIBS_USER: /mnt/cache/R-pkgs + CCACHE_DIR: /mnt/cache/ccache + PLATFORM: ubuntu-2204 + ARCH: amd64 + NCPUS: 2 + volumes: + - amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache + commands: + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages + - rm -rf /mnt/cache/R-pkgs/00LOCK-* + - R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")' + - R -q -e 'packageVersion("bincraft")' + - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi + - $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_ubuntu-2204_amd64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1 + backend_options: + kubernetes: + resources: + requests: + memory: 5Gi + cpu: 3000m + limits: + memory: 18Gi + cpu: 3000m + nodeSelector: + kubernetes.io/arch: amd64 + node.kubernetes.io/instance-type: AX42 + tolerations: + - key: 'CI' + operator: 'Equal' + value: 'true' + effect: 'NoSchedule' +``` + +- [ ] **Step 2: Commit** + +```bash +git add .crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml +git commit -m "feat: add weekly rebuild workflow for ubuntu-2204-amd64" +``` + +--- + +## Task 6: Create remaining 13 rebuild workflows + +**Files:** +- Create: 13 files in `.crow/` (see substitution table below) + +Each file follows the exact same structure as Task 5 with these substitutions: + +| File suffix | PLATFORM | ARCH | Image | Volume | nodeSelector instance-type | +|-------------|----------|------|-------|--------|--------------------------| +| ubuntu-2204-arm64 | ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2204 | (omit) | +| ubuntu-2404-amd64 | ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | amd64-binaries-r-dep-cache-ubuntu2404 | AX42 | +| ubuntu-2404-arm64 | ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2404 | (omit) | +| alpine-321-amd64 | alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | amd64-binaries-r-dep-cache-alpine321 | AX42 | +| alpine-321-arm64 | alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | arm64-binaries-r-dep-cache-alpine321 | (omit) | +| alpine-322-amd64 | alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | amd64-binaries-r-dep-cache-alpine322 | AX42 | +| alpine-322-arm64 | alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | arm64-binaries-r-dep-cache-alpine322 | (omit) | +| alpine-323-amd64 | alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | amd64-binaries-r-dep-cache-alpine323 | AX42 | +| alpine-323-arm64 | alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | arm64-binaries-r-dep-cache-alpine323 | (omit) | +| redhat-8-amd64 | redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | amd64-binaries-r-dep-cache-redhat8 | AX42 | +| redhat-8-arm64 | redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | arm64-binaries-r-dep-cache-redhat8 | (omit) | +| redhat-9-amd64 | redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | amd64-binaries-r-dep-cache-redhat9 | AX42 | +| redhat-9-arm64 | redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | arm64-binaries-r-dep-cache-redhat9 | (omit) | + +**Key substitution points (8 locations):** + +1. `cron:` value — `weekly-rebuild-missing-{suffix}` +2. `evaluate:` value — `'task == "weekly-rebuild-missing-{suffix}"'` +3. `image:` — use the Image column +4. `PLATFORM:` env var — use the PLATFORM column +5. `ARCH:` env var — use the ARCH column +6. `volumes:` — `{ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache` +7. `rds_path` in the R command — `weekly_rebuild_{PLATFORM}_{ARCH}.rds` +8. `nodeSelector` — `kubernetes.io/arch: {ARCH}`, plus `node.kubernetes.io/instance-type: AX42` only for amd64 (omit the instance-type line entirely for arm64) + +- [ ] **Step 1: Create all 13 rebuild workflow files** + +Copy the template from Task 5 and substitute per the table. Example for redhat-9-arm64: + +```yaml +when: + - event: cron + cron: weekly-rebuild-missing-redhat-9-arm64 + - event: manual + evaluate: 'task == "weekly-rebuild-missing-redhat-9-arm64"' + +skip_clone: true + +steps: + - name: 'Rebuild missing binaries' + image: reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 + pull: true + environment: + RED_HAT_DEV_PW: + from_secret: RED_HAT_DEV_PW + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + PGPASS: + from_secret: PGPASS + REPO_RO_TOKEN: + from_secret: REPO_RO_TOKEN + GITHUB_PAT: + from_secret: GITHUB_PAT + GIT_USER: pat-s + R_PKG_CACHE_DIR: /mnt/cache/pkgcache + R_LIBS_USER: /mnt/cache/R-pkgs + CCACHE_DIR: /mnt/cache/ccache + PLATFORM: redhat-9 + ARCH: arm64 + NCPUS: 2 + volumes: + - arm64-binaries-r-dep-cache-redhat9:/mnt/cache + commands: + - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . + - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages + - rm -rf /mnt/cache/R-pkgs/00LOCK-* + - R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")' + - R -q -e 'packageVersion("bincraft")' + - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi + - $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_redhat-9_arm64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1 + backend_options: + kubernetes: + resources: + requests: + memory: 5Gi + cpu: 3000m + limits: + memory: 18Gi + cpu: 3000m + nodeSelector: + kubernetes.io/arch: arm64 + tolerations: + - key: 'CI' + operator: 'Equal' + value: 'true' + effect: 'NoSchedule' +``` + +Note: arm64 workflows do NOT include `node.kubernetes.io/instance-type: AX42` in nodeSelector. + +Repeat for all 13 remaining suffixes from the table, substituting the 8 locations. + +- [ ] **Step 2: Commit** + +```bash +git add .crow/weekly-rebuild-missing-*.yaml +git commit -m "feat: add remaining 13 weekly rebuild workflows for all platform/arch combos" +``` + +--- + +## Task 7: Final verification + +- [ ] **Step 1: Verify all files exist** + +Run: +```bash +ls -1 local/excluded-packages.json local/weekly-missing-binaries-audit.R +ls -1 .crow/weekly-audit-missing-*.yaml | wc -l +ls -1 .crow/weekly-rebuild-missing-*.yaml | wc -l +``` + +Expected: both files exist, 14 audit workflows, 14 rebuild workflows. + +- [ ] **Step 2: Validate JSON** + +Run: +```bash +python3 -c "import json; json.load(open('local/excluded-packages.json')); print('JSON valid')" +``` + +Expected: `JSON valid` + +- [ ] **Step 3: Validate YAML syntax** + +Run: +```bash +python3 -c " +import yaml, glob +for f in sorted(glob.glob('.crow/weekly-*-missing-*.yaml') + glob.glob('.crow/weekly-audit-missing-*.yaml') + glob.glob('.crow/weekly-rebuild-missing-*.yaml')): + yaml.safe_load(open(f)) + print(f'OK: {f}') +" +``` + +Expected: all files print `OK`. + +- [ ] **Step 4: Verify unique cron names** + +Run: +```bash +grep -h 'cron: weekly-' .crow/weekly-*.yaml | sort | uniq -c | sort -rn | head +``` + +Expected: all counts are 1 (no duplicates). + +- [ ] **Step 5: Verify all 14 platform/arch combos covered** + +Run: +```bash +for suffix in ubuntu-2204-amd64 ubuntu-2204-arm64 ubuntu-2404-amd64 ubuntu-2404-arm64 alpine-321-amd64 alpine-321-arm64 alpine-322-amd64 alpine-322-arm64 alpine-323-amd64 alpine-323-arm64 redhat-8-amd64 redhat-8-arm64 redhat-9-amd64 redhat-9-arm64; do + test -f ".crow/weekly-audit-missing-${suffix}.yaml" || echo "MISSING audit: ${suffix}" + test -f ".crow/weekly-rebuild-missing-${suffix}.yaml" || echo "MISSING rebuild: ${suffix}" +done +echo "All checks passed if no MISSING lines above" +``` + +Expected: no MISSING lines, just "All checks passed".