# Weekly Missing Binaries Audit & Rebuild — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add a weekly CI workflow that audits CRAN packages missing binaries for their latest release version, reports them in Forgejo issues by OS family, and rebuilds those without a prior build failure.

**Architecture:** Two independent workflow sets (audit + rebuild), each with one YAML per platform/arch (14 each). A shared R script does the audit logic. An excluded-packages JSON config is the shared blocklist. The audit writes per-platform RDS files consumed by rebuild workflows.

**Tech Stack:** R (bincraft, s3fs, DBI/RPostgres, jsonlite, httr2), Crow/Woodpecker CI YAML, Forgejo API, PostgreSQL, Backblaze S3.

---

## File Structure

| File | Responsibility |
|------|---------------|
| `local/excluded-packages.json` | Excluded packages with reasons (single source of truth) |
| `local/weekly-missing-binaries-audit.R` | Audit logic: S3 check, DB check, issue update, RDS output |
| `.crow/weekly-audit-missing-{platform}-{arch}.yaml` (14 files) | Audit workflow per platform/arch |
| `.crow/weekly-rebuild-missing-{platform}-{arch}.yaml` (14 files) | Rebuild workflow per platform/arch |

## Platform Reference

Used throughout the plan. Each row is one workflow file pair (audit + rebuild).

| PLATFORM env var | ARCH | S3 codename (sub("-","",PLATFORM)) | OS Family | Image |
|-----------------|------|-----------|-----------|-------|
| ubuntu-2204 | amd64 | ubuntu2204 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 |
| ubuntu-2204 | arm64 | ubuntu2204 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 |
| ubuntu-2404 | amd64 | ubuntu2404 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 |
| ubuntu-2404 | arm64 | ubuntu2404 | Ubuntu | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 |
| alpine-321 | amd64 | alpine321 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 |
| alpine-321 | arm64 | alpine321 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 |
| alpine-322 | amd64 | alpine322 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 |
| alpine-322 | arm64 | alpine322 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 |
| alpine-323 | amd64 | alpine323 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 |
| alpine-323 | arm64 | alpine323 | Alpine | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 |
| redhat-8 | amd64 | redhat8 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 |
| redhat-8 | arm64 | redhat8 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 |
| redhat-9 | amd64 | redhat9 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 |
| redhat-9 | arm64 | redhat9 | Red Hat | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 |

---

## Task 1: Create excluded-packages.json

**Files:**
- Create: `local/excluded-packages.json`

- [ ] **Step 1: Create the JSON file**

Extract every package from the inline exclusion list in `.crow/build-all-versions-amd64.yaml` (line 72). The list has two categories: Windows-only packages (comment on line 69) and problematic packages (hang/OOM). Create `local/excluded-packages.json`:

```json
[
  {"package": "RInno", "reason": "windows-only"},
  {"package": "KeyboardSimulator", "reason": "windows-only"},
  {"package": "R2PPT", "reason": "windows-only"},
  {"package": "RWinEdt", "reason": "windows-only"},
  {"package": "blatr", "reason": "windows-only"},
  {"package": "excel.link", "reason": "windows-only"},
  {"package": "spectrino", "reason": "windows-only"},
  {"package": "taskscheduleR", "reason": "windows-only"},
  {"package": "MDSGUI", "reason": "windows-only"},
  {"package": "BiplotGUI", "reason": "windows-only"},
  {"package": "R2wd", "reason": "windows-only"},
  {"package": "rFUSION", "reason": "windows-only"},
  {"package": "MediaNews", "reason": "windows-only"},
  {"package": "doBy", "reason": "hang"},
  {"package": "IDPmisc", "reason": "hang"},
  {"package": "frailtypack", "reason": "hang"},
  {"package": "afex", "reason": "hang"},
  {"package": "FrF2", "reason": "hang"},
  {"package": "DoE.base", "reason": "hang"},
  {"package": "agricolae", "reason": "hang"},
  {"package": "doFuture", "reason": "hang"},
  {"package": "fscaret", "reason": "hang"},
  {"package": "PHYLOGR", "reason": "hang"},
  {"package": "seewave", "reason": "hang"},
  {"package": "pls", "reason": "hang"},
  {"package": "relaimpo", "reason": "hang"},
  {"package": "geepack", "reason": "hang"},
  {"package": "gggenes", "reason": "hang"},
  {"package": "NPCirc", "reason": "hang"},
  {"package": "repmis", "reason": "hang"},
  {"package": "PNDSIBGE", "reason": "hang"},
  {"package": "lidR", "reason": "hang"},
  {"package": "poismf", "reason": "hang"},
  {"package": "neonstore", "reason": "hang"},
  {"package": "MachineShop", "reason": "hang"},
  {"package": "mvst", "reason": "hang"},
  {"package": "MacBehaviour", "reason": "hang"},
  {"package": "mcmcderive", "reason": "hang"},
  {"package": "RGIFT", "reason": "hang"},
  {"package": "KnowBR", "reason": "hang"},
  {"package": "netmeta", "reason": "hang"},
  {"package": "spdep", "reason": "hang"},
  {"package": "Rfast", "reason": "hang"},
  {"package": "compareGroups", "reason": "hang"},
  {"package": "ff", "reason": "hang"},
  {"package": "GsymPoint", "reason": "hang"},
  {"package": "RcppDynProg", "reason": "hang"},
  {"package": "comtradr", "reason": "hang"},
  {"package": "FD", "reason": "hang"},
  {"package": "PearsonDS", "reason": "hang"},
  {"package": "DCluster", "reason": "hang"},
  {"package": "gRc", "reason": "hang"},
  {"package": "mixlm", "reason": "hang"},
  {"package": "geospt", "reason": "hang"},
  {"package": "fdth", "reason": "hang"},
  {"package": "ffmanova", "reason": "hang"},
  {"package": "fiery", "reason": "hang"},
  {"package": "ffscrapr", "reason": "hang"},
  {"package": "cold", "reason": "hang"},
  {"package": "RcmdrPlugin.DoE", "reason": "hang"},
  {"package": "RcmdrPlugin.NMBU", "reason": "hang"},
  {"package": "RcmdrPlugin.RiskDemo", "reason": "hang"},
  {"package": "RcmdrPlugin.TeachStat", "reason": "hang"},
  {"package": "RcmdrPlugin.TeachingDemos", "reason": "hang"},
  {"package": "RcmdrPlugin.UCA", "reason": "hang"},
  {"package": "RcmdrPlugin.WorldFlora", "reason": "hang"},
  {"package": "RcmdrPlugin.aRnova", "reason": "hang"},
  {"package": "RcmdrPlugin.depthTools", "reason": "hang"},
  {"package": "RcmdrPlugin.orloca", "reason": "hang"},
  {"package": "RcmdrPlugin.sos", "reason": "hang"},
  {"package": "RcmdrPlugin.survival", "reason": "hang"},
  {"package": "RcmdrPlugin.temis", "reason": "hang"},
  {"package": "GWlasso", "reason": "hang"},
  {"package": "GWmodelVis", "reason": "hang"}
]
```

- [ ] **Step 2: Commit**

```bash
git add local/excluded-packages.json
git commit -m "feat: add excluded-packages.json for weekly missing binaries workflow"
```

---

## Task 2: Create the audit R script

**Files:**
- Create: `local/weekly-missing-binaries-audit.R`

This is the core logic. It reads env vars `PLATFORM` and `ARCH`, checks S3 + DB, writes an RDS of rebuildable packages, and updates the Forgejo issue for the OS family.

- [ ] **Step 1: Create the R script**

Create `local/weekly-missing-binaries-audit.R` with the full content below:

```r
options(error = function() {
  cat("ERROR:", geterrmessage(), "\n", file = stdout())
  traceback(2)
  q(status = 1)
})

library(DBI, quietly = TRUE)
suppressPackageStartupMessages(library(data.table))

platform <- Sys.getenv("PLATFORM")
arch <- Sys.getenv("ARCH")
stopifnot(nzchar(platform), nzchar(arch))

# S3 codename: remove hyphen from platform string (e.g. "ubuntu-2204" -> "ubuntu2204")
s3_codename <- gsub("-", "", platform)

# OS family for issue grouping
os_family <- if (grepl("^ubuntu", platform)) {
  "Ubuntu"
} else if (grepl("^alpine", platform)) {
  "Alpine"
} else if (grepl("^redhat", platform)) {
  "Red Hat"
} else {
  stop("Unknown platform: ", platform)
}

cat(sprintf("Audit: platform=%s, arch=%s, s3_codename=%s, os_family=%s\n",
            platform, arch, s3_codename, os_family))

### 1. Get current CRAN release packages
cran_release <- as.data.table(tools::CRAN_package_db()[, c("Package", "Version")])
cran_release[, Version := as.character(Version)]
cat(sprintf("CRAN release packages: %d\n", nrow(cran_release)))

### 2. List S3 tarballs for this platform/arch
s3fs::s3_file_system(
  aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"),
  aws_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"),
  endpoint = "https://s3.eu-central-003.backblazeb2.com",
  region_name = "eu-central-003",
  refresh = TRUE
)

s3_path <- sprintf("devxy-rpkgs-binaries/%s/%s/latest/src/contrib", arch, s3_codename)
s3_files <- tryCatch(
  s3fs::s3_dir_ls(s3_path),
  error = function(e) {
    cat(sprintf("Warning: could not list S3 path %s: %s\n", s3_path, conditionMessage(e)))
    character(0)
  }
)

# Parse package name + version from tarball filenames
file_names <- basename(s3_files)
matches <- regexec("^([A-Za-z0-9.]+)_([0-9][^/]*)\\.tar\\.gz$", file_names)
parts <- regmatches(file_names, matches)
parts <- parts[lengths(parts) == 3]

s3_dt <- if (length(parts) > 0) {
  data.table(
    Package = vapply(parts, `[`, character(1), 2),
    Version = vapply(parts, `[`, character(1), 3)
  )
} else {
  data.table(Package = character(0), Version = character(0))
}
cat(sprintf("S3 packages for %s/%s: %d\n", arch, s3_codename, nrow(s3_dt)))

### 3. Find CRAN release packages missing from S3
setkey(cran_release, Package, Version)
setkey(s3_dt, Package, Version)
missing <- cran_release[!s3_dt]
cat(sprintf("Missing binaries (latest CRAN version): %d\n", nrow(missing)))

### 4. Load excluded packages
excluded_json <- jsonlite::fromJSON("local/excluded-packages.json")
excluded_pkgs <- excluded_json$package
missing_not_excluded <- missing[!Package %in% excluded_pkgs]
missing_excluded <- missing[Package %in% excluded_pkgs]
cat(sprintf("Missing after excluding %d blocked packages: %d\n",
            length(excluded_pkgs), nrow(missing_not_excluded)))

### 5. Check DB for prior build failures on these exact versions
con <- DBI::dbConnect(
  RPostgres::Postgres(),
  dbname = "build_metadata",
  host = "r-binaries.devxy.io",
  port = 15432,
  user = "rpkgs",
  password = Sys.getenv("PGPASS"),
  sslmode = "require"
)
on.exit(DBI::dbDisconnect(con), add = TRUE)

errored_pkgs <- as.data.table(DBI::dbGetQuery(
  con,
  sprintf(
    "SELECT name, tag FROM single_builds WHERE error_occurred = TRUE AND platform = '%s' AND arch = '%s'",
    platform, arch
  )
))
setnames(errored_pkgs, c("Package", "Version"))
setkey(errored_pkgs, Package, Version)

# Split: rebuildable vs known failures
known_failures <- missing_not_excluded[errored_pkgs, nomatch = 0, on = c("Package", "Version")]
rebuildable <- missing_not_excluded[!errored_pkgs, on = c("Package", "Version")]

cat(sprintf("Known build failures: %d\n", nrow(known_failures)))
cat(sprintf("Rebuildable (no prior failure): %d\n", nrow(rebuildable)))

### 6. Write RDS for rebuild workflow
cache_dir <- "/mnt/cache/packages"
if (dir.exists(cache_dir)) {
  rds_path <- file.path(cache_dir, sprintf("weekly_rebuild_%s_%s.rds", platform, arch))
  saveRDS(rebuildable$Package, rds_path)
  cat(sprintf("Wrote %d packages to %s\n", nrow(rebuildable), rds_path))
} else {
  cat(sprintf("Cache dir %s does not exist, skipping RDS write\n", cache_dir))
}

### 7. Update Forgejo issue
forgejo_token <- Sys.getenv("FORGEJO_TOKEN")
if (!nzchar(forgejo_token)) {
  cat("FORGEJO_TOKEN not set, skipping issue update\n")
} else {
  base_url <- "https://git.devxy.io/api/v1"
  repo <- "devxy/build-cran-binaries"
  issue_title <- sprintf("Missing package binaries for latest version (%s)", os_family)

  # Helper: make API request
  forgejo_get <- function(path, query = list()) {
    url <- paste0(base_url, path)
    resp <- httr2::request(url) |>
      httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
      httr2::req_url_query(!!!query) |>
      httr2::req_perform()
    httr2::resp_body_json(resp)
  }

  forgejo_patch <- function(path, body) {
    url <- paste0(base_url, path)
    httr2::request(url) |>
      httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
      httr2::req_method("PATCH") |>
      httr2::req_body_json(body) |>
      httr2::req_perform()
  }

  forgejo_post <- function(path, body) {
    url <- paste0(base_url, path)
    httr2::request(url) |>
      httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
      httr2::req_body_json(body) |>
      httr2::req_perform()
  }

  # Build the markdown section for this platform/arch
  build_section <- function() {
    n_missing <- nrow(rebuildable) + nrow(known_failures)
    n_rebuild <- nrow(rebuildable)
    header <- sprintf("### %s (%d missing, %d to rebuild)", arch, n_missing, n_rebuild)

    lines <- header
    if (nrow(rebuildable) > 0) {
      pkg_lines <- sprintf("- %s (%s)", rebuildable$Package, rebuildable$Version)
      lines <- c(lines, "", pkg_lines)
    } else if (nrow(known_failures) == 0) {
      lines <- c(lines, "", "All binaries available.")
    }

    if (nrow(known_failures) > 0) {
      lines <- c(lines, "", "#### Known build failures",
                 sprintf("- %s (%s)", known_failures$Package, known_failures$Version))
    }

    paste(lines, collapse = "\n")
  }

  # Build excluded packages footer
  build_excluded_footer <- function() {
    if (nrow(excluded_json) == 0) return("")
    items <- sprintf("%s (%s)", excluded_json$package, excluded_json$reason)
    paste0("\n---\n\n## Excluded packages\n", paste(items, collapse = ", "))
  }

  new_section <- build_section()

  # Search for existing issue
  issues <- forgejo_get(
    sprintf("/repos/%s/issues", repo),
    query = list(type = "issues", state = "open", q = issue_title, limit = 50)
  )

  # Find exact title match
  existing <- Filter(function(i) i$title == issue_title, issues)

  if (length(existing) > 0) {
    issue <- existing[[1]]
    body <- issue$body

    # Replace or insert the platform section + arch subsection
    platform_header <- sprintf("## %s", platform)
    arch_header <- sprintf("### %s", arch)

    # Split body into lines for manipulation
    body_lines <- strsplit(body, "\n")[[1]]

    # Find the platform section
    platform_start <- which(body_lines == platform_header)

    if (length(platform_start) > 0) {
      platform_start <- platform_start[1]
      # Find end of this platform section (next ## or --- or end)
      remaining <- body_lines[(platform_start + 1):length(body_lines)]
      platform_end_offset <- which(grepl("^## |^---$", remaining))
      platform_end <- if (length(platform_end_offset) > 0) {
        platform_start + platform_end_offset[1] - 1
      } else {
        length(body_lines)
      }

      # Within platform section, find the arch subsection
      section_lines <- body_lines[platform_start:platform_end]
      arch_start_offset <- which(grepl(sprintf("^### %s", arch), section_lines))

      if (length(arch_start_offset) > 0) {
        arch_start <- arch_start_offset[1]
        # Find end of arch subsection (next ### or ## or --- or end of platform section)
        arch_remaining <- section_lines[(arch_start + 1):length(section_lines)]
        arch_end_offset <- which(grepl("^###|^## |^---$", arch_remaining))
        arch_end <- if (length(arch_end_offset) > 0) {
          arch_start + arch_end_offset[1] - 1
        } else {
          length(section_lines)
        }
        # Replace arch subsection within platform section
        section_lines <- c(
          section_lines[1:(arch_start - 1)],
          strsplit(new_section, "\n")[[1]],
          if (arch_end < length(section_lines)) section_lines[(arch_end + 1):length(section_lines)] else character(0)
        )
      } else {
        # Append arch subsection to end of platform section
        section_lines <- c(section_lines, "", strsplit(new_section, "\n")[[1]])
      }

      body_lines <- c(
        body_lines[1:(platform_start - 1)],
        section_lines,
        if (platform_end < length(body_lines)) body_lines[(platform_end + 1):length(body_lines)] else character(0)
      )
    } else {
      # Insert new platform section before "---" (excluded packages footer) or at end
      footer_line <- which(body_lines == "---")
      insert_at <- if (length(footer_line) > 0) footer_line[1] - 1 else length(body_lines)
      body_lines <- c(
        body_lines[1:insert_at],
        "",
        platform_header,
        "",
        strsplit(new_section, "\n")[[1]],
        if (insert_at < length(body_lines)) body_lines[(insert_at + 1):length(body_lines)] else character(0)
      )
    }

    # Update timestamp
    timestamp_pattern <- "^_Last updated:.*_$"
    ts_line <- which(grepl(timestamp_pattern, body_lines))
    new_ts <- sprintf("_Last updated: %s_", Sys.Date())
    if (length(ts_line) > 0) {
      body_lines[ts_line[1]] <- new_ts
    } else {
      body_lines <- c(new_ts, "", body_lines)
    }

    # Rebuild excluded footer
    footer_start <- which(body_lines == "---")
    if (length(footer_start) > 0) {
      body_lines <- c(body_lines[1:(footer_start[1] - 1)],
                       strsplit(build_excluded_footer(), "\n")[[1]])
    } else {
      body_lines <- c(body_lines, strsplit(build_excluded_footer(), "\n")[[1]])
    }

    new_body <- paste(body_lines, collapse = "\n")
    forgejo_patch(
      sprintf("/repos/%s/issues/%d", repo, issue$number),
      list(body = new_body)
    )
    cat(sprintf("Updated issue #%d: %s\n", issue$number, issue_title))
  } else {
    # Create new issue
    body_lines <- c(
      sprintf("_Last updated: %s_", Sys.Date()),
      "",
      sprintf("## %s", platform),
      "",
      new_section,
      build_excluded_footer()
    )
    new_body <- paste(body_lines, collapse = "\n")
    forgejo_post(
      sprintf("/repos/%s/issues", repo),
      list(title = issue_title, body = new_body)
    )
    cat(sprintf("Created new issue: %s\n", issue_title))
  }
}

cat("Audit complete.\n")
```

- [ ] **Step 2: Commit**

```bash
git add local/weekly-missing-binaries-audit.R
git commit -m "feat: add weekly missing binaries audit R script"
```

---

## Task 3: Create audit workflow for ubuntu-2204-amd64

**Files:**
- Create: `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`

This is the template. All other audit workflows follow the same structure with substituted values.

- [ ] **Step 1: Create the workflow YAML**

Create `.crow/weekly-audit-missing-ubuntu-2204-amd64.yaml`:

```yaml
when:
  - event: cron
    cron: weekly-audit-missing-ubuntu-2204-amd64
  - event: manual
    evaluate: 'task == "weekly-audit-missing-ubuntu-2204-amd64"'

skip_clone: true

steps:
  - name: 'Audit missing binaries'
    image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
    pull: true
    environment:
      B2_S3_ACCESS_KEY:
        from_secret: B2_S3_ACCESS_KEY
      B2_S3_SECRET_KEY:
        from_secret: B2_S3_SECRET_KEY
      PGPASS:
        from_secret: PGPASS
      REPO_RO_TOKEN:
        from_secret: REPO_RO_TOKEN
      FORGEJO_TOKEN:
        from_secret: FORGEJO_TOKEN
      GITHUB_PAT:
        from_secret: GITHUB_PAT
      PLATFORM: ubuntu-2204
      ARCH: amd64
      R_LIBS_USER: /mnt/cache/R-pkgs
    volumes:
      - amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache
    commands:
      - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
      - mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs
      - rm -rf /mnt/cache/R-pkgs/00LOCK-*
      - R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))'
      - R -q -e 'source("local/weekly-missing-binaries-audit.R")'
    backend_options:
      kubernetes:
        resources:
          requests:
            memory: 2Gi
            cpu: 2000m
          limits:
            memory: 4Gi
            cpu: 2000m
        nodeSelector:
          kubernetes.io/arch: amd64
        tolerations:
          - key: 'CI'
            operator: 'Equal'
            value: 'true'
            effect: 'NoSchedule'
```

- [ ] **Step 2: Commit**

```bash
git add .crow/weekly-audit-missing-ubuntu-2204-amd64.yaml
git commit -m "feat: add weekly audit workflow for ubuntu-2204-amd64"
```

---

## Task 4: Create remaining 13 audit workflows

**Files:**
- Create: 13 files in `.crow/` (see substitution table below)

Each file follows the exact same structure as Task 3 with these substitutions:

| File suffix | PLATFORM | ARCH | Image | Volume |
|-------------|----------|------|-------|--------|
| ubuntu-2204-arm64 | ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2204 |
| ubuntu-2404-amd64 | ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | amd64-binaries-r-dep-cache-ubuntu2404 |
| ubuntu-2404-arm64 | ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2404 |
| alpine-321-amd64 | alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | amd64-binaries-r-dep-cache-alpine321 |
| alpine-321-arm64 | alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | arm64-binaries-r-dep-cache-alpine321 |
| alpine-322-amd64 | alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | amd64-binaries-r-dep-cache-alpine322 |
| alpine-322-arm64 | alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | arm64-binaries-r-dep-cache-alpine322 |
| alpine-323-amd64 | alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | amd64-binaries-r-dep-cache-alpine323 |
| alpine-323-arm64 | alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | arm64-binaries-r-dep-cache-alpine323 |
| redhat-8-amd64 | redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | amd64-binaries-r-dep-cache-redhat8 |
| redhat-8-arm64 | redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | arm64-binaries-r-dep-cache-redhat8 |
| redhat-9-amd64 | redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | amd64-binaries-r-dep-cache-redhat9 |
| redhat-9-arm64 | redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | arm64-binaries-r-dep-cache-redhat9 |

**Key substitution points in each YAML (6 locations):**

1. `cron:` value — `weekly-audit-missing-{suffix}`
2. `evaluate:` value — `'task == "weekly-audit-missing-{suffix}"'`
3. `image:` — use the Image column
4. `PLATFORM:` env var — use the PLATFORM column
5. `ARCH:` env var — use the ARCH column
6. `volumes:` — `{ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache`
7. `nodeSelector: kubernetes.io/arch:` — use the ARCH column

- [ ] **Step 1: Create all 13 audit workflow files**

Copy the template from Task 3 and substitute the values per the table above. Example for alpine-321-arm64:

```yaml
when:
  - event: cron
    cron: weekly-audit-missing-alpine-321-arm64
  - event: manual
    evaluate: 'task == "weekly-audit-missing-alpine-321-arm64"'

skip_clone: true

steps:
  - name: 'Audit missing binaries'
    image: reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5
    pull: true
    environment:
      B2_S3_ACCESS_KEY:
        from_secret: B2_S3_ACCESS_KEY
      B2_S3_SECRET_KEY:
        from_secret: B2_S3_SECRET_KEY
      PGPASS:
        from_secret: PGPASS
      REPO_RO_TOKEN:
        from_secret: REPO_RO_TOKEN
      FORGEJO_TOKEN:
        from_secret: FORGEJO_TOKEN
      GITHUB_PAT:
        from_secret: GITHUB_PAT
      PLATFORM: alpine-321
      ARCH: arm64
      R_LIBS_USER: /mnt/cache/R-pkgs
    volumes:
      - arm64-binaries-r-dep-cache-alpine321:/mnt/cache
    commands:
      - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
      - mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs
      - rm -rf /mnt/cache/R-pkgs/00LOCK-*
      - R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))'
      - R -q -e 'source("local/weekly-missing-binaries-audit.R")'
    backend_options:
      kubernetes:
        resources:
          requests:
            memory: 2Gi
            cpu: 2000m
          limits:
            memory: 4Gi
            cpu: 2000m
        nodeSelector:
          kubernetes.io/arch: arm64
        tolerations:
          - key: 'CI'
            operator: 'Equal'
            value: 'true'
            effect: 'NoSchedule'
```

Repeat for all 13 remaining suffixes from the table, substituting the 7 locations.

- [ ] **Step 2: Commit**

```bash
git add .crow/weekly-audit-missing-*.yaml
git commit -m "feat: add remaining 13 weekly audit workflows for all platform/arch combos"
```

---

## Task 5: Create rebuild workflow for ubuntu-2204-amd64

**Files:**
- Create: `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`

This is the template for rebuild workflows. It reads the RDS written by the audit and builds each package.

- [ ] **Step 1: Create the workflow YAML**

Create `.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml`:

```yaml
when:
  - event: cron
    cron: weekly-rebuild-missing-ubuntu-2204-amd64
  - event: manual
    evaluate: 'task == "weekly-rebuild-missing-ubuntu-2204-amd64"'

skip_clone: true

steps:
  - name: 'Rebuild missing binaries'
    image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
    pull: true
    environment:
      RED_HAT_DEV_PW:
        from_secret: RED_HAT_DEV_PW
      B2_S3_ACCESS_KEY:
        from_secret: B2_S3_ACCESS_KEY
      B2_S3_SECRET_KEY:
        from_secret: B2_S3_SECRET_KEY
      PGPASS:
        from_secret: PGPASS
      REPO_RO_TOKEN:
        from_secret: REPO_RO_TOKEN
      GITHUB_PAT:
        from_secret: GITHUB_PAT
      GIT_USER: pat-s
      R_PKG_CACHE_DIR: /mnt/cache/pkgcache
      R_LIBS_USER: /mnt/cache/R-pkgs
      CCACHE_DIR: /mnt/cache/ccache
      PLATFORM: ubuntu-2204
      ARCH: amd64
      NCPUS: 2
    volumes:
      - amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache
    commands:
      - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
      - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
      - rm -rf /mnt/cache/R-pkgs/00LOCK-*
      - R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
      - R -q -e 'packageVersion("bincraft")'
      - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
      - $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_ubuntu-2204_amd64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1
    backend_options:
      kubernetes:
        resources:
          requests:
            memory: 5Gi
            cpu: 3000m
          limits:
            memory: 18Gi
            cpu: 3000m
        nodeSelector:
          kubernetes.io/arch: amd64
          node.kubernetes.io/instance-type: AX42
        tolerations:
          - key: 'CI'
            operator: 'Equal'
            value: 'true'
            effect: 'NoSchedule'
```

- [ ] **Step 2: Commit**

```bash
git add .crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml
git commit -m "feat: add weekly rebuild workflow for ubuntu-2204-amd64"
```

---

## Task 6: Create remaining 13 rebuild workflows

**Files:**
- Create: 13 files in `.crow/` (see substitution table below)

Each file follows the exact same structure as Task 5 with these substitutions:

| File suffix | PLATFORM | ARCH | Image | Volume | nodeSelector instance-type |
|-------------|----------|------|-------|--------|--------------------------|
| ubuntu-2204-arm64 | ubuntu-2204 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2204 | (omit) |
| ubuntu-2404-amd64 | ubuntu-2404 | amd64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | amd64-binaries-r-dep-cache-ubuntu2404 | AX42 |
| ubuntu-2404-arm64 | ubuntu-2404 | arm64 | reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 | arm64-binaries-r-dep-cache-ubuntu2404 | (omit) |
| alpine-321-amd64 | alpine-321 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | amd64-binaries-r-dep-cache-alpine321 | AX42 |
| alpine-321-arm64 | alpine-321 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 | arm64-binaries-r-dep-cache-alpine321 | (omit) |
| alpine-322-amd64 | alpine-322 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | amd64-binaries-r-dep-cache-alpine322 | AX42 |
| alpine-322-arm64 | alpine-322 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 | arm64-binaries-r-dep-cache-alpine322 | (omit) |
| alpine-323-amd64 | alpine-323 | amd64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | amd64-binaries-r-dep-cache-alpine323 | AX42 |
| alpine-323-arm64 | alpine-323 | arm64 | reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 | arm64-binaries-r-dep-cache-alpine323 | (omit) |
| redhat-8-amd64 | redhat-8 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | amd64-binaries-r-dep-cache-redhat8 | AX42 |
| redhat-8-arm64 | redhat-8 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 | arm64-binaries-r-dep-cache-redhat8 | (omit) |
| redhat-9-amd64 | redhat-9 | amd64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | amd64-binaries-r-dep-cache-redhat9 | AX42 |
| redhat-9-arm64 | redhat-9 | arm64 | reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 | arm64-binaries-r-dep-cache-redhat9 | (omit) |

**Key substitution points (8 locations):**

1. `cron:` value — `weekly-rebuild-missing-{suffix}`
2. `evaluate:` value — `'task == "weekly-rebuild-missing-{suffix}"'`
3. `image:` — use the Image column
4. `PLATFORM:` env var — use the PLATFORM column
5. `ARCH:` env var — use the ARCH column
6. `volumes:` — `{ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache`
7. `rds_path` in the R command — `weekly_rebuild_{PLATFORM}_{ARCH}.rds`
8. `nodeSelector` — `kubernetes.io/arch: {ARCH}`, plus `node.kubernetes.io/instance-type: AX42` only for amd64 (omit the instance-type line entirely for arm64)

- [ ] **Step 1: Create all 13 rebuild workflow files**

Copy the template from Task 5 and substitute per the table. Example for redhat-9-arm64:

```yaml
when:
  - event: cron
    cron: weekly-rebuild-missing-redhat-9-arm64
  - event: manual
    evaluate: 'task == "weekly-rebuild-missing-redhat-9-arm64"'

skip_clone: true

steps:
  - name: 'Rebuild missing binaries'
    image: reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3
    pull: true
    environment:
      RED_HAT_DEV_PW:
        from_secret: RED_HAT_DEV_PW
      B2_S3_ACCESS_KEY:
        from_secret: B2_S3_ACCESS_KEY
      B2_S3_SECRET_KEY:
        from_secret: B2_S3_SECRET_KEY
      PGPASS:
        from_secret: PGPASS
      REPO_RO_TOKEN:
        from_secret: REPO_RO_TOKEN
      GITHUB_PAT:
        from_secret: GITHUB_PAT
      GIT_USER: pat-s
      R_PKG_CACHE_DIR: /mnt/cache/pkgcache
      R_LIBS_USER: /mnt/cache/R-pkgs
      CCACHE_DIR: /mnt/cache/ccache
      PLATFORM: redhat-9
      ARCH: arm64
      NCPUS: 2
    volumes:
      - arm64-binaries-r-dep-cache-redhat9:/mnt/cache
    commands:
      - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
      - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
      - rm -rf /mnt/cache/R-pkgs/00LOCK-*
      - R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
      - R -q -e 'packageVersion("bincraft")'
      - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
      - $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_redhat-9_arm64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1
    backend_options:
      kubernetes:
        resources:
          requests:
            memory: 5Gi
            cpu: 3000m
          limits:
            memory: 18Gi
            cpu: 3000m
        nodeSelector:
          kubernetes.io/arch: arm64
        tolerations:
          - key: 'CI'
            operator: 'Equal'
            value: 'true'
            effect: 'NoSchedule'
```

Note: arm64 workflows do NOT include `node.kubernetes.io/instance-type: AX42` in nodeSelector.

Repeat for all 13 remaining suffixes from the table, substituting the 8 locations.

- [ ] **Step 2: Commit**

```bash
git add .crow/weekly-rebuild-missing-*.yaml
git commit -m "feat: add remaining 13 weekly rebuild workflows for all platform/arch combos"
```

---

## Task 7: Final verification

- [ ] **Step 1: Verify all files exist**

Run:
```bash
ls -1 local/excluded-packages.json local/weekly-missing-binaries-audit.R
ls -1 .crow/weekly-audit-missing-*.yaml | wc -l
ls -1 .crow/weekly-rebuild-missing-*.yaml | wc -l
```

Expected: both files exist, 14 audit workflows, 14 rebuild workflows.

- [ ] **Step 2: Validate JSON**

Run:
```bash
python3 -c "import json; json.load(open('local/excluded-packages.json')); print('JSON valid')"
```

Expected: `JSON valid`

- [ ] **Step 3: Validate YAML syntax**

Run:
```bash
python3 -c "
import yaml, glob
for f in sorted(glob.glob('.crow/weekly-*-missing-*.yaml') + glob.glob('.crow/weekly-audit-missing-*.yaml') + glob.glob('.crow/weekly-rebuild-missing-*.yaml')):
    yaml.safe_load(open(f))
    print(f'OK: {f}')
"
```

Expected: all files print `OK`.

- [ ] **Step 4: Verify unique cron names**

Run:
```bash
grep -h 'cron: weekly-' .crow/weekly-*.yaml | sort | uniq -c | sort -rn | head
```

Expected: all counts are 1 (no duplicates).

- [ ] **Step 5: Verify all 14 platform/arch combos covered**

Run:
```bash
for suffix in ubuntu-2204-amd64 ubuntu-2204-arm64 ubuntu-2404-amd64 ubuntu-2404-arm64 alpine-321-amd64 alpine-321-arm64 alpine-322-amd64 alpine-322-arm64 alpine-323-amd64 alpine-323-arm64 redhat-8-amd64 redhat-8-arm64 redhat-9-amd64 redhat-9-arm64; do
  test -f ".crow/weekly-audit-missing-${suffix}.yaml" || echo "MISSING audit: ${suffix}"
  test -f ".crow/weekly-rebuild-missing-${suffix}.yaml" || echo "MISSING rebuild: ${suffix}"
done
echo "All checks passed if no MISSING lines above"
```

Expected: no MISSING lines, just "All checks passed".