devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 10 Pull requests 1 Releases Activity
build-cran-binaries/docs/superpowers/plans/2026-04-11-weekly-missing-binaries.md

36 KiB
Raw Blame History

Weekly Missing Binaries Audit & Rebuild — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add a weekly CI workflow that audits CRAN packages missing binaries for their latest release version, reports them in Forgejo issues by OS family, and rebuilds those without a prior build failure.

Architecture: Two independent workflow sets (audit + rebuild), each with one YAML per platform/arch (14 each). A shared R script does the audit logic. An excluded-packages JSON config is the shared blocklist. The audit writes per-platform RDS files consumed by rebuild workflows.

Tech Stack: R (bincraft, s3fs, DBI/RPostgres, jsonlite, httr2), Crow/Woodpecker CI YAML, Forgejo API, PostgreSQL, Backblaze S3.

File Structure

File Responsibility
local/excluded-packages.json Excluded packages with reasons (single source of truth)
local/weekly-missing-binaries-audit.R Audit logic: S3 check, DB check, issue update, RDS output
.crow/weekly-audit-missing-{platform}-{arch}.yaml (14 files) Audit workflow per platform/arch
.crow/weekly-rebuild-missing-{platform}-{arch}.yaml (14 files) Rebuild workflow per platform/arch

Platform Reference

Used throughout the plan. Each row is one workflow file pair (audit + rebuild).

PLATFORM env var ARCH S3 codename (sub("-","",PLATFORM)) OS Family Image
ubuntu-2204 amd64 ubuntu2204 Ubuntu reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
ubuntu-2204 arm64 ubuntu2204 Ubuntu reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
ubuntu-2404 amd64 ubuntu2404 Ubuntu reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3
ubuntu-2404 arm64 ubuntu2404 Ubuntu reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3
alpine-321 amd64 alpine321 Alpine reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5
alpine-321 arm64 alpine321 Alpine reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5
alpine-322 amd64 alpine322 Alpine reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5
alpine-322 arm64 alpine322 Alpine reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5
alpine-323 amd64 alpine323 Alpine reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5
alpine-323 arm64 alpine323 Alpine reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5
redhat-8 amd64 redhat8 Red Hat reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3
redhat-8 arm64 redhat8 Red Hat reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3
redhat-9 amd64 redhat9 Red Hat reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3
redhat-9 arm64 redhat9 Red Hat reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3

Task 1: Create excluded-packages.json

Files:

  • Create: local/excluded-packages.json

  • Step 1: Create the JSON file

Extract every package from the inline exclusion list in .crow/build-all-versions-amd64.yaml (line 72). The list has two categories: Windows-only packages (comment on line 69) and problematic packages (hang/OOM). Create local/excluded-packages.json:

[
  {"package": "RInno", "reason": "windows-only"},
  {"package": "KeyboardSimulator", "reason": "windows-only"},
  {"package": "R2PPT", "reason": "windows-only"},
  {"package": "RWinEdt", "reason": "windows-only"},
  {"package": "blatr", "reason": "windows-only"},
  {"package": "excel.link", "reason": "windows-only"},
  {"package": "spectrino", "reason": "windows-only"},
  {"package": "taskscheduleR", "reason": "windows-only"},
  {"package": "MDSGUI", "reason": "windows-only"},
  {"package": "BiplotGUI", "reason": "windows-only"},
  {"package": "R2wd", "reason": "windows-only"},
  {"package": "rFUSION", "reason": "windows-only"},
  {"package": "MediaNews", "reason": "windows-only"},
  {"package": "doBy", "reason": "hang"},
  {"package": "IDPmisc", "reason": "hang"},
  {"package": "frailtypack", "reason": "hang"},
  {"package": "afex", "reason": "hang"},
  {"package": "FrF2", "reason": "hang"},
  {"package": "DoE.base", "reason": "hang"},
  {"package": "agricolae", "reason": "hang"},
  {"package": "doFuture", "reason": "hang"},
  {"package": "fscaret", "reason": "hang"},
  {"package": "PHYLOGR", "reason": "hang"},
  {"package": "seewave", "reason": "hang"},
  {"package": "pls", "reason": "hang"},
  {"package": "relaimpo", "reason": "hang"},
  {"package": "geepack", "reason": "hang"},
  {"package": "gggenes", "reason": "hang"},
  {"package": "NPCirc", "reason": "hang"},
  {"package": "repmis", "reason": "hang"},
  {"package": "PNDSIBGE", "reason": "hang"},
  {"package": "lidR", "reason": "hang"},
  {"package": "poismf", "reason": "hang"},
  {"package": "neonstore", "reason": "hang"},
  {"package": "MachineShop", "reason": "hang"},
  {"package": "mvst", "reason": "hang"},
  {"package": "MacBehaviour", "reason": "hang"},
  {"package": "mcmcderive", "reason": "hang"},
  {"package": "RGIFT", "reason": "hang"},
  {"package": "KnowBR", "reason": "hang"},
  {"package": "netmeta", "reason": "hang"},
  {"package": "spdep", "reason": "hang"},
  {"package": "Rfast", "reason": "hang"},
  {"package": "compareGroups", "reason": "hang"},
  {"package": "ff", "reason": "hang"},
  {"package": "GsymPoint", "reason": "hang"},
  {"package": "RcppDynProg", "reason": "hang"},
  {"package": "comtradr", "reason": "hang"},
  {"package": "FD", "reason": "hang"},
  {"package": "PearsonDS", "reason": "hang"},
  {"package": "DCluster", "reason": "hang"},
  {"package": "gRc", "reason": "hang"},
  {"package": "mixlm", "reason": "hang"},
  {"package": "geospt", "reason": "hang"},
  {"package": "fdth", "reason": "hang"},
  {"package": "ffmanova", "reason": "hang"},
  {"package": "fiery", "reason": "hang"},
  {"package": "ffscrapr", "reason": "hang"},
  {"package": "cold", "reason": "hang"},
  {"package": "RcmdrPlugin.DoE", "reason": "hang"},
  {"package": "RcmdrPlugin.NMBU", "reason": "hang"},
  {"package": "RcmdrPlugin.RiskDemo", "reason": "hang"},
  {"package": "RcmdrPlugin.TeachStat", "reason": "hang"},
  {"package": "RcmdrPlugin.TeachingDemos", "reason": "hang"},
  {"package": "RcmdrPlugin.UCA", "reason": "hang"},
  {"package": "RcmdrPlugin.WorldFlora", "reason": "hang"},
  {"package": "RcmdrPlugin.aRnova", "reason": "hang"},
  {"package": "RcmdrPlugin.depthTools", "reason": "hang"},
  {"package": "RcmdrPlugin.orloca", "reason": "hang"},
  {"package": "RcmdrPlugin.sos", "reason": "hang"},
  {"package": "RcmdrPlugin.survival", "reason": "hang"},
  {"package": "RcmdrPlugin.temis", "reason": "hang"},
  {"package": "GWlasso", "reason": "hang"},
  {"package": "GWmodelVis", "reason": "hang"}
]
  • Step 2: Commit
git add local/excluded-packages.json
git commit -m "feat: add excluded-packages.json for weekly missing binaries workflow"

Task 2: Create the audit R script

Files:

  • Create: local/weekly-missing-binaries-audit.R

This is the core logic. It reads env vars PLATFORM and ARCH, checks S3 + DB, writes an RDS of rebuildable packages, and updates the Forgejo issue for the OS family.

  • Step 1: Create the R script

Create local/weekly-missing-binaries-audit.R with the full content below:

options(error = function() {
  cat("ERROR:", geterrmessage(), "\n", file = stdout())
  traceback(2)
  q(status = 1)
})

library(DBI, quietly = TRUE)
suppressPackageStartupMessages(library(data.table))

platform <- Sys.getenv("PLATFORM")
arch <- Sys.getenv("ARCH")
stopifnot(nzchar(platform), nzchar(arch))

# S3 codename: remove hyphen from platform string (e.g. "ubuntu-2204" -> "ubuntu2204")
s3_codename <- gsub("-", "", platform)

# OS family for issue grouping
os_family <- if (grepl("^ubuntu", platform)) {
  "Ubuntu"
} else if (grepl("^alpine", platform)) {
  "Alpine"
} else if (grepl("^redhat", platform)) {
  "Red Hat"
} else {
  stop("Unknown platform: ", platform)
}

cat(sprintf("Audit: platform=%s, arch=%s, s3_codename=%s, os_family=%s\n",
            platform, arch, s3_codename, os_family))

### 1. Get current CRAN release packages
cran_release <- as.data.table(tools::CRAN_package_db()[, c("Package", "Version")])
cran_release[, Version := as.character(Version)]
cat(sprintf("CRAN release packages: %d\n", nrow(cran_release)))

### 2. List S3 tarballs for this platform/arch
s3fs::s3_file_system(
  aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"),
  aws_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"),
  endpoint = "https://s3.eu-central-003.backblazeb2.com",
  region_name = "eu-central-003",
  refresh = TRUE
)

s3_path <- sprintf("devxy-rpkgs-binaries/%s/%s/latest/src/contrib", arch, s3_codename)
s3_files <- tryCatch(
  s3fs::s3_dir_ls(s3_path),
  error = function(e) {
    cat(sprintf("Warning: could not list S3 path %s: %s\n", s3_path, conditionMessage(e)))
    character(0)
  }
)

# Parse package name + version from tarball filenames
file_names <- basename(s3_files)
matches <- regexec("^([A-Za-z0-9.]+)_([0-9][^/]*)\\.tar\\.gz$", file_names)
parts <- regmatches(file_names, matches)
parts <- parts[lengths(parts) == 3]

s3_dt <- if (length(parts) > 0) {
  data.table(
    Package = vapply(parts, `[`, character(1), 2),
    Version = vapply(parts, `[`, character(1), 3)
  )
} else {
  data.table(Package = character(0), Version = character(0))
}
cat(sprintf("S3 packages for %s/%s: %d\n", arch, s3_codename, nrow(s3_dt)))

### 3. Find CRAN release packages missing from S3
setkey(cran_release, Package, Version)
setkey(s3_dt, Package, Version)
missing <- cran_release[!s3_dt]
cat(sprintf("Missing binaries (latest CRAN version): %d\n", nrow(missing)))

### 4. Load excluded packages
excluded_json <- jsonlite::fromJSON("local/excluded-packages.json")
excluded_pkgs <- excluded_json$package
missing_not_excluded <- missing[!Package %in% excluded_pkgs]
missing_excluded <- missing[Package %in% excluded_pkgs]
cat(sprintf("Missing after excluding %d blocked packages: %d\n",
            length(excluded_pkgs), nrow(missing_not_excluded)))

### 5. Check DB for prior build failures on these exact versions
con <- DBI::dbConnect(
  RPostgres::Postgres(),
  dbname = "build_metadata",
  host = "r-binaries.devxy.io",
  port = 15432,
  user = "rpkgs",
  password = Sys.getenv("PGPASS"),
  sslmode = "require"
)
on.exit(DBI::dbDisconnect(con), add = TRUE)

errored_pkgs <- as.data.table(DBI::dbGetQuery(
  con,
  sprintf(
    "SELECT name, tag FROM single_builds WHERE error_occurred = TRUE AND platform = '%s' AND arch = '%s'",
    platform, arch
  )
))
setnames(errored_pkgs, c("Package", "Version"))
setkey(errored_pkgs, Package, Version)

# Split: rebuildable vs known failures
known_failures <- missing_not_excluded[errored_pkgs, nomatch = 0, on = c("Package", "Version")]
rebuildable <- missing_not_excluded[!errored_pkgs, on = c("Package", "Version")]

cat(sprintf("Known build failures: %d\n", nrow(known_failures)))
cat(sprintf("Rebuildable (no prior failure): %d\n", nrow(rebuildable)))

### 6. Write RDS for rebuild workflow
cache_dir <- "/mnt/cache/packages"
if (dir.exists(cache_dir)) {
  rds_path <- file.path(cache_dir, sprintf("weekly_rebuild_%s_%s.rds", platform, arch))
  saveRDS(rebuildable$Package, rds_path)
  cat(sprintf("Wrote %d packages to %s\n", nrow(rebuildable), rds_path))
} else {
  cat(sprintf("Cache dir %s does not exist, skipping RDS write\n", cache_dir))
}

### 7. Update Forgejo issue
forgejo_token <- Sys.getenv("FORGEJO_TOKEN")
if (!nzchar(forgejo_token)) {
  cat("FORGEJO_TOKEN not set, skipping issue update\n")
} else {
  base_url <- "https://git.devxy.io/api/v1"
  repo <- "devxy/build-cran-binaries"
  issue_title <- sprintf("Missing package binaries for latest version (%s)", os_family)

  # Helper: make API request
  forgejo_get <- function(path, query = list()) {
    url <- paste0(base_url, path)
    resp <- httr2::request(url) |>
      httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
      httr2::req_url_query(!!!query) |>
      httr2::req_perform()
    httr2::resp_body_json(resp)
  }

  forgejo_patch <- function(path, body) {
    url <- paste0(base_url, path)
    httr2::request(url) |>
      httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
      httr2::req_method("PATCH") |>
      httr2::req_body_json(body) |>
      httr2::req_perform()
  }

  forgejo_post <- function(path, body) {
    url <- paste0(base_url, path)
    httr2::request(url) |>
      httr2::req_headers(Authorization = paste("token", forgejo_token)) |>
      httr2::req_body_json(body) |>
      httr2::req_perform()
  }

  # Build the markdown section for this platform/arch
  build_section <- function() {
    n_missing <- nrow(rebuildable) + nrow(known_failures)
    n_rebuild <- nrow(rebuildable)
    header <- sprintf("### %s (%d missing, %d to rebuild)", arch, n_missing, n_rebuild)

    lines <- header
    if (nrow(rebuildable) > 0) {
      pkg_lines <- sprintf("- %s (%s)", rebuildable$Package, rebuildable$Version)
      lines <- c(lines, "", pkg_lines)
    } else if (nrow(known_failures) == 0) {
      lines <- c(lines, "", "All binaries available.")
    }

    if (nrow(known_failures) > 0) {
      lines <- c(lines, "", "#### Known build failures",
                 sprintf("- %s (%s)", known_failures$Package, known_failures$Version))
    }

    paste(lines, collapse = "\n")
  }

  # Build excluded packages footer
  build_excluded_footer <- function() {
    if (nrow(excluded_json) == 0) return("")
    items <- sprintf("%s (%s)", excluded_json$package, excluded_json$reason)
    paste0("\n---\n\n## Excluded packages\n", paste(items, collapse = ", "))
  }

  new_section <- build_section()

  # Search for existing issue
  issues <- forgejo_get(
    sprintf("/repos/%s/issues", repo),
    query = list(type = "issues", state = "open", q = issue_title, limit = 50)
  )

  # Find exact title match
  existing <- Filter(function(i) i$title == issue_title, issues)

  if (length(existing) > 0) {
    issue <- existing[[1]]
    body <- issue$body

    # Replace or insert the platform section + arch subsection
    platform_header <- sprintf("## %s", platform)
    arch_header <- sprintf("### %s", arch)

    # Split body into lines for manipulation
    body_lines <- strsplit(body, "\n")[[1]]

    # Find the platform section
    platform_start <- which(body_lines == platform_header)

    if (length(platform_start) > 0) {
      platform_start <- platform_start[1]
      # Find end of this platform section (next ## or --- or end)
      remaining <- body_lines[(platform_start + 1):length(body_lines)]
      platform_end_offset <- which(grepl("^## |^---$", remaining))
      platform_end <- if (length(platform_end_offset) > 0) {
        platform_start + platform_end_offset[1] - 1
      } else {
        length(body_lines)
      }

      # Within platform section, find the arch subsection
      section_lines <- body_lines[platform_start:platform_end]
      arch_start_offset <- which(grepl(sprintf("^### %s", arch), section_lines))

      if (length(arch_start_offset) > 0) {
        arch_start <- arch_start_offset[1]
        # Find end of arch subsection (next ### or ## or --- or end of platform section)
        arch_remaining <- section_lines[(arch_start + 1):length(section_lines)]
        arch_end_offset <- which(grepl("^###|^## |^---$", arch_remaining))
        arch_end <- if (length(arch_end_offset) > 0) {
          arch_start + arch_end_offset[1] - 1
        } else {
          length(section_lines)
        }
        # Replace arch subsection within platform section
        section_lines <- c(
          section_lines[1:(arch_start - 1)],
          strsplit(new_section, "\n")[[1]],
          if (arch_end < length(section_lines)) section_lines[(arch_end + 1):length(section_lines)] else character(0)
        )
      } else {
        # Append arch subsection to end of platform section
        section_lines <- c(section_lines, "", strsplit(new_section, "\n")[[1]])
      }

      body_lines <- c(
        body_lines[1:(platform_start - 1)],
        section_lines,
        if (platform_end < length(body_lines)) body_lines[(platform_end + 1):length(body_lines)] else character(0)
      )
    } else {
      # Insert new platform section before "---" (excluded packages footer) or at end
      footer_line <- which(body_lines == "---")
      insert_at <- if (length(footer_line) > 0) footer_line[1] - 1 else length(body_lines)
      body_lines <- c(
        body_lines[1:insert_at],
        "",
        platform_header,
        "",
        strsplit(new_section, "\n")[[1]],
        if (insert_at < length(body_lines)) body_lines[(insert_at + 1):length(body_lines)] else character(0)
      )
    }

    # Update timestamp
    timestamp_pattern <- "^_Last updated:.*_$"
    ts_line <- which(grepl(timestamp_pattern, body_lines))
    new_ts <- sprintf("_Last updated: %s_", Sys.Date())
    if (length(ts_line) > 0) {
      body_lines[ts_line[1]] <- new_ts
    } else {
      body_lines <- c(new_ts, "", body_lines)
    }

    # Rebuild excluded footer
    footer_start <- which(body_lines == "---")
    if (length(footer_start) > 0) {
      body_lines <- c(body_lines[1:(footer_start[1] - 1)],
                       strsplit(build_excluded_footer(), "\n")[[1]])
    } else {
      body_lines <- c(body_lines, strsplit(build_excluded_footer(), "\n")[[1]])
    }

    new_body <- paste(body_lines, collapse = "\n")
    forgejo_patch(
      sprintf("/repos/%s/issues/%d", repo, issue$number),
      list(body = new_body)
    )
    cat(sprintf("Updated issue #%d: %s\n", issue$number, issue_title))
  } else {
    # Create new issue
    body_lines <- c(
      sprintf("_Last updated: %s_", Sys.Date()),
      "",
      sprintf("## %s", platform),
      "",
      new_section,
      build_excluded_footer()
    )
    new_body <- paste(body_lines, collapse = "\n")
    forgejo_post(
      sprintf("/repos/%s/issues", repo),
      list(title = issue_title, body = new_body)
    )
    cat(sprintf("Created new issue: %s\n", issue_title))
  }
}

cat("Audit complete.\n")
  • Step 2: Commit
git add local/weekly-missing-binaries-audit.R
git commit -m "feat: add weekly missing binaries audit R script"

Task 3: Create audit workflow for ubuntu-2204-amd64

Files:

  • Create: .crow/weekly-audit-missing-ubuntu-2204-amd64.yaml

This is the template. All other audit workflows follow the same structure with substituted values.

  • Step 1: Create the workflow YAML

Create .crow/weekly-audit-missing-ubuntu-2204-amd64.yaml:

when:
  - event: cron
    cron: weekly-audit-missing-ubuntu-2204-amd64
  - event: manual
    evaluate: 'task == "weekly-audit-missing-ubuntu-2204-amd64"'

skip_clone: true

steps:
  - name: 'Audit missing binaries'
    image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
    pull: true
    environment:
      B2_S3_ACCESS_KEY:
        from_secret: B2_S3_ACCESS_KEY
      B2_S3_SECRET_KEY:
        from_secret: B2_S3_SECRET_KEY
      PGPASS:
        from_secret: PGPASS
      REPO_RO_TOKEN:
        from_secret: REPO_RO_TOKEN
      FORGEJO_TOKEN:
        from_secret: FORGEJO_TOKEN
      GITHUB_PAT:
        from_secret: GITHUB_PAT
      PLATFORM: ubuntu-2204
      ARCH: amd64
      R_LIBS_USER: /mnt/cache/R-pkgs
    volumes:
      - amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache
    commands:
      - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
      - mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs
      - rm -rf /mnt/cache/R-pkgs/00LOCK-*
      - R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))'
      - R -q -e 'source("local/weekly-missing-binaries-audit.R")'
    backend_options:
      kubernetes:
        resources:
          requests:
            memory: 2Gi
            cpu: 2000m
          limits:
            memory: 4Gi
            cpu: 2000m
        nodeSelector:
          kubernetes.io/arch: amd64
        tolerations:
          - key: 'CI'
            operator: 'Equal'
            value: 'true'
            effect: 'NoSchedule'
  • Step 2: Commit
git add .crow/weekly-audit-missing-ubuntu-2204-amd64.yaml
git commit -m "feat: add weekly audit workflow for ubuntu-2204-amd64"

Task 4: Create remaining 13 audit workflows

Files:

  • Create: 13 files in .crow/ (see substitution table below)

Each file follows the exact same structure as Task 3 with these substitutions:

File suffix PLATFORM ARCH Image Volume
ubuntu-2204-arm64 ubuntu-2204 arm64 reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 arm64-binaries-r-dep-cache-ubuntu2204
ubuntu-2404-amd64 ubuntu-2404 amd64 reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 amd64-binaries-r-dep-cache-ubuntu2404
ubuntu-2404-arm64 ubuntu-2404 arm64 reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 arm64-binaries-r-dep-cache-ubuntu2404
alpine-321-amd64 alpine-321 amd64 reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 amd64-binaries-r-dep-cache-alpine321
alpine-321-arm64 alpine-321 arm64 reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 arm64-binaries-r-dep-cache-alpine321
alpine-322-amd64 alpine-322 amd64 reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 amd64-binaries-r-dep-cache-alpine322
alpine-322-arm64 alpine-322 arm64 reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 arm64-binaries-r-dep-cache-alpine322
alpine-323-amd64 alpine-323 amd64 reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 amd64-binaries-r-dep-cache-alpine323
alpine-323-arm64 alpine-323 arm64 reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 arm64-binaries-r-dep-cache-alpine323
redhat-8-amd64 redhat-8 amd64 reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 amd64-binaries-r-dep-cache-redhat8
redhat-8-arm64 redhat-8 arm64 reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 arm64-binaries-r-dep-cache-redhat8
redhat-9-amd64 redhat-9 amd64 reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 amd64-binaries-r-dep-cache-redhat9
redhat-9-arm64 redhat-9 arm64 reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 arm64-binaries-r-dep-cache-redhat9

Key substitution points in each YAML (6 locations):

  1. cron: value — weekly-audit-missing-{suffix}
  2. evaluate: value — 'task == "weekly-audit-missing-{suffix}"'
  3. image: — use the Image column
  4. PLATFORM: env var — use the PLATFORM column
  5. ARCH: env var — use the ARCH column
  6. volumes: — {ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache
  7. nodeSelector: kubernetes.io/arch: — use the ARCH column
  • Step 1: Create all 13 audit workflow files

Copy the template from Task 3 and substitute the values per the table above. Example for alpine-321-arm64:

when:
  - event: cron
    cron: weekly-audit-missing-alpine-321-arm64
  - event: manual
    evaluate: 'task == "weekly-audit-missing-alpine-321-arm64"'

skip_clone: true

steps:
  - name: 'Audit missing binaries'
    image: reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5
    pull: true
    environment:
      B2_S3_ACCESS_KEY:
        from_secret: B2_S3_ACCESS_KEY
      B2_S3_SECRET_KEY:
        from_secret: B2_S3_SECRET_KEY
      PGPASS:
        from_secret: PGPASS
      REPO_RO_TOKEN:
        from_secret: REPO_RO_TOKEN
      FORGEJO_TOKEN:
        from_secret: FORGEJO_TOKEN
      GITHUB_PAT:
        from_secret: GITHUB_PAT
      PLATFORM: alpine-321
      ARCH: arm64
      R_LIBS_USER: /mnt/cache/R-pkgs
    volumes:
      - arm64-binaries-r-dep-cache-alpine321:/mnt/cache
    commands:
      - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
      - mkdir -p /mnt/cache/packages /mnt/cache/R-pkgs
      - rm -rf /mnt/cache/R-pkgs/00LOCK-*
      - R -q -e 'pak::pak(c("git::https://codefloe.com/rpkgs/bincraft.git", "httr2", "jsonlite"))'
      - R -q -e 'source("local/weekly-missing-binaries-audit.R")'
    backend_options:
      kubernetes:
        resources:
          requests:
            memory: 2Gi
            cpu: 2000m
          limits:
            memory: 4Gi
            cpu: 2000m
        nodeSelector:
          kubernetes.io/arch: arm64
        tolerations:
          - key: 'CI'
            operator: 'Equal'
            value: 'true'
            effect: 'NoSchedule'

Repeat for all 13 remaining suffixes from the table, substituting the 7 locations.

  • Step 2: Commit
git add .crow/weekly-audit-missing-*.yaml
git commit -m "feat: add remaining 13 weekly audit workflows for all platform/arch combos"

Task 5: Create rebuild workflow for ubuntu-2204-amd64

Files:

  • Create: .crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml

This is the template for rebuild workflows. It reads the RDS written by the audit and builds each package.

  • Step 1: Create the workflow YAML

Create .crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml:

when:
  - event: cron
    cron: weekly-rebuild-missing-ubuntu-2204-amd64
  - event: manual
    evaluate: 'task == "weekly-rebuild-missing-ubuntu-2204-amd64"'

skip_clone: true

steps:
  - name: 'Rebuild missing binaries'
    image: reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3
    pull: true
    environment:
      RED_HAT_DEV_PW:
        from_secret: RED_HAT_DEV_PW
      B2_S3_ACCESS_KEY:
        from_secret: B2_S3_ACCESS_KEY
      B2_S3_SECRET_KEY:
        from_secret: B2_S3_SECRET_KEY
      PGPASS:
        from_secret: PGPASS
      REPO_RO_TOKEN:
        from_secret: REPO_RO_TOKEN
      GITHUB_PAT:
        from_secret: GITHUB_PAT
      GIT_USER: pat-s
      R_PKG_CACHE_DIR: /mnt/cache/pkgcache
      R_LIBS_USER: /mnt/cache/R-pkgs
      CCACHE_DIR: /mnt/cache/ccache
      PLATFORM: ubuntu-2204
      ARCH: amd64
      NCPUS: 2
    volumes:
      - amd64-binaries-r-dep-cache-ubuntu2204:/mnt/cache
    commands:
      - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
      - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
      - rm -rf /mnt/cache/R-pkgs/00LOCK-*
      - R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
      - R -q -e 'packageVersion("bincraft")'
      - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
      - $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_ubuntu-2204_amd64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1
    backend_options:
      kubernetes:
        resources:
          requests:
            memory: 5Gi
            cpu: 3000m
          limits:
            memory: 18Gi
            cpu: 3000m
        nodeSelector:
          kubernetes.io/arch: amd64
          node.kubernetes.io/instance-type: AX42
        tolerations:
          - key: 'CI'
            operator: 'Equal'
            value: 'true'
            effect: 'NoSchedule'
  • Step 2: Commit
git add .crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml
git commit -m "feat: add weekly rebuild workflow for ubuntu-2204-amd64"

Task 6: Create remaining 13 rebuild workflows

Files:

  • Create: 13 files in .crow/ (see substitution table below)

Each file follows the exact same structure as Task 5 with these substitutions:

File suffix PLATFORM ARCH Image Volume nodeSelector instance-type
ubuntu-2204-arm64 ubuntu-2204 arm64 reg.devxy.io/rpkgs/build-env-ubuntu:jammy-4.4.3 arm64-binaries-r-dep-cache-ubuntu2204 (omit)
ubuntu-2404-amd64 ubuntu-2404 amd64 reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 amd64-binaries-r-dep-cache-ubuntu2404 AX42
ubuntu-2404-arm64 ubuntu-2404 arm64 reg.devxy.io/rpkgs/build-env-ubuntu:noble-4.4.3 arm64-binaries-r-dep-cache-ubuntu2404 (omit)
alpine-321-amd64 alpine-321 amd64 reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 amd64-binaries-r-dep-cache-alpine321 AX42
alpine-321-arm64 alpine-321 arm64 reg.devxy.io/rpkgs/build-env-alpine:3.21-4.5 arm64-binaries-r-dep-cache-alpine321 (omit)
alpine-322-amd64 alpine-322 amd64 reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 amd64-binaries-r-dep-cache-alpine322 AX42
alpine-322-arm64 alpine-322 arm64 reg.devxy.io/rpkgs/build-env-alpine:3.22-4.5 arm64-binaries-r-dep-cache-alpine322 (omit)
alpine-323-amd64 alpine-323 amd64 reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 amd64-binaries-r-dep-cache-alpine323 AX42
alpine-323-arm64 alpine-323 arm64 reg.devxy.io/rpkgs/build-env-alpine:3.23-4.5 arm64-binaries-r-dep-cache-alpine323 (omit)
redhat-8-amd64 redhat-8 amd64 reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 amd64-binaries-r-dep-cache-redhat8 AX42
redhat-8-arm64 redhat-8 arm64 reg.devxy.io/rpkgs/build-env-redhat:8-4.4.3 arm64-binaries-r-dep-cache-redhat8 (omit)
redhat-9-amd64 redhat-9 amd64 reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 amd64-binaries-r-dep-cache-redhat9 AX42
redhat-9-arm64 redhat-9 arm64 reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3 arm64-binaries-r-dep-cache-redhat9 (omit)

Key substitution points (8 locations):

  1. cron: value — weekly-rebuild-missing-{suffix}
  2. evaluate: value — 'task == "weekly-rebuild-missing-{suffix}"'
  3. image: — use the Image column
  4. PLATFORM: env var — use the PLATFORM column
  5. ARCH: env var — use the ARCH column
  6. volumes: — {ARCH}-binaries-r-dep-cache-{PLATFORM-without-hyphens}:/mnt/cache
  7. rds_path in the R command — weekly_rebuild_{PLATFORM}_{ARCH}.rds
  8. nodeSelector — kubernetes.io/arch: {ARCH}, plus node.kubernetes.io/instance-type: AX42 only for amd64 (omit the instance-type line entirely for arm64)
  • Step 1: Create all 13 rebuild workflow files

Copy the template from Task 5 and substitute per the table. Example for redhat-9-arm64:

when:
  - event: cron
    cron: weekly-rebuild-missing-redhat-9-arm64
  - event: manual
    evaluate: 'task == "weekly-rebuild-missing-redhat-9-arm64"'

skip_clone: true

steps:
  - name: 'Rebuild missing binaries'
    image: reg.devxy.io/rpkgs/build-env-redhat:9-4.4.3
    pull: true
    environment:
      RED_HAT_DEV_PW:
        from_secret: RED_HAT_DEV_PW
      B2_S3_ACCESS_KEY:
        from_secret: B2_S3_ACCESS_KEY
      B2_S3_SECRET_KEY:
        from_secret: B2_S3_SECRET_KEY
      PGPASS:
        from_secret: PGPASS
      REPO_RO_TOKEN:
        from_secret: REPO_RO_TOKEN
      GITHUB_PAT:
        from_secret: GITHUB_PAT
      GIT_USER: pat-s
      R_PKG_CACHE_DIR: /mnt/cache/pkgcache
      R_LIBS_USER: /mnt/cache/R-pkgs
      CCACHE_DIR: /mnt/cache/ccache
      PLATFORM: redhat-9
      ARCH: arm64
      NCPUS: 2
    volumes:
      - arm64-binaries-r-dep-cache-redhat9:/mnt/cache
    commands:
      - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
      - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages
      - rm -rf /mnt/cache/R-pkgs/00LOCK-*
      - R -q -e 'pak::pak("git::https://codefloe.com/rpkgs/bincraft.git")'
      - R -q -e 'packageVersion("bincraft")'
      - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi
      - $XVFB $XVFB_ARGS -- R -q -e "sink(stdout(), type = 'message'); options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); rds_path <- '/mnt/cache/packages/weekly_rebuild_redhat-9_arm64.rds'; if (!file.exists(rds_path)) { cat('No RDS file found at', rds_path, '- nothing to rebuild\n'); q('no') }; pkgs <- readRDS(rds_path); if (length(pkgs) == 0) { cat('RDS is empty - nothing to rebuild\n'); q('no') }; excluded <- jsonlite::fromJSON('local/excluded-packages.json')[['package']]; pkgs <- setdiff(pkgs, excluded); cat(sprintf('Rebuilding %d packages\n', length(pkgs))); n <- length(pkgs); for (i in seq_along(pkgs)) { x <- pkgs[i]; cat(sprintf('[%d/%d] %s\n', i, n, x)); tryCatch(bincraft::build_binary_package(x, tag_limit = 1L, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE), error = function(e) cat(sprintf('ERROR building %s: %s\n', x, conditionMessage(e)))) }" 2>&1
    backend_options:
      kubernetes:
        resources:
          requests:
            memory: 5Gi
            cpu: 3000m
          limits:
            memory: 18Gi
            cpu: 3000m
        nodeSelector:
          kubernetes.io/arch: arm64
        tolerations:
          - key: 'CI'
            operator: 'Equal'
            value: 'true'
            effect: 'NoSchedule'

Note: arm64 workflows do NOT include node.kubernetes.io/instance-type: AX42 in nodeSelector.

Repeat for all 13 remaining suffixes from the table, substituting the 8 locations.

  • Step 2: Commit
git add .crow/weekly-rebuild-missing-*.yaml
git commit -m "feat: add remaining 13 weekly rebuild workflows for all platform/arch combos"

Task 7: Final verification

  • Step 1: Verify all files exist

Run:

ls -1 local/excluded-packages.json local/weekly-missing-binaries-audit.R
ls -1 .crow/weekly-audit-missing-*.yaml | wc -l
ls -1 .crow/weekly-rebuild-missing-*.yaml | wc -l

Expected: both files exist, 14 audit workflows, 14 rebuild workflows.

  • Step 2: Validate JSON

Run:

python3 -c "import json; json.load(open('local/excluded-packages.json')); print('JSON valid')"

Expected: JSON valid

  • Step 3: Validate YAML syntax

Run:

python3 -c "
import yaml, glob
for f in sorted(glob.glob('.crow/weekly-*-missing-*.yaml') + glob.glob('.crow/weekly-audit-missing-*.yaml') + glob.glob('.crow/weekly-rebuild-missing-*.yaml')):
    yaml.safe_load(open(f))
    print(f'OK: {f}')
"

Expected: all files print OK.

  • Step 4: Verify unique cron names

Run:

grep -h 'cron: weekly-' .crow/weekly-*.yaml | sort | uniq -c | sort -rn | head

Expected: all counts are 1 (no duplicates).

  • Step 5: Verify all 14 platform/arch combos covered

Run:

for suffix in ubuntu-2204-amd64 ubuntu-2204-arm64 ubuntu-2404-amd64 ubuntu-2404-arm64 alpine-321-amd64 alpine-321-arm64 alpine-322-amd64 alpine-322-arm64 alpine-323-amd64 alpine-323-arm64 redhat-8-amd64 redhat-8-arm64 redhat-9-amd64 redhat-9-arm64; do
  test -f ".crow/weekly-audit-missing-${suffix}.yaml" || echo "MISSING audit: ${suffix}"
  test -f ".crow/weekly-rebuild-missing-${suffix}.yaml" || echo "MISSING rebuild: ${suffix}"
done
echo "All checks passed if no MISSING lines above"

Expected: no MISSING lines, just "All checks passed".