build-cran-binaries/local/fetch-rebuild-packages-from-issue.R
pat-s 71f3f0c601
Some checks failed
ci/crow/manual/weekly-audit-missing/13 Pipeline was successful
ci/crow/manual/weekly-audit-missing/15 Pipeline was successful
ci/crow/manual/weekly-audit-missing/4 Pipeline was successful
ci/crow/manual/weekly-audit-missing/5 Pipeline was successful
ci/crow/manual/weekly-audit-missing/1 Pipeline was successful
ci/crow/manual/weekly-audit-missing/11 Pipeline was successful
ci/crow/manual/weekly-audit-missing/2 Pipeline was successful
ci/crow/manual/weekly-audit-missing/3 Pipeline was successful
ci/crow/manual/weekly-audit-missing/7 Pipeline was successful
ci/crow/manual/weekly-audit-missing/9 Pipeline was successful
ci/crow/manual/weekly-audit-missing/17 Pipeline was successful
ci/crow/manual/weekly-audit-missing/6 Pipeline was successful
ci/crow/manual/weekly-audit-missing/8 Pipeline was successful
ci/crow/manual/weekly-audit-missing/10 Pipeline was successful
ci/crow/manual/weekly-audit-missing/12 Pipeline was successful
ci/crow/manual/weekly-audit-missing/14 Pipeline was successful
ci/crow/manual/weekly-audit-missing/16 Pipeline was successful
ci/crow/manual/weekly-audit-missing/18 Pipeline was successful
ci/crow/manual/weekly-rebuild-missing/12 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/14 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/16 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/18 Pipeline is pending
ci/crow/manual/weekly-rebuild-missing/1 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/4 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/11 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/15 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/10 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/6 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/8 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/2 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/7 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/5 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/9 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/3 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/17 Pipeline was canceled
ci/crow/manual/weekly-rebuild-missing/13 Pipeline was canceled
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/4 Pipeline was canceled
ci/crow/cron/process-updates/13 Pipeline was canceled
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
fix(audit): replace arch subsection in place instead of appending duplicates (#138)
## Problem

A fresh `weekly-audit-missing` run followed by `weekly-rebuild-missing` reported that almost every package "already exists in the remote bucket", even though the audit had just flagged them as missing.

Root cause: the audit's Forgejo-issue update matched the existing `### <arch>` subsection by its **bare** header (`### arm64`) while writing headers with a ` (N missing, M to rebuild)` suffix.
The equality test never matched, so every run **appended** a new block instead of replacing the old one.
Issue #63 had accumulated 38 arch subsections under `## alpine-323` where 2 are expected (75 total across the alpine platforms; body ~77k lines).

`fetch-rebuild-packages-from-issue.R` reads the **first** matching block, which was the oldest snapshot.
So the rebuild kept re-checking a months-old list (180 packages, mostly already built), while the genuinely-missing packages, ~1921 for alpine-323/arm64 in the freshest block, were never fed to the rebuild and the backlog grew silently.

## Changes

- **`local/weekly-missing-binaries-audit.R`**: match arch subsections by prefix (`^### <arch>( |$)`) and remove **all** blocks for that arch before writing one fresh block. Accumulation now self-heals on every run.
- **`local/dedupe-audit-issue.R`** (new): one-off cleanup that collapses each `## platform` section to the freshest block per arch across the three OS-family issues. Supports `DRY_RUN=1`.
- **`local/fetch-rebuild-packages-from-issue.R`**: prefer the audit's freshly-written RDS (overwritten each run, immune to issue-body drift), falling back to issue parsing when absent.

## Validation

Simulated the dedupe logic against the live #63 body: **75 → 8** arch subsections, body 77k → 28k lines, and the kept alpine-323/arm64 block correctly resolves to `GARCH.X (3.0)` (the stale first block held `2.0`).

## Follow-ups (not in this PR)

- `alpine-321` is audited but has no row in the rebuild matrix (7,230 missing, never rebuilt).
- An `alpine-324` section exists in the issue but is in neither matrix.

Reviewed-on: #138
2026-07-21 08:51:23 +00:00

140 lines
3.8 KiB
R

library(httr2, quietly = TRUE)
forgejo_base <- "https://git.devxy.io/api/v1"
repo <- "devxy/build-cran-binaries"
platform <- Sys.getenv("PLATFORM")
arch <- Sys.getenv("ARCH")
output_file <- Sys.getenv("REBUILD_PKG_LIST", "/tmp/rebuild_pkgs.txt")
if (nchar(platform) == 0) {
stop("PLATFORM env var is not set")
}
if (nchar(arch) == 0) {
stop("ARCH env var is not set")
}
# Prefer the audit's freshly-written RDS. The audit overwrites it each run
# (saveRDS), so unlike the Forgejo issue body it is never subject to the
# duplicate-subsection accumulation bug. Fall back to parsing the issue when the
# RDS is absent (e.g. a fresh runner with no shared cache).
rds_file <- file.path(
Sys.getenv("REBUILD_PKG_RDS_DIR", "/mnt/cache/packages"),
sprintf("weekly_rebuild_%s_%s.rds", platform, arch)
)
if (file.exists(rds_file)) {
pkgs <- tryCatch(as.character(readRDS(rds_file)), error = function(e) NULL)
if (!is.null(pkgs) && length(pkgs) > 0) {
cat(sprintf(
"Using audit RDS %s: %d rebuildable packages for %s/%s\n",
rds_file,
length(pkgs),
platform,
arch
))
writeLines(pkgs, output_file)
cat(sprintf("Wrote package list to %s\n", output_file))
q("no")
}
cat(sprintf(
"RDS %s present but empty/unreadable -- falling back to issue\n",
rds_file
))
}
token <- Sys.getenv("FORGEJO_TOKEN")
if (nchar(token) == 0) {
stop("FORGEJO_TOKEN env var is not set")
}
os_family <- if (grepl("^ubuntu", platform)) {
"Ubuntu"
} else if (grepl("^alpine", platform)) {
"Alpine"
} else if (grepl("^redhat", platform)) {
"Red Hat"
} else {
platform
}
issue_title <- sprintf(
"Missing package binaries for latest version (%s)",
os_family
)
cat(sprintf("Searching for issue: %s\n", issue_title))
search_url <- sprintf(
"%s/repos/%s/issues?type=issues&state=open&q=%s&limit=50",
forgejo_base,
repo,
utils::URLencode(issue_title, reserved = TRUE)
)
search_resp <- request(search_url) |>
req_headers(Authorization = paste("token", token)) |>
req_perform()
issues <- resp_body_json(search_resp, simplifyVector = FALSE)
match_idx <- which(
vapply(issues, function(x) x$title, character(1)) == issue_title
)
if (length(match_idx) == 0) {
cat("No matching issue found - nothing to rebuild\n")
writeLines(character(0), output_file)
q("no")
}
body <- issues[[match_idx[1]]]$body
if (is.null(body) || nchar(body) == 0) {
cat("Issue body is empty - nothing to rebuild\n")
writeLines(character(0), output_file)
q("no")
}
lines <- strsplit(body, "\n", fixed = TRUE)[[1]]
# Find the platform section (## alpine-322)
plat_header <- sprintf("## %s", platform)
plat_idx <- which(lines == plat_header)
if (length(plat_idx) == 0) {
cat(sprintf(
"No section found for platform %s - nothing to rebuild\n",
platform
))
writeLines(character(0), output_file)
q("no")
}
# Find the arch subsection (### arm64 (...))
arch_pattern <- sprintf("^### %s ", arch)
arch_idx <- which(grepl(arch_pattern, lines) & seq_along(lines) > plat_idx[1])
if (length(arch_idx) == 0) {
cat(sprintf(
"No section found for arch %s under %s - nothing to rebuild\n",
arch,
platform
))
writeLines(character(0), output_file)
q("no")
}
ai <- arch_idx[1]
# Find the end of the rebuildable section: stop at ####, ###, ##, ---, or EOF
end_idx <- which(grepl("^####|^###|^##|^---", lines) & seq_along(lines) > ai)
end_idx <- if (length(end_idx) > 0) end_idx[1] - 1 else length(lines)
section_lines <- lines[seq(ai + 1, end_idx)]
# Extract package names from "- PackageName (Version)" lines
pkg_lines <- section_lines[grepl("^- ", section_lines)]
pkgs <- sub("^- ([^ ]+) \\(.*\\)$", "\\1", pkg_lines)
pkgs <- pkgs[nchar(pkgs) > 0 & pkgs != "_None_"]
cat(sprintf(
"Found %d rebuildable packages for %s/%s\n",
length(pkgs),
platform,
arch
))
writeLines(pkgs, output_file)
cat(sprintf("Wrote package list to %s\n", output_file))