feat(build): skip already-built package versions on workflow restart (#91)
All checks were successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
All checks were successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
## Summary When a `build-all-*` workflow is restarted, the build job re-reads the static `pkgs_to_build.rds` that the install-deps step produced once, so it cycles over every package an interrupted run already built. This adds a DB-based skip filter so a restart only processes what is genuinely left. - At job start, `build-all.R` queries the `single_builds` metadata table for `(name, tag)` already built successfully (`error_occurred = FALSE`) on this `platform`/`arch`, and drops those pairs from the chunk before the build loop. It logs how many it skipped. - One indexed query, one round trip, run before the pak forks — no extra S3 listing and no new Python/s3fs memory pressure (`RPostgres`/`DBI` are already used in the container). - Errored versions are intentionally **not** skipped, so transient failures still get retried on restart. ## Dependency Correctness depends on a `error_occurred = FALSE` row meaning the binary is actually published. That guarantee is added in rpkgs/bincraft#56 (success row written only after a confirmed S3 upload). This PR should land together with / after a bincraft release including that fix. Reviewed-on: #91
This commit is contained in:
parent
2dddc5fe43
commit
1cd86b65e1
1 changed files with 70 additions and 45 deletions
|
|
@ -39,6 +39,31 @@ sprintf("# of package versions for this job: %s", nrow(chunk))
|
|||
exclude <- jsonlite::fromJSON("local/excluded-packages.json")[["package"]]
|
||||
chunk <- chunk[!chunk$Package %in% exclude, ]
|
||||
|
||||
# Skip package versions already built in a previous run.
|
||||
# pkgs_to_build.rds is a static snapshot from the install-deps step, so on a
|
||||
# restart it still lists everything an interrupted run already produced. The
|
||||
# metadata DB reflects that progress, so we re-derive the remaining set here.
|
||||
platform <- paste(Sys.getenv("OS"), gsub("[.]", "", Sys.getenv("OS_VERSION")), sep = "-")
|
||||
arch <- Sys.getenv("ARCH")
|
||||
con <- DBI::dbConnect(
|
||||
RPostgres::Postgres(),
|
||||
dbname = "build_metadata",
|
||||
host = "r-binaries.devxy.io",
|
||||
port = 15432,
|
||||
user = "rpkgs",
|
||||
password = Sys.getenv("PGPASS"),
|
||||
sslmode = "require"
|
||||
)
|
||||
built <- DBI::dbGetQuery(
|
||||
con,
|
||||
"SELECT name, tag FROM single_builds WHERE platform = $1 AND arch = $2 AND error_occurred = FALSE",
|
||||
params = list(platform, arch)
|
||||
)
|
||||
DBI::dbDisconnect(con)
|
||||
before <- nrow(chunk)
|
||||
chunk <- chunk[!paste(chunk$Package, chunk$Version) %in% paste(built$name, built$tag), ]
|
||||
sprintf("Skipped %d already-built package versions; %d remaining for this job", before - nrow(chunk), nrow(chunk))
|
||||
|
||||
# Read pre-computed S3 listing from install-deps step
|
||||
# This avoids loading s3fs/reticulate/Python in the build container,
|
||||
# saving significant memory for pak subprocess forks
|
||||
|
|
|
|||
Loading…
Reference in a new issue