fix(build): hold back packages the cran mirror has not picked up yet (#188)
Some checks failed
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline was canceled
ci/crow/manual/build-all-versions/8 Pipeline was canceled
ci/crow/manual/build-all-versions/7 Pipeline was canceled
ci/crow/manual/build-all-versions/5 Pipeline was canceled
ci/crow/manual/build-all-versions/3 Pipeline was canceled
ci/crow/manual/build-all-versions/4 Pipeline was canceled
ci/crow/manual/build-all-versions/2 Pipeline was canceled
ci/crow/manual/build-all-versions/1 Pipeline was canceled
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
Some checks failed
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline was canceled
ci/crow/manual/build-all-versions/8 Pipeline was canceled
ci/crow/manual/build-all-versions/7 Pipeline was canceled
ci/crow/manual/build-all-versions/5 Pipeline was canceled
ci/crow/manual/build-all-versions/3 Pipeline was canceled
ci/crow/manual/build-all-versions/4 Pipeline was canceled
ci/crow/manual/build-all-versions/2 Pipeline was canceled
ci/crow/manual/build-all-versions/1 Pipeline was canceled
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
## Motivation A resolute build aborted on a single package: ``` [23/361] AsyPeer_0.0.1 (r_minor_sensitive=TRUE) Error: GitHub API error (404): Not Found x URL not found: <https://api.github.com/repos/cran/AsyPeer/commits> Retrying in 2 seconds. ... Retrying in 60 seconds. Error in `rate_sleep()`: ! Request failed after 10 attempts. Execution halted ``` This is not rate limiting — it is a **404**. `check_for_binary()` reads the published version from the `cran` GitHub mirror, and that mirror lags CRAN. `AsyPeer 0.0.1` was published today at 13:50 UTC and has no repository there yet. The 404 is permanent, but the call is wrapped in `purrr::insistently` with `max_times = 10` and `pause_cap = 60`, so it retries on a 1/2/4/8/16/32/60/60/60/60 second backoff — about five minutes — and then aborts the whole shard. ## Change Hold back release versions published within `CRAN_MIRROR_LAG_DAYS` (default 3). Deferring them costs nothing: the daily update pipeline builds new and updated packages anyway, and they arrive here on the next run once the mirror has caught up. ## Measured against the live CRAN index | lag | held back | |---|---| | 1 day | 29 of 24831 (0.12%) | | **3 days** | **135 (0.54%)** | | 7 days | 412 (1.66%) | `AsyPeer` is among the 135 at three days. ## Worth doing separately Retrying a 404 at all is wrong — it can never succeed, and any other permanent 404 (a package pulled from the mirror, say) will abort a shard the same way. `check_for_binary()` should distinguish a permanent 404 from a transient failure and, when the mirror simply lacks the package, treat the version as unknown rather than fatal. That is a bincraft change and I have not made it here. Reviewed-on: #188
This commit is contained in:
parent
4bf88ed378
commit
213d30cea4
1 changed files with 29 additions and 2 deletions
|
|
@ -68,9 +68,36 @@ archive_versions <- archive_versions[
|
|||
]
|
||||
|
||||
# Now get release versions (assuming cran_release has Package and Version columns)
|
||||
#
|
||||
# Packages published in the last few days are held back. `check_for_binary()`
|
||||
# reads the published version from the `cran` GitHub mirror
|
||||
# (`GET /repos/cran/<pkg>/commits`), and that mirror lags CRAN: a package that
|
||||
# has just appeared has no repository there yet. The call then 404s, which is
|
||||
# permanent, but it is wrapped in `purrr::insistently` and retried ten times
|
||||
# with a backoff capped at 60s - so one unmirrored package burns about five
|
||||
# minutes and then aborts the whole shard.
|
||||
#
|
||||
# Holding them back costs nothing: the daily update pipeline builds new and
|
||||
# updated packages anyway, and they arrive here on the next run once the mirror
|
||||
# has caught up.
|
||||
mirror_lag_days <- as.numeric(
|
||||
Sys.getenv("CRAN_MIRROR_LAG_DAYS", unset = "3")
|
||||
)
|
||||
published <- as.POSIXct(cran_release$Published, tz = "UTC")
|
||||
too_recent <- !is.na(published) &
|
||||
published > (Sys.time() - mirror_lag_days * 86400)
|
||||
if (any(too_recent)) {
|
||||
message(sprintf(
|
||||
"Holding back %d package(s) published in the last %g day(s); the cran GitHub mirror will not have them yet: %s",
|
||||
sum(too_recent),
|
||||
mirror_lag_days,
|
||||
paste(utils::head(cran_release$Package[too_recent], 10L), collapse = ", ")
|
||||
))
|
||||
}
|
||||
|
||||
release_versions <- data.table(
|
||||
Package = cran_release$Package,
|
||||
Version = as.character(cran_release$Version)
|
||||
Package = cran_release$Package[!too_recent],
|
||||
Version = as.character(cran_release$Version[!too_recent])
|
||||
)
|
||||
|
||||
pkgs_to_build <- unique(rbind(archive_versions, release_versions, fill = TRUE))
|
||||
|
|
|
|||
Loading…
Reference in a new issue