fix(build): hold back packages the cran mirror has not picked up yet (#188)
Some checks failed
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline was canceled
ci/crow/manual/build-all-versions/8 Pipeline was canceled
ci/crow/manual/build-all-versions/7 Pipeline was canceled
ci/crow/manual/build-all-versions/5 Pipeline was canceled
ci/crow/manual/build-all-versions/3 Pipeline was canceled
ci/crow/manual/build-all-versions/4 Pipeline was canceled
ci/crow/manual/build-all-versions/2 Pipeline was canceled
ci/crow/manual/build-all-versions/1 Pipeline was canceled
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful

## Motivation

A resolute build aborted on a single package:

```
[23/361] AsyPeer_0.0.1 (r_minor_sensitive=TRUE)
Error: GitHub API error (404): Not Found
x URL not found: <https://api.github.com/repos/cran/AsyPeer/commits>
Retrying in 2 seconds. ... Retrying in 60 seconds.
Error in `rate_sleep()`: ! Request failed after 10 attempts.
Execution halted
```

This is not rate limiting — it is a **404**. `check_for_binary()` reads the published version from the `cran` GitHub mirror, and that mirror lags CRAN. `AsyPeer 0.0.1` was published today at 13:50 UTC and has no repository there yet.

The 404 is permanent, but the call is wrapped in `purrr::insistently` with `max_times = 10` and `pause_cap = 60`, so it retries on a 1/2/4/8/16/32/60/60/60/60 second backoff — about five minutes — and then aborts the whole shard.

## Change

Hold back release versions published within `CRAN_MIRROR_LAG_DAYS` (default 3).

Deferring them costs nothing: the daily update pipeline builds new and updated packages anyway, and they arrive here on the next run once the mirror has caught up.

## Measured against the live CRAN index

| lag | held back |
|---|---|
| 1 day | 29 of 24831 (0.12%) |
| **3 days** | **135 (0.54%)** |
| 7 days | 412 (1.66%) |

`AsyPeer` is among the 135 at three days.

## Worth doing separately

Retrying a 404 at all is wrong — it can never succeed, and any other permanent 404 (a package pulled from the mirror, say) will abort a shard the same way. `check_for_binary()` should distinguish a permanent 404 from a transient failure and, when the mirror simply lacks the package, treat the version as unknown rather than fatal. That is a bincraft change and I have not made it here.

Reviewed-on: #188
This commit is contained in:
Patrick Schratz 2026-08-31 17:12:56 +00:00 committed by Patrick Schratz
commit 213d30cea4

View file

@ -68,9 +68,36 @@ archive_versions <- archive_versions[
]
# Now get release versions (assuming cran_release has Package and Version columns)
#
# Packages published in the last few days are held back. `check_for_binary()`
# reads the published version from the `cran` GitHub mirror
# (`GET /repos/cran/<pkg>/commits`), and that mirror lags CRAN: a package that
# has just appeared has no repository there yet. The call then 404s, which is
# permanent, but it is wrapped in `purrr::insistently` and retried ten times
# with a backoff capped at 60s - so one unmirrored package burns about five
# minutes and then aborts the whole shard.
#
# Holding them back costs nothing: the daily update pipeline builds new and
# updated packages anyway, and they arrive here on the next run once the mirror
# has caught up.
mirror_lag_days <- as.numeric(
Sys.getenv("CRAN_MIRROR_LAG_DAYS", unset = "3")
)
published <- as.POSIXct(cran_release$Published, tz = "UTC")
too_recent <- !is.na(published) &
published > (Sys.time() - mirror_lag_days * 86400)
if (any(too_recent)) {
message(sprintf(
"Holding back %d package(s) published in the last %g day(s); the cran GitHub mirror will not have them yet: %s",
sum(too_recent),
mirror_lag_days,
paste(utils::head(cran_release$Package[too_recent], 10L), collapse = ", ")
))
}
release_versions <- data.table(
Package = cran_release$Package,
Version = as.character(cran_release$Version)
Package = cran_release$Package[!too_recent],
Version = as.character(cran_release$Version[!too_recent])
)
pkgs_to_build <- unique(rbind(archive_versions, release_versions, fill = TRUE))