fix(build): judge each index by its own source fallbacks (#193)
Some checks failed
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/5 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline was successful
ci/crow/manual/build-all-versions/8 Pipeline is running
ci/crow/manual/build-all-versions/7 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
Some checks failed
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/manual/build-all-versions/5 Pipeline was successful
ci/crow/manual/build-all-versions/6 Pipeline was successful
ci/crow/manual/build-all-versions/8 Pipeline is running
ci/crow/manual/build-all-versions/7 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline failed
ci/crow/manual/build-all-versions/3 Pipeline failed
ci/crow/manual/build-all-versions/2 Pipeline failed
ci/crow/manual/build-all-versions/4 Pipeline failed
## Why #192 made the per-minor objects visible to their pass, and that part works: every slot now reports a populated per-minor cache where it previously reported `0 of 119053`. But the cache-driven skip rate stayed near **zero**, and the builds kept recompiling packages that already exist. Sampling what pipeline 12027 called missing, against the live CDN: ``` orthoDr_0.6.7.tar.gz 4.4/=404 flat=404 <- genuinely missing osmdata_0.2.4.tar.gz 4.4/=200 flat=404 <- already built, rebuilt anyway osmdata_0.2.5.tar.gz 4.4/=200 flat=404 osmdata_0.3.0.tar.gz 4.4/=200 flat=404 osqp_0.6.3.3.tar.gz 4.4/=200 flat=404 outbreaker2_1.1.0.tar.gz 4.4/=200 flat=404 ``` `source_served` is read from the **flat** slot's `PACKAGES.gz` and lists the packages that index serves as CRAN source instead of a binary. It was applied to every path in the listing: ```r existence_cache <- relative_paths[!basename(relative_paths) %in% source_basenames] ``` A package the flat slot serves as source still has a real `4.4/<pkg>.tar.gz`, so dropping that basename deleted a genuine binary from the cache. The overlap with the work list is total rather than partial, which is why the skip rate was ~0% and not merely low: the sensitive candidate list is exactly "sensitive packages with no flat binary", which is the same set this was removing. amd64/resolute listed 21212 per-minor objects and cached only 13572; the missing ~7600 are these. ## What changed - Read each per-minor `PACKAGES.gz` and exclude only that minor's own source fallbacks. - Keep judging flat-slot objects by the flat index. - Log the fallback count per index, so a future collapse is visible rather than silent. All per-minor indexes were confirmed reachable (`4.4`, `4.5`, `4.6` return 200 on the slots checked); an unreadable index falls back to keeping that scope's cache in full, which is the safe direction. ## Verification ``` 4.4 pass sees -> OLD: osqp_0.6.3.3 NEW: osmdata_0.2.4, osqp_0.6.3.3 all assertions PASS ``` Correct in both directions: the old code dropped `4.4/osmdata` (a real binary) *and* kept `4.6/broken` (a real source fallback in the 4.6 index). The new code restores the first and still excludes the second. `prek run -a` passes. Pipelines 12025-12029 were stopped rather than left recompiling what they already have; their uploads persist, so a fresh run inherits them. Reviewed-on: #193
This commit is contained in:
parent
d4f093923a
commit
7e0f3d0ac6
1 changed files with 83 additions and 23 deletions
|
|
@ -154,9 +154,10 @@ index_url <- sprintf(
|
||||||
arch,
|
arch,
|
||||||
codename
|
codename
|
||||||
)
|
)
|
||||||
source_served <- tryCatch(
|
read_source_served <- function(index) {
|
||||||
|
tryCatch(
|
||||||
{
|
{
|
||||||
con_idx <- gzcon(url(index_url, open = "rb"))
|
con_idx <- gzcon(url(index, open = "rb"))
|
||||||
on.exit(close(con_idx), add = TRUE)
|
on.exit(close(con_idx), add = TRUE)
|
||||||
idx <- read.dcf(con_idx, fields = c("Package", "Version", "Built"))
|
idx <- read.dcf(con_idx, fields = c("Package", "Version", "Built"))
|
||||||
sprintf(
|
sprintf(
|
||||||
|
|
@ -167,13 +168,16 @@ source_served <- tryCatch(
|
||||||
},
|
},
|
||||||
error = function(e) {
|
error = function(e) {
|
||||||
cat(sprintf(
|
cat(sprintf(
|
||||||
"WARNING: could not read %s (%s); keeping the full S3 cache\n",
|
"WARNING: could not read %s (%s); keeping that index's cache in full\n",
|
||||||
index_url,
|
index,
|
||||||
conditionMessage(e)
|
conditionMessage(e)
|
||||||
))
|
))
|
||||||
character(0)
|
character(0)
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
source_served <- read_source_served(index_url)
|
||||||
|
|
||||||
binary_cache <- setdiff(file_names, source_served)
|
binary_cache <- setdiff(file_names, source_served)
|
||||||
|
|
||||||
|
|
@ -225,9 +229,65 @@ if (sum(stripped_per_minor) != sum(per_minor_object)) {
|
||||||
))
|
))
|
||||||
}
|
}
|
||||||
source_basenames <- source_served
|
source_basenames <- source_served
|
||||||
existence_cache <- relative_paths[
|
|
||||||
!basename(relative_paths) %in% source_basenames
|
# A source-fallback list describes ONE index, so it may only be applied to that
|
||||||
]
|
# index's objects. Applied to every path it deletes real per-minor binaries from
|
||||||
|
# the cache: a package the flat slot serves as CRAN source still has a genuine
|
||||||
|
# `4.4/<pkg>.tar.gz`, and dropping that name makes every per-minor pass rebuild
|
||||||
|
# it, every run, forever.
|
||||||
|
#
|
||||||
|
# The overlap is total rather than partial, which is why this pinned the skip
|
||||||
|
# rate near 0%: the sensitive candidate list is exactly "sensitive packages with
|
||||||
|
# no flat binary", which is the same set this was removing. amd64/resolute
|
||||||
|
# listed 21212 per-minor objects but cached only 13572, then recompiled osmdata,
|
||||||
|
# osqp and outbreaker2 while their 4.4 binaries sat in the bucket.
|
||||||
|
#
|
||||||
|
# So judge each minor by its own index, and the flat slot by the flat index.
|
||||||
|
is_per_minor_path <- grepl("^[0-9]+\\.[0-9]+/", relative_paths)
|
||||||
|
path_minor <- ifelse(
|
||||||
|
is_per_minor_path,
|
||||||
|
sub("^([0-9]+\\.[0-9]+)/.*$", "\\1", relative_paths),
|
||||||
|
""
|
||||||
|
)
|
||||||
|
minors_present <- sort(unique(path_minor[is_per_minor_path]))
|
||||||
|
per_minor_source <- lapply(minors_present, function(m) {
|
||||||
|
read_source_served(sprintf(
|
||||||
|
"https://cran.rpkgs.com/%s/%s/latest/src/contrib/%s/PACKAGES.gz",
|
||||||
|
arch,
|
||||||
|
codename,
|
||||||
|
m
|
||||||
|
))
|
||||||
|
})
|
||||||
|
names(per_minor_source) <- minors_present
|
||||||
|
cat(sprintf(
|
||||||
|
"Source fallbacks per index: flat=%d%s\n",
|
||||||
|
length(source_basenames),
|
||||||
|
if (length(minors_present)) {
|
||||||
|
paste0(
|
||||||
|
", ",
|
||||||
|
paste(
|
||||||
|
sprintf("%s=%d", minors_present, lengths(per_minor_source)),
|
||||||
|
collapse = ", "
|
||||||
|
)
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
""
|
||||||
|
}
|
||||||
|
))
|
||||||
|
|
||||||
|
served_as_source <- vapply(
|
||||||
|
seq_along(relative_paths),
|
||||||
|
function(i) {
|
||||||
|
scope <- if (nzchar(path_minor[i])) {
|
||||||
|
per_minor_source[[path_minor[i]]]
|
||||||
|
} else {
|
||||||
|
source_basenames
|
||||||
|
}
|
||||||
|
basename(relative_paths[i]) %in% scope
|
||||||
|
},
|
||||||
|
logical(1L)
|
||||||
|
)
|
||||||
|
existence_cache <- relative_paths[!served_as_source]
|
||||||
cat(sprintf(
|
cat(sprintf(
|
||||||
"S3 cache: %d objects, %d served as CRAN source, %d usable binaries\n",
|
"S3 cache: %d objects, %d served as CRAN source, %d usable binaries\n",
|
||||||
length(file_names),
|
length(file_names),
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue