fix(build): judge each index by its own source fallbacks #193

Merged

pat-s merged 1 commit from fix/scope-source-fallback-per-minor into main

2026-09-02 09:13:55 +00:00

pat-s commented

2026-09-02 09:13:15 +00:00

Owner

Why

#192 made the per-minor objects visible to their pass, and that part works: every slot now reports a populated per-minor cache where it previously reported 0 of 119053.

But the cache-driven skip rate stayed near zero, and the builds kept recompiling packages that already exist. Sampling what pipeline 12027 called missing, against the live CDN:

orthoDr_0.6.7.tar.gz      4.4/=404  flat=404   <- genuinely missing
osmdata_0.2.4.tar.gz      4.4/=200  flat=404   <- already built, rebuilt anyway
osmdata_0.2.5.tar.gz      4.4/=200  flat=404
osmdata_0.3.0.tar.gz      4.4/=200  flat=404
osqp_0.6.3.3.tar.gz       4.4/=200  flat=404
outbreaker2_1.1.0.tar.gz  4.4/=200  flat=404

source_served is read from the flat slot's PACKAGES.gz and lists the packages that index serves as CRAN source instead of a binary. It was applied to every path in the listing:

existence_cache <- relative_paths[!basename(relative_paths) %in% source_basenames]

A package the flat slot serves as source still has a real 4.4/<pkg>.tar.gz, so dropping that basename deleted a genuine binary from the cache.

The overlap with the work list is total rather than partial, which is why the skip rate was ~0% and not merely low: the sensitive candidate list is exactly "sensitive packages with no flat binary", which is the same set this was removing. amd64/resolute listed 21212 per-minor objects and cached only 13572; the missing ~7600 are these.

What changed

Read each per-minor PACKAGES.gz and exclude only that minor's own source fallbacks.
Keep judging flat-slot objects by the flat index.
Log the fallback count per index, so a future collapse is visible rather than silent.

All per-minor indexes were confirmed reachable (4.4, 4.5, 4.6 return 200 on the slots checked); an unreadable index falls back to keeping that scope's cache in full, which is the safe direction.

Verification

4.4 pass sees -> OLD: osqp_0.6.3.3          NEW: osmdata_0.2.4, osqp_0.6.3.3
all assertions PASS

Correct in both directions: the old code dropped 4.4/osmdata (a real binary) and kept 4.6/broken (a real source fallback in the 4.6 index). The new code restores the first and still excludes the second.

prek run -a passes.

Pipelines 12025-12029 were stopped rather than left recompiling what they already have; their uploads persist, so a fresh run inherits them.

## Why #192 made the per-minor objects visible to their pass, and that part works: every slot now reports a populated per-minor cache where it previously reported `0 of 119053`. But the cache-driven skip rate stayed near **zero**, and the builds kept recompiling packages that already exist. Sampling what pipeline 12027 called missing, against the live CDN: ``` orthoDr_0.6.7.tar.gz 4.4/=404 flat=404 <- genuinely missing osmdata_0.2.4.tar.gz 4.4/=200 flat=404 <- already built, rebuilt anyway osmdata_0.2.5.tar.gz 4.4/=200 flat=404 osmdata_0.3.0.tar.gz 4.4/=200 flat=404 osqp_0.6.3.3.tar.gz 4.4/=200 flat=404 outbreaker2_1.1.0.tar.gz 4.4/=200 flat=404 ``` `source_served` is read from the **flat** slot's `PACKAGES.gz` and lists the packages that index serves as CRAN source instead of a binary. It was applied to every path in the listing: ```r existence_cache <- relative_paths[!basename(relative_paths) %in% source_basenames] ``` A package the flat slot serves as source still has a real `4.4/<pkg>.tar.gz`, so dropping that basename deleted a genuine binary from the cache. The overlap with the work list is total rather than partial, which is why the skip rate was ~0% and not merely low: the sensitive candidate list is exactly "sensitive packages with no flat binary", which is the same set this was removing. amd64/resolute listed 21212 per-minor objects and cached only 13572; the missing ~7600 are these. ## What changed - Read each per-minor `PACKAGES.gz` and exclude only that minor's own source fallbacks. - Keep judging flat-slot objects by the flat index. - Log the fallback count per index, so a future collapse is visible rather than silent. All per-minor indexes were confirmed reachable (`4.4`, `4.5`, `4.6` return 200 on the slots checked); an unreadable index falls back to keeping that scope's cache in full, which is the safe direction. ## Verification ``` 4.4 pass sees -> OLD: osqp_0.6.3.3 NEW: osmdata_0.2.4, osqp_0.6.3.3 all assertions PASS ``` Correct in both directions: the old code dropped `4.4/osmdata` (a real binary) *and* kept `4.6/broken` (a real source fallback in the 4.6 index). The new code restores the first and still excludes the second. `prek run -a` passes. Pipelines 12025-12029 were stopped rather than left recompiling what they already have; their uploads persist, so a fresh run inherits them.

pat-s added 1 commit

2026-09-02 09:13:15 +00:00

fix(build): judge each index by its own source fallbacks 0940f004d4

The source-fallback list is read from the flat slot's PACKAGES.gz and names the
packages that index serves as CRAN source rather than as a binary. It was then
applied to every path in the listing, including per-minor ones.

A package the flat slot serves as source still has a genuine `4.4/<pkg>.tar.gz`,
so removing that basename deleted a real binary from the cache and made every
per-minor pass rebuild it, every run, forever.

The overlap with the work list is total, not partial: the sensitive candidate
list is exactly "sensitive packages with no flat binary", which is the same set
this removed. That pinned the cache-driven skip rate near zero even after #192
made the per-minor objects visible. amd64/resolute listed 21212 per-minor
objects but cached only 13572, then recompiled osmdata, osqp and outbreaker2
while their 4.4 binaries sat in the bucket returning HTTP 200.

This change will:

- read each per-minor index and exclude only that minor's own source fallbacks
- keep judging flat-slot objects by the flat index
- report the fallback count per index, so a collapse is visible in the log

pat-s merged commit 7e0f3d0ac6 into main

2026-09-02 09:13:55 +00:00

pat-s deleted branch fix/scope-source-fallback-per-minor

2026-09-02 09:13:55 +00:00

pat-s referenced this pull request from a commit

2026-09-02 09:13:57 +00:00

fix(build): judge each index by its own source fallbacks (#193)