devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 11 Pull requests 1 Releases Activity

test(verify): report uneven coverage instead of failing on it #180

Merged
pat-s merged 1 commit from test/parity-as-note into main 2026-08-31 10:00:50 +00:00
Conversation 1 Commits 1 Files changed 1 +9 -5
pat-s commented 2026-08-31 09:59:58 +00:00
Owner

Motivation

The full 16-slot run came back 139 passed, 5 failed. Four of the five were coverage uneven across minors on resolute and alpine324 (both arches) — and they are not defects.

Those slots are built under R 4.5, so their 4.5 union carries ABI-risky packages that only exist as 4.5 builds. A 4.4 or 4.6 client cannot safely load them, which is the whole reason per-minor slots exist. Their absence from the 4.4 and 4.6 indexes is correct behaviour, and failing the run on it blocks four slots that regress nobody.

This is the same mistake as the source-fallback share, which was demoted to a note for the same reason.

Change

Report uneven coverage; do not fail on it.

The two checks answer different questions and should not share an exit code:

  • MAX_REGRESSIONS gates enablement: would routing serve a client source where the generic slot holds a binary of that client's own minor? Must be zero.
  • parity gates the claim: can we advertise full coverage for ABI-sensitive packages? Informative, and currently no.

Verification

amd64/resolute now passes with the shortfall printed as a note:

ok    amd64/resolute R 4.6: no regression against the generic slot
      note: amd64/resolute coverage uneven across minors (vs best 24748): R4.4:-345 R4.6:-352
passed: 8   failed: 0

shellcheck clean.

## Motivation The full 16-slot run came back 139 passed, 5 failed. Four of the five were `coverage uneven across minors` on `resolute` and `alpine324` (both arches) — and they are not defects. Those slots are built under R 4.5, so their 4.5 union carries ABI-risky packages that only exist as 4.5 builds. A 4.4 or 4.6 client cannot safely load them, which is the whole reason per-minor slots exist. Their absence from the 4.4 and 4.6 indexes is correct behaviour, and failing the run on it blocks four slots that regress nobody. This is the same mistake as the source-fallback share, which was demoted to a note for the same reason. ## Change Report uneven coverage; do not fail on it. The two checks answer different questions and should not share an exit code: - **`MAX_REGRESSIONS`** gates *enablement*: would routing serve a client source where the generic slot holds a binary of that client's own minor? Must be zero. - **parity** gates the *claim*: can we advertise full coverage for ABI-sensitive packages? Informative, and currently no. ## Verification `amd64/resolute` now passes with the shortfall printed as a note: ``` ok amd64/resolute R 4.6: no regression against the generic slot note: amd64/resolute coverage uneven across minors (vs best 24748): R4.4:-345 R4.6:-352 passed: 8 failed: 0 ``` `shellcheck` clean. 
Same mistake as the source-fallback share. The packages a non-primary
minor lacks are ABI-risky ones built under the slot's primary minor,
which a client on another minor cannot safely load: their absence is
correct, not a defect. Failing on it blocked resolute and alpine324 on
both arches while they regressed nobody.

Parity gates the claim we make about coverage; MAX_REGRESSIONS gates
whether routing is safe to enable. Only the second belongs in the exit
code.
pat-s merged commit d38a4b5746 into main 2026-08-31 10:00:50 +00:00
pat-s deleted branch test/parity-as-note 2026-08-31 10:00:50 +00:00
pat-s commented 2026-08-31 10:26:10 +00:00
Author
Owner

Added two harness fixes to this branch, since they are corrections to the same script and splitting them costs a second review for no benefit.

fetch() did not pass -L. The router answers an index request with a redirect, so a --live byte comparison was comparing against a 302 body rather than the index a client actually receives.

fetch() did not bypass the edge cache. A purge is asynchronous, so a run started immediately after a reindex reads whatever the edge still holds. This produced a phantom failure worth recording: amd64/alpine323 R 4.5 reported 161 regressions while its PACKAGES.rds was already correct — the .gz was a stale copy. Re-measuring after the purge settled gave zero.

I chased that phantom through two wrong hypotheses (a mixed-version pipeline, then a real bug in union_index_records()) before fetching the .rds and finding it disagreed with the .gz. A verification tool that races the thing it verifies is worse than no tool, so both fixes belong here.

Added two harness fixes to this branch, since they are corrections to the same script and splitting them costs a second review for no benefit. **`fetch()` did not pass `-L`.** The router answers an index request with a redirect, so a `--live` byte comparison was comparing against a 302 body rather than the index a client actually receives. **`fetch()` did not bypass the edge cache.** A purge is asynchronous, so a run started immediately after a reindex reads whatever the edge still holds. This produced a phantom failure worth recording: `amd64/alpine323` R 4.5 reported 161 regressions while its `PACKAGES.rds` was already correct — the `.gz` was a stale copy. Re-measuring after the purge settled gave zero. I chased that phantom through two wrong hypotheses (a mixed-version pipeline, then a real bug in `union_index_records()`) before fetching the `.rds` and finding it disagreed with the `.gz`. A verification tool that races the thing it verifies is worse than no tool, so both fixes belong here.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries!180
No description provided.