test(verify): report uneven coverage instead of failing on it (#180)
## Motivation
The full 16-slot run came back 139 passed, 5 failed. Four of the five were `coverage uneven across minors` on `resolute` and `alpine324` (both arches) — and they are not defects.
Those slots are built under R 4.5, so their 4.5 union carries ABI-risky packages that only exist as 4.5 builds. A 4.4 or 4.6 client cannot safely load them, which is the whole reason per-minor slots exist. Their absence from the 4.4 and 4.6 indexes is correct behaviour, and failing the run on it blocks four slots that regress nobody.
This is the same mistake as the source-fallback share, which was demoted to a note for the same reason.
## Change
Report uneven coverage; do not fail on it.
The two checks answer different questions and should not share an exit code:
- **`MAX_REGRESSIONS`** gates *enablement*: would routing serve a client source where the generic slot holds a binary of that client's own minor? Must be zero.
- **parity** gates the *claim*: can we advertise full coverage for ABI-sensitive packages? Informative, and currently no.
## Verification
`amd64/resolute` now passes with the shortfall printed as a note:
```
ok amd64/resolute R 4.6: no regression against the generic slot
note: amd64/resolute coverage uneven across minors (vs best 24748): R4.4:-345 R4.6:-352
passed: 8 failed: 0
```
`shellcheck` clean.
Reviewed-on: #180
This commit is contained in:
parent
a3004695a7
commit
d38a4b5746
1 changed files with 9 additions and 5 deletions
|
|
@ -43,10 +43,13 @@ MINORS=${MINORS:-"4.4 4.5 4.6"}
|
|||
EXCLUDED_MINORS=${EXCLUDED_MINORS:-"4.3"}
|
||||
# How many Path: targets to HEAD-check per slot/minor. 0 disables.
|
||||
SAMPLE=${SAMPLE:-5}
|
||||
# Largest package-count shortfall a non-primary minor may have against the best
|
||||
# minor on the same slot before coverage counts as uneven. A slot built under
|
||||
# one R minor carries fewer per-minor binaries for the others; until that gap
|
||||
# closes, "full coverage for ABI-sensitive packages" is not a claim we can make.
|
||||
# Package-count shortfall against the best minor on the same slot, above which
|
||||
# coverage is reported as uneven. Reported, not failed on: the packages a
|
||||
# non-primary minor lacks are ABI-risky ones built under the primary minor,
|
||||
# which a client on another minor cannot safely load anyway, so their absence
|
||||
# is correct. This gates the *claim* ("full coverage for ABI-sensitive
|
||||
# packages"), not whether routing is safe to enable - MAX_REGRESSIONS does
|
||||
# that.
|
||||
PARITY_TOLERANCE=${PARITY_TOLERANCE:-25}
|
||||
# Source fallbacks are reported, not failed on. Since bincraft learned to keep
|
||||
# a matching-minor generic binary out of a fallback's shadow, a remaining
|
||||
|
|
@ -299,7 +302,8 @@ for arch in $ARCHES; do
|
|||
[ "$gap" -gt "$PARITY_TOLERANCE" ] && uneven="$uneven R$minor:-$gap"
|
||||
done
|
||||
if [ -n "$uneven" ]; then
|
||||
bad "$slot coverage uneven across minors (vs best $best):$uneven"
|
||||
printf ' note: %s coverage uneven across minors (vs best %s):%s\n' \
|
||||
"$slot" "$best" "$uneven"
|
||||
else
|
||||
ok "$slot coverage parity across minors (best $best, all within $PARITY_TOLERANCE)"
|
||||
fi
|
||||
|
|
|
|||
Loading…
Reference in a new issue