fix(abi): inspect flat objects of packages CRAN has removed #212

Merged

pat-s merged 1 commit from fix/backfill-inspect-archived-packages into main

2026-09-09 18:35:34 +00:00

pat-s commented

2026-09-09 18:27:17 +00:00

Owner

Why

The backfill decided which flat objects were worth inspecting by looking the package up in CRAN's current index:

compiled_names <- cran[NeedsCompilation == "yes", "Package"]
keep <- flat_pkg %in% compiled_names

CRAN lists only what it currently ships. A package CRAN has archived is absent from that index, so it is never recognised as compiled and never inspected, and no verdict is ever recorded for it.

union_index_records() meanwhile reads the slot's own NeedsCompilation, finds yes, and drops the record because the source classifier flags the package and no safe verdict exists to clear it. The repair cannot rescue it either: it works from recorded verdicts, and there is no row.

The result is unrecoverable, and identical in shape to #132 but from a different cause: dropped from every per-minor index, on every re-index, forever.

Evidence

mixedMem and vcpen survived a backfill, two re-indexes and a repair while missing from all three rhel8 per-minor indexes on both architectures:

rhel8 amd64 4.5: missing=3 [mixedMem timbr vcpen]
rhel8 arm64 4.5: missing=2 [mixedMem vcpen]

Neither appears anywhere in either repair log, because neither was ever in the unsafe set. Their per-minor objects are 404 at every minor. And in CRAN's index:

mixedMem  CRAN version=ABSENT  NeedsCompilation=ABSENT
vcpen     CRAN version=ABSENT  NeedsCompilation=ABSENT

Both removed from CRAN. Their flat records in our own index say NeedsCompilation: yes.

What changed

NeedsCompilation now comes from the slot's own flat index, which describes exactly the objects being inspected rather than a different repository's opinion of them.

Fetched from the bucket, not over HTTPS. This runs inside R, so the edge would route the request to a per-minor index and the flat objects would not all be there. Same reason the object listing above it comes from S3 (#203).
Keyed on <package>_<version>, not the package name: a slot can carry several versions and only some of them compile.
Refuses to proceed on an empty read, so an unreadable index cannot silently mean "nothing is compiled" and skip the entire slot.

Effect

Re-running the backfill on a slot will now inspect these objects and record verdicts, after which the index writer keeps the safe ones and the repair can act on the unsafe ones. Already-recorded objects are still skipped, so re-running stays cheap.

## Why The backfill decided which flat objects were worth inspecting by looking the package up in **CRAN's current index**: ```r compiled_names <- cran[NeedsCompilation == "yes", "Package"] keep <- flat_pkg %in% compiled_names ``` CRAN lists only what it currently ships. A package CRAN has **archived** is absent from that index, so it is never recognised as compiled and never inspected, and no verdict is ever recorded for it. `union_index_records()` meanwhile reads the slot's *own* `NeedsCompilation`, finds `yes`, and drops the record because the source classifier flags the package and no safe verdict exists to clear it. The repair cannot rescue it either: it works from recorded verdicts, and there is no row. The result is unrecoverable, and identical in shape to #132 but from a different cause: dropped from every per-minor index, on every re-index, forever. ## Evidence `mixedMem` and `vcpen` survived a backfill, two re-indexes and a repair while missing from all three rhel8 per-minor indexes on **both** architectures: ``` rhel8 amd64 4.5: missing=3 [mixedMem timbr vcpen] rhel8 arm64 4.5: missing=2 [mixedMem vcpen] ``` Neither appears anywhere in either repair log, because neither was ever in the unsafe set. Their per-minor objects are 404 at every minor. And in CRAN's index: ``` mixedMem CRAN version=ABSENT NeedsCompilation=ABSENT vcpen CRAN version=ABSENT NeedsCompilation=ABSENT ``` Both removed from CRAN. Their flat records in our own index say `NeedsCompilation: yes`. ## What changed `NeedsCompilation` now comes from the slot's own flat index, which describes exactly the objects being inspected rather than a different repository's opinion of them. - Fetched **from the bucket**, not over HTTPS. This runs inside R, so the edge would route the request to a per-minor index and the flat objects would not all be there. Same reason the object listing above it comes from S3 (#203). - Keyed on `<package>_<version>`, not the package name: a slot can carry several versions and only some of them compile. - Refuses to proceed on an empty read, so an unreadable index cannot silently mean "nothing is compiled" and skip the entire slot. ## Effect Re-running the backfill on a slot will now inspect these objects and record verdicts, after which the index writer keeps the safe ones and the repair can act on the unsafe ones. Already-recorded objects are still skipped, so re-running stays cheap.

pat-s added 1 commit

2026-09-09 18:27:17 +00:00

fix(abi): inspect flat objects of packages CRAN has removed d9a652904e

The backfill decided which objects were worth inspecting by looking the package
up in CRAN's current index. CRAN lists only what it currently ships, so a
package it has archived is absent, is never recognised as compiled, and is
never inspected. No verdict is ever recorded for it.

`union_index_records()` meanwhile reads the slot's *own* `NeedsCompilation`,
finds `yes`, and drops the record when the source classifier flags the package
and no safe verdict exists to clear it. The repair cannot rescue it either,
because it works from recorded verdicts and there is no row.

The record is therefore dropped from every per-minor index on every re-index,
permanently. `mixedMem` and `vcpen`, both removed from CRAN, were stuck out of
all three rhel8 per-minor indexes on both architectures that way, surviving a
backfill, two re-indexes and a repair.

Read `NeedsCompilation` from the slot's own flat index instead, which describes
exactly the objects being inspected. It is fetched from the bucket rather than
over HTTPS because this runs inside R, where the edge would route the request
to a per-minor index.

Keyed on `<package>_<version>` rather than the package name: a slot can carry
several versions and only some of them compile.

pat-s merged commit 523424bd68 into main

2026-09-09 18:35:34 +00:00

pat-s deleted branch fix/backfill-inspect-archived-packages

2026-09-09 18:35:34 +00:00

pat-s referenced this pull request from a commit

2026-09-09 18:35:35 +00:00

fix(abi): inspect flat objects of packages CRAN has removed (#212)