988 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
automation-bot	5dad51c76a	chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.9.10	2026-10-12 00:32:11 +00:00
pat-s	d992a2b636	fix(ci): give each xvfb-run its own display (#230) ... Every `process-cran-updates-ubuntu-2204-arm64` cron run since pipeline 13744 has failed with `xvfb-run: error: Xvfb failed to start` (e.g. 13829). ## Cause The newer-R loop added in #216 runs two `xvfb-run` calls back to back: the R-minor-sensitive `process_cran_updates` pass, then `local/build-newer-r.R`. Without `-a`, `xvfb-run` always uses display `:99`, and on exit it kills its Xvfb without waiting for it. The second call can therefore start while the old server still holds the `:99` lock, so the new Xvfb refuses to start. Only ubuntu-2204-arm64 hits this because it is the only image with a second R minor installed, so the loop body only runs there. ## Change - Default `XVFB_ARGS` to `-a` (`--auto-servernum`) in `process-updates.yaml`, `build-all-versions.yaml` and `weekly-rebuild-missing.yaml`, so each call picks a free display. - The RHEL 10 `xwfb-run` branch still overrides `XVFB_ARGS` with `-c weston`, so it is unchanged. ## Verification `crow lint .crow/` passes. A local repro was not possible (`docker run` hangs on the dev host), so the next ubuntu-2204-arm64 cron run is the real check. Reviewed-on: #230	2026-10-11 12:12:28 +00:00
automation-bot	3426d5f59f	chore(deps): update dependency nbafrank/uvr to v0.4.7	2026-10-11 00:32:41 +00:00
pat-s	40a0a6e02d	fix(patches): cover every RcppParallel 6.x release with the bundled-TBB patch (#228)	2026-10-04 07:35:01 +00:00
automation-bot	317990a40f	chore(deps): update pre-commit hook posit-dev/air-pre-commit to v0.12.0	2026-10-02 00:32:22 +00:00
automation-bot	5231758c00	chore(deps): update terraform bunnynet to v0.19.1	2026-09-25 00:32:19 +00:00
automation-bot	b18c21aa8d	chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.9.9	2026-09-24 00:32:00 +00:00
automation-bot	e8b0df0c91	chore(deps): update terraform bunnynet to ~> 0.19	2026-09-23 00:32:05 +00:00
pat-s	3962ed6d61	fix(installer): install gettext headers on Alpine (#222)	2026-09-22 12:17:52 +00:00
automation-bot	aa6e5e9dd8	chore(deps): update pre-commit hook davidanson/markdownlint-cli2 to v0.23.3	2026-09-21 00:31:54 +00:00
automation-bot	212065f5a4	chore(deps): update pre-commit hook editorconfig-checker/editorconfig-checker to v4.0.2	2026-09-20 00:31:53 +00:00
automation-bot	caefd1131f	chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.9.8	2026-09-19 00:32:12 +00:00
automation-bot	3325a7bda2	chore(deps): update pre-commit hook rbubley/mirrors-prettier to v3.9.7	2026-09-17 00:32:19 +00:00
automation-bot	b0bade879b	chore(deps): lock file maintenance	2026-09-14 00:32:09 +00:00
pat-s	6b764d2014	fix(build): build packages requiring newer R on compatible minors (#216)	2026-09-13 09:35:50 +00:00
pat-s	d80d3e814e	feat(slots): add a retirement workflow and a drift check (#214) ... ## Why Retiring a slot means three separate things in three separate places, and nothing tied them together. Alpine 3.22 is what that costs. It was removed from `local.rpkgs_slots` in `cdn.tf` at some point, so the edge stopped routing it, but it kept: - a matrix row in **twelve** pipelines - **six** cron jobs on the Crow server - roughly **50000 objects** in the bucket Nightly builds kept producing binaries nobody could reach. Worse, an unrouted slot is not inert: every request falls through to the flat index, so R 4.4 and R 4.6 clients got `LOAD_FAILED` at `dyn.load()` rather than simply nothing. It took a fleet-wide `verify_routing=all` run to notice. ## What changed **`local/check-slots.R`** (`just check-slots`) fails when any `.crow/*.yaml` still targets a slot missing from `local.rpkgs_slots`. `cdn.tf` stays the source of truth; the checker reads it rather than restating the list. **`local/retire-slot.R`** (`just retire-slot <codename>`) does the cleanup: - strips the slot from every pipeline matrix, dropdown option and `OS_VERSION` tag, and moves a dropdown default that pointed at the retired tag - deletes `amd64/<codename>/` and `arm64/<codename>/` from the bucket - prints the cron jobs that are left, with the API call, since they live on the Crow server and cannot be removed from here Dry run by default. It refuses any slot still listed in `cdn.tf`, so it cannot be pointed at a live slot however it is invoked. **`.crow/retire-slot.yaml`** runs the asset half in CI, where the B2 credentials exist. **README** gains a `## Retiring a slot` section with the ordering. Step 4 before step 2 is the one ordering that actively breaks things: the next nightly rebuilds everything just deleted. ## Verification Both scripts were run against the tree as it stood at `523424b`, immediately before #213. The checker names exactly the twelve files that PR edited by hand: ``` .crow/archive-missed-packages.yaml targets unpublished: alpine322 .crow/backfill-flat-safety.yaml targets unpublished: alpine-322 .crow/process-updates.yaml targets unpublished: alpine-322, alpine322 ... 12 file(s) target a slot the edge does not route. ``` `retire-slot.R alpine322 --config-only --apply` reproduces that commit's structural changes byte for byte across all 18 pipeline files, and additionally catches the two `OS_VERSION` dropdowns. The only differences from the hand-made commit are two prose lines, a usage example and a dropdown `description:`, which the script deliberately leaves alone. Guards exercised: a live slot (`alpine323`) is refused, an unknown codename is rejected by name, and the asset half refuses to run without `B2_S3_ACCESS_KEY` rather than treating B2's empty unauthenticated listing as an already-clean slot. `crow lint .crow/` reports all 19 configs valid. `air` and `jarl` are clean on both new scripts. ## Next `alpine321` and `alpine322` still hold objects in the bucket (22922 and 25308 records respectively, both architectures). Once this merges they can be removed with the pipeline this PR adds. Reviewed-on: #214	2026-09-11 13:14:17 +00:00
pat-s	416a2d64d9	chore(alpine): drop the deprecated Alpine 3.22 slot (#213) ... ## Why Alpine 3.22 is deprecated and should be dropped. It was never a published slot to begin with. `local.rpkgs_slots` in `cdn.tf` enumerates eight distros and alpine322 is not one of them, so `UNION_SLOTS` never gated per-minor rewriting for it. The website's `DistroId` union lists the same eight. Every request to the slot therefore fell through to the flat index, which holds R 4.5 builds. ## Evidence The fleet-wide routing verification (pipeline 12591, `verify_routing=all`) measured this end to end. Across all 16 published slots, both architectures, three R minors and both contractual URL forms: **1056 of 1056 install-and-load checks passed.** Every failure in the run was alpine322, identical on amd64 and arm64: ``` amd64/alpine322 R 4.4 via arch|generic|flat: 8 of 11 amd64/alpine322 R 4.6 via arch|generic|flat: 10 of 11 arm64/alpine322 R 4.4 via arch|generic|flat: 8 of 11 arm64/alpine322 R 4.6 via arch|generic|flat: 10 of 11 ``` The tell is that all three forms agree. On a routed slot `arch` and `generic` diverge from `flat`, because that divergence *is* the routing. On alpine322 there is nothing to diverge from. R 4.5 clients were 11 of 11 on every form, because flat happens to be what they need. The data itself is not broken, only unreachable. `alpine322/contrib/4.4/data.table_1.18.6.1.tar.gz` is genuinely `Built: R 4.4.3` and the 4.6 object is `Built: R 4.6.0`. Correct objects, correct per-minor indexes, never served. ## What changed - alpine-322 matrix rows and dropdown options removed from the 12 pipelines that carried them, so `=all` no longer fans out to a slot nobody can reach. - `3.22` removed from the `OS_VERSION` dropdowns in `build-all-versions.yaml` and `repair-built-stamp.yaml`; the latter's default moves to `3.23`. - Stale `3.22` examples updated in `local/packages-to-build.R`. `crow lint .crow/` reports all 18 configs valid. ## Also done outside this diff The six alpine-322 cron jobs are deleted from the Crow server, since a cron whose matrix row is gone keeps firing against nothing: ``` 5 process-cran-updates-alpine-322-amd64 0 5/8 * * * 6 process-cran-updates-alpine-322-arm64 30 5/8 * * * 68 weekly-audit-missing-alpine-322-amd64 20 0 * * 0 69 weekly-audit-missing-alpine-322-arm64 30 0 * * 0 82 weekly-rebuild-missing-alpine-322-amd64 0 4 * * 0 83 weekly-rebuild-missing-alpine-322-arm64 10 4 * * 0 ``` They are listed here so they can be recreated if this is ever reversed. ## Not done The published objects and indexes under `amd64/alpine322` and `arm64/alpine322` are left in the bucket. Deleting them is a separate decision, and nothing routes to them in the meantime. Reviewed-on: #213	2026-09-11 07:27:03 +00:00
pat-s	523424bd68	fix(abi): inspect flat objects of packages CRAN has removed (#212) ... ## Why The backfill decided which flat objects were worth inspecting by looking the package up in **CRAN's current index**: ```r compiled_names <- cran[NeedsCompilation == "yes", "Package"] keep <- flat_pkg %in% compiled_names ``` CRAN lists only what it currently ships. A package CRAN has **archived** is absent from that index, so it is never recognised as compiled and never inspected, and no verdict is ever recorded for it. `union_index_records()` meanwhile reads the slot's *own* `NeedsCompilation`, finds `yes`, and drops the record because the source classifier flags the package and no safe verdict exists to clear it. The repair cannot rescue it either: it works from recorded verdicts, and there is no row. The result is unrecoverable, and identical in shape to #132 but from a different cause: dropped from every per-minor index, on every re-index, forever. ## Evidence `mixedMem` and `vcpen` survived a backfill, two re-indexes and a repair while missing from all three rhel8 per-minor indexes on **both** architectures: ``` rhel8 amd64 4.5: missing=3 [mixedMem timbr vcpen] rhel8 arm64 4.5: missing=2 [mixedMem vcpen] ``` Neither appears anywhere in either repair log, because neither was ever in the unsafe set. Their per-minor objects are 404 at every minor. And in CRAN's index: ``` mixedMem CRAN version=ABSENT NeedsCompilation=ABSENT vcpen CRAN version=ABSENT NeedsCompilation=ABSENT ``` Both removed from CRAN. Their flat records in our own index say `NeedsCompilation: yes`. ## What changed `NeedsCompilation` now comes from the slot's own flat index, which describes exactly the objects being inspected rather than a different repository's opinion of them. - Fetched **from the bucket**, not over HTTPS. This runs inside R, so the edge would route the request to a per-minor index and the flat objects would not all be there. Same reason the object listing above it comes from S3 (#203). - Keyed on `<package>_<version>`, not the package name: a slot can carry several versions and only some of them compile. - Refuses to proceed on an empty read, so an unreadable index cannot silently mean "nothing is compiled" and skip the entire slot. ## Effect Re-running the backfill on a slot will now inspect these objects and record verdicts, after which the index writer keeps the safe ones and the repair can act on the unsafe ones. Already-recorded objects are still skipped, so re-running stays cheap. Reviewed-on: #212	2026-09-09 18:35:33 +00:00
pat-s	03e007069a	fix(ci): do not print a missing timing as "NAs" (#210) ... ## Why A `MISSING_FROM_INDEX` row has no install to time, and `sprintf("%.1fs", NA_real_)` renders it as the literal `NAs`: ``` ROW arch amd64 alpine323 4.5 Rcpp MISSING_FROM_INDEX |NAs ``` Read quickly, `NAs` looks like a unit rather than a missing measurement. ## What changed Rows without a timing print the `built` field alone. Every other row is unchanged. ## Where it showed up Pipeline 12492, the deliberate failure-path check against a slot known to have gaps. That run is otherwise a good result: it caught `Rcpp` missing from the alpine323 4.5 and 4.6 indexes on both contractual URL forms, and it caught the reverse-direction ABI break on the flat slot (objects built under R 4.5.3 failing `dyn.load()` under 4.4), which no previous reprex had surfaced. Reviewed-on: #210	2026-09-09 15:01:29 +00:00
pat-s	a84cca1949	fix(abi): prune the objects recorded broken, not every risky package (#211) ... ## Why A dry run of the prune on `ubuntu-2404-amd64` (pipeline 12494): ``` Dry run: 3013 object(s) left in place. Re-run with --apply to delete. ``` 3013 objects on one slot. The recorded verdicts say roughly **230** of that slot's objects actually fail `dyn.load()` anywhere. The gap is the selector: the prune used `abi_cache_risky_packages()`, the source classifier, which calls **93%** of compiled packages risky where the binaries' own symbols say **4.3%**. Applying it would have deleted ~2800 working binaries and turned each into a source fallback for every client that cannot be routed by R minor. This is the same mistake #129 fixed in the index writer. The prune was written before the per-object verdicts existed (#200, before #128/#129) and was never brought forward. ## What changed - Selection comes from `abi_flat_safety`, every `NOT safe` row for the slot regardless of which minor it fails under, since the flat object is served to clients on all of them. - Matching is by **object**, `<package>_<version>`, not by package name. A package can have a safe flat binary at one version and an unsafe one at another; the old code deleted both. - The explicit `[pkg ...]` override is unchanged in behaviour and still matches by package name, which is what you want when naming a package by hand. - The refusal guard moves with the selector. An empty result now means the slot's flat-safety backfill never ran — the exact state rhel8 was found in today — and pruning nothing while reporting success would have hidden it. ## Not changed Still a dry run unless `PRUNE_APPLY=yes`, and still refuses to delete an object whose package CRAN no longer ships, since there would be no source to fall back to. ## Ordering note A prune converts each deleted binary into a source fallback. bincraft #131 fixes source fallbacks of *archived* versions being advertised as binaries, which breaks `uvr` outright. That should land before any large prune, or this will manufacture new instances of it. Reviewed-on: #211	2026-09-09 15:01:21 +00:00
pat-s	af4cc6f414	feat(ci): verify what clients actually receive from every published slot (#209) ... ## Why Everything built so far reasons about the *binaries*: `abi_flat_safety` records what an object's symbols say it can load under, and the index writer drops records on that verdict. None of it proves what a **client** actually receives, which is the product of the CDN rewrite, the per-minor indexes and the `Path:` field acting together. The reprex that found the `base64enc` failure ran on a workstation, could only reach amd64, and did not survive the session. This makes it a pipeline. ## What it does For every slot, under every R minor the image carries, it installs a fixed package set through each URL form the website documents and checks that each one loads in a fresh subprocess. | form | repository | user agent | | --- | --- | --- | | `arch` | `https://cran.rpkgs.com/<arch>/<codename>/latest` | R's own | | `generic` | `https://cran.rpkgs.com/` | the string the website's snippet builds | | `flat` | the arch URL | `curl/8.5.0`, so the edge cannot read an R minor | `arch` and `generic` are contractual, so a failure there fails the run. `flat` measures what a client hiding its R minor is exposed to: it is reported and never fatal, because pruning the flat slot is a separate decision. Two details that matter: - **CRAN is deliberately absent from `repos`.** A package missing from our index must fail loudly rather than be quietly compiled from source, which would hide exactly the gap this exists to find. - **One process per form.** `available.packages()` caches its index in `tempdir()` keyed by repository URL, and `arch` and `flat` share a URL. In a single process the second form would read the first form's index and prove nothing. ## Verification Run against the live repository in `build-env-ubuntu:noble` under R 4.6.0, mid-repair: ``` amd64/noble R 4.6 via arch: 11 of 11 packages usable amd64/noble R 4.6 via generic: 11 of 11 packages usable amd64/noble R 4.6 via flat: 8 of 11 packages usable ``` The three `flat` failures are `Rcpp`, `data.table` and `yaml`, each `Built: R 4.4.3` and each failing `dyn.load()` under 4.6. That is the unrouted-client exposure, measured rather than estimated. The rows also show the routed forms serving a mix of per-minor and flat objects (`Rcpp` at `R 4.6.0`, `stringi` at `R 4.4.3`), which is the intended behaviour: only objects that cannot load cross-minor get a per-minor copy. Reviewed-on: #209	2026-09-09 12:52:16 +00:00
pat-s	4342fa773a	fix(abi): do not error on the nothing-to-repair path (#208) ... ## Why Pipeline 12480 shows this for slots with nothing to do: ``` Error in sprintf("%s/%s: %d objects recorded, none unsafe. Nothing to repair.", : ``` `RPostgres` returns `count(*)` as `integer64`, and `%d` rejects it. So a slot in the **best possible state** — verdicts recorded, nothing unsafe — failed its pipeline step, while slots with real work to do succeeded. Exactly the wrong signal. ## What changed Formats the count as a string. One line, plus a comment recording why `%d` is wrong here. Reviewed-on: #208	2026-09-09 10:55:15 +00:00
pat-s	e3b7212eaf	feat(abi): skip objects that already have the per-minor builds they need (#207) ... ## Why The fleet repair (12410) was interrupted after 14 of 18 slots, having done: ``` 2522 objects enumerated | 2568 uploads | 1451 symbol overrides ``` Resuming would repeat all of it. The verdict in `abi_flat_safety` describes the **flat** object, and building a per-minor copy does not change that object, so the repair list is identical on the next run. Roughly 2500 objects of duplicated compute. ## What changed `list-unsafe-flat-objects.R` now drops rows that are already repaired: an object counts as repaired once a per-minor build exists for **every** minor that cannot load the flat one. - The bucket is the source of truth, listed directly (it cannot be routed). - Where `unsupported` is empty — unsafe without a recorded reason — a build under every minor the image carries is required, rather than assuming one suffices. - It reports how many were skipped, so an unexpectedly short list is visible rather than silent. - It **fails when the bucket cannot be listed**, because an empty listing would make every object look unrepaired and silently restore the full rebuild. ## Verification The predicate, exercised directly: ``` Rcpp needs[4.4,4.6] -> SKIP (repaired) Rcpp needs[4.4,4.5,4.6] -> REBUILD (4.5 missing) yaml needs[4.4] -> SKIP (repaired) lobstr needs[] -> REBUILD (no recorded minors: all required) ``` ## Note ~70 packages fail to build regardless — `glmmTMB`, `tmbstan`, `OpenMx`, `Cyclops`, `RSiena`, `pdftools`, `odbc` among them — all on `uvr sync --install-system-deps`. They keep their unsafe flat binaries and stay broken for the minors they cannot load under. That is a missing-system-dependency problem in the build images, not an ABI one, and needs separate attention. Reviewed-on: #207	2026-09-09 10:33:41 +00:00
pat-s	ee4562d44c	feat(abi): repair from the recorded verdicts rather than an ad-hoc scan (#206) ... ## Why The reprex found 44 routed-path failures: `Rcpp`, `Matrix` and `RJSONIO` served to R 4.4 from 4.5 builds. Two causes, both now fixed (#205 here, #129 in bincraft) — but the repair itself also needs redoing, and the lists it ran from no longer exist. Those lists came from a one-off local symbol scan, seeded from a candidate set derived on a single slot. That made them a lower bound by construction: the scan found 846 package-slot pairs, while inspecting every object found **4077** unsafe. `abi_flat_safety` now holds the authoritative answer for every slot, written by the flat-safety backfill from the binaries themselves: ``` 95418 inspected, 91325 safe, 4077 unsafe (4.3%) ``` against the 93% of compiled packages the source classifier suspects. ## What it does - `local/list-unsafe-flat-objects.R` prints the objects recorded as unable to load under some minor, for one slot. - The workflow rebuilds each under **every** R minor installed in the image, with **no `--sensitive-only` gate**. Since bincraft v5.3.0 the destination is decided by the built binary's symbols, so a package needing a per-minor slot lands there and a portable one lands in the flat slot. Gating on the source classifier is exactly what let `Rcpp` slip through. ## Safety It **fails loudly when a slot has no verdicts at all**, rather than printing an empty list. An empty repair list is indistinguishable from "nothing to repair" and would read as success while doing nothing — the same shape of silent failure as the missing `PGPASS`. `crow lint` passes. ## After this Run it fleet-wide, re-index, then re-run the reprex. The routed paths are the ones that must come back clean; the `flat` form's failures are the separate prune question. Reviewed-on: #206	2026-09-07 18:46:50 +00:00
pat-s	cceda03935	fix(ci): do not gate the targeted repair on the source classifier (#205) ... ## Why The repair workflow ran `build-one.R --sensitive-only`. That flag skips any version the **source** classifier does not call r-minor-sensitive — and it misses `Rcpp`. So the repair skipped `Rcpp` on every slot, leaving its 4.5-built flat binary as the only copy: ``` flat 200 built under R 4.5.3 4.4/ 404 4.5/ 404 4.6/ 404 ``` Every R 4.4 client on those slots then failed at `dyn.load()`. A real install check across 8 slots x 3 minors x 3 repository forms found **44 such failures**, `Rcpp`, `Matrix` and `RJSONIO` among them. ## What changed Drops `--sensitive-only`. Since bincraft v5.3.0 the destination is decided by the built binary's symbols in `upload_single_binary()`, so building unconditionally is safe and correct: a package that needs a per-minor slot lands there, one that does not lands in the flat slot. Skipping before the build is exactly what removed that decision from the path. `crow lint` passes. ## Companion bincraft #129 is the other half: the index build could not compensate either, because it only considered dropping records whose *package* was flagged risky, and `Rcpp` is not — even though the backfill had recorded that object as unable to load under 4.4. Reviewed-on: #205	2026-09-07 06:20:04 +00:00
pat-s	f3984f337d	fix(ci): authenticate uvr's GitHub resolution in the index workflows (#204) ... ## Why The post-backfill re-index (12257) reported failure. Five slots could not write their indexes: ``` WARN Failed to fetch git Remote pat-s/cranlike@s3 (GitHub API error: 403 Forbidden); falling back to registry resolution ! Attempt 1/3 to update PACKAGES failed (unused argument (built = built)). Retrying in 5s... Error: Failed to update PACKAGES after 3 attempts. Last error: unused argument (built = built) ``` uvr resolves bincraft's dependencies through the GitHub API, including the `pat-s/cranlike@s3` fork. Unauthenticated, that is rate limited. uvr does not fail on the 403 — it **warns and falls back to the registry copy of cranlike**, which has no `built` parameter, so `update_PACKAGES(built = ...)` dies three retries later. Affected: alpine323, rhel8, rhel9, rhel10, noble. The remaining 13 slots resolved the fork fine and indexed correctly, which is what makes this insidious: the same pipeline passes or fails on identical input depending on the rate-limit window. `build-all-versions.yaml` and `rebuild-abi-gap.yaml` already pass `GITHUB_PAT`, which is why they were never affected. ## What changed Adds the `GITHUB_PAT` secret to `reindex.yaml`, `weekly-rebuild-reindex.yaml` and `backfill-flat-safety.yaml`, with a comment recording why its absence is quiet rather than loud. `crow lint` passes on all three. ## Context The re-index itself did what it was meant to. Drops per per-minor index fell from ~1800-2200 to: ``` 4.4: avg 129 4.5: avg 172 4.6: avg 145 ``` so the per-object safety verdicts recovered roughly 93% of the coverage while still dropping every object that genuinely cannot load. This PR is about the five slots that never got to apply it. Reviewed-on: #204	2026-09-05 15:59:16 +00:00
pat-s	844ad9749b	fix(abi): enumerate the flat slot from S3, not through the edge (#203) ... ## Why The first backfill run reported: ``` amd64/resolute: 27105 flat binaries, 8187 compiled and worth inspecting Done: 300 inspected, 7 safe for every minor, 1 unsafe, 292 skipped ``` 292 of 300 skipped, and `27105` is larger than the flat index (~24k). Both have the same cause: the script read `PACKAGES.gz` over HTTPS from inside R, which sends an R User-Agent, so **the edge routed it to a per-minor index**. It then enumerated union records carrying `Path: <minor>`, whose objects do not exist at the flat path, and every download 404'd. An ironic failure given what this whole change is about, and one the trial run existed to catch. ## What changed - List the flat slot with `s3fs::s3_dir_ls()`, keeping only objects with no `<minor>/` prefix and nothing under `Archive/`. The bucket cannot be routed and is authoritative about what is actually in the flat slot. - Fetch each object from S3 too. The edge does not rewrite tarball requests, but it does cache them, and the verdict is about the object. - `NeedsCompilation` still comes from CRAN's own index, which is not behind our edge. ## Verification The next trial run is the check: `flat objects` should now match the flat index rather than exceed it, and `skipped` should be near zero. Reviewed-on: #203	2026-09-05 09:41:57 +00:00
pat-s	c216e44578	feat(abi): backfill per-object flat-slot safety verdicts (#202) ... ## Why bincraft #128 lets `union_index_records()` keep a carried flat record when that specific object is recorded safe. On its own it changes nothing for the records already being dropped, because they are legacy objects with no verdict: ``` 4.4: 16398 dropped across 9 indexes (~1822 each) 4.5: 20076 dropped across 9 indexes (~2230 each) 4.6: 15532 dropped across 9 indexes (~1726 each) ``` Every one of those resolves to CRAN source for clients the routing serves correctly. This backfill inspects the objects already in the bucket and writes their verdicts, which is what actually restores the coverage. ## How the verdict is made By symbols, not heuristics. A binary fails at `dyn.load()` under minor M exactly when it references a symbol M does not export, and the volatile set is derived by diffing the `libR.so` exports of the R builds installed in the image — so a new R release updates it with no curated list to maintain. That matters here because the source-level classifier calls **93%** of compiled packages risky (`LinkingTo: Rcpp` alone is enough), while a symbol scan of the same slot found **17** packages that break under 4.6 and **72** under 4.4. Recording the real answer per object is what makes the difference between ~2000 dropped records and the few dozen that genuinely need dropping. ## Safety - **Refuses to run where fewer than two R minors are installed.** With nothing to compare, every verdict would read "safe" — writing a bucket full of false clearances is far worse than writing none. - Skips packages with no compiled code: they cannot fail this way and are never dropped, so downloading them to prove it is wasted. - Resumable; already-recorded objects cost one query. `crow lint` passes. ## Sequencing Merge and release bincraft #128 first, then run this per slot, then re-index. Verifying the drop counts fall is the point of the exercise, so that re-index is where the fix is confirmed rather than assumed. Reviewed-on: #202	2026-09-05 09:35:47 +00:00
pat-s	e1114bc145	fix(ci): give the index writers the metadata database password (#201) ... ## Why Pipeline 12207 logs one of these per slot: ``` Error: connection to server at "r-binaries.devxy.io" (116.203.165.191), port 15432 failed: fe_sendauth: no password supplied ``` bincraft's index build calls `abi_cache_risky_packages()` to decide which packages may be carried from the flat slot into a per-minor index. `reindex.yaml` and `weekly-rebuild-reindex.yaml` never passed `PGPASS`, so the query cannot connect. It fails soft by design, which is what makes this worth fixing rather than tolerating: the run still reports **success** while using an empty cache-derived risky set. The index is then written as though no package were known risky from the classifier, which is the exact state the cache was added to fix. `build-all-versions.yaml` already passes `PGPASS`, which is why classification behaved correctly there and silently degraded here. ## What changed Adds the `PGPASS` secret to both index-writing workflows, with a comment explaining why its absence is invisible rather than loud. `crow lint` passes on both files. ## Note The tarball-derived half of the risky set still worked, so indexes written without this are not wrong so much as incomplete: they miss packages that are known risky from classification but have no per-minor tarball yet. A re-index after this merges applies the full set. Reviewed-on: #201	2026-09-04 19:26:55 +00:00
pat-s	f9ebb3dba4	fix(abi): prune flat-slot binaries of ABI-risky packages (#200) ... ## Why `base64enc` still fails to install under R 4.6 even after it was built into the 4.6 slot: ``` base64enc.so: undefined symbol: SETLENGTH ``` The per-minor binary is correct and serves correctly to any client the router can identify. But the **flat slot still holds a binary built under R 4.4.3**, and that is what every client receives when it cannot be routed by R minor. That object should not exist. `upload_single_binary()` writes an ABI-risky package only to the per-minor path: ```r if (is_r_minor_sensitive) { remote_tarball_path <- file.path(remote_bin_path, minor_version, tarball_name) } else { remote_tarball_path <- file.path(remote_bin_path, tarball_name) } ``` `base64enc`'s flat binary is a legacy artifact from before it was classified risky. Removing it makes the outcome correct regardless of routing, user agent, downloader or renv configuration, which is the part I could not otherwise guarantee. ## What it does `local/prune-stale-flat-binaries.R` lists a slot's flat objects, intersects them with the risky set, and deletes them. Deleting is safe here: the source fallback republishes the CRAN source under the same key, so the package stays installable everywhere and compiles against the client's own R, while routed clients keep getting the per-minor binary. Safety posture, given this deletes production objects: - **dry run by default**; nothing is removed without `--apply` - takes an **explicit package list**, so the first runs need not trust the classifier cache at all - **refuses to proceed on an empty risky set**, which is indistinguishable from an unreachable cache and would otherwise report success having pruned nothing - **keeps objects whose package CRAN no longer ships**, since there would be no source to fall back to and the package would vanish - **re-indexes and purges in the same run**. This is not optional: a published index naming deleted files serves 404s, which is worse than the binary it replaced. ## Use ```sh # dry run first crow pipeline create devxy/build-cran-binaries \ --var prune_flat=ubuntu-2604-amd64 --var PRUNE_PACKAGES="base64enc lobstr" # then apply crow pipeline create devxy/build-cran-binaries \ --var prune_flat=ubuntu-2604-amd64 --var PRUNE_PACKAGES="base64enc lobstr" \ --var PRUNE_APPLY=yes ``` `crow lint` passes. ## Sequencing Run with an explicit list first, on one slot, and confirm the flat URL then serves source while the per-minor URLs still serve binaries. Only widen to the cache-derived set once the backfill has finished and its output looks sane: the first 250 classifications came back 232 r-minor-sensitive, and a rate that high suggests the source-grep classifier is broader than the set that actually breaks, which is worth checking before pruning thousands of flat binaries. Reviewed-on: #200	2026-09-04 10:54:09 +00:00
pat-s	853b924765	feat(abi): backfill classifier verdicts for pre-per-minor packages (#199) ... ## Why bincraft PR #124 makes the `abi_classification` cache authoritative for the risky-package set, breaking the circularity where a package counted as ABI-risky only once it had already been built per-minor. On its own that changes nothing for the packages actually at risk. The cache only holds verdicts for packages that passed through the build pipeline after the per-minor scheme existed; everything older has no verdict, and no verdict means the flat binary is carried into every per-minor index. That is exactly how a `base64enc` built under R 4.4.3 reached an R 4.6.1 client: ``` base64enc.so: undefined symbol: SETLENGTH ``` This is the backfill that gives those packages a verdict. ## What it does `local/backfill-abi-classification.R` enumerates a slot's flat index and runs each compiled package through `bincraft::classify_r_minor_sensitive()`, which looks the verdict up in the cache, and on a miss downloads the CRAN source, classifies it and stores the result. The verdict is a property of the source, independent of OS, arch and R minor, so **one run populates the cache for every slot**. The arch/codename arguments only choose which flat index to enumerate. Only `NeedsCompilation=yes` packages are classified: a package with no compiled code has no symbols to bind and cannot fail this way, so proving it by download would be wasted work. On amd64/resolute that is 4585 packages rather than 23218. Re-running is cheap and resumable: cached verdicts return without touching the network, so a second run costs one query per package and downloads only the remainder. ## Verification The enumeration was run against the live indexes and matches an independent symbol scan of the same slot exactly: ``` flat binaries: 23218 | due classification: 4585 base64enc included: TRUE lobstr included: TRUE ``` 4585 is the same population I scanned with `nm -D --undefined-only` when establishing the blast radius, and both packages confirmed to break under 4.6 are in it. `crow lint` passes. ## Sequencing Merge after bincraft #124 and after a bincraft release the pipelines pick up, otherwise the verdicts are written but nothing reads them yet. Writing them early is harmless. Then re-index the slots so the per-minor indexes are rebuilt with the widened risky set. Expect coverage in the per-minor indexes to *drop* slightly as newly-recognised risky packages are dropped rather than served from a mismatched minor; those then need per-minor builds, either through the normal pipeline or the targeted workflow in devxy/build-cran-binaries#198. Reviewed-on: #199	2026-09-04 10:34:49 +00:00
pat-s	26406a3e86	feat(ci): add a targeted per-R-minor rebuild workflow (#198) ... ## Why A user hit this on amd64/resolute under R 4.6.1: ``` base64enc.so: undefined symbol: SETLENGTH ``` R 4.6 no longer exports `SETLENGTH`: ``` R 4.4: missing -> R_ClosureEnv R_ClosureFormals R_MakeMissingBinding R_PromiseEnv R_PromiseValue R_mkClosure R 4.5: missing -> R_MakeMissingBinding R_PromiseEnv R_PromiseValue R 4.6: missing -> R_PromiseEnv R_PromiseValue SETLENGTH ``` `base64enc` calls `SETLENGTH` (`base64.c:206`, `253`), its flat binary was built under R 4.4.3, and it has no per-minor copy at all, so the union index hands that binary to a 4.6 client. The safety net missed it because the risky set is defined circularly: ```r risky_packages_across_minors <- function(...) { # a package is "risky" iff it already has a tarball in some per-minor slot ``` A package counts as ABI-risky only if it has *already* been built per-minor. `base64enc` never was, and having no new CRAN version it never enters the build list to be classified. So the per-minor protection only covers packages that passed through the pipeline after the scheme existed. ## Scope Scanning all 4,585 compiled binaries in amd64/resolute's flat index for symbols R 4.6 does not export found 10 referencing `SETLENGTH`. Most already have per-minor builds and are served correctly. Two do not: ``` base64enc_0.1-6 flat=200 4.4=404 4.5=404 4.6=404 lobstr_1.2.1 flat=200 4.4=404 4.5=200 4.6=404 ``` (`iotools` lacks a 4.5 copy but 4.5 still exports `SETLENGTH`, so it does not break.) ## What this adds Rebuilding one package into one minor was only possible with `just rebuild` from a workstation holding the S3 and Postgres credentials. This adds the CI path: ```sh crow pipeline create devxy/build-cran-binaries \ --var rebuild_abi_gap=ubuntu-2604-amd64 \ --var ABI_PACKAGES="base64enc@0.1-6 lobstr@1.2.1" \ --var ABI_R_MINOR=4.6 ``` - reuses `reindex.yaml`'s slot matrix verbatim, so the slot-to-image mapping cannot drift - runs `build-one.R --sensitive-only`, which reclassifies each version and skips anything not ABI-risky rather than writing it to the wrong slot - mounts no shared cache volume, so it cannot disturb the build pipeline's per-agent caches - attempts every package, then exits non-zero if any failed `crow lint` passes. ## Not in this PR This is the remediation path, not the fix. The circular `risky_packages_across_minors()` definition is still in place, so new instances of this class can still appear. That is the follow-up. Reviewed-on: #198	2026-09-04 10:24:47 +00:00
automation-bot	d3f24c89df	chore(deps): update pre-commit hook editorconfig-checker/editorconfig-checker to v4 (#197) ... This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [editorconfig-checker/editorconfig-checker](https://github.com/editorconfig-checker/editorconfig-checker) | repository | major | `v3.11.3` → `v4.0.1` | Note: The `pre-commit` manager in Renovate is not supported by the `pre-commit` maintainers or community. Please do not report any problems there, instead [create a Discussion in the Renovate repository](https://github.com/renovatebot/renovate/discussions/new) if you have any questions. --- ### Release Notes <details> <summary>editorconfig-checker/editorconfig-checker (editorconfig-checker/editorconfig-checker)</summary> ### [`v4.0.1`](https://github.com/editorconfig-checker/editorconfig-checker/releases/tag/v4.0.1) [Compare Source](https://github.com/editorconfig-checker/editorconfig-checker/compare/v4.0.0...v4.0.1) ##### Bug Fixes - bump golang.org/x/mod to v0.40.0 ([3d2215c](3d2215c10a)) - **ci:** build docker images for all our platforms ([a34a6ee](a34a6eeac7)) - update pre-commit-config to use v4 of our repo ([fcd68bf](fcd68bffda)) ### [`v4.0.0`](https://github.com/editorconfig-checker/editorconfig-checker/releases/tag/v4.0.0) [Compare Source](https://github.com/editorconfig-checker/editorconfig-checker/compare/v3.11.3...v4.0.0) ##### ⚠ BREAKING CHANGES - internalize our packages ([#&#8203;619](https://github.com/editorconfig-checker/editorconfig-checker/issues/619))<br>It will no longer be possible to import our internal packages in your code. If you rely on one of our packages please let us know by opening an issue. - migrate to goreleaser dockers\_v2. ([#&#8203;617](https://github.com/editorconfig-checker/editorconfig-checker/issues/617))<br>This means we stop publishing the arch-suffixed images like `3.11.2-amd64` or `3.11.2-arm64v8`. All these architectures are still provided in the multiarch tag `3.11.2`. - rename ec to editorconfig-checker ([#&#8203;371](https://github.com/editorconfig-checker/editorconfig-checker/issues/371)) - `.ecrc` is no longer discovered as a default config file name. Rename `.ecrc` to `.editorconfig-checker.json`, or pass the file explicitly with `--config .ecrc`. - Docker image tags no longer carry a leading `v`. Pull `3.12.0`, `3.12` and `3` instead of `v3.12.0`, `v3.12` and `v3` - the deprecated `SpacesAftertabs` configuration key has been removed, use `SpacesAfterTabs` instead. - stop accepting latin1 files as utf-8. ([#&#8203;598](https://github.com/editorconfig-checker/editorconfig-checker/issues/598))<br>Previously `charset=utf-8` was satisfied with files encoded as `latin1`. While most of the `latin1` range is identical to `utf-8`, not all is. If you request `utf-8` but have `latin1` encoded files expect these files to be reported as violations now. ##### Features - add safe EditorConfig autofixes ([#&#8203;612](https://github.com/editorconfig-checker/editorconfig-checker/issues/612)) ([2f4962c](2f4962c596)) - auto-detect `github-actions` output format in CI ([#&#8203;606](https://github.com/editorconfig-checker/editorconfig-checker/issues/606)) ([271c500](271c500a31)) - remove support for the deprecated `.ecrc` config file name ([#&#8203;603](https://github.com/editorconfig-checker/editorconfig-checker/issues/603)) ([2062751](2062751696)) ##### Bug Fixes - add missing space in wrong indentation type error message ([#&#8203;610](https://github.com/editorconfig-checker/editorconfig-checker/issues/610)) ([e0f1b56](e0f1b561a2)) - allow builds with Go 1.26 ([#&#8203;615](https://github.com/editorconfig-checker/editorconfig-checker/issues/615)) ([343018a](343018a4a3)) - consolidate error blocks that are interleaved with other errors ([#&#8203;601](https://github.com/editorconfig-checker/editorconfig-checker/issues/601)) ([76b6313](76b6313bc2)) - correct end line in codeclimate formatter output ([#&#8203;602](https://github.com/editorconfig-checker/editorconfig-checker/issues/602)) ([2476e7c](2476e7c671)) - correct the -disable-end-of-line flag description ([#&#8203;600](https://github.com/editorconfig-checker/editorconfig-checker/issues/600)) ([1615efc](1615efcaed)) - drop a path whose content type could not be read ([#&#8203;599](https://github.com/editorconfig-checker/editorconfig-checker/issues/599)) ([46995b1](46995b1db0)) - handle `unset` for `end_of_line` and `insert_final_newline` ([#&#8203;594](https://github.com/editorconfig-checker/editorconfig-checker/issues/594)) ([123ba31](123ba319dc)), closes [#&#8203;580](https://github.com/editorconfig-checker/editorconfig-checker/issues/580) - remove ambiguity between potential folder and file matches ([#&#8203;611](https://github.com/editorconfig-checker/editorconfig-checker/issues/611)) ([4c4f7d4](4c4f7d4c30)) - remove leading `v` from Docker image tags ([#&#8203;607](https://github.com/editorconfig-checker/editorconfig-checker/issues/607)) ([3827a6c](3827a6c595)) - remove retired goreportcard badge ([2a9dd32](2a9dd32f6e)) - stop accepting latin1 files as utf-8. ([#&#8203;598](https://github.com/editorconfig-checker/editorconfig-checker/issues/598))\<br>Previously `charset=utf-8` was satisfied with files encoded as `latin1`. While most of the `latin1` range is identical to `utf-8`, not all is. If you request `utf-8` but have `latin1` encoded files expect these files to be reported as violations now. ([4d84a77](4d84a778c9)) - typo in config, `SpacesAftertabs` => `SpacesAfterTabs` ([#&#8203;604](https://github.com/editorconfig-checker/editorconfig-checker/issues/604)) ([bbc3f03](bbc3f03507)) ##### Miscellaneous Chores - internalize our packages ([#&#8203;619](https://github.com/editorconfig-checker/editorconfig-checker/issues/619))\<br>It will no longer be possible to import our internal packages in your code. If you rely on one of our packages please let us know by opening an issue. ([d42bcb7](d42bcb7af0)) - rename ec to editorconfig-checker ([#&#8203;371](https://github.com/editorconfig-checker/editorconfig-checker/issues/371)) ([0e3b27f](0e3b27f7e2)) ##### Continuous Integration - migrate to goreleaser dockers\_v2. ([#&#8203;617](https://github.com/editorconfig-checker/editorconfig-checker/issues/617))\<br>This means we stop publishing the arch-suffixed images like `3.11.2-amd64` or `3.11.2-arm64v8`. All these architectures are still provided in the multiarch tag `3.11.2`. ([d4c1442](d4c14421f8)) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM (`* 0-3 * * *`) - Automerge - Between 12:00 AM and 03:59 AM (`* 0-3 * * *`) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yODMuMCIsInVwZGF0ZWRJblZlciI6IjQzLjI4OC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> Reviewed-on: #197	2026-09-04 09:51:46 +00:00
automation-bot	e4ba623ddc	chore(deps): update pre-commit hook editorconfig-checker/editorconfig-checker to v3.11.3	2026-09-04 00:32:36 +00:00
pat-s	2c9bd598fb	fix(ci): install RPostgres in build shards (#195) ... Reviewed-on: #195	2026-09-03 20:09:20 +00:00
pat-s	a650f1d18a	fix(build): scope the errored-package filter to the snapshot's R minor (#194) ... ## Why After #192 and #193 the builds stopped recompiling what they already had, but every slot still showed a per-minor gap that a successful run did not close: ``` arm64/resolute 3302 amd64/resolute 1783 amd64/alpine324 1163 arm64/alpine324 1104 arm64/rhel8 572 amd64/rhel8 532 amd64/rhel10 453 arm64/rhel10 176 (others <= 13) ``` amd64/alpine324 had just completed successfully, yet 655 ABI-risky packages were still absent from its 4.6 index. Sampling 25 of them against that run's log: ``` attempted: 0 errored: 0 uploaded: 0 never attempted: 25 ``` They were filtered out before the build loop. The candidate list drops every package with an error row for this platform/arch: ```sql SELECT name, tag FROM single_builds WHERE error_occurred = TRUE and platform='%s' and arch='%s' ``` There is no `r_version` in that predicate, and the row is keyed by `(name, tag, platform, arch)` only. So a single failure under one minor cancels the package's builds under every other minor, permanently. The index data matches that shape rather than a build-failure shape: of the 508 ABI-risky packages missing from alpine324's 4.4 index, **505 are also missing from 4.6** - a 99% overlap. Genuine per-minor failures would not correlate that way (only 3 were 4.4-specific). ## What changed Adds the `substring(r_version from '^[0-9]+[.][0-9]+') = '<minor>'` scoping already used by `check_package_error()` (#119) and the per-minor error query (#189). Errors specific to another minor are still respected downstream: `filter_packages_with_errors()` runs per pass with that pass's own `r_version`, so a package that genuinely fails under 4.6 is still skipped there. ## Verification Rendered query: ```sql SELECT name, tag FROM single_builds WHERE error_occurred = TRUE and platform='alpine-324' and arch='amd64' and substring(r_version from '^[0-9]+[.][0-9]+') = '4.5' ``` Routing itself is unaffected and remains green: `verify-r-minor-routing.sh` reports 232 passed / 0 failed across all 16 slots, and `--live` reported 328 passed / 0 failed. Reviewed-on: #194	2026-09-03 13:37:04 +00:00
pat-s	7e0f3d0ac6	fix(build): judge each index by its own source fallbacks (#193) ... ## Why #192 made the per-minor objects visible to their pass, and that part works: every slot now reports a populated per-minor cache where it previously reported `0 of 119053`. But the cache-driven skip rate stayed near **zero**, and the builds kept recompiling packages that already exist. Sampling what pipeline 12027 called missing, against the live CDN: ``` orthoDr_0.6.7.tar.gz 4.4/=404 flat=404 <- genuinely missing osmdata_0.2.4.tar.gz 4.4/=200 flat=404 <- already built, rebuilt anyway osmdata_0.2.5.tar.gz 4.4/=200 flat=404 osmdata_0.3.0.tar.gz 4.4/=200 flat=404 osqp_0.6.3.3.tar.gz 4.4/=200 flat=404 outbreaker2_1.1.0.tar.gz 4.4/=200 flat=404 ``` `source_served` is read from the **flat** slot's `PACKAGES.gz` and lists the packages that index serves as CRAN source instead of a binary. It was applied to every path in the listing: ```r existence_cache <- relative_paths[!basename(relative_paths) %in% source_basenames] ``` A package the flat slot serves as source still has a real `4.4/<pkg>.tar.gz`, so dropping that basename deleted a genuine binary from the cache. The overlap with the work list is total rather than partial, which is why the skip rate was ~0% and not merely low: the sensitive candidate list is exactly "sensitive packages with no flat binary", which is the same set this was removing. amd64/resolute listed 21212 per-minor objects and cached only 13572; the missing ~7600 are these. ## What changed - Read each per-minor `PACKAGES.gz` and exclude only that minor's own source fallbacks. - Keep judging flat-slot objects by the flat index. - Log the fallback count per index, so a future collapse is visible rather than silent. All per-minor indexes were confirmed reachable (`4.4`, `4.5`, `4.6` return 200 on the slots checked); an unreadable index falls back to keeping that scope's cache in full, which is the safe direction. ## Verification ``` 4.4 pass sees -> OLD: osqp_0.6.3.3 NEW: osmdata_0.2.4, osqp_0.6.3.3 all assertions PASS ``` Correct in both directions: the old code dropped `4.4/osmdata` (a real binary) *and* kept `4.6/broken` (a real source fallback in the 4.6 index). The new code restores the first and still excludes the second. `prek run -a` passes. Pipelines 12025-12029 were stopped rather than left recompiling what they already have; their uploads persist, so a fresh run inherits them. Reviewed-on: #193	2026-09-02 09:13:54 +00:00
pat-s	d4f093923a	fix(build): strip the s3 scheme so per-minor objects reach their pass (#192) ... ## Why #191 gave the existence cache a per-minor view of the slot, but the paths it filters never carried a minor prefix, so the fix could not take effect. `s3fs::s3_dir_ls()` returns keys with the `s3://` scheme attached: ``` s3://devxy-rpkgs-binaries/amd64/resolute/latest/src/contrib/4.4/oeli_0.7.6.tar.gz ``` The contrib prefix was stripped as a *fixed substring*, so it was removed from the middle of the key and the scheme survived: ``` s3://4.4/oeli_0.7.6.tar.gz ``` That leading `s3://` defeats both `^<minor>/` and `^[0-9]+\.[0-9]+/`, so all 21212 per-minor objects on amd64/resolute were classified as flat-slot objects. Pipeline 12011 shows it exactly: ``` sensitive pass: 0 files (0 of 119053 objects apply to this pass) primary pass: 119053 files (119053 of 119053 objects apply to this pass) ``` Each sensitive pass therefore ran with an empty cache and recompiled all 10248 sensitive packages it already had. ## What changed - `local/packages-to-build.R`: anchor the contrib prefix and swallow an optional `s3://` with it. - `local/packages-to-build.R`: abort when the strip leaves fewer per-minor paths than the listing held. The failure mode is silent and only surfaces as a multi-hour rebuild, and both counts derive from the same listing so they must agree exactly. - `local/packages-to-build.R` / `local/build-all.R`: mark the cache with a `slot_relative` attribute and read that, instead of sniffing for a `/`. A slot-relative cache for a slot with no per-minor or Archive object holds bare names too, and would have been misread as legacy and used unfiltered, which makes a per-minor pass believe the flat slot's binaries are its own and build nothing. - `scripts/purge_cdn_zone.sh`: indent the jq continuation lines by a multiple of two. This is unrelated, but it fails editorconfig-checker on `main` and blocks `prek run -a` for everyone. jq ignores the whitespace, and both response shapes still resolve. ## Verification Replaying the real key shapes through the old and new code: ``` sensitive(4.6) primary OLD paths: 0 8 <- reproduces production NEW paths: 3 3 Invariant NEW: listing=4 stripped=4 -> PASS Invariant OLD: listing=4 stripped=0 -> ABORT (would have caught this) ``` Per-minor `Archive/` objects are attributed to their minor, and the flat bucket keeps its own `Archive/`. `prek run -a` passes. Pipelines 12011-12015 were stopped rather than left to spend hours recompiling what they already had. Their uploads are not lost, so a fresh run inherits them. Reviewed-on: #192	2026-09-02 07:11:10 +00:00
pat-s	8f97cf99ef	fix(build): give the existence cache a per-minor view of the slot (#191) ... ## Motivation A restarted run recompiled everything it had already built. ``` attempted: 8683 already exists in S3: 3 actually compiled: 8683 ``` `amd64/resolute` had uploaded roughly 3000 binaries before being restarted, and rebuilt all of them. ## Cause This is a regression from #189, which I introduced. `s3_cache.rds` is derived from the same `file_names` that #189 filtered: ```r binary_cache <- setdiff(file_names, source_served) saveRDS(binary_cache, "/mnt/cache/packages/s3_cache.rds") ``` Removing per-minor objects from `file_names` was right for the candidate list and wrong for the existence cache. `build_binary_package()` was handed a cache containing no per-minor object at all, so its "not present in the remote bucket" check answered not-present for every one of them. The two consumers need opposite views of one listing: - the **candidate list** must ignore per-minor objects, or a package present under one minor prunes itself from every other minor's work; - the **existence cache** must not ignore them, or the work is redone. Conflating them was wrong in both directions: before #189 a per-minor pass believed the flat slot's binaries were its own and built nothing; after #189 it believed it had nothing and rebuilt the lot. ## Change The cache keeps paths relative to the slot, and `build-all.R` selects the part matching the pass it is running. `build_binary_package()` compares basenames and cannot distinguish `4.4/curl_1.0.tar.gz` from `curl_1.0.tar.gz`, so the choice has to be made where the running R minor is known. Verified: | pass | selects | |---|---| | primary (writes flat) | `curl_1.0`, `jsonlite_2.0`, `Archive/curl_0.9`, `PACKAGES.gz` | | `--sensitive-only` under 4.4 | `curl_1.0`, `rlang_1.3.0` (from `4.4/` only) | | `--sensitive-only` under 4.6 | `rlang_1.3.0` (from `4.6/` only) | | legacy basename cache | passthrough, unfiltered | `Archive/` counts as present for the primary pass: those are versions built and later superseded. ## Backwards compatibility A cache written before this change holds bare basenames. Filtering those by path would select nothing and trigger the very rebuild this prevents, so they are detected and used unfiltered; #190's staleness check replaces them on the next pipeline. ## Note on the running builds The five pipelines currently running are recompiling packages they already have. Their output is still correct — the binaries carry the right `Built` stamp for their slot — but the work is wasted. They should be restarted once this is merged. Reviewed-on: #191	2026-09-01 21:54:13 +00:00
pat-s	642e07e1d6	fix(build): recompute a stale package snapshot, not just a missing one (#190) ... ## Motivation `arm64/alpine324` reported nothing to build while thousands were missing: ``` line 49: Precomputed 7192 package versions (6871 r-minor-sensitive) <- install-deps agent line 99: Total# of remaining package versions: 43 (sensitive_only=TRUE) <- a build shard line 101: Skipped 0 package versions already attempted under R 4.4; 0 remaining ``` Both numbers come from the **same pipeline**. The same run's index step dropped 2407 packages as missing for 4.4 and 2436 for 4.6. ## Cause ```r if (!all(file.exists(package_cache_files))) { ... recompute ... } ``` Existence is not freshness. The snapshot describes S3 and CRAN state when it was written, and the cache volume is per-agent — the file's own comment says so. An agent that ran an earlier pipeline keeps serving that pipeline's answer forever, and no later fix to how the snapshot is computed (#189) can reach it. ## Change Recompute when the snapshot is stale as well as when it is missing. Keyed on the pipeline when the CI exposes an identifier (`CI_PIPELINE_NUMBER`, `CI_BUILD_NUMBER`, `CI_PIPELINE_ID`), so a new pipeline recomputes once per agent and its shards then share the result. Off CI, or when none is set, an age check with a two hour default (`PACKAGE_SNAPSHOT_TTL_HOURS`). ## Verification | scenario | decision | |---|---| | files missing | RECOMPUTE | | same pipeline id | reuse | | **new pipeline id** | **RECOMPUTE** | | no CI var, recent file | reuse | | no CI var, aged out | RECOMPUTE | I could not confirm which identifier Crow actually sets — none is referenced anywhere in this repo — so all three are tried and the age check backs them up. If none is present the behaviour is the age path, which is still correct, just coarser. Reviewed-on: #190	2026-08-31 22:11:17 +00:00
pat-s	94e6c697cf	fix(build): stop per-minor objects masking the per-minor candidate list (#189) ... ## Motivation `arm64/alpine324` (pipeline 11953) finished in minutes having uploaded 57 packages, and reported: ``` Skipped 0 package versions already attempted under R 4.4; 0 remaining Skipped 0 package versions already attempted under R 4.6; 3 remaining ``` The same run's index step dropped **2407** packages as missing for 4.4 and **2436** for 4.6. Nothing to build, and thousands missing — the candidate list is wrong. ## Two omissions **1. Per-minor objects mask the per-minor candidates.** ```r s3_pkgs <- s3fs::s3_dir_ls(".../latest/src/contrib", recurse = TRUE) file_names <- basename(s3_pkgs) ``` `recurse = TRUE` walks `4.4/`, `4.5/`, `4.6/`; `basename()` throws the directory away. `4.5/curl_1.0.tar.gz` and `curl_1.0.tar.gz` collapse to one name, so a package present under **any** R minor counts as built for **all** of them — pruning exactly the packages a per-minor pass exists to build. Per-minor objects are now excluded, and presence in a specific minor is decided downstream where the running R version is known: `build-all.R` filters on it, and `build_binary_package()` checks the per-minor path per package and skips what is already there. `Archive/` is kept. Those are versions built and later superseded; dropping them would make every archived version look unbuilt. Validated against real path shapes: | path | | |---|---| | `curl_1.0.tar.gz` | keep | | `4.4/curl_1.0.tar.gz` | exclude | | `4.6/rlang_1.3.0.tar.gz` | exclude | | `Archive/curl/curl_0.9.tar.gz` | keep | | `PACKAGES.gz` | keep | **2. The error query ignores `r_version`.** ```sql SELECT error_occurred FROM single_builds WHERE name = $1 AND tag = $2 AND platform = $3 AND arch = $4 ``` A failure under the primary minor drops the package from every other minor's candidate list. This is the same omission fixed in `local/build-all.R` (#187) and in bincraft's `check_package_error()` (rpkgs/bincraft#119). This is the third and last consumer of that table — I have grepped the rest; `bincraft::R/cran-internal.R` also reads it, but to list packages present rather than to skip, where the R minor does not apply. ## Expected effect The per-minor passes get real candidate lists. Expect slots that reported "0 remaining" to report thousands, and correspondingly long runs. There is a cost: the list is no longer pruned by per-minor presence, so each pass asks `build_binary_package()` about packages that may already exist, and it answers `already exists in S3 ... Skipping build` per package. Slower per pass, and correct — the pruning it replaces was removing the wrong things. Reviewed-on: #189	2026-08-31 21:36:11 +00:00
pat-s	213d30cea4	fix(build): hold back packages the cran mirror has not picked up yet (#188) ... ## Motivation A resolute build aborted on a single package: ``` [23/361] AsyPeer_0.0.1 (r_minor_sensitive=TRUE) Error: GitHub API error (404): Not Found x URL not found: <https://api.github.com/repos/cran/AsyPeer/commits> Retrying in 2 seconds. ... Retrying in 60 seconds. Error in `rate_sleep()`: ! Request failed after 10 attempts. Execution halted ``` This is not rate limiting — it is a **404**. `check_for_binary()` reads the published version from the `cran` GitHub mirror, and that mirror lags CRAN. `AsyPeer 0.0.1` was published today at 13:50 UTC and has no repository there yet. The 404 is permanent, but the call is wrapped in `purrr::insistently` with `max_times = 10` and `pause_cap = 60`, so it retries on a 1/2/4/8/16/32/60/60/60/60 second backoff — about five minutes — and then aborts the whole shard. ## Change Hold back release versions published within `CRAN_MIRROR_LAG_DAYS` (default 3). Deferring them costs nothing: the daily update pipeline builds new and updated packages anyway, and they arrive here on the next run once the mirror has caught up. ## Measured against the live CRAN index | lag | held back | |---|---| | 1 day | 29 of 24831 (0.12%) | | **3 days** | **135 (0.54%)** | | 7 days | 412 (1.66%) | `AsyPeer` is among the 135 at three days. ## Worth doing separately Retrying a 404 at all is wrong — it can never succeed, and any other permanent 404 (a package pulled from the mirror, say) will abort a shard the same way. `check_for_binary()` should distinguish a permanent 404 from a transient failure and, when the mirror simply lacks the package, treat the version as unknown rather than fatal. That is a bincraft change and I have not made it here. Reviewed-on: #188	2026-08-31 17:12:56 +00:00
pat-s	4bf88ed378	fix(build): scope the already-attempted skip to the running R minor (#187) ... ## Motivation The run meant to close the 4.6 gap on `amd64/resolute` barely built anything: ``` [1] "Skipped 2334 already-attempted package versions; 59 remaining for this job" ``` `single_builds` records `r_version` per attempt — `store_build_metadata()` both writes and queries it — but the skip query here ignored that column: ```sql SELECT name, tag FROM single_builds WHERE platform = $1 AND arch = $2 ``` So a non-primary pass skipped every package the **primary** pass had already attempted under a different minor. `build-all.R --sensitive-only` running under R 4.6 skipped packages that had only ever been built for 4.5. That is the reason the per-minor slots never fill, and why the backlog cannot be worked off by rebuilding: `amd64/resolute` serves a 4.6 client 22322 packages against the 4.5 slot's 26346. It is also, ultimately, why an R 4.6.1 client got a 4.5-built `rlang` and `undefined symbol: SETLENGTH`. Every other fix in this chain addressed a consequence; this is the cause. ## Change Scope the skip to the R minor the pass is running under. Matched on the `major.minor` prefix rather than the full `r_version` string, so a patch bump (4.6.0 → 4.6.1) does not re-attempt the entire catalogue. Verified the prefix extraction against `4.5.3`, `4.6.0`, `4.4.3` and a bare `4.6`, and that the derivation matches what `store_build_metadata()` records. ## Expected effect The non-primary passes stop skipping wholesale. The first run per slot will be long, since it works off a backlog that has been accumulating for as long as the per-minor slots have existed. ## Verification - `local/build-all.R` parses. - Minor derivation checked under R 4.6.1: `4.6`. - Real effect is only observable from a run; the number to watch is the "Skipped N ... M remaining" line, which should show a far larger `M` for a non-primary pass. Reviewed-on: #187	2026-08-31 13:34:33 +00:00
pat-s	448349d075	revert(build): drop 4.6.0 as a primary R version option (#186) ... #183 added `4.6.0` so the per-minor pass could run under 4.6. It was unnecessary, and it is a footgun. **Unnecessary:** `build-all-versions` already loops every installed interpreter and runs `local/build-all.R --sensitive-only` for each non-primary minor. The 4.6 pass happens when the pipeline runs with the slot's normal `R_VERSION=4.5.3`. I proposed #183 without reading that loop closely enough. **Footgun:** `R_VERSION` selects the *primary* minor, and the primary build lands in the **generic** slot. Selecting `4.6.0` for a slot whose generic binaries are 4.5-built would publish 4.6 binaries there and break every 4.5 client — the mirror image of the bug that started all this. The gaps are being filled by running the pipeline as it already stands (11928 on amd64/resolute). Reviewed-on: #186	2026-08-31 13:29:03 +00:00
pat-s	4413f499f1	test(verify): follow redirects, and allow the guard's deliberate drops (#185) ... Two checks that no longer matched the system. `fetch()` did not pass `-L`, so every `--live` check against a routed index read a 302 body rather than the index a client receives — 48 spurious failures against production. It also did not bypass the edge cache. Both were fixed on the branch behind #180, but that PR merged at `+9/-5`, capturing only the parity commit, so neither reached `main`. The union check asserted flat ⊆ every per-minor index. Since rpkgs/bincraft#116, that is deliberately false: `amd64/resolute` drops 2228 packages from its 4.6 index because their only binary was built under another R minor. Those absences **are** the fix working. It now asserts the thing that must hold — no generic package built under *this* minor may go missing — and reports the deliberate drops as context. Verified against production: `amd64/resolute` goes from 5 failures to 14 passed / 0 failed. Reviewed-on: #185	2026-08-31 13:01:31 +00:00
pat-s	0a6c155dca	feat(cdn): enable per-R-minor routing in production (#184) ... ## Motivation Everything built today is unreachable until this is set. ``` > install.packages("rlang") trying URL 'https://cran.rpkgs.com/amd64/resolute/latest/src/contrib/rlang_1.3.0.tar.gz' > library(rlang) undefined symbol: SETLENGTH ``` No `4.6/` in that path. With `UNION_SLOTS` empty the client resolves against the generic index and never reaches a per-minor binary: | artifact | size | |---|---| | generic, R 4.5-built | **2079570** — what R downloaded | | `4.6/`, R 4.6-built | 2075106 — correct, unused | The working binary has existed since 12:13 today. Nothing routes anyone to it. ## Change Sets production `UNION_SLOTS` to all 16 slots, from the same `local.rpkgs_slots` the staging zone uses. ## Verified before enabling Against the staging zone, which runs the identical script against the identical origin: | check | result | |---|---| | regressions against the generic slot | 0 across all 16 slots | | R minor served the per-minor index | 48/48 | | excluded R minor sent to CRAN | 16/16 | | client with no R minor still gets generic | 16/16 | | tarball never rewritten | 16/16 | ## Trade-off, stated plainly Coverage on a non-primary minor drops where the per-minor build backlog has not been worked off. `amd64/resolute` serves a 4.6 client 22169 packages rather than the generic slot's 24310. Those ~2100 are ABI-risky packages built under another R minor. They are exactly the ones that would install and then fail at load, so the drop trades a confusing runtime crash for an honest "not available". It shrinks as the 4.6 builds land. If that trade is unwelcome for some slots, `local.rpkgs_slots` can be narrowed to a subset — `amd64/rhel10` and `amd64/alpine323` have the smallest backlogs — and widened as builds catch up. ## After applying ```sh BASE=https://cran.rpkgs.com scripts/verify-r-minor-routing.sh --live ``` and the reported case directly: ```sh docker run --rm --platform linux/amd64 reg.devxy.io/r/r-ubuntu:4.6-resolute \ R -q -e 'install.packages("rlang"); library(rlang); cat("loaded OK\n")' ``` Reviewed-on: #184	2026-08-31 12:52:29 +00:00
pat-s	5f901312a6	feat(build): allow the per-minor pass to run under R 4.6 (#183) ... ## Motivation The supported window is the latest R minor plus the two previous, which the build images install as `R_VERSION_LATEST=4.6.0`, `PREV1=4.5.3`, `PREV2=4.4.3`. This pipeline's `R_VERSION` offered only the latter two, so **no pipeline could run `local/build-all.R --sensitive-only` under 4.6** and its per-minor slots kept a backlog. That backlog is the live bug. `rlang` is built for 4.4 and 4.5 on `amd64/resolute` but never for 4.6, so an R 4.6.1 client is served the generic 4.5.3 binary and dies with `undefined symbol: SETLENGTH`. 2709 records across the 16 slots are in that state. ## Why not weekly-rebuild-missing I tried that first (#182) and it is the wrong tool, for two independent reasons: - `weekly-missing-binaries-audit.R` reads only `/latest/src/contrib/PACKAGES.gz` and has no `r_minor` awareness, so its candidate list can only contain packages missing from the **generic** slot. - `rebuild-missing.R:73` says it outright: *"rebuild passes no `is_r_minor_sensitive`, so it only ever targets the flat"*. Running it under 4.6 built with the right interpreter and wrote to the wrong slot. It built almost nothing, and I verified it contaminated nothing: `amd64/resolute`'s flat slot is 22503 records at 4.5 and zero at 4.6. #182 should be closed. `build-all-versions` already runs `--sensitive-only`, documented as "the extra per-minor passes under non-primary R versions". It only needed the option. ## Change Adds `4.6.0` to `R_VERSION`. Default unchanged. ```sh crow pipeline create devxy/build-cran-binaries \ --var target_arch=amd64 --var OS=ubuntu --var OS_VERSION=resolute --var R_VERSION=4.6.0 ``` ## Follow-up worth doing separately The audit has no per-minor awareness, so this gap is invisible to every existing check and will silently reopen. Nothing measures per-minor completeness today except `scripts/verify-r-minor-routing.sh`, which was written for routing rather than coverage. Reviewed-on: #183	2026-08-31 12:29:54 +00:00
pat-s	a2923bf063	fix(cdn): purge the staging zone too (#181) ... ## Motivation `cran-rpkgs-test` was added as a second pull zone on the same B2 origin, but it was never added to `BUNNY_PULLZONES`. It therefore keeps serving pre-reindex indexes behind the same ~370 day `cache_expiration_time` as production, and nothing ever refreshes it. That is not cosmetic. The zone exists to be measured, and a verification run against it measures whatever the edge still holds: ``` production: regressions=0 AGHmatrix Path=NA Built=R 4.5.3; x86_64-pc staging : regressions=161 AGHmatrix Path=4.5 Built=(none) ``` Same objects, same origin, 161 phantom regressions. I chased that number through two wrong diagnoses before noticing production and staging disagreed. ## Change Add `cran-rpkgs-test.b-cdn.net` to the purge list in both reindex pipelines. ## Note A `Cache-Control: no-cache` request header is not a substitute. It was added to the verification script and did **not** clear this: bunny does not honour it for these objects. Purging is the mechanism that works. Reviewed-on: #181	2026-08-31 10:31:35 +00:00
pat-s	d38a4b5746	test(verify): report uneven coverage instead of failing on it (#180) ... ## Motivation The full 16-slot run came back 139 passed, 5 failed. Four of the five were `coverage uneven across minors` on `resolute` and `alpine324` (both arches) — and they are not defects. Those slots are built under R 4.5, so their 4.5 union carries ABI-risky packages that only exist as 4.5 builds. A 4.4 or 4.6 client cannot safely load them, which is the whole reason per-minor slots exist. Their absence from the 4.4 and 4.6 indexes is correct behaviour, and failing the run on it blocks four slots that regress nobody. This is the same mistake as the source-fallback share, which was demoted to a note for the same reason. ## Change Report uneven coverage; do not fail on it. The two checks answer different questions and should not share an exit code: - **`MAX_REGRESSIONS`** gates *enablement*: would routing serve a client source where the generic slot holds a binary of that client's own minor? Must be zero. - **parity** gates the *claim*: can we advertise full coverage for ABI-sensitive packages? Informative, and currently no. ## Verification `amd64/resolute` now passes with the shortfall printed as a note: ``` ok amd64/resolute R 4.6: no regression against the generic slot note: amd64/resolute coverage uneven across minors (vs best 24748): R4.4:-345 R4.6:-352 passed: 8 failed: 0 ``` `shellcheck` clean. Reviewed-on: #180	2026-08-31 10:00:49 +00:00
pat-s	a3004695a7	test(verify): gate on regressions against the generic slot, not fallback rate (#179) ... ## Motivation The readiness check added in #175 failed a slot when more than 10% of its per-minor entries were source fallbacks. That stopped being a meaningful signal once rpkgs/bincraft#113 and #114 landed. Since bincraft keeps a matching-minor generic binary out of a fallback's shadow, a surviving fallback means the generic slot's binary was built under a **different** minor — unsafe for that client anyway. Serving source there is correct, just slow. Failing on that share blocks slots that are genuinely ready: `amd64/noble` sits at 53% for 4.5 and 4.6 while regressing nobody. ## Change Gate on the thing that actually decides enablement: packages a client of minor M would receive as **source** through per-minor routing while the generic slot holds a binary built under **M itself**. That is strictly worse than not routing at all, and must be zero. Fallback share is still printed, as context rather than a verdict. ## Verification Measured across every reindexed slot and minor after the `reindex=all` run: zero regressions everywhere. | slot | 4.4 | 4.5 | 4.6 | |---|---|---|---| | amd64/noble | 0 | 0 | 0 | | amd64/jammy | 0 | 0 | 0 | | amd64/rhel9 | 0 | 0 | 0 | | amd64/rhel10 | 0 | 0 | 0 | | amd64/resolute | 0 | 0 | 0 | | arm64/noble | 0 | 0 | 0 | `shellcheck` clean; script exercised against the live indexes. Reviewed-on: #179	2026-08-31 09:55:44 +00:00