Commit graph build-cran-binaries/local/archive-missed-pkgs.R
Author SHA1 Message Date
767dd9ee5d
fix(abi): inspect flat objects of packages CRAN has removed
The backfill decided which objects were worth inspecting by looking the package
up in CRAN's current index. CRAN lists only what it currently ships, so a
package it has archived is absent, is never recognised as compiled, and is
never inspected. No verdict is ever recorded for it.

`union_index_records()` meanwhile reads the slot's *own* `NeedsCompilation`,
finds `yes`, and drops the record when the source classifier flags the package
and no safe verdict exists to clear it. The repair cannot rescue it either,
because it works from recorded verdicts and there is no row.

The record is therefore dropped from every per-minor index on every re-index,
permanently. `mixedMem` and `vcpen`, both removed from CRAN, were stuck out of
all three rhel8 per-minor indexes on both architectures that way, surviving a
backfill, two re-indexes and a repair.

Read `NeedsCompilation` from the slot's own flat index instead, which describes
exactly the objects being inspected. It is fetched from the bucket rather than
over HTTPS because this runs inside R, where the edge would route the request
to a per-minor index.

Keyed on `<package>_<version>` rather than the package name: a slot can carry
several versions and only some of them compile.
2026-09-09 18:26:38 +00:00
d558e27c11 feat: R-minor-sensitive binary builds (full + iterative) (#84)
Some checks failed
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline failed
## Summary

Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered.

## What's in here

**Detection / precompute**

- `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests.
- `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions.

**Full build**

- `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset.
- `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor.

**Iterative build**

- All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added.

**Tooling / housekeeping**

- Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`.
- Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts.
- Design + implementation docs under `docs/superpowers/`.

## Requires before merge

- A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected.

Reviewed-on: #84
2026-06-14 08:04:40 +00:00
6b1ae4e555
chore: cleanup workflows
Some checks failed
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline failed
ci/crow/manual/build-all-versions-install-deps Pipeline was successful
ci/crow/manual/build-all-versions/7 Pipeline is running
ci/crow/manual/build-all-versions/3 Pipeline is running
ci/crow/manual/build-all-versions/4 Pipeline is running
ci/crow/manual/build-all-versions/9 Pipeline is running
ci/crow/manual/build-all-versions/1 Pipeline is running
ci/crow/manual/build-all-versions/8 Pipeline is running
ci/crow/manual/build-all-versions/2 Pipeline is running
ci/crow/manual/build-all-versions/5 Pipeline is running
ci/crow/manual/build-all-versions/10 Pipeline is running
ci/crow/manual/build-all-versions/6 Pipeline is running
2025-06-17 21:03:49 +02:00
321e3d3205
refactor: bincraftR -> bincraft
Some checks failed
ci/crow/cron/process-updates-alpine-320-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-320-amd64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-322-amd64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-322-arm64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-9-amd64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-321-amd64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-8-amd64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-320-arm64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-9-arm64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-8-arm64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-320-amd64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-321-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline failed
ci/crow/cron/process-updates-alpine-321-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-321-arm64 Pipeline was successful
2025-06-08 15:39:54 +02:00
16e773b655
feat: migrate to hel1 location
Some checks failed
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline failed
ci/crow/cron/update-package-index-redhat-9-amd64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-8-amd64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-320-amd64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-321-amd64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-9-arm64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-8-arm64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-320-arm64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-321-arm64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-320-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-321-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-320-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-321-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
2025-04-29 16:48:57 +02:00
6ef3573ca1
switch to hetzner 3 2024-12-23 14:36:45 +01:00
ad768bce27
use new s3 bucket 2024-12-22 14:32:09 +01:00
2588b2512f
update local/ scripts 2024-11-25 10:04:23 +01:00
febe099e45
make missed package script more flexible 2024-11-20 20:17:52 +01:00
0cda5405ac
feat: add script to archive multiple pkgs manually 2024-11-04 14:42:31 +01:00