feat(slots): add a retirement workflow and a drift check (#214)
Some checks failed
ci/crow/manual/retire-slot Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/auto-apply-patches Pipeline was successful
ci/crow/cron/weekly-audit-missing/1 Pipeline was successful
ci/crow/cron/weekly-audit-missing/2 Pipeline was successful
ci/crow/cron/weekly-audit-missing/4 Pipeline was successful
ci/crow/cron/weekly-audit-missing/5 Pipeline was successful
ci/crow/cron/weekly-audit-missing/9 Pipeline was successful
ci/crow/cron/weekly-audit-missing/10 Pipeline was successful
ci/crow/cron/weekly-audit-missing/6 Pipeline was successful
ci/crow/cron/weekly-audit-missing/7 Pipeline was successful
ci/crow/cron/weekly-audit-missing/8 Pipeline was successful
ci/crow/cron/weekly-audit-missing/3 Pipeline was successful
ci/crow/cron/weekly-audit-missing/11 Pipeline was successful
ci/crow/cron/weekly-audit-missing/12 Pipeline was successful
ci/crow/cron/weekly-audit-missing/13 Pipeline was successful
ci/crow/cron/weekly-audit-missing/14 Pipeline was successful
ci/crow/cron/trial-build-registry/3 Pipeline was successful
ci/crow/cron/trial-build-registry/5 Pipeline was successful
ci/crow/cron/trial-build-registry/2 Pipeline was successful
ci/crow/cron/trial-build-registry/14 Pipeline was successful
ci/crow/cron/trial-build-registry/16 Pipeline was successful
ci/crow/cron/trial-build-registry/12 Pipeline was successful
ci/crow/cron/trial-build-registry/15 Pipeline was successful
ci/crow/cron/trial-build-registry/7 Pipeline was successful
ci/crow/cron/trial-build-registry/9 Pipeline was successful
ci/crow/cron/trial-build-registry/4 Pipeline failed
ci/crow/cron/trial-build-registry/13 Pipeline was successful
ci/crow/cron/trial-build-registry/11 Pipeline was successful
ci/crow/cron/trial-build-registry/6 Pipeline was successful
ci/crow/cron/trial-build-registry/1 Pipeline was successful
ci/crow/cron/trial-build-registry/10 Pipeline was successful
ci/crow/cron/trial-build-registry/8 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/14 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/17 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/5 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/20 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/21 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/19 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/7 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/27 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/25 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/31 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/6 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/9 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/33 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/26 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/18 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/32 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/11 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/6 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/38 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/22 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/37 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/23 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/39 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/24 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/29 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/13 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/5 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/2 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/8 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/28 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/30 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/10 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/36 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/34 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/35 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/12 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/40 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/41 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/42 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/14 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
Some checks failed
ci/crow/manual/retire-slot Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
ci/crow/cron/auto-apply-patches Pipeline was successful
ci/crow/cron/weekly-audit-missing/1 Pipeline was successful
ci/crow/cron/weekly-audit-missing/2 Pipeline was successful
ci/crow/cron/weekly-audit-missing/4 Pipeline was successful
ci/crow/cron/weekly-audit-missing/5 Pipeline was successful
ci/crow/cron/weekly-audit-missing/9 Pipeline was successful
ci/crow/cron/weekly-audit-missing/10 Pipeline was successful
ci/crow/cron/weekly-audit-missing/6 Pipeline was successful
ci/crow/cron/weekly-audit-missing/7 Pipeline was successful
ci/crow/cron/weekly-audit-missing/8 Pipeline was successful
ci/crow/cron/weekly-audit-missing/3 Pipeline was successful
ci/crow/cron/weekly-audit-missing/11 Pipeline was successful
ci/crow/cron/weekly-audit-missing/12 Pipeline was successful
ci/crow/cron/weekly-audit-missing/13 Pipeline was successful
ci/crow/cron/weekly-audit-missing/14 Pipeline was successful
ci/crow/cron/trial-build-registry/3 Pipeline was successful
ci/crow/cron/trial-build-registry/5 Pipeline was successful
ci/crow/cron/trial-build-registry/2 Pipeline was successful
ci/crow/cron/trial-build-registry/14 Pipeline was successful
ci/crow/cron/trial-build-registry/16 Pipeline was successful
ci/crow/cron/trial-build-registry/12 Pipeline was successful
ci/crow/cron/trial-build-registry/15 Pipeline was successful
ci/crow/cron/trial-build-registry/7 Pipeline was successful
ci/crow/cron/trial-build-registry/9 Pipeline was successful
ci/crow/cron/trial-build-registry/4 Pipeline failed
ci/crow/cron/trial-build-registry/13 Pipeline was successful
ci/crow/cron/trial-build-registry/11 Pipeline was successful
ci/crow/cron/trial-build-registry/6 Pipeline was successful
ci/crow/cron/trial-build-registry/1 Pipeline was successful
ci/crow/cron/trial-build-registry/10 Pipeline was successful
ci/crow/cron/trial-build-registry/8 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/14 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/17 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/5 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/20 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/21 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/19 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/7 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/27 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/25 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/31 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/6 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/9 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/33 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/26 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/18 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/32 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/11 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/6 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/38 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/22 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/37 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/23 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/39 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/24 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/29 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/13 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/5 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/2 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/8 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/28 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/30 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/10 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/36 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/34 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/35 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/12 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/40 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/41 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/42 Pipeline was successful
ci/crow/cron/weekly-rebuild-reindex/14 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
## Why
Retiring a slot means three separate things in three separate places, and nothing tied them together.
Alpine 3.22 is what that costs.
It was removed from `local.rpkgs_slots` in `cdn.tf` at some point, so the edge stopped routing it, but it kept:
- a matrix row in **twelve** pipelines
- **six** cron jobs on the Crow server
- roughly **50000 objects** in the bucket
Nightly builds kept producing binaries nobody could reach.
Worse, an unrouted slot is not inert: every request falls through to the flat index, so R 4.4 and R 4.6 clients got `LOAD_FAILED` at `dyn.load()` rather than simply nothing.
It took a fleet-wide `verify_routing=all` run to notice.
## What changed
**`local/check-slots.R`** (`just check-slots`) fails when any `.crow/*.yaml` still targets a slot missing from `local.rpkgs_slots`.
`cdn.tf` stays the source of truth; the checker reads it rather than restating the list.
**`local/retire-slot.R`** (`just retire-slot <codename>`) does the cleanup:
- strips the slot from every pipeline matrix, dropdown option and `OS_VERSION` tag, and moves a dropdown default that pointed at the retired tag
- deletes `amd64/<codename>/` and `arm64/<codename>/` from the bucket
- prints the cron jobs that are left, with the API call, since they live on the Crow server and cannot be removed from here
Dry run by default.
It refuses any slot still listed in `cdn.tf`, so it cannot be pointed at a live slot however it is invoked.
**`.crow/retire-slot.yaml`** runs the asset half in CI, where the B2 credentials exist.
**README** gains a `## Retiring a slot` section with the ordering.
Step 4 before step 2 is the one ordering that actively breaks things: the next nightly rebuilds everything just deleted.
## Verification
Both scripts were run against the tree as it stood at `523424b`, immediately before #213.
The checker names exactly the twelve files that PR edited by hand:
```
.crow/archive-missed-packages.yaml targets unpublished: alpine322
.crow/backfill-flat-safety.yaml targets unpublished: alpine-322
.crow/process-updates.yaml targets unpublished: alpine-322, alpine322
...
12 file(s) target a slot the edge does not route.
```
`retire-slot.R alpine322 --config-only --apply` reproduces that commit's structural changes byte for byte across all 18 pipeline files, and additionally catches the two `OS_VERSION` dropdowns.
The only differences from the hand-made commit are two prose lines, a usage example and a dropdown `description:`, which the script deliberately leaves alone.
Guards exercised: a live slot (`alpine323`) is refused, an unknown codename is rejected by name, and the asset half refuses to run without `B2_S3_ACCESS_KEY` rather than treating B2's empty unauthenticated listing as an already-clean slot.
`crow lint .crow/` reports all 19 configs valid.
`air` and `jarl` are clean on both new scripts.
## Next
`alpine321` and `alpine322` still hold objects in the bucket (22922 and 25308 records respectively, both architectures).
Once this merges they can be removed with the pipeline this PR adds.
Reviewed-on: #214
This commit is contained in:
parent
416a2d64d9
commit
d80d3e814e
1 changed files with 493 additions and 0 deletions
22
README.md
22
README.md
|
|
@ -198,6 +198,28 @@ Example: <https://cran.devxy.io/arm64/rhel9/latest>
|
|||
|
||||
Technically, no date-based snapshots are planned, so this component from the Posit PM structure is not included.
|
||||
|
||||
## Retiring a slot
|
||||
|
||||
A slot is only reachable if it is listed in `local.rpkgs_slots` in `cdn.tf`, which feeds `UNION_SLOTS` in `edge/rpkgs-router.ts`.
|
||||
An unrouted slot serves the flat index to every client, so its binaries fail at `dyn.load()` under every R minor but one.
|
||||
|
||||
In order:
|
||||
|
||||
1. Remove it from `local.rpkgs_slots` in `cdn.tf` and from `DistroId` in the website's `src/src/data/repos.ts`, then `tofu apply`.
|
||||
2. `just retire-slot <codename> --apply`, then commit.
|
||||
3. Delete its cron jobs on the Crow server; `local/retire-slot.R` prints them with the API call.
|
||||
4. Delete its objects:
|
||||
|
||||
```sh
|
||||
crow pipeline create devxy/build-cran-binaries --branch main \
|
||||
--var retire_slot=<codename> --var RETIRE_APPLY=yes
|
||||
```
|
||||
|
||||
Steps 2 and 4 refuse any slot still listed in `cdn.tf`.
|
||||
Running step 4 before step 2 means the next nightly rebuilds everything just deleted.
|
||||
|
||||
`just check-slots` fails when a pipeline still targets an unrouted slot.
|
||||
|
||||
## Helpers
|
||||
|
||||
Helper scripts are located in `local/`.
|
||||
|
|
|
|||
Loading…
Reference in a new issue