feat(slots): add a retirement workflow and a drift check #214

pat-s · 2026-09-11T08:17:31Z

pat-s commented

2026-09-11 08:17:31 +00:00

Why

Retiring a slot means three separate things in three separate places, and nothing tied them together.

Alpine 3.22 is what that costs.
It was removed from local.rpkgs_slots in cdn.tf at some point, so the edge stopped routing it, but it kept:

a matrix row in twelve pipelines
six cron jobs on the Crow server
roughly 50000 objects in the bucket

Nightly builds kept producing binaries nobody could reach.
Worse, an unrouted slot is not inert: every request falls through to the flat index, so R 4.4 and R 4.6 clients got LOAD_FAILED at dyn.load() rather than simply nothing.
It took a fleet-wide verify_routing=all run to notice.

What changed

local/check-slots.R (just check-slots) fails when any .crow/*.yaml still targets a slot missing from local.rpkgs_slots.
cdn.tf stays the source of truth; the checker reads it rather than restating the list.

local/retire-slot.R (just retire-slot <codename>) does the cleanup:

strips the slot from every pipeline matrix, dropdown option and OS_VERSION tag, and moves a dropdown default that pointed at the retired tag
deletes amd64/<codename>/ and arm64/<codename>/ from the bucket
prints the cron jobs that are left, with the API call, since they live on the Crow server and cannot be removed from here

Dry run by default.
It refuses any slot still listed in cdn.tf, so it cannot be pointed at a live slot however it is invoked.

.crow/retire-slot.yaml runs the asset half in CI, where the B2 credentials exist.

README gains a ## Retiring a slot section with the ordering.
Step 4 before step 2 is the one ordering that actively breaks things: the next nightly rebuilds everything just deleted.

Verification

Both scripts were run against the tree as it stood at 523424b, immediately before #213.

The checker names exactly the twelve files that PR edited by hand:

.crow/archive-missed-packages.yaml            targets unpublished: alpine322
.crow/backfill-flat-safety.yaml               targets unpublished: alpine-322
.crow/process-updates.yaml                    targets unpublished: alpine-322, alpine322
...
12 file(s) target a slot the edge does not route.

retire-slot.R alpine322 --config-only --apply reproduces that commit's structural changes byte for byte across all 18 pipeline files, and additionally catches the two OS_VERSION dropdowns.
The only differences from the hand-made commit are two prose lines, a usage example and a dropdown description:, which the script deliberately leaves alone.

Guards exercised: a live slot (alpine323) is refused, an unknown codename is rejected by name, and the asset half refuses to run without B2_S3_ACCESS_KEY rather than treating B2's empty unauthenticated listing as an already-clean slot.

crow lint .crow/ reports all 19 configs valid.
air and jarl are clean on both new scripts.

## Why Retiring a slot means three separate things in three separate places, and nothing tied them together. Alpine 3.22 is what that costs. It was removed from `local.rpkgs_slots` in `cdn.tf` at some point, so the edge stopped routing it, but it kept: - a matrix row in **twelve** pipelines - **six** cron jobs on the Crow server - roughly **50000 objects** in the bucket Nightly builds kept producing binaries nobody could reach. Worse, an unrouted slot is not inert: every request falls through to the flat index, so R 4.4 and R 4.6 clients got `LOAD_FAILED` at `dyn.load()` rather than simply nothing. It took a fleet-wide `verify_routing=all` run to notice. ## What changed **`local/check-slots.R`** (`just check-slots`) fails when any `.crow/*.yaml` still targets a slot missing from `local.rpkgs_slots`. `cdn.tf` stays the source of truth; the checker reads it rather than restating the list. **`local/retire-slot.R`** (`just retire-slot <codename>`) does the cleanup: - strips the slot from every pipeline matrix, dropdown option and `OS_VERSION` tag, and moves a dropdown default that pointed at the retired tag - deletes `amd64/<codename>/` and `arm64/<codename>/` from the bucket - prints the cron jobs that are left, with the API call, since they live on the Crow server and cannot be removed from here Dry run by default. It refuses any slot still listed in `cdn.tf`, so it cannot be pointed at a live slot however it is invoked. **`.crow/retire-slot.yaml`** runs the asset half in CI, where the B2 credentials exist. **README** gains a `## Retiring a slot` section with the ordering. Step 4 before step 2 is the one ordering that actively breaks things: the next nightly rebuilds everything just deleted. ## Verification Both scripts were run against the tree as it stood at `523424b`, immediately before #213. The checker names exactly the twelve files that PR edited by hand: ``` .crow/archive-missed-packages.yaml targets unpublished: alpine322 .crow/backfill-flat-safety.yaml targets unpublished: alpine-322 .crow/process-updates.yaml targets unpublished: alpine-322, alpine322 ... 12 file(s) target a slot the edge does not route. ``` `retire-slot.R alpine322 --config-only --apply` reproduces that commit's structural changes byte for byte across all 18 pipeline files, and additionally catches the two `OS_VERSION` dropdowns. The only differences from the hand-made commit are two prose lines, a usage example and a dropdown `description:`, which the script deliberately leaves alone. Guards exercised: a live slot (`alpine323`) is refused, an unknown codename is rejected by name, and the asset half refuses to run without `B2_S3_ACCESS_KEY` rather than treating B2's empty unauthenticated listing as an already-clean slot. `crow lint .crow/` reports all 19 configs valid. `air` and `jarl` are clean on both new scripts. ## Next `alpine321` and `alpine322` still hold objects in the bucket (22922 and 25308 records respectively, both architectures). Once this merges they can be removed with the pipeline this PR adds.

pat-s added 1 commit

2026-09-11 08:17:31 +00:00

feat(slots): add a retirement workflow and a drift check bfe56ffd22

Retiring a slot means three separate things in three separate places, and
nothing tied them together.
Alpine 3.22 is what that costs: it was removed from `local.rpkgs_slots` in
`cdn.tf` at some point, so the edge stopped routing it, but it kept a matrix
row in twelve pipelines, six cron jobs on the Crow server and ~50000 objects in
the bucket.
Nightly builds kept producing binaries nobody could reach, and because an
unrouted slot serves the flat index to every client, R 4.4 and R 4.6 users got
`LOAD_FAILED` rather than nothing.

This change will:

- Add `local/check-slots.R`, which fails when any `.crow/*.yaml` still targets
  a slot missing from `local.rpkgs_slots`. Run it with `just check-slots`.
- Add `local/retire-slot.R`, which strips a slot from the pipeline matrices and
  deletes its objects from the bucket. Dry run by default, and it refuses any
  slot still listed in `cdn.tf`, so it cannot be pointed at a live slot.
- Add `.crow/retire-slot.yaml`, which runs the asset half in CI where the B2
  credentials exist.
- Document the ordering in README, including the cron jobs, which live on the
  Crow server and cannot be removed from here.

The check is one-directional on purpose.
A pipeline targeting an unpublished slot is an error; a published slot missing
from *some* pipeline is only reported, because several pipelines cover a subset
deliberately.

Verified against the tree as it stood before #213: the checker names exactly the
twelve files that PR edited by hand, and `retire-slot.R --config-only` reproduces
that commit's structural changes byte for byte across all 18 pipeline files.
The only differences are two prose lines, a usage example and a dropdown
`description:`, which the script leaves alone because rewriting free text
automatically is guesswork.

pat-s reviewed

2026-09-11 08:20:12 +00:00

README.md

					
				@ -198,6 +198,42 @@ Example: <https://cran.devxy.io/arm64/rhel9/latest>

				Technically, no date-based snapshots are planned, so this component from the Posit PM structure is not included.

				## Retiring a slot

remove that verbose info and pin it down to the absolute essentials

pat-s marked this conversation as resolved

local/check-slots.R Outdated

					
				@ -0,0 +1,184 @@

				### Fail when a pipeline still targets a slot the edge does not route.

remove execessive comments

pat-s marked this conversation as resolved

local/retire-slot.R

					
				@ -0,0 +1,267 @@

				### Retire a slot: strip it from the pipelines and delete its objects from S3.

remove excessive comments

pat-s marked this conversation as resolved

pat-s referenced this pull request from a commit

2026-09-11 08:21:25 +00:00

refactor(slots): cut the comments down to the essentials

pat-s added 1 commit

2026-09-11 08:21:25 +00:00

refactor(slots): cut the comments down to the essentials cc9d7b6046

Addresses review on #214.

This change will:

- Trim the header comments of `local/check-slots.R`, `local/retire-slot.R` and
  `.crow/retire-slot.yaml` to what a reader needs, dropping the narrative about
  how Alpine 3.22 got into that state.
- Reduce the README section to the ordered steps and the two constraints that
  matter.

Behaviour is unchanged: against the tree at 523424b the script still reproduces
#213's structural changes, differing only in the same prose-only file, and both
guards still fire.

pat-s merged commit d80d3e814e into main

2026-09-11 13:14:18 +00:00

pat-s deleted branch feat/retire-slot-workflow

2026-09-11 13:14:18 +00:00

pat-s referenced this pull request from a commit

2026-09-11 13:14:19 +00:00

feat(slots): add a retirement workflow and a drift check (#214)

Sign in to join this conversation.

No reviewers

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

devxy/build-cran-binaries!214

				`@ -0,0 +1,184 @@`
				`### Fail when a pipeline still targets a slot the edge does not route.`

				`@ -0,0 +1,267 @@`
				`### Retire a slot: strip it from the pipelines and delete its objects from S3.`

Rows
Columns