feat(slots): add a retirement workflow and a drift check (#214)

## Why

Retiring a slot means three separate things in three separate places, and nothing tied them together.

Alpine 3.22 is what that costs.
It was removed from `local.rpkgs_slots` in `cdn.tf` at some point, so the edge stopped routing it, but it kept:

- a matrix row in **twelve** pipelines
- **six** cron jobs on the Crow server
- roughly **50000 objects** in the bucket

Nightly builds kept producing binaries nobody could reach.
Worse, an unrouted slot is not inert: every request falls through to the flat index, so R 4.4 and R 4.6 clients got `LOAD_FAILED` at `dyn.load()` rather than simply nothing.
It took a fleet-wide `verify_routing=all` run to notice.

## What changed

**`local/check-slots.R`** (`just check-slots`) fails when any `.crow/*.yaml` still targets a slot missing from `local.rpkgs_slots`.
`cdn.tf` stays the source of truth; the checker reads it rather than restating the list.

**`local/retire-slot.R`** (`just retire-slot <codename>`) does the cleanup:

- strips the slot from every pipeline matrix, dropdown option and `OS_VERSION` tag, and moves a dropdown default that pointed at the retired tag
- deletes `amd64/<codename>/` and `arm64/<codename>/` from the bucket
- prints the cron jobs that are left, with the API call, since they live on the Crow server and cannot be removed from here

Dry run by default.
It refuses any slot still listed in `cdn.tf`, so it cannot be pointed at a live slot however it is invoked.

**`.crow/retire-slot.yaml`** runs the asset half in CI, where the B2 credentials exist.

**README** gains a `## Retiring a slot` section with the ordering.
Step 4 before step 2 is the one ordering that actively breaks things: the next nightly rebuilds everything just deleted.

## Verification

Both scripts were run against the tree as it stood at `523424b`, immediately before #213.

The checker names exactly the twelve files that PR edited by hand:

```
.crow/archive-missed-packages.yaml            targets unpublished: alpine322
.crow/backfill-flat-safety.yaml               targets unpublished: alpine-322
.crow/process-updates.yaml                    targets unpublished: alpine-322, alpine322
...
12 file(s) target a slot the edge does not route.
```

`retire-slot.R alpine322 --config-only --apply` reproduces that commit's structural changes byte for byte across all 18 pipeline files, and additionally catches the two `OS_VERSION` dropdowns.
The only differences from the hand-made commit are two prose lines, a usage example and a dropdown `description:`, which the script deliberately leaves alone.

Guards exercised: a live slot (`alpine323`) is refused, an unknown codename is rejected by name, and the asset half refuses to run without `B2_S3_ACCESS_KEY` rather than treating B2's empty unauthenticated listing as an already-clean slot.

`crow lint .crow/` reports all 19 configs valid.
`air` and `jarl` are clean on both new scripts.

## Next

`alpine321` and `alpine322` still hold objects in the bucket (22922 and 25308 records respectively, both architectures).
Once this merges they can be removed with the pipeline this PR adds.

Reviewed-on: #214

.crow/retire-slot.yaml

@@ -0,0 +1,72 @@
+# Delete a retired slot's objects from the bucket.
+#
+# The config half is a commit, so it runs on a workstation:
+#   just retire-slot <codename> --apply
+#
+# This half needs the B2 credentials, which only exist in CI.
+# `local/retire-slot.R` refuses any slot still listed in `local.rpkgs_slots`,
+# so this cannot delete a live slot. Both architectures go together.
+#
+# Dry run unless RETIRE_APPLY is 'yes':
+#
+#   crow pipeline create devxy/build-cran-binaries --branch main \
+#     --var retire_slot=alpine322 --var RETIRE_APPLY=yes
+
+variables:
+  # A manual pipeline creation instantiates every file in .crow/, so the
+  # default must be a value the script never accepts.
+  retire_slot:
+    description: "Codename of the slot to retire (e.g. alpine322), or 'none'."
+    default: none
+  RETIRE_APPLY:
+    description: "Set to 'yes' to delete. Anything else reports and changes nothing."
+    options:
+      - 'no'
+      - 'yes'
+    default: 'no'
+
+when:
+  - event: manual
+    evaluate: 'retire_slot != "none"'
+
+skip_clone: true
+
+labels:
+  group: rpkgs-amd64
+
+steps:
+  - name: 'Delete the retired slot from the bucket'
+    image: reg.devxy.io/rpkgs/build-env-alpine:3.24
+    pull: true
+    environment:
+      OTEL_R_TRACES_EXPORTER: none
+      OTEL_R_LOGS_EXPORTER: none
+      OTEL_R_METRICS_EXPORTER: none
+      B2_S3_ACCESS_KEY:
+        from_secret: B2_S3_ACCESS_KEY
+      B2_S3_SECRET_KEY:
+        from_secret: B2_S3_SECRET_KEY
+      REPO_RO_TOKEN:
+        from_secret: REPO_RO_TOKEN
+      R_LIBS_USER: /tmp/R-pkgs
+      # Pipeline variables are substituted into the YAML, not exported, so they
+      # have to be bound here to be readable as env vars in the script below.
+      RETIRE_SLOT: ${retire_slot}
+      RETIRE_APPLY: ${RETIRE_APPLY}
+    commands:
+      - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
+      - mkdir -p /tmp/R-pkgs
+      - /opt/R/4.5.3/bin/Rscript local/install-bincraft.R
+      - |
+        set -eu
+        APPLY=""
+        if [ "$$RETIRE_APPLY" = "yes" ]; then
+          APPLY="--apply"
+        fi
+        /opt/R/4.5.3/bin/Rscript local/retire-slot.R "$$RETIRE_SLOT" --assets-only $$APPLY
+    backend_options:
+      docker:
+        resources:
+          requests:
+            memory: 2Gi
+            cpu: 1000m

README.md

@@ -198,6 +198,28 @@ Example: <https://cran.devxy.io/arm64/rhel9/latest>
 
 Technically, no date-based snapshots are planned, so this component from the Posit PM structure is not included.
 
+## Retiring a slot
+
+A slot is only reachable if it is listed in `local.rpkgs_slots` in `cdn.tf`, which feeds `UNION_SLOTS` in `edge/rpkgs-router.ts`.
+An unrouted slot serves the flat index to every client, so its binaries fail at `dyn.load()` under every R minor but one.
+
+In order:
+
+1. Remove it from `local.rpkgs_slots` in `cdn.tf` and from `DistroId` in the website's `src/src/data/repos.ts`, then `tofu apply`.
+2. `just retire-slot <codename> --apply`, then commit.
+3. Delete its cron jobs on the Crow server; `local/retire-slot.R` prints them with the API call.
+4. Delete its objects:
+
+   ```sh
+   crow pipeline create devxy/build-cran-binaries --branch main \
+     --var retire_slot=<codename> --var RETIRE_APPLY=yes
+   ```
+
+Steps 2 and 4 refuse any slot still listed in `cdn.tf`.
+Running step 4 before step 2 means the next nightly rebuilds everything just deleted.
+
+`just check-slots` fails when a pipeline still targets an unrouted slot.
+
 ## Helpers
 
 Helper scripts are located in `local/`.

justfile

@@ -87,3 +87,11 @@ edge-test:
         denoland/deno:latest \
         deno test -A edge/rpkgs-router.test.ts
     fi
+
+# fail if any pipeline still targets a slot the edge does not route
+check-slots:
+    Rscript local/check-slots.R
+
+# strip a retired slot from .crow/*.yaml (dry run; pass --apply to write)
+retire-slot codename *flags:
+    Rscript local/retire-slot.R {{ codename }} --config-only {{ flags }}

local/check-slots.R

@@ -0,0 +1,151 @@
+### Fail when a pipeline targets a slot that is not in `local.rpkgs_slots`.
+#
+#   Rscript local/check-slots.R
+#
+# That list feeds `UNION_SLOTS`, which `edge/rpkgs-router.ts` gates per-R-minor
+# routing on. An unrouted slot serves the flat index to every client, so its
+# binaries fail at dyn.load() under every R minor but one.
+#
+# One-directional: a published slot missing from some pipeline is only
+# reported, since several pipelines cover a subset on purpose.
+
+options(error = function() {
+  cat("ERROR:", geterrmessage(), "\n", file = stdout())
+  q(status = 1)
+})
+
+# Parses the distro list out of the setproduct's second array.
+published_codenames <- function(path = "cdn.tf") {
+  if (!file.exists(path)) {
+    stop(
+      sprintf("%s not found; run this from the repository root.", path),
+      call. = FALSE
+    )
+  }
+  lines <- readLines(path, warn = FALSE)
+  start <- grep("rpkgs_slots\\s*=", lines)
+  if (length(start) != 1L) {
+    stop(
+      "Could not find exactly one `rpkgs_slots` local in cdn.tf.",
+      call. = FALSE
+    )
+  }
+  block <- paste(lines[start:min(start + 10L, length(lines))], collapse = "\n")
+  arrays <- regmatches(block, gregexpr("\\[[^]]*\\]", block))[[1L]]
+  if (length(arrays) < 2L) {
+    stop(
+      "`rpkgs_slots` no longer has the two-array setproduct shape this parses.",
+      call. = FALSE
+    )
+  }
+  codenames <- gsub(
+    '"',
+    "",
+    regmatches(arrays[[2L]], gregexpr('"[^"]+"', arrays[[2L]]))[[1L]]
+  )
+  if (length(codenames) == 0L) {
+    stop("Parsed an empty distro list from cdn.tf.", call. = FALSE)
+  }
+  codenames
+}
+
+UBUNTU_CODENAMES <- c(
+  resolute = "ubuntu-2604",
+  noble = "ubuntu-2404",
+  jammy = "ubuntu-2204"
+)
+
+as_pipeline_os <- function(codename) {
+  ifelse(
+    codename %in% names(UBUNTU_CODENAMES),
+    UBUNTU_CODENAMES[codename],
+    ifelse(
+      grepl("^rhel[0-9]+$", codename),
+      sub("^rhel", "redhat-", codename),
+      ifelse(
+        grepl("^alpine[0-9]+$", codename),
+        sub("^alpine([0-9])([0-9]+)$", "alpine-\\1\\2", codename),
+        NA_character_
+      )
+    )
+  )
+}
+
+# Structural lines only, so a codename in a comment is not a finding.
+slots_in_pipeline <- function(path) {
+  lines <- readLines(path, warn = FALSE)
+  row <- regmatches(
+    lines,
+    regexec("^\\s*-?\\s*(OS|CODENAME|OS_ID):\\s*(\\S+)\\s*$", lines)
+  )
+  opt <- regmatches(
+    lines,
+    regexec("^\\s*-\\s*([a-z0-9]+-[a-z0-9]+)-(amd64|arm64)\\s*$", lines)
+  )
+  hits <- c(
+    vapply(
+      row,
+      function(m) if (length(m) == 3L) m[[3L]] else NA_character_,
+      ""
+    ),
+    vapply(opt, function(m) if (length(m) == 4L) m[[2L]] else NA_character_, "")
+  )
+  hits <- hits[!is.na(hits)]
+  # `OS: alpine` (build-all-versions' bare image name) is not a slot.
+  unique(hits[grepl("^(alpine|redhat|ubuntu)-?[0-9]", hits)])
+}
+
+codenames <- published_codenames()
+published_os <- unname(as_pipeline_os(codenames))
+if (anyNA(published_os)) {
+  stop(
+    sprintf(
+      "Unrecognised codename(s) in cdn.tf: %s",
+      paste(codenames[is.na(published_os)], collapse = ", ")
+    ),
+    call. = FALSE
+  )
+}
+# Files spell a slot either way, so accept both.
+published <- unique(c(published_os, codenames))
+
+cat(sprintf(
+  "Published slots (cdn.tf): %s\n\n",
+  paste(sort(published_os), collapse = ", ")
+))
+
+files <- sort(list.files(".crow", pattern = "\\.yaml$", full.names = TRUE))
+findings <- list()
+for (f in files) {
+  stale <- setdiff(slots_in_pipeline(f), published)
+  if (length(stale) > 0L) {
+    findings[[f]] <- stale
+    cat(sprintf(
+      "%-45s targets unpublished: %s\n",
+      f,
+      paste(sort(stale), collapse = ", ")
+    ))
+  }
+}
+
+uncovered <- setdiff(
+  published_os,
+  unique(unlist(lapply(files, slots_in_pipeline)))
+)
+if (length(uncovered) > 0L) {
+  cat(sprintf(
+    "\nNote: %s appear(s) in no pipeline at all.\n",
+    paste(sort(uncovered), collapse = ", ")
+  ))
+}
+
+if (length(findings) == 0L) {
+  cat("All pipeline slots are published and routed.\n")
+  quit(status = 0)
+}
+
+cat(sprintf(
+  "\n%d file(s) target an unrouted slot. Retire it with:\n  just retire-slot <codename>\n",
+  length(findings)
+))
+quit(status = 1)

local/retire-slot.R

@@ -0,0 +1,240 @@
+### Retire a slot: strip it from the pipelines and delete its objects from S3.
+#
+#   Rscript local/retire-slot.R <codename> [--apply] [--config-only|--assets-only]
+#
+# Retiring a distro retires both architectures, because `local.rpkgs_slots` in
+# `cdn.tf` is a setproduct over them.
+#
+# Order: remove the slot from `local.rpkgs_slots` (and the website's
+# `DistroId`) and apply that first, then run this. Deleting objects while a
+# pipeline still targets the slot just means the next nightly rebuilds them.
+#
+# Cron jobs live on the Crow server, not here, and are printed at the end.
+#
+# Only structural YAML is edited. Prose mentioning the slot is left alone, so
+# grep for the name afterwards.
+#
+# Dry run by default.
+
+options(error = function() {
+  cat("ERROR:", geterrmessage(), "\n", file = stdout())
+  q(status = 1)
+})
+
+args <- commandArgs(trailingOnly = TRUE)
+apply_changes <- "--apply" %in% args
+config_only <- "--config-only" %in% args
+assets_only <- "--assets-only" %in% args
+args <- args[!args %in% c("--apply", "--config-only", "--assets-only")]
+if (length(args) != 1L || (config_only && assets_only)) {
+  stop(
+    "usage: retire-slot.R <codename> [--apply] [--config-only|--assets-only]",
+    call. = FALSE
+  )
+}
+codename <- args[[1L]]
+
+# The bucket and `CODENAME:` say `alpine322`; `OS:` and the options say
+# `alpine-322`. Both have to go.
+pipeline_os <- if (grepl("^rhel[0-9]+$", codename)) {
+  sub("^rhel", "redhat-", codename)
+} else if (grepl("^alpine[0-9]+$", codename)) {
+  sub("^alpine", "alpine-", codename)
+} else {
+  # Single bracket so an unknown codename yields NA, not an error.
+  c(resolute = "ubuntu-2604", noble = "ubuntu-2404", jammy = "ubuntu-2204")[
+    codename
+  ]
+}
+if (is.na(pipeline_os) || length(pipeline_os) == 0L) {
+  stop(sprintf("Unrecognised codename '%s'.", codename), call. = FALSE)
+}
+
+# The `OS_VERSION` dropdowns are keyed on the image tag, not the slot name.
+image_tag <- if (grepl("^alpine-", pipeline_os)) {
+  sub("^alpine-([0-9])([0-9]+)$", "\\1.\\2", pipeline_os)
+} else if (grepl("^redhat-", pipeline_os)) {
+  sub("^redhat-", "", pipeline_os)
+} else {
+  codename
+}
+
+# Refuse a slot that is still routed: deleting its objects would leave the edge
+# serving indexes that name files which no longer exist.
+tf <- readLines("cdn.tf", warn = FALSE)
+start <- grep("rpkgs_slots\\s*=", tf)
+if (length(start) != 1L) {
+  stop(
+    "Could not find exactly one `rpkgs_slots` local in cdn.tf.",
+    call. = FALSE
+  )
+}
+block <- paste(tf[start:min(start + 10L, length(tf))], collapse = "\n")
+if (grepl(sprintf('"%s"', codename), block, fixed = TRUE)) {
+  stop(
+    sprintf(
+      "'%s' is still listed in `local.rpkgs_slots` in cdn.tf, so the edge still routes it. Remove it there (and from the website's DistroId union) and apply the OpenTofu change first.", # nolint
+      codename
+    ),
+    call. = FALSE
+  )
+}
+
+cat(sprintf(
+  "Retiring %s (pipelines call it '%s', images tag it '%s')\n\n",
+  codename,
+  pipeline_os,
+  image_tag
+))
+
+## 1. Pipeline definitions ----------------------------------------------------
+
+# Line-based on purpose: a YAML round trip would reflow all 18 files.
+strip_slot <- function(lines) {
+  keep <- rep(TRUE, length(lines))
+  i <- 1L
+  while (i <= length(lines)) {
+    line <- lines[[i]]
+    row <- regexec(
+      sprintf(
+        "^(\\s*)-\\s*(OS|CODENAME):\\s*(%s|%s)\\s*$",
+        pipeline_os,
+        codename
+      ),
+      line
+    )
+    m <- regmatches(line, row)[[1L]]
+    if (length(m) == 4L) {
+      indent <- nchar(m[[2L]])
+      keep[[i]] <- FALSE
+      j <- i + 1L
+      while (j <= length(lines)) {
+        nxt <- lines[[j]]
+        if (
+          !nzchar(trimws(nxt)) || nchar(sub("^(\\s*).*$", "\\1", nxt)) <= indent
+        ) {
+          break
+        }
+        keep[[j]] <- FALSE
+        j <- j + 1L
+      }
+      i <- j
+      next
+    }
+    if (grepl(sprintf("^\\s*-\\s*%s-(amd64|arm64)\\s*$", pipeline_os), line)) {
+      keep[[i]] <- FALSE
+    }
+    if (grepl(sprintf("^\\s*-\\s*'%s'\\s*$", image_tag), line)) {
+      keep[[i]] <- FALSE
+    }
+    i <- i + 1L
+  }
+  out <- lines[keep]
+  # A dropdown cannot default to an option that no longer exists.
+  bad_default <- grepl(sprintf("^\\s*default:\\s*'%s'\\s*$", image_tag), out)
+  if (any(bad_default)) {
+    for (k in which(bad_default)) {
+      opts <- grep("^\\s*-\\s*'[^']+'\\s*$", out[seq_len(k - 1L)])
+      if (length(opts) == 0L) {
+        next
+      }
+      # Walk back to this block's first option, not the nearest one.
+      first <- max(opts)
+      while (first > 1L && grepl("^\\s*-\\s*'[^']+'\\s*$", out[[first - 1L]])) {
+        first <- first - 1L
+      }
+      replacement <- sub("^\\s*-\\s*'([^']+)'\\s*$", "\\1", out[[first]])
+      out[[k]] <- sub("'[^']+'", sprintf("'%s'", replacement), out[[k]])
+      cat(sprintf("  default moved to '%s'\n", replacement))
+    }
+  }
+  out
+}
+
+if (!assets_only) {
+  files <- sort(list.files(".crow", pattern = "\\.yaml$", full.names = TRUE))
+  touched <- 0L
+  for (f in files) {
+    lines <- readLines(f, warn = FALSE)
+    out <- strip_slot(lines)
+    if (length(out) == length(lines)) {
+      next
+    }
+    touched <- touched + 1L
+    cat(sprintf(
+      "  %s %-45s -%d lines\n",
+      if (apply_changes) "EDIT   " else "would edit",
+      f,
+      length(lines) - length(out)
+    ))
+    if (apply_changes) {
+      writeLines(out, f)
+    }
+  }
+  if (touched == 0L) {
+    cat("  No pipeline references the slot.\n")
+  } else if (apply_changes) {
+    cat(sprintf(
+      "  %d file(s) edited. Verify with `crow lint .crow/` and `Rscript local/check-slots.R`.\n",
+      touched
+    ))
+  }
+  cat("\n")
+}
+
+## 2. Bucket objects ----------------------------------------------------------
+
+if (!config_only) {
+  library(s3fs, quietly = TRUE)
+  if (!nzchar(Sys.getenv("B2_S3_ACCESS_KEY"))) {
+    stop(
+      "B2_S3_ACCESS_KEY is unset. B2 lists nothing without credentials, which is indistinguishable from an already-clean slot.", # nolint
+      call. = FALSE
+    )
+  }
+  bucket <- "devxy-rpkgs-binaries"
+  s3fs::s3_file_system(
+    aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"),
+    aws_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"),
+    endpoint = "https://s3.eu-central-003.backblazeb2.com",
+    region_name = "eu-central-003",
+    refresh = TRUE
+  )
+
+  for (arch in c("amd64", "arm64")) {
+    prefix <- sprintf("%s/%s/%s", bucket, arch, codename)
+    if (!s3fs::s3_dir_exists(prefix)) {
+      cat(sprintf("  %s/%s: nothing in the bucket.\n", arch, codename))
+      next
+    }
+    objects <- s3fs::s3_dir_ls(prefix, recurse = TRUE)
+    cat(sprintf(
+      "  %s %s/%s: %d object(s)\n",
+      if (apply_changes) "DELETE" else "would delete",
+      arch,
+      codename,
+      length(objects)
+    ))
+    if (apply_changes && length(objects) > 0L) {
+      s3fs::s3_dir_delete(prefix)
+      cat(sprintf("  deleted s3://%s\n", prefix))
+    }
+  }
+  cat("\n")
+}
+
+## 3. Crons ------------------------------------------------------------------
+
+if (!apply_changes) {
+  cat("Dry run: nothing was edited or deleted. Re-run with --apply.\n\n")
+}
+
+cat("Crons still to remove on the Crow server (they are not in this repo):\n")
+cat(sprintf(
+  "  crow-at ci.devxy.io repo cron ls devxy/build-cran-binaries | grep %s\n",
+  pipeline_os
+))
+cat("  # `crow repo cron rm` 404s here; the API base is /api/v1\n")
+cat("  eval \"$(crow-at --export ci.devxy.io)\"\n")
+cat("  curl -s -X DELETE -H \"Authorization: Bearer $CROW_TOKEN\" \\\n")
+cat("    https://ci.devxy.io/api/v1/repos/5/cron/<cron-id>\n")