devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 12 Pull requests Releases Activity

fix(ci): pass the metadata DB to per-minor reindexes and fail on errors #234

Merged
pat-s merged 2 commits from chore/explicit-metadata-db-reindex into main 2026-10-17 07:46:10 +00:00
Conversation 0 Commits 2 Files changed 22 +113 -38
pat-s commented 2026-10-17 07:21:20 +00:00
Owner

Step 1 of rpkgs/bincraft#211: pass the metadata database explicitly so bincraft can drop its implicit production defaults.

Changes

  • Pass metadata_db_* explicitly to every per-minor upload_package_index() call in reindex.yaml, process-updates.yaml, weekly-rebuild-reindex.yaml, build-all-versions.yaml and prune-stale-flat-binaries.yaml.
  • Replace || true in those per-minor loops with a list of failed minors: every minor is still attempted, then the step exits 1 naming the ones that failed.
  • Give the "Upload package indexes" step of build-all-versions.yaml the PGPASS secret. It never had it, so its per-minor reindexes always aborted and || true hid that.
  • Run the CDN purge in process-updates.yaml and prune-stale-flat-binaries.yaml even when a reindex failed, as reindex.yaml and weekly-rebuild-reindex.yaml already do, because the generic index was still replaced.
  • local/repair-built-stamp.R passes the database too, and fails once all minors are tried instead of only printing the error.

The generic (non-per-minor) calls are unchanged: they never read the database.

Behaviour change

A per-minor reindex that fails now turns the pipeline red instead of passing silently.
Expect build-all-versions runs to start reindexing per-minor slots for real, since they now have the password.

Ordering

Merge this before rpkgs/bincraft's companion PR is released: local/install-bincraft.R installs the latest bincraft tag, and the explicit arguments work with both the current and the next release.

Step 1 of rpkgs/bincraft#211: pass the metadata database explicitly so bincraft can drop its implicit production defaults. ## Changes - Pass `metadata_db_*` explicitly to every per-minor `upload_package_index()` call in `reindex.yaml`, `process-updates.yaml`, `weekly-rebuild-reindex.yaml`, `build-all-versions.yaml` and `prune-stale-flat-binaries.yaml`. - Replace `|| true` in those per-minor loops with a list of failed minors: every minor is still attempted, then the step exits 1 naming the ones that failed. - Give the "Upload package indexes" step of `build-all-versions.yaml` the `PGPASS` secret. It never had it, so its per-minor reindexes always aborted and `|| true` hid that. - Run the CDN purge in `process-updates.yaml` and `prune-stale-flat-binaries.yaml` even when a reindex failed, as `reindex.yaml` and `weekly-rebuild-reindex.yaml` already do, because the generic index was still replaced. - `local/repair-built-stamp.R` passes the database too, and fails once all minors are tried instead of only printing the error. The generic (non-per-minor) calls are unchanged: they never read the database. ## Behaviour change A per-minor reindex that fails now turns the pipeline red instead of passing silently. Expect build-all-versions runs to start reindexing per-minor slots for real, since they now have the password. ## Ordering Merge this before rpkgs/bincraft's companion PR is released: `local/install-bincraft.R` installs the latest bincraft tag, and the explicit arguments work with both the current and the next release. 
- Pass `metadata_db_*` explicitly to every per-minor `upload_package_index()` call, so bincraft can drop its implicit production defaults (rpkgs/bincraft#211).
- Replace `|| true` in the per-minor reindex loops with a failure list, so every minor is still attempted but a failed reindex fails the step.
- Give the "Upload package indexes" step of build-all-versions the `PGPASS` secret, without which its per-minor reindexes always aborted.
- Run the CDN purge in process-updates and prune-stale-flat-binaries even when a reindex failed, as reindex.yaml already does.
- Fail local/repair-built-stamp.R when a per-minor reindex fails instead of only printing the error.
Part of https://codefloe.com/rpkgs/bincraft/issues/225. Stacked on #234.

Every connection to the metadata database used `sslmode = 'require'`, which encrypts but does not authenticate the server, so anyone able to impersonate `r-binaries.devxy.io:15432` would receive `PGPASS`.

## Changes

- `.crow/build-all-versions.yaml`, `process-updates.yaml`, `prune-stale-flat-binaries.yaml`, `reindex.yaml`, `weekly-rebuild-reindex.yaml`: `metadata_db_sslmode = 'verify-full'` in every bincraft call.
- `local/*.R`: the same for bincraft calls; scripts that call `DBI::dbConnect()` directly also pass `sslrootcert = bincraft::postgres_sslrootcert("verify-full")`, because libpq looks for `~/.postgresql/root.crt` rather than the system trust store.
- `README.md`: note that the endpoint serves a Let's Encrypt certificate and how to verify it.
- air reformatted three `=` assignments to `<-` in `local/missing-cran-packages-db.R` and `local/packages-without-any-binary.R` while the hook ran on those files.

## Ordering: merge last

1. devxy/k8s-talos#417 must be live: today the server presents an in-cluster certificate from a private CA, which `verify-full` rejects.
2. https://codefloe.com/rpkgs/bincraft/pulls/226 must be released: `local/install-bincraft.R` installs the latest tag, and older releases neither pass a root certificate nor export `postgres_sslrootcert()`.
3. #234 must be merged.

Merging early fails every database connection in CI, and since #234 per-minor reindex failures fail the pipeline.

Reviewed-on: #235
pat-s merged commit 3618168572 into main 2026-10-17 07:46:10 +00:00
pat-s deleted branch chore/explicit-metadata-db-reindex 2026-10-17 07:46:10 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries!234
No description provided.