fix(ci): bound pkgcache _metadata growth to stop macmini disk-fill #110

Merged

pat-s merged 5 commits from t3code/smarter-disk-pruning-macmini into main

2026-07-13 09:31:29 +00:00

pat-s commented

2026-07-03 07:33:56 +00:00

Owner

Problem

The arm64 build-all pipeline fills the macmini (gaia) host disk despite an 8h prune.
Root cause is not images or job volumes: it is the persistent dep-cache volume, specifically pkgcache/R/pkgcache/_metadata, which grew to ~165 GB.
{pkgcache} mints a new content hash for the "patched" binaries repo on every PACKAGES change, so each per-package build writes a fresh ~70 MB pkgs-<hash>.rds (+ patched-<hash>/) that is never evicted (2407 snapshots observed).
When the disk hits 100% OrbStack stops and the on-host prune can no longer connect to the daemon, so it never self-heals.

Change (Workstream A of the disk-fill fix)

Add trim_pkgcache_metadata() to local/r-minor-helpers.R: keeps the newest keep (default 20) patched-*/pkgs-*.rds entries under _metadata, deleting only entries older than min_age_secs (default 600s) so it never races the up-to-4 concurrent split-jobs sharing the volume.
Preserves pkg/ downloads and the stable CRAN/BioC/INLA repo dirs.
No-op when R_PKG_CACHE_DIR is empty (amd64) or _metadata is absent (first run).
Call it every 25 packages inside the build loop in local/build-all.R.
Add a defensive start-of-run cleanup of _metadata/patched-* + pkgs-*.rds to the two workflows that mount the persistent volume (build-all-versions.yaml, build-all-versions-install-deps.yaml).

Only these paths are touched; process-updates.yaml/weekly-rebuild-missing.yaml (no persistent volume) are unchanged.

Follow-ups (separate workstreams): on-host self-healing prune watcher + OrbStack disk cap (ansible), and Prometheus/Grafana alerting (k8s-talos).
Upstream: bincraft patched-repo hash churn is the true source fix.

New unit tests (6) for the helper; full suite 24/24 green.

## Problem The arm64 `build-all` pipeline fills the macmini (gaia) host disk despite an 8h prune. Root cause is not images or job volumes: it is the persistent dep-cache volume, specifically `pkgcache/R/pkgcache/_metadata`, which grew to ~165 GB. `{pkgcache}` mints a new content hash for the "patched" binaries repo on every PACKAGES change, so each per-package build writes a fresh ~70 MB `pkgs-<hash>.rds` (+ `patched-<hash>/`) that is never evicted (2407 snapshots observed). When the disk hits 100% OrbStack stops and the on-host prune can no longer connect to the daemon, so it never self-heals. ## Change (Workstream A of the disk-fill fix) - Add `trim_pkgcache_metadata()` to `local/r-minor-helpers.R`: keeps the newest `keep` (default 20) `patched-*`/`pkgs-*.rds` entries under `_metadata`, deleting only entries older than `min_age_secs` (default 600s) so it never races the up-to-4 concurrent split-jobs sharing the volume. Preserves `pkg/` downloads and the stable CRAN/BioC/INLA repo dirs. No-op when `R_PKG_CACHE_DIR` is empty (amd64) or `_metadata` is absent (first run). - Call it every 25 packages inside the build loop in `local/build-all.R`. - Add a defensive start-of-run cleanup of `_metadata/patched-*` + `pkgs-*.rds` to the two workflows that mount the persistent volume (`build-all-versions.yaml`, `build-all-versions-install-deps.yaml`). Only these paths are touched; `process-updates.yaml`/`weekly-rebuild-missing.yaml` (no persistent volume) are unchanged. Follow-ups (separate workstreams): on-host self-healing prune watcher + OrbStack disk cap (ansible), and Prometheus/Grafana alerting (k8s-talos). Upstream: bincraft patched-repo hash churn is the true source fix. New unit tests (6) for the helper; full suite 24/24 green.

pat-s added 4 commits

2026-07-03 07:33:56 +00:00

fix: recompute package snapshot when missing from cache 26ac988717

The install-deps step precomputes pkgs_to_build.rds, r_minor_sensitive_pkgs.rds
and s3_cache.rds into /mnt/cache, but that cache volume is per-agent.
A build job scheduled on a fresh agent (or racing install-deps) finds the
snapshot absent and fails at readRDS.

Guard the reads in build-all.R: when any snapshot file is missing, source
packages-to-build.R and save the derived files (atomic temp+rename so a
concurrent job never reads a half-written rds). The first job on an agent
repopulates the shared cache for subsequent jobs; jobs that also miss redo
the work.

feat(build-all): add trim_pkgcache_metadata helper to bound pkgcache _metadata dd7f467b6a

feat(build-all): trim pkgcache _metadata every 25 packages during full builds b53a24614b

fix(ci): clear stale pkgcache metadata at start of build-all runs c338ce2f10

pat-s added 1 commit

2026-07-13 09:30:29 +00:00

Merge remote-tracking branch 'origin/main' into t3code/smarter-disk-pruning-macmini 6afa4e2405

# Conflicts:
#	.crow/build-all-versions-install-deps.yaml
#	.crow/build-all-versions.yaml

pat-s merged commit a4b274f281 into main

2026-07-13 09:31:29 +00:00

pat-s deleted branch t3code/smarter-disk-pruning-macmini

2026-07-13 09:31:30 +00:00

pat-s referenced this pull request from a commit

2026-07-13 09:31:30 +00:00

fix(ci): bound pkgcache _metadata growth to stop macmini disk-fill (#110)

No reviewers

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

devxy/build-cran-binaries!110

No description provided.

Rows
Columns