fix(build): recompute a stale package snapshot, not just a missing one #190

Merged

pat-s merged 1 commit from fix/snapshot-staleness into main

2026-08-31 22:11:18 +00:00

pat-s commented

2026-08-31 21:55:26 +00:00

Owner

Motivation

arm64/alpine324 reported nothing to build while thousands were missing:

line  49: Precomputed 7192 package versions (6871 r-minor-sensitive)   <- install-deps agent
line  99: Total# of remaining package versions: 43 (sensitive_only=TRUE) <- a build shard
line 101: Skipped 0 package versions already attempted under R 4.4; 0 remaining

Both numbers come from the same pipeline. The same run's index step dropped 2407 packages as missing for 4.4 and 2436 for 4.6.

Cause

if (!all(file.exists(package_cache_files))) { ... recompute ... }

Existence is not freshness. The snapshot describes S3 and CRAN state when it was written, and the cache volume is per-agent — the file's own comment says so. An agent that ran an earlier pipeline keeps serving that pipeline's answer forever, and no later fix to how the snapshot is computed (#189) can reach it.

Change

Recompute when the snapshot is stale as well as when it is missing. Keyed on the pipeline when the CI exposes an identifier (CI_PIPELINE_NUMBER, CI_BUILD_NUMBER, CI_PIPELINE_ID), so a new pipeline recomputes once per agent and its shards then share the result. Off CI, or when none is set, an age check with a two hour default (PACKAGE_SNAPSHOT_TTL_HOURS).

Verification

scenario	decision
files missing	RECOMPUTE
same pipeline id	reuse
new pipeline id	RECOMPUTE
no CI var, recent file	reuse
no CI var, aged out	RECOMPUTE

I could not confirm which identifier Crow actually sets — none is referenced anywhere in this repo — so all three are tried and the age check backs them up. If none is present the behaviour is the age path, which is still correct, just coarser.

## Motivation `arm64/alpine324` reported nothing to build while thousands were missing: ``` line 49: Precomputed 7192 package versions (6871 r-minor-sensitive) <- install-deps agent line 99: Total# of remaining package versions: 43 (sensitive_only=TRUE) <- a build shard line 101: Skipped 0 package versions already attempted under R 4.4; 0 remaining ``` Both numbers come from the **same pipeline**. The same run's index step dropped 2407 packages as missing for 4.4 and 2436 for 4.6. ## Cause ```r if (!all(file.exists(package_cache_files))) { ... recompute ... } ``` Existence is not freshness. The snapshot describes S3 and CRAN state when it was written, and the cache volume is per-agent — the file's own comment says so. An agent that ran an earlier pipeline keeps serving that pipeline's answer forever, and no later fix to how the snapshot is computed (#189) can reach it. ## Change Recompute when the snapshot is stale as well as when it is missing. Keyed on the pipeline when the CI exposes an identifier (`CI_PIPELINE_NUMBER`, `CI_BUILD_NUMBER`, `CI_PIPELINE_ID`), so a new pipeline recomputes once per agent and its shards then share the result. Off CI, or when none is set, an age check with a two hour default (`PACKAGE_SNAPSHOT_TTL_HOURS`). ## Verification | scenario | decision | |---|---| | files missing | RECOMPUTE | | same pipeline id | reuse | | **new pipeline id** | **RECOMPUTE** | | no CI var, recent file | reuse | | no CI var, aged out | RECOMPUTE | I could not confirm which identifier Crow actually sets — none is referenced anywhere in this repo — so all three are tried and the age check backs them up. If none is present the behaviour is the age path, which is still correct, just coarser.

pat-s added 1 commit

2026-08-31 21:55:26 +00:00

fix(build): recompute a stale package snapshot, not just a missing one 85d14ffed5

The guard asked only whether the snapshot files exist. They describe S3
and CRAN state at the moment they were written, and the cache volume is
per-agent, so an agent that ran an earlier pipeline keeps serving that
pipeline's answer indefinitely.

arm64/alpine324 reported '0 remaining' for both 4.4 and 4.6 from a
snapshot listing 43 r-minor-sensitive packages, while the install-deps
step of the same pipeline had just computed 6871 on another agent. Both
numbers appear in one log: line 49 says 6871, line 99 says 43.

That is why the slot looked finished in minutes while its own index step
dropped 2407 and 2436 packages as missing.

Keyed on the pipeline when the CI exposes an identifier, so a new
pipeline recomputes once per agent and its shards share the result;
otherwise an age check with a two hour default. Verified across five
cases: missing, same pipeline, new pipeline, recent without a CI
variable, and aged out.

pat-s merged commit 642e07e1d6 into main

2026-08-31 22:11:18 +00:00

pat-s deleted branch fix/snapshot-staleness

2026-08-31 22:11:18 +00:00

pat-s referenced this pull request from a commit

2026-08-31 22:11:19 +00:00

fix(build): recompute a stale package snapshot, not just a missing one (#190)