| Filename | Latest commit message | Latest commit date |
|---|---|---|
pat-s
8f97cf99ef
Some checks failed
ci/crow/manual/build-all-versions-install-deps/2 Pipeline was successful
ci/crow/manual/build-all-versions-install-deps/1 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/manual/build-all-versions/1 Pipeline was canceled
ci/crow/manual/build-all-versions/7 Pipeline was canceled
ci/crow/manual/build-all-versions/4 Pipeline was canceled
ci/crow/manual/build-all-versions/5 Pipeline was canceled
ci/crow/manual/build-all-versions/8 Pipeline was canceled
ci/crow/manual/build-all-versions/6 Pipeline was canceled
ci/crow/manual/build-all-versions/3 Pipeline was canceled
ci/crow/manual/build-all-versions/2 Pipeline was canceled
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
## Motivation A restarted run recompiled everything it had already built. ``` attempted: 8683 already exists in S3: 3 actually compiled: 8683 ``` `amd64/resolute` had uploaded roughly 3000 binaries before being restarted, and rebuilt all of them. ## Cause This is a regression from #189, which I introduced. `s3_cache.rds` is derived from the same `file_names` that #189 filtered: ```r binary_cache <- setdiff(file_names, source_served) saveRDS(binary_cache, "/mnt/cache/packages/s3_cache.rds") ``` Removing per-minor objects from `file_names` was right for the candidate list and wrong for the existence cache. `build_binary_package()` was handed a cache containing no per-minor object at all, so its "not present in the remote bucket" check answered not-present for every one of them. The two consumers need opposite views of one listing: - the **candidate list** must ignore per-minor objects, or a package present under one minor prunes itself from every other minor's work; - the **existence cache** must not ignore them, or the work is redone. Conflating them was wrong in both directions: before #189 a per-minor pass believed the flat slot's binaries were its own and built nothing; after #189 it believed it had nothing and rebuilt the lot. ## Change The cache keeps paths relative to the slot, and `build-all.R` selects the part matching the pass it is running. `build_binary_package()` compares basenames and cannot distinguish `4.4/curl_1.0.tar.gz` from `curl_1.0.tar.gz`, so the choice has to be made where the running R minor is known. Verified: | pass | selects | |---|---| | primary (writes flat) | `curl_1.0`, `jsonlite_2.0`, `Archive/curl_0.9`, `PACKAGES.gz` | | `--sensitive-only` under 4.4 | `curl_1.0`, `rlang_1.3.0` (from `4.4/` only) | | `--sensitive-only` under 4.6 | `rlang_1.3.0` (from `4.6/` only) | | legacy basename cache | passthrough, unfiltered | `Archive/` counts as present for the primary pass: those are versions built and later superseded. ## Backwards compatibility A cache written before this change holds bare basenames. Filtering those by path would select nothing and trigger the very rebuild this prevents, so they are detected and used unfiltered; #190's staleness check replaces them on the next pipeline. ## Note on the running builds The five pipelines currently running are recompiling packages they already have. Their output is still correct — the binaries carry the right `Built` stamp for their slot — but the work is wasted. They should be restarted once this is merged. Reviewed-on: #191 |
||
| .. | ||
| patches | ||
| tests | ||
| archive-missed-pkgs.R | ||
| build-all.R | ||
| build-one.R | ||
| check-cdn-cache.sh | ||
| check-NA.R | ||
| debug-packages-writing.R | ||
| dedupe-audit-issue.R | ||
| detect-duplicates.R | ||
| excluded-packages.json | ||
| failing-builds-classify.R | ||
| failing-builds-report.R | ||
| fetch-rebuild-packages-from-issue.R | ||
| find-R-api-packages.sh | ||
| install-bincraft.R | ||
| label-removed-cran-packages.R | ||
| last-processed-by-platform.R | ||
| migrate-s3-hetzner-to-backblaze.sh | ||
| missing-cran-packages-db.R | ||
| missing-cran-packages.R | ||
| missing-packages-in-index.R | ||
| packages-to-build.R | ||
| packages-without-any-binary.R | ||
| proposal-tracking-lib.R | ||
| proposal-tracking.R | ||
| propose-patches.R | ||
| query-pkgs-without-old-versions.R | ||
| r-minor-helpers.R | ||
| rebuild-missing-helpers.R | ||
| rebuild-missing.R | ||
| repair-built-stamp.R | ||
| test-package-loading.R | ||
| trial-build-patch.R | ||
| trial-build-registry.R | ||
| uvr-install.sh | ||
| validate-patches.R | ||
| weekly-missing-binaries-audit.R | ||