devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 12 Pull requests Releases Activity

perf: use ccache and parallel make, and skip already-built newer-R candidates in bulk #240

Merged
pat-s merged 2 commits from perf/ccache-and-newer-r-prefilter into main 2026-10-23 13:18:00 +00:00
Conversation 0 Commits 2 Files changed 7 +234
pat-s commented 2026-10-23 13:16:50 +00:00
Owner

Most of a process-updates run is per-package overhead, not compiling.
In pipeline 14290 (ubuntu-2204-amd64) the build-newer-r.R loop spent about 4s on each of the ~1,100 candidates that already had a binary, and ccache was configured but never actually used.

Changes

  • local/compiler-cache.sh (new): sourced by the build pipelines right after the clone.
    • Prepends ccache's masquerade dir to PATH (/usr/lib/ccache/bin on Alpine, /usr/lib64/ccache on RHEL, /usr/lib/ccache on Ubuntu). The images call bare gcc/g++ in Makeconf and /root/.R/Makevars, so this catches every compile, CMake-driven vendored builds included, without replacing the images' Makevars through R_MAKEVARS_USER.
    • Sets CCACHE_BASEDIR=/tmp and CCACHE_NOHASHDIR=1. R compiles each package in a fresh /tmp/RtmpXXXXXX dir and its default -g puts that dir into the hash, so without these nothing ever hits.
    • Sets CCACHE_COMPILERCHECK=content so an image rebuild doesn't invalidate the cache.
    • Sets MAKEFLAGS=-j$MAKE_JOBS (default 2).
  • process-updates.yaml, weekly-rebuild-missing.yaml: source the script and set MAKE_JOBS: 3 to match their 3-CPU limit.
  • build-all-versions.yaml: source the script (2-CPU limit, so the default of 2 applies).
  • local/newer-r-helpers.R: add newer_r_outstanding(), which applies bincraft's skip decision in bulk. A candidate is dropped only when a binary that isn't a CRAN source sits at the path its cached ABI verdict routes it to. No cached verdict, or a slot that could not be listed, leaves the package for bincraft.
  • local/build-newer-r.R: before the loop, list the flat slot and this interpreter's per-minor slot once each, read the ABI verdicts in one query, filter with newer_r_outstanding(), and log how many were skipped.
  • local/tests/test-newer-r-helpers.R: tests for the routing, a sensitive package that only has a flat copy, source fallbacks, unlisted slots and empty inputs.

Verification

  • ccache, in the build-env-ubuntu:jammy, build-env-redhat:9 and build-env-alpine:3.24 images: installing fastmatch twice from different temp dirs gave 4/4 cache hits on the second build. The same test without CCACHE_BASEDIR/CCACHE_NOHASHDIR gave 0 hits.
  • All files in local/tests pass; crow lint .crow/ and prek run -a pass for the changed files.
  • The bulk listing and verdict query have not run against the real bucket and DB yet, because the credentials only exist in CI. The next nightly run will show it: look for the "N of M already carry a binary at their routed path" line.

Behaviour change

  • Builds that compile now run parallel make and reuse cached objects for unchanged translation units (version bumps, retries, weekly rebuilds). R 4.4 and R 4.5 passes do not share cache entries, because the R headers differ.
  • build-newer-r.R no longer calls bincraft for candidates that already have a binary at their routed path. Everything else is handled exactly as before.
  • A handful of CRAN packages break under parallel make. If one appears, set MAKE_JOBS: 1 for that pipeline.
Most of a `process-updates` run is per-package overhead, not compiling. In pipeline 14290 (ubuntu-2204-amd64) the `build-newer-r.R` loop spent about 4s on each of the ~1,100 candidates that already had a binary, and ccache was configured but never actually used. ## Changes - `local/compiler-cache.sh` (new): sourced by the build pipelines right after the clone. - Prepends ccache's masquerade dir to `PATH` (`/usr/lib/ccache/bin` on Alpine, `/usr/lib64/ccache` on RHEL, `/usr/lib/ccache` on Ubuntu). The images call bare `gcc`/`g++` in Makeconf and `/root/.R/Makevars`, so this catches every compile, CMake-driven vendored builds included, without replacing the images' Makevars through `R_MAKEVARS_USER`. - Sets `CCACHE_BASEDIR=/tmp` and `CCACHE_NOHASHDIR=1`. R compiles each package in a fresh `/tmp/RtmpXXXXXX` dir and its default `-g` puts that dir into the hash, so without these nothing ever hits. - Sets `CCACHE_COMPILERCHECK=content` so an image rebuild doesn't invalidate the cache. - Sets `MAKEFLAGS=-j$MAKE_JOBS` (default 2). - `process-updates.yaml`, `weekly-rebuild-missing.yaml`: source the script and set `MAKE_JOBS: 3` to match their 3-CPU limit. - `build-all-versions.yaml`: source the script (2-CPU limit, so the default of 2 applies). - `local/newer-r-helpers.R`: add `newer_r_outstanding()`, which applies bincraft's skip decision in bulk. A candidate is dropped only when a binary that isn't a CRAN source sits at the path its cached ABI verdict routes it to. No cached verdict, or a slot that could not be listed, leaves the package for bincraft. - `local/build-newer-r.R`: before the loop, list the flat slot and this interpreter's per-minor slot once each, read the ABI verdicts in one query, filter with `newer_r_outstanding()`, and log how many were skipped. - `local/tests/test-newer-r-helpers.R`: tests for the routing, a sensitive package that only has a flat copy, source fallbacks, unlisted slots and empty inputs. ## Verification - ccache, in the `build-env-ubuntu:jammy`, `build-env-redhat:9` and `build-env-alpine:3.24` images: installing `fastmatch` twice from different temp dirs gave 4/4 cache hits on the second build. The same test without `CCACHE_BASEDIR`/`CCACHE_NOHASHDIR` gave 0 hits. - All files in `local/tests` pass; `crow lint .crow/` and `prek run -a` pass for the changed files. - The bulk listing and verdict query have not run against the real bucket and DB yet, because the credentials only exist in CI. The next nightly run will show it: look for the "N of M already carry a binary at their routed path" line. ## Behaviour change - Builds that compile now run parallel make and reuse cached objects for unchanged translation units (version bumps, retries, weekly rebuilds). R 4.4 and R 4.5 passes do not share cache entries, because the R headers differ. - `build-newer-r.R` no longer calls bincraft for candidates that already have a binary at their routed path. Everything else is handled exactly as before. - A handful of CRAN packages break under parallel make. If one appears, set `MAKE_JOBS: 1` for that pipeline. 
Prepend ccache's masquerade dir to PATH in the build pipelines and set the ccache options that make hits possible for R's per-package temp dirs, plus MAKEFLAGS matched to each step's CPU limit.
Decide the already-built candidates from one listing per slot and one ABI-verdict query instead of a DB connection and several S3 requests per package; anything undecidable still goes through bincraft.
pat-s merged commit 773add042c into main 2026-10-23 13:18:00 +00:00
pat-s deleted branch perf/ccache-and-newer-r-prefilter 2026-10-23 13:18:01 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
devxy/build-cran-binaries!240
No description provided.