Author	SHA1	Message	Date
pat-s	a126d74cd3	fix(build): keep source fallbacks out of the S3 package cache (#159) ... ## Problem This is the gap flagged in rpkgs/bincraft#106. `build_binary_package()` has a fast path that compares against `s3_package_cache` instead of querying S3 per package, and that cache is produced here: ```r s3_pkgs <- s3fs::s3_dir_ls(".../latest/src/contrib", recurse = TRUE) saveRDS(basename(s3_pkgs), "/mnt/cache/packages/s3_cache.rds") ``` A raw bucket listing cannot tell a binary from a package whose build failed and was published as its CRAN source — the two occupy the same key. So every source fallback reads as "already built" and is skipped for good. That is how `alpine324` accumulated ~13.5k of them. The same listing feeds `s3_dt`, which is subtracted from the build list at line 194 (`pkgs <- pkgs_no_error[!s3_dt]`). That one matters more: it excludes the very packages that need building, before `build_binary_package()` is even called. ## What this changes Drops from the listing every object the slot's own index reports as served from source. bincraft leaves the `Built` stamp off exactly those records (rpkgs/bincraft#105), so the index already carries the answer and no credentials, downloads or extra API calls are needed. Both consumers are fixed: the saved cache and `s3_dt`. The cache stays a plain filename vector, so the build container still needs no `s3fs`/reticulate — that was the point of saving it in the first place. Two deliberately conservative edges: - archived objects have no index record, so they are kept. Unknown means binary, never "rebuild it". - if the index cannot be read, the full listing is kept and a warning is printed, so a CDN blip cannot mass-schedule a rebuild. ## Verification The script parses, and the new block run against the live indices: ``` amd64/alpine324: index=24235 source-served=13542 e.g. AATtools_0.0.3.tar.gz, ABCDscores_7.0.0.tar.gz amd64/noble: index=24681 source-served=0 ``` `alpine324` is re-indexed by bincraft 5.1.1, so 13 542 objects drop out and those packages become buildable. `noble` has not been re-indexed yet, so every record still carries `Built`, nothing is dropped, and its behaviour is exactly what it is today — the safe failure mode this relies on. The new log line makes it visible per run: ``` S3 cache: N objects, M served as CRAN source, K usable binaries ``` ## Sequencing Needs rpkgs/bincraft#106 (and a release) before a rebuild actually builds: this fixes the bulk build path's list, #106 fixes the per-package pre-build skip. Reviewed-on: #159	2026-08-09 16:27:51 +00:00
pat-s	ee15c50f53	refactor: migrate package installation from pak to uvr (#147) ... `bincraft` dropped pak in favour of uvr (5.0.x, "Dependencies and their system requirements are now installed with `uvr` instead of pak during `build_binary_package()`"), so the pipelines, helper scripts and images in this repo move with it. ## Approach uvr is project-scoped in a way pak is not: `uvr add` refuses to run outside a project and always installs into `.uvr/library/`, and only `uvr sync` honours `--library`. So there is no one-line `pak::pak(...)` equivalent. `local/uvr-install.sh` encapsulates the dance — bootstrap a pinned uvr, mint a throwaway project under `TMPDIR`, `uvr add --no-install`, then `uvr sync --library <target>`. Keeping the project outside the checkout also keeps `uvr init`'s `.Rprofile` from hijacking `.libPaths()` for every other R call in the pipeline. This matches what bincraft itself does (`uvr sync --install-system-deps --library <lib>`), and bincraft requires `uvr` on `PATH`, which the bootstrap provides: every pipeline that calls `bincraft::` runs `install-bincraft.R` (and therefore the bootstrap) first. ## Changes | File | Change | | --- | --- | | `local/uvr-install.sh` | **New.** The single replacement for `pak::pak(...)`. Bootstraps uvr `v0.4.4`, resolves the R interpreter from `UVR_R_BIN`/`R_VERSION`/`PATH`, pins the manifest to that R's exact version, and syncs into `UVR_TARGET_LIB`/`R_LIBS_USER`. | | `local/install-bincraft.R` | Installs `forgejo::codefloe.com/rpkgs/bincraft@<tag>` instead of a `git::` URL; keeps the `git ls-remote` tag resolution. Exports `UVR_R_BIN`/`UVR_TARGET_LIB` from `R.home()`/`.libPaths()[1]` so the per-R-minor passes target their own R and library. | | `.crow/auto-apply-patches.yaml`, `.crow/weekly-patch-proposals.yaml`, `.crow/weekly-audit-missing.yaml`, `.crow/weekly-rebuild-missing.yaml`, `.crow/build-all-versions-install-deps.yaml` | `pak::pak(...)` → `UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh ...`. The explicit `UVR_R_BIN` matters in `build-all-versions-install-deps.yaml`, which has no `R_VERSION` in its step environment. | | `.crow/build-all-versions.yaml`, `.crow/process-updates.yaml`, `.crow/weekly-rebuild-missing.yaml` | `R_PKG_CACHE_DIR` → `UVR_CACHE_DIR` + `UVR_PACKAGES_DIR` on the same `/mnt/cache` volume, preserving the amd64-off/arm64-on split. Drops the `rm -rf .../pkgcache/_metadata/...` cleanup. | | `local/r-minor-helpers.R`, `local/build-all.R`, `local/tests/test-trim-pkgcache.R` | Removes `trim_pkgcache_metadata()`, its every-25-packages call and its tests. uvr's cache does not mint a fresh ~70 MB snapshot per `PACKAGES` change. | | `.crow/build-all-versions-install-deps.yaml` | Drops `pak::sysreqs_db_update()`; uvr resolves sysreqs from its vendored `r-system-requirements` rules via `--install-system-deps`. | | `docker/Containerfile-shiny-app` | Bootstraps uvr and drives both dependency installs through one uvr project with `UVR_LIBRARY` pointed at the image's R library. | | `docker/build-one.Dockerfile` | Ships `uvr-install.sh` at `/work/local/` so `install-bincraft.R` finds it. | | `docker/reprex/alpine.sh` | Replaces `pak::local_install_deps()` with DESCRIPTION parsing + `uvr add`. | | `local/test-package-loading.R` | Installs via the helper instead of `pak::pkg_install()`. | | `README.md` | Documents uvr for sysreq inference, archived-version installs and cache clearing. | | `renovate.json` | Tracks the `UVR_PIN` in `uvr-install.sh` via `github-releases`. | ## Behaviour notes - **`weekly-audit-missing` still takes bincraft from the default branch**, not the latest release tag, matching what the `git::` pak call did. Called out in a comment rather than silently changed. - **The uvr pin is repo-wide.** bincraft resolves `uvr` from `PATH` and pins no version of its own, so `UVR_PIN` in `uvr-install.sh` governs the whole pipeline. - **Persistent caches now also benefit bincraft**, which reads `UVR_CACHE_DIR`/`UVR_PACKAGES_DIR` from the inherited pipeline environment. - **`uvr sync` will not prune the shared library.** Pruning is disabled whenever `--library` is passed (`do_prune = prune && library_override.is_none()`), so `/mnt/cache/R-pkgs` keeps bincraft and its dependencies. The wipe-on-ABI-mismatch path is *not* similarly guarded, which is why the helper pins the manifest to the active R's exact version. - **`plans/` and `specs/` are untouched** — they are dated records of decisions made in June 2026 and describe bincraft's then-pak-based internals; rewriting them would misstate history. ## Verification `shellcheck`, all pre-commit hooks (on this commit's file range) and the `local/tests/` suite (100 assertions) pass. Not yet exercised in CI: the build-env images do not ship `uvr`, so the per-step `curl install.sh` bootstrap is untested against a real image. Worth a manual `build-all-versions-install-deps` run before merging. Reviewed-on: #147	2026-07-31 12:19:26 +00:00
pat-s	d558e27c11	feat: R-minor-sensitive binary builds (full + iterative) (#84) ... ## Summary Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered. ## What's in here **Detection / precompute** - `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests. - `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions. **Full build** - `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset. - `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor. **Iterative build** - All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added. **Tooling / housekeeping** - Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`. - Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts. - Design + implementation docs under `docs/superpowers/`. ## Requires before merge - A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected. Reviewed-on: #84	2026-06-14 08:04:40 +00:00
pat-s	8a98cf96a6	fix: save S3 listing in precompute step, avoid loading s3fs in build step ... s3fs uses reticulate/Python which significantly increases R process memory. When pak tries to fork R to install package dependencies, the enlarged process can't be duplicated within the container memory limit. Now the install-deps step saves the S3 file listing as s3_cache.rds, and the build step reads it with readRDS() — no s3fs loading needed.	2026-04-17 20:05:26 +02:00
pat-s	87d3f54b21	fix: pass NULL to set_codename() which requires an explicit argument	2026-04-17 18:55:40 +02:00
pat-s	92084e51ed	fix: use bincraft::set_codename() for S3 path in precompute step ... The precompute script was constructing S3 paths as "redhat10" while bincraft uses "rhel10" as the codename. This mismatch caused the precompute to find no existing packages in S3, producing ~77K false positive package versions that all get skipped at build time. Now uses bincraft::set_codename() to ensure path alignment. Also fixes nrow() vs length() in install-deps summary message.	2026-04-17 18:54:29 +02:00
pat-s	21ec8a67dd	feat: save package+version pairs instead of just package names ... packages-to-build.R now saves the full (Package, Version) data.table so the build step can pass specific tags to build_binary_package(), eliminating redundant per-package tag discovery and S3 checks. Also reduces archive versions from 9 to 4 (+ 1 release = 5 total).	2026-04-17 18:20:00 +02:00
pat-s	aa2a6edef4	fix: write error message to stdout so it appears in CI logs	2026-04-09 23:06:10 +02:00
pat-s	852f732933	fix: compute platform for all distros and add error handler to surface failures ... platform was only set inside an alpine-specific if block, causing undefined variable errors for ubuntu/redhat builds that silently exited with code 1.	2026-04-09 22:59:46 +02:00
pat-s	9568d4dbfa	fix: migrate packages-to-build.R S3 config from Hetzner to Backblaze ... The S3 credentials, endpoint, and bucket name were still pointing to the old Hetzner storage after the Backblaze migration, causing silent build failures.	2026-04-09 22:53:49 +02:00
pat-s	30753d8871	use new postgres db and user	2025-06-25 15:53:40 +02:00
pat-s	6b1ae4e555	chore: cleanup workflows	2025-06-17 21:03:49 +02:00
pat-s	05e0ecdae5	new build approach	2025-06-17 01:15:56 +02:00