| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| ee15c50f53 |
refactor: migrate package installation from pak to uvr (#147)
Some checks failed
ci/crow/cron/weekly-rebuild-missing/2 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/14 Pipeline failed
ci/crow/cron/weekly-rebuild-missing/9 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/3 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/10 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/13 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/11 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/4 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/12 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/1 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/16 Pipeline was successful
ci/crow/cron/weekly-rebuild-missing/15 Pipeline was successful
ci/crow/cron/process-updates/10 Pipeline was successful
ci/crow/cron/process-updates/4 Pipeline was successful
ci/crow/cron/process-updates/13 Pipeline was successful
ci/crow/cron/process-updates/14 Pipeline was successful
ci/crow/cron/process-updates/15 Pipeline was successful
ci/crow/cron/process-updates/16 Pipeline was successful
ci/crow/cron/process-updates/11 Pipeline was successful
ci/crow/cron/process-updates/18 Pipeline was successful
ci/crow/cron/process-updates/17 Pipeline was successful
ci/crow/cron/process-updates/12 Pipeline was successful
ci/crow/cron/process-updates/6 Pipeline was successful
ci/crow/cron/process-updates/5 Pipeline was successful
ci/crow/cron/process-updates/1 Pipeline was successful
ci/crow/cron/process-updates/2 Pipeline was successful
ci/crow/cron/process-updates/7 Pipeline was successful
ci/crow/cron/process-updates/8 Pipeline was successful
ci/crow/cron/process-updates/9 Pipeline was successful
ci/crow/cron/process-updates/3 Pipeline was successful
`bincraft` dropped pak in favour of uvr (5.0.x, "Dependencies and their system requirements are now installed with `uvr` instead of pak during `build_binary_package()`"), so the pipelines, helper scripts and images in this repo move with it. ## Approach uvr is project-scoped in a way pak is not: `uvr add` refuses to run outside a project and always installs into `.uvr/library/`, and only `uvr sync` honours `--library`. So there is no one-line `pak::pak(...)` equivalent. `local/uvr-install.sh` encapsulates the dance — bootstrap a pinned uvr, mint a throwaway project under `TMPDIR`, `uvr add --no-install`, then `uvr sync --library <target>`. Keeping the project outside the checkout also keeps `uvr init`'s `.Rprofile` from hijacking `.libPaths()` for every other R call in the pipeline. This matches what bincraft itself does (`uvr sync --install-system-deps --library <lib>`), and bincraft requires `uvr` on `PATH`, which the bootstrap provides: every pipeline that calls `bincraft::` runs `install-bincraft.R` (and therefore the bootstrap) first. ## Changes | File | Change | | --- | --- | | `local/uvr-install.sh` | **New.** The single replacement for `pak::pak(...)`. Bootstraps uvr `v0.4.4`, resolves the R interpreter from `UVR_R_BIN`/`R_VERSION`/`PATH`, pins the manifest to that R's exact version, and syncs into `UVR_TARGET_LIB`/`R_LIBS_USER`. | | `local/install-bincraft.R` | Installs `forgejo::codefloe.com/rpkgs/bincraft@<tag>` instead of a `git::` URL; keeps the `git ls-remote` tag resolution. Exports `UVR_R_BIN`/`UVR_TARGET_LIB` from `R.home()`/`.libPaths()[1]` so the per-R-minor passes target their own R and library. | | `.crow/auto-apply-patches.yaml`, `.crow/weekly-patch-proposals.yaml`, `.crow/weekly-audit-missing.yaml`, `.crow/weekly-rebuild-missing.yaml`, `.crow/build-all-versions-install-deps.yaml` | `pak::pak(...)` → `UVR_R_BIN=/opt/R/$R_VERSION/bin/R local/uvr-install.sh ...`. The explicit `UVR_R_BIN` matters in `build-all-versions-install-deps.yaml`, which has no `R_VERSION` in its step environment. | | `.crow/build-all-versions.yaml`, `.crow/process-updates.yaml`, `.crow/weekly-rebuild-missing.yaml` | `R_PKG_CACHE_DIR` → `UVR_CACHE_DIR` + `UVR_PACKAGES_DIR` on the same `/mnt/cache` volume, preserving the amd64-off/arm64-on split. Drops the `rm -rf .../pkgcache/_metadata/...` cleanup. | | `local/r-minor-helpers.R`, `local/build-all.R`, `local/tests/test-trim-pkgcache.R` | Removes `trim_pkgcache_metadata()`, its every-25-packages call and its tests. uvr's cache does not mint a fresh ~70 MB snapshot per `PACKAGES` change. | | `.crow/build-all-versions-install-deps.yaml` | Drops `pak::sysreqs_db_update()`; uvr resolves sysreqs from its vendored `r-system-requirements` rules via `--install-system-deps`. | | `docker/Containerfile-shiny-app` | Bootstraps uvr and drives both dependency installs through one uvr project with `UVR_LIBRARY` pointed at the image's R library. | | `docker/build-one.Dockerfile` | Ships `uvr-install.sh` at `/work/local/` so `install-bincraft.R` finds it. | | `docker/reprex/alpine.sh` | Replaces `pak::local_install_deps()` with DESCRIPTION parsing + `uvr add`. | | `local/test-package-loading.R` | Installs via the helper instead of `pak::pkg_install()`. | | `README.md` | Documents uvr for sysreq inference, archived-version installs and cache clearing. | | `renovate.json` | Tracks the `UVR_PIN` in `uvr-install.sh` via `github-releases`. | ## Behaviour notes - **`weekly-audit-missing` still takes bincraft from the default branch**, not the latest release tag, matching what the `git::` pak call did. Called out in a comment rather than silently changed. - **The uvr pin is repo-wide.** bincraft resolves `uvr` from `PATH` and pins no version of its own, so `UVR_PIN` in `uvr-install.sh` governs the whole pipeline. - **Persistent caches now also benefit bincraft**, which reads `UVR_CACHE_DIR`/`UVR_PACKAGES_DIR` from the inherited pipeline environment. - **`uvr sync` will not prune the shared library.** Pruning is disabled whenever `--library` is passed (`do_prune = prune && library_override.is_none()`), so `/mnt/cache/R-pkgs` keeps bincraft and its dependencies. The wipe-on-ABI-mismatch path is *not* similarly guarded, which is why the helper pins the manifest to the active R's exact version. - **`plans/` and `specs/` are untouched** — they are dated records of decisions made in June 2026 and describe bincraft's then-pak-based internals; rewriting them would misstate history. ## Verification `shellcheck`, all pre-commit hooks (on this commit's file range) and the `local/tests/` suite (100 assertions) pass. Not yet exercised in CI: the build-env images do not ship `uvr`, so the per-step `curl install.sh` bootstrap is untested against a real image. Worth a manual `build-all-versions-install-deps` run before merging. Reviewed-on: #147 |
|||
| d558e27c11 |
feat: R-minor-sensitive binary builds (full + iterative) (#84)
## Summary
Builds R-minor-sensitive CRAN packages once per installed R minor version (into per-minor S3 slots `…/contrib/<x.y>/`) and everything else once into the generic slot, driven by bincraft 4.2.0's ABI classifier. Both the full and iterative pipelines are covered.
## What's in here
**Detection / precompute**
- `local/r-minor-helpers.R` — pure `classify_from_metadata()` (NeedsCompilation / risky `LinkingTo`) + `parse_build_args()`, with unit tests.
- `local/packages-to-build.R` — adds a per-package `r_minor_sensitive` flag: cheap CRAN-metadata rules first, source download + `bincraft::needs_per_minor_recompile()` only for the ambiguous compiled subset (fail-safe to sensitive). Classified once per package, applied to all versions.
**Full build**
- `local/build-all.R` — passes the per-row `is_r_minor_sensitive` flag; new `--sensitive-only` mode builds just the risky subset.
- `.crow/build-all-versions-{amd64,arm64}.yaml` — install-deps persists the sensitive subset; build step runs a sensitive-only pass under each non-primary `/opt/R/*` minor; new step uploads the generic index plus a per-minor index for each minor.
**Iterative build**
- All 14 `.crow/process-updates-*.yaml` — primary pass uses `r_minor_detection = 'classifier'`; a sensitive-only multi-R pass builds risky updates under each other minor; per-minor index upload added.
**Tooling / housekeeping**
- Pins bincraft `v4.1.1` → `v4.2.0` across all workflows; removes the superseded standalone `build-r-minor-sensitive-packages.yaml`.
- Adds prek/pre-commit hooks (prettier, markdownlint, editorconfig-checker, yamllint, air) and applies them repo-wide; excludes the verbatim GPL `LICENSE.md` and auxiliary shell scripts.
- Design + implementation docs under `docs/superpowers/`.
## Requires before merge
- A `v4.2.0` git tag must be pushed on the bincraft repo (codefloe.com/rpkgs/bincraft) — the workflow install steps pin `@v4.2.0`. The full-build install-deps clones `main`, so it is unaffected.
Reviewed-on: #84
|
|||
|
8a98cf96a6 |
fix: save S3 listing in precompute step, avoid loading s3fs in build step
Some checks failed
ci/crow/cron/update-package-index-alpine-322-arm64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-323-amd64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-323-arm64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-8-amd64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-8-arm64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-9-amd64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-9-arm64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2204-arm64 Pipeline was successful
ci/crow/manual/build-all-versions-arm64/3 Pipeline failed
ci/crow/cron/update-package-index-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2404-arm64 Pipeline was successful
ci/crow/manual/build-all-versions-arm64/4 Pipeline failed
ci/crow/manual/build-all-versions-arm64/1 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/manual/build-all-versions-amd64/2 Pipeline failed
ci/crow/manual/build-all-versions-amd64/4 Pipeline failed
ci/crow/manual/build-all-versions-amd64/1 Pipeline failed
ci/crow/manual/build-all-versions-amd64/3 Pipeline failed
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
s3fs uses reticulate/Python which significantly increases R process memory. When pak tries to fork R to install package dependencies, the enlarged process can't be duplicated within the container memory limit. Now the install-deps step saves the S3 file listing as s3_cache.rds, and the build step reads it with readRDS() — no s3fs loading needed. |
|||
|
87d3f54b21 |
fix: pass NULL to set_codename() which requires an explicit argument
All checks were successful
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
|
|||
|
92084e51ed |
fix: use bincraft::set_codename() for S3 path in precompute step
Some checks failed
ci/crow/manual/build-all-versions-amd64/4 Pipeline is pending
ci/crow/manual/build-all-versions-install-deps-amd64 Pipeline was successful
ci/crow/manual/build-all-versions-amd64/3 Pipeline failed
ci/crow/manual/build-all-versions-amd64/2 Pipeline failed
ci/crow/manual/build-all-versions-amd64/1 Pipeline failed
The precompute script was constructing S3 paths as "redhat10" while bincraft uses "rhel10" as the codename. This mismatch caused the precompute to find no existing packages in S3, producing ~77K false positive package versions that all get skipped at build time. Now uses bincraft::set_codename() to ensure path alignment. Also fixes nrow() vs length() in install-deps summary message. |
|||
|
21ec8a67dd |
feat: save package+version pairs instead of just package names
packages-to-build.R now saves the full (Package, Version) data.table so the build step can pass specific tags to build_binary_package(), eliminating redundant per-package tag discovery and S3 checks. Also reduces archive versions from 9 to 4 (+ 1 release = 5 total). |
|||
|
aa2a6edef4 |
fix: write error message to stdout so it appears in CI logs
Some checks failed
ci/crow/manual/build-all-versions-amd64/1 Pipeline failed
ci/crow/manual/build-all-versions-amd64/2 Pipeline failed
ci/crow/manual/build-all-versions-amd64/5 Pipeline failed
ci/crow/manual/build-all-versions-amd64/4 Pipeline failed
ci/crow/manual/build-all-versions-amd64/3 Pipeline failed
|
|||
|
852f732933 |
fix: compute platform for all distros and add error handler to surface failures
Some checks failed
ci/crow/manual/build-all-versions-amd64/5 Pipeline failed
ci/crow/manual/build-all-versions-amd64/2 Pipeline failed
ci/crow/manual/build-all-versions-amd64/4 Pipeline failed
ci/crow/manual/build-all-versions-amd64/1 Pipeline failed
ci/crow/manual/build-all-versions-amd64/3 Pipeline failed
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
platform was only set inside an alpine-specific if block, causing undefined variable errors for ubuntu/redhat builds that silently exited with code 1. |
|||
|
9568d4dbfa |
fix: migrate packages-to-build.R S3 config from Hetzner to Backblaze
Some checks failed
ci/crow/manual/build-all-versions-amd64/3 Pipeline failed
ci/crow/manual/build-all-versions-amd64/1 Pipeline failed
ci/crow/manual/build-all-versions-amd64/5 Pipeline failed
ci/crow/manual/build-all-versions-amd64/2 Pipeline failed
ci/crow/manual/build-all-versions-amd64/4 Pipeline failed
The S3 credentials, endpoint, and bucket name were still pointing to the old Hetzner storage after the Backblaze migration, causing silent build failures. |
|||
|
30753d8871 |
use new postgres db and user
Some checks failed
ci/crow/manual/build-all-versions-install-deps Pipeline was successful
ci/crow/manual/build-all-versions-arm64/2 Pipeline was successful
ci/crow/manual/build-all-versions-arm64/5 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-321-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/manual/build-all-versions-arm64/4 Pipeline failed
ci/crow/manual/build-all-versions-arm64/3 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-321-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline failed
ci/crow/manual/build-all-versions-arm64/1 Pipeline failed
|
|||
|
6b1ae4e555 |
chore: cleanup workflows
Some checks failed
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline failed
ci/crow/manual/build-all-versions-install-deps Pipeline was successful
ci/crow/manual/build-all-versions/7 Pipeline is running
ci/crow/manual/build-all-versions/3 Pipeline is running
ci/crow/manual/build-all-versions/4 Pipeline is running
ci/crow/manual/build-all-versions/9 Pipeline is running
ci/crow/manual/build-all-versions/1 Pipeline is running
ci/crow/manual/build-all-versions/8 Pipeline is running
ci/crow/manual/build-all-versions/2 Pipeline is running
ci/crow/manual/build-all-versions/5 Pipeline is running
ci/crow/manual/build-all-versions/10 Pipeline is running
ci/crow/manual/build-all-versions/6 Pipeline is running
|
|||
|
05e0ecdae5 |
new build approach
Some checks failed
ci/crow/manual/alpine-322-arm64-install-deps Pipeline was successful
ci/crow/cron/process-updates-alpine-321-arm64 Pipeline was successful
ci/crow/cron/process-updates-alpine-320-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-321-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-320-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/manual/build-alpine-322-arm64-2 Pipeline failed
ci/crow/manual/build-alpine-322-arm64-1 Pipeline failed
ci/crow/manual/build-alpine-322-arm64-3 Pipeline failed
ci/crow/manual/build-alpine-322-arm64-4 Pipeline failed
ci/crow/manual/build-alpine-322-arm64-5 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline was successful
|
| Author | SHA1 | Date | |
|---|---|---|---|
| ee15c50f53 | |||
| d558e27c11 | |||
|
8a98cf96a6 |
|||
|
87d3f54b21 |
|||
|
92084e51ed |
|||
|
21ec8a67dd |
|||
|
aa2a6edef4 |
|||
|
852f732933 |
|||
|
9568d4dbfa |
|||
|
30753d8871 |
|||
|
6b1ae4e555 |
|||
|
05e0ecdae5 |