From 59c3e0960c527c0f26306fb8b6a048672b86b797 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 18:53:17 +0200 Subject: [PATCH 01/14] docs: design for R-minor-sensitive binary builds Spec for integrating bincraft's ABI classifier (PR #49) into the iterative and full build pipelines so only R-minor-sensitive packages are built per installed R minor, leveraging the multi-R exec-env images. --- ...6-06-13-r-minor-sensitive-builds-design.md | 148 ++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md diff --git a/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md b/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md new file mode 100644 index 0000000..81f828f --- /dev/null +++ b/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md @@ -0,0 +1,148 @@ +# R-minor-sensitive binary builds — design + +## Problem + +CRAN binaries are currently built once, under a single R minor version, and served from a +single generic slot (`…/latest/src/contrib/`). +That is wrong for the minority of packages whose compiled code reaches into volatile R +internals: a binary built under R 4.5 will fail to load under R 4.4 with an +`undefined symbol` error. + +bincraft v4.1.0+ can now detect these packages automatically via the ABI classifier +(`abi_classify()` / `needs_per_minor_recompile()`, added in bincraft PR #49, +). +The exec-env images now ship multiple R minor versions under `/opt/R/`. +This design uses both to build R-minor-specific binaries for the sensitive packages only, +in both the iterative (`process-updates-*`) and full (`build-all-versions-*`) pipelines. + +## Background: how bincraft already behaves + +- `abi_classify(path)` returns a tier: `pure-r` (~78.6%), `safe-compiled` (~7.7%), or + `risky` (~13.6%). `needs_per_minor_recompile(path)` is the boolean wrapper + (`TRUE` iff `risky`). Both need the package source (DESCRIPTION + `src/`). +- `build_binary_package(…, is_r_minor_sensitive = TRUE)` uploads the artifact into a + per-minor slot `…/latest/src/contrib//` and records `r_version` in the + build metadata. With `FALSE` it uses the generic slot. The minor is derived from the + **running interpreter** (`R.version`), so producing a 4.4 binary requires running + `/opt/R/4.4.x/bin/R`. +- `process_cran_updates()` orchestrates the iterative flow with a single + `is_r_minor_sensitive` bool for the whole run and no per-package classification or + R-version loop. +- `upload_package_index()` writes/uploads `PACKAGES*` for the **generic slot only**; it + has no per-minor support. The current standalone r-minor workflow never builds a + per-minor index, so per-minor slots are effectively unservable today. + +## Core model + +Collapse the three tiers to one boolean per package: + +``` +r_minor_sensitive := (abi_classify(pkg)$tier == "risky") +``` + +| group | tiers | built under | slot | +| ------------------------------ | ---------------------- | -------------------- | ---------------------------- | +| non-sensitive (~86%) | pure-r, safe-compiled | primary R only | generic `contrib/` | +| sensitive (~14%) | risky | every installed minor| per-minor `contrib//` | + +- **Primary R** = the existing `R_VERSION` env var in each workflow. Its pass builds the + non-sensitive packages (generic slot) *and* the sensitive packages for its own minor slot. +- **Extra minors** = every other R version discovered by scanning `/opt/R/` at runtime. + Each runs a **sensitive-only** pass. An image with a single R version degrades cleanly + to just the primary pass. +- The loop over minors always lives at the shell/script layer (one `/opt/R//bin/R` + invocation per minor), never inside a single `build_binary_package()` call. + +Classification granularity: classify **once per package** (its release version) and apply +the resulting flag to all archived versions of that package. Tier rarely changes across +recent versions; this avoids multiplying source downloads. + +## Full build — `build-all-versions-*` + `local/` + +### install-deps step (`local/packages-to-build.R`) + +After computing `pkgs_to_build`, add an `r_minor_sensitive` logical column: + +1. Pull `NeedsCompilation` and `LinkingTo` from `tools::CRAN_package_db()` (already loaded). +2. Resolve cheaply, no download: + - `NeedsCompilation != "yes"` → not sensitive (rule 1, pure-r). + - `LinkingTo` references any `bincraft::abi_risky_linking_deps()` entry → sensitive + (rule 2). +3. For the remaining compiled, non-LinkingTo-risky packages only: download the source and + call `bincraft::needs_per_minor_recompile()` (rules 3/4). +4. Join the per-package flag onto every `(Package, Version)` row. + +Outputs: +- `pkgs_to_build.rds` — now carries the `r_minor_sensitive` column. +- `r_minor_sensitive_pkgs.rds` — the sensitive subset, for the extra-minor passes. + +### build step (`local/build-all.R` + `.crow/build-all-versions-{amd64,arm64}.yaml`) + +- `build-all.R` gains an optional `--sensitive-only` mode (or an arg flag). In normal mode + it builds the full chunk, passing `is_r_minor_sensitive = ` per package. In + sensitive-only mode it reads `r_minor_sensitive_pkgs.rds`, intersects with its chunk, and + builds those with `is_r_minor_sensitive = TRUE`. +- The workflow step keeps the existing matrix split. After the primary `Rscript build-all.R` + invocation, a shell loop discovers non-primary `/opt/R/*` minors and runs + `Rscript build-all.R --sensitive-only ` under each. +- Index upload step: generic index as today, plus a per-minor index for each minor slot + that received artifacts (depends on the `upload_package_index()` enhancement below). + +## Iterative build — `process-updates-*` + +Driven by the bincraft enhancement below; the single build step becomes: + +1. Primary-R pass: + `process_cran_updates(…, r_minor_detection = "classifier")`. + Each updated/new package is classified; risky → primary minor slot, rest → generic slot. + Removed-package handling stays as-is. +2. Shell loop over non-primary `/opt/R/*` minors: + `process_cran_updates(…, r_minor_detection = "classifier", r_minor_sensitive_only = TRUE)`. + Builds only risky updates into their respective minor slots. +3. Index upload extended to cover each touched minor slot in addition to the generic slot. + +## Required bincraft enhancements (separate PR, coordinated release) + +1. `build_binary_package()`: + - Accept `is_r_minor_sensitive = "auto"` — classify the source it already clones via + `abi_classify()` and route the artifact to the per-minor slot iff `risky`. + - Add `r_minor_sensitive_only` — when `TRUE`, skip (return `"skipped"`) non-risky + packages early, after classification, before building. +2. `process_cran_updates()`: + - Add `r_minor_detection = c("none", "issue", "classifier")` (default `"none"` to + preserve current behavior; `"issue"` is today's `filter_r_minor_sensitive` path). + - Add `r_minor_sensitive_only`, threaded down to `build_binary_package()`. + - With `"classifier"`, pass `is_r_minor_sensitive = "auto"` per package instead of a + single run-wide bool. +3. `upload_package_index()`: + - Add per-minor slot support: write/upload `PACKAGES*` (and `Meta/archive.rds`) under + `…/contrib//`, mirroring the generic-slot logic. Invocable per minor. + +These three are the only bincraft changes; detection itself (PR #49) is already merged. + +## Scope and cleanup + +- In scope: `.crow/process-updates-*` (iterative) and `.crow/build-all-versions-*` + + `local/build-all.R` + `local/packages-to-build.R` (full), across amd64 and arm64 and all + platforms (alpine, ubuntu, redhat). The per-platform workflow files share the same edit. +- Removed: `.crow/build-r-minor-sensitive-packages.yaml` — superseded by the integrated + flow (it was manual, alpine-3.21-only, and issue-list-driven). +- Out of scope (call out, do not change here): `weekly-rebuild-missing-*`, + `archive-missed-packages`, and the audit workflows. Revisit separately if per-minor + rebuilds are wanted there too. + +## Open risks / notes + +- **R-version discovery**: assumes `/opt/R//bin/R` layout and that the primary + `R_VERSION` is one of the installed versions. Parse minor as `major.minor` from each + discovered version; dedupe by minor (build once per minor even if two patch releases + coexist). +- **Per-minor index correctness**: clients resolving `bin//contrib//` require the + per-minor `PACKAGES` to exist; the `upload_package_index()` enhancement is a hard + dependency for the sensitive artifacts to be usable. Verify against a real client + install before declaring done. +- **Classification cost**: bounded by downloading sources only for the compiled, + non-LinkingTo-risky subset in install-deps. Worth measuring on a full run; if still too + heavy, consider caching classifications keyed by package+version in the metadata DB. +- **Double download**: install-deps classification downloads some sources that the build + step re-downloads. Acceptable for now; the metadata-DB cache above would also remove this. -- 2.54.0 From d328f261d248bcaf439b45cfe0bcc6b66873ef2f Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 19:10:55 +0200 Subject: [PATCH 02/14] docs: implementation plans for R-minor-sensitive builds Split into two sequenced plans: bincraft 4.2.0 enhancements (per-minor index + classifier-driven process_cran_updates) and the pipeline integration (full + iterative). Spec revised to classify at the orchestration layer instead of a build_binary_package "auto" mode. --- ...026-06-13-bincraft-r-minor-enhancements.md | 470 +++++++++++++++++ .../2026-06-13-pipeline-r-minor-builds.md | 471 ++++++++++++++++++ ...6-06-13-r-minor-sensitive-builds-design.md | 33 +- 3 files changed, 961 insertions(+), 13 deletions(-) create mode 100644 docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md create mode 100644 docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md diff --git a/docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md b/docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md new file mode 100644 index 0000000..b52735a --- /dev/null +++ b/docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md @@ -0,0 +1,470 @@ +# bincraft R-minor Enhancements Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Teach bincraft to (a) build/serve per-R-minor package indexes and (b) drive `process_cran_updates()` from the ABI classifier so only risky packages are built per R minor. + +**Architecture:** Two additive, low-risk changes to the bincraft R package. `upload_package_index()` gains an `r_minor` argument so a per-minor slot (`…/contrib//`) gets its own `PACKAGES*`. `process_cran_updates()` gains `r_minor_detection`/`r_minor_sensitive_only` and classifies each candidate via `needs_per_minor_recompile()` (already in the package), passing a concrete `is_r_minor_sensitive` logical to the unchanged `build_binary_package()`. + +**Tech Stack:** R package, `testthat` (3e) + `mockery` for tests, `roxygen2` for docs, `s3fs`/`cranlike` for the index. Repo: `https://codefloe.com/rpkgs/bincraft` (this plan is executed in a clone of that repo, NOT in build-cran-binaries). + +**Pre-req:** PR #49 (ABI classifier) is already merged on `main`; `abi_classify()`, `needs_per_minor_recompile()`, `abi_risky_linking_deps()` are exported. + +--- + +### Task 1: `package_index_remote_dir()` pure helper + +**Files:** +- Modify: `R/package_index.R` (top of file, before `add_to_package_index`) +- Test: `tests/testthat/test-package_index.R` (new) + +- [ ] **Step 1: Write the failing test** + +```r +# tests/testthat/test-package_index.R +test_that("package_index_remote_dir builds the generic slot when r_minor is NULL", { + expect_identical( + package_index_remote_dir("bucket", "amd64", "alpine323"), + file.path("bucket", "amd64", "alpine323", "latest", "src", "contrib") + ) +}) + +test_that("package_index_remote_dir appends the minor slot when r_minor is set", { + expect_identical( + package_index_remote_dir("bucket", "amd64", "alpine323", r_minor = "4.4"), + file.path("bucket", "amd64", "alpine323", "latest", "src", "contrib", "4.4") + ) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-package_index.R")'` +Expected: FAIL — `could not find function "package_index_remote_dir"`. + +- [ ] **Step 3: Add the helper** + +```r +# R/package_index.R — add near the top, above add_to_package_index() +#' Build the S3 remote contrib dir for a package index +#' +#' @param r_minor Optional `"major.minor"` string (e.g. `"4.4"`). When non-NULL +#' the path points at the per-minor slot. +#' @keywords internal +package_index_remote_dir <- function(s3_bucket, arch, codename, r_minor = NULL) { + base <- file.path(s3_bucket, arch, codename, "latest", "src", "contrib") + if (is.null(r_minor)) { + base + } else { + file.path(base, r_minor) + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-package_index.R")'` +Expected: PASS (2/2). + +- [ ] **Step 5: Commit** + +```bash +git add R/package_index.R tests/testthat/test-package_index.R +git commit -m "feat(index): add package_index_remote_dir helper for per-minor slots" +``` + +--- + +### Task 2: per-minor support in `upload_package_index()` + +**Files:** +- Modify: `R/package_index.R` (`upload_package_index`, signature + the `remote_bin_dir` block at lines ~117-124; `add_to_package_index` for consistency) + +- [ ] **Step 1: Add `r_minor` to the signature and use the helper** + +In `upload_package_index()`, add `r_minor = NULL` to the argument list (after `arch = NULL`). Replace the inline `remote_bin_dir <- file.path(s3_bucket, arch, codename, "latest", "src", "contrib")` block with: + +```r + remote_bin_dir <- package_index_remote_dir(s3_bucket, arch, codename, r_minor) +``` + +Do the same replacement in `add_to_package_index()` (it has the identical inline construction); add `r_minor = NULL` to its signature too. + +- [ ] **Step 2: Add the roxygen param** + +Above `upload_package_index` and `add_to_package_index`, add: + +```r +#' @param r_minor Optional `"major.minor"` string. When set, the index is +#' written/read under the per-minor slot `…/contrib//` instead of the +#' generic `…/contrib/` slot. +``` + +- [ ] **Step 3: Regenerate docs** + +Run: `R -q -e 'devtools::document()'` +Expected: updated `man/upload_package_index.Rd`, `man/add_to_package_index.Rd`, no errors. + +- [ ] **Step 4: Verify package still loads and existing tests pass** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_dir("tests/testthat")'` +Expected: PASS — no regressions; the new path test from Task 1 still green. + +- [ ] **Step 5: Commit** + +```bash +git add R/package_index.R man/ +git commit -m "feat(index): upload/update PACKAGES for a per-minor slot via r_minor" +``` + +--- + +### Task 3: `classify_r_minor_sensitive()` internal helper + +**Files:** +- Modify: `R/process_cran_updates.R` (add helper near the top) +- Test: `tests/testthat/test-process_cran_updates.R` (new) + +- [ ] **Step 1: Write the failing test** + +```r +# tests/testthat/test-process_cran_updates.R +test_that("classify_r_minor_sensitive returns TRUE for a risky package", { + skip_if_not_installed("mockery") + mockery::stub(classify_r_minor_sensitive, "clone_repository", function(pkg, tag, url, dest) { + dir.create(dest, recursive = TRUE, showWarnings = FALSE) + writeLines( + c("Package: dummy", "Version: 1.0", "NeedsCompilation: yes", "LinkingTo: Rcpp"), + file.path(dest, "DESCRIPTION") + ) + dir.create(file.path(dest, "src")) + writeLines("// Rcpp glue", file.path(dest, "src", "x.cpp")) + }) + expect_true(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran")) +}) + +test_that("classify_r_minor_sensitive returns FALSE for a pure-r package", { + skip_if_not_installed("mockery") + mockery::stub(classify_r_minor_sensitive, "clone_repository", function(pkg, tag, url, dest) { + dir.create(dest, recursive = TRUE, showWarnings = FALSE) + writeLines(c("Package: dummy", "Version: 1.0", "NeedsCompilation: no"), + file.path(dest, "DESCRIPTION")) + }) + expect_false(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran")) +}) + +test_that("classify_r_minor_sensitive fails safe to TRUE on clone error", { + skip_if_not_installed("mockery") + mockery::stub(classify_r_minor_sensitive, "clone_repository", function(...) stop("boom")) + expect_true(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran")) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` +Expected: FAIL — `could not find function "classify_r_minor_sensitive"`. + +- [ ] **Step 3: Add the helper** + +```r +# R/process_cran_updates.R — add above process_cran_updates() +#' Classify a single CRAN package for R-minor sensitivity +#' +#' Clones the package source to a temp dir and runs +#' [needs_per_minor_recompile()]. Fails safe to `TRUE` (build per minor) if the +#' clone or classification errors, so a possibly-ABI-fragile binary is never +#' served from the cross-minor generic slot by mistake. +#' @keywords internal +classify_r_minor_sensitive <- function( + package_name, + tag, + source_org_url = "https://github.com/cran", + local_clone_dir = tempdir() +) { + dest <- file.path( + local_clone_dir, + sprintf("classify_%s_%s", package_name, tag) + ) + on.exit(unlink(dest, recursive = TRUE, force = TRUE), add = TRUE) + tryCatch( + { + clone_repository(package_name, tag, source_org_url, dest) + isTRUE(as.logical(needs_per_minor_recompile(dest))) + }, + error = function(e) { + log_warn(sprintf( + "{.fun classify_r_minor_sensitive}: failed for {.pkg %s} {.field %s}: %s. Treating as r-minor-sensitive.", # nolint + package_name, + tag, + conditionMessage(e) + )) + TRUE + } + ) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` +Expected: PASS (3/3). + +- [ ] **Step 5: Commit** + +```bash +git add R/process_cran_updates.R tests/testthat/test-process_cran_updates.R +git commit -m "feat(updates): add classify_r_minor_sensitive helper" +``` + +--- + +### Task 4: wire classifier detection into `process_cran_updates()` + +**Files:** +- Modify: `R/process_cran_updates.R` (`process_cran_updates` signature + the build loop at lines ~262-289) +- Modify: `tests/testthat/test-process_cran_updates.R` (add routing tests) + +- [ ] **Step 1: Write the failing tests** + +```r +# append to tests/testthat/test-process_cran_updates.R +test_that("classifier mode passes per-package is_r_minor_sensitive to build", { + skip_if_not_installed("mockery") + recorded <- list() + mockery::stub(process_cran_updates, "get_updated_cran_packages", + function(...) data.frame(name = c("riskypkg", "purepkg"), + version = c("1.0", "2.0"), stringsAsFactors = FALSE)) + mockery::stub(process_cran_updates, "get_new_cran_packages", + function(...) data.frame(name = character(), version = character())) + mockery::stub(process_cran_updates, "tools::CRAN_package_db", + function(...) data.frame(Package = character(), OS_type = character())) + mockery::stub(process_cran_updates, "classify_r_minor_sensitive", + function(name, tag, ...) name == "riskypkg") + mockery::stub(process_cran_updates, "build_binary_package", + function(name, tag, ..., is_r_minor_sensitive) { + recorded[[name]] <<- is_r_minor_sensitive + invisible(TRUE) + }) + + process_cran_updates( + platform = "alpine-323", process_removed = FALSE, + r_minor_detection = "classifier", + s3_endpoint = "x", s3_region = "x", s3_bucket = "x" + ) + + expect_true(recorded[["riskypkg"]]) + expect_false(recorded[["purepkg"]]) +}) + +test_that("r_minor_sensitive_only drops non-risky candidates", { + skip_if_not_installed("mockery") + built <- character() + mockery::stub(process_cran_updates, "get_updated_cran_packages", + function(...) data.frame(name = c("riskypkg", "purepkg"), + version = c("1.0", "2.0"), stringsAsFactors = FALSE)) + mockery::stub(process_cran_updates, "get_new_cran_packages", + function(...) data.frame(name = character(), version = character())) + mockery::stub(process_cran_updates, "tools::CRAN_package_db", + function(...) data.frame(Package = character(), OS_type = character())) + mockery::stub(process_cran_updates, "classify_r_minor_sensitive", + function(name, tag, ...) name == "riskypkg") + mockery::stub(process_cran_updates, "build_binary_package", + function(name, tag, ..., is_r_minor_sensitive) { built <<- c(built, name); invisible(TRUE) }) + + process_cran_updates( + platform = "alpine-323", process_removed = FALSE, + r_minor_detection = "classifier", r_minor_sensitive_only = TRUE, + s3_endpoint = "x", s3_region = "x", s3_bucket = "x" + ) + + expect_identical(built, "riskypkg") +}) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` +Expected: FAIL — `unused argument (r_minor_detection = ...)`. + +- [ ] **Step 3: Add the parameters** + +In the `process_cran_updates()` signature, after `filter_r_minor_sensitive = FALSE,` add: + +```r + r_minor_detection = c("none", "issue", "classifier"), + r_minor_sensitive_only = FALSE, +``` + +Immediately after the three `stop()` validation lines at the top of the body, add: + +```r + r_minor_detection <- match.arg(r_minor_detection) + # back-compat: the old boolean maps onto the issue-list path + if (isTRUE(filter_r_minor_sensitive) && r_minor_detection == "none") { + r_minor_detection <- "issue" + } +``` + +- [ ] **Step 4: Replace the issue-filter gate** + +The existing block reads `if (filter_r_minor_sensitive) { all_pkgs <- get_r_minor_sensitive_packages(...) }`. Change its condition to: + +```r + if (r_minor_detection == "issue") { + all_pkgs <- get_r_minor_sensitive_packages( + r_minor_packages_forge_type, + r_minor_packages_issue_url, + interval, + updated_packages = updated_pkgs, + new_packages = new_pkgs + ) + } +``` + +Also change the two later `if (filter_r_minor_sensitive)` log branches to `if (r_minor_detection != "none")`. + +- [ ] **Step 5: Replace the build loop** + +Replace the `purrr::walk2(all_pkgs$name, all_pkgs$version, ~ { build_binary_package(... is_r_minor_sensitive = filter_r_minor_sensitive ...) })` block with: + +```r + if (nrow(all_pkgs) > 0L) { + sensitive <- switch( + r_minor_detection, + classifier = vapply( + seq_len(nrow(all_pkgs)), + function(i) { + classify_r_minor_sensitive( + all_pkgs$name[i], + all_pkgs$version[i], + local_clone_dir = local_clone_dir + ) + }, + logical(1L) + ), + issue = rep(TRUE, nrow(all_pkgs)), + none = rep(FALSE, nrow(all_pkgs)) + ) + + if (isTRUE(r_minor_sensitive_only)) { + all_pkgs <- all_pkgs[sensitive, , drop = FALSE] + sensitive <- sensitive[sensitive] + } + + purrr::pwalk( + list(all_pkgs$name, all_pkgs$version, sensitive), + function(.name, .version, .sensitive) { + build_binary_package( + .name, + .version, + platform = platform, + upload = upload, + archive = archive, + force = force, + store_build_metadata = store_build_metadata, + s3_endpoint = s3_endpoint, + s3_bucket = s3_bucket, + s3_region = s3_region, + s3_access_key_id = s3_access_key_id, + s3_secret_access_key = s3_secret_access_key, + is_r_minor_sensitive = .sensitive, + metadata_db_type = metadata_db_type, + metadata_db_host = metadata_db_host, + metadata_db_name = metadata_db_name, + metadata_db_table = metadata_db_table, + metadata_db_port = metadata_db_port, + metadata_db_user = metadata_db_user, + metadata_db_password = metadata_db_password, + metadata_db_sslmode = metadata_db_sslmode + ) + } + ) + } else { + log_info("No packages to process after filtering Windows-only packages") + } +``` + +- [ ] **Step 6: Run tests to verify they pass** + +Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` +Expected: PASS (5/5 — 3 from Task 3 + 2 new). + +- [ ] **Step 7: Commit** + +```bash +git add R/process_cran_updates.R tests/testthat/test-process_cran_updates.R +git commit -m "feat(updates): classifier-driven per-package r-minor routing + sensitive-only pass" +``` + +--- + +### Task 5: docs, roxygen params, version bump, NEWS + +**Files:** +- Create: `man-roxygen/param-r_minor_detection.R`, `man-roxygen/param-r_minor_sensitive_only.R` +- Modify: `R/process_cran_updates.R` (roxygen `@template` lines), `DESCRIPTION`, `NEWS.md` + +- [ ] **Step 1: Add the man templates** + +```r +# man-roxygen/param-r_minor_detection.R +#' @param r_minor_detection How to decide which packages are R-minor-sensitive. +#' `"none"` (default) builds everything into the generic slot. `"issue"` uses +#' the curated tracking issue (the legacy `filter_r_minor_sensitive` path). +#' `"classifier"` classifies each candidate via [needs_per_minor_recompile()] +#' and routes only `risky` packages to the per-minor slot. +``` + +```r +# man-roxygen/param-r_minor_sensitive_only.R +#' @param r_minor_sensitive_only When `TRUE`, only R-minor-sensitive packages are +#' built (used for the additional per-minor passes under non-primary R versions). +``` + +- [ ] **Step 2: Reference the templates** + +In the roxygen block above `process_cran_updates`, add: + +```r +#' @template param-r_minor_detection +#' @template param-r_minor_sensitive_only +``` + +- [ ] **Step 3: Bump version and NEWS** + +In `DESCRIPTION` set `Version: 4.2.0`. Add to the top of `NEWS.md`: + +```markdown +# bincraft 4.2.0 + +* `process_cran_updates()` gains `r_minor_detection` (`"none"`/`"issue"`/`"classifier"`) + and `r_minor_sensitive_only`, classifying each candidate via the ABI classifier and + routing only `risky` packages to per-minor slots. +* `upload_package_index()` / `add_to_package_index()` gain an `r_minor` argument to + write/serve a per-minor `PACKAGES*` index under `…/contrib//`. +``` + +- [ ] **Step 4: Regenerate docs and run full check** + +Run: `R -q -e 'devtools::document()'` +Run: `R -q -e 'devtools::test()'` +Expected: docs regenerate clean; all tests PASS. + +- [ ] **Step 5: Commit, push, open PR** + +```bash +git add DESCRIPTION NEWS.md man-roxygen/ man/ R/process_cran_updates.R +git commit -m "docs(release): bincraft 4.2.0 — per-minor index + classifier-driven updates" +git push -u origin +fj -H codefloe.com pr create --base main --head --body "" "feat: per-minor index + classifier-driven r-minor builds (4.2.0)" +``` + +--- + +## Self-Review + +- **Spec coverage:** §"Required bincraft enhancements" item 1 (process_cran_updates) → Tasks 3-5; item 2 (upload_package_index per-minor) → Tasks 1-2. `build_binary_package` unchanged per the spec — no task, correct. +- **Type consistency:** helper named `package_index_remote_dir` (Tasks 1-2), `classify_r_minor_sensitive` (Tasks 3-4) consistently; params `r_minor_detection`/`r_minor_sensitive_only`/`r_minor` consistent across signature, tests, and docs. +- **Placeholders:** none — every code/edit step shows the actual code; ``/`` in the final push step are deliberate operator inputs. +- **Release dependency:** the pipeline plan pins bincraft `4.2.0`, matching the bump here. diff --git a/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md b/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md new file mode 100644 index 0000000..6123278 --- /dev/null +++ b/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md @@ -0,0 +1,471 @@ +# Pipeline R-minor Builds Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the full (`build-all-versions-*`) and iterative (`process-updates-*`) pipelines build R-minor-sensitive packages once per installed R minor, and everything else once into the generic slot. + +**Architecture:** A precompute step classifies packages (cheap CRAN-metadata rules + source download only for the ambiguous compiled subset) and emits a sensitivity flag. Build scripts pass a concrete `is_r_minor_sensitive` per package on the primary-R pass and re-run a sensitive-only pass under each additional `/opt/R/*` minor. Per-minor `PACKAGES` indexes are uploaded for each touched slot. + +**Tech Stack:** R scripts under `local/`, crow/woodpecker CI YAML under `.crow/`, bincraft `>= 4.2.0` (see `2026-06-13-bincraft-r-minor-enhancements.md`). + +**HARD DEPENDENCY:** bincraft `4.2.0` (the companion plan) must be merged and released first — Tasks 4-6 call `r_minor_detection`/`upload_package_index(r_minor=)` which only exist there. Tasks 1-3 (this repo's R scripts) can be written and unit-tested before the release. + +--- + +### Task 1: pure metadata classifier helper + +**Files:** +- Create: `local/r-minor-helpers.R` +- Test: `local/tests/test-r-minor-helpers.R` + +- [ ] **Step 1: Write the failing test** + +```r +# local/tests/test-r-minor-helpers.R +source(file.path("local", "r-minor-helpers.R")) + +test_that("pure-r (NeedsCompilation != yes) is not sensitive", { + expect_identical(classify_from_metadata("no", NA, c("Rcpp")), "not-sensitive") + expect_identical(classify_from_metadata("", "Rcpp", c("Rcpp")), "not-sensitive") +}) + +test_that("LinkingTo a risky dep is sensitive (version constraints stripped)", { + expect_identical(classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), "sensitive") + expect_identical(classify_from_metadata("yes", "R6,\n cpp11", c("Rcpp", "cpp11")), "sensitive") +}) + +test_that("compiled but no risky LinkingTo is ambiguous (needs source)", { + expect_identical(classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), "ambiguous") + expect_identical(classify_from_metadata("yes", NA, c("Rcpp")), "ambiguous") +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'` +Expected: FAIL — `cannot open file 'local/r-minor-helpers.R'` / function not found. + +- [ ] **Step 3: Write the helper** + +```r +# local/r-minor-helpers.R +# Metadata-only ABI triage so the full build avoids downloading every source. +# Mirrors bincraft::abi_classify rules 1-2; "ambiguous" packages still need a +# source grep via bincraft::needs_per_minor_recompile(). + +classify_from_metadata <- function(needs_compilation, linking_to, risky_deps) { + nc <- if (length(needs_compilation) == 0L || is.na(needs_compilation)) { + "" + } else { + tolower(trimws(needs_compilation)) + } + if (!identical(nc, "yes")) { + return("not-sensitive") + } + lt <- if (length(linking_to) == 0L || is.na(linking_to)) "" else linking_to + linked <- trimws(unlist(strsplit(lt, "[,\n]"))) + linked <- sub("\\s*\\(.*\\)$", "", linked) # strip "(>= x)" constraints + linked <- linked[nzchar(linked)] + if (any(linked %in% risky_deps)) { + return("sensitive") + } + "ambiguous" +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'` +Expected: PASS (3/3). + +- [ ] **Step 5: Commit** + +```bash +git add local/r-minor-helpers.R local/tests/test-r-minor-helpers.R +git commit -m "feat(local): metadata-only ABI triage helper" +``` + +--- + +### Task 2: add `r_minor_sensitive` to `packages-to-build.R` + +**Files:** +- Modify: `local/packages-to-build.R` (after `pkgs` is finalized, ~line 142) + +- [ ] **Step 1: Append the classification block** + +At the end of `local/packages-to-build.R` (after `setorder(pkgs, Package, Version)`), add: + +```r +### R-minor sensitivity (classify once per package, applied to all versions) +source(file.path("local", "r-minor-helpers.R")) +risky_deps <- bincraft::abi_risky_linking_deps() + +release_meta <- data.table( + Package = cran_release$Package, + NeedsCompilation = cran_release$NeedsCompilation, + LinkingTo = cran_release$LinkingTo +) + +meta <- release_meta[Package %in% unique(pkgs$Package)] +meta[, triage := mapply( + classify_from_metadata, + NeedsCompilation, + LinkingTo, + MoreArgs = list(risky_deps = risky_deps) +)] + +# Only the "ambiguous" compiled packages need a source grep. +ambiguous <- meta[triage == "ambiguous", Package] +sensitive_ambiguous <- character() +if (length(ambiguous) > 0L) { + tmp_src <- file.path(tempdir(), "abi_src") + dir.create(tmp_src, showWarnings = FALSE, recursive = TRUE) + sens <- vapply(ambiguous, function(pkg) { + out <- tryCatch({ + dl <- utils::download.packages( + pkg, destdir = tmp_src, + repos = "https://cloud.r-project.org", quiet = TRUE + ) + isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L]))) + }, error = function(e) TRUE) # fail safe: treat as sensitive + out + }, logical(1L)) + sensitive_ambiguous <- ambiguous[sens] +} + +sensitive_pkgs <- unique(c(meta[triage == "sensitive", Package], sensitive_ambiguous)) +pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs] +sprintf("R-minor-sensitive packages: %s of %s", length(sensitive_pkgs), uniqueN(pkgs$Package)) +``` + +- [ ] **Step 2: Verify the script parses and the column is added (offline smoke)** + +Run: +```bash +R -q -e ' + library(data.table) + source("local/r-minor-helpers.R") + pkgs <- data.table(Package = c("A","B"), Version = c("1","1")) + cran_release <- data.frame(Package = c("A","B"), + NeedsCompilation = c("no","yes"), LinkingTo = c(NA,"Rcpp"), + stringsAsFactors = FALSE) + risky_deps <- c("Rcpp") + release_meta <- as.data.table(cran_release) + meta <- release_meta[Package %in% unique(pkgs$Package)] + meta[, triage := mapply(classify_from_metadata, NeedsCompilation, LinkingTo, + MoreArgs = list(risky_deps = risky_deps))] + sensitive_pkgs <- meta[triage == "sensitive", Package] + pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs] + stopifnot(identical(pkgs$r_minor_sensitive, c(FALSE, TRUE))) + cat("OK\n")' +``` +Expected: prints `OK` (B classified sensitive via LinkingTo, A not). + +- [ ] **Step 3: Persist the sensitive subset in install-deps** + +In `.crow/build-all-versions-install-deps-amd64.yaml` and `...-arm64.yaml`, change the precompute command (line ~38) from saving only `pkgs_to_build.rds` to also saving the subset: + +```yaml + - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))" +``` + +- [ ] **Step 4: Commit** + +```bash +git add local/packages-to-build.R .crow/build-all-versions-install-deps-amd64.yaml .crow/build-all-versions-install-deps-arm64.yaml +git commit -m "feat(full): classify r-minor sensitivity in install-deps precompute" +``` + +--- + +### Task 3: per-package flag + `--sensitive-only` in `build-all.R` + +**Files:** +- Modify: `local/build-all.R` +- Test: `local/tests/test-build-all-args.R` (new — covers the arg/flag parsing seam only) + +- [ ] **Step 1: Write the failing test for the arg parser** + +```r +# local/tests/test-build-all-args.R +source(file.path("local", "r-minor-helpers.R")) + +test_that("parse_build_args splits flags from positionals", { + a <- parse_build_args(c("--sensitive-only", "4", "2", "8")) + expect_true(a$sensitive_only) + expect_identical(a$split_into, 4L) + expect_identical(a$split_index, 2L) + expect_identical(a$ncpus, 8L) + + b <- parse_build_args(c("4", "2", "8")) + expect_false(b$sensitive_only) + expect_identical(b$split_into, 4L) +}) +``` + +- [ ] **Step 2: Add `parse_build_args` to the helpers and run the test** + +Append to `local/r-minor-helpers.R`: + +```r +parse_build_args <- function(args) { + sensitive_only <- "--sensitive-only" %in% args + pos <- args[!startsWith(args, "--")] + list( + sensitive_only = sensitive_only, + split_into = as.integer(pos[1L]), + split_index = as.integer(pos[2L]), + ncpus = as.integer(pos[3L]) + ) +} +``` + +Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-build-all-args.R")'` +Expected: PASS (1/1, 5 expectations). + +- [ ] **Step 3: Rewire `build-all.R` to use the parser, the flag, and the per-row sensitivity** + +Replace the arg-parsing header and the `mapply` build loop of `local/build-all.R`. New top: + +```r +sink(stdout(), type = "message") +options(crayon.enabled = TRUE, future.globals.onReference = NULL) +source(file.path("local", "r-minor-helpers.R")) + +args <- commandArgs(trailingOnly = TRUE) +parsed <- parse_build_args(args) +split_into <- parsed$split_into +split_index <- parsed$split_index +ncpus <- parsed$ncpus +sensitive_only <- parsed$sensitive_only +options(Ncpus = ncpus) + +library(bincraft, quietly = TRUE) +library(future) +plan("sequential") + +pkgs <- if (sensitive_only) { + readRDS("/mnt/cache/packages/r_minor_sensitive_pkgs.rds") +} else { + readRDS("/mnt/cache/packages/pkgs_to_build.rds") +} +# Back-compat: tolerate an older RDS without the column (treat all as non-sensitive) +if (is.null(pkgs$r_minor_sensitive)) pkgs$r_minor_sensitive <- FALSE +sprintf("Total# of remaining package versions: %s (sensitive_only=%s)", nrow(pkgs), sensitive_only) +``` + +Keep the existing chunk-split, exclude-list, and `s3_cache` lines unchanged. Then replace the `mapply(...)` call with: + +```r +n <- nrow(chunk) +mapply(function(pkg, ver, sens, i) { + cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens)) + bincraft::build_binary_package( + pkg, + tag = ver, + is_r_minor_sensitive = isTRUE(sens), + s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", + s3_region = "eu-central-003", + s3_bucket = "devxy-rpkgs-binaries", + s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), + s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), + s3_package_cache = s3_cache, + metadata_db_host = "r-binaries.devxy.io", + metadata_db_name = "build_metadata", + metadata_db_table = "single_builds", + metadata_db_user = "rpkgs", + metadata_db_password = Sys.getenv("PGPASS"), + metadata_db_sslmode = "require", + metadata_db_port = 15432, + archive = TRUE, + upload = TRUE, + store_build_metadata = TRUE + ) +}, chunk$Package, chunk$Version, chunk$r_minor_sensitive, seq_len(n)) +``` + +Note: in `--sensitive-only` mode every row already has `r_minor_sensitive == TRUE`, so the same loop builds them into per-minor slots; no special-casing needed. + +- [ ] **Step 4: Verify the script parses** + +Run: `R -q -e 'invisible(parse(file="local/build-all.R")); cat("parsed OK\n")'` +Expected: `parsed OK`. + +- [ ] **Step 5: Commit** + +```bash +git add local/build-all.R local/r-minor-helpers.R local/tests/test-build-all-args.R +git commit -m "feat(full): per-package r-minor flag and --sensitive-only mode in build-all.R" +``` + +--- + +### Task 4: multi-R loop + per-minor index in `build-all-versions-*.yaml` + +> Depends on bincraft 4.2.0 (`upload_package_index(r_minor=)`). + +**Files:** +- Modify: `.crow/build-all-versions-amd64.yaml`, `.crow/build-all-versions-arm64.yaml` + +- [ ] **Step 1: Add the sensitive-only multi-R pass after the primary build** + +In each file's `commands:` (after the existing `Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS` line and before the `process_unarchived_pkgs` line), insert this multiline command: + +```yaml + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive pass under R $RV ===" + R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$RBIN" Rscript local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true + done +``` + +(`$XVFB`/`$XVFB_ARGS` are already defined on the preceding line in this workflow.) Note the primary pass still runs `build-all.R` without `--sensitive-only`, building everything and routing sensitive packages into the primary minor slot. + +- [ ] **Step 2: Add a per-minor index upload step** + +Append a new step after the build step (mirroring the existing CDN/index pattern). Add to `commands` of a new `Upload per-minor indexes` step (or extend the existing index handling) the following, which uploads the generic index plus one per discovered minor: + +```yaml + - | + CODENAME=$(/opt/R/$R_VERSION/bin/Rscript -e "cat(bincraft::set_codename(NULL))") + # generic slot + /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done +``` + +- [ ] **Step 3: Validate YAML** + +Run: `R -q -e 'invisible(lapply(c(".crow/build-all-versions-amd64.yaml",".crow/build-all-versions-arm64.yaml"), yaml::yaml.load_file)); cat("yaml OK\n")'` +Expected: `yaml OK`. + +- [ ] **Step 4: Commit** + +```bash +git add .crow/build-all-versions-amd64.yaml .crow/build-all-versions-arm64.yaml +git commit -m "feat(full): sensitive-only multi-R passes and per-minor index upload" +``` + +--- + +### Task 5: classifier + multi-R in all `process-updates-*.yaml` + +> Depends on bincraft 4.2.0 (`r_minor_detection`, `upload_package_index(r_minor=)`). + +**Files (14):** every `.crow/process-updates--.yaml`: +`alpine-322`, `alpine-323`, `redhat-8`, `redhat-9`, `redhat-10`, `ubuntu-2204`, `ubuntu-2404` × `amd64`, `arm64`. + +The transform is identical in shape; only the already-present `platform=`, `R_VERSION`, `ARCH`, and the index `codename=` differ per file (leave those as-is). + +- [ ] **Step 1: Edit the `process_cran_updates` call (primary pass)** + +In each file, inside the single long `bincraft::process_cran_updates(...)` argument list, add `r_minor_detection = 'classifier', ` immediately before `s3_endpoint = ...`. Leave all other args unchanged. + +- [ ] **Step 2: Add the sensitive-only multi-R pass** + +Immediately after the `process_cran_updates(...)` command line, insert (preserve the per-file `''` string and `$INTERVAL`): + +```yaml + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$RBIN" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done +``` + +Replace `` with that file's existing platform string (e.g. `alpine-323`, `redhat-9`, `ubuntu-2404`). `process_removed = FALSE` on the extra passes (removals are handled once by the primary pass). + +- [ ] **Step 3: Add per-minor index upload** + +After the existing `upload_package_index(codename = "", ...)` line in each file, append (reusing the file's existing `codename`): + +```yaml + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "", r_minor = "'"$RMINOR"'", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' || true + done +``` + +Replace `` with the file's existing codename value (e.g. `alpine323`, `rhel9`, `ubuntu2404`). + +- [ ] **Step 4: Verify every file changed and parses** + +Run: +```bash +R -q -e 'fs <- Sys.glob(".crow/process-updates-*.yaml"); invisible(lapply(fs, yaml::yaml.load_file)); cat(length(fs), "files parse OK\n")' +grep -L "r_minor_detection = 'classifier'" .crow/process-updates-*.yaml +``` +Expected: `14 files parse OK`, and the `grep -L` prints **nothing** (every file contains the classifier arg). + +- [ ] **Step 5: Commit** + +```bash +git add .crow/process-updates-*.yaml +git commit -m "feat(updates): classifier-driven r-minor builds + per-minor index across platforms" +``` + +--- + +### Task 6: pin bincraft 4.2.0, remove the standalone workflow + +**Files:** +- Modify: all workflows pinning bincraft (`grep -rln "bincraft.git@v" .crow/`), and `.crow/build-all-versions-install-deps-*.yaml` (which installs from `main`) +- Delete: `.crow/build-r-minor-sensitive-packages.yaml` + +- [ ] **Step 1: Bump the pin** + +Run to find pins: `grep -rn "bincraft" .crow/ | grep -E "@v4\.1\.1|bincraft.git"`. In every `process-updates-*` file, change `bincraft.git@v4.1.1` (and the `packageVersion("bincraft") != "4.1.1"` guard) to `4.2.0`. + +- [ ] **Step 2: Remove the superseded workflow** + +```bash +git rm .crow/build-r-minor-sensitive-packages.yaml +``` + +- [ ] **Step 3: Verify no stale references** + +Run: `grep -rn "build-r-minor-sensitive\|@v4.1.1\|!= \"4.1.1\"" .crow/` +Expected: no matches. + +- [ ] **Step 4: Commit** + +```bash +git add .crow/ +git commit -m "chore: pin bincraft 4.2.0 and drop superseded standalone r-minor workflow" +``` + +--- + +### Task 7: integration smoke test (manual, gated on bincraft 4.2.0 release) + +- [ ] **Step 1: Run one iterative platform manually** via crow against a short interval and confirm: a known risky package (e.g. one LinkingTo Rcpp) lands under `…/contrib//` for each installed minor, and a pure-r package lands only in the generic slot. + +- [ ] **Step 2: Confirm a client install resolves the per-minor slot.** From an R `4.4` and an R `4.5` container: +```r +install.packages("", repos = "https://cran.devxy.io/") +library() # must load without "undefined symbol" +``` +Expected: loads under both minors. If the per-minor `PACKAGES` is missing, revisit bincraft Task 2. + +- [ ] **Step 3: Spot-check the full build** on one platform with a small `SPLIT_INTO`, verifying the sensitive-only extra passes ran and produced per-minor artifacts. + +--- + +## Self-Review + +- **Spec coverage:** install-deps precompute → Tasks 1-2; `build-all.R` per-row flag + `--sensitive-only` → Task 3; full-build multi-R loop + per-minor index → Task 4; iterative classifier + multi-R + per-minor index → Task 5; remove standalone workflow + version pin → Task 6; per-minor index client-serviceability risk → Task 7 verification. +- **Type/name consistency:** `classify_from_metadata` and `parse_build_args` live in `local/r-minor-helpers.R` and are used in Tasks 2-3; the RDS column is `r_minor_sensitive` everywhere; the subset file is `/mnt/cache/packages/r_minor_sensitive_pkgs.rds` in install-deps (Task 2) and `build-all.R` (Task 3). +- **Placeholders:** ``/`` in Task 5 are explicit per-file substitutions (the values already exist in each file), not unfilled blanks. No "TBD"/"handle errors" placeholders. +- **Dependency ordering:** Tasks 1-3 are pure R-script work, unit-testable now; Tasks 4-6 are gated on bincraft 4.2.0; Task 7 is post-release verification. diff --git a/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md b/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md index 81f828f..248a1a7 100644 --- a/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md +++ b/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md @@ -103,22 +103,29 @@ Driven by the bincraft enhancement below; the single build step becomes: ## Required bincraft enhancements (separate PR, coordinated release) -1. `build_binary_package()`: - - Accept `is_r_minor_sensitive = "auto"` — classify the source it already clones via - `abi_classify()` and route the artifact to the per-minor slot iff `risky`. - - Add `r_minor_sensitive_only` — when `TRUE`, skip (return `"skipped"`) non-risky - packages early, after classification, before building. -2. `process_cran_updates()`: +`build_binary_package()` needs **no change** — it already accepts a concrete +`is_r_minor_sensitive` logical and routes the slot accordingly. Classification stays at +the orchestration layer (mirroring the full-build precompute), which keeps +`build_binary_package`'s pre-build S3 skip-check and source clone untouched. + +1. `process_cran_updates()`: - Add `r_minor_detection = c("none", "issue", "classifier")` (default `"none"` to preserve current behavior; `"issue"` is today's `filter_r_minor_sensitive` path). - - Add `r_minor_sensitive_only`, threaded down to `build_binary_package()`. - - With `"classifier"`, pass `is_r_minor_sensitive = "auto"` per package instead of a - single run-wide bool. -3. `upload_package_index()`: - - Add per-minor slot support: write/upload `PACKAGES*` (and `Meta/archive.rds`) under - `…/contrib//`, mirroring the generic-slot logic. Invocable per minor. + - Add `r_minor_sensitive_only` (default `FALSE`). + - With `"classifier"`: for each candidate `(name, version)`, clone the source to a temp + dir and call `bincraft::needs_per_minor_recompile()`; pass the resulting concrete + logical as `is_r_minor_sensitive` to `build_binary_package()`. When + `r_minor_sensitive_only = TRUE`, drop non-risky candidates before building. + - Implemented via a small internal helper `classify_r_minor_sensitive(name, tag, + source_org_url, local_clone_dir)` returning a logical — the unit-testable seam. +2. `upload_package_index()`: + - Add an `r_minor = NULL` argument. When non-NULL (e.g. `"4.4"`), point the remote dir + at `…/contrib//` and write/upload `PACKAGES*` (and `Meta/archive.rds`) there, + mirroring the generic-slot logic. Extract the remote-dir construction into a pure + helper `package_index_remote_dir(s3_bucket, arch, codename, r_minor = NULL)` — the + unit-testable seam. -These three are the only bincraft changes; detection itself (PR #49) is already merged. +These two are the only bincraft changes; detection itself (PR #49) is already merged. ## Scope and cleanup -- 2.54.0 From dbf26d7786515451ba8af4cff24640587ea4e448 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 20:46:49 +0200 Subject: [PATCH 03/14] feat(local): metadata-only ABI triage helper --- local/r-minor-helpers.R | 22 ++++++++++++++++++++++ local/tests/test-r-minor-helpers.R | 16 ++++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 local/r-minor-helpers.R create mode 100644 local/tests/test-r-minor-helpers.R diff --git a/local/r-minor-helpers.R b/local/r-minor-helpers.R new file mode 100644 index 0000000..ffe56fb --- /dev/null +++ b/local/r-minor-helpers.R @@ -0,0 +1,22 @@ +# Metadata-only ABI triage so the full build avoids downloading every source. +# Mirrors bincraft::abi_classify rules 1-2; "ambiguous" packages still need a +# source grep via bincraft::needs_per_minor_recompile(). + +classify_from_metadata <- function(needs_compilation, linking_to, risky_deps) { + nc <- if (length(needs_compilation) == 0L || is.na(needs_compilation)) { + "" + } else { + tolower(trimws(needs_compilation)) + } + if (!identical(nc, "yes")) { + return("not-sensitive") + } + lt <- if (length(linking_to) == 0L || is.na(linking_to)) "" else linking_to + linked <- trimws(unlist(strsplit(lt, "[,\n]"))) + linked <- sub("\\s*\\(.*\\)$", "", linked) # strip "(>= x)" constraints + linked <- linked[nzchar(linked)] + if (any(linked %in% risky_deps)) { + return("sensitive") + } + "ambiguous" +} diff --git a/local/tests/test-r-minor-helpers.R b/local/tests/test-r-minor-helpers.R new file mode 100644 index 0000000..d68e99b --- /dev/null +++ b/local/tests/test-r-minor-helpers.R @@ -0,0 +1,16 @@ +source(file.path("..", "r-minor-helpers.R")) + +test_that("pure-r (NeedsCompilation != yes) is not sensitive", { + expect_identical(classify_from_metadata("no", NA, c("Rcpp")), "not-sensitive") + expect_identical(classify_from_metadata("", "Rcpp", c("Rcpp")), "not-sensitive") +}) + +test_that("LinkingTo a risky dep is sensitive (version constraints stripped)", { + expect_identical(classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), "sensitive") + expect_identical(classify_from_metadata("yes", "R6,\n cpp11", c("Rcpp", "cpp11")), "sensitive") +}) + +test_that("compiled but no risky LinkingTo is ambiguous (needs source)", { + expect_identical(classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), "ambiguous") + expect_identical(classify_from_metadata("yes", NA, c("Rcpp")), "ambiguous") +}) -- 2.54.0 From 34173a6722fcd7fbbb51200de43371079fc387a0 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 20:48:54 +0200 Subject: [PATCH 04/14] feat(full): classify r-minor sensitivity in install-deps precompute --- ...build-all-versions-install-deps-amd64.yaml | 2 +- ...build-all-versions-install-deps-arm64.yaml | 2 +- local/packages-to-build.R | 41 +++++++++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/.crow/build-all-versions-install-deps-amd64.yaml b/.crow/build-all-versions-install-deps-amd64.yaml index 4d50ddf..b1f33e4 100644 --- a/.crow/build-all-versions-install-deps-amd64.yaml +++ b/.crow/build-all-versions-install-deps-amd64.yaml @@ -35,7 +35,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - git clone -q https://codefloe.com/rpkgs/bincraft.git /tmp/bincraft - /opt/R/$R_VERSION/bin/R -q -e 'pak::sysreqs_db_update(); pak::local_install("/tmp/bincraft"); pak::pak(c("RPostgres", "s3fs", "data.table", "future", "jsonlite")); packageVersion("bincraft")' - - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); sprintf('Precomputed %s package versions to build', nrow(pkgs))" + - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))" backend_options: docker: resources: diff --git a/.crow/build-all-versions-install-deps-arm64.yaml b/.crow/build-all-versions-install-deps-arm64.yaml index 2b3b881..0ef90e3 100644 --- a/.crow/build-all-versions-install-deps-arm64.yaml +++ b/.crow/build-all-versions-install-deps-arm64.yaml @@ -34,7 +34,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - git clone -q https://codefloe.com/rpkgs/bincraft.git /tmp/bincraft - /opt/R/$R_VERSION/bin/R -q -e 'pak::sysreqs_db_update(); pak::local_install("/tmp/bincraft"); pak::pak(c("RPostgres", "s3fs", "data.table", "future", "jsonlite")); packageVersion("bincraft")' - - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); sprintf('Precomputed %s package versions to build', nrow(pkgs))" + - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))" backend_options: docker: resources: diff --git a/local/packages-to-build.R b/local/packages-to-build.R index 91ee96e..3b23276 100644 --- a/local/packages-to-build.R +++ b/local/packages-to-build.R @@ -140,3 +140,44 @@ pkgs <- pkgs_no_error[!s3_dt] # Deduplicate pkgs <- unique(pkgs) setorder(pkgs, Package, Version) + +### R-minor sensitivity (classify once per package, applied to all versions) +source(file.path("local", "r-minor-helpers.R")) +risky_deps <- bincraft::abi_risky_linking_deps() + +release_meta <- data.table( + Package = cran_release$Package, + NeedsCompilation = cran_release$NeedsCompilation, + LinkingTo = cran_release$LinkingTo +) + +meta <- release_meta[Package %in% unique(pkgs$Package)] +meta[, triage := mapply( + classify_from_metadata, + NeedsCompilation, + LinkingTo, + MoreArgs = list(risky_deps = risky_deps) +)] + +# Only the "ambiguous" compiled packages need a source grep. +ambiguous <- meta[triage == "ambiguous", Package] +sensitive_ambiguous <- character() +if (length(ambiguous) > 0L) { + tmp_src <- file.path(tempdir(), "abi_src") + dir.create(tmp_src, showWarnings = FALSE, recursive = TRUE) + sens <- vapply(ambiguous, function(pkg) { + out <- tryCatch({ + dl <- utils::download.packages( + pkg, destdir = tmp_src, + repos = "https://cloud.r-project.org", quiet = TRUE + ) + isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L]))) + }, error = function(e) TRUE) # fail safe: treat as sensitive + out + }, logical(1L)) + sensitive_ambiguous <- ambiguous[sens] +} + +sensitive_pkgs <- unique(c(meta[triage == "sensitive", Package], sensitive_ambiguous)) +pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs] +sprintf("R-minor-sensitive packages: %s of %s", length(sensitive_pkgs), uniqueN(pkgs$Package)) -- 2.54.0 From a33986a8d668d67e52ff27d46de048b3db190f25 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 20:51:42 +0200 Subject: [PATCH 05/14] feat(full): per-package r-minor flag and --sensitive-only mode in build-all.R --- local/build-all.R | 27 ++++++++++++++++++--------- local/r-minor-helpers.R | 11 +++++++++++ local/tests/test-build-all-args.R | 13 +++++++++++++ 3 files changed, 42 insertions(+), 9 deletions(-) create mode 100644 local/tests/test-build-all-args.R diff --git a/local/build-all.R b/local/build-all.R index 102d18c..8c181f3 100644 --- a/local/build-all.R +++ b/local/build-all.R @@ -1,10 +1,13 @@ sink(stdout(), type = "message") options(crayon.enabled = TRUE, future.globals.onReference = NULL) +source(file.path("local", "r-minor-helpers.R")) args <- commandArgs(trailingOnly = TRUE) -split_into <- as.integer(args[1]) -split_index <- as.integer(args[2]) -ncpus <- as.integer(args[3]) +parsed <- parse_build_args(args) +split_into <- parsed$split_into +split_index <- parsed$split_index +ncpus <- parsed$ncpus +sensitive_only <- parsed$sensitive_only options(Ncpus = ncpus) # Load bincraft eagerly to avoid lazy-load memory spike during first build call @@ -12,9 +15,14 @@ library(bincraft, quietly = TRUE) library(future) plan("sequential") -# Read precomputed package+version pairs -pkgs <- readRDS("/mnt/cache/packages/pkgs_to_build.rds") -sprintf("Total# of remaining package versions: %s", nrow(pkgs)) +pkgs <- if (sensitive_only) { + readRDS("/mnt/cache/packages/r_minor_sensitive_pkgs.rds") +} else { + readRDS("/mnt/cache/packages/pkgs_to_build.rds") +} +# Back-compat: tolerate an older RDS without the column (treat all as non-sensitive) +if (is.null(pkgs$r_minor_sensitive)) pkgs$r_minor_sensitive <- FALSE +sprintf("Total# of remaining package versions: %s (sensitive_only=%s)", nrow(pkgs), sensitive_only) # Split into chunks for this worker chunks <- split(pkgs, cut(seq_len(nrow(pkgs)), split_into, labels = FALSE)) @@ -32,11 +40,12 @@ s3_cache <- readRDS("/mnt/cache/packages/s3_cache.rds") sprintf("S3 cache: %s files", length(s3_cache)) n <- nrow(chunk) -mapply(function(pkg, ver, i) { - cat(sprintf("[%d/%d] %s_%s\n", i, n, pkg, ver)) +mapply(function(pkg, ver, sens, i) { + cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens)) bincraft::build_binary_package( pkg, tag = ver, + is_r_minor_sensitive = isTRUE(sens), s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", @@ -54,4 +63,4 @@ mapply(function(pkg, ver, i) { upload = TRUE, store_build_metadata = TRUE ) -}, chunk$Package, chunk$Version, seq_len(n)) +}, chunk$Package, chunk$Version, chunk$r_minor_sensitive, seq_len(n)) diff --git a/local/r-minor-helpers.R b/local/r-minor-helpers.R index ffe56fb..65aafdf 100644 --- a/local/r-minor-helpers.R +++ b/local/r-minor-helpers.R @@ -20,3 +20,14 @@ classify_from_metadata <- function(needs_compilation, linking_to, risky_deps) { } "ambiguous" } + +parse_build_args <- function(args) { + sensitive_only <- "--sensitive-only" %in% args + pos <- args[!startsWith(args, "--")] + list( + sensitive_only = sensitive_only, + split_into = as.integer(pos[1L]), + split_index = as.integer(pos[2L]), + ncpus = as.integer(pos[3L]) + ) +} diff --git a/local/tests/test-build-all-args.R b/local/tests/test-build-all-args.R new file mode 100644 index 0000000..3308fc3 --- /dev/null +++ b/local/tests/test-build-all-args.R @@ -0,0 +1,13 @@ +source(file.path("..", "r-minor-helpers.R")) + +test_that("parse_build_args splits flags from positionals", { + a <- parse_build_args(c("--sensitive-only", "4", "2", "8")) + expect_true(a$sensitive_only) + expect_identical(a$split_into, 4L) + expect_identical(a$split_index, 2L) + expect_identical(a$ncpus, 8L) + + b <- parse_build_args(c("4", "2", "8")) + expect_false(b$sensitive_only) + expect_identical(b$split_into, 4L) +}) -- 2.54.0 From 4d10f400a658b204c2f8aa7c42f7c033ff266bd2 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 21:14:36 +0200 Subject: [PATCH 06/14] ci: add prek/pre-commit hooks (prettier, markdownlint, editorconfig, yamllint, air) --- .pre-commit-config.yaml | 37 +++++++++++++++++++++++++++++++++++++ .yamllint.yaml | 19 +++++++++++++++++++ 2 files changed, 56 insertions(+) create mode 100644 .pre-commit-config.yaml create mode 100644 .yamllint.yaml diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..7ff6df3 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,37 @@ +# cSpell:ignore autofix autoupdate +repos: + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v6.0.0 + hooks: + - id: end-of-file-fixer + - id: trailing-whitespace + args: + - --markdown-linebreak-ext=md + - repo: https://github.com/DavidAnson/markdownlint-cli2 + rev: v0.22.1 + hooks: + - id: markdownlint-cli2 + - repo: https://github.com/rbubley/mirrors-prettier + rev: v3.8.4 + hooks: + - id: prettier + - repo: https://github.com/posit-dev/air-pre-commit + rev: 0.8.2 + hooks: + - id: air-format + - repo: https://github.com/editorconfig-checker/editorconfig-checker + rev: v3.7.0 + hooks: + - id: editorconfig-checker + - repo: https://github.com/adrienverge/yamllint.git + rev: v1.38.0 + hooks: + - id: yamllint + args: [--strict, -c=.yamllint.yaml] + - repo: local + hooks: + - id: yaml-file-extension + name: Check if YAML files has *.yaml extension. + entry: YAML filenames must have .yaml extension. + language: fail + files: .yml$ diff --git a/.yamllint.yaml b/.yamllint.yaml new file mode 100644 index 0000000..d0370a3 --- /dev/null +++ b/.yamllint.yaml @@ -0,0 +1,19 @@ +rules: + comments: + require-starting-space: false + ignore-shebangs: true + min-spaces-from-content: 1 + braces: + min-spaces-inside-empty: 0 + max-spaces-inside-empty: 0 + min-spaces-inside: 0 + max-spaces-inside: 1 + document-start: + present: false + indentation: + spaces: 2 + indent-sequences: true + # .crow/* workflows embed long single-line R commands (>1k chars) + line-length: disable + new-lines: + type: unix -- 2.54.0 From 67960b7b3b80e9f316f2830df21d4233ba9c4fea Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 21:14:36 +0200 Subject: [PATCH 07/14] style: air-format and prettier the new r-minor build files --- ...build-all-versions-install-deps-amd64.yaml | 2 +- ...build-all-versions-install-deps-arm64.yaml | 2 +- local/build-all.R | 64 ++++++++++------- local/packages-to-build.R | 72 +++++++++++++------ local/tests/test-r-minor-helpers.R | 20 ++++-- 5 files changed, 106 insertions(+), 54 deletions(-) diff --git a/.crow/build-all-versions-install-deps-amd64.yaml b/.crow/build-all-versions-install-deps-amd64.yaml index b1f33e4..285a4e2 100644 --- a/.crow/build-all-versions-install-deps-amd64.yaml +++ b/.crow/build-all-versions-install-deps-amd64.yaml @@ -10,7 +10,7 @@ labels: steps: - name: 'Install deps and bincraft' - image: "reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}" + image: 'reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}' pull: true environment: REPO_RO_TOKEN: diff --git a/.crow/build-all-versions-install-deps-arm64.yaml b/.crow/build-all-versions-install-deps-arm64.yaml index 0ef90e3..98decea 100644 --- a/.crow/build-all-versions-install-deps-arm64.yaml +++ b/.crow/build-all-versions-install-deps-arm64.yaml @@ -9,7 +9,7 @@ labels: steps: - name: 'Install deps and bincraft' - image: "reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}" + image: 'reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}' pull: true environment: REPO_RO_TOKEN: diff --git a/local/build-all.R b/local/build-all.R index 8c181f3..4e7b44e 100644 --- a/local/build-all.R +++ b/local/build-all.R @@ -21,8 +21,14 @@ pkgs <- if (sensitive_only) { readRDS("/mnt/cache/packages/pkgs_to_build.rds") } # Back-compat: tolerate an older RDS without the column (treat all as non-sensitive) -if (is.null(pkgs$r_minor_sensitive)) pkgs$r_minor_sensitive <- FALSE -sprintf("Total# of remaining package versions: %s (sensitive_only=%s)", nrow(pkgs), sensitive_only) +if (is.null(pkgs$r_minor_sensitive)) { + pkgs$r_minor_sensitive <- FALSE +} +sprintf( + "Total# of remaining package versions: %s (sensitive_only=%s)", + nrow(pkgs), + sensitive_only +) # Split into chunks for this worker chunks <- split(pkgs, cut(seq_len(nrow(pkgs)), split_into, labels = FALSE)) @@ -40,27 +46,33 @@ s3_cache <- readRDS("/mnt/cache/packages/s3_cache.rds") sprintf("S3 cache: %s files", length(s3_cache)) n <- nrow(chunk) -mapply(function(pkg, ver, sens, i) { - cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens)) - bincraft::build_binary_package( - pkg, - tag = ver, - is_r_minor_sensitive = isTRUE(sens), - s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", - s3_region = "eu-central-003", - s3_bucket = "devxy-rpkgs-binaries", - s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), - s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), - s3_package_cache = s3_cache, - metadata_db_host = "r-binaries.devxy.io", - metadata_db_name = "build_metadata", - metadata_db_table = "single_builds", - metadata_db_user = "rpkgs", - metadata_db_password = Sys.getenv("PGPASS"), - metadata_db_sslmode = "require", - metadata_db_port = 15432, - archive = TRUE, - upload = TRUE, - store_build_metadata = TRUE - ) -}, chunk$Package, chunk$Version, chunk$r_minor_sensitive, seq_len(n)) +mapply( + function(pkg, ver, sens, i) { + cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens)) + bincraft::build_binary_package( + pkg, + tag = ver, + is_r_minor_sensitive = isTRUE(sens), + s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", + s3_region = "eu-central-003", + s3_bucket = "devxy-rpkgs-binaries", + s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), + s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), + s3_package_cache = s3_cache, + metadata_db_host = "r-binaries.devxy.io", + metadata_db_name = "build_metadata", + metadata_db_table = "single_builds", + metadata_db_user = "rpkgs", + metadata_db_password = Sys.getenv("PGPASS"), + metadata_db_sslmode = "require", + metadata_db_port = 15432, + archive = TRUE, + upload = TRUE, + store_build_metadata = TRUE + ) + }, + chunk$Package, + chunk$Version, + chunk$r_minor_sensitive, + seq_len(n) +) diff --git a/local/packages-to-build.R b/local/packages-to-build.R index 3b23276..7a3652d 100644 --- a/local/packages-to-build.R +++ b/local/packages-to-build.R @@ -17,7 +17,11 @@ suppressPackageStartupMessages(library(data.table)) arch = Sys.getenv("ARCH") # target: alpine-322, ubuntu-2404, redhat-9, etc. -platform = paste(Sys.getenv("OS"), gsub("[.]", "", Sys.getenv("OS_VERSION")), sep = "-") +platform = paste( + Sys.getenv("OS"), + gsub("[.]", "", Sys.getenv("OS_VERSION")), + sep = "-" +) # Use bincraft's codename detection for S3 paths (e.g. "rhel10" not "redhat10") codename = bincraft::set_codename(NULL) @@ -34,7 +38,9 @@ con <- DBI::dbConnect( cran_archive = tools::CRAN_archive_db() cran_release = tools::CRAN_package_db() # Subset cran_archive to only those packages -cran_archive_in_release <- cran_archive[names(cran_archive) %in% cran_release$Package] +cran_archive_in_release <- cran_archive[ + names(cran_archive) %in% cran_release$Package +] archive_versions <- rbindlist( lapply(names(cran_archive), function(pkg) { @@ -126,11 +132,15 @@ query_error <- function(pkg, ver) { # Fetch all relevant columns from the database errored_pkgs <- DBI::dbGetQuery( con, - sprintf("SELECT name, tag FROM single_builds WHERE error_occurred = TRUE and platform='%s' and arch='%s'", platform, arch) + sprintf( + "SELECT name, tag FROM single_builds WHERE error_occurred = TRUE and platform='%s' and arch='%s'", + platform, + arch + ) ) errored_pkgs <- as.data.table(errored_pkgs) setkey(pkgs_to_build, Package, Version) -setnames(errored_pkgs, c("Package","Version")) +setnames(errored_pkgs, c("Package", "Version")) setkey(errored_pkgs, Package, Version) ### Final subsetting @@ -152,12 +162,14 @@ release_meta <- data.table( ) meta <- release_meta[Package %in% unique(pkgs$Package)] -meta[, triage := mapply( - classify_from_metadata, - NeedsCompilation, - LinkingTo, - MoreArgs = list(risky_deps = risky_deps) -)] +meta[, + triage := mapply( + classify_from_metadata, + NeedsCompilation, + LinkingTo, + MoreArgs = list(risky_deps = risky_deps) + ) +] # Only the "ambiguous" compiled packages need a source grep. ambiguous <- meta[triage == "ambiguous", Package] @@ -165,19 +177,35 @@ sensitive_ambiguous <- character() if (length(ambiguous) > 0L) { tmp_src <- file.path(tempdir(), "abi_src") dir.create(tmp_src, showWarnings = FALSE, recursive = TRUE) - sens <- vapply(ambiguous, function(pkg) { - out <- tryCatch({ - dl <- utils::download.packages( - pkg, destdir = tmp_src, - repos = "https://cloud.r-project.org", quiet = TRUE - ) - isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L]))) - }, error = function(e) TRUE) # fail safe: treat as sensitive - out - }, logical(1L)) + sens <- vapply( + ambiguous, + function(pkg) { + out <- tryCatch( + { + dl <- utils::download.packages( + pkg, + destdir = tmp_src, + repos = "https://cloud.r-project.org", + quiet = TRUE + ) + isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L]))) + }, + error = function(e) TRUE + ) # fail safe: treat as sensitive + out + }, + logical(1L) + ) sensitive_ambiguous <- ambiguous[sens] } -sensitive_pkgs <- unique(c(meta[triage == "sensitive", Package], sensitive_ambiguous)) +sensitive_pkgs <- unique(c( + meta[triage == "sensitive", Package], + sensitive_ambiguous +)) pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs] -sprintf("R-minor-sensitive packages: %s of %s", length(sensitive_pkgs), uniqueN(pkgs$Package)) +sprintf( + "R-minor-sensitive packages: %s of %s", + length(sensitive_pkgs), + uniqueN(pkgs$Package) +) diff --git a/local/tests/test-r-minor-helpers.R b/local/tests/test-r-minor-helpers.R index d68e99b..ac7b0a6 100644 --- a/local/tests/test-r-minor-helpers.R +++ b/local/tests/test-r-minor-helpers.R @@ -2,15 +2,27 @@ source(file.path("..", "r-minor-helpers.R")) test_that("pure-r (NeedsCompilation != yes) is not sensitive", { expect_identical(classify_from_metadata("no", NA, c("Rcpp")), "not-sensitive") - expect_identical(classify_from_metadata("", "Rcpp", c("Rcpp")), "not-sensitive") + expect_identical( + classify_from_metadata("", "Rcpp", c("Rcpp")), + "not-sensitive" + ) }) test_that("LinkingTo a risky dep is sensitive (version constraints stripped)", { - expect_identical(classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), "sensitive") - expect_identical(classify_from_metadata("yes", "R6,\n cpp11", c("Rcpp", "cpp11")), "sensitive") + expect_identical( + classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), + "sensitive" + ) + expect_identical( + classify_from_metadata("yes", "R6,\n cpp11", c("Rcpp", "cpp11")), + "sensitive" + ) }) test_that("compiled but no risky LinkingTo is ambiguous (needs source)", { - expect_identical(classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), "ambiguous") + expect_identical( + classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), + "ambiguous" + ) expect_identical(classify_from_metadata("yes", NA, c("Rcpp")), "ambiguous") }) -- 2.54.0 From cf851c1c302d40a6c12037f9a58e1bbcf160df4a Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 21:23:36 +0200 Subject: [PATCH 08/14] ci: configure hook exclusions and fix README lint (MD040/MD056) --- .editorconfig | 4 ++++ .editorconfig-checker.json | 8 +++++++ .markdownlint-cli2.yaml | 4 ++++ .pre-commit-config.yaml | 9 ++++++++ .prettierignore | 2 ++ README.md | 46 +++++++++++++++++++------------------- 6 files changed, 50 insertions(+), 23 deletions(-) create mode 100644 .editorconfig-checker.json create mode 100644 .markdownlint-cli2.yaml create mode 100644 .prettierignore diff --git a/.editorconfig b/.editorconfig index 734c5ee..82fac74 100644 --- a/.editorconfig +++ b/.editorconfig @@ -11,6 +11,10 @@ insert_final_newline = true [*.md] trim_trailing_whitespace = false +# Markdown code fences hold verbatim content (logs, snippets) whose +# indentation must not be forced to the source indent rules. +indent_style = unset +indent_size = unset # Ignore paths [/collections/ansible_collections/community/**] diff --git a/.editorconfig-checker.json b/.editorconfig-checker.json new file mode 100644 index 0000000..e5f1357 --- /dev/null +++ b/.editorconfig-checker.json @@ -0,0 +1,8 @@ +{ + "Exclude": [ + "^LICENSE\\.md$", + "^benchmark/", + "^docker/reprex/", + "^\\.crow/build-r-minor-sensitive-packages\\.yaml$" + ] +} diff --git a/.markdownlint-cli2.yaml b/.markdownlint-cli2.yaml new file mode 100644 index 0000000..66e7cc8 --- /dev/null +++ b/.markdownlint-cli2.yaml @@ -0,0 +1,4 @@ +# Rules live in .markdownlint.yaml; this file only sets cli2 options. +ignores: + - LICENSE.md + - docs/superpowers/** diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 7ff6df3..9d3a2e1 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,4 +1,13 @@ # cSpell:ignore autofix autoupdate +# Excluded: verbatim GPL license, auxiliary shell scripts with intentional +# in-string formatting, and the soon-to-be-removed standalone workflow. +exclude: | + (?x)^( + LICENSE\.md| + benchmark/| + docker/reprex/| + \.crow/build-r-minor-sensitive-packages\.yaml + ) repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v6.0.0 diff --git a/.prettierignore b/.prettierignore new file mode 100644 index 0000000..7830219 --- /dev/null +++ b/.prettierignore @@ -0,0 +1,2 @@ +LICENSE.md +.crow/build-r-minor-sensitive-packages.yaml diff --git a/README.md b/README.md index fd03609..278a475 100644 --- a/README.md +++ b/README.md @@ -186,19 +186,19 @@ Below is a collection of raw errors observed during the build process:
-``` +```text * installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n" ``` Tag does not have a NAMESPACE file and hence cannot be built. -``` +```text "* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n" ``` Dependency not available: Either because the dependency was not declared or errored itself during installation. -``` +```text In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n 162 | double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n | \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n 277 | ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n | \033[01;31m\033[K^~\033[m\033[K\n | \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n ``` @@ -230,26 +230,26 @@ For others it might be due to exotic external dependencies which require manual Help in resolving these issues are highly welcome! -| Name | Platform | Arch | Reason | Solved via | Date Created | Date Solved | -| ------------- | -------------- | ------- | -------------------------------- | ---------- | ------------ | ----------- | -| Apollonius | redhat-8 | | gmp missing | 2024-11-15 | | -| doBy | alpine-320 | amd | hangs | 2024-11-23 | | -| later | ubuntu 22 & 24 | amd | hangs for early versions - xfvb? | 2024-11-23 | | -| CoTiMA | ubuntu 22 & 24 | amd | OOM? | 2024-11-23 | | -| FrF2 | alpine | arm | hangs | 2024-11-23 | | -| FrF2.catlg128 | alpine | arm | hangs | 2024-11-23 | | -| DoE.base | alpine | arm/amd | hangs | 2024-11-23 | | -| eha | alpine | amd | hangs | 2024-11-23 | | -| gRain | alpine | amd | hangs | 2024-11-23 | | -| gRbase | alpine | amd | hangs | 2024-11-24 | | -| IDPmisc | alpine | amd | hangs | 2024-11-24 | | -| RVAideMemoire | alpine | amd | hangs | 2024-11-25 | | -| pbkrtest | alpine | arm | hangs | 2024-11-25 | | -| seewave | alpine | arm | hangs | 2024-11-25 | | -| spdep | alpine | arm | hangs | 2024-12-03 | | -| compareGroups | alpine | arm | hangs | 2024-12-11 | | -| SNPassoc | alpine | arm | hangs | 2024-12-11 | | -| surveillance | alpine | arm | hangs | 2024-12-13 | | +| Name | Platform | Arch | Reason | Date Created | Date Solved | +| ------------- | -------------- | ------- | -------------------------------- | ------------ | ----------- | +| Apollonius | redhat-8 | | gmp missing | 2024-11-15 | | +| doBy | alpine-320 | amd | hangs | 2024-11-23 | | +| later | ubuntu 22 & 24 | amd | hangs for early versions - xfvb? | 2024-11-23 | | +| CoTiMA | ubuntu 22 & 24 | amd | OOM? | 2024-11-23 | | +| FrF2 | alpine | arm | hangs | 2024-11-23 | | +| FrF2.catlg128 | alpine | arm | hangs | 2024-11-23 | | +| DoE.base | alpine | arm/amd | hangs | 2024-11-23 | | +| eha | alpine | amd | hangs | 2024-11-23 | | +| gRain | alpine | amd | hangs | 2024-11-23 | | +| gRbase | alpine | amd | hangs | 2024-11-24 | | +| IDPmisc | alpine | amd | hangs | 2024-11-24 | | +| RVAideMemoire | alpine | amd | hangs | 2024-11-25 | | +| pbkrtest | alpine | arm | hangs | 2024-11-25 | | +| seewave | alpine | arm | hangs | 2024-11-25 | | +| spdep | alpine | arm | hangs | 2024-12-03 | | +| compareGroups | alpine | arm | hangs | 2024-12-11 | | +| SNPassoc | alpine | arm | hangs | 2024-12-11 | | +| surveillance | alpine | arm | hangs | 2024-12-13 | | ## CDN Settings -- 2.54.0 From 251bba8f23381cb30ed3fd198910713eaed4654c Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 21:23:37 +0200 Subject: [PATCH 09/14] style: apply air and prettier formatting repo-wide --- .crow/archive-missed-packages.yaml | 1 - .crow/build-all-versions-amd64.yaml | 4 +- .crow/build-all-versions-arm64.yaml | 2 +- local/archive-missed-pkgs.R | 26 ++- local/check-NA.R | 9 +- local/debug-packages-writing.R | 70 ++++--- local/detect-duplicates.R | 7 +- local/excluded-packages.json | 150 +++++++------- local/fetch-rebuild-packages-from-issue.R | 52 +++-- local/label-removed-cran-packages.R | 14 +- local/last-processed-by-platform.R | 2 +- local/missing-cran-packages-db.R | 48 +++-- local/missing-cran-packages.R | 48 +++-- local/missing-packages-in-index.R | 117 ++++++++--- local/packages-without-any-binary.R | 25 ++- local/weekly-missing-binaries-audit.R | 229 ++++++++++++++++------ renovate.json | 8 +- 17 files changed, 554 insertions(+), 258 deletions(-) diff --git a/.crow/archive-missed-packages.yaml b/.crow/archive-missed-packages.yaml index bb9ecb4..88c199e 100644 --- a/.crow/archive-missed-packages.yaml +++ b/.crow/archive-missed-packages.yaml @@ -6,7 +6,6 @@ when: - event: cron cron: archive-missed-packages - matrix: include: - CODENAME: jammy diff --git a/.crow/build-all-versions-amd64.yaml b/.crow/build-all-versions-amd64.yaml index 796c812..30def98 100644 --- a/.crow/build-all-versions-amd64.yaml +++ b/.crow/build-all-versions-amd64.yaml @@ -39,7 +39,7 @@ depends_on: steps: - name: 'Build binaries' - image: "reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}" + image: 'reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}' pull: true environment: RED_HAT_DEV_PW: @@ -57,7 +57,7 @@ steps: # normal env vars GIT_USER: pat-s # set the location of the 'pkgcache' cache dir which persists the R package dependencies needed to install the packages themselves - R_PKG_CACHE_DIR: "" + R_PKG_CACHE_DIR: '' R_LIBS_USER: /mnt/cache/R-pkgs CCACHE_DIR: /mnt/cache/ccache NCPUS: 2 diff --git a/.crow/build-all-versions-arm64.yaml b/.crow/build-all-versions-arm64.yaml index 37992df..c3a2977 100644 --- a/.crow/build-all-versions-arm64.yaml +++ b/.crow/build-all-versions-arm64.yaml @@ -54,7 +54,7 @@ depends_on: steps: - name: 'Build binaries' - image: "reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}" + image: 'reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}' pull: true environment: RED_HAT_DEV_PW: diff --git a/local/archive-missed-pkgs.R b/local/archive-missed-pkgs.R index 57e92d7..f817c3e 100644 --- a/local/archive-missed-pkgs.R +++ b/local/archive-missed-pkgs.R @@ -11,16 +11,34 @@ s3fs::s3_file_system( arch = "arm64" os = "alpine321" -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/%s/%s/latest/src/contrib", arch, os)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/%s/%s/latest/src/contrib", + arch, + os +)) length(unique(sapply(strsplit(basename(files), "_"), function(x) x[1]))) -non_archived = unique(sapply(strsplit(basename(files), "_"), function(x) x[1])[duplicated(sapply(strsplit(basename(files), "_"), function(x) x[1]))]) +non_archived = unique(sapply(strsplit(basename(files), "_"), function(x) { + x[1] +})[duplicated(sapply(strsplit(basename(files), "_"), function(x) x[1]))]) # RInno is not built because it only exists for Windows non_archived = setdiff(non_archived, "RInno") # future::plan("sequential", workers = 8) future::plan("sequential", workers = 1) # future.apply::future_lapply(non_archived, function(x) archive_package(x, codename = os, arch = arch)) -lapply(non_archived, function(x) archive_package(x, codename = os, arch = arch, s3_region = "hel1", s3_endpoint = "https://hel1.your-objectstorage.com", s3_bucket = "devxy-r-package-binaries-hel1", s3_access_key_id = Sys.getenv("HETZNER_S3_ACCESS_KEY_K3S", s3_secret_access_key = Sys.getenv("HETZNER_S3_SECRET_KEY_K3S")))) +lapply(non_archived, function(x) { + archive_package( + x, + codename = os, + arch = arch, + s3_region = "hel1", + s3_endpoint = "https://hel1.your-objectstorage.com", + s3_bucket = "devxy-r-package-binaries-hel1", + s3_access_key_id = Sys.getenv( + "HETZNER_S3_ACCESS_KEY_K3S", + s3_secret_access_key = Sys.getenv("HETZNER_S3_SECRET_KEY_K3S") + ) + ) +}) # archive_package() - # grep("duckdb_", files, value = T) diff --git a/local/check-NA.R b/local/check-NA.R index 7405db4..95c158e 100644 --- a/local/check-NA.R +++ b/local/check-NA.R @@ -1,4 +1,3 @@ - ### Scope: Check whether there are any incomplete entries in PACKAGES.rds. These will result in NA when calling available.packages() foo = available.packages("https://cran.devxy.io/amd64/jammy/latest/src/contrib") sum((is.na(foo[, "Version"]))) @@ -17,7 +16,9 @@ foo = available.packages("https://cran.devxy.io/amd64/rhel9/latest/src/contrib") sum((is.na(foo[, "Version"]))) which((is.na(foo[, "Version"]))) -foo = available.packages("https://cran.devxy.io/amd64/alpine320/latest/src/contrib") +foo = available.packages( + "https://cran.devxy.io/amd64/alpine320/latest/src/contrib" +) sum((is.na(foo[, "Version"]))) which((is.na(foo[, "Version"]))) @@ -40,6 +41,8 @@ foo = available.packages("https://cran.devxy.io/arm64/rhel9/latest/src/contrib") sum((is.na(foo[, "Version"]))) which((is.na(foo[, "Version"]))) -foo = available.packages("https://cran.devxy.io/arm64/alpine320/latest/src/contrib") +foo = available.packages( + "https://cran.devxy.io/arm64/alpine320/latest/src/contrib" +) sum((is.na(foo[, "Version"]))) which((is.na(foo[, "Version"]))) diff --git a/local/debug-packages-writing.R b/local/debug-packages-writing.R index 5a0fd24..533ae2b 100644 --- a/local/debug-packages-writing.R +++ b/local/debug-packages-writing.R @@ -2,42 +2,58 @@ library(future) future::plan(multisession) future::plan(sequential) time = Sys.time() -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/rhel9/latest/src/contrib/Archive"), recurse = T) +files <- s3fs::s3_dir_ls( + sprintf( + "devxy-r-package-binaries-hel1/arm64/rhel9/latest/src/contrib/Archive" + ), + recurse = T +) Sys.time() - time - archive <- llply(dirs, function(dir) { - files <- list.files(dir, recursive = FALSE, full.names = TRUE, pattern = "*.tar.gz") + files <- list.files( + dir, + recursive = FALSE, + full.names = TRUE, + pattern = "*.tar.gz" + ) if (length(files) == 0) { print(paste0("Error: Empty directory: ", dir)) return(NULL) } info <- file.info(files) - tryCatch({ - rownames(info) <- paste0(basename(dirname(files)), "/", basename(files)) - }, error = function(e) { - print(paste0("Error: Exception catched for Archived directory: ", dir)) - print(e) - return(NULL) - }) + tryCatch( + { + rownames(info) <- paste0(basename(dirname(files)), "/", basename(files)) + }, + error = function(e) { + print(paste0("Error: Exception catched for Archived directory: ", dir)) + print(e) + return(NULL) + } + ) info }) -tryCatch({ - rownames(info) <- paste0(basename(dirname(files)), "/", basename(files)) -}, error = function(e) { - print(paste0("Error: Exception catched for Archived directory: ", dir)) - print(e) - return(NULL) -}) +tryCatch( + { + rownames(info) <- paste0(basename(dirname(files)), "/", basename(files)) + }, + error = function(e) { + print(paste0("Error: Exception catched for Archived directory: ", dir)) + print(e) + return(NULL) + } +) - - -curl::curl_download("https://cran.devxy.io/amd64/rhel9/latest/src/contrib/PACKAGES.db", "PACKAGES.db") +curl::curl_download( + "https://cran.devxy.io/amd64/rhel9/latest/src/contrib/PACKAGES.db", + "PACKAGES.db" +) con = DBI::dbConnect(RSQLite::SQLite(), "PACKAGES.db") df = DBI::dbReadTable(con, "packages") @@ -50,6 +66,14 @@ which(grepl("digest", df$Package)) system("cat /tmp/PACKAGES | grep '^Package: digest' | wc -l") -curl::curl_fetch_memory("https://cloud.r-project.org/src/contrib/Meta/archive.rds") -remotes = readRDS(url("https://cloud.r-project.org/src/contrib/Meta/archive.rds", "rb")) -packages = readRDS(url("https://cran.devxy.io/arm64/noble/latest/src/contrib/PACKAGES.rds", "rb")) +curl::curl_fetch_memory( + "https://cloud.r-project.org/src/contrib/Meta/archive.rds" +) +remotes = readRDS(url( + "https://cloud.r-project.org/src/contrib/Meta/archive.rds", + "rb" +)) +packages = readRDS(url( + "https://cran.devxy.io/arm64/noble/latest/src/contrib/PACKAGES.rds", + "rb" +)) diff --git a/local/detect-duplicates.R b/local/detect-duplicates.R index 076499d..f7e2c18 100644 --- a/local/detect-duplicates.R +++ b/local/detect-duplicates.R @@ -37,10 +37,13 @@ future.apply::future_lapply( files <- s3fs::s3_dir_ls( sprintf( "devxy-r-package-binaries-hel1/%s/%s/latest/src/contrib", - arch, codename + arch, + codename ) ) - if (length(files) == 0) return(NULL) + if (length(files) == 0) { + return(NULL) + } files_b <- basename(files) pkg_names <- sapply(strsplit(files_b, "_"), `[`, 1) dupes <- unique(pkg_names[duplicated(pkg_names)]) diff --git a/local/excluded-packages.json b/local/excluded-packages.json index dccfc0b..6489998 100644 --- a/local/excluded-packages.json +++ b/local/excluded-packages.json @@ -1,77 +1,77 @@ [ - {"package": "RInno", "reason": "windows-only"}, - {"package": "KeyboardSimulator", "reason": "windows-only"}, - {"package": "R2PPT", "reason": "windows-only"}, - {"package": "RWinEdt", "reason": "windows-only"}, - {"package": "blatr", "reason": "windows-only"}, - {"package": "excel.link", "reason": "windows-only"}, - {"package": "spectrino", "reason": "windows-only"}, - {"package": "taskscheduleR", "reason": "windows-only"}, - {"package": "MDSGUI", "reason": "windows-only"}, - {"package": "BiplotGUI", "reason": "windows-only"}, - {"package": "R2wd", "reason": "windows-only"}, - {"package": "rFUSION", "reason": "windows-only"}, - {"package": "MediaNews", "reason": "windows-only"}, - {"package": "doBy", "reason": "hang"}, - {"package": "IDPmisc", "reason": "hang"}, - {"package": "frailtypack", "reason": "hang"}, - {"package": "afex", "reason": "hang"}, - {"package": "FrF2", "reason": "hang"}, - {"package": "DoE.base", "reason": "hang"}, - {"package": "agricolae", "reason": "hang"}, - {"package": "doFuture", "reason": "hang"}, - {"package": "fscaret", "reason": "hang"}, - {"package": "PHYLOGR", "reason": "hang"}, - {"package": "seewave", "reason": "hang"}, - {"package": "pls", "reason": "hang"}, - {"package": "relaimpo", "reason": "hang"}, - {"package": "geepack", "reason": "hang"}, - {"package": "gggenes", "reason": "hang"}, - {"package": "NPCirc", "reason": "hang"}, - {"package": "repmis", "reason": "hang"}, - {"package": "PNDSIBGE", "reason": "hang"}, - {"package": "lidR", "reason": "hang"}, - {"package": "poismf", "reason": "hang"}, - {"package": "neonstore", "reason": "hang"}, - {"package": "MachineShop", "reason": "hang"}, - {"package": "mvst", "reason": "hang"}, - {"package": "MacBehaviour", "reason": "hang"}, - {"package": "mcmcderive", "reason": "hang"}, - {"package": "RGIFT", "reason": "hang"}, - {"package": "KnowBR", "reason": "hang"}, - {"package": "netmeta", "reason": "hang"}, - {"package": "spdep", "reason": "hang"}, - {"package": "Rfast", "reason": "hang"}, - {"package": "compareGroups", "reason": "hang"}, - {"package": "ff", "reason": "hang"}, - {"package": "GsymPoint", "reason": "hang"}, - {"package": "RcppDynProg", "reason": "hang"}, - {"package": "comtradr", "reason": "hang"}, - {"package": "FD", "reason": "hang"}, - {"package": "PearsonDS", "reason": "hang"}, - {"package": "DCluster", "reason": "hang"}, - {"package": "gRc", "reason": "hang"}, - {"package": "mixlm", "reason": "hang"}, - {"package": "geospt", "reason": "hang"}, - {"package": "fdth", "reason": "hang"}, - {"package": "ffmanova", "reason": "hang"}, - {"package": "fiery", "reason": "hang"}, - {"package": "ffscrapr", "reason": "hang"}, - {"package": "cold", "reason": "hang"}, - {"package": "RcmdrPlugin.DoE", "reason": "hang"}, - {"package": "RcmdrPlugin.NMBU", "reason": "hang"}, - {"package": "RcmdrPlugin.RiskDemo", "reason": "hang"}, - {"package": "RcmdrPlugin.ROC", "reason": "hang"}, - {"package": "RcmdrPlugin.TeachStat", "reason": "hang"}, - {"package": "RcmdrPlugin.TeachingDemos", "reason": "hang"}, - {"package": "RcmdrPlugin.UCA", "reason": "hang"}, - {"package": "RcmdrPlugin.WorldFlora", "reason": "hang"}, - {"package": "RcmdrPlugin.aRnova", "reason": "hang"}, - {"package": "RcmdrPlugin.depthTools", "reason": "hang"}, - {"package": "RcmdrPlugin.orloca", "reason": "hang"}, - {"package": "RcmdrPlugin.sos", "reason": "hang"}, - {"package": "RcmdrPlugin.survival", "reason": "hang"}, - {"package": "RcmdrPlugin.temis", "reason": "hang"}, - {"package": "GWlasso", "reason": "hang"}, - {"package": "GWmodelVis", "reason": "hang"} + { "package": "RInno", "reason": "windows-only" }, + { "package": "KeyboardSimulator", "reason": "windows-only" }, + { "package": "R2PPT", "reason": "windows-only" }, + { "package": "RWinEdt", "reason": "windows-only" }, + { "package": "blatr", "reason": "windows-only" }, + { "package": "excel.link", "reason": "windows-only" }, + { "package": "spectrino", "reason": "windows-only" }, + { "package": "taskscheduleR", "reason": "windows-only" }, + { "package": "MDSGUI", "reason": "windows-only" }, + { "package": "BiplotGUI", "reason": "windows-only" }, + { "package": "R2wd", "reason": "windows-only" }, + { "package": "rFUSION", "reason": "windows-only" }, + { "package": "MediaNews", "reason": "windows-only" }, + { "package": "doBy", "reason": "hang" }, + { "package": "IDPmisc", "reason": "hang" }, + { "package": "frailtypack", "reason": "hang" }, + { "package": "afex", "reason": "hang" }, + { "package": "FrF2", "reason": "hang" }, + { "package": "DoE.base", "reason": "hang" }, + { "package": "agricolae", "reason": "hang" }, + { "package": "doFuture", "reason": "hang" }, + { "package": "fscaret", "reason": "hang" }, + { "package": "PHYLOGR", "reason": "hang" }, + { "package": "seewave", "reason": "hang" }, + { "package": "pls", "reason": "hang" }, + { "package": "relaimpo", "reason": "hang" }, + { "package": "geepack", "reason": "hang" }, + { "package": "gggenes", "reason": "hang" }, + { "package": "NPCirc", "reason": "hang" }, + { "package": "repmis", "reason": "hang" }, + { "package": "PNDSIBGE", "reason": "hang" }, + { "package": "lidR", "reason": "hang" }, + { "package": "poismf", "reason": "hang" }, + { "package": "neonstore", "reason": "hang" }, + { "package": "MachineShop", "reason": "hang" }, + { "package": "mvst", "reason": "hang" }, + { "package": "MacBehaviour", "reason": "hang" }, + { "package": "mcmcderive", "reason": "hang" }, + { "package": "RGIFT", "reason": "hang" }, + { "package": "KnowBR", "reason": "hang" }, + { "package": "netmeta", "reason": "hang" }, + { "package": "spdep", "reason": "hang" }, + { "package": "Rfast", "reason": "hang" }, + { "package": "compareGroups", "reason": "hang" }, + { "package": "ff", "reason": "hang" }, + { "package": "GsymPoint", "reason": "hang" }, + { "package": "RcppDynProg", "reason": "hang" }, + { "package": "comtradr", "reason": "hang" }, + { "package": "FD", "reason": "hang" }, + { "package": "PearsonDS", "reason": "hang" }, + { "package": "DCluster", "reason": "hang" }, + { "package": "gRc", "reason": "hang" }, + { "package": "mixlm", "reason": "hang" }, + { "package": "geospt", "reason": "hang" }, + { "package": "fdth", "reason": "hang" }, + { "package": "ffmanova", "reason": "hang" }, + { "package": "fiery", "reason": "hang" }, + { "package": "ffscrapr", "reason": "hang" }, + { "package": "cold", "reason": "hang" }, + { "package": "RcmdrPlugin.DoE", "reason": "hang" }, + { "package": "RcmdrPlugin.NMBU", "reason": "hang" }, + { "package": "RcmdrPlugin.RiskDemo", "reason": "hang" }, + { "package": "RcmdrPlugin.ROC", "reason": "hang" }, + { "package": "RcmdrPlugin.TeachStat", "reason": "hang" }, + { "package": "RcmdrPlugin.TeachingDemos", "reason": "hang" }, + { "package": "RcmdrPlugin.UCA", "reason": "hang" }, + { "package": "RcmdrPlugin.WorldFlora", "reason": "hang" }, + { "package": "RcmdrPlugin.aRnova", "reason": "hang" }, + { "package": "RcmdrPlugin.depthTools", "reason": "hang" }, + { "package": "RcmdrPlugin.orloca", "reason": "hang" }, + { "package": "RcmdrPlugin.sos", "reason": "hang" }, + { "package": "RcmdrPlugin.survival", "reason": "hang" }, + { "package": "RcmdrPlugin.temis", "reason": "hang" }, + { "package": "GWlasso", "reason": "hang" }, + { "package": "GWmodelVis", "reason": "hang" } ] diff --git a/local/fetch-rebuild-packages-from-issue.R b/local/fetch-rebuild-packages-from-issue.R index d944dd0..26e544d 100644 --- a/local/fetch-rebuild-packages-from-issue.R +++ b/local/fetch-rebuild-packages-from-issue.R @@ -1,15 +1,21 @@ library(httr2, quietly = TRUE) forgejo_base <- "https://git.devxy.io/api/v1" -repo <- "devxy/build-cran-binaries" -platform <- Sys.getenv("PLATFORM") -arch <- Sys.getenv("ARCH") -token <- Sys.getenv("FORGEJO_TOKEN") -output_file <- Sys.getenv("REBUILD_PKG_LIST", "/tmp/rebuild_pkgs.txt") +repo <- "devxy/build-cran-binaries" +platform <- Sys.getenv("PLATFORM") +arch <- Sys.getenv("ARCH") +token <- Sys.getenv("FORGEJO_TOKEN") +output_file <- Sys.getenv("REBUILD_PKG_LIST", "/tmp/rebuild_pkgs.txt") -if (nchar(platform) == 0) stop("PLATFORM env var is not set") -if (nchar(arch) == 0) stop("ARCH env var is not set") -if (nchar(token) == 0) stop("FORGEJO_TOKEN env var is not set") +if (nchar(platform) == 0) { + stop("PLATFORM env var is not set") +} +if (nchar(arch) == 0) { + stop("ARCH env var is not set") +} +if (nchar(token) == 0) { + stop("FORGEJO_TOKEN env var is not set") +} os_family <- if (grepl("^ubuntu", platform)) { "Ubuntu" @@ -21,12 +27,16 @@ os_family <- if (grepl("^ubuntu", platform)) { platform } -issue_title <- sprintf("Missing package binaries for latest version (%s)", os_family) +issue_title <- sprintf( + "Missing package binaries for latest version (%s)", + os_family +) cat(sprintf("Searching for issue: %s\n", issue_title)) search_url <- sprintf( "%s/repos/%s/issues?type=issues&state=open&q=%s&limit=50", - forgejo_base, repo, + forgejo_base, + repo, utils::URLencode(issue_title, reserved = TRUE) ) search_resp <- request(search_url) |> @@ -34,7 +44,9 @@ search_resp <- request(search_url) |> req_perform() issues <- resp_body_json(search_resp, simplifyVector = FALSE) -match_idx <- which(vapply(issues, function(x) x$title, character(1)) == issue_title) +match_idx <- which( + vapply(issues, function(x) x$title, character(1)) == issue_title +) if (length(match_idx) == 0) { cat("No matching issue found - nothing to rebuild\n") @@ -55,7 +67,10 @@ lines <- strsplit(body, "\n", fixed = TRUE)[[1]] plat_header <- sprintf("## %s", platform) plat_idx <- which(lines == plat_header) if (length(plat_idx) == 0) { - cat(sprintf("No section found for platform %s - nothing to rebuild\n", platform)) + cat(sprintf( + "No section found for platform %s - nothing to rebuild\n", + platform + )) writeLines(character(0), output_file) q("no") } @@ -64,7 +79,11 @@ if (length(plat_idx) == 0) { arch_pattern <- sprintf("^### %s ", arch) arch_idx <- which(grepl(arch_pattern, lines) & seq_along(lines) > plat_idx[1]) if (length(arch_idx) == 0) { - cat(sprintf("No section found for arch %s under %s - nothing to rebuild\n", arch, platform)) + cat(sprintf( + "No section found for arch %s under %s - nothing to rebuild\n", + arch, + platform + )) writeLines(character(0), output_file) q("no") } @@ -82,6 +101,11 @@ pkg_lines <- section_lines[grepl("^- ", section_lines)] pkgs <- sub("^- ([^ ]+) \\(.*\\)$", "\\1", pkg_lines) pkgs <- pkgs[nchar(pkgs) > 0 & pkgs != "_None_"] -cat(sprintf("Found %d rebuildable packages for %s/%s\n", length(pkgs), platform, arch)) +cat(sprintf( + "Found %d rebuildable packages for %s/%s\n", + length(pkgs), + platform, + arch +)) writeLines(pkgs, output_file) cat(sprintf("Wrote package list to %s\n", output_file)) diff --git a/local/label-removed-cran-packages.R b/local/label-removed-cran-packages.R index 92cd940..12dff9d 100644 --- a/local/label-removed-cran-packages.R +++ b/local/label-removed-cran-packages.R @@ -6,9 +6,13 @@ archived <- quickcode::archivedPkg() |> pull(name) # Query all distinct pkgs in the DB -con <- DBI::dbConnect(RPostgres::Postgres(), - dbname = "build_metadata", host = "r-binaries.devxy.io", - port = 15432, user = "r_binaries", password = Sys.getenv("PGPASS"), +con <- DBI::dbConnect( + RPostgres::Postgres(), + dbname = "build_metadata", + host = "r-binaries.devxy.io", + port = 15432, + user = "r_binaries", + password = Sys.getenv("PGPASS"), sslmode = "require" ) @@ -22,7 +26,9 @@ to_process <- pkgs_db[pkgs_db %in% archived] # for all matches, set 'removed = TRUE' sapply(to_process, function(.x) { - DBI::dbExecute(con, "UPDATE single_builds SET removed = 'TRUE' where name = $1", + DBI::dbExecute( + con, + "UPDATE single_builds SET removed = 'TRUE' where name = $1", params = list(.x) ) }) diff --git a/local/last-processed-by-platform.R b/local/last-processed-by-platform.R index f913610..398b6c1 100644 --- a/local/last-processed-by-platform.R +++ b/local/last-processed-by-platform.R @@ -7,4 +7,4 @@ query_metadata_table() |> group_by(platform, arch) |> arrange(desc(timestamp)) |> select(name, timestamp) |> - filter(row_number()==1) + filter(row_number() == 1) diff --git a/local/missing-cran-packages-db.R b/local/missing-cran-packages-db.R index 56b3d94..882bca4 100644 --- a/local/missing-cran-packages-db.R +++ b/local/missing-cran-packages-db.R @@ -3,22 +3,34 @@ library(bincraft) library(dplyr) library(DBI) -con <- DBI::dbConnect(RPostgres::Postgres(), - dbname = "build_metadata", host = "r-binaries.devxy.io", - port = 15432, user = "r_binaries", password = Sys.getenv("PGPASS"), +con <- DBI::dbConnect( + RPostgres::Postgres(), + dbname = "build_metadata", + host = "r-binaries.devxy.io", + port = 15432, + user = "r_binaries", + password = Sys.getenv("PGPASS"), sslmode = "require" ) `%nin%` <- Negate(`%in%`) -new_packages <- get_new_cran_packages(lubridate::interval(lubridate::today(), lubridate::today() - 2))$name -removed_pkgs <- get_removed_cran_packages(lubridate::interval(lubridate::today(), lubridate::today() - 2))$name +new_packages <- get_new_cran_packages(lubridate::interval( + lubridate::today(), + lubridate::today() - 2 +))$name +removed_pkgs <- get_removed_cran_packages(lubridate::interval( + lubridate::today(), + lubridate::today() - 2 +))$name # filter windows packages # filter new packages from the last X days -cran_pkgs <- unique(tools::CRAN_package_db() |> - filter(`OS_type` != "windows" | is.na(`OS_type`)) |> - filter(`Package` %nin% new_packages) |> - pull(Package)) +cran_pkgs <- unique( + tools::CRAN_package_db() |> + filter(`OS_type` != "windows" | is.na(`OS_type`)) |> + filter(`Package` %nin% new_packages) |> + pull(Package) +) arch <- "arm64" platform <- "redhat-8" @@ -28,13 +40,24 @@ platform <- "alpine-321" platform <- "ubuntu-2204" platform <- "ubuntu-2404" -data <- dbGetQuery(con, "SELECT name FROM single_builds WHERE platform = $1 AND arch = $2 and removed = FALSE;", params = list(platform, arch)) -removed_pkgs = get_removed_cran_packages(lubridate::interval(lubridate::today(), lubridate::today() - 2))$name +data <- dbGetQuery( + con, + "SELECT name FROM single_builds WHERE platform = $1 AND arch = $2 and removed = FALSE;", + params = list(platform, arch) +) +removed_pkgs = get_removed_cran_packages(lubridate::interval( + lubridate::today(), + lubridate::today() - 2 +))$name pkgs_db <- unique(data$name) pkgs_db = setdiff(pkgs_db, removed_pkgs) pkgs <- setdiff(cran_pkgs, pkgs_db) -formatted_pkgs <- gsub('"', "'", capture.output(dput(as.character(na.omit(pkgs[1:length(pkgs)]))))) +formatted_pkgs <- gsub( + '"', + "'", + capture.output(dput(as.character(na.omit(pkgs[1:length(pkgs)])))) +) formatted_pkgs_one_line <- paste(formatted_pkgs, collapse = " ") cat(formatted_pkgs_one_line, "\n", sep = "") @@ -44,7 +67,6 @@ clipr::write_clip(formatted_pkgs_one_line) # sapply(pkgs, function(x) which(grepl(sprintf("^%s$", x), cran_pkgs))) - s3fs::s3_file_system( aws_access_key_id = Sys.getenv("HETZNER_S3_ACCESS_KEY_K3S"), aws_secret_access_key = Sys.getenv("HETZNER_S3_SECRET_KEY_K3S"), diff --git a/local/missing-cran-packages.R b/local/missing-cran-packages.R index ab71b5b..ca5cad8 100644 --- a/local/missing-cran-packages.R +++ b/local/missing-cran-packages.R @@ -10,19 +10,39 @@ s3fs::s3_file_system( ) s3fs::s3_dir_ls("s3://devxy-r-package-binaries") -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/jammy/latest/src/contrib")) # done -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/noble/latest/src/contrib")) # done -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/rhel8/latest/src/contrib")) # done -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/rhel9/latest/src/contrib")) # done -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/alpine320/latest/src/contrib")) # done +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/jammy/latest/src/contrib" +)) # done +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/noble/latest/src/contrib" +)) # done +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/rhel8/latest/src/contrib" +)) # done +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/rhel9/latest/src/contrib" +)) # done +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/alpine320/latest/src/contrib" +)) # done -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/jammy/latest/src/contrib")) # done -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/noble/latest/src/contrib")) # done -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/rhel8/latest/src/contrib")) # done -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/rhel9/latest/src/contrib")) # done -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/alpine320/latest/src/contrib")) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/jammy/latest/src/contrib" +)) # done +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/noble/latest/src/contrib" +)) # done +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/rhel8/latest/src/contrib" +)) # done +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/rhel9/latest/src/contrib" +)) # done +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/alpine320/latest/src/contrib" +)) -files_b=basename(files) +files_b = basename(files) files_b_split = sapply(strsplit(basename(files_b), "_"), function(x) x[1]) cran_pkgs <- tools::CRAN_package_db() |> @@ -30,7 +50,11 @@ cran_pkgs <- tools::CRAN_package_db() |> pull(Package) pkgs = setdiff(cran_pkgs, files_b_split) -formatted_pkgs <- gsub('"', "'", capture.output(dput(as.character(na.omit(pkgs[1:900]))))) +formatted_pkgs <- gsub( + '"', + "'", + capture.output(dput(as.character(na.omit(pkgs[1:900])))) +) formatted_pkgs_one_line <- paste(formatted_pkgs, collapse = " ") cat(formatted_pkgs_one_line, "\n", sep = "") diff --git a/local/missing-packages-in-index.R b/local/missing-packages-in-index.R index 810c89d..6453888 100644 --- a/local/missing-packages-in-index.R +++ b/local/missing-packages-in-index.R @@ -7,42 +7,109 @@ s3fs::s3_file_system( region_name = "hel1", ) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/jammy/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/noble/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/rhel8/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/rhel9/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/alpine320/latest/src/contrib")) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/jammy/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/noble/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/rhel8/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/rhel9/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/alpine320/latest/src/contrib" +)) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/jammy/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/noble/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/rhel8/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/rhel9/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/alpine320/latest/src/contrib")) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/jammy/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/noble/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/rhel8/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/rhel9/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/alpine320/latest/src/contrib" +)) -files_b=basename(files) +files_b = basename(files) files_b_split = sapply(strsplit(basename(files_b), "_"), function(x) x[1]) -files_b_split = setdiff(files_b_split, c("PACKAGES", "PACKAGES.gz", "PACKAGES.rds", "PACKAGES.db", "Archive")) +files_b_split = setdiff( + files_b_split, + c("PACKAGES", "PACKAGES.gz", "PACKAGES.rds", "PACKAGES.db", "Archive") +) # windows-only -files_b_split = setdiff(files_b_split, c("PACKAGES", "PACKAGES.gz", "PACKAGES.rds", "PACKAGES.db", "Archive", "RInno", "KeyboardSimulator", "R2PPT", "RWinEdt", "blatr", "excel.link", "spectrino", "taskscheduleR", "MDSGUI", "BiplotGUI", "R2wd")) -pkgs_index = available.packages("https://cran.devxy.io/amd64/jammy/latest/src/contrib")[, "Package"] -pkgs_index_b=basename(pkgs_index) +files_b_split = setdiff( + files_b_split, + c( + "PACKAGES", + "PACKAGES.gz", + "PACKAGES.rds", + "PACKAGES.db", + "Archive", + "RInno", + "KeyboardSimulator", + "R2PPT", + "RWinEdt", + "blatr", + "excel.link", + "spectrino", + "taskscheduleR", + "MDSGUI", + "BiplotGUI", + "R2wd" + ) +) +pkgs_index = available.packages( + "https://cran.devxy.io/amd64/jammy/latest/src/contrib" +)[, "Package"] +pkgs_index_b = basename(pkgs_index) if (length(files_b_split) != length(pkgs_index_b)) { pkgs_missing = setdiff(files_b_split, pkgs_index_b) message("Packages missing in index but present in S3:") pkgs_missing - formatted_pkgs <- gsub('"', "'", capture.output(dput(as.character(na.omit(pkgs_missing[1:900]))))) + formatted_pkgs <- gsub( + '"', + "'", + capture.output(dput(as.character(na.omit(pkgs_missing[1:900])))) + ) formatted_pkgs_one_line <- paste(formatted_pkgs, collapse = " ") cat(formatted_pkgs_one_line, "\n", sep = "") } -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/noble/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/rhel8/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/rhel9/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/amd64/alpine320/latest/src/contrib")) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/noble/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/rhel8/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/rhel9/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/amd64/alpine320/latest/src/contrib" +)) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/jammy/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/noble/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/rhel8/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/rhel9/latest/src/contrib")) -files <- s3fs::s3_dir_ls(sprintf("devxy-r-package-binaries-hel1/arm64/alpine320/latest/src/contrib")) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/jammy/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/noble/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/rhel8/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/rhel9/latest/src/contrib" +)) +files <- s3fs::s3_dir_ls(sprintf( + "devxy-r-package-binaries-hel1/arm64/alpine320/latest/src/contrib" +)) diff --git a/local/packages-without-any-binary.R b/local/packages-without-any-binary.R index a6994ad..f42196f 100644 --- a/local/packages-without-any-binary.R +++ b/local/packages-without-any-binary.R @@ -1,17 +1,24 @@ ### Lists packages without any successful binaries, i.e. pkgs for which all builds errored (per platform & arch) library(DBI) library(dplyr) -con <- DBI::dbConnect(RPostgres::Postgres(), - dbname = "build_metadata", host = "r-binaries.devxy.io", - port = 15432, user = "r_binaries", password = Sys.getenv("PGPASS"), +con <- DBI::dbConnect( + RPostgres::Postgres(), + dbname = "build_metadata", + host = "r-binaries.devxy.io", + port = 15432, + user = "r_binaries", + password = Sys.getenv("PGPASS"), sslmode = "require" ) -cran_pkgs <- tools::CRAN_package_db() |> - filter(`Date/Publication` <= "2024-12-02") |> +cran_pkgs <- tools::CRAN_package_db() |> + filter(`Date/Publication` <= "2024-12-02") |> pull(Package) -data <- dbGetQuery(con, "SELECT name,platform,arch,removed,error_occurred FROM single_builds;") +data <- dbGetQuery( + con, + "SELECT name,platform,arch,removed,error_occurred FROM single_builds;" +) pkgs <- data |> filter(platform == "alpine-320", arch == "arm64") |> @@ -22,7 +29,11 @@ pkgs <- data |> pkgs = setdiff(cran_pkgs, pkgs) # Format, remove line breaks, and print as a single line -formatted_pkgs <- gsub('"', "'", capture.output(dput(as.character(na.omit(pkgs[1:900]))))) +formatted_pkgs <- gsub( + '"', + "'", + capture.output(dput(as.character(na.omit(pkgs[1:900])))) +) formatted_pkgs_one_line <- paste(formatted_pkgs, collapse = " ") diff --git a/local/weekly-missing-binaries-audit.R b/local/weekly-missing-binaries-audit.R index c6b67e5..6b32037 100644 --- a/local/weekly-missing-binaries-audit.R +++ b/local/weekly-missing-binaries-audit.R @@ -15,10 +15,14 @@ library(httr2, quietly = TRUE) # Environment / config # --------------------------------------------------------------------------- platform <- Sys.getenv("PLATFORM") -arch <- Sys.getenv("ARCH") +arch <- Sys.getenv("ARCH") -if (nchar(platform) == 0) stop("PLATFORM env var is not set") -if (nchar(arch) == 0) stop("ARCH env var is not set") +if (nchar(platform) == 0) { + stop("PLATFORM env var is not set") +} +if (nchar(arch) == 0) { + stop("ARCH env var is not set") +} # Map platform names to S3 codenames s3_codename <- gsub("-", "", platform) @@ -37,8 +41,13 @@ os_family <- if (grepl("^ubuntu", platform)) { platform } -cat(sprintf("Platform: %s | Arch: %s | S3 codename: %s | OS family: %s\n", - platform, arch, s3_codename, os_family)) +cat(sprintf( + "Platform: %s | Arch: %s | S3 codename: %s | OS family: %s\n", + platform, + arch, + s3_codename, + os_family +)) # --------------------------------------------------------------------------- # 1. Query PostgreSQL for known build failures (before s3fs init to avoid @@ -47,19 +56,20 @@ cat(sprintf("Platform: %s | Arch: %s | S3 codename: %s | OS family: %s\n", cat("Connecting to PostgreSQL...\n") con <- DBI::dbConnect( RPostgres::Postgres(), - dbname = "build_metadata", - host = "r-binaries.devxy.io", - port = 15432, - user = "rpkgs", + dbname = "build_metadata", + host = "r-binaries.devxy.io", + port = 15432, + user = "rpkgs", password = Sys.getenv("PGPASS"), - sslmode = "require" + sslmode = "require" ) errored_pkgs <- DBI::dbGetQuery( con, sprintf( "SELECT name, tag FROM single_builds WHERE error_occurred = TRUE AND platform = '%s' AND arch = '%s'", - platform, arch + platform, + arch ) ) DBI::dbDisconnect(con) @@ -69,7 +79,12 @@ if (nrow(errored_dt) > 0) { setnames(errored_dt, c("Package", "Version")) setkey(errored_dt, Package, Version) } -cat(sprintf("Found %d known build failures for %s/%s\n", nrow(errored_dt), platform, arch)) +cat(sprintf( + "Found %d known build failures for %s/%s\n", + nrow(errored_dt), + platform, + arch +)) # --------------------------------------------------------------------------- # 2. CRAN release packages (uses curl internally - must come after PG) @@ -86,28 +101,36 @@ cran_dt <- data.table( # --------------------------------------------------------------------------- cat("Connecting to S3...\n") s3fs::s3_file_system( - aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), + aws_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), aws_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), - endpoint = "https://s3.eu-central-003.backblazeb2.com", - region_name = "eu-central-003", - refresh = TRUE + endpoint = "https://s3.eu-central-003.backblazeb2.com", + region_name = "eu-central-003", + refresh = TRUE ) -s3_path <- sprintf("devxy-rpkgs-binaries/%s/%s/latest/src/contrib", arch, s3_codename) +s3_path <- sprintf( + "devxy-rpkgs-binaries/%s/%s/latest/src/contrib", + arch, + s3_codename +) cat(sprintf("Listing S3 path: %s\n", s3_path)) s3_pkgs <- tryCatch( s3fs::s3_dir_ls(s3_path, recurse = FALSE), error = function(e) { - cat(sprintf("WARNING: Could not list S3 path %s: %s\n", s3_path, conditionMessage(e))) + cat(sprintf( + "WARNING: Could not list S3 path %s: %s\n", + s3_path, + conditionMessage(e) + )) character(0) } ) file_names <- basename(s3_pkgs) -matches <- regexec("^([A-Za-z0-9.]+)_([0-9][^/]*)\\.tar\\.gz$", file_names) -parts <- regmatches(file_names, matches) -parts <- parts[sapply(parts, length) == 3] +matches <- regexec("^([A-Za-z0-9.]+)_([0-9][^/]*)\\.tar\\.gz$", file_names) +parts <- regmatches(file_names, matches) +parts <- parts[sapply(parts, length) == 3] if (length(parts) == 0) { s3_dt <- data.table(Package = character(0), Version = character(0)) } else { @@ -117,13 +140,18 @@ if (length(parts) == 0) { ) } -cat(sprintf("S3 contains %d tarballs for %s/%s\n", nrow(s3_dt), arch, s3_codename)) +cat(sprintf( + "S3 contains %d tarballs for %s/%s\n", + nrow(s3_dt), + arch, + s3_codename +)) # --------------------------------------------------------------------------- # 4. Find missing packages (CRAN release version not in S3) # --------------------------------------------------------------------------- setkey(cran_dt, Package, Version) -setkey(s3_dt, Package, Version) +setkey(s3_dt, Package, Version) missing_dt <- cran_dt[!s3_dt] cat(sprintf("%d CRAN release packages missing from S3\n", nrow(missing_dt))) @@ -132,9 +160,15 @@ cat(sprintf("%d CRAN release packages missing from S3\n", nrow(missing_dt))) # --------------------------------------------------------------------------- script_dir <- tryCatch( dirname(normalizePath( - if (exists("ofile", envir = sys.frame(1), inherits = FALSE)) sys.frame(1)$ofile - else commandArgs(trailingOnly = FALSE)[grepl("--file=", commandArgs(trailingOnly = FALSE))] |> - sub("--file=", "", x = _), + if (exists("ofile", envir = sys.frame(1), inherits = FALSE)) { + sys.frame(1)$ofile + } else { + commandArgs(trailingOnly = FALSE)[grepl( + "--file=", + commandArgs(trailingOnly = FALSE) + )] |> + sub("--file=", "", x = _) + }, mustWork = FALSE )), error = function(e) "local" @@ -143,13 +177,23 @@ excluded_path <- file.path(script_dir, "excluded-packages.json") excluded_pkgs <- data.table(package = character(0), reason = character(0)) if (file.exists(excluded_path)) { - excluded_raw <- jsonlite::fromJSON(excluded_path) + excluded_raw <- jsonlite::fromJSON(excluded_path) excluded_pkgs <- as.data.table(excluded_raw) - cat(sprintf("Loaded %d excluded packages from %s\n", nrow(excluded_pkgs), excluded_path)) + cat(sprintf( + "Loaded %d excluded packages from %s\n", + nrow(excluded_pkgs), + excluded_path + )) missing_dt <- missing_dt[!Package %in% excluded_pkgs$package] - cat(sprintf("%d packages remain after removing exclusions\n", nrow(missing_dt))) + cat(sprintf( + "%d packages remain after removing exclusions\n", + nrow(missing_dt) + )) } else { - cat(sprintf("No excluded-packages.json found at %s -- skipping exclusion step\n", excluded_path)) + cat(sprintf( + "No excluded-packages.json found at %s -- skipping exclusion step\n", + excluded_path + )) } # --------------------------------------------------------------------------- @@ -157,24 +201,34 @@ if (file.exists(excluded_path)) { # --------------------------------------------------------------------------- if (nrow(errored_dt) > 0) { known_failures_dt <- missing_dt[errored_dt, nomatch = 0] - rebuildable_dt <- missing_dt[!errored_dt] + rebuildable_dt <- missing_dt[!errored_dt] } else { known_failures_dt <- missing_dt[0] - rebuildable_dt <- missing_dt + rebuildable_dt <- missing_dt } -cat(sprintf("Rebuildable: %d | Known failures: %d\n", - nrow(rebuildable_dt), nrow(known_failures_dt))) +cat(sprintf( + "Rebuildable: %d | Known failures: %d\n", + nrow(rebuildable_dt), + nrow(known_failures_dt) +)) # --------------------------------------------------------------------------- # 7. Write rebuildable package names to cache RDS # --------------------------------------------------------------------------- -cache_dir <- "/mnt/cache/packages" -cache_file <- file.path(cache_dir, sprintf("weekly_rebuild_%s_%s.rds", platform, arch)) +cache_dir <- "/mnt/cache/packages" +cache_file <- file.path( + cache_dir, + sprintf("weekly_rebuild_%s_%s.rds", platform, arch) +) if (dir.exists(cache_dir)) { saveRDS(rebuildable_dt$Package, cache_file) - cat(sprintf("Wrote %d rebuildable packages to %s\n", nrow(rebuildable_dt), cache_file)) + cat(sprintf( + "Wrote %d rebuildable packages to %s\n", + nrow(rebuildable_dt), + cache_file + )) } else { cat(sprintf("Cache dir %s does not exist -- skipping RDS write\n", cache_dir)) } @@ -186,29 +240,47 @@ forgejo_token <- Sys.getenv("FORGEJO_TOKEN") if (nchar(forgejo_token) == 0) { cat("FORGEJO_TOKEN not set -- skipping issue update\n") } else { - issue_title <- sprintf("Missing package binaries for latest version (%s)", os_family) + issue_title <- sprintf( + "Missing package binaries for latest version (%s)", + os_family + ) - n_missing <- nrow(missing_dt) - n_rebuild <- nrow(rebuildable_dt) + n_missing <- nrow(missing_dt) + n_rebuild <- nrow(rebuildable_dt) - arch_lines <- sprintf("### %s (%d missing, %d to rebuild)", arch, n_missing, n_rebuild) + arch_lines <- sprintf( + "### %s (%d missing, %d to rebuild)", + arch, + n_missing, + n_rebuild + ) if (nrow(rebuildable_dt) > 0) { - arch_lines <- c(arch_lines, + arch_lines <- c( + arch_lines, paste0("- ", rebuildable_dt$Package, " (", rebuildable_dt$Version, ")") ) } else { arch_lines <- c(arch_lines, "_None_") } if (nrow(known_failures_dt) > 0) { - arch_lines <- c(arch_lines, + arch_lines <- c( + arch_lines, "", "#### Known build failures", - paste0("- ", known_failures_dt$Package, " (", known_failures_dt$Version, ")") + paste0( + "- ", + known_failures_dt$Package, + " (", + known_failures_dt$Version, + ")" + ) ) } build_excluded_footer <- function() { - if (nrow(excluded_pkgs) == 0) return(character(0)) + if (nrow(excluded_pkgs) == 0) { + return(character(0)) + } entries <- paste( paste0(excluded_pkgs$package, " (", excluded_pkgs$reason, ")"), collapse = ", " @@ -217,11 +289,12 @@ if (nchar(forgejo_token) == 0) { } forgejo_base <- "https://git.devxy.io/api/v1" - repo <- "devxy/build-cran-binaries" + repo <- "devxy/build-cran-binaries" search_url <- sprintf( "%s/repos/%s/issues?type=issues&state=open&q=%s&limit=50", - forgejo_base, repo, + forgejo_base, + repo, utils::URLencode(issue_title, reserved = TRUE) ) search_resp <- httr2::request(search_url) |> @@ -230,15 +303,19 @@ if (nchar(forgejo_token) == 0) { existing_issues <- httr2::resp_body_json(search_resp, simplifyVector = FALSE) - match_idx <- which(sapply(existing_issues, function(x) x$title) == issue_title) + match_idx <- which( + sapply(existing_issues, function(x) x$title) == issue_title + ) today_str <- format(Sys.Date(), "%Y-%m-%d") if (length(match_idx) > 0) { # ---- Update existing issue ---- issue_number <- existing_issues[[match_idx[1]]]$number - old_body <- existing_issues[[match_idx[1]]]$body - if (is.null(old_body)) old_body <- "" + old_body <- existing_issues[[match_idx[1]]]$body + if (is.null(old_body)) { + old_body <- "" + } lines <- strsplit(old_body, "\n", fixed = TRUE)[[1]] @@ -258,7 +335,11 @@ if (nchar(forgejo_token) == 0) { if (length(plat_idx) == 0) { # Platform section missing -- insert before --- footer footer_idx <- which(lines == "---") - insert_at <- if (length(footer_idx) > 0) footer_idx[length(footer_idx)] else length(lines) + 1 + insert_at <- if (length(footer_idx) > 0) { + footer_idx[length(footer_idx)] + } else { + length(lines) + 1 + } new_plat_block <- c(sprintf("## %s", platform), "", arch_lines, "") lines <- c( @@ -271,10 +352,14 @@ if (nchar(forgejo_token) == 0) { # End of platform section: next ## or --- at a higher level, or EOF next_section <- which(grepl("^## |^---", lines) & seq_along(lines) > pi) - plat_end <- if (length(next_section) > 0) next_section[1] - 1 else length(lines) + plat_end <- if (length(next_section) > 0) { + next_section[1] - 1 + } else { + length(lines) + } - plat_lines <- lines[seq(pi, plat_end)] - arch_local_idx <- which(plat_lines == arch_header) + plat_lines <- lines[seq(pi, plat_end)] + arch_local_idx <- which(plat_lines == arch_header) if (length(arch_local_idx) == 0) { # Append arch subsection at end of platform block @@ -285,11 +370,13 @@ if (nchar(forgejo_token) == 0) { lines[seq(plat_end + 1, length(lines))] ) } else { - ai <- pi + arch_local_idx[1] - 1 # absolute line index + ai <- pi + arch_local_idx[1] - 1 # absolute line index # End of arch subsection - next_arch <- which(grepl("^### |^## |^---", lines) & seq_along(lines) > ai) - arch_end <- if (length(next_arch) > 0) next_arch[1] - 1 else plat_end + next_arch <- which( + grepl("^### |^## |^---", lines) & seq_along(lines) > ai + ) + arch_end <- if (length(next_arch) > 0) next_arch[1] - 1 else plat_end lines <- c( lines[seq_len(ai - 1)], @@ -302,8 +389,12 @@ if (nchar(forgejo_token) == 0) { # Rebuild excluded footer excl_hdr_idx <- which(lines == "## Excluded packages") if (length(excl_hdr_idx) > 0) { - pre_dash <- which(lines == "---" & seq_along(lines) < excl_hdr_idx[1]) - remove_from <- if (length(pre_dash) > 0) pre_dash[length(pre_dash)] else excl_hdr_idx[1] + pre_dash <- which(lines == "---" & seq_along(lines) < excl_hdr_idx[1]) + remove_from <- if (length(pre_dash) > 0) { + pre_dash[length(pre_dash)] + } else { + excl_hdr_idx[1] + } lines <- lines[seq_len(remove_from - 1)] } footer <- build_excluded_footer() @@ -313,10 +404,15 @@ if (nchar(forgejo_token) == 0) { new_body <- paste(lines, collapse = "\n") - patch_url <- sprintf("%s/repos/%s/issues/%d", forgejo_base, repo, issue_number) + patch_url <- sprintf( + "%s/repos/%s/issues/%d", + forgejo_base, + repo, + issue_number + ) httr2::request(patch_url) |> httr2::req_headers( - Authorization = paste("token", forgejo_token), + Authorization = paste("token", forgejo_token), `Content-Type` = "application/json" ) |> httr2::req_body_json(list(body = new_body)) |> @@ -324,7 +420,6 @@ if (nchar(forgejo_token) == 0) { httr2::req_perform() cat(sprintf("Updated Forgejo issue #%d: %s\n", issue_number, issue_title)) - } else { # ---- Create new issue ---- body_lines <- c( @@ -342,17 +437,21 @@ if (nchar(forgejo_token) == 0) { post_url <- sprintf("%s/repos/%s/issues", forgejo_base, repo) create_resp <- httr2::request(post_url) |> httr2::req_headers( - Authorization = paste("token", forgejo_token), + Authorization = paste("token", forgejo_token), `Content-Type` = "application/json" ) |> httr2::req_body_json(list( title = issue_title, - body = paste(body_lines, collapse = "\n") + body = paste(body_lines, collapse = "\n") )) |> httr2::req_perform() new_issue <- httr2::resp_body_json(create_resp) - cat(sprintf("Created Forgejo issue #%d: %s\n", new_issue$number, issue_title)) + cat(sprintf( + "Created Forgejo issue #%d: %s\n", + new_issue$number, + issue_title + )) } } diff --git a/renovate.json b/renovate.json index 97ca6ad..53c7306 100644 --- a/renovate.json +++ b/renovate.json @@ -1,11 +1,7 @@ { "$schema": "https://docs.renovatebot.com/renovate-schema.json", - "extends": [ - "local>devxy/renovate-config" - ], - "ignorePaths": [ - "docker/**" - ], + "extends": ["local>devxy/renovate-config"], + "ignorePaths": ["docker/**"], "customManagers": [ { "customType": "regex", -- 2.54.0 From 90dbcc224f9c5e27cc0ef79ee3062b47fea2b295 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 21:40:09 +0200 Subject: [PATCH 10/14] feat(full): sensitive-only multi-R passes and per-minor index upload --- .crow/build-all-versions-amd64.yaml | 53 +++++++++++++++++++++++++++++ .crow/build-all-versions-arm64.yaml | 53 +++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+) diff --git a/.crow/build-all-versions-amd64.yaml b/.crow/build-all-versions-amd64.yaml index 30def98..7a400c9 100644 --- a/.crow/build-all-versions-amd64.yaml +++ b/.crow/build-all-versions-amd64.yaml @@ -70,6 +70,15 @@ steps: # pkgs = readRDS('/mnt/cache/pkgs_amd64.rds'); - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive pass under R $RV ===" + R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$(dirname "$RBIN")/Rscript" local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true + done # archive missed packages - /opt/R/$R_VERSION/bin/R -q -e "bincraft::process_unarchived_pkgs(paste(Sys.getenv('OS'), Sys.getenv('OS_VERSION')), Sys.getenv('ARCH'), workers = $NCPUS)" backend_options: @@ -97,3 +106,47 @@ steps: operator: 'Equal' value: 'true' effect: 'NoSchedule' + - name: 'Upload package indexes' + image: 'reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}' + pull: true + environment: + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + R_LIBS_USER: /mnt/cache/R-pkgs + volumes: + - ${ARCH}-binaries-r-dep-cache-${OS}-${OS_VERSION//./}:/mnt/cache + commands: + - | + CODENAME=$(/opt/R/$R_VERSION/bin/Rscript -e "cat(bincraft::set_codename(NULL))") + /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done + backend_options: + docker: + resources: + requests: + memory: 5Gi + cpu: 1000m + limits: + memory: 20Gi + cpu: 2000m + kubernetes: + resources: + requests: + memory: 5Gi + cpu: 1000m + limits: + memory: 20Gi + cpu: 2000m + nodeSelector: + kubernetes.io/arch: ${ARCH} + node.kubernetes.io/instance-type: ${K8S_INSTANCE_TYPE} + tolerations: + - key: 'CI' + operator: 'Equal' + value: 'true' + effect: 'NoSchedule' diff --git a/.crow/build-all-versions-arm64.yaml b/.crow/build-all-versions-arm64.yaml index c3a2977..aa92de2 100644 --- a/.crow/build-all-versions-arm64.yaml +++ b/.crow/build-all-versions-arm64.yaml @@ -83,6 +83,15 @@ steps: - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- /opt/R/$R_VERSION/bin/Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive pass under R $RV ===" + R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$(dirname "$RBIN")/Rscript" local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true + done # archive missed packages - /opt/R/$R_VERSION/bin/R -q -e "bincraft::process_unarchived_pkgs(paste(Sys.getenv('OS'), Sys.getenv('OS_VERSION')), Sys.getenv('ARCH'), workers = $NCPUS)" backend_options: @@ -110,3 +119,47 @@ steps: operator: 'Equal' value: 'true' effect: 'NoSchedule' + - name: 'Upload package indexes' + image: 'reg.devxy.io/rpkgs/build-env-${OS}:${OS_VERSION}' + pull: true + environment: + B2_S3_ACCESS_KEY: + from_secret: B2_S3_ACCESS_KEY + B2_S3_SECRET_KEY: + from_secret: B2_S3_SECRET_KEY + R_LIBS_USER: /mnt/cache/R-pkgs + volumes: + - ${ARCH}-binaries-r-dep-cache-${OS}-${OS_VERSION//./}:/mnt/cache + commands: + - | + CODENAME=$(/opt/R/$R_VERSION/bin/Rscript -e "cat(bincraft::set_codename(NULL))") + /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done + backend_options: + docker: + resources: + requests: + memory: 5Gi + cpu: 1000m + limits: + memory: 20Gi + cpu: 2000m + kubernetes: + resources: + requests: + memory: 5Gi + cpu: 1000m + limits: + memory: 20Gi + cpu: 2000m + nodeSelector: + kubernetes.io/arch: ${ARCH} + node.kubernetes.io/instance-type: ${K8S_INSTANCE_TYPE} + tolerations: + - key: 'CI' + operator: 'Equal' + value: 'true' + effect: 'NoSchedule' -- 2.54.0 From 771d642531ee4c9fca91deec874b088de1dc8521 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 21:43:16 +0200 Subject: [PATCH 11/14] feat(updates): classifier-driven r-minor builds + per-minor index across platforms --- .crow/process-updates-alpine-322-amd64.yaml | 16 +++++++++++++++- .crow/process-updates-alpine-322-arm64.yaml | 16 +++++++++++++++- .crow/process-updates-alpine-323-amd64.yaml | 16 +++++++++++++++- .crow/process-updates-alpine-323-arm64.yaml | 16 +++++++++++++++- .crow/process-updates-redhat-10-amd64.yaml | 16 +++++++++++++++- .crow/process-updates-redhat-10-arm64.yaml | 16 +++++++++++++++- .crow/process-updates-redhat-8-amd64.yaml | 16 +++++++++++++++- .crow/process-updates-redhat-8-arm64.yaml | 16 +++++++++++++++- .crow/process-updates-redhat-9-amd64.yaml | 16 +++++++++++++++- .crow/process-updates-redhat-9-arm64.yaml | 16 +++++++++++++++- .crow/process-updates-ubuntu-2204-amd64.yaml | 16 +++++++++++++++- .crow/process-updates-ubuntu-2204-arm64.yaml | 16 +++++++++++++++- .crow/process-updates-ubuntu-2404-amd64.yaml | 16 +++++++++++++++- .crow/process-updates-ubuntu-2404-arm64.yaml | 16 +++++++++++++++- 14 files changed, 210 insertions(+), 14 deletions(-) diff --git a/.crow/process-updates-alpine-322-amd64.yaml b/.crow/process-updates-alpine-322-amd64.yaml index 1927d57..5b1836e 100644 --- a/.crow/process-updates-alpine-322-amd64.yaml +++ b/.crow/process-updates-alpine-322-amd64.yaml @@ -56,8 +56,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-322', process_updated = TRUE, process_new = FALSE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-322', process_updated = TRUE, process_new = FALSE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-322', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "alpine322", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'alpine322', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done backend_options: kubernetes: resources: diff --git a/.crow/process-updates-alpine-322-arm64.yaml b/.crow/process-updates-alpine-322-arm64.yaml index c89a22f..48037e9 100644 --- a/.crow/process-updates-alpine-322-arm64.yaml +++ b/.crow/process-updates-alpine-322-arm64.yaml @@ -55,8 +55,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-322', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-322', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-322', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "alpine322", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'alpine322', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done - name: Purge CDN cache image: reg.devxy.io/docker.io/library/alpine:3.24 diff --git a/.crow/process-updates-alpine-323-amd64.yaml b/.crow/process-updates-alpine-323-amd64.yaml index 3a73df5..0262eda 100644 --- a/.crow/process-updates-alpine-323-amd64.yaml +++ b/.crow/process-updates-alpine-323-amd64.yaml @@ -56,8 +56,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-323', process_updated = TRUE, process_new = FALSE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-323', process_updated = TRUE, process_new = FALSE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-323', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "alpine323", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'alpine323', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done backend_options: kubernetes: resources: diff --git a/.crow/process-updates-alpine-323-arm64.yaml b/.crow/process-updates-alpine-323-arm64.yaml index 2401859..ceb01be 100644 --- a/.crow/process-updates-alpine-323-arm64.yaml +++ b/.crow/process-updates-alpine-323-arm64.yaml @@ -55,8 +55,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-323', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-323', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'alpine-323', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "alpine323", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'alpine323', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done - name: Purge CDN cache image: reg.devxy.io/docker.io/library/alpine:3.24 diff --git a/.crow/process-updates-redhat-10-amd64.yaml b/.crow/process-updates-redhat-10-amd64.yaml index a4f0cd1..51267d8 100644 --- a/.crow/process-updates-redhat-10-amd64.yaml +++ b/.crow/process-updates-redhat-10-amd64.yaml @@ -58,8 +58,22 @@ steps: - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - - $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-10', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-10', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-10', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "rhel10", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'rhel10', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done backend_options: kubernetes: resources: diff --git a/.crow/process-updates-redhat-10-arm64.yaml b/.crow/process-updates-redhat-10-arm64.yaml index a786070..9818a86 100644 --- a/.crow/process-updates-redhat-10-arm64.yaml +++ b/.crow/process-updates-redhat-10-arm64.yaml @@ -56,8 +56,22 @@ steps: - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - - $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-10', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - $XVFB $XVFB_ARGS -- /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-10', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-10', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "rhel10", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'rhel10', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done - name: Purge CDN cache image: reg.devxy.io/docker.io/library/alpine:3.24 diff --git a/.crow/process-updates-redhat-8-amd64.yaml b/.crow/process-updates-redhat-8-amd64.yaml index 09e5d47..9621c70 100644 --- a/.crow/process-updates-redhat-8-amd64.yaml +++ b/.crow/process-updates-redhat-8-amd64.yaml @@ -56,8 +56,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-8', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-8', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-8', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "rhel8", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'rhel8', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done backend_options: kubernetes: resources: diff --git a/.crow/process-updates-redhat-8-arm64.yaml b/.crow/process-updates-redhat-8-arm64.yaml index aa51437..240ca28 100644 --- a/.crow/process-updates-redhat-8-arm64.yaml +++ b/.crow/process-updates-redhat-8-arm64.yaml @@ -55,8 +55,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-8', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-8', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-8', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "rhel8", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'rhel8', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done - name: Purge CDN cache image: reg.devxy.io/docker.io/library/alpine:3.24 diff --git a/.crow/process-updates-redhat-9-amd64.yaml b/.crow/process-updates-redhat-9-amd64.yaml index bf2b54e..1023a31 100644 --- a/.crow/process-updates-redhat-9-amd64.yaml +++ b/.crow/process-updates-redhat-9-amd64.yaml @@ -57,8 +57,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-9', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-9', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-9', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "rhel9", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'rhel9', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done backend_options: kubernetes: resources: diff --git a/.crow/process-updates-redhat-9-arm64.yaml b/.crow/process-updates-redhat-9-arm64.yaml index 0356998..8b21ac6 100644 --- a/.crow/process-updates-redhat-9-arm64.yaml +++ b/.crow/process-updates-redhat-9-arm64.yaml @@ -55,8 +55,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-9', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-9', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'redhat-9', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "rhel9", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'rhel9', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done - name: Purge CDN cache image: reg.devxy.io/docker.io/library/alpine:3.24 diff --git a/.crow/process-updates-ubuntu-2204-amd64.yaml b/.crow/process-updates-ubuntu-2204-amd64.yaml index 8940c70..2a578c9 100644 --- a/.crow/process-updates-ubuntu-2204-amd64.yaml +++ b/.crow/process-updates-ubuntu-2204-amd64.yaml @@ -57,8 +57,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2204', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2204', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2204', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "jammy", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'jammy', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done backend_options: kubernetes: resources: diff --git a/.crow/process-updates-ubuntu-2204-arm64.yaml b/.crow/process-updates-ubuntu-2204-arm64.yaml index 2757c68..dc95720 100644 --- a/.crow/process-updates-ubuntu-2204-arm64.yaml +++ b/.crow/process-updates-ubuntu-2204-arm64.yaml @@ -55,8 +55,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2204', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2204', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2204', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "jammy", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'jammy', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done - name: Purge CDN cache image: reg.devxy.io/docker.io/library/alpine:3.24 diff --git a/.crow/process-updates-ubuntu-2404-amd64.yaml b/.crow/process-updates-ubuntu-2404-amd64.yaml index 3dd25c9..d9d9c0d 100644 --- a/.crow/process-updates-ubuntu-2404-amd64.yaml +++ b/.crow/process-updates-ubuntu-2404-amd64.yaml @@ -56,8 +56,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2404', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2404', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2404', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "noble", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'noble', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done backend_options: kubernetes: resources: diff --git a/.crow/process-updates-ubuntu-2404-arm64.yaml b/.crow/process-updates-ubuntu-2404-arm64.yaml index 69027ae..db49476 100644 --- a/.crow/process-updates-ubuntu-2404-arm64.yaml +++ b/.crow/process-updates-ubuntu-2404-arm64.yaml @@ -55,8 +55,22 @@ steps: - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise - - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2404', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - xvfb-run /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = 'error', repos = structure(c(getOption('repos'),INLA='https://inla.r-inla-download.org/R/stable'))); progressr::handlers('cli'); progressr::handlers(global = TRUE); options(future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2404', process_updated = TRUE, process_new = TRUE, process_removed = TRUE, r_minor_detection = 'classifier', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" + - | + PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) + for RBIN in /opt/R/*/bin/R; do + RV=$(basename "$(dirname "$(dirname "$RBIN")")") + RMINOR=$(echo "$RV" | cut -d. -f1-2) + [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue + echo "=== R-minor-sensitive update pass under R $RV ===" + xvfb-run "$(dirname "$RBIN")/R" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = 'ubuntu-2404', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true + done - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "noble", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' + - | + for RBIN in /opt/R/*/bin/R; do + RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) + "$(dirname "$RBIN")/R" -q -e "library(bincraft); upload_package_index(codename = 'noble', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true + done - name: Purge CDN cache image: reg.devxy.io/docker.io/library/alpine:3.24 -- 2.54.0 From 5bade188c2937811ed9fcca80550223e389e04a4 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 21:45:18 +0200 Subject: [PATCH 12/14] chore: pin bincraft 4.2.0 and drop superseded standalone r-minor workflow Bumps the bincraft git-tag pin and version guard from v4.1.1 to v4.2.0 across all .crow workflows, removes .crow/build-r-minor-sensitive-packages.yaml (now superseded by the integrated full + iterative flow), and drops its now-orphaned hook-exclude entries. --- .crow/archive-missed-packages.yaml | 2 +- .crow/build-r-minor-sensitive-packages.yaml | 80 ------------------- .crow/process-updates-alpine-322-amd64.yaml | 2 +- .crow/process-updates-alpine-322-arm64.yaml | 2 +- .crow/process-updates-alpine-323-amd64.yaml | 2 +- .crow/process-updates-alpine-323-arm64.yaml | 2 +- .crow/process-updates-redhat-10-amd64.yaml | 2 +- .crow/process-updates-redhat-10-arm64.yaml | 2 +- .crow/process-updates-redhat-8-amd64.yaml | 2 +- .crow/process-updates-redhat-8-arm64.yaml | 2 +- .crow/process-updates-redhat-9-amd64.yaml | 2 +- .crow/process-updates-redhat-9-arm64.yaml | 2 +- .crow/process-updates-ubuntu-2204-amd64.yaml | 2 +- .crow/process-updates-ubuntu-2204-arm64.yaml | 2 +- .crow/process-updates-ubuntu-2404-amd64.yaml | 2 +- .crow/process-updates-ubuntu-2404-arm64.yaml | 2 +- ...ekly-rebuild-missing-alpine-322-amd64.yaml | 2 +- ...ekly-rebuild-missing-alpine-322-arm64.yaml | 2 +- ...ekly-rebuild-missing-alpine-323-amd64.yaml | 2 +- ...ekly-rebuild-missing-alpine-323-arm64.yaml | 2 +- ...eekly-rebuild-missing-redhat-10-amd64.yaml | 2 +- ...eekly-rebuild-missing-redhat-10-arm64.yaml | 2 +- ...weekly-rebuild-missing-redhat-8-amd64.yaml | 2 +- ...weekly-rebuild-missing-redhat-8-arm64.yaml | 2 +- ...weekly-rebuild-missing-redhat-9-amd64.yaml | 2 +- ...weekly-rebuild-missing-redhat-9-arm64.yaml | 2 +- ...kly-rebuild-missing-ubuntu-2204-amd64.yaml | 2 +- ...kly-rebuild-missing-ubuntu-2204-arm64.yaml | 2 +- ...kly-rebuild-missing-ubuntu-2404-amd64.yaml | 2 +- ...kly-rebuild-missing-ubuntu-2404-arm64.yaml | 2 +- .editorconfig-checker.json | 7 +- .pre-commit-config.yaml | 7 +- .prettierignore | 1 - 33 files changed, 33 insertions(+), 120 deletions(-) delete mode 100644 .crow/build-r-minor-sensitive-packages.yaml diff --git a/.crow/archive-missed-packages.yaml b/.crow/archive-missed-packages.yaml index 88c199e..cb4de27 100644 --- a/.crow/archive-missed-packages.yaml +++ b/.crow/archive-missed-packages.yaml @@ -62,7 +62,7 @@ steps: GIT_USER: pat-s R_VERSION: 4.5.3 commands: - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1", dependencies = TRUE)' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0", dependencies = TRUE)' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - /opt/R/$R_VERSION/bin/R -q -e 'bincraft::process_unarchived_pkgs(Sys.getenv("CODENAME"), Sys.getenv("ARCH"), workers = 2L)' backend_options: diff --git a/.crow/build-r-minor-sensitive-packages.yaml b/.crow/build-r-minor-sensitive-packages.yaml deleted file mode 100644 index a0c3c2d..0000000 --- a/.crow/build-r-minor-sensitive-packages.yaml +++ /dev/null @@ -1,80 +0,0 @@ -when: - - event: manual - evaluate: 'build == "r-minor"' -# - event: manual -# evaluate: 'build == "alpine-322-arm64"' -# - event: manual -# evaluate: 'build == "alpine-322-arm64-1"' -# - event: manual -# evaluate: 'build == "all-arm64"' - -# skip_clone: true - -# depends_on: -# - alpine-322-arm64-install-deps - -matrix: - include: - - os: alpine - os_version: 3.21 - r_version: 4.5.3 - - os: alpine - os_version: 3.21 - r_version: 4.4.3 - # - os: alpine - # os_version: 3.21 - # r_version: 4.3 -steps: - - name: 'Build R-minor-sensitive binaries' - image: "docker.io/devxygmbh/rpkgs-build-env-${os}:${os_version}" - pull: true - environment: - RED_HAT_DEV_PW: - from_secret: RED_HAT_DEV_PW - B2_S3_ACCESS_KEY: - from_secret: B2_S3_ACCESS_KEY - B2_S3_SECRET_KEY: - from_secret: B2_S3_SECRET_KEY - PGPASS: - from_secret: PGPASS - REPO_RO_TOKEN: - from_secret: REPO_RO_TOKEN - GITHUB_PAT: - from_secret: GITHUB_PAT - # normal env vars - GIT_USER: pat-s - NTFY_SERVER: - from_secret: NTFY_SERVER - NTFY_TOPIC: - from_secret: NTFY_TOPIC - NTFY_AUTH: TRUE - NTFY_PASSWORD: - from_secret: ntfy_token - # set the location of the 'pkgcache' cache dir which persists the R package dependencies needed to install the packages themselves - R_PKG_CACHE_DIR: /mnt/cache/pkgcache - R_LIBS_USER: /tmp/R-libs:/mnt/cache/R-pkgs - CCACHE_DIR: /mnt/cache/ccache - NCPUS: 2 - STRATEGY: sequential - commands: - - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - mkdir -p /tmp/R-libs - - git clone -q https://codefloe.com/rpkgs/bincraft.git /tmp/bincraft && /opt/R/$R_VERSION/bin/R CMD INSTALL --library=/tmp/R-libs /tmp/bincraft && /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run) - - $XVFB -- /opt/R/$R_VERSION/bin/R -q -e "options(crayon.enabled = TRUE, Ncpus = $NCPUS, future.globals.onReference = NULL); pak::sysreqs_db_update(); pkgs = bincraft::get_r_minor_sensitive_packages(r_minor_packages_issue_url = 'https://git.devxy.io/api/v1/repos/devxy/build-cran-binaries/issues/29'); foo = lapply(pkgs, function(x) bincraft::build_binary_package(x, is_debug = FALSE, is_r_minor_sensitive = TRUE, force = FALSE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE))" 2>&1 - backend_options: - kubernetes: - resources: - requests: - memory: 5Gi - cpu: 1000m - limits: - memory: 20Gi - cpu: 2000m - tolerations: - - key: 'CI' - operator: 'Equal' - value: 'true' - effect: 'NoSchedule' diff --git a/.crow/process-updates-alpine-322-amd64.yaml b/.crow/process-updates-alpine-322-amd64.yaml index 5b1836e..2e00d4c 100644 --- a/.crow/process-updates-alpine-322-amd64.yaml +++ b/.crow/process-updates-alpine-322-amd64.yaml @@ -52,7 +52,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-alpine-322-arm64.yaml b/.crow/process-updates-alpine-322-arm64.yaml index 48037e9..0b56996 100644 --- a/.crow/process-updates-alpine-322-arm64.yaml +++ b/.crow/process-updates-alpine-322-arm64.yaml @@ -51,7 +51,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-alpine-323-amd64.yaml b/.crow/process-updates-alpine-323-amd64.yaml index 0262eda..4b19037 100644 --- a/.crow/process-updates-alpine-323-amd64.yaml +++ b/.crow/process-updates-alpine-323-amd64.yaml @@ -52,7 +52,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-alpine-323-arm64.yaml b/.crow/process-updates-alpine-323-arm64.yaml index ceb01be..3e20a90 100644 --- a/.crow/process-updates-alpine-323-arm64.yaml +++ b/.crow/process-updates-alpine-323-arm64.yaml @@ -51,7 +51,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-redhat-10-amd64.yaml b/.crow/process-updates-redhat-10-amd64.yaml index 51267d8..81075d0 100644 --- a/.crow/process-updates-redhat-10-amd64.yaml +++ b/.crow/process-updates-redhat-10-amd64.yaml @@ -53,7 +53,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-redhat-10-arm64.yaml b/.crow/process-updates-redhat-10-arm64.yaml index 9818a86..a2b7e06 100644 --- a/.crow/process-updates-redhat-10-arm64.yaml +++ b/.crow/process-updates-redhat-10-arm64.yaml @@ -51,7 +51,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-redhat-8-amd64.yaml b/.crow/process-updates-redhat-8-amd64.yaml index 9621c70..f148d44 100644 --- a/.crow/process-updates-redhat-8-amd64.yaml +++ b/.crow/process-updates-redhat-8-amd64.yaml @@ -52,7 +52,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-redhat-8-arm64.yaml b/.crow/process-updates-redhat-8-arm64.yaml index 240ca28..ee3f79b 100644 --- a/.crow/process-updates-redhat-8-arm64.yaml +++ b/.crow/process-updates-redhat-8-arm64.yaml @@ -51,7 +51,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-redhat-9-amd64.yaml b/.crow/process-updates-redhat-9-amd64.yaml index 1023a31..063f978 100644 --- a/.crow/process-updates-redhat-9-amd64.yaml +++ b/.crow/process-updates-redhat-9-amd64.yaml @@ -53,7 +53,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-redhat-9-arm64.yaml b/.crow/process-updates-redhat-9-arm64.yaml index 8b21ac6..327a22a 100644 --- a/.crow/process-updates-redhat-9-arm64.yaml +++ b/.crow/process-updates-redhat-9-arm64.yaml @@ -51,7 +51,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-ubuntu-2204-amd64.yaml b/.crow/process-updates-ubuntu-2204-amd64.yaml index 2a578c9..7d00e9e 100644 --- a/.crow/process-updates-ubuntu-2204-amd64.yaml +++ b/.crow/process-updates-ubuntu-2204-amd64.yaml @@ -53,7 +53,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-ubuntu-2204-arm64.yaml b/.crow/process-updates-ubuntu-2204-arm64.yaml index dc95720..7728e63 100644 --- a/.crow/process-updates-ubuntu-2204-arm64.yaml +++ b/.crow/process-updates-ubuntu-2204-arm64.yaml @@ -51,7 +51,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-ubuntu-2404-amd64.yaml b/.crow/process-updates-ubuntu-2404-amd64.yaml index d9d9c0d..58f5be5 100644 --- a/.crow/process-updates-ubuntu-2404-amd64.yaml +++ b/.crow/process-updates-ubuntu-2404-amd64.yaml @@ -52,7 +52,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/process-updates-ubuntu-2404-arm64.yaml b/.crow/process-updates-ubuntu-2404-arm64.yaml index db49476..5da7620 100644 --- a/.crow/process-updates-ubuntu-2404-arm64.yaml +++ b/.crow/process-updates-ubuntu-2404-arm64.yaml @@ -51,7 +51,7 @@ steps: commands: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - rm -rf /mnt/cache/R-pkgs/00LOCK-r-pkg-binaries /mnt/cache/R-pkgs/00LOCK-bincraft /mnt/cache/R-pkgs/00LOCK-pak /mnt/cache/pkgcache/R/pkgcache /mnt/cache/pkgcache/R/pkgcache /mnt/cache/R-pkgs/pkgcache /mnt/cache/R-pkgs/00LOCK-pak/mnt/cache/R-pkgs/00LOCK-RPostgres /mnt/cache/R-pkgs/bincraft - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages # options(future.globals.onReference = NULL): for some reason s3fs::file_delete() throws 'Error: Detected a non-exportable reference ('externalptr') in one of the globals ('FUN' of class 'function') used in the future expression' otherwise diff --git a/.crow/weekly-rebuild-missing-alpine-322-amd64.yaml b/.crow/weekly-rebuild-missing-alpine-322-amd64.yaml index ba7735c..e7b3904 100644 --- a/.crow/weekly-rebuild-missing-alpine-322-amd64.yaml +++ b/.crow/weekly-rebuild-missing-alpine-322-amd64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-alpine-322-arm64.yaml b/.crow/weekly-rebuild-missing-alpine-322-arm64.yaml index 076b267..26624c4 100644 --- a/.crow/weekly-rebuild-missing-alpine-322-arm64.yaml +++ b/.crow/weekly-rebuild-missing-alpine-322-arm64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-alpine-323-amd64.yaml b/.crow/weekly-rebuild-missing-alpine-323-amd64.yaml index e6f95c2..2b05036 100644 --- a/.crow/weekly-rebuild-missing-alpine-323-amd64.yaml +++ b/.crow/weekly-rebuild-missing-alpine-323-amd64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-alpine-323-arm64.yaml b/.crow/weekly-rebuild-missing-alpine-323-arm64.yaml index 5c1e732..8c87a02 100644 --- a/.crow/weekly-rebuild-missing-alpine-323-arm64.yaml +++ b/.crow/weekly-rebuild-missing-alpine-323-arm64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-redhat-10-amd64.yaml b/.crow/weekly-rebuild-missing-redhat-10-amd64.yaml index 6a70915..87e21f0 100644 --- a/.crow/weekly-rebuild-missing-redhat-10-amd64.yaml +++ b/.crow/weekly-rebuild-missing-redhat-10-amd64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-redhat-10-arm64.yaml b/.crow/weekly-rebuild-missing-redhat-10-arm64.yaml index dae5001..2b2cf88 100644 --- a/.crow/weekly-rebuild-missing-redhat-10-arm64.yaml +++ b/.crow/weekly-rebuild-missing-redhat-10-arm64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-redhat-8-amd64.yaml b/.crow/weekly-rebuild-missing-redhat-8-amd64.yaml index 33326d2..4ef638e 100644 --- a/.crow/weekly-rebuild-missing-redhat-8-amd64.yaml +++ b/.crow/weekly-rebuild-missing-redhat-8-amd64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-redhat-8-arm64.yaml b/.crow/weekly-rebuild-missing-redhat-8-arm64.yaml index fee0bb3..d02673b 100644 --- a/.crow/weekly-rebuild-missing-redhat-8-arm64.yaml +++ b/.crow/weekly-rebuild-missing-redhat-8-arm64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-redhat-9-amd64.yaml b/.crow/weekly-rebuild-missing-redhat-9-amd64.yaml index b78dd37..c857116 100644 --- a/.crow/weekly-rebuild-missing-redhat-9-amd64.yaml +++ b/.crow/weekly-rebuild-missing-redhat-9-amd64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-redhat-9-arm64.yaml b/.crow/weekly-rebuild-missing-redhat-9-arm64.yaml index d339587..c7968ba 100644 --- a/.crow/weekly-rebuild-missing-redhat-9-arm64.yaml +++ b/.crow/weekly-rebuild-missing-redhat-9-arm64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml b/.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml index ee5c094..6a2d625 100644 --- a/.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml +++ b/.crow/weekly-rebuild-missing-ubuntu-2204-amd64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-ubuntu-2204-arm64.yaml b/.crow/weekly-rebuild-missing-ubuntu-2204-arm64.yaml index 40b3b71..d14c278 100644 --- a/.crow/weekly-rebuild-missing-ubuntu-2204-arm64.yaml +++ b/.crow/weekly-rebuild-missing-ubuntu-2204-arm64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-ubuntu-2404-amd64.yaml b/.crow/weekly-rebuild-missing-ubuntu-2404-amd64.yaml index 973b7c9..e9002f8 100644 --- a/.crow/weekly-rebuild-missing-ubuntu-2404-amd64.yaml +++ b/.crow/weekly-rebuild-missing-ubuntu-2404-amd64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.crow/weekly-rebuild-missing-ubuntu-2404-arm64.yaml b/.crow/weekly-rebuild-missing-ubuntu-2404-arm64.yaml index eaf0a94..1f17c45 100644 --- a/.crow/weekly-rebuild-missing-ubuntu-2404-arm64.yaml +++ b/.crow/weekly-rebuild-missing-ubuntu-2404-arm64.yaml @@ -39,7 +39,7 @@ steps: - git clone -q https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git . - mkdir -p /mnt/cache/pkgcache /mnt/cache/R-pkgs /mnt/cache/ccache /mnt/cache/packages - rm -rf /mnt/cache/R-pkgs/00LOCK-* - - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.1.1") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.1.1")' + - /opt/R/$R_VERSION/bin/R -q -e 'if (!requireNamespace("bincraft", quietly = TRUE) || packageVersion("bincraft") != "4.2.0") pak::pak("git::https://codefloe.com/rpkgs/bincraft.git@v4.2.0")' - /opt/R/$R_VERSION/bin/R -q -e 'packageVersion("bincraft")' - XVFB=$(command -v xwfb-run 2>/dev/null || command -v xvfb-run); XVFB_ARGS=""; if command -v xwfb-run >/dev/null 2>&1; then dnf install -y -q weston 2>/dev/null; XVFB_ARGS="-c weston"; fi - /opt/R/$R_VERSION/bin/R -q -e 'pak::pak("httr2")' diff --git a/.editorconfig-checker.json b/.editorconfig-checker.json index e5f1357..f7abef3 100644 --- a/.editorconfig-checker.json +++ b/.editorconfig-checker.json @@ -1,8 +1,3 @@ { - "Exclude": [ - "^LICENSE\\.md$", - "^benchmark/", - "^docker/reprex/", - "^\\.crow/build-r-minor-sensitive-packages\\.yaml$" - ] + "Exclude": ["^LICENSE\\.md$", "^benchmark/", "^docker/reprex/"] } diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 9d3a2e1..c7edc32 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,12 +1,11 @@ # cSpell:ignore autofix autoupdate -# Excluded: verbatim GPL license, auxiliary shell scripts with intentional -# in-string formatting, and the soon-to-be-removed standalone workflow. +# Excluded: verbatim GPL license and auxiliary shell scripts with intentional +# in-string formatting (reformatting would corrupt their output). exclude: | (?x)^( LICENSE\.md| benchmark/| - docker/reprex/| - \.crow/build-r-minor-sensitive-packages\.yaml + docker/reprex/ ) repos: - repo: https://github.com/pre-commit/pre-commit-hooks diff --git a/.prettierignore b/.prettierignore index 7830219..0e4b780 100644 --- a/.prettierignore +++ b/.prettierignore @@ -1,2 +1 @@ LICENSE.md -.crow/build-r-minor-sensitive-packages.yaml -- 2.54.0 From c7a8c5d9f8db69567bf007f92ac1253ec95c7240 Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 21:49:09 +0200 Subject: [PATCH 13/14] refactor(full): drop dead R_VERSION override in sensitive-only pass --- .crow/build-all-versions-amd64.yaml | 2 +- .crow/build-all-versions-arm64.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.crow/build-all-versions-amd64.yaml b/.crow/build-all-versions-amd64.yaml index 7a400c9..0c74ef7 100644 --- a/.crow/build-all-versions-amd64.yaml +++ b/.crow/build-all-versions-amd64.yaml @@ -77,7 +77,7 @@ steps: RMINOR=$(echo "$RV" | cut -d. -f1-2) [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue echo "=== R-minor-sensitive pass under R $RV ===" - R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$(dirname "$RBIN")/Rscript" local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true + $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$(dirname "$RBIN")/Rscript" local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true done # archive missed packages - /opt/R/$R_VERSION/bin/R -q -e "bincraft::process_unarchived_pkgs(paste(Sys.getenv('OS'), Sys.getenv('OS_VERSION')), Sys.getenv('ARCH'), workers = $NCPUS)" diff --git a/.crow/build-all-versions-arm64.yaml b/.crow/build-all-versions-arm64.yaml index aa92de2..aac08c3 100644 --- a/.crow/build-all-versions-arm64.yaml +++ b/.crow/build-all-versions-arm64.yaml @@ -90,7 +90,7 @@ steps: RMINOR=$(echo "$RV" | cut -d. -f1-2) [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue echo "=== R-minor-sensitive pass under R $RV ===" - R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$(dirname "$RBIN")/Rscript" local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true + $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$(dirname "$RBIN")/Rscript" local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true done # archive missed packages - /opt/R/$R_VERSION/bin/R -q -e "bincraft::process_unarchived_pkgs(paste(Sys.getenv('OS'), Sys.getenv('OS_VERSION')), Sys.getenv('ARCH'), workers = $NCPUS)" -- 2.54.0 From ab8f24e067642ae8b3d91ad44791627bca218eea Mon Sep 17 00:00:00 2001 From: pat-s Date: Sat, 13 Jun 2026 22:03:33 +0200 Subject: [PATCH 14/14] chore: drop superpowers spec/plans from PR (kept locally, gitignored) --- ...026-06-13-bincraft-r-minor-enhancements.md | 470 ----------------- .../2026-06-13-pipeline-r-minor-builds.md | 471 ------------------ ...6-06-13-r-minor-sensitive-builds-design.md | 155 ------ 3 files changed, 1096 deletions(-) delete mode 100644 docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md delete mode 100644 docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md delete mode 100644 docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md diff --git a/docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md b/docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md deleted file mode 100644 index b52735a..0000000 --- a/docs/superpowers/plans/2026-06-13-bincraft-r-minor-enhancements.md +++ /dev/null @@ -1,470 +0,0 @@ -# bincraft R-minor Enhancements Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Teach bincraft to (a) build/serve per-R-minor package indexes and (b) drive `process_cran_updates()` from the ABI classifier so only risky packages are built per R minor. - -**Architecture:** Two additive, low-risk changes to the bincraft R package. `upload_package_index()` gains an `r_minor` argument so a per-minor slot (`…/contrib//`) gets its own `PACKAGES*`. `process_cran_updates()` gains `r_minor_detection`/`r_minor_sensitive_only` and classifies each candidate via `needs_per_minor_recompile()` (already in the package), passing a concrete `is_r_minor_sensitive` logical to the unchanged `build_binary_package()`. - -**Tech Stack:** R package, `testthat` (3e) + `mockery` for tests, `roxygen2` for docs, `s3fs`/`cranlike` for the index. Repo: `https://codefloe.com/rpkgs/bincraft` (this plan is executed in a clone of that repo, NOT in build-cran-binaries). - -**Pre-req:** PR #49 (ABI classifier) is already merged on `main`; `abi_classify()`, `needs_per_minor_recompile()`, `abi_risky_linking_deps()` are exported. - ---- - -### Task 1: `package_index_remote_dir()` pure helper - -**Files:** -- Modify: `R/package_index.R` (top of file, before `add_to_package_index`) -- Test: `tests/testthat/test-package_index.R` (new) - -- [ ] **Step 1: Write the failing test** - -```r -# tests/testthat/test-package_index.R -test_that("package_index_remote_dir builds the generic slot when r_minor is NULL", { - expect_identical( - package_index_remote_dir("bucket", "amd64", "alpine323"), - file.path("bucket", "amd64", "alpine323", "latest", "src", "contrib") - ) -}) - -test_that("package_index_remote_dir appends the minor slot when r_minor is set", { - expect_identical( - package_index_remote_dir("bucket", "amd64", "alpine323", r_minor = "4.4"), - file.path("bucket", "amd64", "alpine323", "latest", "src", "contrib", "4.4") - ) -}) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-package_index.R")'` -Expected: FAIL — `could not find function "package_index_remote_dir"`. - -- [ ] **Step 3: Add the helper** - -```r -# R/package_index.R — add near the top, above add_to_package_index() -#' Build the S3 remote contrib dir for a package index -#' -#' @param r_minor Optional `"major.minor"` string (e.g. `"4.4"`). When non-NULL -#' the path points at the per-minor slot. -#' @keywords internal -package_index_remote_dir <- function(s3_bucket, arch, codename, r_minor = NULL) { - base <- file.path(s3_bucket, arch, codename, "latest", "src", "contrib") - if (is.null(r_minor)) { - base - } else { - file.path(base, r_minor) - } -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-package_index.R")'` -Expected: PASS (2/2). - -- [ ] **Step 5: Commit** - -```bash -git add R/package_index.R tests/testthat/test-package_index.R -git commit -m "feat(index): add package_index_remote_dir helper for per-minor slots" -``` - ---- - -### Task 2: per-minor support in `upload_package_index()` - -**Files:** -- Modify: `R/package_index.R` (`upload_package_index`, signature + the `remote_bin_dir` block at lines ~117-124; `add_to_package_index` for consistency) - -- [ ] **Step 1: Add `r_minor` to the signature and use the helper** - -In `upload_package_index()`, add `r_minor = NULL` to the argument list (after `arch = NULL`). Replace the inline `remote_bin_dir <- file.path(s3_bucket, arch, codename, "latest", "src", "contrib")` block with: - -```r - remote_bin_dir <- package_index_remote_dir(s3_bucket, arch, codename, r_minor) -``` - -Do the same replacement in `add_to_package_index()` (it has the identical inline construction); add `r_minor = NULL` to its signature too. - -- [ ] **Step 2: Add the roxygen param** - -Above `upload_package_index` and `add_to_package_index`, add: - -```r -#' @param r_minor Optional `"major.minor"` string. When set, the index is -#' written/read under the per-minor slot `…/contrib//` instead of the -#' generic `…/contrib/` slot. -``` - -- [ ] **Step 3: Regenerate docs** - -Run: `R -q -e 'devtools::document()'` -Expected: updated `man/upload_package_index.Rd`, `man/add_to_package_index.Rd`, no errors. - -- [ ] **Step 4: Verify package still loads and existing tests pass** - -Run: `R -q -e 'devtools::load_all("."); testthat::test_dir("tests/testthat")'` -Expected: PASS — no regressions; the new path test from Task 1 still green. - -- [ ] **Step 5: Commit** - -```bash -git add R/package_index.R man/ -git commit -m "feat(index): upload/update PACKAGES for a per-minor slot via r_minor" -``` - ---- - -### Task 3: `classify_r_minor_sensitive()` internal helper - -**Files:** -- Modify: `R/process_cran_updates.R` (add helper near the top) -- Test: `tests/testthat/test-process_cran_updates.R` (new) - -- [ ] **Step 1: Write the failing test** - -```r -# tests/testthat/test-process_cran_updates.R -test_that("classify_r_minor_sensitive returns TRUE for a risky package", { - skip_if_not_installed("mockery") - mockery::stub(classify_r_minor_sensitive, "clone_repository", function(pkg, tag, url, dest) { - dir.create(dest, recursive = TRUE, showWarnings = FALSE) - writeLines( - c("Package: dummy", "Version: 1.0", "NeedsCompilation: yes", "LinkingTo: Rcpp"), - file.path(dest, "DESCRIPTION") - ) - dir.create(file.path(dest, "src")) - writeLines("// Rcpp glue", file.path(dest, "src", "x.cpp")) - }) - expect_true(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran")) -}) - -test_that("classify_r_minor_sensitive returns FALSE for a pure-r package", { - skip_if_not_installed("mockery") - mockery::stub(classify_r_minor_sensitive, "clone_repository", function(pkg, tag, url, dest) { - dir.create(dest, recursive = TRUE, showWarnings = FALSE) - writeLines(c("Package: dummy", "Version: 1.0", "NeedsCompilation: no"), - file.path(dest, "DESCRIPTION")) - }) - expect_false(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran")) -}) - -test_that("classify_r_minor_sensitive fails safe to TRUE on clone error", { - skip_if_not_installed("mockery") - mockery::stub(classify_r_minor_sensitive, "clone_repository", function(...) stop("boom")) - expect_true(classify_r_minor_sensitive("dummy", "1.0", "https://github.com/cran")) -}) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` -Expected: FAIL — `could not find function "classify_r_minor_sensitive"`. - -- [ ] **Step 3: Add the helper** - -```r -# R/process_cran_updates.R — add above process_cran_updates() -#' Classify a single CRAN package for R-minor sensitivity -#' -#' Clones the package source to a temp dir and runs -#' [needs_per_minor_recompile()]. Fails safe to `TRUE` (build per minor) if the -#' clone or classification errors, so a possibly-ABI-fragile binary is never -#' served from the cross-minor generic slot by mistake. -#' @keywords internal -classify_r_minor_sensitive <- function( - package_name, - tag, - source_org_url = "https://github.com/cran", - local_clone_dir = tempdir() -) { - dest <- file.path( - local_clone_dir, - sprintf("classify_%s_%s", package_name, tag) - ) - on.exit(unlink(dest, recursive = TRUE, force = TRUE), add = TRUE) - tryCatch( - { - clone_repository(package_name, tag, source_org_url, dest) - isTRUE(as.logical(needs_per_minor_recompile(dest))) - }, - error = function(e) { - log_warn(sprintf( - "{.fun classify_r_minor_sensitive}: failed for {.pkg %s} {.field %s}: %s. Treating as r-minor-sensitive.", # nolint - package_name, - tag, - conditionMessage(e) - )) - TRUE - } - ) -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` -Expected: PASS (3/3). - -- [ ] **Step 5: Commit** - -```bash -git add R/process_cran_updates.R tests/testthat/test-process_cran_updates.R -git commit -m "feat(updates): add classify_r_minor_sensitive helper" -``` - ---- - -### Task 4: wire classifier detection into `process_cran_updates()` - -**Files:** -- Modify: `R/process_cran_updates.R` (`process_cran_updates` signature + the build loop at lines ~262-289) -- Modify: `tests/testthat/test-process_cran_updates.R` (add routing tests) - -- [ ] **Step 1: Write the failing tests** - -```r -# append to tests/testthat/test-process_cran_updates.R -test_that("classifier mode passes per-package is_r_minor_sensitive to build", { - skip_if_not_installed("mockery") - recorded <- list() - mockery::stub(process_cran_updates, "get_updated_cran_packages", - function(...) data.frame(name = c("riskypkg", "purepkg"), - version = c("1.0", "2.0"), stringsAsFactors = FALSE)) - mockery::stub(process_cran_updates, "get_new_cran_packages", - function(...) data.frame(name = character(), version = character())) - mockery::stub(process_cran_updates, "tools::CRAN_package_db", - function(...) data.frame(Package = character(), OS_type = character())) - mockery::stub(process_cran_updates, "classify_r_minor_sensitive", - function(name, tag, ...) name == "riskypkg") - mockery::stub(process_cran_updates, "build_binary_package", - function(name, tag, ..., is_r_minor_sensitive) { - recorded[[name]] <<- is_r_minor_sensitive - invisible(TRUE) - }) - - process_cran_updates( - platform = "alpine-323", process_removed = FALSE, - r_minor_detection = "classifier", - s3_endpoint = "x", s3_region = "x", s3_bucket = "x" - ) - - expect_true(recorded[["riskypkg"]]) - expect_false(recorded[["purepkg"]]) -}) - -test_that("r_minor_sensitive_only drops non-risky candidates", { - skip_if_not_installed("mockery") - built <- character() - mockery::stub(process_cran_updates, "get_updated_cran_packages", - function(...) data.frame(name = c("riskypkg", "purepkg"), - version = c("1.0", "2.0"), stringsAsFactors = FALSE)) - mockery::stub(process_cran_updates, "get_new_cran_packages", - function(...) data.frame(name = character(), version = character())) - mockery::stub(process_cran_updates, "tools::CRAN_package_db", - function(...) data.frame(Package = character(), OS_type = character())) - mockery::stub(process_cran_updates, "classify_r_minor_sensitive", - function(name, tag, ...) name == "riskypkg") - mockery::stub(process_cran_updates, "build_binary_package", - function(name, tag, ..., is_r_minor_sensitive) { built <<- c(built, name); invisible(TRUE) }) - - process_cran_updates( - platform = "alpine-323", process_removed = FALSE, - r_minor_detection = "classifier", r_minor_sensitive_only = TRUE, - s3_endpoint = "x", s3_region = "x", s3_bucket = "x" - ) - - expect_identical(built, "riskypkg") -}) -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` -Expected: FAIL — `unused argument (r_minor_detection = ...)`. - -- [ ] **Step 3: Add the parameters** - -In the `process_cran_updates()` signature, after `filter_r_minor_sensitive = FALSE,` add: - -```r - r_minor_detection = c("none", "issue", "classifier"), - r_minor_sensitive_only = FALSE, -``` - -Immediately after the three `stop()` validation lines at the top of the body, add: - -```r - r_minor_detection <- match.arg(r_minor_detection) - # back-compat: the old boolean maps onto the issue-list path - if (isTRUE(filter_r_minor_sensitive) && r_minor_detection == "none") { - r_minor_detection <- "issue" - } -``` - -- [ ] **Step 4: Replace the issue-filter gate** - -The existing block reads `if (filter_r_minor_sensitive) { all_pkgs <- get_r_minor_sensitive_packages(...) }`. Change its condition to: - -```r - if (r_minor_detection == "issue") { - all_pkgs <- get_r_minor_sensitive_packages( - r_minor_packages_forge_type, - r_minor_packages_issue_url, - interval, - updated_packages = updated_pkgs, - new_packages = new_pkgs - ) - } -``` - -Also change the two later `if (filter_r_minor_sensitive)` log branches to `if (r_minor_detection != "none")`. - -- [ ] **Step 5: Replace the build loop** - -Replace the `purrr::walk2(all_pkgs$name, all_pkgs$version, ~ { build_binary_package(... is_r_minor_sensitive = filter_r_minor_sensitive ...) })` block with: - -```r - if (nrow(all_pkgs) > 0L) { - sensitive <- switch( - r_minor_detection, - classifier = vapply( - seq_len(nrow(all_pkgs)), - function(i) { - classify_r_minor_sensitive( - all_pkgs$name[i], - all_pkgs$version[i], - local_clone_dir = local_clone_dir - ) - }, - logical(1L) - ), - issue = rep(TRUE, nrow(all_pkgs)), - none = rep(FALSE, nrow(all_pkgs)) - ) - - if (isTRUE(r_minor_sensitive_only)) { - all_pkgs <- all_pkgs[sensitive, , drop = FALSE] - sensitive <- sensitive[sensitive] - } - - purrr::pwalk( - list(all_pkgs$name, all_pkgs$version, sensitive), - function(.name, .version, .sensitive) { - build_binary_package( - .name, - .version, - platform = platform, - upload = upload, - archive = archive, - force = force, - store_build_metadata = store_build_metadata, - s3_endpoint = s3_endpoint, - s3_bucket = s3_bucket, - s3_region = s3_region, - s3_access_key_id = s3_access_key_id, - s3_secret_access_key = s3_secret_access_key, - is_r_minor_sensitive = .sensitive, - metadata_db_type = metadata_db_type, - metadata_db_host = metadata_db_host, - metadata_db_name = metadata_db_name, - metadata_db_table = metadata_db_table, - metadata_db_port = metadata_db_port, - metadata_db_user = metadata_db_user, - metadata_db_password = metadata_db_password, - metadata_db_sslmode = metadata_db_sslmode - ) - } - ) - } else { - log_info("No packages to process after filtering Windows-only packages") - } -``` - -- [ ] **Step 6: Run tests to verify they pass** - -Run: `R -q -e 'devtools::load_all("."); testthat::test_file("tests/testthat/test-process_cran_updates.R")'` -Expected: PASS (5/5 — 3 from Task 3 + 2 new). - -- [ ] **Step 7: Commit** - -```bash -git add R/process_cran_updates.R tests/testthat/test-process_cran_updates.R -git commit -m "feat(updates): classifier-driven per-package r-minor routing + sensitive-only pass" -``` - ---- - -### Task 5: docs, roxygen params, version bump, NEWS - -**Files:** -- Create: `man-roxygen/param-r_minor_detection.R`, `man-roxygen/param-r_minor_sensitive_only.R` -- Modify: `R/process_cran_updates.R` (roxygen `@template` lines), `DESCRIPTION`, `NEWS.md` - -- [ ] **Step 1: Add the man templates** - -```r -# man-roxygen/param-r_minor_detection.R -#' @param r_minor_detection How to decide which packages are R-minor-sensitive. -#' `"none"` (default) builds everything into the generic slot. `"issue"` uses -#' the curated tracking issue (the legacy `filter_r_minor_sensitive` path). -#' `"classifier"` classifies each candidate via [needs_per_minor_recompile()] -#' and routes only `risky` packages to the per-minor slot. -``` - -```r -# man-roxygen/param-r_minor_sensitive_only.R -#' @param r_minor_sensitive_only When `TRUE`, only R-minor-sensitive packages are -#' built (used for the additional per-minor passes under non-primary R versions). -``` - -- [ ] **Step 2: Reference the templates** - -In the roxygen block above `process_cran_updates`, add: - -```r -#' @template param-r_minor_detection -#' @template param-r_minor_sensitive_only -``` - -- [ ] **Step 3: Bump version and NEWS** - -In `DESCRIPTION` set `Version: 4.2.0`. Add to the top of `NEWS.md`: - -```markdown -# bincraft 4.2.0 - -* `process_cran_updates()` gains `r_minor_detection` (`"none"`/`"issue"`/`"classifier"`) - and `r_minor_sensitive_only`, classifying each candidate via the ABI classifier and - routing only `risky` packages to per-minor slots. -* `upload_package_index()` / `add_to_package_index()` gain an `r_minor` argument to - write/serve a per-minor `PACKAGES*` index under `…/contrib//`. -``` - -- [ ] **Step 4: Regenerate docs and run full check** - -Run: `R -q -e 'devtools::document()'` -Run: `R -q -e 'devtools::test()'` -Expected: docs regenerate clean; all tests PASS. - -- [ ] **Step 5: Commit, push, open PR** - -```bash -git add DESCRIPTION NEWS.md man-roxygen/ man/ R/process_cran_updates.R -git commit -m "docs(release): bincraft 4.2.0 — per-minor index + classifier-driven updates" -git push -u origin -fj -H codefloe.com pr create --base main --head --body "" "feat: per-minor index + classifier-driven r-minor builds (4.2.0)" -``` - ---- - -## Self-Review - -- **Spec coverage:** §"Required bincraft enhancements" item 1 (process_cran_updates) → Tasks 3-5; item 2 (upload_package_index per-minor) → Tasks 1-2. `build_binary_package` unchanged per the spec — no task, correct. -- **Type consistency:** helper named `package_index_remote_dir` (Tasks 1-2), `classify_r_minor_sensitive` (Tasks 3-4) consistently; params `r_minor_detection`/`r_minor_sensitive_only`/`r_minor` consistent across signature, tests, and docs. -- **Placeholders:** none — every code/edit step shows the actual code; ``/`` in the final push step are deliberate operator inputs. -- **Release dependency:** the pipeline plan pins bincraft `4.2.0`, matching the bump here. diff --git a/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md b/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md deleted file mode 100644 index 6123278..0000000 --- a/docs/superpowers/plans/2026-06-13-pipeline-r-minor-builds.md +++ /dev/null @@ -1,471 +0,0 @@ -# Pipeline R-minor Builds Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make the full (`build-all-versions-*`) and iterative (`process-updates-*`) pipelines build R-minor-sensitive packages once per installed R minor, and everything else once into the generic slot. - -**Architecture:** A precompute step classifies packages (cheap CRAN-metadata rules + source download only for the ambiguous compiled subset) and emits a sensitivity flag. Build scripts pass a concrete `is_r_minor_sensitive` per package on the primary-R pass and re-run a sensitive-only pass under each additional `/opt/R/*` minor. Per-minor `PACKAGES` indexes are uploaded for each touched slot. - -**Tech Stack:** R scripts under `local/`, crow/woodpecker CI YAML under `.crow/`, bincraft `>= 4.2.0` (see `2026-06-13-bincraft-r-minor-enhancements.md`). - -**HARD DEPENDENCY:** bincraft `4.2.0` (the companion plan) must be merged and released first — Tasks 4-6 call `r_minor_detection`/`upload_package_index(r_minor=)` which only exist there. Tasks 1-3 (this repo's R scripts) can be written and unit-tested before the release. - ---- - -### Task 1: pure metadata classifier helper - -**Files:** -- Create: `local/r-minor-helpers.R` -- Test: `local/tests/test-r-minor-helpers.R` - -- [ ] **Step 1: Write the failing test** - -```r -# local/tests/test-r-minor-helpers.R -source(file.path("local", "r-minor-helpers.R")) - -test_that("pure-r (NeedsCompilation != yes) is not sensitive", { - expect_identical(classify_from_metadata("no", NA, c("Rcpp")), "not-sensitive") - expect_identical(classify_from_metadata("", "Rcpp", c("Rcpp")), "not-sensitive") -}) - -test_that("LinkingTo a risky dep is sensitive (version constraints stripped)", { - expect_identical(classify_from_metadata("yes", "Rcpp (>= 1.0)", c("Rcpp")), "sensitive") - expect_identical(classify_from_metadata("yes", "R6,\n cpp11", c("Rcpp", "cpp11")), "sensitive") -}) - -test_that("compiled but no risky LinkingTo is ambiguous (needs source)", { - expect_identical(classify_from_metadata("yes", "R6", c("Rcpp", "cpp11")), "ambiguous") - expect_identical(classify_from_metadata("yes", NA, c("Rcpp")), "ambiguous") -}) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'` -Expected: FAIL — `cannot open file 'local/r-minor-helpers.R'` / function not found. - -- [ ] **Step 3: Write the helper** - -```r -# local/r-minor-helpers.R -# Metadata-only ABI triage so the full build avoids downloading every source. -# Mirrors bincraft::abi_classify rules 1-2; "ambiguous" packages still need a -# source grep via bincraft::needs_per_minor_recompile(). - -classify_from_metadata <- function(needs_compilation, linking_to, risky_deps) { - nc <- if (length(needs_compilation) == 0L || is.na(needs_compilation)) { - "" - } else { - tolower(trimws(needs_compilation)) - } - if (!identical(nc, "yes")) { - return("not-sensitive") - } - lt <- if (length(linking_to) == 0L || is.na(linking_to)) "" else linking_to - linked <- trimws(unlist(strsplit(lt, "[,\n]"))) - linked <- sub("\\s*\\(.*\\)$", "", linked) # strip "(>= x)" constraints - linked <- linked[nzchar(linked)] - if (any(linked %in% risky_deps)) { - return("sensitive") - } - "ambiguous" -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-r-minor-helpers.R")'` -Expected: PASS (3/3). - -- [ ] **Step 5: Commit** - -```bash -git add local/r-minor-helpers.R local/tests/test-r-minor-helpers.R -git commit -m "feat(local): metadata-only ABI triage helper" -``` - ---- - -### Task 2: add `r_minor_sensitive` to `packages-to-build.R` - -**Files:** -- Modify: `local/packages-to-build.R` (after `pkgs` is finalized, ~line 142) - -- [ ] **Step 1: Append the classification block** - -At the end of `local/packages-to-build.R` (after `setorder(pkgs, Package, Version)`), add: - -```r -### R-minor sensitivity (classify once per package, applied to all versions) -source(file.path("local", "r-minor-helpers.R")) -risky_deps <- bincraft::abi_risky_linking_deps() - -release_meta <- data.table( - Package = cran_release$Package, - NeedsCompilation = cran_release$NeedsCompilation, - LinkingTo = cran_release$LinkingTo -) - -meta <- release_meta[Package %in% unique(pkgs$Package)] -meta[, triage := mapply( - classify_from_metadata, - NeedsCompilation, - LinkingTo, - MoreArgs = list(risky_deps = risky_deps) -)] - -# Only the "ambiguous" compiled packages need a source grep. -ambiguous <- meta[triage == "ambiguous", Package] -sensitive_ambiguous <- character() -if (length(ambiguous) > 0L) { - tmp_src <- file.path(tempdir(), "abi_src") - dir.create(tmp_src, showWarnings = FALSE, recursive = TRUE) - sens <- vapply(ambiguous, function(pkg) { - out <- tryCatch({ - dl <- utils::download.packages( - pkg, destdir = tmp_src, - repos = "https://cloud.r-project.org", quiet = TRUE - ) - isTRUE(as.logical(bincraft::needs_per_minor_recompile(dl[1L, 2L]))) - }, error = function(e) TRUE) # fail safe: treat as sensitive - out - }, logical(1L)) - sensitive_ambiguous <- ambiguous[sens] -} - -sensitive_pkgs <- unique(c(meta[triage == "sensitive", Package], sensitive_ambiguous)) -pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs] -sprintf("R-minor-sensitive packages: %s of %s", length(sensitive_pkgs), uniqueN(pkgs$Package)) -``` - -- [ ] **Step 2: Verify the script parses and the column is added (offline smoke)** - -Run: -```bash -R -q -e ' - library(data.table) - source("local/r-minor-helpers.R") - pkgs <- data.table(Package = c("A","B"), Version = c("1","1")) - cran_release <- data.frame(Package = c("A","B"), - NeedsCompilation = c("no","yes"), LinkingTo = c(NA,"Rcpp"), - stringsAsFactors = FALSE) - risky_deps <- c("Rcpp") - release_meta <- as.data.table(cran_release) - meta <- release_meta[Package %in% unique(pkgs$Package)] - meta[, triage := mapply(classify_from_metadata, NeedsCompilation, LinkingTo, - MoreArgs = list(risky_deps = risky_deps))] - sensitive_pkgs <- meta[triage == "sensitive", Package] - pkgs[, r_minor_sensitive := Package %in% sensitive_pkgs] - stopifnot(identical(pkgs$r_minor_sensitive, c(FALSE, TRUE))) - cat("OK\n")' -``` -Expected: prints `OK` (B classified sensitive via LinkingTo, A not). - -- [ ] **Step 3: Persist the sensitive subset in install-deps** - -In `.crow/build-all-versions-install-deps-amd64.yaml` and `...-arm64.yaml`, change the precompute command (line ~38) from saving only `pkgs_to_build.rds` to also saving the subset: - -```yaml - - /opt/R/$R_VERSION/bin/R -q -e "source('local/packages-to-build.R'); saveRDS(pkgs, '/mnt/cache/packages/pkgs_to_build.rds'); saveRDS(pkgs[r_minor_sensitive == TRUE], '/mnt/cache/packages/r_minor_sensitive_pkgs.rds'); sprintf('Precomputed %s package versions (%s r-minor-sensitive)', nrow(pkgs), nrow(pkgs[r_minor_sensitive == TRUE]))" -``` - -- [ ] **Step 4: Commit** - -```bash -git add local/packages-to-build.R .crow/build-all-versions-install-deps-amd64.yaml .crow/build-all-versions-install-deps-arm64.yaml -git commit -m "feat(full): classify r-minor sensitivity in install-deps precompute" -``` - ---- - -### Task 3: per-package flag + `--sensitive-only` in `build-all.R` - -**Files:** -- Modify: `local/build-all.R` -- Test: `local/tests/test-build-all-args.R` (new — covers the arg/flag parsing seam only) - -- [ ] **Step 1: Write the failing test for the arg parser** - -```r -# local/tests/test-build-all-args.R -source(file.path("local", "r-minor-helpers.R")) - -test_that("parse_build_args splits flags from positionals", { - a <- parse_build_args(c("--sensitive-only", "4", "2", "8")) - expect_true(a$sensitive_only) - expect_identical(a$split_into, 4L) - expect_identical(a$split_index, 2L) - expect_identical(a$ncpus, 8L) - - b <- parse_build_args(c("4", "2", "8")) - expect_false(b$sensitive_only) - expect_identical(b$split_into, 4L) -}) -``` - -- [ ] **Step 2: Add `parse_build_args` to the helpers and run the test** - -Append to `local/r-minor-helpers.R`: - -```r -parse_build_args <- function(args) { - sensitive_only <- "--sensitive-only" %in% args - pos <- args[!startsWith(args, "--")] - list( - sensitive_only = sensitive_only, - split_into = as.integer(pos[1L]), - split_index = as.integer(pos[2L]), - ncpus = as.integer(pos[3L]) - ) -} -``` - -Run: `R -q -e 'library(testthat); testthat::test_file("local/tests/test-build-all-args.R")'` -Expected: PASS (1/1, 5 expectations). - -- [ ] **Step 3: Rewire `build-all.R` to use the parser, the flag, and the per-row sensitivity** - -Replace the arg-parsing header and the `mapply` build loop of `local/build-all.R`. New top: - -```r -sink(stdout(), type = "message") -options(crayon.enabled = TRUE, future.globals.onReference = NULL) -source(file.path("local", "r-minor-helpers.R")) - -args <- commandArgs(trailingOnly = TRUE) -parsed <- parse_build_args(args) -split_into <- parsed$split_into -split_index <- parsed$split_index -ncpus <- parsed$ncpus -sensitive_only <- parsed$sensitive_only -options(Ncpus = ncpus) - -library(bincraft, quietly = TRUE) -library(future) -plan("sequential") - -pkgs <- if (sensitive_only) { - readRDS("/mnt/cache/packages/r_minor_sensitive_pkgs.rds") -} else { - readRDS("/mnt/cache/packages/pkgs_to_build.rds") -} -# Back-compat: tolerate an older RDS without the column (treat all as non-sensitive) -if (is.null(pkgs$r_minor_sensitive)) pkgs$r_minor_sensitive <- FALSE -sprintf("Total# of remaining package versions: %s (sensitive_only=%s)", nrow(pkgs), sensitive_only) -``` - -Keep the existing chunk-split, exclude-list, and `s3_cache` lines unchanged. Then replace the `mapply(...)` call with: - -```r -n <- nrow(chunk) -mapply(function(pkg, ver, sens, i) { - cat(sprintf("[%d/%d] %s_%s (r_minor_sensitive=%s)\n", i, n, pkg, ver, sens)) - bincraft::build_binary_package( - pkg, - tag = ver, - is_r_minor_sensitive = isTRUE(sens), - s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", - s3_region = "eu-central-003", - s3_bucket = "devxy-rpkgs-binaries", - s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), - s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"), - s3_package_cache = s3_cache, - metadata_db_host = "r-binaries.devxy.io", - metadata_db_name = "build_metadata", - metadata_db_table = "single_builds", - metadata_db_user = "rpkgs", - metadata_db_password = Sys.getenv("PGPASS"), - metadata_db_sslmode = "require", - metadata_db_port = 15432, - archive = TRUE, - upload = TRUE, - store_build_metadata = TRUE - ) -}, chunk$Package, chunk$Version, chunk$r_minor_sensitive, seq_len(n)) -``` - -Note: in `--sensitive-only` mode every row already has `r_minor_sensitive == TRUE`, so the same loop builds them into per-minor slots; no special-casing needed. - -- [ ] **Step 4: Verify the script parses** - -Run: `R -q -e 'invisible(parse(file="local/build-all.R")); cat("parsed OK\n")'` -Expected: `parsed OK`. - -- [ ] **Step 5: Commit** - -```bash -git add local/build-all.R local/r-minor-helpers.R local/tests/test-build-all-args.R -git commit -m "feat(full): per-package r-minor flag and --sensitive-only mode in build-all.R" -``` - ---- - -### Task 4: multi-R loop + per-minor index in `build-all-versions-*.yaml` - -> Depends on bincraft 4.2.0 (`upload_package_index(r_minor=)`). - -**Files:** -- Modify: `.crow/build-all-versions-amd64.yaml`, `.crow/build-all-versions-arm64.yaml` - -- [ ] **Step 1: Add the sensitive-only multi-R pass after the primary build** - -In each file's `commands:` (after the existing `Rscript local/build-all.R $SPLIT_INTO $SPLIT_INDEX $NCPUS` line and before the `process_unarchived_pkgs` line), insert this multiline command: - -```yaml - - | - PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) - for RBIN in /opt/R/*/bin/R; do - RV=$(basename "$(dirname "$(dirname "$RBIN")")") - RMINOR=$(echo "$RV" | cut -d. -f1-2) - [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue - echo "=== R-minor-sensitive pass under R $RV ===" - R_VERSION="$RV" $XVFB $XVFB_ARGS -n $SPLIT_INDEX -- "$RBIN" Rscript local/build-all.R --sensitive-only $SPLIT_INTO $SPLIT_INDEX $NCPUS 2>&1 || true - done -``` - -(`$XVFB`/`$XVFB_ARGS` are already defined on the preceding line in this workflow.) Note the primary pass still runs `build-all.R` without `--sensitive-only`, building everything and routing sensitive packages into the primary minor slot. - -- [ ] **Step 2: Add a per-minor index upload step** - -Append a new step after the build step (mirroring the existing CDN/index pattern). Add to `commands` of a new `Upload per-minor indexes` step (or extend the existing index handling) the following, which uploads the generic index plus one per discovered minor: - -```yaml - - | - CODENAME=$(/opt/R/$R_VERSION/bin/Rscript -e "cat(bincraft::set_codename(NULL))") - # generic slot - /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" - for RBIN in /opt/R/*/bin/R; do - RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) - /opt/R/$R_VERSION/bin/R -q -e "bincraft::upload_package_index(codename = '$CODENAME', r_minor = '$RMINOR', s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'))" || true - done -``` - -- [ ] **Step 3: Validate YAML** - -Run: `R -q -e 'invisible(lapply(c(".crow/build-all-versions-amd64.yaml",".crow/build-all-versions-arm64.yaml"), yaml::yaml.load_file)); cat("yaml OK\n")'` -Expected: `yaml OK`. - -- [ ] **Step 4: Commit** - -```bash -git add .crow/build-all-versions-amd64.yaml .crow/build-all-versions-arm64.yaml -git commit -m "feat(full): sensitive-only multi-R passes and per-minor index upload" -``` - ---- - -### Task 5: classifier + multi-R in all `process-updates-*.yaml` - -> Depends on bincraft 4.2.0 (`r_minor_detection`, `upload_package_index(r_minor=)`). - -**Files (14):** every `.crow/process-updates--.yaml`: -`alpine-322`, `alpine-323`, `redhat-8`, `redhat-9`, `redhat-10`, `ubuntu-2204`, `ubuntu-2404` × `amd64`, `arm64`. - -The transform is identical in shape; only the already-present `platform=`, `R_VERSION`, `ARCH`, and the index `codename=` differ per file (leave those as-is). - -- [ ] **Step 1: Edit the `process_cran_updates` call (primary pass)** - -In each file, inside the single long `bincraft::process_cran_updates(...)` argument list, add `r_minor_detection = 'classifier', ` immediately before `s3_endpoint = ...`. Leave all other args unchanged. - -- [ ] **Step 2: Add the sensitive-only multi-R pass** - -Immediately after the `process_cran_updates(...)` command line, insert (preserve the per-file `''` string and `$INTERVAL`): - -```yaml - - | - PRIMARY_MINOR=$(echo "$R_VERSION" | cut -d. -f1-2) - for RBIN in /opt/R/*/bin/R; do - RV=$(basename "$(dirname "$(dirname "$RBIN")")") - RMINOR=$(echo "$RV" | cut -d. -f1-2) - [ "$RMINOR" = "$PRIMARY_MINOR" ] && continue - echo "=== R-minor-sensitive update pass under R $RV ===" - xvfb-run "$RBIN" -q -e "options(crayon.enabled = TRUE, Ncpus = 4, future.globals.onReference = NULL); bincraft::process_cran_updates(interval = $INTERVAL, platform = '', process_updated = TRUE, process_new = FALSE, process_removed = FALSE, r_minor_detection = 'classifier', r_minor_sensitive_only = TRUE, s3_endpoint = 'https://s3.eu-central-003.backblazeb2.com', s3_region = 'eu-central-003', s3_bucket = 'devxy-rpkgs-binaries', s3_access_key_id = Sys.getenv('B2_S3_ACCESS_KEY'), s3_secret_access_key = Sys.getenv('B2_S3_SECRET_KEY'), metadata_db_host = 'r-binaries.devxy.io', metadata_db_name = 'build_metadata', metadata_db_table = 'single_builds', metadata_db_user = 'rpkgs', metadata_db_password = Sys.getenv('PGPASS'), metadata_db_sslmode = 'require', metadata_db_port = 15432, archive = TRUE, upload = TRUE, store_build_metadata = TRUE)" || true - done -``` - -Replace `` with that file's existing platform string (e.g. `alpine-323`, `redhat-9`, `ubuntu-2404`). `process_removed = FALSE` on the extra passes (removals are handled once by the primary pass). - -- [ ] **Step 3: Add per-minor index upload** - -After the existing `upload_package_index(codename = "", ...)` line in each file, append (reusing the file's existing `codename`): - -```yaml - - | - for RBIN in /opt/R/*/bin/R; do - RMINOR=$(basename "$(dirname "$(dirname "$RBIN")")" | cut -d. -f1-2) - /opt/R/$R_VERSION/bin/R -q -e 'library(bincraft); upload_package_index(codename = "", r_minor = "'"$RMINOR"'", s3_endpoint = "https://s3.eu-central-003.backblazeb2.com", s3_region = "eu-central-003", s3_bucket = "devxy-rpkgs-binaries", s3_access_key_id = Sys.getenv("B2_S3_ACCESS_KEY"), s3_secret_access_key = Sys.getenv("B2_S3_SECRET_KEY"))' || true - done -``` - -Replace `` with the file's existing codename value (e.g. `alpine323`, `rhel9`, `ubuntu2404`). - -- [ ] **Step 4: Verify every file changed and parses** - -Run: -```bash -R -q -e 'fs <- Sys.glob(".crow/process-updates-*.yaml"); invisible(lapply(fs, yaml::yaml.load_file)); cat(length(fs), "files parse OK\n")' -grep -L "r_minor_detection = 'classifier'" .crow/process-updates-*.yaml -``` -Expected: `14 files parse OK`, and the `grep -L` prints **nothing** (every file contains the classifier arg). - -- [ ] **Step 5: Commit** - -```bash -git add .crow/process-updates-*.yaml -git commit -m "feat(updates): classifier-driven r-minor builds + per-minor index across platforms" -``` - ---- - -### Task 6: pin bincraft 4.2.0, remove the standalone workflow - -**Files:** -- Modify: all workflows pinning bincraft (`grep -rln "bincraft.git@v" .crow/`), and `.crow/build-all-versions-install-deps-*.yaml` (which installs from `main`) -- Delete: `.crow/build-r-minor-sensitive-packages.yaml` - -- [ ] **Step 1: Bump the pin** - -Run to find pins: `grep -rn "bincraft" .crow/ | grep -E "@v4\.1\.1|bincraft.git"`. In every `process-updates-*` file, change `bincraft.git@v4.1.1` (and the `packageVersion("bincraft") != "4.1.1"` guard) to `4.2.0`. - -- [ ] **Step 2: Remove the superseded workflow** - -```bash -git rm .crow/build-r-minor-sensitive-packages.yaml -``` - -- [ ] **Step 3: Verify no stale references** - -Run: `grep -rn "build-r-minor-sensitive\|@v4.1.1\|!= \"4.1.1\"" .crow/` -Expected: no matches. - -- [ ] **Step 4: Commit** - -```bash -git add .crow/ -git commit -m "chore: pin bincraft 4.2.0 and drop superseded standalone r-minor workflow" -``` - ---- - -### Task 7: integration smoke test (manual, gated on bincraft 4.2.0 release) - -- [ ] **Step 1: Run one iterative platform manually** via crow against a short interval and confirm: a known risky package (e.g. one LinkingTo Rcpp) lands under `…/contrib//` for each installed minor, and a pure-r package lands only in the generic slot. - -- [ ] **Step 2: Confirm a client install resolves the per-minor slot.** From an R `4.4` and an R `4.5` container: -```r -install.packages("", repos = "https://cran.devxy.io/") -library() # must load without "undefined symbol" -``` -Expected: loads under both minors. If the per-minor `PACKAGES` is missing, revisit bincraft Task 2. - -- [ ] **Step 3: Spot-check the full build** on one platform with a small `SPLIT_INTO`, verifying the sensitive-only extra passes ran and produced per-minor artifacts. - ---- - -## Self-Review - -- **Spec coverage:** install-deps precompute → Tasks 1-2; `build-all.R` per-row flag + `--sensitive-only` → Task 3; full-build multi-R loop + per-minor index → Task 4; iterative classifier + multi-R + per-minor index → Task 5; remove standalone workflow + version pin → Task 6; per-minor index client-serviceability risk → Task 7 verification. -- **Type/name consistency:** `classify_from_metadata` and `parse_build_args` live in `local/r-minor-helpers.R` and are used in Tasks 2-3; the RDS column is `r_minor_sensitive` everywhere; the subset file is `/mnt/cache/packages/r_minor_sensitive_pkgs.rds` in install-deps (Task 2) and `build-all.R` (Task 3). -- **Placeholders:** ``/`` in Task 5 are explicit per-file substitutions (the values already exist in each file), not unfilled blanks. No "TBD"/"handle errors" placeholders. -- **Dependency ordering:** Tasks 1-3 are pure R-script work, unit-testable now; Tasks 4-6 are gated on bincraft 4.2.0; Task 7 is post-release verification. diff --git a/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md b/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md deleted file mode 100644 index 248a1a7..0000000 --- a/docs/superpowers/specs/2026-06-13-r-minor-sensitive-builds-design.md +++ /dev/null @@ -1,155 +0,0 @@ -# R-minor-sensitive binary builds — design - -## Problem - -CRAN binaries are currently built once, under a single R minor version, and served from a -single generic slot (`…/latest/src/contrib/`). -That is wrong for the minority of packages whose compiled code reaches into volatile R -internals: a binary built under R 4.5 will fail to load under R 4.4 with an -`undefined symbol` error. - -bincraft v4.1.0+ can now detect these packages automatically via the ABI classifier -(`abi_classify()` / `needs_per_minor_recompile()`, added in bincraft PR #49, -). -The exec-env images now ship multiple R minor versions under `/opt/R/`. -This design uses both to build R-minor-specific binaries for the sensitive packages only, -in both the iterative (`process-updates-*`) and full (`build-all-versions-*`) pipelines. - -## Background: how bincraft already behaves - -- `abi_classify(path)` returns a tier: `pure-r` (~78.6%), `safe-compiled` (~7.7%), or - `risky` (~13.6%). `needs_per_minor_recompile(path)` is the boolean wrapper - (`TRUE` iff `risky`). Both need the package source (DESCRIPTION + `src/`). -- `build_binary_package(…, is_r_minor_sensitive = TRUE)` uploads the artifact into a - per-minor slot `…/latest/src/contrib//` and records `r_version` in the - build metadata. With `FALSE` it uses the generic slot. The minor is derived from the - **running interpreter** (`R.version`), so producing a 4.4 binary requires running - `/opt/R/4.4.x/bin/R`. -- `process_cran_updates()` orchestrates the iterative flow with a single - `is_r_minor_sensitive` bool for the whole run and no per-package classification or - R-version loop. -- `upload_package_index()` writes/uploads `PACKAGES*` for the **generic slot only**; it - has no per-minor support. The current standalone r-minor workflow never builds a - per-minor index, so per-minor slots are effectively unservable today. - -## Core model - -Collapse the three tiers to one boolean per package: - -``` -r_minor_sensitive := (abi_classify(pkg)$tier == "risky") -``` - -| group | tiers | built under | slot | -| ------------------------------ | ---------------------- | -------------------- | ---------------------------- | -| non-sensitive (~86%) | pure-r, safe-compiled | primary R only | generic `contrib/` | -| sensitive (~14%) | risky | every installed minor| per-minor `contrib//` | - -- **Primary R** = the existing `R_VERSION` env var in each workflow. Its pass builds the - non-sensitive packages (generic slot) *and* the sensitive packages for its own minor slot. -- **Extra minors** = every other R version discovered by scanning `/opt/R/` at runtime. - Each runs a **sensitive-only** pass. An image with a single R version degrades cleanly - to just the primary pass. -- The loop over minors always lives at the shell/script layer (one `/opt/R//bin/R` - invocation per minor), never inside a single `build_binary_package()` call. - -Classification granularity: classify **once per package** (its release version) and apply -the resulting flag to all archived versions of that package. Tier rarely changes across -recent versions; this avoids multiplying source downloads. - -## Full build — `build-all-versions-*` + `local/` - -### install-deps step (`local/packages-to-build.R`) - -After computing `pkgs_to_build`, add an `r_minor_sensitive` logical column: - -1. Pull `NeedsCompilation` and `LinkingTo` from `tools::CRAN_package_db()` (already loaded). -2. Resolve cheaply, no download: - - `NeedsCompilation != "yes"` → not sensitive (rule 1, pure-r). - - `LinkingTo` references any `bincraft::abi_risky_linking_deps()` entry → sensitive - (rule 2). -3. For the remaining compiled, non-LinkingTo-risky packages only: download the source and - call `bincraft::needs_per_minor_recompile()` (rules 3/4). -4. Join the per-package flag onto every `(Package, Version)` row. - -Outputs: -- `pkgs_to_build.rds` — now carries the `r_minor_sensitive` column. -- `r_minor_sensitive_pkgs.rds` — the sensitive subset, for the extra-minor passes. - -### build step (`local/build-all.R` + `.crow/build-all-versions-{amd64,arm64}.yaml`) - -- `build-all.R` gains an optional `--sensitive-only` mode (or an arg flag). In normal mode - it builds the full chunk, passing `is_r_minor_sensitive = ` per package. In - sensitive-only mode it reads `r_minor_sensitive_pkgs.rds`, intersects with its chunk, and - builds those with `is_r_minor_sensitive = TRUE`. -- The workflow step keeps the existing matrix split. After the primary `Rscript build-all.R` - invocation, a shell loop discovers non-primary `/opt/R/*` minors and runs - `Rscript build-all.R --sensitive-only ` under each. -- Index upload step: generic index as today, plus a per-minor index for each minor slot - that received artifacts (depends on the `upload_package_index()` enhancement below). - -## Iterative build — `process-updates-*` - -Driven by the bincraft enhancement below; the single build step becomes: - -1. Primary-R pass: - `process_cran_updates(…, r_minor_detection = "classifier")`. - Each updated/new package is classified; risky → primary minor slot, rest → generic slot. - Removed-package handling stays as-is. -2. Shell loop over non-primary `/opt/R/*` minors: - `process_cran_updates(…, r_minor_detection = "classifier", r_minor_sensitive_only = TRUE)`. - Builds only risky updates into their respective minor slots. -3. Index upload extended to cover each touched minor slot in addition to the generic slot. - -## Required bincraft enhancements (separate PR, coordinated release) - -`build_binary_package()` needs **no change** — it already accepts a concrete -`is_r_minor_sensitive` logical and routes the slot accordingly. Classification stays at -the orchestration layer (mirroring the full-build precompute), which keeps -`build_binary_package`'s pre-build S3 skip-check and source clone untouched. - -1. `process_cran_updates()`: - - Add `r_minor_detection = c("none", "issue", "classifier")` (default `"none"` to - preserve current behavior; `"issue"` is today's `filter_r_minor_sensitive` path). - - Add `r_minor_sensitive_only` (default `FALSE`). - - With `"classifier"`: for each candidate `(name, version)`, clone the source to a temp - dir and call `bincraft::needs_per_minor_recompile()`; pass the resulting concrete - logical as `is_r_minor_sensitive` to `build_binary_package()`. When - `r_minor_sensitive_only = TRUE`, drop non-risky candidates before building. - - Implemented via a small internal helper `classify_r_minor_sensitive(name, tag, - source_org_url, local_clone_dir)` returning a logical — the unit-testable seam. -2. `upload_package_index()`: - - Add an `r_minor = NULL` argument. When non-NULL (e.g. `"4.4"`), point the remote dir - at `…/contrib//` and write/upload `PACKAGES*` (and `Meta/archive.rds`) there, - mirroring the generic-slot logic. Extract the remote-dir construction into a pure - helper `package_index_remote_dir(s3_bucket, arch, codename, r_minor = NULL)` — the - unit-testable seam. - -These two are the only bincraft changes; detection itself (PR #49) is already merged. - -## Scope and cleanup - -- In scope: `.crow/process-updates-*` (iterative) and `.crow/build-all-versions-*` + - `local/build-all.R` + `local/packages-to-build.R` (full), across amd64 and arm64 and all - platforms (alpine, ubuntu, redhat). The per-platform workflow files share the same edit. -- Removed: `.crow/build-r-minor-sensitive-packages.yaml` — superseded by the integrated - flow (it was manual, alpine-3.21-only, and issue-list-driven). -- Out of scope (call out, do not change here): `weekly-rebuild-missing-*`, - `archive-missed-packages`, and the audit workflows. Revisit separately if per-minor - rebuilds are wanted there too. - -## Open risks / notes - -- **R-version discovery**: assumes `/opt/R//bin/R` layout and that the primary - `R_VERSION` is one of the installed versions. Parse minor as `major.minor` from each - discovered version; dedupe by minor (build once per minor even if two patch releases - coexist). -- **Per-minor index correctness**: clients resolving `bin//contrib//` require the - per-minor `PACKAGES` to exist; the `upload_package_index()` enhancement is a hard - dependency for the sensitive artifacts to be usable. Verify against a real client - install before declaring done. -- **Classification cost**: bounded by downloading sources only for the compiled, - non-LinkingTo-risky subset in install-deps. Worth measuring on a full run; if still too - heavy, consider caching classifications keyed by package+version in the metadata DB. -- **Double download**: install-deps classification downloads some sources that the build - step re-downloads. Acceptable for now; the metadata-DB cache above would also remove this. -- 2.54.0