devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
0
Commit graph build-cran-binaries/.gitignore
Author SHA1 Message Date
72440f9cf5 feat(stats): collect CDN access logs and publish download statistics (#251)
All checks were successful
ci/crow/manual/download-stats Pipeline was successful
ci/crow/cron/download-stats Pipeline was successful
Adds visibility into how cran.rpkgs.com is actually used, from the Bunny access logs, without sending anything to a third party.

## What it does

- `.crow/download-stats.yaml` runs hourly: `stats/collect.ts` reads the last 70 hours of cran-rpkgs logs (Bunny keeps three days) into a new `download_stats` schema in the build metadata database, then `stats/publish.ts` writes aggregated counts to `stats/downloads.json` in the binaries bucket and purges the CDN copy.
- `stats/classify.ts` labels each request (download, index, redirect reason, renv/pak probe, scanner noise) by mirroring `edge/rpkgs-router.ts`, and splits cloud/CI networks (by ASN) from everything else.
- Downloads are de-duplicated per client, package version and UTC day; clients are a salted HMAC of the anonymized /24 prefix, ASN and R platform, so no IP is stored or published.
- Our own CI runners (`STATS_OWN_NETWORKS`) are excluded; collection windows are recorded so the dashboard can show gaps.
- `cdn.tf` declares the zone's existing logging settings and adds `json` to the CORS extensions so rpkgs.com can read the file.
- `stats/README.md` documents every rule.

The rpkgs.com `/stats` page that renders this is in rpkgs/website.

## Before merging

- Add Crow secrets `STATS_CLIENT_SALT` and `STATS_OWN_NETWORKS` (values in OpenBao at `development/agent-dev/rpkgs-download-stats`).
- `tofu apply` for the CORS change.
- After merge, register the `download-stats` cron (`0 * * * *`).

## Verification

- `bun test stats`: 15 tests pass.
- Ran `collect.ts` locally against the live API and database: 14,273 events inserted, a second run inserted 0.
- `publish.ts --out` produced the JSON the website was previewed with; the upload and purge path has only run in CI so far (B2 credentials are CI-only).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reviewed-on: #251
1d ago
55a18fd87d feat: patch registry + wiring for per-package patching (#103)
Some checks failed
ci/crow/cron/process-updates/15 Pipeline failed
ci/crow/cron/process-updates/14 Pipeline failed
ci/crow/cron/process-updates/16 Pipeline failed
ci/crow/cron/process-updates/18 Pipeline failed
ci/crow/cron/process-updates/11 Pipeline failed
ci/crow/cron/process-updates/12 Pipeline failed
ci/crow/cron/process-updates/17 Pipeline failed
ci/crow/cron/process-updates/6 Pipeline failed
ci/crow/cron/process-updates/5 Pipeline failed
## Summary

Adds the curated **patch registry** and wiring that drives bincraft's new package-patching mechanism (see bincraft PR `feat/package-patching`).
Lets specific packages be patched (env/configure/Makevars overrides or source diffs) before pak installs them — including as transitive dependencies — so compiler-/OS-specific failures like RcppParallel's bundled TBB stop cascading.

## What's included

- `local/patches/registry.json` — initial entry: RcppParallel with `RCPP_PARALLEL_USE_TBB=0` for alpine / ubuntu-2604, plus `local/patches/README.md` schema docs.
- `local/validate-patches.R` — validates schema, referenced patch files, and ambiguous overlaps; clean failure + exit 1 (no stacktrace).
- `.pre-commit-config.yaml` — a `validate-patches` hook (re-runs when the registry or the validator changes).
- `local/build-one.R` / `local/build-all.R` — pass `patches = "local/patches"` to `bincraft::build_binary_package()`.
- `specs/2026-06-30-package-patching-design.md` and `plans/2026-06-30-package-patching-implementation.md`.

## ⚠️ Merge ordering (blocker)

This PR adds a `patches = ...` argument to `build_binary_package()` calls.
The `.crow/*.yaml` workflows currently pin bincraft **v4.2.3**, which does not accept that argument — CI will error with `unused argument (patches=...)` until:

1. bincraft **v4.3.0** is released (PR `feat/package-patching`), and
2. the pin is bumped in `.crow/build-all-versions-install-deps.yaml`, `.crow/build-all-versions.yaml`, and `.crow/process-updates.yaml`.

The `.crow` pin bump will be added to this PR once bincraft v4.3.0 is tagged. Do not merge before then.

Reviewed-on: #103
4mo ago
8c5362b7ab
refactor: R_VERSION syntax
Some checks failed
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline was successful
ci/crow/cron/update-package-index-redhat-10-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-10-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-8-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-8-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-9-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-9-arm64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2204-amd64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2204-arm64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2404-amd64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2404-arm64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-322-amd64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-323-amd64 Pipeline was successful
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline failed
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/update-package-index-alpine-323-arm64 Pipeline failed
5mo ago
6faa3772d1 refactor(ci): use multi-R-version images across workflows (#70)
Some checks failed
ci/crow/cron/update-package-index-alpine-322-arm64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-323-amd64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-323-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-10-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-10-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-8-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-8-arm64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-9-amd64 Pipeline is pending
ci/crow/cron/update-package-index-redhat-9-arm64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2204-amd64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2204-arm64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2404-amd64 Pipeline is pending
ci/crow/cron/update-package-index-ubuntu-2404-arm64 Pipeline is pending
ci/crow/cron/update-package-index-alpine-322-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-322-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-322-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-323-amd64 Pipeline failed
ci/crow/cron/process-updates-alpine-323-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline failed
ci/crow/cron/process-updates-redhat-10-amd64 Pipeline failed
ci/crow/cron/process-updates-redhat-10-arm64 Pipeline failed
## Summary

- Switches all 64 `.crow/*.yaml` workflows, the `Justfile`, and the commented-out `build-all-versions-install-deps.yaml` template to the new multi-R-version container images.
- Each image tag drops `-${R_VERSION}` (e.g. `build-env-alpine:3.22-4.5` → `build-env-alpine:3.22`); workflows pick an R version by calling `/opt/R/${R_VERSION}/bin/R` explicitly.
- Pattern-1 workflows (hard-coded image tag) gain a single `R_VERSION:` env var per file.
- Pattern-2 workflows (`build-all-versions-*`, matrix-driven) continue to receive `R_VERSION` via `crow pipeline create --var`.

## Platform → R_VERSION mapping

| Platform | Image | R_VERSION |
| --- | --- | --- |
| alpine 3.22 / 3.23 | `build-env-alpine:3.{22,23}` | 4.5.3 |
| ubuntu jammy / noble | `build-env-ubuntu:{jammy,noble}` | 4.4.3 |
| redhat 8 / 9 | `build-env-redhat:{8,9}` | 4.4.3 |
| redhat 10 | `build-env-redhat:10` | 4.5.3 |

The `alpine-321` audit-only workflows fall back to `build-env-alpine:3.23` since no 3.21 image exists in the new scheme; the audits only query S3/CRAN, so the container OS does not affect correctness.

## Bug fixes folded in

1. All six `weekly-audit-missing-alpine-*.yaml` files were pinned to `alpine:3.23-4.5` regardless of platform.
   Now each one points at its own alpine image.
2. All fourteen `update-package-index-*.yaml` files were pinned to `build-env-ubuntu:noble-4.4` (or `noble-4.5` for `redhat-10-amd64`) regardless of platform.
   Now each one uses its own platform's image.
3. `process-updates-ubuntu-2404-{amd64,arm64}.yaml` had `R_VERSION` of `4.4` (no patch) baked into the tag.
   Aligned to `4.4.3`, matching the audit and rebuild counterparts.

## Smoke targets (post-merge)

One workflow per shape, ready to dry-run:

- `process-updates-alpine-322-amd64.yaml`
- `weekly-rebuild-missing-redhat-9-amd64.yaml`
- `weekly-audit-missing-ubuntu-2204-amd64.yaml`
- `update-package-index-redhat-10-amd64.yaml`
- `archive-missed-packages.yaml`
- `build-all-versions-amd64.yaml` (with its install-deps predecessor)
- `build-r-minor-sensitive-packages.yaml`

## Follow-ups (out of scope, surfaced during review)

- `process-updates-redhat-10-{amd64,arm64}.yaml` set `codename = "rhel9"` in their `upload_package_index(...)` calls.
  Pre-existing bug — should be `"rhel10"`. The `update-package-index-redhat-10-*` counterparts already use `"rhel10"` correctly.
- `update-package-index-redhat-8-amd64.yaml` is missing the `packageVersion("bincraft")` diagnostic line that every sibling file has.
  Pre-existing asymmetry between amd64 and arm64.

## Spec & plan

- Spec: `docs/superpowers/specs/2026-05-25-multi-r-version-images-design.md`
- Plan: `docs/superpowers/plans/2026-05-25-multi-r-version-images.md`

Reviewed-on: #70
5mo ago
27e1471bca
gitignore
1y ago
425e31c880
skip more alpine pkgs
Some checks failed
ci/crow/cron/update-package-index-ubuntu-2404-arm64 Pipeline was successful
ci/crow/cron/update-package-index-ubuntu-2204-arm64 Pipeline was successful
ci/crow/cron/images/3 Pipeline failed
ci/crow/cron/images/1 Pipeline failed
ci/crow/cron/images/2 Pipeline failed
ci/crow/cron/images/4 Pipeline failed
ci/crow/cron/images/6 Pipeline failed
ci/crow/cron/images/5 Pipeline failed
ci/crow/cron/process-updates-alpine-321-arm64 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2204-arm64 Pipeline is running
ci/crow/cron/process-updates-redhat-9-arm64 Pipeline is running
ci/crow/cron/process-updates-ubuntu-2404-arm64 Pipeline is running
ci/crow/cron/process-updates-redhat-8-arm64 Pipeline is running
ci/crow/cron/process-updates-alpine-320-amd64 Pipeline failed
ci/crow/manual/alpine-322-arm64-install-deps Pipeline was successful
ci/crow/manual/alpine-322-amd64-install-deps Pipeline was successful
ci/crow/manual/build-alpine-322-amd64-5 Pipeline failed
ci/crow/cron/process-updates-ubuntu-2404-amd64 Pipeline was successful
ci/crow/cron/process-updates-ubuntu-2204-amd64 Pipeline was successful
ci/crow/manual/build-alpine-322-arm64-1 Pipeline failed
ci/crow/cron/process-updates-redhat-8-amd64 Pipeline was successful
ci/crow/manual/build-alpine-322-amd64-1 Pipeline failed
ci/crow/cron/process-updates-alpine-320-arm64 Pipeline was successful
ci/crow/cron/process-updates-redhat-9-amd64 Pipeline was successful
ci/crow/cron/process-updates-alpine-321-amd64 Pipeline was successful
ci/crow/manual/build-alpine-322-arm64-3 Pipeline failed
ci/crow/manual/build-alpine-322-arm64-2 Pipeline failed
ci/crow/manual/build-alpine-322-amd64-3 Pipeline failed
ci/crow/manual/build-alpine-322-amd64-2 Pipeline failed
ci/crow/manual/build-alpine-322-amd64-4 Pipeline failed
1y ago
11040e75dd
docs: add 'infrastructure' section
1y ago
576125d377
feat: add docs website
1y ago
a65b93a367
chore: clean local/
2y ago
40571c7e3e
chore: ignore docs
2y ago
6ca496b0ac
gitignore
2y ago
a40e81e783
gitignore
2y ago
f851b38106
gitignore
2y ago
d983f391fb
move sqlite to s3, mature metadata summary functions
2y ago
5e32452d09
move to R package
2y ago