feat(stats): collect CDN access logs and publish download statistics #251

Merged

pat-s merged 2 commits from feat/download-stats into main

2026-11-14 12:11:01 +00:00

pat-s commented

2026-11-14 12:03:23 +00:00

Owner

Adds visibility into how cran.rpkgs.com is actually used, from the Bunny access logs, without sending anything to a third party.

What it does

.crow/download-stats.yaml runs hourly: stats/collect.ts reads the last 70 hours of cran-rpkgs logs (Bunny keeps three days) into a new download_stats schema in the build metadata database, then stats/publish.ts writes aggregated counts to stats/downloads.json in the binaries bucket and purges the CDN copy.
stats/classify.ts labels each request (download, index, redirect reason, renv/pak probe, scanner noise) by mirroring edge/rpkgs-router.ts, and splits cloud/CI networks (by ASN) from everything else.
Downloads are de-duplicated per client, package version and UTC day; clients are a salted HMAC of the anonymized /24 prefix, ASN and R platform, so no IP is stored or published.
Our own CI runners (STATS_OWN_NETWORKS) are excluded; collection windows are recorded so the dashboard can show gaps.
cdn.tf declares the zone's existing logging settings and adds json to the CORS extensions so rpkgs.com can read the file.
stats/README.md documents every rule.

The rpkgs.com /stats page that renders this is in rpkgs/website.

Before merging

Add Crow secrets STATS_CLIENT_SALT and STATS_OWN_NETWORKS (values in OpenBao at development/agent-dev/rpkgs-download-stats).
tofu apply for the CORS change.
After merge, register the download-stats cron (0 * * * *).

Verification

bun test stats: 15 tests pass.
Ran collect.ts locally against the live API and database: 14,273 events inserted, a second run inserted 0.
publish.ts --out produced the JSON the website was previewed with; the upload and purge path has only run in CI so far (B2 credentials are CI-only).

🤖 Generated with Claude Code

Adds visibility into how cran.rpkgs.com is actually used, from the Bunny access logs, without sending anything to a third party. ## What it does - `.crow/download-stats.yaml` runs hourly: `stats/collect.ts` reads the last 70 hours of cran-rpkgs logs (Bunny keeps three days) into a new `download_stats` schema in the build metadata database, then `stats/publish.ts` writes aggregated counts to `stats/downloads.json` in the binaries bucket and purges the CDN copy. - `stats/classify.ts` labels each request (download, index, redirect reason, renv/pak probe, scanner noise) by mirroring `edge/rpkgs-router.ts`, and splits cloud/CI networks (by ASN) from everything else. - Downloads are de-duplicated per client, package version and UTC day; clients are a salted HMAC of the anonymized /24 prefix, ASN and R platform, so no IP is stored or published. - Our own CI runners (`STATS_OWN_NETWORKS`) are excluded; collection windows are recorded so the dashboard can show gaps. - `cdn.tf` declares the zone's existing logging settings and adds `json` to the CORS extensions so rpkgs.com can read the file. - `stats/README.md` documents every rule. The rpkgs.com `/stats` page that renders this is in rpkgs/website. ## Before merging - Add Crow secrets `STATS_CLIENT_SALT` and `STATS_OWN_NETWORKS` (values in OpenBao at `development/agent-dev/rpkgs-download-stats`). - `tofu apply` for the CORS change. - After merge, register the `download-stats` cron (`0 * * * *`). ## Verification - `bun test stats`: 15 tests pass. - Ran `collect.ts` locally against the live API and database: 14,273 events inserted, a second run inserted 0. - `publish.ts --out` produced the JSON the website was previewed with; the upload and purge path has only run in CI so far (B2 credentials are CI-only). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

pat-s added 1 commit

2026-11-14 12:03:24 +00:00

feat(stats): collect CDN access logs and publish download statistics 2614d40bd9

Add an hourly download-stats pipeline that reads the cran-rpkgs Bunny
access logs into a download_stats schema in the build metadata database
and publishes aggregated counts to /stats/downloads.json for the
rpkgs.com /stats page.

- Classify each request (download, index, redirect reason, probe, noise)
  mirroring the edge router, and split cloud/CI networks from the rest.
- De-duplicate downloads per client, package version and day, and key
  clients on a salted HMAC so no IP is stored or published.
- Exclude our own CI runners and record run windows to show gaps.
- Declare the zone's logging settings and allow CORS for JSON.

pat-s force-pushed feat/download-stats from 2614d40bd9 to a4c32725fd

2026-11-14 12:10:04 +00:00

Compare

pat-s merged commit 72440f9cf5 into main

2026-11-14 12:11:01 +00:00

pat-s deleted branch feat/download-stats

2026-11-14 12:11:01 +00:00

pat-s referenced this pull request from a commit

2026-11-14 12:11:02 +00:00

feat(stats): collect CDN access logs and publish download statistics (#251)