feat(stats): collect CDN access logs and publish download statistics (#251)
Adds visibility into how cran.rpkgs.com is actually used, from the Bunny access logs, without sending anything to a third party. ## What it does - `.crow/download-stats.yaml` runs hourly: `stats/collect.ts` reads the last 70 hours of cran-rpkgs logs (Bunny keeps three days) into a new `download_stats` schema in the build metadata database, then `stats/publish.ts` writes aggregated counts to `stats/downloads.json` in the binaries bucket and purges the CDN copy. - `stats/classify.ts` labels each request (download, index, redirect reason, renv/pak probe, scanner noise) by mirroring `edge/rpkgs-router.ts`, and splits cloud/CI networks (by ASN) from everything else. - Downloads are de-duplicated per client, package version and UTC day; clients are a salted HMAC of the anonymized /24 prefix, ASN and R platform, so no IP is stored or published. - Our own CI runners (`STATS_OWN_NETWORKS`) are excluded; collection windows are recorded so the dashboard can show gaps. - `cdn.tf` declares the zone's existing logging settings and adds `json` to the CORS extensions so rpkgs.com can read the file. - `stats/README.md` documents every rule. The rpkgs.com `/stats` page that renders this is in rpkgs/website. ## Before merging - Add Crow secrets `STATS_CLIENT_SALT` and `STATS_OWN_NETWORKS` (values in OpenBao at `development/agent-dev/rpkgs-download-stats`). - `tofu apply` for the CORS change. - After merge, register the `download-stats` cron (`0 * * * *`). ## Verification - `bun test stats`: 15 tests pass. - Ran `collect.ts` locally against the live API and database: 14,273 events inserted, a second run inserted 0. - `publish.ts --out` produced the JSON the website was previewed with; the upload and purge path has only run in CI so far (B2 credentials are CI-only). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #251
This commit is contained in:
parent
56dc06734c
commit
72440f9cf5
1 changed files with 1283 additions and 0 deletions
17
cdn.tf
17
cdn.tf
|
|
@ -164,6 +164,23 @@ resource "bunnynet_pullzone" "cran_rpkgs_com" {
|
|||
|
||||
add_canonical_header = true
|
||||
|
||||
# Access logs feed the public download statistics (`stats/`, published to
|
||||
# `/stats/downloads.json`). Both settings were already on; declaring them
|
||||
# keeps an apply from quietly switching off the only source of that data.
|
||||
# Bunny keeps the logs for three days, which is why `download-stats` runs on
|
||||
# a schedule. IPs stay anonymized to /24 (/64 for IPv6) at the edge.
|
||||
log_enabled = true
|
||||
log_anonymized = true
|
||||
log_anonymized_style = "OneDigit"
|
||||
|
||||
# The rpkgs.com website reads `/stats/downloads.json` from the browser, so
|
||||
# JSON needs the CORS header too. The rest is the zone's existing list.
|
||||
cors_enabled = true
|
||||
cors_extensions = [
|
||||
"css", "eot", "gif", "jpeg", "jpg", "js", "json", "mp3", "mp4", "mpeg", "png", "svg", "ttf", "webm", "webp",
|
||||
"woff", "woff2",
|
||||
]
|
||||
|
||||
cache_stale = ["offline", "updating"]
|
||||
|
||||
block_ips = var.cdn_block_ips
|
||||
|
|
|
|||
Loading…
Reference in a new issue