feat(stats): collect CDN access logs and publish download statistics (#251)
Adds visibility into how cran.rpkgs.com is actually used, from the Bunny access logs, without sending anything to a third party. ## What it does - `.crow/download-stats.yaml` runs hourly: `stats/collect.ts` reads the last 70 hours of cran-rpkgs logs (Bunny keeps three days) into a new `download_stats` schema in the build metadata database, then `stats/publish.ts` writes aggregated counts to `stats/downloads.json` in the binaries bucket and purges the CDN copy. - `stats/classify.ts` labels each request (download, index, redirect reason, renv/pak probe, scanner noise) by mirroring `edge/rpkgs-router.ts`, and splits cloud/CI networks (by ASN) from everything else. - Downloads are de-duplicated per client, package version and UTC day; clients are a salted HMAC of the anonymized /24 prefix, ASN and R platform, so no IP is stored or published. - Our own CI runners (`STATS_OWN_NETWORKS`) are excluded; collection windows are recorded so the dashboard can show gaps. - `cdn.tf` declares the zone's existing logging settings and adds `json` to the CORS extensions so rpkgs.com can read the file. - `stats/README.md` documents every rule. The rpkgs.com `/stats` page that renders this is in rpkgs/website. ## Before merging - Add Crow secrets `STATS_CLIENT_SALT` and `STATS_OWN_NETWORKS` (values in OpenBao at `development/agent-dev/rpkgs-download-stats`). - `tofu apply` for the CORS change. - After merge, register the `download-stats` cron (`0 * * * *`). ## Verification - `bun test stats`: 15 tests pass. - Ran `collect.ts` locally against the live API and database: 14,273 events inserted, a second run inserted 0. - `publish.ts --out` produced the JSON the website was previewed with; the upload and purge path has only run in CI so far (B2 credentials are CI-only). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #251
This commit is contained in:
parent
56dc06734c
commit
72440f9cf5
1 changed files with 1283 additions and 0 deletions
79
.crow/download-stats.yaml
Normal file
79
.crow/download-stats.yaml
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
# Collect CDN access logs and publish the public download statistics.
|
||||
#
|
||||
# Each run reads the last 70 hours of cran-rpkgs access logs from Bunny into
|
||||
# `download_stats.events` (build metadata Postgres), then rewrites
|
||||
# https://cran.rpkgs.com/stats/downloads.json, which the rpkgs.com /stats page
|
||||
# reads. Bunny keeps logs for only three days, so a run every hour leaves room
|
||||
# for two days of failed runs before anything is lost. Re-reading the window is
|
||||
# safe: events are keyed by Bunny's request id.
|
||||
#
|
||||
# The classification rules and what is excluded are documented in
|
||||
# `stats/README.md`.
|
||||
#
|
||||
# Register the `download-stats` cron (hourly, `0 * * * *`) in the crow UI, or
|
||||
# run manually:
|
||||
# crow pipeline create --branch main --var download_stats=true devxy/build-cran-binaries
|
||||
#
|
||||
# Secrets:
|
||||
# STATS_CLIENT_SALT HMAC key for the client hash. Rotating it resets
|
||||
# distinct-client counts, so keep it stable.
|
||||
# STATS_OWN_NETWORKS Comma-separated CIDRs of our own CI runners.
|
||||
# Both live in OpenBao at development/agent-dev/rpkgs-download-stats.
|
||||
variables:
|
||||
download_stats:
|
||||
description: 'Collect and publish download statistics. Also gates this pipeline.'
|
||||
options:
|
||||
- 'true'
|
||||
- 'false'
|
||||
default: 'false'
|
||||
|
||||
when:
|
||||
- event: manual
|
||||
evaluate: 'download_stats == "true"'
|
||||
- event: cron
|
||||
cron: download-stats
|
||||
|
||||
skip_clone: true
|
||||
|
||||
labels:
|
||||
group: rpkgs-amd64
|
||||
|
||||
steps:
|
||||
- name: 'Collect and publish download stats'
|
||||
image: oven/bun:1.4.4-alpine
|
||||
pull: true
|
||||
environment:
|
||||
BUNNYNET_API_KEY:
|
||||
from_secret: BUNNYNET_API_KEY
|
||||
PGPASS:
|
||||
from_secret: PGPASS
|
||||
B2_S3_ACCESS_KEY:
|
||||
from_secret: B2_S3_ACCESS_KEY
|
||||
B2_S3_SECRET_KEY:
|
||||
from_secret: B2_S3_SECRET_KEY
|
||||
STATS_CLIENT_SALT:
|
||||
from_secret: STATS_CLIENT_SALT
|
||||
STATS_OWN_NETWORKS:
|
||||
from_secret: STATS_OWN_NETWORKS
|
||||
REPO_RO_TOKEN:
|
||||
from_secret: REPO_RO_TOKEN
|
||||
commands:
|
||||
- apk add --no-cache -q git
|
||||
- git clone -q --depth 1 --branch "$CI_COMMIT_BRANCH" https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
||||
- bun test stats
|
||||
- bun stats/collect.ts
|
||||
- bun stats/publish.ts
|
||||
backend_options:
|
||||
kubernetes:
|
||||
resources:
|
||||
requests:
|
||||
memory: 256Mi
|
||||
cpu: 250m
|
||||
limits:
|
||||
memory: 1Gi
|
||||
cpu: 1000m
|
||||
tolerations:
|
||||
- key: 'CI'
|
||||
operator: 'Equal'
|
||||
value: 'true'
|
||||
effect: 'NoSchedule'
|
||||
Loading…
Reference in a new issue