pat-s
72440f9cf5
Adds visibility into how cran.rpkgs.com is actually used, from the Bunny access logs, without sending anything to a third party. ## What it does - `.crow/download-stats.yaml` runs hourly: `stats/collect.ts` reads the last 70 hours of cran-rpkgs logs (Bunny keeps three days) into a new `download_stats` schema in the build metadata database, then `stats/publish.ts` writes aggregated counts to `stats/downloads.json` in the binaries bucket and purges the CDN copy. - `stats/classify.ts` labels each request (download, index, redirect reason, renv/pak probe, scanner noise) by mirroring `edge/rpkgs-router.ts`, and splits cloud/CI networks (by ASN) from everything else. - Downloads are de-duplicated per client, package version and UTC day; clients are a salted HMAC of the anonymized /24 prefix, ASN and R platform, so no IP is stored or published. - Our own CI runners (`STATS_OWN_NETWORKS`) are excluded; collection windows are recorded so the dashboard can show gaps. - `cdn.tf` declares the zone's existing logging settings and adds `json` to the CORS extensions so rpkgs.com can read the file. - `stats/README.md` documents every rule. The rpkgs.com `/stats` page that renders this is in rpkgs/website. ## Before merging - Add Crow secrets `STATS_CLIENT_SALT` and `STATS_OWN_NETWORKS` (values in OpenBao at `development/agent-dev/rpkgs-download-stats`). - `tofu apply` for the CORS change. - After merge, register the `download-stats` cron (`0 * * * *`). ## Verification - `bun test stats`: 15 tests pass. - Ran `collect.ts` locally against the live API and database: 14,273 events inserted, a second run inserted 0. - `publish.ts --out` produced the JSON the website was previewed with; the upload and purge path has only run in CI so far (B2 credentials are CI-only). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #251
63 lines
4.1 KiB
Markdown
63 lines
4.1 KiB
Markdown
# Download statistics
|
|
|
|
This directory turns the cran.rpkgs.com CDN access logs into the public download numbers on [rpkgs.com/stats](https://rpkgs.com/stats).
|
|
The `download-stats` pipeline runs it every hour.
|
|
|
|
## Data flow
|
|
|
|
1. `collect.ts` reads the last 70 hours of access logs from the Bunny Logging API v2 and inserts each request into `download_stats.events` in the build metadata database.
|
|
2. `classify.ts` derives what each request was (package download, index fetch, redirect, probe, noise) from its path, status and User-Agent.
|
|
3. `publish.ts` aggregates the events into `stats/downloads.json` in the binaries bucket and purges its CDN copy, so it is served at <https://cran.rpkgs.com/stats/downloads.json>.
|
|
|
|
Bunny keeps access logs for three days.
|
|
Each hourly run re-reads almost all of that window, and events are keyed by Bunny's request id, so re-reading never double counts and up to two days of failed runs lose nothing.
|
|
Every run is recorded in `download_stats.runs`, and the dashboard marks any hour that no run covered.
|
|
|
|
## Privacy
|
|
|
|
Bunny anonymizes IPs at the edge to /24 (IPv4) or /64 (IPv6) before they reach the logs.
|
|
Not even that prefix is stored: each event keeps only `client`, an HMAC (keyed with `STATS_CLIENT_SALT`) of the prefix, the network (ASN) and the client platform from the User-Agent.
|
|
The public JSON contains counts only, never a client hash, IP or User-Agent.
|
|
Countries seen from fewer than three clients are folded into "other".
|
|
|
|
## What counts
|
|
|
|
| Metric | Definition |
|
|
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| Download | A distinct (client, package, version) served on one UTC day. Raw tarball requests are reported separately, because one CI job can re-download the same package many times a day. |
|
|
| Client | A distinct `client` hash that fetched an index or a package. Several machines behind one /24 count once, and one laptop on two networks counts twice, so this is an estimate. |
|
|
| Client-day | A client active on a given day, roughly one install session. |
|
|
|
|
The same R session seen with and without renv's User-Agent prefix counts as one client, since `clientPlatform()` keys on the R part of the User-Agent.
|
|
|
|
## What is separated or excluded
|
|
|
|
- **Cloud and CI networks** are kept but reported separately: Microsoft/GitHub, AWS, Google, Hetzner, OVH and other hosting ASNs, plus any network whose name says hosting, cloud or data centre.
|
|
- **Our own CI runners** (`STATS_OWN_NETWORKS`) are excluded from every public number and only counted as excluded requests.
|
|
- **Internal redirects** (`minor_routing`, `slot_resolution`, `https`) are not counted, only the request they lead to.
|
|
- **Redirects away from us** are counted as their own metrics: `unsupported_r_minor` (demand for R versions we do not build), `archive_fallback` (old versions we have no binary for), `unknown_distro` and `macos`.
|
|
- **Probes**: renv and pak request `__api__/status` to check for Posit Package Manager.
|
|
- **Noise**: vulnerability scanners, crawlers and anything outside the repository layout.
|
|
|
|
## Changing the rules
|
|
|
|
Change `classify.ts`, add a test case to `classify.test.ts` and bump `RULESET`.
|
|
Then recompute every stored event:
|
|
|
|
```sh
|
|
bun stats/collect.ts --reclassify
|
|
```
|
|
|
|
The raw path, status, User-Agent and network are stored for exactly this reason.
|
|
Whether a request came from our own runners is decided at ingest, so changing `STATS_OWN_NETWORKS` only affects new events.
|
|
|
|
## Running locally
|
|
|
|
The secrets are in OpenBao: the Bunny key at `development/agent-dev/bunny`, the database password at `development/agent-dev/rpkgs-metadata-db` and the salt and own networks at `development/agent-dev/rpkgs-download-stats`.
|
|
|
|
```sh
|
|
bun test stats
|
|
bun stats/collect.ts
|
|
# write the JSON locally instead of uploading it
|
|
bun stats/publish.ts --out downloads.json
|
|
```
|