build-cran-binaries/stats

Download statistics

This directory turns the cran.rpkgs.com CDN access logs into the public download numbers on rpkgs.com/stats. The download-stats pipeline runs it every hour.

Data flow

1. collect.ts reads the last 70 hours of access logs from the Bunny Logging API v2 and inserts each request into download_stats.events in the build metadata database.
2. classify.ts derives what each request was (package download, index fetch, redirect, probe, noise) from its path, status and User-Agent.
3. publish.ts aggregates the events into stats/downloads.json in the binaries bucket and purges its CDN copy, so it is served at https://cran.rpkgs.com/stats/downloads.json.

Bunny keeps access logs for three days. Each hourly run re-reads almost all of that window, and events are keyed by Bunny's request id, so re-reading never double counts and up to two days of failed runs lose nothing. Every run is recorded in download_stats.runs, and the dashboard marks any hour that no run covered.

Privacy

Bunny anonymizes IPs at the edge to /24 (IPv4) or /64 (IPv6) before they reach the logs. Not even that prefix is stored: each event keeps only client, an HMAC (keyed with STATS_CLIENT_SALT) of the prefix, the network (ASN) and the client platform from the User-Agent. The public JSON contains counts only, never a client hash, IP or User-Agent. Countries seen from fewer than three clients are folded into "other".

What counts

Metric	Definition
Download	A distinct (client, package, version) served on one UTC day. Raw tarball requests are reported separately, because one CI job can re-download the same package many times a day.
Client	A distinct client hash that fetched an index or a package. Several machines behind one /24 count once, and one laptop on two networks counts twice, so this is an estimate.
Client-day	A client active on a given day, roughly one install session.

The same R session seen with and without renv's User-Agent prefix counts as one client, since clientPlatform() keys on the R part of the User-Agent.

What is separated or excluded

• Cloud and CI networks are kept but reported separately: Microsoft/GitHub, AWS, Google, Hetzner, OVH and other hosting ASNs, plus any network whose name says hosting, cloud or data centre.
• Our own CI runners (STATS_OWN_NETWORKS) are excluded from every public number and only counted as excluded requests.
• Internal redirects (minor_routing, slot_resolution, https) are not counted, only the request they lead to.
• Redirects away from us are counted as their own metrics: unsupported_r_minor (demand for R versions we do not build), archive_fallback (old versions we have no binary for), unknown_distro and macos.
• Probes: renv and pak request __api__/status to check for Posit Package Manager.
• Noise: vulnerability scanners, crawlers and anything outside the repository layout.

Changing the rules

Change classify.ts, add a test case to classify.test.ts and bump RULESET. Then recompute every stored event:

bun stats/collect.ts --reclassify

The raw path, status, User-Agent and network are stored for exactly this reason. Whether a request came from our own runners is decided at ingest, so changing STATS_OWN_NETWORKS only affects new events.

Running locally

The secrets are in OpenBao: the Bunny key at development/agent-dev/bunny, the database password at development/agent-dev/rpkgs-metadata-db and the salt and own networks at development/agent-dev/rpkgs-download-stats.

bun test stats
bun stats/collect.ts
# write the JSON locally instead of uploading it
bun stats/publish.ts --out downloads.json