pat-s
72440f9cf5
Adds visibility into how cran.rpkgs.com is actually used, from the Bunny access logs, without sending anything to a third party. ## What it does - `.crow/download-stats.yaml` runs hourly: `stats/collect.ts` reads the last 70 hours of cran-rpkgs logs (Bunny keeps three days) into a new `download_stats` schema in the build metadata database, then `stats/publish.ts` writes aggregated counts to `stats/downloads.json` in the binaries bucket and purges the CDN copy. - `stats/classify.ts` labels each request (download, index, redirect reason, renv/pak probe, scanner noise) by mirroring `edge/rpkgs-router.ts`, and splits cloud/CI networks (by ASN) from everything else. - Downloads are de-duplicated per client, package version and UTC day; clients are a salted HMAC of the anonymized /24 prefix, ASN and R platform, so no IP is stored or published. - Our own CI runners (`STATS_OWN_NETWORKS`) are excluded; collection windows are recorded so the dashboard can show gaps. - `cdn.tf` declares the zone's existing logging settings and adds `json` to the CORS extensions so rpkgs.com can read the file. - `stats/README.md` documents every rule. The rpkgs.com `/stats` page that renders this is in rpkgs/website. ## Before merging - Add Crow secrets `STATS_CLIENT_SALT` and `STATS_OWN_NETWORKS` (values in OpenBao at `development/agent-dev/rpkgs-download-stats`). - `tofu apply` for the CORS change. - After merge, register the `download-stats` cron (`0 * * * *`). ## Verification - `bun test stats`: 15 tests pass. - Ran `collect.ts` locally against the live API and database: 14,273 events inserted, a second run inserted 0. - `publish.ts --out` produced the JSON the website was previewed with; the upload and purge path has only run in CI so far (B2 credentials are CI-only). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #251
79 lines
2.5 KiB
YAML
79 lines
2.5 KiB
YAML
# Collect CDN access logs and publish the public download statistics.
|
|
#
|
|
# Each run reads the last 70 hours of cran-rpkgs access logs from Bunny into
|
|
# `download_stats.events` (build metadata Postgres), then rewrites
|
|
# https://cran.rpkgs.com/stats/downloads.json, which the rpkgs.com /stats page
|
|
# reads. Bunny keeps logs for only three days, so a run every hour leaves room
|
|
# for two days of failed runs before anything is lost. Re-reading the window is
|
|
# safe: events are keyed by Bunny's request id.
|
|
#
|
|
# The classification rules and what is excluded are documented in
|
|
# `stats/README.md`.
|
|
#
|
|
# Register the `download-stats` cron (hourly, `0 * * * *`) in the crow UI, or
|
|
# run manually:
|
|
# crow pipeline create --branch main --var download_stats=true devxy/build-cran-binaries
|
|
#
|
|
# Secrets:
|
|
# STATS_CLIENT_SALT HMAC key for the client hash. Rotating it resets
|
|
# distinct-client counts, so keep it stable.
|
|
# STATS_OWN_NETWORKS Comma-separated CIDRs of our own CI runners.
|
|
# Both live in OpenBao at development/agent-dev/rpkgs-download-stats.
|
|
variables:
|
|
download_stats:
|
|
description: 'Collect and publish download statistics. Also gates this pipeline.'
|
|
options:
|
|
- 'true'
|
|
- 'false'
|
|
default: 'false'
|
|
|
|
when:
|
|
- event: manual
|
|
evaluate: 'download_stats == "true"'
|
|
- event: cron
|
|
cron: download-stats
|
|
|
|
skip_clone: true
|
|
|
|
labels:
|
|
group: rpkgs-amd64
|
|
|
|
steps:
|
|
- name: 'Collect and publish download stats'
|
|
image: oven/bun:1.4.4-alpine
|
|
pull: true
|
|
environment:
|
|
BUNNYNET_API_KEY:
|
|
from_secret: BUNNYNET_API_KEY
|
|
PGPASS:
|
|
from_secret: PGPASS
|
|
B2_S3_ACCESS_KEY:
|
|
from_secret: B2_S3_ACCESS_KEY
|
|
B2_S3_SECRET_KEY:
|
|
from_secret: B2_S3_SECRET_KEY
|
|
STATS_CLIENT_SALT:
|
|
from_secret: STATS_CLIENT_SALT
|
|
STATS_OWN_NETWORKS:
|
|
from_secret: STATS_OWN_NETWORKS
|
|
REPO_RO_TOKEN:
|
|
from_secret: REPO_RO_TOKEN
|
|
commands:
|
|
- apk add --no-cache -q git
|
|
- git clone -q --depth 1 --branch "$CI_COMMIT_BRANCH" https://pat-s:$$REPO_RO_TOKEN@git.devxy.io/devxy/build-cran-binaries.git .
|
|
- bun test stats
|
|
- bun stats/collect.ts
|
|
- bun stats/publish.ts
|
|
backend_options:
|
|
kubernetes:
|
|
resources:
|
|
requests:
|
|
memory: 256Mi
|
|
cpu: 250m
|
|
limits:
|
|
memory: 1Gi
|
|
cpu: 1000m
|
|
tolerations:
|
|
- key: 'CI'
|
|
operator: 'Equal'
|
|
value: 'true'
|
|
effect: 'NoSchedule'
|