pat-s
72440f9cf5
Adds visibility into how cran.rpkgs.com is actually used, from the Bunny access logs, without sending anything to a third party. ## What it does - `.crow/download-stats.yaml` runs hourly: `stats/collect.ts` reads the last 70 hours of cran-rpkgs logs (Bunny keeps three days) into a new `download_stats` schema in the build metadata database, then `stats/publish.ts` writes aggregated counts to `stats/downloads.json` in the binaries bucket and purges the CDN copy. - `stats/classify.ts` labels each request (download, index, redirect reason, renv/pak probe, scanner noise) by mirroring `edge/rpkgs-router.ts`, and splits cloud/CI networks (by ASN) from everything else. - Downloads are de-duplicated per client, package version and UTC day; clients are a salted HMAC of the anonymized /24 prefix, ASN and R platform, so no IP is stored or published. - Our own CI runners (`STATS_OWN_NETWORKS`) are excluded; collection windows are recorded so the dashboard can show gaps. - `cdn.tf` declares the zone's existing logging settings and adds `json` to the CORS extensions so rpkgs.com can read the file. - `stats/README.md` documents every rule. The rpkgs.com `/stats` page that renders this is in rpkgs/website. ## Before merging - Add Crow secrets `STATS_CLIENT_SALT` and `STATS_OWN_NETWORKS` (values in OpenBao at `development/agent-dev/rpkgs-download-stats`). - `tofu apply` for the CORS change. - After merge, register the `download-stats` cron (`0 * * * *`). ## Verification - `bun test stats`: 15 tests pass. - Ran `collect.ts` locally against the live API and database: 14,273 events inserted, a second run inserted 0. - `publish.ts --out` produced the JSON the website was previewed with; the upload and purge path has only run in CI so far (B2 credentials are CI-only). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #251
75 lines
2.3 KiB
TypeScript
75 lines
2.3 KiB
TypeScript
import { SQL } from 'bun';
|
|
|
|
/**
|
|
* Download events live in their own `download_stats` schema in the build
|
|
* metadata database, next to (but never joined with) `single_builds`.
|
|
*
|
|
* No IP address is stored, not even the /24 prefix Bunny logs: only a salted
|
|
* HMAC of it (`client`) and whether it matched one of our own networks.
|
|
*/
|
|
const SCHEMA = `
|
|
create schema if not exists download_stats;
|
|
|
|
create table if not exists download_stats.events (
|
|
request_id text primary key,
|
|
ts timestamptz not null,
|
|
zone_id integer not null,
|
|
host text,
|
|
status smallint not null,
|
|
cache_status text,
|
|
bytes bigint not null,
|
|
path text not null,
|
|
user_agent text,
|
|
country text,
|
|
asn integer,
|
|
asn_org text,
|
|
client text not null,
|
|
own boolean not null,
|
|
ruleset smallint not null,
|
|
kind text not null,
|
|
outcome text not null,
|
|
redirect_reason text,
|
|
slot text,
|
|
arch text,
|
|
os text,
|
|
r_minor text,
|
|
r_version text,
|
|
package text,
|
|
version text,
|
|
tool text not null,
|
|
network text not null
|
|
);
|
|
|
|
create index if not exists events_ts_idx on download_stats.events (ts);
|
|
create index if not exists events_package_idx on download_stats.events (package) where kind = 'download';
|
|
|
|
-- One row per collection run. The union of successful windows is what the
|
|
-- dashboard reports as covered; anything outside it is shown as a gap.
|
|
create table if not exists download_stats.runs (
|
|
id bigserial primary key,
|
|
zone_id integer not null,
|
|
started_at timestamptz not null default now(),
|
|
window_from timestamptz not null,
|
|
window_to timestamptz not null,
|
|
fetched integer not null,
|
|
inserted integer not null
|
|
);
|
|
`;
|
|
|
|
export function connect(): SQL {
|
|
const password = process.env.PGPASS;
|
|
if (!password) throw new Error('PGPASS is not set');
|
|
return new SQL({
|
|
hostname: process.env.PGHOST ?? 'r-binaries.devxy.io',
|
|
port: Number(process.env.PGPORT ?? 15432),
|
|
database: process.env.PGDATABASE ?? 'build_metadata',
|
|
username: process.env.PGUSER ?? 'rpkgs',
|
|
password,
|
|
tls: true,
|
|
max: 2,
|
|
});
|
|
}
|
|
|
|
export async function migrate(sql: SQL): Promise<void> {
|
|
await sql.unsafe(SCHEMA);
|
|
}
|