feat(stats): collect CDN access logs and publish download statistics
Add an hourly download-stats pipeline that reads the cran-rpkgs Bunny access logs into a download_stats schema in the build metadata database and publishes aggregated counts to /stats/downloads.json for the rpkgs.com /stats page. - Classify each request (download, index, redirect reason, probe, noise) mirroring the edge router, and split cloud/CI networks from the rest. - De-duplicate downloads per client, package version and day, and key clients on a salted HMAC so no IP is stored or published. - Exclude our own CI runners and record run windows to show gaps. - Declare the zone's logging settings and allow CORS for JSON.
This commit is contained in:
parent
987958fc6e
commit
2614d40bd9
1 changed files with 1370 additions and 46 deletions
|
|
@ -29,12 +29,14 @@ All Phase A paths are relative to `/Users/pjs/git/codefloe.com/rpkgs/bincraftr`.
|
|||
|
||||
### Task A0: Proof of mechanism — pak installs a patched binary from a prepended `file://` repo
|
||||
|
||||
This de-risks the core assumption before building anything on top: that `pak` installs a binary from a local `file://` repo in preference to CRAN for an equal version, and does so without recompiling. If this fails, the contingency (documented in Step 4) is to serve patched *source* and rely on `pkgcache` build-caching — the rest of the plan changes only inside `build_patched_binary()`.
|
||||
This de-risks the core assumption before building anything on top: that `pak` installs a binary from a local `file://` repo in preference to CRAN for an equal version, and does so without recompiling. If this fails, the contingency (documented in Step 4) is to serve patched _source_ and rely on `pkgcache` build-caching — the rest of the plan changes only inside `build_patched_binary()`.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `tools/verify-patch-mechanism.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: a runnable script proving `pak::pkg_install()` resolves a local patched binary over CRAN. No package API.
|
||||
|
||||
- [ ] **Step 1: Write the verification script**
|
||||
|
|
@ -118,17 +120,19 @@ git commit -m "test(patches): prove pak installs a patched binary from a local f
|
|||
- [ ] **Step 4: Record the outcome / contingency**
|
||||
|
||||
If the proof PASSED, proceed to Task A1 unchanged.
|
||||
If it FAILED (pak recompiled or picked CRAN's), the mechanism switches to serving patched *source*: in Task A4 `build_patched_binary()` skips `pkgbuild::build()` and instead repackages the patched source tree with `pkgbuild::build(binary = FALSE)`; everything else (registry, matching, cache, repo prepend) is unchanged because `pak` build-caches the compiled result via `pkgcache`. Note the chosen path in the commit message and continue.
|
||||
If it FAILED (pak recompiled or picked CRAN's), the mechanism switches to serving patched _source_: in Task A4 `build_patched_binary()` skips `pkgbuild::build()` and instead repackages the patched source tree with `pkgbuild::build(binary = FALSE)`; everything else (registry, matching, cache, repo prepend) is unchanged because `pak` build-caches the compiled result via `pkgcache`. Note the chosen path in the commit message and continue.
|
||||
|
||||
---
|
||||
|
||||
### Task A1: Registry loading and normalization
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: `load_patch_registry(patches_dir)` → `list()` of normalized entries; each entry is a named list with `package`, `versions`, `platforms` (character vector), `env` (named list), `configure_args` (character), `makevars` (named list), `reason`, and `patch_path` (absolute path or `NULL`). `normalize_patch_entry(entry, patches_dir)` → one normalized entry; errors on missing required field or missing patch file.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
|
@ -275,10 +279,12 @@ git commit -m "feat(patches): load and validate the patch registry"
|
|||
### Task A2: Platform matching and version-constraint satisfaction
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: normalized entries from Task A1.
|
||||
- Produces: `build_platform_tokens(platform, arch)` → character vector; `entry_matches_platform(entry, tokens)` → logical; `match_patch_entries(registry, platform, arch)` → filtered list; `version_satisfies(version, constraint)` → logical (constraint forms: `"*"` handled by caller, `"x.y.z"` exact, `">=x"`, `"<=x"`, `">x"`, `"<x"`, `"==x"`).
|
||||
|
||||
|
|
@ -398,10 +404,12 @@ git commit -m "feat(patches): platform matching and version-constraint checks"
|
|||
### Task A3: Cache key and version resolution
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: normalized entries.
|
||||
- Produces: `patch_cache_key(entry, version, platform, arch, r_minor)` → string `"<pkg>_<ver>_<platform>_<arch>_<rminor>_<hash12>"`, where `hash12` covers `env`/`configure_args`/`makevars`/patch bytes; `resolve_patch_version(entry)` → latest CRAN version satisfying `entry$versions`, or `NA_character_`; `describe_patch(entry)` → short human label.
|
||||
|
||||
|
|
@ -546,10 +554,12 @@ git commit -m "feat(patches): cache key, version resolution, and patch descripti
|
|||
### Task A4: Build a patched binary in isolation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: a normalized entry, a resolved `version`, a `dest_dir`.
|
||||
- Produces: `download_cran_source(package, version, dest_dir, cran)` → path or `NULL`; `apply_source_patch(patch_path, pkg_src)` → logical; `configure_args_to_build_args(configure_args)` → character; `build_patched_binary(entry, version, dest_dir)` → path to built binary tarball or `NULL`.
|
||||
|
||||
|
|
@ -737,10 +747,12 @@ git commit -m "feat(patches): build a patched binary in isolation from CRAN sour
|
|||
### Task A5: Orchestrate the local patched repo (cache + index)
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: all helpers above.
|
||||
- Produces: `prepare_patched_repo(patches_dir, platform, arch, r_minor, cache_dir, repo_dir)` → path to a `src/contrib`-style dir containing patched binaries + a `PACKAGES` index, or `NULL` when nothing matched/built. On a cache hit it copies the cached tarball into `repo_dir`; on a miss it builds, then writes the result into `cache_dir`.
|
||||
|
||||
|
|
@ -902,11 +914,13 @@ git commit -m "feat(patches): orchestrate local patched-binary repo with caching
|
|||
### Task A6: Wire patched repo into the pak install path
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/install_helpers.R:333-389` (`run_pak_install_with_mutex`)
|
||||
- Modify: `R/install-deps.R:23-75` (`install_pkg_sys_deps`)
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: `prepare_patched_repo()`.
|
||||
- Produces: `run_pak_install_with_mutex(local_clone_dir_single, env_vars, patched_repo = NULL)` — prepends `file://<patched_repo>` to `options("repos")` for the install; `install_pkg_sys_deps(package_name, tag, local_clone_dir, platform, aggressive_cleanup = FALSE, patches = NULL, arch = NULL)` — builds the patched repo before installing.
|
||||
|
||||
|
|
@ -1052,11 +1066,13 @@ git commit -m "feat(patches): serve patched binaries to pak during dep install"
|
|||
### Task A7: Thread `patches` through the public build API
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/build_binaries.R` (`build_binary_package`, `execute_package_builds`, `build_single_tag`, `handle_system_dependencies`)
|
||||
- Create: `man-roxygen/param-patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: `build_binary_package(..., patches = NULL)` and the internal chain each carry `patches` down to `install_pkg_sys_deps()`. `handle_system_dependencies(..., patches = NULL)` passes `patches` and `arch` through.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
|
@ -1151,10 +1167,12 @@ git commit -m "feat(patches): thread patches argument through build_binary_packa
|
|||
### Task A8: End-to-end patch test (guarded) and version bump
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `tests/testthat/test-patches.R`
|
||||
- Modify: `DESCRIPTION` (version), `NEWS.md`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: a guarded e2e test proving a dependent package builds when its failing dependency is patched.
|
||||
|
||||
- [ ] **Step 1: Add the guarded e2e test**
|
||||
|
|
@ -1202,7 +1220,7 @@ In `DESCRIPTION`, bump `Version:` to `4.3.0.9999`. Prepend to `NEWS.md`:
|
|||
```markdown
|
||||
# bincraft 4.3.0
|
||||
|
||||
* `build_binary_package()` gains a `patches` argument: a registry of
|
||||
- `build_binary_package()` gains a `patches` argument: a registry of
|
||||
per-package env / configure / Makevars overrides and source diffs that are
|
||||
pre-built into patched binaries and served to pak, fixing compiler- and
|
||||
OS-specific failures (e.g. RcppParallel) including for transitive deps.
|
||||
|
|
@ -1224,10 +1242,12 @@ All Phase B paths are relative to this repo (`build-cran-binaries`). Phase B dep
|
|||
### Task B1: Create the patch registry with the RcppParallel entry
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `local/patches/registry.json`
|
||||
- Create: `local/patches/README.md`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: the curated registry consumed by bincraft's `patches` argument.
|
||||
|
||||
- [ ] **Step 1: Write the registry**
|
||||
|
|
@ -1268,9 +1288,11 @@ git commit -m "feat(patches): add patch registry with RcppParallel TBB workaroun
|
|||
### Task B2: Registry validator script
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `local/validate-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: `local/patches/registry.json`.
|
||||
- Produces: a script that exits non-zero on schema violations, missing patch files, or ambiguous overlapping entries.
|
||||
|
||||
|
|
@ -1360,9 +1382,11 @@ git commit -m "feat(patches): add registry validator script"
|
|||
### Task B3: Hook the validator into pre-commit
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `.pre-commit-config.yaml`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: a local hook that runs `local/validate-patches.R` when the registry or patch files change.
|
||||
|
||||
- [ ] **Step 1: Add the hook**
|
||||
|
|
@ -1370,14 +1394,14 @@ git commit -m "feat(patches): add registry validator script"
|
|||
Add a `repo: local` hook entry to `.pre-commit-config.yaml`:
|
||||
|
||||
```yaml
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: validate-patches
|
||||
name: validate patch registry
|
||||
entry: Rscript local/validate-patches.R
|
||||
language: system
|
||||
files: ^local/patches/
|
||||
pass_filenames: false
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: validate-patches
|
||||
name: validate patch registry
|
||||
entry: Rscript local/validate-patches.R
|
||||
language: system
|
||||
files: ^local/patches/
|
||||
pass_filenames: false
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify the hook runs**
|
||||
|
|
@ -1397,10 +1421,12 @@ git commit -m "ci(patches): validate patch registry in pre-commit"
|
|||
### Task B4: Pass `patches` through the build entry points
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `local/build-one.R:99-119` (the `build_binary_package` call)
|
||||
- Modify: `local/build-all.R:125-145` (the `build_binary_package` call)
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: the new bincraft `patches` argument (Phase A) and `local/patches/`.
|
||||
- Produces: both entry points pass `patches = "local/patches"`.
|
||||
|
||||
|
|
@ -1437,9 +1463,11 @@ git commit -m "feat(patches): pass patch registry to bincraft build calls"
|
|||
### Task B5: Document the feature in the README
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `README.md`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: a short "Patching packages" section explaining the registry and linking the design spec.
|
||||
|
||||
- [ ] **Step 1: Add a README section**
|
||||
|
|
@ -1458,6 +1486,7 @@ git commit -m "docs(patches): document the package patching workflow"
|
|||
## Self-Review
|
||||
|
||||
**Spec coverage:**
|
||||
|
||||
- Registry in this repo, passed to bincraft → Task A7 (`patches` arg), B1 (registry), B4 (wiring). ✓
|
||||
- Mechanism in bincraft (resolve → pre-build → prepend repo → install) → A4–A6. ✓
|
||||
- Both tiers, env-first → A4 (`env`, `configure_args`, `makevars`, `patch` all applied; env is the cheap default and the RcppParallel entry uses only env). ✓
|
||||
|
|
|
|||
Loading…
Reference in a new issue