fix(abi): inspect flat objects of packages CRAN has removed
The backfill decided which objects were worth inspecting by looking the package up in CRAN's current index. CRAN lists only what it currently ships, so a package it has archived is absent, is never recognised as compiled, and is never inspected. No verdict is ever recorded for it. `union_index_records()` meanwhile reads the slot's *own* `NeedsCompilation`, finds `yes`, and drops the record when the source classifier flags the package and no safe verdict exists to clear it. The repair cannot rescue it either, because it works from recorded verdicts and there is no row. The record is therefore dropped from every per-minor index on every re-index, permanently. `mixedMem` and `vcpen`, both removed from CRAN, were stuck out of all three rhel8 per-minor indexes on both architectures that way, surviving a backfill, two re-indexes and a repair. Read `NeedsCompilation` from the slot's own flat index instead, which describes exactly the objects being inspected. It is fetched from the bucket rather than over HTTPS because this runs inside R, where the edge would route the request to a per-minor index. Keyed on `<package>_<version>` rather than the package name: a slot can carry several versions and only some of them compile.
This commit is contained in:
parent
03e007069a
commit
767dd9ee5d
1 changed files with 133 additions and 60 deletions
|
|
@ -29,12 +29,14 @@ All Phase A paths are relative to `/Users/pjs/git/codefloe.com/rpkgs/bincraftr`.
|
|||
|
||||
### Task A0: Proof of mechanism — pak installs a patched binary from a prepended `file://` repo
|
||||
|
||||
This de-risks the core assumption before building anything on top: that `pak` installs a binary from a local `file://` repo in preference to CRAN for an equal version, and does so without recompiling. If this fails, the contingency (documented in Step 4) is to serve patched *source* and rely on `pkgcache` build-caching — the rest of the plan changes only inside `build_patched_binary()`.
|
||||
This de-risks the core assumption before building anything on top: that `pak` installs a binary from a local `file://` repo in preference to CRAN for an equal version, and does so without recompiling. If this fails, the contingency (documented in Step 4) is to serve patched _source_ and rely on `pkgcache` build-caching — the rest of the plan changes only inside `build_patched_binary()`.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `tools/verify-patch-mechanism.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: a runnable script proving `pak::pkg_install()` resolves a local patched binary over CRAN. No package API.
|
||||
|
||||
- [ ] **Step 1: Write the verification script**
|
||||
|
|
@ -118,17 +120,19 @@ git commit -m "test(patches): prove pak installs a patched binary from a local f
|
|||
- [ ] **Step 4: Record the outcome / contingency**
|
||||
|
||||
If the proof PASSED, proceed to Task A1 unchanged.
|
||||
If it FAILED (pak recompiled or picked CRAN's), the mechanism switches to serving patched *source*: in Task A4 `build_patched_binary()` skips `pkgbuild::build()` and instead repackages the patched source tree with `pkgbuild::build(binary = FALSE)`; everything else (registry, matching, cache, repo prepend) is unchanged because `pak` build-caches the compiled result via `pkgcache`. Note the chosen path in the commit message and continue.
|
||||
If it FAILED (pak recompiled or picked CRAN's), the mechanism switches to serving patched _source_: in Task A4 `build_patched_binary()` skips `pkgbuild::build()` and instead repackages the patched source tree with `pkgbuild::build(binary = FALSE)`; everything else (registry, matching, cache, repo prepend) is unchanged because `pak` build-caches the compiled result via `pkgcache`. Note the chosen path in the commit message and continue.
|
||||
|
||||
---
|
||||
|
||||
### Task A1: Registry loading and normalization
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: `load_patch_registry(patches_dir)` → `list()` of normalized entries; each entry is a named list with `package`, `versions`, `platforms` (character vector), `env` (named list), `configure_args` (character), `makevars` (named list), `reason`, and `patch_path` (absolute path or `NULL`). `normalize_patch_entry(entry, patches_dir)` → one normalized entry; errors on missing required field or missing patch file.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
|
@ -275,10 +279,12 @@ git commit -m "feat(patches): load and validate the patch registry"
|
|||
### Task A2: Platform matching and version-constraint satisfaction
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: normalized entries from Task A1.
|
||||
- Produces: `build_platform_tokens(platform, arch)` → character vector; `entry_matches_platform(entry, tokens)` → logical; `match_patch_entries(registry, platform, arch)` → filtered list; `version_satisfies(version, constraint)` → logical (constraint forms: `"*"` handled by caller, `"x.y.z"` exact, `">=x"`, `"<=x"`, `">x"`, `"<x"`, `"==x"`).
|
||||
|
||||
|
|
@ -398,10 +404,12 @@ git commit -m "feat(patches): platform matching and version-constraint checks"
|
|||
### Task A3: Cache key and version resolution
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: normalized entries.
|
||||
- Produces: `patch_cache_key(entry, version, platform, arch, r_minor)` → string `"<pkg>_<ver>_<platform>_<arch>_<rminor>_<hash12>"`, where `hash12` covers `env`/`configure_args`/`makevars`/patch bytes; `resolve_patch_version(entry)` → latest CRAN version satisfying `entry$versions`, or `NA_character_`; `describe_patch(entry)` → short human label.
|
||||
|
||||
|
|
@ -546,10 +554,12 @@ git commit -m "feat(patches): cache key, version resolution, and patch descripti
|
|||
### Task A4: Build a patched binary in isolation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: a normalized entry, a resolved `version`, a `dest_dir`.
|
||||
- Produces: `download_cran_source(package, version, dest_dir, cran)` → path or `NULL`; `apply_source_patch(patch_path, pkg_src)` → logical; `configure_args_to_build_args(configure_args)` → character; `build_patched_binary(entry, version, dest_dir)` → path to built binary tarball or `NULL`.
|
||||
|
||||
|
|
@ -737,10 +747,12 @@ git commit -m "feat(patches): build a patched binary in isolation from CRAN sour
|
|||
### Task A5: Orchestrate the local patched repo (cache + index)
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: all helpers above.
|
||||
- Produces: `prepare_patched_repo(patches_dir, platform, arch, r_minor, cache_dir, repo_dir)` → path to a `src/contrib`-style dir containing patched binaries + a `PACKAGES` index, or `NULL` when nothing matched/built. On a cache hit it copies the cached tarball into `repo_dir`; on a miss it builds, then writes the result into `cache_dir`.
|
||||
|
||||
|
|
@ -902,11 +914,13 @@ git commit -m "feat(patches): orchestrate local patched-binary repo with caching
|
|||
### Task A6: Wire patched repo into the pak install path
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/install_helpers.R:333-389` (`run_pak_install_with_mutex`)
|
||||
- Modify: `R/install-deps.R:23-75` (`install_pkg_sys_deps`)
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: `prepare_patched_repo()`.
|
||||
- Produces: `run_pak_install_with_mutex(local_clone_dir_single, env_vars, patched_repo = NULL)` — prepends `file://<patched_repo>` to `options("repos")` for the install; `install_pkg_sys_deps(package_name, tag, local_clone_dir, platform, aggressive_cleanup = FALSE, patches = NULL, arch = NULL)` — builds the patched repo before installing.
|
||||
|
||||
|
|
@ -1052,11 +1066,13 @@ git commit -m "feat(patches): serve patched binaries to pak during dep install"
|
|||
### Task A7: Thread `patches` through the public build API
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `R/build_binaries.R` (`build_binary_package`, `execute_package_builds`, `build_single_tag`, `handle_system_dependencies`)
|
||||
- Create: `man-roxygen/param-patches.R`
|
||||
- Test: `tests/testthat/test-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: `build_binary_package(..., patches = NULL)` and the internal chain each carry `patches` down to `install_pkg_sys_deps()`. `handle_system_dependencies(..., patches = NULL)` passes `patches` and `arch` through.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
|
@ -1151,10 +1167,12 @@ git commit -m "feat(patches): thread patches argument through build_binary_packa
|
|||
### Task A8: End-to-end patch test (guarded) and version bump
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `tests/testthat/test-patches.R`
|
||||
- Modify: `DESCRIPTION` (version), `NEWS.md`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: a guarded e2e test proving a dependent package builds when its failing dependency is patched.
|
||||
|
||||
- [ ] **Step 1: Add the guarded e2e test**
|
||||
|
|
@ -1202,7 +1220,7 @@ In `DESCRIPTION`, bump `Version:` to `4.3.0.9999`. Prepend to `NEWS.md`:
|
|||
```markdown
|
||||
# bincraft 4.3.0
|
||||
|
||||
* `build_binary_package()` gains a `patches` argument: a registry of
|
||||
- `build_binary_package()` gains a `patches` argument: a registry of
|
||||
per-package env / configure / Makevars overrides and source diffs that are
|
||||
pre-built into patched binaries and served to pak, fixing compiler- and
|
||||
OS-specific failures (e.g. RcppParallel) including for transitive deps.
|
||||
|
|
@ -1224,10 +1242,12 @@ All Phase B paths are relative to this repo (`build-cran-binaries`). Phase B dep
|
|||
### Task B1: Create the patch registry with the RcppParallel entry
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `local/patches/registry.json`
|
||||
- Create: `local/patches/README.md`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: the curated registry consumed by bincraft's `patches` argument.
|
||||
|
||||
- [ ] **Step 1: Write the registry**
|
||||
|
|
@ -1268,9 +1288,11 @@ git commit -m "feat(patches): add patch registry with RcppParallel TBB workaroun
|
|||
### Task B2: Registry validator script
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `local/validate-patches.R`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: `local/patches/registry.json`.
|
||||
- Produces: a script that exits non-zero on schema violations, missing patch files, or ambiguous overlapping entries.
|
||||
|
||||
|
|
@ -1360,9 +1382,11 @@ git commit -m "feat(patches): add registry validator script"
|
|||
### Task B3: Hook the validator into pre-commit
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `.pre-commit-config.yaml`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: a local hook that runs `local/validate-patches.R` when the registry or patch files change.
|
||||
|
||||
- [ ] **Step 1: Add the hook**
|
||||
|
|
@ -1370,14 +1394,14 @@ git commit -m "feat(patches): add registry validator script"
|
|||
Add a `repo: local` hook entry to `.pre-commit-config.yaml`:
|
||||
|
||||
```yaml
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: validate-patches
|
||||
name: validate patch registry
|
||||
entry: Rscript local/validate-patches.R
|
||||
language: system
|
||||
files: ^local/patches/
|
||||
pass_filenames: false
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: validate-patches
|
||||
name: validate patch registry
|
||||
entry: Rscript local/validate-patches.R
|
||||
language: system
|
||||
files: ^local/patches/
|
||||
pass_filenames: false
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify the hook runs**
|
||||
|
|
@ -1397,10 +1421,12 @@ git commit -m "ci(patches): validate patch registry in pre-commit"
|
|||
### Task B4: Pass `patches` through the build entry points
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `local/build-one.R:99-119` (the `build_binary_package` call)
|
||||
- Modify: `local/build-all.R:125-145` (the `build_binary_package` call)
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Consumes: the new bincraft `patches` argument (Phase A) and `local/patches/`.
|
||||
- Produces: both entry points pass `patches = "local/patches"`.
|
||||
|
||||
|
|
@ -1437,9 +1463,11 @@ git commit -m "feat(patches): pass patch registry to bincraft build calls"
|
|||
### Task B5: Document the feature in the README
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `README.md`
|
||||
|
||||
**Interfaces:**
|
||||
|
||||
- Produces: a short "Patching packages" section explaining the registry and linking the design spec.
|
||||
|
||||
- [ ] **Step 1: Add a README section**
|
||||
|
|
@ -1458,6 +1486,7 @@ git commit -m "docs(patches): document the package patching workflow"
|
|||
## Self-Review
|
||||
|
||||
**Spec coverage:**
|
||||
|
||||
- Registry in this repo, passed to bincraft → Task A7 (`patches` arg), B1 (registry), B4 (wiring). ✓
|
||||
- Mechanism in bincraft (resolve → pre-build → prepend repo → install) → A4–A6. ✓
|
||||
- Both tiers, env-first → A4 (`env`, `configure_args`, `makevars`, `patch` all applied; env is the cheap default and the RcppParallel entry uses only env). ✓
|
||||
|
|
|
|||
Loading…
Reference in a new issue