build-cran-binaries/specs/2026-06-30-package-patching-design.md

# Design: Dynamic per-package patching during binary builds

Date: 2026-06-30
Status: Approved (pending spec review)

## Problem

Some CRAN packages fail to compile on specific build platforms due to compiler- or OS-specific issues that have nothing to do with the package being built.
The canonical example is `RcppParallel`: its bundled Intel TBB sources fail to build on musl (Alpine) and on newer OS/compiler combinations.
Observed failure on `ubuntu-2604` ("resolute") with `g++ 15.2.0`:

```
../build/common.inc:74: *** "" is not supported. Add build/.inc file with os-specific settings . Stop.
make: *** [Makevars:163: tbb] Error 2
ERROR: compilation failed for package 'RcppParallel'
```

Because `RcppParallel` is a dependency of many packages, a single such failure cascades: every dependent package (e.g. `rts2`) also fails, even though nothing is wrong with the dependent itself.

Today there is no way to intervene.
A package can only be **excluded** (`local/excluded-packages.json`), which is all-or-nothing and does not help dependents.

## Goal

Allow a curated set of packages to be "patched" — via lightweight build-time overrides or, when necessary, real source diffs — **before** they are installed, whether the package is a direct build target or a transitive dependency pulled in by `pak`.

## Key constraint that drives the design

When `RcppParallel` fails here, it is being installed as a **transitive dependency** by `pak`, inside `bincraft::build_binary_package()`.
`pak` downloads, configures, and compiles it in one subprocess; this repo never touches that source.
For a fix to reach a dependency-of-a-dependency, the fixed package must be visible to `pak` itself, where `pak`'s repositories/sources are configured — which is inside `bincraft`.

Decisions taken during brainstorming:

- **Mechanism lives in `bincraft`** (the engine), because only there can transitive deps be influenced.
- **Patch tiers: both, env-overrides first.** Support cheap per-package env vars / configure args / Makevars (version-independent) *and* true source diffs (version-pinned), preferring the lightweight override.
- **Registry data lives in this repo** (`build-cran-binaries`) and is passed into `bincraft`, keeping `bincraft` as pure mechanism and the frequently-changing policy data with operational config.

## Approaches considered

| Approach | How pak sees the fix | Verdict |
|---|---|---|
| A. Patched **source** repo — drop patched `.tar.gz` source into a local repo, prepend it | pak recompiles from your source | Simple, but env-tier overrides leak globally (one subprocess builds everything) and the dep recompiles on every dependent build |
| **B. Pre-built patched binary repo (chosen)** | pak installs a ready binary by repo priority | Per-package scoping is free; no recompile; the binary is a cacheable/uploadable artifact that fits the existing system |
| C. pkgdepends per-build hook | intercept each build | No clean per-package pre-compile hook exists; fragile |

Chosen: **B**.

## Architecture

### Registry (this repo)

```
local/patches/
  registry.json          # the manifest
  RcppParallel/
    fix.patch            # optional source diff, referenced by an entry
```

`registry.json` is an array of entries:

```json
[
  {
    "package": "RcppParallel",
    "versions": "*",
    "platforms": ["alpine", "ubuntu-2604"],
    "env": { "RCPP_PARALLEL_USE_TBB": "0" },
    "configure_args": [],
    "makevars": {},
    "patch": null,
    "reason": "bundled TBB fails to build on musl / newer compilers"
  }
]
```

Field semantics:

- `package` (string, required): CRAN package name.
- `versions` (string, required): `"*"` for any, a constraint such as `">=5.1.0"`, or an exact version `"5.1.11-2"`.
  Env-tier fixes are typically `"*"`; source diffs are normally exact or lower-bounded because a diff is pinned to the source it was generated against.
- `platforms` (array of strings, required): matched against the running build's platform tokens — distro family (`alpine`, `ubuntu`, `redhat`), codename (`ubuntu-2604`, `alpine-324`), and arch (`amd64`, `arm64`).
  An entry matches if any listed token matches any build token.
  `["*"]` matches all platforms.
- `env` (object, optional): environment variables exported only for this package's isolated build.
- `configure_args` (array, optional): passed as `--configure-args` to the isolated build.
- `makevars` (object, optional): key/value pairs written into a package-local Makevars for the isolated build.
- `patch` (string or null, optional): path (relative to `local/patches/`) to a unified diff applied to the unpacked CRAN source before building.
- `reason` (string, required): human explanation, surfaced in logs and metadata.

A fix is any combination of `env`, `configure_args`, `makevars`, and `patch`.
"Env-first" is an authoring guideline (prefer the lightweight override) and an ordering of effort, not a runtime branch — all present fields are applied together for the isolated build.

### Flow (inside bincraft, around existing pak resolution)

1. **Resolve** the dependency set (dry-run) to learn the concrete versions `pak` will install.
   Reuse bincraft's existing resolution where possible (e.g. a `pkgdepends` proposal: `$resolve()` → inspect resolution → ... → `$solve()` / `$install()` after the local repo is prepended).
2. For each resolved package that matches a registry entry (name + `versions` + `platforms`): obtain a **patched binary** for the exact `version × platform × arch × R-minor`:
   - **Cache hit** (local `/mnt/cache/patched-binaries/` or S3): fetch it into the local repo.
   - **Cache miss**: download the CRAN **source** for that version, apply the source `patch` (if any) to the unpacked tree, build the binary in isolation with `env` / `configure_args` / `makevars` applied, then place the binary in the local repo and write it to the cache (and S3 if uploading is enabled).
3. **Prepend** the local binary repo (`file://…`) to `pak`'s repo list, and regenerate its `PACKAGES` index.
4. Run the **normal install**.
   `pak` resolves the patched binary for the matched package — direct or transitive — because it wins on repo priority for an equal version, and installs it without recompiling.

### Caching (essential)

`RcppParallel` is a dependency of dozens of packages; without caching the fix would be rebuilt on every dependent build.
Patched binaries are keyed by:

```
<package>_<version>_<platform>_<arch>_<rminor>_<patchhash>
```

`patchhash` is a hash of the normalized registry entry plus the referenced diff file contents.
Editing a patch therefore changes the hash and auto-invalidates stale cached binaries.

- Local cache: `/mnt/cache/patched-binaries/`.
- Optional S3 cache for cross-build reuse: a dedicated `…/patched/` slot under the existing arch/codename structure, mirroring how normal binaries are stored.

### S3 upload

Patched binaries **are** uploaded to S3 (in addition to the local cache) so they are reused across CI jobs and machines, not just within one container.
They live in a separate `patched/` slot and are not published into the user-facing `src/contrib` index — they are an internal build accelerator, not a distributed artifact.

## Error handling

- **Source diff fails to apply** (CRAN moved past the pinned version): log a clear warning, skip that entry, and proceed.
  The package builds unpatched (status quo) and may fail.
  The skipped/failed-to-apply patch is surfaced in build metadata.
- **Pre-build of the patched binary fails**: log a warning, skip, proceed.
- **No version or platform match**: skip silently (the entry simply does not apply to this build).
- **Overlapping entries for one package**: the most specific entry wins (a concrete `platforms`/`versions` beats `"*"`).
  Genuine ambiguity (two equally specific, conflicting entries) is a validation error reported before the build.

## Observability

- One log line per applied patch, e.g.: `Applying patch to RcppParallel 5.1.11-2 [env: RCPP_PARALLEL_USE_TBB=0]: bundled TBB fails on musl / newer compilers`
- The set of applied patches (package, version, `patchhash`) is recorded in the Postgres build-metadata row for the build, so it is queryable later.

## Testing

- **Unit (registry):** parsing and matching — version constraints, platform token matching, precedence/specificity, and detection of ambiguous overlaps.
- **Unit (cache key):** `patchhash` changes when the entry or diff changes; is stable otherwise.
- **Integration:** `RcppParallel` on `resolute` (and/or Alpine) fails to build without a registry entry and succeeds with one; a dependent package such as `rts2` succeeds once the dependency is patched.
- **Failure path:** an entry pinned to an old version against a newer CRAN release → graceful skip with a warning, build continues.

## Out of scope

- Shipping a default registry inside `bincraft` (registry is repo-local for now; a baseline-in-engine + repo-override model can come later if needed).
- Publishing patched binaries into the public `src/contrib` index.
- Automatic detection of which packages need patches — entries are curated by hand.

## Split of work

- **bincraft:** the mechanism — registry ingestion, resolution hook, isolated patched-binary build, caching/upload, local-repo prepend, logging, metadata recording.
  A new `patches` argument on `build_binary_package()`.
- **build-cran-binaries (this repo):** the `local/patches/` registry and diffs, passing `patches = "local/patches"` through `build-one.R` / `build-all.R`, and documentation.