pat-s
1bca6004fd
Follow-up to #124 (merged): the auto-patch pipeline required a separate write-scoped `REPO_RW_TOKEN` to push the branch. Reuse the existing `FORGEJO_TOKEN` instead. - `propose-patches.R --open-pr` now pushes `auto/registry-patch-proposals` over HTTPS with `FORGEJO_TOKEN` (the same token used for the PR API); the read-only `origin` clone URL can't push, so it builds an authenticated URL explicitly. - Drop `REPO_RW_TOKEN` from `.crow/auto-apply-patches.yaml` and the docs. No new secret needed: the pipeline's secrets are now `PGPASS`, `REPO_RO_TOKEN`, and `FORGEJO_TOKEN` (all existing). `FORGEJO_TOKEN` must have repository write scope for the push to succeed. 105 tests pass; all pre-commit hooks pass. Reviewed-on: #125
120 lines
9.4 KiB
Markdown
120 lines
9.4 KiB
Markdown
# Patch Registry
|
|
|
|
This directory contains the curated registry of per-package build-time patches consumed by bincraft's `patches` argument.
|
|
|
|
## Schema
|
|
|
|
The registry is defined in `registry.json` as an array of patch entries. Each entry specifies lightweight build-time overrides (environment variables, configure arguments, Makevars) and optionally a source diff to apply before building.
|
|
|
|
### Field semantics
|
|
|
|
| Field | Type | Required | Description |
|
|
| ---------------- | ---------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| `package` | string | yes | CRAN package name. |
|
|
| `versions` | string | yes | `"*"` for any, a constraint such as `">=5.1.0"`, or an exact version `"5.1.11-2"`. Env-tier fixes are typically `"*"`; source diffs are normally exact or lower-bounded because a diff is pinned to the source it was generated against. |
|
|
| `platforms` | array of strings | yes | Matched against the running build's platform tokens — distro family (`alpine`, `ubuntu`, `redhat`), codename (`ubuntu-2604`, `alpine-324`), and arch (`amd64`, `arm64`). An entry matches if any listed token matches any build token. `["*"]` matches all platforms. |
|
|
| `env` | object | no | Environment variables exported only for this package's isolated build. |
|
|
| `configure_args` | array | no | Arguments passed as `--configure-args` to the isolated build. |
|
|
| `makevars` | object | no | Key/value pairs written into a package-local Makevars for the isolated build. |
|
|
| `patch` | string or null | no | Path (relative to `local/patches/`) to a unified diff applied to the unpacked CRAN source before building. |
|
|
| `reason` | string | yes | Human explanation, surfaced in logs and metadata. |
|
|
|
|
## Adding an entry
|
|
|
|
To add a new patch entry:
|
|
|
|
1. Add an object to the array in `registry.json` with the fields documented above.
|
|
Start with lightweight overrides (environment variables, configure arguments, Makevars) before resorting to source diffs.
|
|
|
|
2. If a source diff is needed, place it in `local/patches/<package>/<file>.patch` and reference its path in the `patch` field.
|
|
For example, a diff for `RcppParallel` would go in `local/patches/RcppParallel/fix.patch` and be referenced as `"patch": "RcppParallel/fix.patch"`.
|
|
|
|
3. The `reason` field should clearly explain why the patch is needed and what problem it solves.
|
|
|
|
## Validation
|
|
|
|
The registry is validated and applied by bincraft during the build process.
|
|
For manual validation, run the validator from the repo root:
|
|
|
|
```bash
|
|
Rscript local/validate-patches.R
|
|
```
|
|
|
|
This validates the schema, referenced patch-file existence, and checks for duplicate entries across platforms and versions.
|
|
|
|
## Triaging failures into entries
|
|
|
|
`local/failing-builds-report.R` turns recorded build failures into triaged patch suggestions instead of hand-scraping Crow logs (issue #115, steps 1 + 2).
|
|
It is read-only: it queries `single_builds WHERE error_occurred`, groups failures by a normalised error fingerprint, classifies each group against the known signature set in `local/failing-builds-classify.R`, and prints a report.
|
|
|
|
```bash
|
|
# All platforms/arches; needs the DB password.
|
|
PGPASS=... Rscript local/failing-builds-report.R
|
|
# Restrict scope and also emit a machine-readable report.
|
|
PGPASS=... Rscript local/failing-builds-report.R --platform alpine-321 --arch amd64 --json report.json
|
|
```
|
|
|
|
Each group is tagged `AUTO-PROPOSABLE` (a known env/makevars lever, or an already-curated package patch, safe to pre-fill as a `registry.json` entry) or `HUMAN TRIAGE` (unknown signature, or a fix that needs a novel source diff).
|
|
For auto-proposable groups it prints a ready-to-review registry entry; still run `validate-patches.R` and an isolated trial build before merging.
|
|
Novel source diffs and unknown signatures stay human-reviewed by design.
|
|
|
|
Add a new signature by appending a rule to `build_signatures()` in `local/failing-builds-classify.R`; the pure helpers are covered by `local/tests/test-failing-builds-classify.R`.
|
|
|
|
### Proposing entries (step 3: propose, do not apply)
|
|
|
|
`local/propose-patches.R` takes the auto-proposable candidates one step further: for each classified, safe fix affecting a package with no current entry, it emits a pre-filled `registry.json` entry and validates the candidate set against a temporary merged registry (the real registry is never touched unless you ask).
|
|
The human gate stays: it never merges.
|
|
|
|
```bash
|
|
# Default: print candidates + validation, take no action.
|
|
PGPASS=... Rscript local/propose-patches.R
|
|
# Append the candidates to registry.json + the proposals ledger (you commit + open the PR).
|
|
PGPASS=... Rscript local/propose-patches.R --write
|
|
# Or post/update a Forgejo tracking issue instead (needs FORGEJO_TOKEN).
|
|
PGPASS=... FORGEJO_TOKEN=... Rscript local/propose-patches.R --open-issue
|
|
```
|
|
|
|
The acceptance criteria before merging a proposal are: `validate-patches.R` passes (checked automatically), and an isolated trial build succeeds.
|
|
Run the trial build inside the failing platform's build-env image; it uploads/archives nothing and writes no metadata:
|
|
|
|
```bash
|
|
Rscript local/trial-build-patch.R <package>
|
|
```
|
|
|
|
#### Autonomous PR + trial-build gate
|
|
|
|
`--open-pr` closes the loop: it writes the top-N candidates (by failure volume) onto the reused `auto/registry-patch-proposals` branch and opens/updates a single PR.
|
|
`.crow/auto-apply-patches.yaml` runs this on a cron.
|
|
`FORGEJO_TOKEN` is used for both the branch push and the PR (no separate write-scoped secret).
|
|
|
|
```bash
|
|
# Bounded batch; opens/updates one PR.
|
|
PGPASS=... FORGEJO_TOKEN=... Rscript local/propose-patches.R --open-pr --limit 10
|
|
```
|
|
|
|
The merge gate is `.crow/trial-build-registry.yaml`: matrixed over the build-env images, each platform trial-builds only the entries the branch **adds** (`local/trial-build-registry.R`, which diffs the registry against `main`) and is green only if every new entry builds.
|
|
The repo uses no `pull_request` triggers, so this gate runs manually or on a cron against the auto-patch branch (`--var patch_branch=...`); wiring it to fire on the PR needs `event: pull_request` enabled on the forge.
|
|
|
|
`--write` and `--open-issue` also append to `local/patches/proposals-log.json`, a ledger of what was proposed.
|
|
|
|
### Feedback loop (step 4)
|
|
|
|
`local/proposal-tracking.R` reports the signature hit rate, proposed-vs-merged status (a proposal counts as merged once its package appears in the registry), and retirement candidates (registry entries whose package no longer appears in any current failure, so the upstream cause was likely fixed).
|
|
It also surfaces the classifier's **blind spots**: the unclassified failures (candidates for a new signature) and the groups blocked on a dependency build, so the unknown buckets get the same visibility as the proposals.
|
|
It is read-only on the DB, with an optional Forgejo issue as the only write.
|
|
|
|
```bash
|
|
# Print the metrics + blind spots.
|
|
PGPASS=... Rscript local/proposal-tracking.R --json metrics.json
|
|
# Post/update a "needs signatures" tracking issue with the unclassified failures.
|
|
PGPASS=... FORGEJO_TOKEN=... Rscript local/proposal-tracking.R --open-issue
|
|
```
|
|
|
|
The unclassified groups are the natural place to discover which new signatures are worth adding to `build_signatures()`.
|
|
The pure metric/ledger helpers live in `local/proposal-tracking-lib.R` and are covered by `local/tests/test-proposal-tracking-lib.R`.
|
|
|
|
### Scheduled run
|
|
|
|
`.crow/weekly-patch-proposals.yaml` runs both steps weekly (register the `weekly-patch-proposals` cron in the crow UI).
|
|
It posts/updates two Forgejo issues -- one with the auto-proposable entries, one with the unclassified/blocked failures -- and logs the feedback-loop metrics.
|
|
It clones read-only; the only writes are the two tracking issues.
|