Three separate reasons the escape hatch added in #9 would not have worked.
Declare FORCE_REBUILD as a top-level variable so the UI offers it as a
dropdown on a manual run, rather than requiring a hand-typed key.
Accept the value from either trigger path. `crow pipeline start -p` injects
into the step environment, while the UI and `crow pipeline create --var`
pass a substitution variable, and a fresh run of main is a create. Reading
only the bare shell variable would have ignored the flag in exactly the case
it exists for.
Test the value rather than emptiness. A declared variable substitutes its
default, so the previous `[ -n ... ]` check would have seen the literal
string "false" and rebuilt everything on every cron run.
Set overwrite: true on the upload. plugin-s3 skips an object that already
exists when overwrite is false, logging "File already exists, skipping
upload" and still exiting green, so a forced rebuild would have recompiled
every package and published none of them. Normal runs are unaffected, since
dedup means only not-yet-published versions are ever built.
Also drop the Net debug step, the commented-out Pull upstream step and a
stray `# docker info`. The MTU workaround the debug step was diagnosing
stays, as does the commented-out alpine-320 matrix entry, which records a
deliberately disabled platform rather than dead scaffolding.
The escape hatch added in #9 only read a bare $FORCE_REBUILD, which covers
`crow pipeline start -p FORCE_REBUILD=1`, where the parameter is injected
into the step environment. It does not cover
`crow pipeline create --branch main --var FORCE_REBUILD=1`, which passes a
crow substitution variable instead.
Triggering a fresh run of main is a `create`, so the flag would have been
silently ignored in exactly the case it is needed, and the run would skip
every Alpine version it was meant to rebuild.
Read the substituted ${FORCE_REBUILD} into FORCE_REBUILD_VAR and accept
either source. Crow blanks an unsupplied substitution variable rather than
failing, as recorded in 8eacc0a, so a plain cron run still takes the dedup
path.
## Motivation
R on the Alpine images was built without graphics support: `capabilities()` reports `cairo`, `png`, `jpeg` and `tiff` as `FALSE`, while Ubuntu reports all four `TRUE`.
The images install `R-dev` but never the graphics headers, and R's `configure` treats those as optional and does not fail when they are absent, so the build succeeds and ships an R that cannot draw.
Reproduced in a clean `alpine:3.22` container with the old package set:
```
Additional capabilities: PNG, NLS, ICU
Capabilities skipped: JPEG, TIFF, cairo
```
One correction to the original diagnosis: `--with-x=no` was not the cause, and libpng was in fact found.
R's bitmap and `pangocairo` checks run independently of `R_X11` (`configure.ac:2604` gates on `want_cairo`, not `use_X11`).
`capabilities("png")` needs a device *backend*, and with X11 off and cairo absent there was none, which is why all four read `FALSE` despite configure listing PNG.
So the fix is purely the missing `-dev` packages.
X11 stays off, which is correct for headless images, and cairo covers `png()` / `jpeg()` / `tiff()` / `svg()`.
## Changes
- `builder/Dockerfile.alpine-{320,321,322,323,324}` — install `pkgconf cairo-dev pango-dev fontconfig-dev freetype-dev libpng-dev jpeg-dev tiff-dev zlib-dev`; pass `--with-cairo --with-libpng --with-jpeglib --with-libtiff` explicitly so the intent sits next to `--with-x=no`; set `REQUIRED_CAPABILITIES`.
- `builder/build.sh` — add `verify_capabilities()` between `compile_r` and `package_r`. A build that loses a graphics device now fails instead of being packaged and uploaded. Opt-in via `REQUIRED_CAPABILITIES`, so every non-Alpine platform is untouched.
- `.crow/build.yaml` — add a `FORCE_REBUILD` escape hatch to the dedup step. Artifact filenames carry only the R version, so the HEAD probe would otherwise keep skipping every Alpine version and serving the stale broken binaries.
- `CLAUDE.md` — record the three gotchas.
## Verification
Compiled R 4.5.1 in the patched `alpine-322` image, through the real `build.sh`:
```
jpeg png tiff tcltk X11 aqua
TRUE TRUE TRUE TRUE FALSE FALSE
NLS Rprof profmem cairo ICU long.double
TRUE TRUE TRUE TRUE TRUE TRUE
All required capabilities are present.
```
The devices actually render, not just report `TRUE` (`default bitmapType: cairo`):
```
png -> t.png (2154 bytes)
jpeg -> t.jpeg (4489 bytes)
tiff -> t.tiff (120160 bytes)
svg -> t.svg (9381 bytes)
```
The guard was exercised in both directions against that real build: it passes on the four graphics capabilities, and returns non-zero naming the missing entry when additionally asked for `X11`.
`crow lint .crow/` passes and `bash -n builder/build.sh` is clean.
## Behaviour change
Alpine R binaries gain the cairo, png, jpeg and tiff devices, and link against `libcairo`, `libpangocairo`, `libpng`, `libjpeg` and `libtiff`.
The `.apk` metadata already declared those runtime dependencies, so consumer images need no change.
Any Alpine build that loses a graphics device now fails the pipeline rather than uploading silently.
**Needs a follow-up after merge:** the first run must have `FORCE_REBUILD` set, otherwise dedup skips every existing Alpine version and the broken artifacts stay in the bucket.
Reviewed-on: #9
The merged HEAD-probe check still rebuilt everything: crow runs its own
${VAR} substitution over the commands before the shell executes, so the
shell variables ${BASE} and ${VERSION} were blanked to empty (only matrix
vars resolve at that stage). Every probed URL was therefore malformed and
returned non-200, leaving r-versions-existing.txt empty.
Use bare $name for shell variables and build the filename with printf, which
avoids the ${VERSION}_1 brace requirement entirely. Matrix vars (${PLATFORM},
${PLATFORM_ID}, ${ARCH_ID}) keep braces since crow resolves those correctly.
Verified URL construction against the live bucket for apk, deb and rpm.
The existing-version check listed the bucket with an anonymous curl, but
Backblaze B2 requires authentication for the list-bucket API, so it returned
AccessDenied and r-versions-existing.txt was always empty -> every version
was rebuilt despite already existing in s3.
Signing the request is not workable here: the B2 credentials are scoped to
the plugin-s3 image in crow and are not injected into this plain alpine step
(curl saw --user ":").
Individual artifacts are public-read, so probe each expected object URL with
an anonymous HEAD instead of listing the bucket. The filename mirrors what
nfpm produces (version/release pinned to 1): r-<ver>_1_<arch>.{apk,deb} for
alpine/ubuntu and R-<ver>-1-1.<arch>.rpm for el*. Verified against the live
bucket for all three packager families.
Reviewed-on: #8
Backblaze B2 requires authentication for the list-bucket API; anonymous
access only works for individual public-read objects. The unauthenticated
curl returned AccessDenied, so r-version-s3.txt was always empty and every
R version was rebuilt despite already existing in s3.
Sign the listing request with --aws-sigv4 using the same B2 credentials as
the upload step. Also fix the broken version extraction: the old
substr($NF, 3, 5) ran against the prefixed key and yielded garbage, so the
Build step's grep could never match. Print the actual version on a match
and use a literal index() instead of a regex comparison.
Reviewed-on: #7
## Summary
The crow build pipeline previously built only the single latest stable R version (from the brew formula).
It now builds the latest patch release of each of the **last four minor R versions** and uploads any that are missing from S3, so older supported minors stay available rather than only the newest stable.
- **Get R versions to build**: derive the four versions from posit's `versions.json` (the same source upstream's `test/get_r_versions.py` uses) via jq — keep only `x.y.z` releases, take the latest patch per minor, then the top four minors (e.g. `4.6.0, 4.5.3, 4.4.3, 4.3.3`).
- **Check which R versions already exist in s3**: iterate the four versions against the S3 listing for the platform/arch.
- **Build**: build only the versions that are missing. In steady state nothing builds; when a new minor lands, only that one is built.
## awk fix
The existence check used an awk regex (`$0 ~ ver`), so the unescaped dots in a version made `4.4.3` spuriously match `4.3.3` (the substring `4/4.3` satisfies the pattern).
Harmless with a single version, but across four adjacent versions it would wrongly skip builds.
Switched to literal substring matching via awk `index($0, ver)`.
Reviewed-on: #6
Path MTU to external destinations is ~1280 from the new cluster
(PMTUD broken upstream, packets >1280 are silently dropped). Force
eth0 MTU to 1280 at the start of every step (needs privileged), drop
the DinD inner bridge from 1420 to 1280, and wrap each apk add in a
short retry loop to absorb any remaining transient failures.
Workflow-side workaround; the proper fix is to lower the CNI/host
docker daemon MTU at the infra layer.
- Fetch R version from Homebrew JSON API with jq instead of scraping
the raw formula (was silently producing an empty version file that
propagated to downstream steps).
- Replace grep -oP with grep -oE in the S3 listing step; BusyBox grep
in the alpine image has no Perl-regex support.
- Add ephemeral-storage requests/limits to the Build step so the
docker-cli apk extract doesn't fail with I/O errors under node
disk pressure.
