From 1362e0bd3a7d8eaf2b90ece71d34293b024bb28f Mon Sep 17 00:00:00 2001 From: pat-s Date: Fri, 18 Sep 2026 07:04:22 +0000 Subject: [PATCH] fix(alpine): build R with cairo, png, jpeg and tiff graphics devices (#9) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation R on the Alpine images was built without graphics support: `capabilities()` reports `cairo`, `png`, `jpeg` and `tiff` as `FALSE`, while Ubuntu reports all four `TRUE`. The images install `R-dev` but never the graphics headers, and R's `configure` treats those as optional and does not fail when they are absent, so the build succeeds and ships an R that cannot draw. Reproduced in a clean `alpine:3.22` container with the old package set: ``` Additional capabilities: PNG, NLS, ICU Capabilities skipped: JPEG, TIFF, cairo ``` One correction to the original diagnosis: `--with-x=no` was not the cause, and libpng was in fact found. R's bitmap and `pangocairo` checks run independently of `R_X11` (`configure.ac:2604` gates on `want_cairo`, not `use_X11`). `capabilities("png")` needs a device *backend*, and with X11 off and cairo absent there was none, which is why all four read `FALSE` despite configure listing PNG. So the fix is purely the missing `-dev` packages. X11 stays off, which is correct for headless images, and cairo covers `png()` / `jpeg()` / `tiff()` / `svg()`. ## Changes - `builder/Dockerfile.alpine-{320,321,322,323,324}` — install `pkgconf cairo-dev pango-dev fontconfig-dev freetype-dev libpng-dev jpeg-dev tiff-dev zlib-dev`; pass `--with-cairo --with-libpng --with-jpeglib --with-libtiff` explicitly so the intent sits next to `--with-x=no`; set `REQUIRED_CAPABILITIES`. - `builder/build.sh` — add `verify_capabilities()` between `compile_r` and `package_r`. A build that loses a graphics device now fails instead of being packaged and uploaded. Opt-in via `REQUIRED_CAPABILITIES`, so every non-Alpine platform is untouched. - `.crow/build.yaml` — add a `FORCE_REBUILD` escape hatch to the dedup step. Artifact filenames carry only the R version, so the HEAD probe would otherwise keep skipping every Alpine version and serving the stale broken binaries. - `CLAUDE.md` — record the three gotchas. ## Verification Compiled R 4.5.1 in the patched `alpine-322` image, through the real `build.sh`: ``` jpeg png tiff tcltk X11 aqua TRUE TRUE TRUE TRUE FALSE FALSE NLS Rprof profmem cairo ICU long.double TRUE TRUE TRUE TRUE TRUE TRUE All required capabilities are present. ``` The devices actually render, not just report `TRUE` (`default bitmapType: cairo`): ``` png -> t.png (2154 bytes) jpeg -> t.jpeg (4489 bytes) tiff -> t.tiff (120160 bytes) svg -> t.svg (9381 bytes) ``` The guard was exercised in both directions against that real build: it passes on the four graphics capabilities, and returns non-zero naming the missing entry when additionally asked for `X11`. `crow lint .crow/` passes and `bash -n builder/build.sh` is clean. ## Behaviour change Alpine R binaries gain the cairo, png, jpeg and tiff devices, and link against `libcairo`, `libpangocairo`, `libpng`, `libjpeg` and `libtiff`. The `.apk` metadata already declared those runtime dependencies, so consumer images need no change. Any Alpine build that loses a graphics device now fails the pipeline rather than uploading silently. **Needs a follow-up after merge:** the first run must have `FORCE_REBUILD` set, otherwise dedup skips every existing Alpine version and the broken artifacts stay in the bucket. Reviewed-on: https://git.devxy.io/devxy/r-builds/pulls/9 --- .crow/build.yaml | 22 +++++++++++++----- CLAUDE.md | 7 ++++++ builder/Dockerfile.alpine-320 | 14 +++++++++++- builder/Dockerfile.alpine-321 | 14 +++++++++++- builder/Dockerfile.alpine-322 | 14 +++++++++++- builder/Dockerfile.alpine-323 | 14 +++++++++++- builder/Dockerfile.alpine-324 | 14 +++++++++++- builder/build.sh | 42 +++++++++++++++++++++++++++++++++++ 8 files changed, 130 insertions(+), 11 deletions(-) diff --git a/.crow/build.yaml b/.crow/build.yaml index 18f126f..acdb7a3 100644 --- a/.crow/build.yaml +++ b/.crow/build.yaml @@ -178,12 +178,22 @@ steps: esac BASE="https://s3.eu-central-003.backblazeb2.com/devxy-r-builds/${PLATFORM_ID}" : > r-versions-existing.txt - for VERSION in $(cat r-versions-to-build.txt); do - URL="$BASE/$(printf "$FMT" "$VERSION" "${ARCH_ID}")" - if [ "$(curl -s -o /dev/null -w '%{http_code}' -I "$URL")" = "200" ]; then - echo "$VERSION" >> r-versions-existing.txt - fi - done + # Escape hatch for when the builder image itself changes in a way that + # alters the resulting R binary (a newly enabled graphics device, a + # different BLAS). The artifact filename carries only the R version, so + # dedup would otherwise keep the stale binaries forever. Set FORCE_REBUILD + # to a non-empty value as a pipeline variable on a manual run, or hardcode + # it here temporarily; uploads overwrite in place. + if [ -n "$FORCE_REBUILD" ]; then + echo "FORCE_REBUILD is set, ignoring artifacts already in the bucket" + else + for VERSION in $(cat r-versions-to-build.txt); do + URL="$BASE/$(printf "$FMT" "$VERSION" "${ARCH_ID}")" + if [ "$(curl -s -o /dev/null -w '%{http_code}' -I "$URL")" = "200" ]; then + echo "$VERSION" >> r-versions-existing.txt + fi + done + fi - echo "Already present in s3:"; cat r-versions-existing.txt backend_options: kubernetes: diff --git a/CLAUDE.md b/CLAUDE.md index 911139b..9d6c279 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,4 +10,11 @@ CI pipelines (Crow, in `.crow/`) that build R binaries and upload them to Backbl ## Gotchas - **B2 requires authentication for the list-bucket API.** Anonymous GET only works for individual public-read objects — an empty bucket listing usually means missing credentials, not an empty bucket. This silently broke rebuild dedup once. +- **R's configure silently disables graphics devices when headers are missing.** +It never fails the build, so a missing `-dev` package yields a working R whose `capabilities()` reports `cairo`, `png`, `jpeg` and `tiff` as `FALSE`. +Installing the runtime libraries on the target system afterwards cannot bring compiled-out devices back; the binary has to be rebuilt. +`builder/build.sh` therefore asserts `REQUIRED_CAPABILITIES` (set per Dockerfile) after `make install`. +- **The `--with-x=no` on the Alpine images does not disable cairo.** R's bitmap and pangocairo checks run independently of `R_X11`, so `png()` and `svg()` work headless via cairo. +- **Artifact filenames carry only the R version, not the builder image.** +When a builder change alters the resulting binary, dedup would keep serving the stale artifact forever; set `FORCE_REBUILD` in `.crow/build.yaml` to rebuild and overwrite in place. - When a pipeline fails, fetch the Crow logs yourself instead of asking the user to paste them, and monitor reruns in the background until green. diff --git a/builder/Dockerfile.alpine-320 b/builder/Dockerfile.alpine-320 index b83ef43..726fa04 100644 --- a/builder/Dockerfile.alpine-320 +++ b/builder/Dockerfile.alpine-320 @@ -5,6 +5,13 @@ ENV OS_IDENTIFIER alpine-320 RUN set -x \ && apk add -q R-dev curl ca-certificates bash g++ tzdata openjdk17 texmf-dist texlive-full tar sed patch tcl tk tk-dev xvfb libdeflate libdeflate-dev +# Graphics devices. R's configure only enables the cairo/png/jpeg/tiff devices +# when the corresponding headers are present at compile time, and pkgconf must +# exist for the pangocairo check to run at all. Without these, capabilities() +# reports cairo/png/jpeg/tiff as FALSE, and installing the runtime libraries +# afterwards cannot bring the compiled-out devices back. +RUN apk add -q pkgconf cairo-dev pango-dev fontconfig-dev freetype-dev libpng-dev jpeg-dev tiff-dev zlib-dev + # Install s5cmd for S3 uploads (much faster than AWS CLI) RUN ARCH=$(uname -m); if [ "$ARCH" = "x86_64" ]; then S5CMD_ARCH="64bit"; else S5CMD_ARCH="arm64"; fi && \ curl -sL "https://github.com/peak/s5cmd/releases/download/v2.3.0/s5cmd_2.3.0_Linux-${S5CMD_ARCH}.tar.gz" | tar xz -C /usr/local/bin s5cmd @@ -18,7 +25,12 @@ RUN chmod 0777 /opt # Override the default pager used by R ENV PAGER /usr/bin/pager -ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-blas --with-lapack" +# X11 stays off: these are headless images, and the cairo/bitmap devices are +# detected independently of X11, so png()/svg() still work via cairo. +ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-cairo --with-libpng --with-jpeglib --with-libtiff --with-blas --with-lapack" + +# Fail the build if a graphics device silently drops out again. +ENV REQUIRED_CAPABILITIES "cairo,png,jpeg,tiff" COPY package.alpine-320 /package.sh COPY build.sh . diff --git a/builder/Dockerfile.alpine-321 b/builder/Dockerfile.alpine-321 index 5d86190..e2298ef 100644 --- a/builder/Dockerfile.alpine-321 +++ b/builder/Dockerfile.alpine-321 @@ -5,6 +5,13 @@ ENV OS_IDENTIFIER alpine-321 RUN set -x \ && apk add -q R-dev curl ca-certificates bash g++ tzdata openjdk17 texmf-dist texlive-full tar sed patch tcl tk tk-dev xvfb libdeflate libdeflate-dev +# Graphics devices. R's configure only enables the cairo/png/jpeg/tiff devices +# when the corresponding headers are present at compile time, and pkgconf must +# exist for the pangocairo check to run at all. Without these, capabilities() +# reports cairo/png/jpeg/tiff as FALSE, and installing the runtime libraries +# afterwards cannot bring the compiled-out devices back. +RUN apk add -q pkgconf cairo-dev pango-dev fontconfig-dev freetype-dev libpng-dev jpeg-dev tiff-dev zlib-dev + # Install s5cmd for S3 uploads (much faster than AWS CLI) RUN ARCH=$(uname -m); if [ "$ARCH" = "x86_64" ]; then S5CMD_ARCH="64bit"; else S5CMD_ARCH="arm64"; fi && \ curl -sL "https://github.com/peak/s5cmd/releases/download/v2.3.0/s5cmd_2.3.0_Linux-${S5CMD_ARCH}.tar.gz" | tar xz -C /usr/local/bin s5cmd @@ -18,7 +25,12 @@ RUN chmod 0777 /opt # Override the default pager used by R ENV PAGER /usr/bin/pager -ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-blas --with-lapack" +# X11 stays off: these are headless images, and the cairo/bitmap devices are +# detected independently of X11, so png()/svg() still work via cairo. +ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-cairo --with-libpng --with-jpeglib --with-libtiff --with-blas --with-lapack" + +# Fail the build if a graphics device silently drops out again. +ENV REQUIRED_CAPABILITIES "cairo,png,jpeg,tiff" COPY package.alpine-321 /package.sh COPY build.sh . diff --git a/builder/Dockerfile.alpine-322 b/builder/Dockerfile.alpine-322 index 0a3695a..23a7793 100644 --- a/builder/Dockerfile.alpine-322 +++ b/builder/Dockerfile.alpine-322 @@ -5,6 +5,13 @@ ENV OS_IDENTIFIER alpine-322 RUN set -x \ && apk add -q R-dev curl ca-certificates bash g++ tzdata openjdk17 texmf-dist texlive-full tar sed patch tcl tk tk-dev xvfb libdeflate libdeflate-dev +# Graphics devices. R's configure only enables the cairo/png/jpeg/tiff devices +# when the corresponding headers are present at compile time, and pkgconf must +# exist for the pangocairo check to run at all. Without these, capabilities() +# reports cairo/png/jpeg/tiff as FALSE, and installing the runtime libraries +# afterwards cannot bring the compiled-out devices back. +RUN apk add -q pkgconf cairo-dev pango-dev fontconfig-dev freetype-dev libpng-dev jpeg-dev tiff-dev zlib-dev + # Install s5cmd for S3 uploads (much faster than AWS CLI) RUN ARCH=$(uname -m); if [ "$ARCH" = "x86_64" ]; then S5CMD_ARCH="64bit"; else S5CMD_ARCH="arm64"; fi && \ curl -sL "https://github.com/peak/s5cmd/releases/download/v2.3.0/s5cmd_2.3.0_Linux-${S5CMD_ARCH}.tar.gz" | tar xz -C /usr/local/bin s5cmd @@ -18,7 +25,12 @@ RUN chmod 0777 /opt # Override the default pager used by R ENV PAGER /usr/bin/pager -ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-blas --with-lapack" +# X11 stays off: these are headless images, and the cairo/bitmap devices are +# detected independently of X11, so png()/svg() still work via cairo. +ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-cairo --with-libpng --with-jpeglib --with-libtiff --with-blas --with-lapack" + +# Fail the build if a graphics device silently drops out again. +ENV REQUIRED_CAPABILITIES "cairo,png,jpeg,tiff" COPY package.alpine-322 /package.sh COPY build.sh . diff --git a/builder/Dockerfile.alpine-323 b/builder/Dockerfile.alpine-323 index 9e7fa0b..82b8c5a 100644 --- a/builder/Dockerfile.alpine-323 +++ b/builder/Dockerfile.alpine-323 @@ -5,6 +5,13 @@ ENV OS_IDENTIFIER alpine-323 RUN set -x \ && apk add -q R-dev curl ca-certificates bash g++ tzdata openjdk17 texmf-dist texlive-full tar sed patch tcl tk tk-dev xvfb libdeflate libdeflate-dev +# Graphics devices. R's configure only enables the cairo/png/jpeg/tiff devices +# when the corresponding headers are present at compile time, and pkgconf must +# exist for the pangocairo check to run at all. Without these, capabilities() +# reports cairo/png/jpeg/tiff as FALSE, and installing the runtime libraries +# afterwards cannot bring the compiled-out devices back. +RUN apk add -q pkgconf cairo-dev pango-dev fontconfig-dev freetype-dev libpng-dev jpeg-dev tiff-dev zlib-dev + # Install s5cmd for S3 uploads (much faster than AWS CLI) RUN ARCH=$(uname -m); if [ "$ARCH" = "x86_64" ]; then S5CMD_ARCH="64bit"; else S5CMD_ARCH="arm64"; fi && \ curl -sL "https://github.com/peak/s5cmd/releases/download/v2.3.0/s5cmd_2.3.0_Linux-${S5CMD_ARCH}.tar.gz" | tar xz -C /usr/local/bin s5cmd @@ -18,7 +25,12 @@ RUN chmod 0777 /opt # Override the default pager used by R ENV PAGER /usr/bin/pager -ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-blas --with-lapack" +# X11 stays off: these are headless images, and the cairo/bitmap devices are +# detected independently of X11, so png()/svg() still work via cairo. +ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-cairo --with-libpng --with-jpeglib --with-libtiff --with-blas --with-lapack" + +# Fail the build if a graphics device silently drops out again. +ENV REQUIRED_CAPABILITIES "cairo,png,jpeg,tiff" COPY package.alpine-322 /package.sh COPY build.sh . diff --git a/builder/Dockerfile.alpine-324 b/builder/Dockerfile.alpine-324 index b3daa01..c95ce40 100644 --- a/builder/Dockerfile.alpine-324 +++ b/builder/Dockerfile.alpine-324 @@ -5,6 +5,13 @@ ENV OS_IDENTIFIER alpine-324 RUN set -x \ && apk add -q R-dev curl ca-certificates bash g++ tzdata openjdk17 texmf-dist texlive-full tar sed patch tcl tk tk-dev xvfb libdeflate libdeflate-dev +# Graphics devices. R's configure only enables the cairo/png/jpeg/tiff devices +# when the corresponding headers are present at compile time, and pkgconf must +# exist for the pangocairo check to run at all. Without these, capabilities() +# reports cairo/png/jpeg/tiff as FALSE, and installing the runtime libraries +# afterwards cannot bring the compiled-out devices back. +RUN apk add -q pkgconf cairo-dev pango-dev fontconfig-dev freetype-dev libpng-dev jpeg-dev tiff-dev zlib-dev + # Install s5cmd for S3 uploads (much faster than AWS CLI) RUN ARCH=$(uname -m); if [ "$ARCH" = "x86_64" ]; then S5CMD_ARCH="64bit"; else S5CMD_ARCH="arm64"; fi && \ curl -sL "https://github.com/peak/s5cmd/releases/download/v2.3.0/s5cmd_2.3.0_Linux-${S5CMD_ARCH}.tar.gz" | tar xz -C /usr/local/bin s5cmd @@ -18,7 +25,12 @@ RUN chmod 0777 /opt # Override the default pager used by R ENV PAGER /usr/bin/pager -ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-blas --with-lapack" +# X11 stays off: these are headless images, and the cairo/bitmap devices are +# detected independently of X11, so png()/svg() still work via cairo. +ENV CONFIGURE_OPTIONS "--enable-R-shlib --with-tcltk --enable-memory-profiling --with-x=no --with-cairo --with-libpng --with-jpeglib --with-libtiff --with-blas --with-lapack" + +# Fail the build if a graphics device silently drops out again. +ENV REQUIRED_CAPABILITIES "cairo,png,jpeg,tiff" COPY package.alpine-324 /package.sh COPY build.sh . diff --git a/builder/build.sh b/builder/build.sh index 427da6e..7a8e59e 100755 --- a/builder/build.sh +++ b/builder/build.sh @@ -209,6 +209,47 @@ local({ EOF } +# verify_capabilities() +# Guard against a graphics device being silently compiled out. R's configure +# only enables the cairo/png/jpeg/tiff devices when the matching headers were +# present at compile time, and it does not fail when they are missing, so a +# missing -dev package produces a working R that simply cannot draw. Once that +# tarball is published, installing the runtime libraries on the target system +# cannot bring the devices back; the build has to be redone. +# +# Dockerfiles opt in by setting REQUIRED_CAPABILITIES to a comma-separated list +# of capabilities() names, e.g. "cairo,png,jpeg,tiff". +verify_capabilities() { + if [ -z "${REQUIRED_CAPABILITIES}" ]; then + return 0 + fi + + echo "Verifying required R capabilities: ${REQUIRED_CAPABILITIES}" + + # The full capabilities() table goes to stderr so it shows up in the build + # log; stdout carries only the space-separated list of missing entries. + local missing + missing=$("${R_INSTALL_PATH}/bin/Rscript" --vanilla -e ' + required <- trimws(strsplit(Sys.getenv("REQUIRED_CAPABILITIES"), ",")[[1]]) + required <- required[nzchar(required)] + caps <- capabilities() + writeLines(capture.output(print(caps)), con = stderr()) + ok <- (required %in% names(caps)) & (unname(caps[required]) %in% TRUE) + cat(paste(required[!ok], collapse = " "), "\n", sep = "") + ') || { + echo "ERROR: could not query capabilities() from the freshly built R" >&2 + return 1 + } + + if [ -n "${missing}" ]; then + echo "ERROR: R ${R_VERSION} for ${OS_IDENTIFIER} is missing required capabilities: ${missing}" >&2 + echo "ERROR: install the matching -dev packages in builder/Dockerfile.${OS_IDENTIFIER} and rebuild." >&2 + return 1 + fi + + echo "All required capabilities are present." +} + # check for packager script ## If it exists this build is ready for packaging with nFPM, so run the script ## else do nothing @@ -236,6 +277,7 @@ set_up_environment fetch_r_source $R_VERSION patch_r $R_VERSION compile_r $R_VERSION +verify_capabilities package_r $R_VERSION archive_r upload_r $R_VERSION