devxy/build-cran-binaries

Watch

Fork 0

feat: serve R >= 4.6 custom binary types (R_PLATFORM_PKGTYPE) alongside UA routing #227

New issue

Open

opened 2026-10-02 13:41:34 +00:00 by pat-s · 1 comment

pat-s commented

2026-10-02 13:41:34 +00:00

Owner

Summary

Support R ≥ 4.6 custom binary types (R_PLATFORM_PKGTYPE) on cran.rpkgs.com and the binary store, so clients can find binaries by path instead of relying only on User-Agent sniffing.

Background

Simon Urbanek pointed this out in the r-devel thread on the User-Agent patch (https://stat.ethz.ch/pipermail/r-devel/2026-September/084687.html).
It shipped in R 4.6.0, in a series of Simon's commits from 2026-02-03 (R trunk r89368–r89372, plus r89499 on 2026-02-28):

R_PLATFORM_PKGTYPE env var overrides .Platform$pkgType (src/main/platform.c, r89368).
contrib.url() maps <system>.binary.<build> to bin/<system>/<build>/contrib/<x.y> (r89369).
install.packages() / write_PACKAGES() support custom binaries named .tar.{gz,bz2,xz,zst,zstd} (r89370).
When Version is identical, a newer Built timestamp now counts as an update, so a rebuilt binary is picked up without a version bump (r89371).
At install time, the platform in Built: is checked against the running R (r89499, .is.built.compatible() in utils/R/unix/mac.install.R).

Verified on R 4.6.1:

R_PLATFORM_PKGTYPE=linux.binary.amd64-noble Rscript -e 'contrib.url("https://cran.rpkgs.com", "binary")'
# https://cran.rpkgs.com/bin/linux/amd64-noble/contrib/4.6

With the variable set, getOption("pkgType") becomes "both".
install.packages() then reads both bin/linux/<build>/contrib/<x.y> and src/contrib and prefers the binary.
Without it, Linux R stays on "source" and only requests src/contrib, which is what the edge router handles today.

Proposal

Serve both routes from the same storage, without duplicating tarballs:

Path route (new): /bin/linux/<build>/contrib/<x.y>/… maps to the existing slot (<arch>/<os>/latest/src/contrib plus the per-minor <x.y>/ union index from the per-minor routing design).
This is a deterministic path rewrite, so it needs no Vary: User-Agent and caches cleanly.
User-Agent route (existing): keep edge/rpkgs-router.ts for src/contrib requests from R < 4.6 and from R ≥ 4.6 without the env var.
CRAN fallback (existing): an unknown platform still goes to CRAN sources.

Open questions / requirements

<build> naming. It must match [[:alnum:]_-]+.
Options: our slot names (amd64-noble, arm64-alpine324) or Simon's example form (x86_64-ubuntu-noble), which is derived from R.version$arch and os-release ID + VERSION_CODENAME/VERSION_ID.
The derived form is more likely to be adopted by others (rig, rocker, Posit builds) and can be computed from os-release without a lookup table.
The edge can accept aliases.
File: field in PACKAGES. For custom types, download.packages() defaults the filename to <pkg>_<ver>.tar.xz unless the record has File: (utils/R/packages.R:901-908).
Our tarballs are .tar.gz, so the index served on the path route must carry File:, or the edge must alias the name.
The compatibility check is coarse. .is.built.compatible() only compares arch and linux vs non-linux.
x86_64-pc-linux-musl vs x86_64-pc-linux-gnu passes, so a misrouted glibc binary on Alpine is not caught.
Our Built: stamps (see the Built-triple / paquetier coupling) must stay accurate, and R builds can add a stricter options(R.platform.is.compatible = function(b.platform) …).
Built-timestamp updates. With the new tie-break, a fixed source-fallback rebuild with the same version reaches users via update.packages(), as long as Built is bumped.
Upstream bug (minor). utils/R/packages2.R:579 uses [[:lower::]] (extra colon), so grepl() never matches.
The install.packages.check.source branch is therefore skipped for every *.binary type, including mac.binary.
This is worth reporting on r-devel.

Making the information available by default

The path route only helps when R_PLATFORM_PKGTYPE is set.
To cover as many clients as possible:

Set it in Renviron.site in every image we publish (reg.devxy.io/r/*) and in the bincraft build images.
Ship a one-line installer / docs snippet that derives the value from os-release, for users on stock R.
Propose it to R distributors (rig, rocker, Posit R builds, distro R packages), who per R's NEWS are the intended setters.
Keep the User-Agent route as a permanent fallback, so not setting the variable never makes things worse than today.

Tasks

Decide the <build> naming scheme and aliases.
Prototype the /bin/linux/<build>/contrib/<x.y> rewrite in edge/rpkgs-router.ts with tests.
Make sure the served PACKAGES carries File: (or alias .tar.xz to .tar.gz).
End-to-end test with R_PLATFORM_PKGTYPE on R 4.6 for noble, an el slot and alpine324, on amd64 and arm64.
Set R_PLATFORM_PKGTYPE in our R images.
Document the setup on cran.rpkgs.com.
Report the [[:lower::]] typo upstream.

## Summary Support R ≥ 4.6 custom binary types (`R_PLATFORM_PKGTYPE`) on cran.rpkgs.com and the binary store, so clients can find binaries by path instead of relying only on User-Agent sniffing. ## Background Simon Urbanek pointed this out in the r-devel thread on the User-Agent patch (https://stat.ethz.ch/pipermail/r-devel/2026-September/084687.html). It shipped in R 4.6.0, in a series of Simon's commits from 2026-02-03 (R trunk r89368–r89372, plus r89499 on 2026-02-28): - `R_PLATFORM_PKGTYPE` env var overrides `.Platform$pkgType` (`src/main/platform.c`, r89368). - `contrib.url()` maps `<system>.binary.<build>` to `bin/<system>/<build>/contrib/<x.y>` (r89369). - `install.packages()` / `write_PACKAGES()` support custom binaries named `.tar.{gz,bz2,xz,zst,zstd}` (r89370). - When `Version` is identical, a newer `Built` timestamp now counts as an update, so a rebuilt binary is picked up without a version bump (r89371). - At install time, the platform in `Built:` is checked against the running R (r89499, `.is.built.compatible()` in `utils/R/unix/mac.install.R`). Verified on R 4.6.1: ```sh R_PLATFORM_PKGTYPE=linux.binary.amd64-noble Rscript -e 'contrib.url("https://cran.rpkgs.com", "binary")' # https://cran.rpkgs.com/bin/linux/amd64-noble/contrib/4.6 ``` With the variable set, `getOption("pkgType")` becomes `"both"`. `install.packages()` then reads both `bin/linux/<build>/contrib/<x.y>` and `src/contrib` and prefers the binary. Without it, Linux R stays on `"source"` and only requests `src/contrib`, which is what the edge router handles today. ## Proposal Serve both routes from the same storage, without duplicating tarballs: 1. **Path route (new):** `/bin/linux/<build>/contrib/<x.y>/…` maps to the existing slot (`<arch>/<os>/latest/src/contrib` plus the per-minor `<x.y>/` union index from the per-minor routing design). This is a deterministic path rewrite, so it needs no `Vary: User-Agent` and caches cleanly. 2. **User-Agent route (existing):** keep `edge/rpkgs-router.ts` for `src/contrib` requests from R < 4.6 and from R ≥ 4.6 without the env var. 3. **CRAN fallback (existing):** an unknown platform still goes to CRAN sources. ## Open questions / requirements - **`<build>` naming.** It must match `[[:alnum:]_-]+`. Options: our slot names (`amd64-noble`, `arm64-alpine324`) or Simon's example form (`x86_64-ubuntu-noble`), which is derived from `R.version$arch` and os-release `ID` + `VERSION_CODENAME`/`VERSION_ID`. The derived form is more likely to be adopted by others (rig, rocker, Posit builds) and can be computed from os-release without a lookup table. The edge can accept aliases. - **`File:` field in PACKAGES.** For custom types, `download.packages()` defaults the filename to `<pkg>_<ver>.tar.xz` unless the record has `File:` (`utils/R/packages.R:901-908`). Our tarballs are `.tar.gz`, so the index served on the path route must carry `File:`, or the edge must alias the name. - **The compatibility check is coarse.** `.is.built.compatible()` only compares arch and linux vs non-linux. `x86_64-pc-linux-musl` vs `x86_64-pc-linux-gnu` passes, so a misrouted glibc binary on Alpine is **not** caught. Our `Built:` stamps (see the Built-triple / paquetier coupling) must stay accurate, and R builds can add a stricter `options(R.platform.is.compatible = function(b.platform) …)`. - **Built-timestamp updates.** With the new tie-break, a fixed source-fallback rebuild with the same version reaches users via `update.packages()`, as long as `Built` is bumped. - **Upstream bug (minor).** `utils/R/packages2.R:579` uses `[[:lower::]]` (extra colon), so `grepl()` never matches. The `install.packages.check.source` branch is therefore skipped for every `*.binary` type, including `mac.binary`. This is worth reporting on r-devel. ## Making the information available by default The path route only helps when `R_PLATFORM_PKGTYPE` is set. To cover as many clients as possible: - Set it in `Renviron.site` in every image we publish (`reg.devxy.io/r/*`) and in the bincraft build images. - Ship a one-line installer / docs snippet that derives the value from os-release, for users on stock R. - Propose it to R distributors (rig, rocker, Posit R builds, distro R packages), who per R's NEWS are the intended setters. - Keep the User-Agent route as a permanent fallback, so not setting the variable never makes things worse than today. ## Tasks - [ ] Decide the `<build>` naming scheme and aliases. - [ ] Prototype the `/bin/linux/<build>/contrib/<x.y>` rewrite in `edge/rpkgs-router.ts` with tests. - [ ] Make sure the served PACKAGES carries `File:` (or alias `.tar.xz` to `.tar.gz`). - [ ] End-to-end test with `R_PLATFORM_PKGTYPE` on R 4.6 for noble, an el slot and alpine324, on amd64 and arm64. - [ ] Set `R_PLATFORM_PKGTYPE` in our R images. - [ ] Document the setup on cran.rpkgs.com. - [ ] Report the `[[:lower::]]` typo upstream.

pat-s commented

2026-10-02 13:47:16 +00:00

Author

Owner

Follow-up: backwards compatibility and unconfigured R

Existing paths stay canonical

The existing /<arch>/<os>/latest/src/contrib[/<x.y>] paths remain the storage layout and keep working unchanged.
/bin/linux/<build>/contrib/<x.y> is only an edge rewrite onto them; nothing is moved or duplicated.
If the <build> naming uses the os-release form (x86_64-ubuntu-noble), the edge also accepts slot-style aliases (amd64-noble), so every published form resolves.
Router tests must cover old paths, new paths and aliases.

What a stock R actually sends (measured)

R 4.6.1, Rscript --vanilla, default libcurl method, against a local header-logging server:

/src/contrib/PACKAGES.rds  libcurl/8.22.0
/src/contrib/PACKAGES.gz   libcurl/8.22.0

libcurl.c replaces any HTTPUserAgent starting with R ( by libcurl/<ver>, so an unconfigured Linux R sends no R version, platform or distro.
User-Agent routing therefore only works for clients that already set a custom HTTPUserAgent (R/x.y R (…)).
For an unconfigured R, neither route works; both need one piece of client-side configuration.

With R_PLATFORM_PKGTYPE=linux.binary.x86_64-ubuntu-noble, the same R requests /bin/linux/x86_64-ubuntu-noble/contrib/4.6/PACKAGES.rds, so the path route carries arch, distro, release and R minor without any User-Agent.

Consequence for the design

Since some configuration is always needed, set it once per R installation at the distributor level, not per user:

our images and bincraft build images (Renviron.site);
rig, rocker, Posit r-builds, and distro packages (Debian r-base, Fedora/EPEL R, Alpine R), which R's NEWS names as the intended setters;
a documented one-liner for everyone else.

Unconfigured clients keep falling through to CRAN sources, which is no worse than today.

## Follow-up: backwards compatibility and unconfigured R ### Existing paths stay canonical The existing `/<arch>/<os>/latest/src/contrib[/<x.y>]` paths remain the storage layout and keep working unchanged. `/bin/linux/<build>/contrib/<x.y>` is only an edge rewrite onto them; nothing is moved or duplicated. If the `<build>` naming uses the os-release form (`x86_64-ubuntu-noble`), the edge also accepts slot-style aliases (`amd64-noble`), so every published form resolves. Router tests must cover old paths, new paths and aliases. ### What a stock R actually sends (measured) R 4.6.1, `Rscript --vanilla`, default `libcurl` method, against a local header-logging server: ``` /src/contrib/PACKAGES.rds libcurl/8.22.0 /src/contrib/PACKAGES.gz libcurl/8.22.0 ``` `libcurl.c` replaces any `HTTPUserAgent` starting with `R (` by `libcurl/<ver>`, so an unconfigured Linux R sends **no** R version, platform or distro. User-Agent routing therefore only works for clients that already set a custom `HTTPUserAgent` (`R/x.y R (…)`). For an unconfigured R, neither route works; both need one piece of client-side configuration. With `R_PLATFORM_PKGTYPE=linux.binary.x86_64-ubuntu-noble`, the same R requests `/bin/linux/x86_64-ubuntu-noble/contrib/4.6/PACKAGES.rds`, so the path route carries arch, distro, release and R minor without any User-Agent. ### Consequence for the design Since some configuration is always needed, set it once per R installation at the distributor level, not per user: - our images and bincraft build images (`Renviron.site`); - rig, rocker, Posit r-builds, and distro packages (Debian `r-base`, Fedora/EPEL `R`, Alpine `R`), which R's NEWS names as the intended setters; - a documented one-liner for everyone else. Unconfigured clients keep falling through to CRAN sources, which is no worse than today.

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

devxy/build-cran-binaries#227

No description provided.

Rows
Columns