build-cran-binaries/README.md

---
gitea: none
include_toc: true
---

# build-binaries

This project offers a framework for creating R package binaries on Linux across various architectures and distributions.

It achieves this through the integration of several components:

- **R package `bincraft`**
- **Containerfiles** that define the build toolchain for each distribution
- **S3 storage** for storing the compiled binaries
- **PostgreSQL database** for recording build logs

## R Package

The R package [`bincraft`](https://gitlab.com/devxy/r-package-binaries/bincraft) powers everything.
It provides the following functionality:

- build binaries
- archive packages following the CRAN-like directory structure
- upload package binaries to S3
- update the package index files (`PACKAGES*`)
- store build metadata, including error logs, in a PostgreSQL database

See the function reference on the [pkgdown site](https://devxy.gitlab.io/r-package-binaries/bincraft/) for a full overview.

The focus of the R package is on usability rather than minimizing dependencies.
The individual containerfiles include the package along with its dependencies.
Bundling more R packages upfront helps reducing the number of additional packages needed when installing the dependencies for the individual packages to be built.

## Containerfiles

The toolchain in the containerfile of each distribution is a core element for the build success of each package.
The C compiler settings should be close to the recommended settings from CRAN and allow compatibility for most CRAN packages.

Especially Alpine is tricky as CRAN currently does not test R packages for Alpine compatibility (i.e. working with MUSL instead of GLIBC).
Due to the different C library (MUSL instead of GLIBC), many R packages that include C/C++ code encounter errors.

## Build Process

Tags for each package can be built in parallel via {future} through `build_binary_package()`.
`build_binary_package()` builds all available tags of an R package by default.
When setting `tag = <X.Y.Z>` or the special value `tag = "latest"`, only these tags will be built.

For every package+tag combination:

1. Checkout tag(s) from GitHub CRAN mirror (e.g. <https://github.com/ggplot2>)
1. Build binaries
1. Upload binaries to S3
1. Archive old package versions and keep the latest one in the root
1. Delete local binaries after successful upload

## Patching packages

Some CRAN packages fail to compile on specific platforms due to compiler- or OS-specific issues unrelated to the package itself.
The canonical example is `RcppParallel`, whose bundled TBB is linked in a way that lets a system TBB on the build host shadow it, so the published binary depends on a library the consumer does not have.
Because such packages are often transitive dependencies of many others, a single failure cascades: all dependents fail even though nothing is wrong with the dependent itself.

To address this, frequently-failing packages can be "patched" before they are installed — whether as a direct build target or a transitive dependency pulled in by `pak`.

The patch registry lives in `local/patches/registry.json`.
Each entry specifies a package and the platforms/versions it applies to, along with either lightweight build-time overrides (environment variables, configure arguments, Makevars) or a source diff (for deeper fixes).
See `local/patches/README.md` for the complete schema.

Patching uses a two-tier approach:

1. **Lightweight overrides:** environment variables, configure arguments, or Makevars settings applied during build — typically version-independent and fast.
2. **Source diffs:** unified diff patches applied to the unpacked source before building — more powerful but version-pinned.

The system is implemented in `bincraft`: when a package needs patching, `bincraft` pre-builds it with the patch and serves the patched binary to `pak`, ensuring transitive dependents receive the fixed package.
This way, the fix cascades to all packages that depend on it.

For the design rationale and architecture, see `specs/2026-06-30-package-patching-design.md`.

## Build Environment

Binaries are built on a mixed-architecture Kubernetes cluster using CI.
Dedicated `arm64`/`amd64` nodes are utilized to efficiently build the binaries.
After all binaries for a specific architecture/OS combination are built, CRON jobs handle the processing of daily change operations.
This elastic server architecture offers a robust and performant backend while minimizing costs.

## Technical Details

### Creating/Updating the PACKAGES Index Files

Currently, the {cranlike} and {desc} packages only work with files on a local file system.
This is infeasible if the goal is to store binaries in S3.
Storing binaries permanently on a disk-based file system would incur significantly higher costs, especially when operating in the cloud.

Hence, [{cranlike}](https://devxy.gitlab.io/r-package-binaries/bincraft/) and [{desc}](https://github.com/pat-s/desc/tree/description-from-remote) were forked to handle files in S3 (through {s3fs}).

### Resources

For efficient binary building, reasonably sized instances with high-performance CPUs are crucial to complete tasks within a reasonable timeframe.

During the process, it was observed that a single build can consume up to 14 GB of memory for certain packages.
While the majority of packages require less than 2 GB of memory, the exact RAM requirements for individual packages are unpredictable.
To prevent the risk of running out of memory (OOM) during builds, the memory allocation for individual builds should not be set below 14 GB.

Parallel builds were tested initially but led to occasional issues with parallel writes and rate limits.
Currently, all builds are executed sequentially. It is important to note that parallelism is applied at the tag level, not at the package level, and this behavior cannot be modified at present.

### Dependency Cache

A build cache for both R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a persistent volume for each architecture/OS combination to speed up dependency installation.

When updating the PACKAGES\* index files (`PACKAGES`, `PACKAGES.gz`, `PACKAGES.rds`), processing all CRAN packages (21k+) takes around 40 minutes.
Processing updates with an existing database file takes around 5 minutes.

### Inferring System Dependencies

R package dependencies and their system dependencies are installed through {pak}.
{pak} allows for parallel downloads and installation, significantly speeding up package installation compared to `install.packages()`.
Additionally, it automatically infers package dependencies using JSON rules from [rstudio/r-system-requirements](https://github.com/rstudio/r-system-requirements).
Not all R packages specify required system dependencies in their DESCRIPTION file, and not all listed dependencies have existing rules in `rstudio/r-system-requirements`.
For Alpine, no rules existed until recently, establishing a foundation for semi-automated package installation on Alpine Linux.

## Metadata Database

The build metadata is stored in a PostgreSQL database.
The database has a public endpoint at `r-binaries.devxy.io` and port `15432`.
It contains one table named `single_builds`, which holds the build metadata for each package:

| package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size | removed |
| ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- | ------- |

Column types:

| Column Name       | Data Type                   |
| ----------------- | --------------------------- |
| `package_name`    | character varying(255)      |
| `tag`             | character varying(255)      |
| `platform`        | character varying(255)      |
| `error_occurred`  | boolean                     |
| `build_timestamp` | timestamp without time zone |
| `build_duration`  | numeric(1000,2)             |
| `error`           | text                        |
| `size`            | numeric(1000,2)             |
| `removed`         | boolean                     |

Alternatively, use `\d+ single_builds`.
**Note**: There is currently no read-only role available to connect to the DB as a viewer.

A Shiny dashboard providing a search functionality of the database and grouped statistics is available at <https://app.devxy.io/app/r-package-binaries-dashboard> with the source repo living at <https://gitlab.com/devxy/r-package-binaries/shiny>.

## Support for Archived Versions

### `remotes::install_version()`

Is supported by writing `Meta/archive.rds` during each package index update, listing all available archived packages.

### `pak::pak(package@version)`

`pak` searches for `Archive/<package>` and can install all versions it finds.

Ensure to use a clean cache if other repositories have been used previously.
If in doubt or when testing, call `pak::meta_clean(force = TRUE)`.

## Lessons Learned

- The newest R version needs to be used to build binaries.
  Some packages depend on the "recommended" packages and attempt to install them as dependencies.
  This fails for older R versions, e.g., if R 4.0.5 tries to install `Matrix` from 4.4.x.
  Generally, no issues installing packages built with newer versions were observed yet, hence it should be fine to build with the newest version and install these packages using any version.

- Graphical R packages are challenging.
  Most can be processed by starting R with `xvfb-run R`, but some still encounter issues and get stuck during processing.

- A few dozen packages rely on exotic external dependencies that must be installed from source.
  Including all of these would significantly increase the container image size for minimal gain.
  A common external dependency on which many R packages depend is JAGS.
  Because of this, it has been included in the Containerfiles to enable successful builds for several dozen R packages.

- Catching build failures at all possible build stages is difficult.
  Timeouts may occur, dependencies can fail to install, and some tags in the GitHub mirror might not include valid DESCRIPTION files.
  It is crucial to catch errors, continue the build, and make the build process as robust as possible.

## URL Composition and Platform Identifiers

Platform identifiers have been aligned with those used in <https://github.com/rstudio/r-system-requirements> to ensure proper recognition by the automatic syslib dependency installer of `pak`, specifically via the environment variable `PKG_SYSREQS_PLATFORM`:

- redhat-9
- redhat-8
- ubuntu-2204
- ubuntu-2404
- alpine-320
- alpine-321

The final repository URL is structured slightly differently and follows the format of the Posit Packagemanager:

`https://<domain>/<arch>/<OS>/latest`

Example: <https://cran.devxy.io/arm64/rhel9/latest>

Technically, no date-based snapshots are planned, so this component from the Posit PM structure is not included.

## Helpers

Helper scripts are located in `local/`.
These scripts can assist in various situations, such as manually processing packages or filtering specific information from the metadata database.

## Common Errors

Below is a collection of raw errors observed during the build process:

<details>

```text
* installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n"
```

Tag does not have a NAMESPACE file and hence cannot be built.

```text
"* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n"
```

Dependency not available: Either because the dependency was not declared or errored itself during installation.

```text
 In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n  162 |   double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n      |                                                             \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n  277 |   ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n      |                               \033[01;31m\033[K^~\033[m\033[K\n      |                               \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.2/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n
```

Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17.

---

When inferring dependencies:

```sh
internal error 1 in memDecompress
```

Solution:

```sh
rm -rf  /mnt/cache/R-pkgs/pak /mnt/cache/pkgcache/ /root/.cache/R/
R -q -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'
```

</details>

## Packages with unresolved build failures

Some packages are intentionally skipped right for "full builds" (i.e. building of all versions) as they result in getting stuck indefinitely.
This applies mostly to very old package versions and the most recent version can usually be built successfully.

For others it might be due to exotic external dependencies which require manual installation.

Help in resolving these issues are highly welcome!

| Name          | Platform       | Arch    | Reason                           | Date Created | Date Solved |
| ------------- | -------------- | ------- | -------------------------------- | ------------ | ----------- |
| Apollonius    | redhat-8       |         | gmp missing                      | 2024-11-15   |             |
| doBy          | alpine-320     | amd     | hangs                            | 2024-11-23   |             |
| later         | ubuntu 22 & 24 | amd     | hangs for early versions - xfvb? | 2024-11-23   |             |
| CoTiMA        | ubuntu 22 & 24 | amd     | OOM?                             | 2024-11-23   |             |
| FrF2          | alpine         | arm     | hangs                            | 2024-11-23   |             |
| FrF2.catlg128 | alpine         | arm     | hangs                            | 2024-11-23   |             |
| DoE.base      | alpine         | arm/amd | hangs                            | 2024-11-23   |             |
| eha           | alpine         | amd     | hangs                            | 2024-11-23   |             |
| gRain         | alpine         | amd     | hangs                            | 2024-11-23   |             |
| gRbase        | alpine         | amd     | hangs                            | 2024-11-24   |             |
| IDPmisc       | alpine         | amd     | hangs                            | 2024-11-24   |             |
| RVAideMemoire | alpine         | amd     | hangs                            | 2024-11-25   |             |
| pbkrtest      | alpine         | arm     | hangs                            | 2024-11-25   |             |
| seewave       | alpine         | arm     | hangs                            | 2024-11-25   |             |
| spdep         | alpine         | arm     | hangs                            | 2024-12-03   |             |
| compareGroups | alpine         | arm     | hangs                            | 2024-12-11   |             |
| SNPassoc      | alpine         | arm     | hangs                            | 2024-12-11   |             |
| surveillance  | alpine         | arm     | hangs                            | 2024-12-13   |             |

## CDN Settings

A CDN is used in front of the S3 bucket to efficiently distribute the binaries globally.

A "Perma-Cache" is enabled for three different regions around the world (DE, US, Asia).
Once a binary is requested for the first time from a specific location, the asset is copied to the perma-cache and served from there for subsequent requests.

The `CacheControl = "no-cache"` header is set for all PACKAGES\* files to ensure users always receive the latest version, as these files change daily.
Additionally, the cache for these files is purged after every update.