Table of contents

• build-binaries
• R Package
• Containerfiles
• Build Process
• Build Environment
• Technical Details
• Creating/Updating the PACKAGES Index Files
• Resources
• Dependency Cache
• Inferring System Dependencies
• Metadata Database
• Support for Archived Versions
• remotes::install_version()
• pak::pak(package@version)
• Lessons Learned
• URL Composition and Platform Identifiers
• Helpers
• Common Errors
• Packages with unresolved build failures
• CDN Settings

build-binaries

This project offers a framework for creating R package binaries on Linux across various architectures and distributions.

It achieves this through the integration of several components:

• R package bincraftR
• Containerfiles that define the build toolchain for each distribution
• S3 storage for storing the compiled binaries
• PostgreSQL database for recording build logs

R Package

The R package bincraftR powers everything. It provides the following functionality:

• build binaries
• archive packages following the CRAN-like directory structure
• upload package binaries to S3
• update the package index files (PACKAGES*)
• store build metadata, including error logs, in a PostgreSQL database

See the function reference on the pkgdown site for a full overview.

The focus of the R package is on usability rather than minimizing dependencies. The individual containerfiles include the package along with its dependencies. Bundling more R packages upfront helps reducing the number of additional packages needed when installing the dependencies for the individual packages to be built.

Containerfiles

The toolchain in the containerfile of each distribution is a core element for the build success of each package. The C compiler settings should be close to the recommended settings from CRAN and allow compatibility for most CRAN packages.

Especially Alpine is tricky as CRAN currently does not test R packages for Alpine compatibility (i.e. working with MUSL instead of GLIBC). Due to the different C library (MUSL instead of GLIBC), many R packages that include C/C++ code encounter errors.

Build Process

Tags for each package can be built in parallel via {future} through build_binary_package(). build_binary_package() builds all available tags of an R package by default. When setting tag = <X.Y.Z> or the special value tag = "latest", only these tags will be built.

For every package+tag combination:

1. Checkout tag(s) from GitHub CRAN mirror (e.g. https://github.com/ggplot2)
2. Build binaries
3. Upload binaries to S3
4. Archive old package versions and keep the latest one in the root
5. Delete local binaries after successful upload

Build Environment

Binaries are built on a mixed-architecture Kubernetes cluster using CI. Dedicated arm64/amd64 nodes are utilized to efficiently build the binaries. After all binaries for a specific architecture/OS combination are built, CRON jobs handle the processing of daily change operations. This elastic server architecture offers a robust and performant backend while minimizing costs.

Technical Details

Creating/Updating the PACKAGES Index Files

Currently, the {cranlike} and {desc} packages only work with files on a local file system. This is infeasible if the goal is to store binaries in S3. Storing binaries permanently on a disk-based file system would incur significantly higher costs, especially when operating in the cloud.

Hence, {cranlike} and {desc} were forked to handle files in S3 (through {s3fs}).

Resources

For efficient binary building, reasonably sized instances with high-performance CPUs are crucial to complete tasks within a reasonable timeframe.

During the process, it was observed that a single build can consume up to 14 GB of memory for certain packages. While the majority of packages require less than 2 GB of memory, the exact RAM requirements for individual packages are unpredictable. To prevent the risk of running out of memory (OOM) during builds, the memory allocation for individual builds should not be set below 14 GB.

Parallel builds were tested initially but led to occasional issues with parallel writes and rate limits. Currently, all builds are executed sequentially. It is important to note that parallelism is applied at the tag level, not at the package level, and this behavior cannot be modified at present.

Dependency Cache

A build cache for both R packages (/mnt/cache/R-pkgs) and ccache (/mnt/cache/ccache) is stored in a persistent volume for each architecture/OS combination to speed up dependency installation.

When updating the PACKAGES* index files (PACKAGES, PACKAGES.gz, PACKAGES.rds), processing all CRAN packages (21k+) takes around 40 minutes. Processing updates with an existing database file takes around 5 minutes.

Inferring System Dependencies

R package dependencies and their system dependencies are installed through {pak}. {pak} allows for parallel downloads and installation, significantly speeding up package installation compared to install.packages(). Additionally, it automatically infers package dependencies using JSON rules from rstudio/r-system-requirements. Not all R packages specify required system dependencies in their DESCRIPTION file, and not all listed dependencies have existing rules in rstudio/r-system-requirements. For Alpine, no rules existed until recently, establishing a foundation for semi-automated package installation on Alpine Linux.

Metadata Database

The build metadata is stored in a PostgreSQL database. The database has a public endpoint at r-binaries.devxy.io and port 15432. It contains one table named single_builds, which holds the build metadata for each package:

package_name	tag	platform	error_occurred	build_timestamp	build_duration	error	size	removed

Column types:

Column Name	Data Type
package_name	character varying(255)
tag	character varying(255)
platform	character varying(255)
error_occurred	boolean
build_timestamp	timestamp without time zone
build_duration	numeric(1000,2)
error	text
size	numeric(1000,2)
removed	boolean

Alternatively, use \d+ single_builds. Note: There is currently no read-only role available to connect to the DB as a viewer.

A Shiny dashboard providing a search functionality of the database and grouped statistics is available at https://app.devxy.io/app/r-package-binaries-dashboard with the source repo living at https://gitlab.com/devxy/r-package-binaries/shiny.

Support for Archived Versions

remotes::install_version()

Is supported by writing Meta/archive.rds during each package index update, listing all available archived packages.

pak::pak(package@version)

pak searches for Archive/<package> and can install all versions it finds.

Ensure to use a clean cache if other repositories have been used previously. If in doubt or when testing, call pak::meta_clean(force = TRUE).

Lessons Learned

• The newest R version needs to be used to build binaries. Some packages depend on the "recommended" packages and attempt to install them as dependencies. This fails for older R versions, e.g., if R 4.0.5 tries to install Matrix from 4.4.x. Generally, no issues installing packages built with newer versions were observed yet, hence it should be fine to build with the newest version and install these packages using any version.

• Graphical R packages are challenging. Most can be processed by starting R with xvfb-run R, but some still encounter issues and get stuck during processing.

• A few dozen packages rely on exotic external dependencies that must be installed from source. Including all of these would significantly increase the container image size for minimal gain. A common external dependency on which many R packages depend is JAGS. Because of this, it has been included in the Containerfiles to enable successful builds for several dozen R packages.

• Catching build failures at all possible build stages is difficult. Timeouts may occur, dependencies can fail to install, and some tags in the GitHub mirror might not include valid DESCRIPTION files. It is crucial to catch errors, continue the build, and make the build process as robust as possible.

URL Composition and Platform Identifiers

Platform identifiers have been aligned with those used in https://github.com/rstudio/r-system-requirements to ensure proper recognition by the automatic syslib dependency installer of pak, specifically via the environment variable PKG_SYSREQS_PLATFORM:

• redhat-9
• redhat-8
• ubuntu-2204
• ubuntu-2404
• alpine-320
• alpine-321

The final repository URL is structured slightly differently and follows the format of the Posit Packagemanager:

https://<domain>/<arch>/<OS>/latest

Example: https://cran.devxy.io/arm64/rhel9/latest

Technically, no date-based snapshots are planned, so this component from the Posit PM structure is not included.

Helpers

Helper scripts are located in local/. These scripts can assist in various situations, such as manually processing packages or filtering specific information from the metadata database.

Common Errors

Below is a collection of raw errors observed during the build process:

* installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n"

Tag does not have a NAMESPACE file and hence cannot be built.

"* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n"

Dependency not available: Either because the dependency was not declared or errored itself during installation.

 In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n  162 |   double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n      |                                                             \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n  277 |   ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n      |                               \033[01;31m\033[K^~\033[m\033[K\n      |                               \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n

Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17.

---

When inferring dependencies:

internal error 1 in memDecompress

Solution:

rm -rf  /mnt/cache/R-pkgs/pak /mnt/cache/pkgcache/ /root/.cache/R/
R -q -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'

Packages with unresolved build failures

Some packages are intentionally skipped right for "full builds" (i.e. building of all versions) as they result in getting stuck indefinitely. This applies mostly to very old package versions and the most recent version can usually be built successfully.

For others it might be due to exotic external dependencies which require manual installation.

Help in resolving these issues are highly welcome!

Name	Platform	Arch	Reason	Solved via	Date Created	Date Solved
Apollonius	redhat-8		gmp missing	2024-11-15
doBy	alpine-320	amd	hangs	2024-11-23
later	ubuntu 22 & 24	amd	hangs for early versions - xfvb?	2024-11-23
CoTiMA	ubuntu 22 & 24	amd	OOM?	2024-11-23
FrF2	alpine	arm	hangs	2024-11-23
FrF2.catlg128	alpine	arm	hangs	2024-11-23
DoE.base	alpine	arm/amd	hangs	2024-11-23
eha	alpine	amd	hangs	2024-11-23
gRain	alpine	amd	hangs	2024-11-23
gRbase	alpine	amd	hangs	2024-11-24
IDPmisc	alpine	amd	hangs	2024-11-24
RVAideMemoire	alpine	amd	hangs	2024-11-25
pbkrtest	alpine	arm	hangs	2024-11-25
seewave	alpine	arm	hangs	2024-11-25
spdep	alpine	arm	hangs	2024-12-03
compareGroups	alpine	arm	hangs	2024-12-11
SNPassoc	alpine	arm	hangs	2024-12-11
surveillance	alpine	arm	hangs	2024-12-13

CDN Settings

A CDN is used in front of the S3 bucket to efficiently distribute the binaries globally.

A "Perma-Cache" is enabled for three different regions around the world (DE, US, Asia). Once a binary is requested for the first time from a specific location, the asset is copied to the perma-cache and served from there for subsequent requests.

The CacheControl = "no-cache" header is set for all PACKAGES* files to ensure users always receive the latest version, as these files change daily. Additionally, the cache for these files is purged after every update.