build-cran-binaries/README.md

# bincraftR

[TOC]

This project offers a framework for creating R package binaries on Linux across various architectures and distributions.

It achieves this through the integration of several components:

- **R package `bincraftR`**
- **Containerfiles** that define the build toolchain for each distribution
- **S3 storage** for storing the compiled binaries
- **PostgreSQL database** for recording build logs

## R Package

The R package `bincraftR` is the engine behind everything.
It provides functions that can:

- build binaries
- archive packages following the CRAN-like directory structure
- upload package binaries to S3
- update the package index files (`PACKAGES*`)
- store build metadata, including error logs, in a PostgreSQL database

See the function reference on the pkgdown site for a full overview.

The focus of the R package is on usability rather than minimizing dependencies.
The individual containerfiles include the package along with its dependencies.
Bundling more R packages upfront helps reduce the number of additional packages needed when installing the dependencies for building packages.

## Containerfiles

The toolchain in the containerfile of each distribution is a very important element for the build success of the packages.
The C compiler settings should be close to the recommended settings from CRAN and allow compatibility for most CRAN packages.

Here, especially Alpine is tricky as CRAN does not test R packages for Alpine.
Since Alpine uses a different C library (MUSL instead of GLIBC), many R packages that include C/C++ code encounter errors.

## Build Process

Tags for each package can be built in parallel via {future} through `build_binary_package()`.
`build_binary_package()` builds all available tags of an R package by default.
When setting `tag = <X.Y.Z>` or the special value `tag = "latest"`, only these tags will be built.

For every package+tag combination:

1. Checkout tag(s) from GitHub CRAN mirror (e.g. <https://github.com/ggplot2>)
1. Build binaries
1. Upload binaries to S3
1. Archive old package versions and keep the latest one in the root
1. Delete local binaries after successful upload

## Build Environment

Binaries are built on a mixed-architecture Kubernetes cluster using CI. 
Dedicated arm64 and amd64 nodes are utilized to efficiently build the binaries. 
After all binaries for a specific architecture/OS combination are built, CRON jobs handle the processing of daily change operations. 
This elastic server architecture offers a robust and performant backend while minimizing costs.

## Technical Details

### Creating/Updating the PACKAGES Index Files

Currently, the {cranlike} and {desc} packages only work with files on a local file system.
This is infeasible if the goal is to store binaries in S3.
Storing binaries permanently on a disk-based file system would incur significantly higher costs, especially when operating in the cloud.

Hence, modified versions of {cranlike} and {desc} were created that are able to handle files in S3 (through {s3fs}).

### Resources

Reasonably sized instances with performant CPUs are important to build binaries in a reasonable time.
While building binaries, it was found that a single process might need up to 14 GB of memory, as certain packages on CRAN require that much to build.
While this applies to only a few packages and most do not exceed 2 GB of memory, the exact RAM requirement for each individual package is unknown.
To ensure that any package can be processed without the risk of running out of memory (OOM), a safety margin of using 16 GB of memory is the suggested minimum requirement.
This means that a VM with 16 GB of memory can build binaries sequentially.
With 32 GB of memory, two cores can be used to process multiple packages in parallel.

Important: the parallelism applies at the tag level, not at the package level, and this behavior cannot currently be changed.

### Dependency Cache

A build cache for both R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a persistent volume for each architecture/OS combination.
Additionally, the PACKAGES index files are persisted to speed up adding new packages when calling `upload_package_index()`.
Otherwise, the entire (SQLite) database would need to be created from scratch, which takes considerable time and requires numerous API calls to Backblaze.

Processing all CRAN packages (approximately 21k) takes around 40 minutes, while processing updates with an existing database file takes around 5 minutes.

### Inferring System Dependencies

R package dependencies and their system dependencies are installed through {pak}.
{pak} allows for parallel downloads and installation, significantly speeding up package installation compared to `install.packages()`.
Additionally, it automatically infers package dependencies using JSON rules from [rstudio/r-system-requirements](https://github.com/rstudio/r-system-requirements).
Not all R packages specify required system dependencies in their DESCRIPTION file, and not all listed dependencies have existing rules in `rstudio/r-system-requirements`.
For Alpine, no rules existed until recently, establishing a foundation for semi-automated package installation on Alpine Linux.

## Metadata Database

The build metadata is stored in a PostgreSQL database.
The database has a public endpoint at `r-binaries.devxy.io` and port `15432`.
The database contains one table named `single_builds`, which holds the build metadata for each package:

| package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size |
| ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- |

Column types:

- `package_name`: character varying(255)
- `tag`: character varying(255)
- `platform`: character varying(255)
- `error_occurred`: boolean
- `build_timestamp`: timestamp without time zone
- `build_duration`: numeric(1000,2)
- `error`: text
- `size`: numeric(1000,2)

Alternatively, use `\d+ single_builds`.

A shiny dashboard providing a search functionality of the database and grouped statistics is available in `shiny/`.

The self-hosted database is running on a Kubernetes Cluster in HA mode.

## Support for Archived Versions

### `remotes::install_version()`

`remotes::install_version()` searches for a `Meta/archive.rds` file in the `/src/contrib` directory.
This file must be a list of data frames containing information about the archived versions of the package.

Example:

```r
con <- gzcon(url(sprintf("%s/src/contrib/Meta/archive.rds",
                           c("CRAN" = "https://cloud.r-project.org")), "rb"))
foo = readRDS(con)
foo[[1]]
```

### `pak::pak(package@version)`

`pak` searches for `Archive/<package>` and can install all versions listed in it.
Ensure to use a clean cache if other repositories have been used previously.
If in doubt or when testing, call `pak::meta_clean(force = TRUE)`.

## Lessons Learned

- The newest R version needs to be used to build binaries.
  The reason is that some packages depend on the "recommended" packages and attempt to install them as dependencies.
  This fails for older R versions, e.g., if R 4.0.5 tries to install `Matrix` from 4.4.x.

- Graphical R packages are challenging.
  Most can be processed by starting R with `xvfb-run R`, but some still encounter issues and get stuck during processing.

- A few dozen packages rely on exotic external dependencies that must be installed from source.
  Including all of these would significantly increase the container image size for minimal gain.
  A common external dependency on which many R packages depend is JAGS.
  Because of this, it has been included in the Containerfiles to enable successful builds for several dozen R packages.

- Catching build failures at all possible build stages is difficult.
  Timeouts may occur, dependencies can fail to install, and some tags in the GitHub mirror might not include valid DESCRIPTION files.
  It is crucial to catch errors, continue the build, and make the build process as robust as possible.

## URL Composition and Platform Identifiers

Platform identifiers have been aligned with those used in <https://github.com/rstudio/r-system-requirements> to ensure proper recognition by the automatic syslib dependency installer of `pak`, specifically via the environment variable `PKG_SYSREQS_PLATFORM`:

- redhat-9
- redhat-8
- ubuntu-2204
- ubuntu-2404
- alpine-320

The final repository URL is structured slightly differently and follows the format of the Posit Packagemanager:

`https://<domain>/<arch>/<OS>/latest`

Example: <https://cran.devxy.io/arm64/rhel9/latest>

Technically, no date-based snapshots are planned, so this component from the Posit PM structure is not included.

## Helpers

Helper scripts are located in `local/`. 
These scripts can assist in various situations, such as manually processing packages or filtering specific information from the metadata database.

## Common Errors

Below is a collection of raw errors observed during the build process:
<details>

```
* installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n"
```

Tag does not have a NAMESPACE file and hence cannot be built.

```
"* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n"
```

Dependency not available: Either because the dependency was not declared or errored itself during installation.

```
 In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n  162 |   double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n      |                                                             \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n  277 |   ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n      |                               \033[01;31m\033[K^~\033[m\033[K\n      |                               \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n
```

Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17.

---

When inferring dependencies:

```sh
internal error 1 in memDecompress
```

Solution:

```sh
rm -rf /mnt/cache/R-pkgs/pkgcache/ /mnt/cache/R-pkgs/pak /mnt/cache/pkgcache/ /root/.cache/R/
R -q -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'
```

</details>

## Packages Skipped on Purpose

Some packages have been intentionally skipped after multiple build attempts. 
The reasons for this vary, and ideally, solutions can be found over the long term. 
Contributions to help resolve these issues are highly welcome!

## CDN Settings

A CDN is used in front of the S3 bucket to efficiently distribute the binaries globally. 

A "Perma-Cache" is enabled for three different regions around the world (DE, US, Asia). 
Once a binary is requested for the first time from a specific location, the asset is copied to the perma-cache and served from there for subsequent requests.

The `CacheControl = "no-cache"` header is set for all PACKAGES* files to ensure users always receive the latest version, as these files change daily.

A monthly traffic limit of 50 TB is set on cran.devxy.io to prevent abuse and manage costs.