devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 11 Pull requests 1 Releases Activity

docs: large README update

Browse source
This commit is contained in:
Patrick Schratz 2024-11-14 23:38:58 +01:00
commit bf46450b1a
Signed by: pat-s
GPG key ID: 3C6318841EF78925
1 changed files with 143 additions and 74 deletions
Download patch file Download diff file Expand all files Collapse all files

221
README.md
View file

@ -1,63 +1,104 @@
# README
# bincraftR
## Workflow
[TOC]
Most of the work is done through the R package `bincraftR`.
This project offers a framework for creating R package binaries on Linux across various architectures and distributions.
- `build_binary_package()`
- `install_system_dependencies()`
- `build_single_tag()`
- `upload_single_binary_to_s3()`
It achieves this through the integration of several components:
### Preparation of environment
- **R package `bincraftR`**
- **Containerfiles** that define the build toolchain for each distribution
- **S3 storage** for storing the compiled binaries
- **PostgreSQL database** for recording build logs
- Installation of R interpreter: done by downloading a prebuilt R binary for the specific OS.
- Installation of R and system dependencies:
- -> `install_system_dependencies()`
- Invoked through `build_binary_package()`
- Uses {pak} to install both R package dependencies and syslib dependencies
## R Package
### Building binaries
The R package `bincraftR` is the engine behind everything.
It provides functions that can:
Tags for each package are built in parallel via {future} by executing `build_binary_package()` via `future_mapply()`.
- build binaries
- archive packages following the CRAN-like directory structure
- upload package binaries to S3
- update the package index files (`PACKAGES*`)
- store build metadata, including error logs, in a PostgreSQL database
The build process in detail:
See the function reference on the pkgdown site for a full overview.
For every package+tag combination, do
The focus of the R package is on usability rather than minimizing dependencies.
The individual containerfiles include the package along with its dependencies.
Bundling more R packages upfront helps reduce the number of additional packages needed when installing the dependencies for building packages.
1. Checkout tag from GitHub CRAN mirror
2. Build binary package
3. Upload package binary to S3
4. Delete package binary (to free up space on PVC)
## Containerfiles
### Building the PACKAGES index file
The toolchain in the containerfile of each distribution is a very important element for the build success of the packages.
The C compiler settings should be close to the recommended settings from CRAN and allow compatibility for most CRAN packages.
The packages {cranlike} and {desc} only work with sources on a local file system.
This is infeasible for our approach with the data stored only in S3.
Otherwise, all binaries need to be present permanently on a static file system.
This would incur a lot of costs.
Here, especially Alpine is tricky as CRAN does not test R packages for Alpine.
Since Alpine uses a different C library (MUSL instead of GLIBC), many R packages that include C/C++ code encounter errors.
Hence, modified versions of {cranlike} and {desc} were created which are able to deal with files in S3 through {s3fs}.
## Build Process
## Resources
Tags for each package can be built in parallel via {future} through `build_binary_package()`.
`build_binary_package()` builds all available tags of an R package by default.
When setting `tag = <X.Y.Z>` or the special value `tag = "latest"`, only these tags will be built.
Binaries are build in parallel on a 16 Core 32 GB instance with 6 processes. This gives each process ~ 5 GB of memory for building the individual package+tag combination.
This looks excessive on the first look but some packages take up to 5 GB of memory when compiling from source.
Hence, we need to allocation these resources to ensure a smooth build process which does not result in an OOMKilled of the pod.
For every package+tag combination:
## Cache
1. Checkout tag(s) from GitHub CRAN mirror (e.g. <https://github.com/ggplot2>)
1. Build binaries
1. Upload binaries to S3
1. Archive old package versions and keep the latest one in the root
1. Delete local binaries after successful upload
A build cache for R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a PVC with a size of 25 GB.
Also, the PACKAGES index files are persistet to speed up adding new packages when calling `upload_package_index()`.
Otherwise, the whole DB needs to be created from scratch which takes quite long and requires many API calls against backblaze.
## Build Environment
## Build metadata database
Binaries are built on a mixed-architecture Kubernetes cluster using CI.
Dedicated arm64 and amd64 nodes are utilized to efficiently build the binaries.
After all binaries for a specific architecture/OS combination are built, CRON jobs handle the processing of daily change operations.
This elastic server architecture offers a robust and performant backend while minimizing costs.
The build metadata is stored in a Postgres database.
The DB has a public endpoint at `r-binaries.devxy.io` and port `15432`.
The following tables are used:
## Technical Details
- `single_builds` contains the build metadata for each package:
### Creating/Updating the PACKAGES Index Files
Currently, the {cranlike} and {desc} packages only work with files on a local file system.
This is infeasible if the goal is to store binaries in S3.
Storing binaries permanently on a disk-based file system would incur significantly higher costs, especially when operating in the cloud.
Hence, modified versions of {cranlike} and {desc} were created that are able to handle files in S3 (through {s3fs}).
### Resources
Reasonably sized instances with performant CPUs are important to build binaries in a reasonable time.
While building binaries, it was found that a single process might need up to 14 GB of memory, as certain packages on CRAN require that much to build.
While this applies to only a few packages and most do not exceed 2 GB of memory, the exact RAM requirement for each individual package is unknown.
To ensure that any package can be processed without the risk of running out of memory (OOM), a safety margin of using 16 GB of memory is the suggested minimum requirement.
This means that a VM with 16 GB of memory can build binaries sequentially.
With 32 GB of memory, two cores can be used to process multiple packages in parallel.
Important: the parallelism applies at the tag level, not at the package level, and this behavior cannot currently be changed.
### Dependency Cache
A build cache for both R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a persistent volume for each architecture/OS combination.
Additionally, the PACKAGES index files are persisted to speed up adding new packages when calling `upload_package_index()`.
Otherwise, the entire (SQLite) database would need to be created from scratch, which takes considerable time and requires numerous API calls to Backblaze.
Processing all CRAN packages (approximately 21k) takes around 40 minutes, while processing updates with an existing database file takes around 5 minutes.
### Inferring System Dependencies
R package dependencies and their system dependencies are installed through {pak}.
{pak} allows for parallel downloads and installation, significantly speeding up package installation compared to `install.packages()`.
Additionally, it automatically infers package dependencies using JSON rules from [rstudio/r-system-requirements](https://github.com/rstudio/r-system-requirements).
Not all R packages specify required system dependencies in their DESCRIPTION file, and not all listed dependencies have existing rules in `rstudio/r-system-requirements`.
For Alpine, no rules existed until recently, establishing a foundation for semi-automated package installation on Alpine Linux.
## Metadata Database
The build metadata is stored in a PostgreSQL database.
The database has a public endpoint at `r-binaries.devxy.io` and port `15432`.
The database contains one table named `single_builds`, which holds the build metadata for each package:
| package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size |
| ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- |
@ -66,26 +107,29 @@ Column types:
- `package_name`: character varying(255)
- `tag`: character varying(255)
- `platform` character varying(255)
- `platform`: character varying(255)
- `error_occurred`: boolean
- `build_timestamp`: timestamp without time zone
- `build_duration`: numeric(1000,2)
- `error`: text
- `size`: numeric(1000,2)
Alternatively, use `\d+ single_builds`
Alternatively, use `\d+ single_builds`.
## Support for archived versions
A shiny dashboard providing a search functionality of the database and grouped statistics is available in `shiny/`.
The self-hosted database is running on a Kubernetes Cluster in HA mode.
## Support for Archived Versions
### `remotes::install_version()`
`remotes::install_version()` is searching for a `Meta/archive.rds` file in the `/src/contrib` directory.
This file must be a list of dataframes containing the information about the archived versions of the package.
`remotes::install_version()` searches for a `Meta/archive.rds` file in the `/src/contrib` directory.
This file must be a list of data frames containing information about the archived versions of the package.
Example:
```r
con <- gzcon(url(sprintf("%s/src/contrib/Meta/archive.rds",
c("CRAN" = "https://cloud.r-project.org")), "rb"))
foo = readRDS(con)
@ -94,36 +138,55 @@ foo[[1]]
### `pak::pak(package@version)`
`pak` searches for `Archive/<package>` and is able to install all versions existing there.
Ensure to use a clean cache if other repos have been used before, i.e. call `pak::meta_clean(force = TRUE)` if in doubt or when testing.
`pak` searches for `Archive/<package>` and can install all versions listed in it.
Ensure to use a clean cache if other repositories have been used previously.
If in doubt or when testing, call `pak::meta_clean(force = TRUE)`.
## Platforms
## Lessons Learned
The following platforms are supported:
- The newest R version needs to be used to build binaries.
The reason is that some packages depend on the "recommended" packages and attempt to install them as dependencies.
This fails for older R versions, e.g., if R 4.0.5 tries to install `Matrix` from 4.4.x.
- RHEL9
- Graphical R packages are challenging.
Most can be processed by starting R with `xvfb-run R`, but some still encounter issues and get stuck during processing.
## Lessons learned
- A few dozen packages rely on exotic external dependencies that must be installed from source.
Including all of these would significantly increase the container image size for minimal gain.
A common external dependency on which many R packages depend is JAGS.
Because of this, it has been included in the Containerfiles to enable successful builds for several dozen R packages.
- The newest R version needs to be used to build binaries. The reason is that some packages depend on the "recommended" packages and try to install them as a dep. This fails for older R versions, e.g. if R 4.0.5 tries to install `Matrix` from 4.4.x
- Catching build failures at all possible build stages is difficult.
Timeouts may occur, dependencies can fail to install, and some tags in the GitHub mirror might not include valid DESCRIPTION files.
It is crucial to catch errors, continue the build, and make the build process as robust as possible.
## Platforms
## URL Composition and Platform Identifiers
- Identifier must match the ones used in <https://github.com/rstudio/r-system-requirements> to be picked up correctly by the automatic syslib dependency installer of `pak`, more specifically by the env var `PKG_SYSREQS_PLATFORM`
Platform identifiers have been aligned with those used in <https://github.com/rstudio/r-system-requirements> to ensure proper recognition by the automatic syslib dependency installer of `pak`, specifically via the environment variable `PKG_SYSREQS_PLATFORM`:
- redhat-9
- redhat-8
- ubuntu-2204
- ubuntu-2404
- alpine-320
The final repository URL is structured slightly differently and follows the format of the Posit Packagemanager:
`https://<domain>/<arch>/<OS>/latest`
Example: <https://cran.devxy.io/arm64/rhel9/latest>
Technically, no date-based snapshots are planned, so this component from the Posit PM structure is not included.
## Helpers
Some common helper functions to ease usage live in `exec.R`.
Helper scripts are located in `local/`.
These scripts can assist in various situations, such as manually processing packages or filtering specific information from the metadata database.
## Common Errors
240 pkgs in 5h (200-449)
costs for 5h cax41: 0.235 EUR (0.047)
200 pkgs in 2h 40 min (449 - 649)
costs for 5h cax41: 0.129 EUR (0.047)
## Common errors
Below is a collection of raw errors observed during the build process:
<details>
```
* installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n"
@ -135,38 +198,44 @@ Tag does not have a NAMESPACE file and hence cannot be built.
"* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n"
```
Dependency not available: Either because the dependency was not declared or errored itself during installation.
Dependency not available: Either because the dependency was not declared or errored itself during installation.
```
In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n 162 | double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n | \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n 277 | ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n | \033[01;31m\033[K^~\033[m\033[K\n | \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n
```
Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17.
```
Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17.
---
When inferring dependencies:
When inferring dependencies:
```sh
internal error 1 in memDecompress
```
Solution:
Solution:
```sh
rm -rf /mnt/cache/R-pkgs/pkgcache/ /mnt/cache/R-pkgs/pak /mnt/cache/pkgcache/ /root/.cache/R/
R -q -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'
```
</details>
## Packages skipped on purpose
## Packages Skipped on Purpose
- `biplotbootGUI` - requires Bwidget which is hardly installable on any distribution and causes a lot of issues due to the need of a virtual frame buffer
Some packages have been intentionally skipped after multiple build attempts.
The reasons for this vary, and ideally, solutions can be found over the long term.
Contributions to help resolve these issues are highly welcome!
## CDN Settings
## CDN
A CDN is used in front of the S3 bucket to efficiently distribute the binaries globally.
- Perma-cache enabled for three different regions (DE, US, Asia) + smart cache for other edge servers
- `CacheControl = "no-cache"` for all PACKAGES* files to ensure users always get the latest version (as these files change daily)
-
A "Perma-Cache" is enabled for three different regions around the world (DE, US, Asia).
Once a binary is requested for the first time from a specific location, the asset is copied to the perma-cache and served from there for subsequent requests.
The `CacheControl = "no-cache"` header is set for all PACKAGES* files to ensure users always receive the latest version, as these files change daily.
A monthly traffic limit of 50 TB is set on cran.devxy.io to prevent abuse and manage costs.