From bf46450b1a95911cf356d3dac993eee885d460e3 Mon Sep 17 00:00:00 2001 From: pat-s Date: Thu, 14 Nov 2024 23:38:58 +0100 Subject: [PATCH] docs: large README update --- README.md | 221 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 145 insertions(+), 76 deletions(-) diff --git a/README.md b/README.md index 15a9abd..d735581 100644 --- a/README.md +++ b/README.md @@ -1,63 +1,104 @@ -# README +# bincraftR -## Workflow +[TOC] -Most of the work is done through the R package `bincraftR`. +This project offers a framework for creating R package binaries on Linux across various architectures and distributions. -- `build_binary_package()` - - `install_system_dependencies()` - - `build_single_tag()` - - `upload_single_binary_to_s3()` +It achieves this through the integration of several components: -### Preparation of environment +- **R package `bincraftR`** +- **Containerfiles** that define the build toolchain for each distribution +- **S3 storage** for storing the compiled binaries +- **PostgreSQL database** for recording build logs -- Installation of R interpreter: done by downloading a prebuilt R binary for the specific OS. -- Installation of R and system dependencies: - - -> `install_system_dependencies()` - - Invoked through `build_binary_package()` - - Uses {pak} to install both R package dependencies and syslib dependencies +## R Package -### Building binaries +The R package `bincraftR` is the engine behind everything. +It provides functions that can: -Tags for each package are built in parallel via {future} by executing `build_binary_package()` via `future_mapply()`. +- build binaries +- archive packages following the CRAN-like directory structure +- upload package binaries to S3 +- update the package index files (`PACKAGES*`) +- store build metadata, including error logs, in a PostgreSQL database -The build process in detail: +See the function reference on the pkgdown site for a full overview. -For every package+tag combination, do +The focus of the R package is on usability rather than minimizing dependencies. +The individual containerfiles include the package along with its dependencies. +Bundling more R packages upfront helps reduce the number of additional packages needed when installing the dependencies for building packages. -1. Checkout tag from GitHub CRAN mirror -2. Build binary package -3. Upload package binary to S3 -4. Delete package binary (to free up space on PVC) +## Containerfiles -### Building the PACKAGES index file +The toolchain in the containerfile of each distribution is a very important element for the build success of the packages. +The C compiler settings should be close to the recommended settings from CRAN and allow compatibility for most CRAN packages. -The packages {cranlike} and {desc} only work with sources on a local file system. -This is infeasible for our approach with the data stored only in S3. -Otherwise, all binaries need to be present permanently on a static file system. -This would incur a lot of costs. +Here, especially Alpine is tricky as CRAN does not test R packages for Alpine. +Since Alpine uses a different C library (MUSL instead of GLIBC), many R packages that include C/C++ code encounter errors. -Hence, modified versions of {cranlike} and {desc} were created which are able to deal with files in S3 through {s3fs}. +## Build Process -## Resources +Tags for each package can be built in parallel via {future} through `build_binary_package()`. +`build_binary_package()` builds all available tags of an R package by default. +When setting `tag = ` or the special value `tag = "latest"`, only these tags will be built. -Binaries are build in parallel on a 16 Core 32 GB instance with 6 processes. This gives each process ~ 5 GB of memory for building the individual package+tag combination. -This looks excessive on the first look but some packages take up to 5 GB of memory when compiling from source. -Hence, we need to allocation these resources to ensure a smooth build process which does not result in an OOMKilled of the pod. +For every package+tag combination: -## Cache +1. Checkout tag(s) from GitHub CRAN mirror (e.g. ) +1. Build binaries +1. Upload binaries to S3 +1. Archive old package versions and keep the latest one in the root +1. Delete local binaries after successful upload -A build cache for R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a PVC with a size of 25 GB. -Also, the PACKAGES index files are persistet to speed up adding new packages when calling `upload_package_index()`. -Otherwise, the whole DB needs to be created from scratch which takes quite long and requires many API calls against backblaze. +## Build Environment -## Build metadata database +Binaries are built on a mixed-architecture Kubernetes cluster using CI. +Dedicated arm64 and amd64 nodes are utilized to efficiently build the binaries. +After all binaries for a specific architecture/OS combination are built, CRON jobs handle the processing of daily change operations. +This elastic server architecture offers a robust and performant backend while minimizing costs. -The build metadata is stored in a Postgres database. -The DB has a public endpoint at `r-binaries.devxy.io` and port `15432`. -The following tables are used: +## Technical Details -- `single_builds` contains the build metadata for each package: +### Creating/Updating the PACKAGES Index Files + +Currently, the {cranlike} and {desc} packages only work with files on a local file system. +This is infeasible if the goal is to store binaries in S3. +Storing binaries permanently on a disk-based file system would incur significantly higher costs, especially when operating in the cloud. + +Hence, modified versions of {cranlike} and {desc} were created that are able to handle files in S3 (through {s3fs}). + +### Resources + +Reasonably sized instances with performant CPUs are important to build binaries in a reasonable time. +While building binaries, it was found that a single process might need up to 14 GB of memory, as certain packages on CRAN require that much to build. +While this applies to only a few packages and most do not exceed 2 GB of memory, the exact RAM requirement for each individual package is unknown. +To ensure that any package can be processed without the risk of running out of memory (OOM), a safety margin of using 16 GB of memory is the suggested minimum requirement. +This means that a VM with 16 GB of memory can build binaries sequentially. +With 32 GB of memory, two cores can be used to process multiple packages in parallel. + +Important: the parallelism applies at the tag level, not at the package level, and this behavior cannot currently be changed. + +### Dependency Cache + +A build cache for both R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a persistent volume for each architecture/OS combination. +Additionally, the PACKAGES index files are persisted to speed up adding new packages when calling `upload_package_index()`. +Otherwise, the entire (SQLite) database would need to be created from scratch, which takes considerable time and requires numerous API calls to Backblaze. + +Processing all CRAN packages (approximately 21k) takes around 40 minutes, while processing updates with an existing database file takes around 5 minutes. + +### Inferring System Dependencies + +R package dependencies and their system dependencies are installed through {pak}. +{pak} allows for parallel downloads and installation, significantly speeding up package installation compared to `install.packages()`. +Additionally, it automatically infers package dependencies using JSON rules from [rstudio/r-system-requirements](https://github.com/rstudio/r-system-requirements). +Not all R packages specify required system dependencies in their DESCRIPTION file, and not all listed dependencies have existing rules in `rstudio/r-system-requirements`. +For Alpine, no rules existed until recently, establishing a foundation for semi-automated package installation on Alpine Linux. + +## Metadata Database + +The build metadata is stored in a PostgreSQL database. +The database has a public endpoint at `r-binaries.devxy.io` and port `15432`. +The database contains one table named `single_builds`, which holds the build metadata for each package: | package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size | | ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- | @@ -66,26 +107,29 @@ Column types: - `package_name`: character varying(255) - `tag`: character varying(255) -- `platform` character varying(255) +- `platform`: character varying(255) - `error_occurred`: boolean - `build_timestamp`: timestamp without time zone - `build_duration`: numeric(1000,2) - `error`: text - `size`: numeric(1000,2) -Alternatively, use `\d+ single_builds` +Alternatively, use `\d+ single_builds`. -## Support for archived versions +A shiny dashboard providing a search functionality of the database and grouped statistics is available in `shiny/`. + +The self-hosted database is running on a Kubernetes Cluster in HA mode. + +## Support for Archived Versions ### `remotes::install_version()` -`remotes::install_version()` is searching for a `Meta/archive.rds` file in the `/src/contrib` directory. -This file must be a list of dataframes containing the information about the archived versions of the package. +`remotes::install_version()` searches for a `Meta/archive.rds` file in the `/src/contrib` directory. +This file must be a list of data frames containing information about the archived versions of the package. Example: ```r - con <- gzcon(url(sprintf("%s/src/contrib/Meta/archive.rds", c("CRAN" = "https://cloud.r-project.org")), "rb")) foo = readRDS(con) @@ -94,36 +138,55 @@ foo[[1]] ### `pak::pak(package@version)` -`pak` searches for `Archive/` and is able to install all versions existing there. -Ensure to use a clean cache if other repos have been used before, i.e. call `pak::meta_clean(force = TRUE)` if in doubt or when testing. +`pak` searches for `Archive/` and can install all versions listed in it. +Ensure to use a clean cache if other repositories have been used previously. +If in doubt or when testing, call `pak::meta_clean(force = TRUE)`. -## Platforms +## Lessons Learned -The following platforms are supported: +- The newest R version needs to be used to build binaries. + The reason is that some packages depend on the "recommended" packages and attempt to install them as dependencies. + This fails for older R versions, e.g., if R 4.0.5 tries to install `Matrix` from 4.4.x. -- RHEL9 +- Graphical R packages are challenging. + Most can be processed by starting R with `xvfb-run R`, but some still encounter issues and get stuck during processing. -## Lessons learned +- A few dozen packages rely on exotic external dependencies that must be installed from source. + Including all of these would significantly increase the container image size for minimal gain. + A common external dependency on which many R packages depend is JAGS. + Because of this, it has been included in the Containerfiles to enable successful builds for several dozen R packages. -- The newest R version needs to be used to build binaries. The reason is that some packages depend on the "recommended" packages and try to install them as a dep. This fails for older R versions, e.g. if R 4.0.5 tries to install `Matrix` from 4.4.x +- Catching build failures at all possible build stages is difficult. + Timeouts may occur, dependencies can fail to install, and some tags in the GitHub mirror might not include valid DESCRIPTION files. + It is crucial to catch errors, continue the build, and make the build process as robust as possible. -## Platforms +## URL Composition and Platform Identifiers -- Identifier must match the ones used in to be picked up correctly by the automatic syslib dependency installer of `pak`, more specifically by the env var `PKG_SYSREQS_PLATFORM` +Platform identifiers have been aligned with those used in to ensure proper recognition by the automatic syslib dependency installer of `pak`, specifically via the environment variable `PKG_SYSREQS_PLATFORM`: + +- redhat-9 +- redhat-8 +- ubuntu-2204 +- ubuntu-2404 +- alpine-320 + +The final repository URL is structured slightly differently and follows the format of the Posit Packagemanager: + +`https://///latest` + +Example: + +Technically, no date-based snapshots are planned, so this component from the Posit PM structure is not included. ## Helpers -Some common helper functions to ease usage live in `exec.R`. +Helper scripts are located in `local/`. +These scripts can assist in various situations, such as manually processing packages or filtering specific information from the metadata database. +## Common Errors - -240 pkgs in 5h (200-449) -costs for 5h cax41: 0.235 EUR (0.047) - -200 pkgs in 2h 40 min (449 - 649) -costs for 5h cax41: 0.129 EUR (0.047) - -## Common errors +Below is a collection of raw errors observed during the build process: +
``` * installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n" @@ -135,38 +198,44 @@ Tag does not have a NAMESPACE file and hence cannot be built. "* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n" ``` -Dependency not available: Either because the dependency was not declared or errored itself during installation. +Dependency not available: Either because the dependency was not declared or errored itself during installation. ``` In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n 162 | double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n | \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n 277 | ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n | \033[01;31m\033[K^~\033[m\033[K\n | \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n - ``` - - Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17. +``` +Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17. --- -When inferring dependencies: +When inferring dependencies: ```sh internal error 1 in memDecompress ``` -Solution: +Solution: ```sh rm -rf /mnt/cache/R-pkgs/pkgcache/ /mnt/cache/R-pkgs/pak /mnt/cache/pkgcache/ /root/.cache/R/ R -q -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))' ``` +
-## Packages skipped on purpose +## Packages Skipped on Purpose -- `biplotbootGUI` - requires Bwidget which is hardly installable on any distribution and causes a lot of issues due to the need of a virtual frame buffer +Some packages have been intentionally skipped after multiple build attempts. +The reasons for this vary, and ideally, solutions can be found over the long term. +Contributions to help resolve these issues are highly welcome! +## CDN Settings -## CDN +A CDN is used in front of the S3 bucket to efficiently distribute the binaries globally. -- Perma-cache enabled for three different regions (DE, US, Asia) + smart cache for other edge servers -- `CacheControl = "no-cache"` for all PACKAGES* files to ensure users always get the latest version (as these files change daily) -- +A "Perma-Cache" is enabled for three different regions around the world (DE, US, Asia). +Once a binary is requested for the first time from a specific location, the asset is copied to the perma-cache and served from there for subsequent requests. + +The `CacheControl = "no-cache"` header is set for all PACKAGES* files to ensure users always receive the latest version, as these files change daily. + +A monthly traffic limit of 50 TB is set on cran.devxy.io to prevent abuse and manage costs.