# README ## Workflow Most of the work is done through the R package `bincraftR`. - `build_binary_package()` - `install_system_dependencies()` - `build_single_tag()` - `upload_single_binary_to_s3()` ### Preparation of environment - Installation of R interpreter: done by downloading a prebuilt R binary for the specific OS. - Installation of R and system dependencies: - -> `install_system_dependencies()` - Invoked through `build_binary_package()` - Uses {pak} to install both R package dependencies and syslib dependencies ### Building binaries Tags for each package are built in parallel via {future} by executing `build_binary_package()` via `future_mapply()`. The build process in detail: For every package+tag combination, do 1. Checkout tag from GitHub CRAN mirror 2. Build binary package 3. Upload package binary to S3 4. Delete package binary (to free up space on PVC) ### Building the PACKAGES index file The packages {cranlike} and {desc} only work with sources on a local file system. This is infeasible for our approach with the data stored only in S3. Otherwise, all binaries need to be present permanently on a static file system. This would incur a lot of costs. Hence, modified versions of {cranlike} and {desc} were created which are able to deal with files in S3 through {s3fs}. ## Resources Binaries are build in parallel on a 16 Core 32 GB instance with 6 processes. This gives each process ~ 5 GB of memory for building the individual package+tag combination. This looks excessive on the first look but some packages take up to 5 GB of memory when compiling from source. Hence, we need to allocation these resources to ensure a smooth build process which does not result in an OOMKilled of the pod. ## Cache A build cache for R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a PVC with a size of 25 GB. Also, the PACKAGES index files are persistet to speed up adding new packages when calling `upload_package_index()`. Otherwise, the whole DB needs to be created from scratch which takes quite long and requires many API calls against backblaze. ## Build metadata database The build metadata is stored in a Postgres database. The DB has a public endpoint at `postgres-arm-binaries-r.devxy.io` and port `15432`. The following tables are used: - `single_builds` contains the build metadata for each package: | package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size | | ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- | Column types: - `package_name`: character varying(255) - `tag`: character varying(255) - `platform` character varying(255) - `error_occurred`: boolean - `build_timestamp`: timestamp without time zone - `build_duration`: numeric(1000,2) - `error`: text - `size`: numeric(1000,2) Alternatively, use `\d+ single_builds` ## Support for archived versions ### `remotes::install_version()` `remotes::install_version()` is searching for a `Meta/archive.rds` file in the `/src/contrib` directory. This file must be a list of dataframes containing the information about the archived versions of the package. Example: ```r con <- gzcon(url(sprintf("%s/src/contrib/Meta/archive.rds", c("CRAN" = "https://cloud.r-project.org")), "rb")) foo = readRDS(con) foo[[1]] ``` ### `pak::pak(package@version)` `pak` searches for `Archive/` and is able to install all versions existing there. Ensure to use a clean cache if other repos have been used before, i.e. call `pak::meta_clean(force = TRUE)` if in doubt or when testing. ## Platforms The following platforms are supported: - RHEL9 ## Lessons learned - The newest R version needs to be used to build binaries. The reason is that some packages depend on the "recommended" packages and try to install them as a dep. This fails for older R versions, e.g. if R 4.0.5 tries to install `Matrix` from 4.4.x ## Platforms - Identifier must match the ones used in to be picked up correctly by the automatic syslib dependency installer of `pak`, more specifically by the env var `PKG_SYSREQS_PLATFORM` ## Helpers Some common helper functions to ease usage live in `exec.R`. 240 pkgs in 5h (200-449) costs for 5h cax41: 0.235 EUR (0.047) 200 pkgs in 2h 40 min (449 - 649) costs for 5h cax41: 0.129 EUR (0.047) ## Common errors ``` * installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n" ``` Tag does not have a NAMESPACE file and hence cannot be built. ``` "* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n" ``` Dependency not available: Either because the dependency was not declared or errored itself during installation. ``` In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n 162 | double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n | \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n 277 | ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n | \033[01;31m\033[K^~\033[m\033[K\n | \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n ``` Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17. ## Packages skipped on purpose - `biplotbootGUI` - requires Bwidget which is hardly installable on any distribution and causes a lot of issues due to the need of a virtual frame buffer ## CDN - Perma-cache enabled for three different regions (DE, US, Asia) + smart cache for other edge servers - `CacheControl = "no-cache"` for all PACKAGES* files to ensure users always get the latest version (as these files change daily) -