diff --git a/README.md b/README.md index 83004e3..438c8f2 100644 --- a/README.md +++ b/README.md @@ -2,42 +2,78 @@ ## Workflow -1. "Prepare" - 1. Install R interpreter - 2. Add required repos to resolve syslibs - 3. Install pak - 4. Install awscli for remote s3 sync -2. "Build" - 1. For every package, do - 1. Clone from GitHub CRAN mirror - 2. Install all system dependencies via `pak::local_install_dev_deps()` - 3. Checkout each available tag - - 4. Build binary package - 5. Create Archive/ directory - 6. Move all versions besides the most recent one to Archive/ (for source and binary) - 7. Create `PACKAGES` metadata file using `cranlike` package - 8. Store build metadata in sqlite DB -3. "Upload" - 1. Sync all binaries and sources to S3 bucket -4. `-4-create-archive-rds.sh` - 1. Create `Meta/archive.rds` file for each package so that `remotes::install_version()` can install archived versions - 2. Upload `Meta/archive.rds` to S3 bucket +Most of the work is done through the R package `rBinaries`. -## Build metadata +- `build_binary_package()` + - `install_system_dependencies()` + - `build_single_tag()` + - `upload_single_binary_to_s3()` -The build metadata is stored in a sqlite database. +### Preparation of environment + +- Installation of R interpreter: done by downloading a prebuilt R binary for the specific OS. +- Installation of R and system dependencies: + - -> `install_system_dependencies()` + - Invoked through `build_binary_package()` + - Uses {pak} to install both R package dependencies and syslib dependencies + +### Building binaries + +Tags for each package are built in parallel via {future} by executing `build_binary_package()` via `future_mapply()`. + +The build process in detail: + +For every package+tag combination, do + +1. Checkout tag from GitHub CRAN mirror +2. Build binary package +3. Upload package binary to S3 +4. Delete package binary (to free up space on PVC) + +### Building the PACKAGES index file + +The packages {cranlike} and {desc} only work with sources on a local file system. +This is infeasible for our approach with the data stored only in S3. +Otherwise, all binaries need to be present permanently on a static file system. +This would incur a lot of costs. + +Hence, modified versions of {cranlike} and {desc} were created which are able to deal with files in S3 through {s3fs}. + +## Resources + +Binaries are build in parallel on a 16 Core 32 GB instance with 6 processes. This gives each process ~ 5 GB of memory for building the individual package+tag combination. +This looks excessive on the first look but some packages take up to 5 GB of memory when compiling from source. +Hence, we need to allocation these resources to ensure a smooth build process which does not result in an OOMKilled of the pod. + +## Cache + +A build cache for R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a PVC with a size of 25 GB. +Also, the PACKAGES index files are persistet to speed up adding new packages when calling `upload_package_index()`. +Otherwise, the whole DB needs to be created from scratch which takes quite long and requires many API calls against backblaze. + +## Build metadata database + +The build metadata is stored in a Postgres database. +The DB has a public endpoint at `postgres-arm-binaries-r.devxy.io` and port `15432`. The following tables are used: -- `metadata` contains the build metadata for each package: +- `single_builds` contains the build metadata for each package: -| package_name | tag | platform | error_occurred | build_timestamp | -| ------------ | --- | -------- | -------------- | --------------- | +| package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size | +| ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- | -- `metadata_summary` contains aggregated build metadata for each package and platform: +Column types: -| package_name | platform | successful_builds | unsuccessful_builds | total_builds | percentage_successful_builds | average_build_time_per_tag | -| ------------ | -------- | ----------------- | ------------------- | ------------ | ---------------------------- | -------------------------- | +- `package_name`: character varying(255) +- `tag`: character varying(255) +- `platform` character varying(255) +- `error_occurred`: boolean +- `build_timestamp`: timestamp without time zone +- `build_duration`: numeric(1000,2) +- `error`: text +- `size`: numeric(1000,2) + +Alternatively, use `\d+ single_builds` ## Support for archived versions @@ -69,20 +105,42 @@ The following platforms are supported: ## Lessons learned -- Newer R versions have issues compiling C code from older package Versions. Using an older R version to build the binaries is recommended. -- R 4.0.5 is used to build all binaries for R 4.x -- Binaries which cannot be built using R 4.0.5 often also require older system library or compiler versions and are out of scope for this project. +- The newest R version needs to be used to build binaries. The reason is that some packages depend on the "recommended" packages and try to install them as a dep. This fails for older R versions, e.g. if R 4.0.5 tries to install `Matrix` from 4.4.x - -## Platforms +## Platforms - Identifier must match the ones used in to be picked up correctly by the automatic syslib dependency installer of `pak`, more specifically by the env var `PKG_SYSREQS_PLATFORM` ## Helpers -Check the row number of a package by name +Some common helper functions to ease usage live in `exec.R`. + + + +240 pkgs in 5h (200-449) +costs for 5h cax41: 0.235 EUR (0.047) + +200 pkgs in 2h 40 min (449 - 649) +costs for 5h cax41: 0.129 EUR (0.047) + +## Common errors + +``` +* installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n" +``` + +Tag does not have a NAMESPACE file and hence cannot be built. + +``` +"* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n" +``` + +Dependency not available: Either because the dependency was not declared or errored itself during installation. + +``` + In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n 162 | double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n | \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n 277 | ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n | \033[01;31m\033[K^~\033[m\033[K\n | \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n + ``` + + Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17. + -```r -pkgs = tools::CRAN_package_db() -which(pkgs$Package == "ABCanalysis") -``` \ No newline at end of file