172 lines
7 KiB
Markdown
172 lines
7 KiB
Markdown
# README
|
|
|
|
## Workflow
|
|
|
|
Most of the work is done through the R package `bincraftR`.
|
|
|
|
- `build_binary_package()`
|
|
- `install_system_dependencies()`
|
|
- `build_single_tag()`
|
|
- `upload_single_binary_to_s3()`
|
|
|
|
### Preparation of environment
|
|
|
|
- Installation of R interpreter: done by downloading a prebuilt R binary for the specific OS.
|
|
- Installation of R and system dependencies:
|
|
- -> `install_system_dependencies()`
|
|
- Invoked through `build_binary_package()`
|
|
- Uses {pak} to install both R package dependencies and syslib dependencies
|
|
|
|
### Building binaries
|
|
|
|
Tags for each package are built in parallel via {future} by executing `build_binary_package()` via `future_mapply()`.
|
|
|
|
The build process in detail:
|
|
|
|
For every package+tag combination, do
|
|
|
|
1. Checkout tag from GitHub CRAN mirror
|
|
2. Build binary package
|
|
3. Upload package binary to S3
|
|
4. Delete package binary (to free up space on PVC)
|
|
|
|
### Building the PACKAGES index file
|
|
|
|
The packages {cranlike} and {desc} only work with sources on a local file system.
|
|
This is infeasible for our approach with the data stored only in S3.
|
|
Otherwise, all binaries need to be present permanently on a static file system.
|
|
This would incur a lot of costs.
|
|
|
|
Hence, modified versions of {cranlike} and {desc} were created which are able to deal with files in S3 through {s3fs}.
|
|
|
|
## Resources
|
|
|
|
Binaries are build in parallel on a 16 Core 32 GB instance with 6 processes. This gives each process ~ 5 GB of memory for building the individual package+tag combination.
|
|
This looks excessive on the first look but some packages take up to 5 GB of memory when compiling from source.
|
|
Hence, we need to allocation these resources to ensure a smooth build process which does not result in an OOMKilled of the pod.
|
|
|
|
## Cache
|
|
|
|
A build cache for R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a PVC with a size of 25 GB.
|
|
Also, the PACKAGES index files are persistet to speed up adding new packages when calling `upload_package_index()`.
|
|
Otherwise, the whole DB needs to be created from scratch which takes quite long and requires many API calls against backblaze.
|
|
|
|
## Build metadata database
|
|
|
|
The build metadata is stored in a Postgres database.
|
|
The DB has a public endpoint at `r-binaries.devxy.io` and port `15432`.
|
|
The following tables are used:
|
|
|
|
- `single_builds` contains the build metadata for each package:
|
|
|
|
| package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size |
|
|
| ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- |
|
|
|
|
Column types:
|
|
|
|
- `package_name`: character varying(255)
|
|
- `tag`: character varying(255)
|
|
- `platform` character varying(255)
|
|
- `error_occurred`: boolean
|
|
- `build_timestamp`: timestamp without time zone
|
|
- `build_duration`: numeric(1000,2)
|
|
- `error`: text
|
|
- `size`: numeric(1000,2)
|
|
|
|
Alternatively, use `\d+ single_builds`
|
|
|
|
## Support for archived versions
|
|
|
|
### `remotes::install_version()`
|
|
|
|
`remotes::install_version()` is searching for a `Meta/archive.rds` file in the `/src/contrib` directory.
|
|
This file must be a list of dataframes containing the information about the archived versions of the package.
|
|
|
|
Example:
|
|
|
|
```r
|
|
|
|
con <- gzcon(url(sprintf("%s/src/contrib/Meta/archive.rds",
|
|
c("CRAN" = "https://cloud.r-project.org")), "rb"))
|
|
foo = readRDS(con)
|
|
foo[[1]]
|
|
```
|
|
|
|
### `pak::pak(package@version)`
|
|
|
|
`pak` searches for `Archive/<package>` and is able to install all versions existing there.
|
|
Ensure to use a clean cache if other repos have been used before, i.e. call `pak::meta_clean(force = TRUE)` if in doubt or when testing.
|
|
|
|
## Platforms
|
|
|
|
The following platforms are supported:
|
|
|
|
- RHEL9
|
|
|
|
## Lessons learned
|
|
|
|
- The newest R version needs to be used to build binaries. The reason is that some packages depend on the "recommended" packages and try to install them as a dep. This fails for older R versions, e.g. if R 4.0.5 tries to install `Matrix` from 4.4.x
|
|
|
|
## Platforms
|
|
|
|
- Identifier must match the ones used in <https://github.com/rstudio/r-system-requirements> to be picked up correctly by the automatic syslib dependency installer of `pak`, more specifically by the env var `PKG_SYSREQS_PLATFORM`
|
|
|
|
## Helpers
|
|
|
|
Some common helper functions to ease usage live in `exec.R`.
|
|
|
|
|
|
|
|
240 pkgs in 5h (200-449)
|
|
costs for 5h cax41: 0.235 EUR (0.047)
|
|
|
|
200 pkgs in 2h 40 min (449 - 649)
|
|
costs for 5h cax41: 0.129 EUR (0.047)
|
|
|
|
## Common errors
|
|
|
|
```
|
|
* installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n"
|
|
```
|
|
|
|
Tag does not have a NAMESPACE file and hence cannot be built.
|
|
|
|
```
|
|
"* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n"
|
|
```
|
|
|
|
Dependency not available: Either because the dependency was not declared or errored itself during installation.
|
|
|
|
```
|
|
In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n 162 | double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n | \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n 277 | ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n | \033[01;31m\033[K^~\033[m\033[K\n | \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n
|
|
```
|
|
|
|
Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17.
|
|
|
|
|
|
---
|
|
|
|
When inferring dependencies:
|
|
|
|
```sh
|
|
internal error 1 in memDecompress
|
|
```
|
|
|
|
Solution:
|
|
|
|
```sh
|
|
rm -rf /mnt/cache/R-pkgs/pkgcache/ /mnt/cache/R-pkgs/pak /mnt/cache/pkgcache/ /root/.cache/R/
|
|
R -q -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'
|
|
```
|
|
|
|
|
|
## Packages skipped on purpose
|
|
|
|
- `biplotbootGUI` - requires Bwidget which is hardly installable on any distribution and causes a lot of issues due to the need of a virtual frame buffer
|
|
|
|
|
|
## CDN
|
|
|
|
- Perma-cache enabled for three different regions (DE, US, Asia) + smart cache for other edge servers
|
|
- `CacheControl = "no-cache"` for all PACKAGES* files to ensure users always get the latest version (as these files change daily)
|
|
-
|