update README
This commit is contained in:
parent
e61126ee4d
commit
f75d330bb0
GPG key ID:
3C6318841EF78925
1 changed files with 99 additions and 41 deletions
136
README.md
136
README.md
|
|
@ -2,42 +2,78 @@
|
|||
|
||||
## Workflow
|
||||
|
||||
1. "Prepare"
|
||||
1. Install R interpreter
|
||||
2. Add required repos to resolve syslibs
|
||||
3. Install pak
|
||||
4. Install awscli for remote s3 sync
|
||||
2. "Build"
|
||||
1. For every package, do
|
||||
1. Clone from GitHub CRAN mirror
|
||||
2. Install all system dependencies via `pak::local_install_dev_deps()`
|
||||
3. Checkout each available tag
|
||||
<!-- 4. Build source package -->
|
||||
4. Build binary package
|
||||
5. Create Archive/ directory
|
||||
6. Move all versions besides the most recent one to Archive/ (for source and binary)
|
||||
7. Create `PACKAGES` metadata file using `cranlike` package
|
||||
8. Store build metadata in sqlite DB
|
||||
3. "Upload"
|
||||
1. Sync all binaries and sources to S3 bucket
|
||||
4. `-4-create-archive-rds.sh`
|
||||
1. Create `Meta/archive.rds` file for each package so that `remotes::install_version()` can install archived versions
|
||||
2. Upload `Meta/archive.rds` to S3 bucket
|
||||
Most of the work is done through the R package `rBinaries`.
|
||||
|
||||
## Build metadata
|
||||
- `build_binary_package()`
|
||||
- `install_system_dependencies()`
|
||||
- `build_single_tag()`
|
||||
- `upload_single_binary_to_s3()`
|
||||
|
||||
The build metadata is stored in a sqlite database.
|
||||
### Preparation of environment
|
||||
|
||||
- Installation of R interpreter: done by downloading a prebuilt R binary for the specific OS.
|
||||
- Installation of R and system dependencies:
|
||||
- -> `install_system_dependencies()`
|
||||
- Invoked through `build_binary_package()`
|
||||
- Uses {pak} to install both R package dependencies and syslib dependencies
|
||||
|
||||
### Building binaries
|
||||
|
||||
Tags for each package are built in parallel via {future} by executing `build_binary_package()` via `future_mapply()`.
|
||||
|
||||
The build process in detail:
|
||||
|
||||
For every package+tag combination, do
|
||||
|
||||
1. Checkout tag from GitHub CRAN mirror
|
||||
2. Build binary package
|
||||
3. Upload package binary to S3
|
||||
4. Delete package binary (to free up space on PVC)
|
||||
|
||||
### Building the PACKAGES index file
|
||||
|
||||
The packages {cranlike} and {desc} only work with sources on a local file system.
|
||||
This is infeasible for our approach with the data stored only in S3.
|
||||
Otherwise, all binaries need to be present permanently on a static file system.
|
||||
This would incur a lot of costs.
|
||||
|
||||
Hence, modified versions of {cranlike} and {desc} were created which are able to deal with files in S3 through {s3fs}.
|
||||
|
||||
## Resources
|
||||
|
||||
Binaries are build in parallel on a 16 Core 32 GB instance with 6 processes. This gives each process ~ 5 GB of memory for building the individual package+tag combination.
|
||||
This looks excessive on the first look but some packages take up to 5 GB of memory when compiling from source.
|
||||
Hence, we need to allocation these resources to ensure a smooth build process which does not result in an OOMKilled of the pod.
|
||||
|
||||
## Cache
|
||||
|
||||
A build cache for R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a PVC with a size of 25 GB.
|
||||
Also, the PACKAGES index files are persistet to speed up adding new packages when calling `upload_package_index()`.
|
||||
Otherwise, the whole DB needs to be created from scratch which takes quite long and requires many API calls against backblaze.
|
||||
|
||||
## Build metadata database
|
||||
|
||||
The build metadata is stored in a Postgres database.
|
||||
The DB has a public endpoint at `postgres-arm-binaries-r.devxy.io` and port `15432`.
|
||||
The following tables are used:
|
||||
|
||||
- `metadata` contains the build metadata for each package:
|
||||
- `single_builds` contains the build metadata for each package:
|
||||
|
||||
| package_name | tag | platform | error_occurred | build_timestamp |
|
||||
| ------------ | --- | -------- | -------------- | --------------- |
|
||||
| package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size |
|
||||
| ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- |
|
||||
|
||||
- `metadata_summary` contains aggregated build metadata for each package and platform:
|
||||
Column types:
|
||||
|
||||
| package_name | platform | successful_builds | unsuccessful_builds | total_builds | percentage_successful_builds | average_build_time_per_tag |
|
||||
| ------------ | -------- | ----------------- | ------------------- | ------------ | ---------------------------- | -------------------------- |
|
||||
- `package_name`: character varying(255)
|
||||
- `tag`: character varying(255)
|
||||
- `platform` character varying(255)
|
||||
- `error_occurred`: boolean
|
||||
- `build_timestamp`: timestamp without time zone
|
||||
- `build_duration`: numeric(1000,2)
|
||||
- `error`: text
|
||||
- `size`: numeric(1000,2)
|
||||
|
||||
Alternatively, use `\d+ single_builds`
|
||||
|
||||
## Support for archived versions
|
||||
|
||||
|
|
@ -69,20 +105,42 @@ The following platforms are supported:
|
|||
|
||||
## Lessons learned
|
||||
|
||||
- Newer R versions have issues compiling C code from older package Versions. Using an older R version to build the binaries is recommended.
|
||||
- R 4.0.5 is used to build all binaries for R 4.x
|
||||
- Binaries which cannot be built using R 4.0.5 often also require older system library or compiler versions and are out of scope for this project.
|
||||
- The newest R version needs to be used to build binaries. The reason is that some packages depend on the "recommended" packages and try to install them as a dep. This fails for older R versions, e.g. if R 4.0.5 tries to install `Matrix` from 4.4.x
|
||||
|
||||
|
||||
## Platforms
|
||||
## Platforms
|
||||
|
||||
- Identifier must match the ones used in <https://github.com/rstudio/r-system-requirements> to be picked up correctly by the automatic syslib dependency installer of `pak`, more specifically by the env var `PKG_SYSREQS_PLATFORM`
|
||||
|
||||
## Helpers
|
||||
|
||||
Check the row number of a package by name
|
||||
Some common helper functions to ease usage live in `exec.R`.
|
||||
|
||||
|
||||
|
||||
240 pkgs in 5h (200-449)
|
||||
costs for 5h cax41: 0.235 EUR (0.047)
|
||||
|
||||
200 pkgs in 2h 40 min (449 - 649)
|
||||
costs for 5h cax41: 0.129 EUR (0.047)
|
||||
|
||||
## Common errors
|
||||
|
||||
```
|
||||
* installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n"
|
||||
```
|
||||
|
||||
Tag does not have a NAMESPACE file and hence cannot be built.
|
||||
|
||||
```
|
||||
"* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n"
|
||||
```
|
||||
|
||||
Dependency not available: Either because the dependency was not declared or errored itself during installation.
|
||||
|
||||
```
|
||||
In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n 162 | double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n | \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n 277 | ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n | \033[01;31m\033[K^~\033[m\033[K\n | \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n
|
||||
```
|
||||
|
||||
Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17.
|
||||
|
||||
|
||||
```r
|
||||
pkgs = tools::CRAN_package_db()
|
||||
which(pkgs$Package == "ABCanalysis")
|
||||
```
|
||||
Loading…
Reference in a new issue