update README

README.md

@@ -2,42 +2,78 @@
 
 ## Workflow
 
-1. "Prepare"
-   1. Install R interpreter
-   2. Add required repos to resolve syslibs
-   3. Install pak
-   4. Install awscli for remote s3 sync
-2. "Build"
-   1. For every package, do
-      1. Clone from GitHub CRAN mirror
-      2. Install all system dependencies via `pak::local_install_dev_deps()`
-      3. Checkout each available tag
-      <!-- 4. Build source package -->
-      4. Build binary package
-      5. Create Archive/ directory
-      6. Move all versions besides the most recent one to Archive/ (for source and binary)
-      7. Create `PACKAGES` metadata file using `cranlike` package
-      8. Store build metadata in sqlite DB
-3. "Upload"
-   1. Sync all binaries and sources to S3 bucket
-4. `-4-create-archive-rds.sh`
-   1. Create `Meta/archive.rds` file for each package so that `remotes::install_version()` can install archived versions
-   2. Upload `Meta/archive.rds` to S3 bucket
+Most of the work is done through the R package `rBinaries`.
 
-## Build metadata
+- `build_binary_package()`
+  - `install_system_dependencies()`
+  - `build_single_tag()`
+  - `upload_single_binary_to_s3()`
 
-The build metadata is stored in a sqlite database.
+### Preparation of environment
+
+- Installation of R interpreter: done by downloading a prebuilt R binary for the specific OS.
+- Installation of R and system dependencies:
+  - -> `install_system_dependencies()`
+  - Invoked through `build_binary_package()`
+  - Uses {pak} to install both R package dependencies and syslib dependencies
+
+### Building binaries
+
+Tags for each package are built in parallel via {future} by executing `build_binary_package()` via `future_mapply()`.
+
+The build process in detail:
+
+For every package+tag combination, do
+
+1.  Checkout tag from GitHub CRAN mirror
+2.  Build binary package
+3.  Upload package binary to S3
+4.  Delete package binary (to free up space on PVC)
+
+### Building the PACKAGES index file
+
+The packages {cranlike} and {desc} only work with sources on a local file system.
+This is infeasible for our approach with the data stored only in S3.
+Otherwise, all binaries need to be present permanently on a static file system. 
+This would incur a lot of costs.
+
+Hence, modified versions of {cranlike} and {desc} were created which are able to deal with files in S3 through {s3fs}.
+
+## Resources
+
+Binaries are build in parallel on a 16 Core 32 GB instance with 6 processes. This gives each process ~ 5 GB of memory for building the individual package+tag combination.
+This looks excessive on the first look but some packages take up to 5 GB of memory when compiling from source.
+Hence, we need to allocation these resources to ensure a smooth build process which does not result in an OOMKilled of the pod.
+
+## Cache
+
+A build cache for R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a PVC with a size of 25 GB. 
+Also, the PACKAGES index files are persistet to speed up adding new packages when calling `upload_package_index()`.
+Otherwise, the whole DB needs to be created from scratch which takes quite long and requires many API calls against backblaze.
+
+## Build metadata database
+
+The build metadata is stored in a Postgres database.
+The DB has a public endpoint at `postgres-arm-binaries-r.devxy.io` and port `15432`.
 The following tables are used:
 
-- `metadata` contains the build metadata for each package:
+- `single_builds` contains the build metadata for each package:
 
-| package_name | tag | platform | error_occurred | build_timestamp |
-| ------------ | --- | -------- | -------------- | --------------- |
+| package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size |
+| ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- |
 
-- `metadata_summary` contains aggregated build metadata for each package and platform:
+Column types:
 
-| package_name | platform | successful_builds | unsuccessful_builds | total_builds | percentage_successful_builds | average_build_time_per_tag |
-| ------------ | -------- | ----------------- | ------------------- | ------------ | ---------------------------- | -------------------------- |
+- `package_name`: character varying(255)
+- `tag`: character varying(255)
+- `platform` character varying(255)
+- `error_occurred`: boolean
+- `build_timestamp`: timestamp without time zone
+- `build_duration`: numeric(1000,2)
+- `error`: text
+- `size`: numeric(1000,2)
+
+Alternatively, use `\d+ single_builds`
 
 ## Support for archived versions
 
@@ -69,20 +105,42 @@ The following platforms are supported:
 
 ## Lessons learned
 
-- Newer R versions have issues compiling C code from older package Versions. Using an older R version to build the binaries is recommended.
-- R 4.0.5 is used to build all binaries for R 4.x
-- Binaries which cannot be built using R 4.0.5 often also require older system library or compiler versions and are out of scope for this project.
+- The newest R version needs to be used to build binaries. The reason is that some packages depend on the "recommended" packages and try to install them as a dep. This fails for older R versions, e.g. if R 4.0.5 tries to install `Matrix` from 4.4.x
 
-
-## Platforms 
+## Platforms
 
 - Identifier must match the ones used in <https://github.com/rstudio/r-system-requirements> to be picked up correctly by the automatic syslib dependency installer of `pak`, more specifically by the env var `PKG_SYSREQS_PLATFORM`
 
 ## Helpers
 
-Check the row number of a package by name
+Some common helper functions to ease usage live in `exec.R`.
+
+
+
+240 pkgs in 5h (200-449)
+costs for 5h cax41: 0.235 EUR (0.047)
+
+200 pkgs in 2h 40 min (449 - 649)
+costs for 5h cax41: 0.129 EUR (0.047)
+
+## Common errors
+
+```
+* installing to library '/tmp/Rtmp7WPw19/temp_libpath114b846b58'\n* installing *source* package 'ade4' ...\n** using staged installation\nERROR: a 'NAMESPACE' file is required\n* removing '/tmp/Rtmp7WPw19/temp_libpath114b846b58/ade4'\n"
+```
+
+Tag does not have a NAMESPACE file and hence cannot be built.
+
+```
+"* installing to library '/tmp/RtmpLcCitS/temp_libpath1146aabbe92'\nERROR: dependency 'tripack' is not available for package 'alphahull'\n* removing '/tmp/RtmpLcCitS/temp_libpath1146aabbe92/alphahull'\n"
+```
+
+Dependency not available: Either because the dependency was not declared or errored itself during installation. 
+
+```
+ In function '\033[01m\033[KRcpp::List solveRRBLUP(const mat&, const mat&, const mat&)\033[m\033[K':\n\033[01m\033[KMME.cpp:162:61:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope\n  162 |   double ll = -0.5*(double(optRes[\"objective\"])+df+df*log(2*\033[01;31m\033[KPI\033[m\033[K/df));\n      |                                                             \033[01;31m\033[K^~\033[m\033[K\n\033[01m\033[KMME.cpp:\033[m\033[K In function '\033[01m\033[KRcpp::List solveRRBLUPMV(const mat&, const mat&, const mat&, int, double)\033[m\033[K':\n\033[01m\033[KMME.cpp:277:31:\033[m\033[K \033[01;31m\033[Kerror: \033[m\033[K'\033[01m\033[KPI\033[m\033[K' was not declared in this scope; did you mean '\033[01m\033[KHI\033[m\033[K'?\n  277 |   ll -= double(n*m)/2.0*log(2*\033[01;31m\033[KPI\033[m\033[K);\n      |                               \033[01;31m\033[K^~\033[m\033[K\n      |                               \033[32m\033[KHI\033[m\033[K\nmake: *** [/opt/R/4.4.1/lib/R/etc/Makeconf:204: MME.o] Error 1\nERROR: compilation failed for package 'AlphaSimR'\n* removing '/tmp/RtmpclI5CE/temp_libpath11135d215d5/AlphaSimR'\n
+ ```
+
+ Compiler error: Possible reasons: too old CXX code which cannot be compiled anymore with CXX14 or CXX17.
+
 
-```r
-pkgs = tools::CRAN_package_db()
-which(pkgs$Package == "ABCanalysis")
-```