devxy/build-cran-binaries
Watch
1
0
Fork
You've already forked build-cran-binaries
 0
Code Issues 10 Pull requests Releases Activity

update README

Browse source
This commit is contained in:
Patrick Schratz 2024-12-13 14:33:29 +01:00
commit 6a62efe646
Signed by: pat-s
GPG key ID: 3C6318841EF78925
1 changed files with 38 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

74
README.md
View file

@ -13,8 +13,8 @@ It achieves this through the integration of several components:
## R Package
The R package `bincraftR` is the engine behind everything.
It provides functions that can:
The R package [`bincraftR`](https://gitlab.com/devxy/r-package-binaries/bincraftr) powers everything.
It provides the following functioanlity:
- build binaries
- archive packages following the CRAN-like directory structure
@ -53,7 +53,7 @@ For every package+tag combination:
## Build Environment
Binaries are built on a mixed-architecture Kubernetes cluster using CI.
Dedicated arm64 and amd64 nodes are utilized to efficiently build the binaries.
Dedicated `arm64`/`amd64` nodes are utilized to efficiently build the binaries.
After all binaries for a specific architecture/OS combination are built, CRON jobs handle the processing of daily change operations.
This elastic server architecture offers a robust and performant backend while minimizing costs.
@ -69,22 +69,21 @@ Hence, modified versions of {cranlike} and {desc} were created that are able to
### Resources
Reasonably sized instances with performant CPUs are important to build binaries in a reasonable time.
While building binaries, it was found that a single process might need up to 14 GB of memory, as certain packages on CRAN require that much to build.
While this applies to only a few packages and most do not exceed 2 GB of memory, the exact RAM requirement for each individual package is unknown.
To ensure that any package can be processed without the risk of running out of memory (OOM), a safety margin of using 16 GB of memory is the suggested minimum requirement.
This means that a VM with 16 GB of memory can build binaries sequentially.
With 32 GB of memory, two cores can be used to process multiple packages in parallel.
For efficient binary building, reasonably sized instances with high-performance CPUs are crucial to complete tasks within a reasonable timeframe.
Important: the parallelism applies at the tag level, not at the package level, and this behavior cannot currently be changed.
During the process, it was observed that a single build can consume up to 14 GB of memory for certain packages.
While the majority of packages require less than 2 GB of memory, the exact RAM requirements for individual packages are unpredictable.
To prevent the risk of running out of memory (OOM) during builds, the memory allocation for individual builds should not be set below 14 GB.
Parallel builds were tested initially but led to occasional issues with parallel writes and rate limits.
Currently, all builds are executed sequentially. It is important to note that parallelism is applied at the tag level, not at the package level, and this behavior cannot be modified at present.
### Dependency Cache
A build cache for both R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a persistent volume for each architecture/OS combination.
Additionally, the PACKAGES index files are persisted to speed up adding new packages when calling `upload_package_index()`.
Otherwise, the entire (SQLite) database would need to be created from scratch, which takes considerable time and requires numerous API calls to Backblaze.
A build cache for both R packages (`/mnt/cache/R-pkgs`) and `ccache` (`/mnt/cache/ccache`) is stored in a persistent volume for each architecture/OS combination to speed up dependency installation.
Processing all CRAN packages (approximately 21k) takes around 40 minutes, while processing updates with an existing database file takes around 5 minutes.
When updating the PACKAGES\* index files (`PACKAGES`, `PACKAGES.gz`, `PACKAGES.rds`), processing all CRAN packages (21k+) takes around 40 minutes.
Processing updates with an existing database file takes around 5 minutes.
### Inferring System Dependencies
@ -98,7 +97,7 @@ For Alpine, no rules existed until recently, establishing a foundation for semi-
The build metadata is stored in a PostgreSQL database.
The database has a public endpoint at `r-binaries.devxy.io` and port `15432`.
The database contains one table named `single_builds`, which holds the build metadata for each package:
It contains one table named `single_builds`, which holds the build metadata for each package:
| package_name | tag | platform | error_occurred | build_timestamp | build_duration | error | size | removed |
| ------------ | --- | -------- | -------------- | --------------- | -------------- | ----- | ---- | ------- |
@ -116,10 +115,9 @@ Column types:
- `removed`: boolean
Alternatively, use `\d+ single_builds`.
**Note**: There is currently no read-only role available to connect to the DB as a viewer.
A shiny dashboard providing a search functionality of the database and grouped statistics is available in `shiny/`.
The self-hosted database is running on a Kubernetes Cluster in HA mode.
A Shiny dashboard providing a search functionality of the database and grouped statistics is available at <https://shinyproxy.devxy.io/app/r-package-binaries-dashboard> with the source repo living at <https://gitlab.com/devxy/r-package-binaries/shiny>.
## Support for Archived Versions
@ -131,8 +129,7 @@ This file must be a list of data frames containing information about the archive
Example:
```r
con <- gzcon(url(sprintf("%s/src/contrib/Meta/archive.rds",
c("CRAN" = "https://cloud.r-project.org")), "rb"))
con <- gzcon(url(sprintf("%s/src/contrib/Meta/archive.rds", c("CRAN" = "https://cloud.r-project.org")), "rb"))
foo = readRDS(con)
foo[[1]]
```
@ -219,7 +216,7 @@ internal error 1 in memDecompress
Solution:
```sh
rm -rf /mnt/cache/R-pkgs/pkgcache/ /mnt/cache/R-pkgs/pak /mnt/cache/pkgcache/ /root/.cache/R/
rm -rf /mnt/cache/R-pkgs/pak /mnt/cache/pkgcache/ /root/.cache/R/
R -q -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'
```
@ -231,19 +228,25 @@ Some packages have been intentionally skipped after multiple build attempts.
The reasons for this vary, and ideally, solutions can be found over the long term.
Contributions to help resolve these issues are highly welcome!
| Name | Platform | Arch | Reason | Solved via | Date Created | Date Solved |
| ------------- | -------------- | ------- | ------------- | ---------- | ------------ | ----------- |
| Apollonius | redhat-8 | | gmp missing | 2024-11-15 | |
| doBy | alpine-320 | amd | hangs | 2024-11-23 | |
| later | ubuntu 22 & 24 | amd | hangs - xfvb? | 2024-11-23 | |
| CoTiMA | ubuntu 22 & 24 | amd | OOM? | 2024-11-23 | |
| FrF2 | alpine | arm | hangs | 2024-11-23 | |
| FrF2.catlg128 | alpine | arm | hangs | 2024-11-23 | |
| DoE.base | alpine | arm/amd | hangs | 2024-11-23 | |
| eha | alpine | amd | hangs | 2024-11-23 | |
| gRain | alpine | amd | hangs | 2024-11-23 | |
| gRbase | alpine | amd | hangs | 2024-11-24 | |
| IDPmisc | alpine | amd | hangs | 2024-11-24 | |
| Name | Platform | Arch | Reason | Solved via | Date Created | Date Solved |
| ------------- | -------------- | ------- | -------------------------------- | ---------- | ------------ | ----------- |
| Apollonius | redhat-8 | | gmp missing | 2024-11-15 | |
| doBy | alpine-320 | amd | hangs | 2024-11-23 | |
| later | ubuntu 22 & 24 | amd | hangs for early versions - xfvb? | 2024-11-23 | |
| CoTiMA | ubuntu 22 & 24 | amd | OOM? | 2024-11-23 | |
| FrF2 | alpine | arm | hangs | 2024-11-23 | |
| FrF2.catlg128 | alpine | arm | hangs | 2024-11-23 | |
| DoE.base | alpine | arm/amd | hangs | 2024-11-23 | |
| eha | alpine | amd | hangs | 2024-11-23 | |
| gRain | alpine | amd | hangs | 2024-11-23 | |
| gRbase | alpine | amd | hangs | 2024-11-24 | |
| IDPmisc | alpine | amd | hangs | 2024-11-24 | |
| RVAideMemoire | alpine | amd | hangs | 2024-11-25 | |
| pbkrtest | alpine | arm | hangs | 2024-11-25 | |
| seewave | alpine | arm | hangs | 2024-11-25 | |
| spdep | alpine | arm | hangs | 2024-12-03 | |
| compareGroups | alpine | arm | hangs | 2024-12-11 | |
| SNPassoc | alpine | arm | hangs | 2024-12-11 | |
## CDN Settings
@ -253,5 +256,4 @@ A "Perma-Cache" is enabled for three different regions around the world (DE, US,
Once a binary is requested for the first time from a specific location, the asset is copied to the perma-cache and served from there for subsequent requests.
The `CacheControl = "no-cache"` header is set for all PACKAGES\* files to ensure users always receive the latest version, as these files change daily.
A monthly traffic limit of 50 TB is set on cran.devxy.io to prevent abuse and manage costs.
Additionally, the cache for these files is purged after every update.