chore: move docs/ and dockerfiles to own repo
This commit is contained in:
parent
daa105ff6b
commit
e2d6240d36
GPG key ID:
3C6318841EF78925
17 changed files with 0 additions and 721 deletions
1
docs/.gitignore
vendored
1
docs/.gitignore
vendored
|
|
@ -1 +0,0 @@
|
|||
/.quarto/
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
color:
|
||||
palette:
|
||||
dark: "#1d262f"
|
||||
orange: "#FF9B00"
|
||||
links: "#15428c"
|
||||
# background: blue
|
||||
# foreground: dark-grey
|
||||
primary: '#1d262f' # hyperlinks, active states, primary action buttons, etc.
|
||||
tertiary: "#FF9B00" # hover states, accents, and wells.
|
||||
|
||||
logo:
|
||||
medium:
|
||||
light: logo.png
|
||||
dark: logo.png
|
||||
|
||||
defaults:
|
||||
bootstrap:
|
||||
defaults:
|
||||
navbar-fg: "#1d262f"
|
||||
navbar-bg: "#ffffff"
|
||||
|
||||
typography:
|
||||
fonts:
|
||||
- family: Jura
|
||||
source: bunny
|
||||
- family: Inter
|
||||
source: bunny
|
||||
- family: Montserrat
|
||||
source: bunny
|
||||
base: Inter
|
||||
headings: Montserrat
|
||||
link:
|
||||
decoration: underline
|
||||
color: links
|
||||
monospace:
|
||||
color: primary
|
||||
background-color: "#f0f1f3"
|
||||
size: 0.8rem
|
||||
# decoration:
|
||||
|
||||
meta:
|
||||
link:
|
||||
home: https://www.devxy.io
|
||||
gitlab: https://gitlab.com/devxy
|
||||
bluesky: https://bsky.app/profile/devxy.bsky.social
|
||||
linkedin: https://www.linkedin.com/company/105100347
|
||||
description: Where Data Science and DevOps meet 🤝️
|
||||
founded: 2024
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
project:
|
||||
type: website
|
||||
|
||||
website:
|
||||
title: "CRAN R Package Binaries"
|
||||
reader-mode: false
|
||||
favicon: favicon.ico
|
||||
back-to-top-navigation: true
|
||||
navbar:
|
||||
right:
|
||||
- icon: gitlab
|
||||
href: https://gitlab.com/devxy
|
||||
- icon: browser-safari
|
||||
href: https://www.devxy.io
|
||||
- icon: bluesky
|
||||
href: https://www.devxy.io
|
||||
sidebar:
|
||||
logo: ""
|
||||
style: "docked"
|
||||
collapse-level: 4
|
||||
search: true
|
||||
border: true
|
||||
pinned: true
|
||||
contents:
|
||||
- text: 'Introduction'
|
||||
file: index.qmd
|
||||
- text: "---"
|
||||
- text: 'Configuration'
|
||||
file: configuration/index.qmd
|
||||
- section: "**Use Cases**"
|
||||
contents:
|
||||
- text: "CI/CD"
|
||||
file: cicd/index.qmd
|
||||
- text: "R Development Environments"
|
||||
file: r-dev-environments/index.qmd
|
||||
- text: "Containers"
|
||||
file: containers/index.qmd
|
||||
- section: "**Technical Background**"
|
||||
contents:
|
||||
- text: "Infrastructure"
|
||||
file: infrastructure/index.qmd
|
||||
# - text: "Continuous Updates"
|
||||
# file: 032-continuous-updates.qmd
|
||||
# - section: "**Miscellaneous**"
|
||||
# contents:
|
||||
# - text: "Private Mirror"
|
||||
# file: 040-private-mirror.qmd
|
||||
# - text: "Fair Usage"
|
||||
# file: 041-fair-usage.qmd
|
||||
- text: "---"
|
||||
- text: 'Contributing & Help'
|
||||
file: "contributing/index.qmd"
|
||||
|
||||
search:
|
||||
location: sidebar
|
||||
type: "textbox"
|
||||
limit: 5
|
||||
copy-button: true
|
||||
collapse-after: 2
|
||||
format:
|
||||
html:
|
||||
theme: cosmo
|
||||
css: styles.css
|
||||
fontsize: 1rem
|
||||
toc: true
|
||||
grid:
|
||||
sidebar-width: 300px
|
||||
include-in-header:
|
||||
text: |
|
||||
<script defer src="https://umami.devxy.io/web-info" data-website-id="5e495624-ac45-4f17-ab6a-6b30032f46bb"></script>
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
---
|
||||
title: CI/CD
|
||||
---
|
||||
|
||||
Continuous Integration / continous delivery (CI/CD) is a basic tool in todays Data Science toolstack to automate processes and execute repetite tasks on schedule.
|
||||
|
||||
Running tasks in CI/CD should be fast and reliable, no matter the language, architecture or computing environment at hand.
|
||||
As CI/CD requires installing many packages over and over again, having binaries and optimized distributions helps to reduce runtime.
|
||||
|
||||
This page presents snippets for well and lesser known CI/CD engines to help getting started with the R package binaries of this project.
|
||||
|
||||
::: {.callout-note}
|
||||
The presented snippets rely on external projects and libraries which get updated regurarly.
|
||||
There is no guarantee to work out-of-the box at all times.
|
||||
If you found an error, please open a pull request in the linked repo.
|
||||
:::
|
||||
|
||||
## GitHub Actions
|
||||
|
||||
### VM
|
||||
|
||||
GitHub Actions can be run directly on a VM or in a containered context.
|
||||
The available VM images can be found [here](https://github.com/actions/runner-images#available-images).
|
||||
The only Linux distribution available is Ubuntu.
|
||||
|
||||
For VM workflows, [r-lib/actions](https://github.com/r-lib/actions/blob/v2/examples) provides many examples for different use cases.
|
||||
|
||||
To use package binaries from this project, use the following config for `r-lib/actions/setup-r@v2`:
|
||||
|
||||
```yaml
|
||||
- uses: r-lib/actions/setup-r@v2
|
||||
with:
|
||||
r-version: ${{ matrix.config.r }}
|
||||
http-user-agent: ${{ matrix.config.http-user-agent }}
|
||||
Ncpus: 2
|
||||
cran: 'https://cran.devxy.io/amd64/noble/latest'
|
||||
```
|
||||
|
||||
(`Ncpus: 2` has been set to allow for parallel installations.)
|
||||
|
||||
### Container
|
||||
|
||||
An alternative is to run Actions in a containerized context.
|
||||
This spins up a container of the selected image in the VM and allows running on other distributions than Ubuntu, e.g. on Alpine.
|
||||
To avoid having to install R first and configure a custom repository every time, [specialized images](https://hub.docker.com/r/devxygmbh/r-alpine) are provided which already have everything in place:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
container:
|
||||
runs-on: ubuntu-latest
|
||||
container: devxygmbh:r-alpine
|
||||
steps:
|
||||
- run: |
|
||||
R -q -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/devel/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'
|
||||
R -q -e 'getOption("repos")'
|
||||
R -q -e 'options(Ncpus = 2); pak::local_install_dev_deps()'
|
||||
```
|
||||
|
||||
::: {.callout-note}
|
||||
When running in a containerized context, the predefined actions from `r-lib/actions` cannot be used.
|
||||
Shell/R commands must be used directly.
|
||||
:::
|
||||
|
||||
::: {.callout-note}
|
||||
Running containerized in Alpine should be substantially faster than running in a VM context.
|
||||
:::
|
||||
|
||||
::: {.callout-tip}
|
||||
(91 packages)
|
||||
|
||||
- Containerized: **1m 21s**
|
||||
- VM: 2m 11s
|
||||
|
||||
Both runs are based on the assumption that caching is not used. With caching enabled, the performance of both approaches should be roughly comparable.
|
||||
|
||||
Source: [pat-s/workflow-compare](https://github.com/pat-s/workflow-compare)
|
||||
:::
|
||||
|
||||
|
||||
## GitLab Runner
|
||||
|
||||
WIP
|
||||
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
title: Configuration
|
||||
---
|
||||
|
||||
In R on Linux, packages are downloaded from CRAN as source packages.
|
||||
The default CRAN mirror is `https://cloud.r-project.org`.
|
||||
The default repository URL needs to be **actively changed** to one providing binaries for the respective Linux distribution.
|
||||
This can be done either for all users of the system, on a project level or for specific users only.
|
||||
All options will be shown in the following.
|
||||
|
||||
## URL Scheme
|
||||
|
||||
The URL scheme for this project is the same for all operating systems and only differs in the OS and architecture identifiers.
|
||||
|
||||
::: {.callout-note icon=false}
|
||||
|
||||
## URL scheme
|
||||
|
||||
`https://cran.devxy.io/<architecture>/<os>/latest`
|
||||
:::
|
||||
|
||||
For example, the URL for Alpine Linux 3.21 for `arm64` would be `https://cran.devxy.io/arm64/alpine321/latest`.
|
||||
|
||||
The architecture identifiers are:
|
||||
|
||||
- `arm64`
|
||||
- `amd64`
|
||||
|
||||
The OS identifiers are:
|
||||
|
||||
- `jammy` (Ubuntu 22.04)
|
||||
- `noble` (Ubuntu 24.04)
|
||||
- `rhel8` (RedHat Enterprise Linux 8)
|
||||
- `rhel9` (RedHat Enterprise Linux 9)
|
||||
- `alpine320` (Alpine Linux 3.20)
|
||||
- `alpine321` (Alpine Linux 3.21)
|
||||
|
||||
## Repository Configuration
|
||||
|
||||
### System-wide
|
||||
|
||||
To configure the default R repository for all users of a system, create/edit `$R_HOME/etc/Rprofile.site`.
|
||||
The location of this file depends on how R has been installed.
|
||||
To find its location, execute `R -q -e "paste0(Sys.getenv('R_HOME'),'/etc/Rprofile.site')"`.
|
||||
|
||||
**At the bottom** of the file, add the following:
|
||||
|
||||
```r
|
||||
options(repos = structure(c(CRAN = "<URL>")))
|
||||
```
|
||||
|
||||
The name "CRAN" is optional.
|
||||
It is the canonical default and in some cases also used for special actions (e.g. in Posit Workbench and [{pak}](https://github.com/r-lib/pak)).
|
||||
Unless you have a good reason, it is advisable to stick with that name.
|
||||
|
||||
### User-wide
|
||||
|
||||
To change the default repository option for a specific *user*, add the line from above to `~/.Rprofile`.
|
||||
|
||||
### Project-wide
|
||||
|
||||
To change the default repository option for a specific *project*, add the line from above to `.Rprofile` in the project root directory in which R is started.
|
||||
|
||||
## Verification
|
||||
|
||||
When starting R, execute `getOption("repos")` to verify the repository is correctly configured.
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
---
|
||||
title: Containers
|
||||
---
|
||||
|
||||
The [rocker](https://hub.docker.com/u/rocker) project provides container images for R since several years.
|
||||
Since recently, the images are multi-arch, i.e. `arm64` is supported alongside `amd64`.
|
||||
|
||||
Yet, other distributions than Ubuntu are missing and don't use the devXY binaries out of the box.
|
||||
To simplify running containers that make use of the devXY R binaries implicitly, the following images are provided:
|
||||
|
||||
- [devxygmbh/r-alpine](https://hub.docker.com/r/devxygmbh/r-alpine)
|
||||
- [devxygmbh/r-ubuntu](https://hub.docker.com/r/devxygmbh/r-ubuntu)
|
||||
|
||||
These have the devXY repositories configured by default and can be used efficiently without further configuration changes.
|
||||
|
||||
In contrast to the minimal `rocker/r-ver` images, the `r-ubuntu` images contain a few additional system libraries which are essential for troublefree R usage.
|
||||
|
||||
The following screencast shows how quickly the `r-alpine` image can be used to install the full {tidyverse} package universe in just under 30s:
|
||||
|
||||
<div style="padding:59.12% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/1035389676?badge=0&autopause=0&player_id=0&app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write" style="position:absolute;top:0;left:0;width:100%;height:100%;" title="Alpine package binaries: Install tidyverse"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script>
|
||||
|
||||
::: {.callout-note}
|
||||
The performance of this screencast (and your local experience) depends on a few factors: local download speed, cached binaries in the CDN, proximity to a local edge node of the CDN and your CPU performance and finally the `Ncpus` option in R, defining how many installations are performed in parallel (by default 1).
|
||||
:::
|
||||
|
|
@ -1 +0,0 @@
|
|||
WIP
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
---
|
||||
title: Contributing & Help
|
||||
---
|
||||
|
||||
Contributions are welcome.
|
||||
The project is open-source and GPL licensed.
|
||||
|
||||
The most helpful tasks are to investigate in detail why specific packages failed to build a binary.
|
||||
It usually comes down to a missing system dependency (which is either not declared in the package's DESCRIPTION file or exotic and must be installed from source) or a C code compile issue.
|
||||
|
||||
Please open an issue in the [`r-package-binaries/support`](https://gitlab.com/devxy/r-package-binaries/support) repo.
|
||||
|
||||
::: {.callout-note}
|
||||
The actual repo executing the builds ([devxy/build-binaries](https://git.devxy.io/devxy/build-binaries)) is hosted in a private git instance to be able to use specific private CI runners and is not easily usable for creating public issues, hence the dedicated repo on GitLab.
|
||||
:::
|
||||
BIN
docs/favicon.ico
BIN
docs/favicon.ico
Binary file not shown.
|
Before Width: | Height: | Size: 11 KiB |
|
|
@ -1,24 +0,0 @@
|
|||
---
|
||||
title: "CRAN R Package Binaries"
|
||||
---
|
||||
|
||||
This project provides R package binaries for various Linux distributions and architectures.
|
||||
While it provides binaries for [CRAN packages](https://cran.r-project.org/), it is not affiliated with CRAN in any way.
|
||||
|
||||
Binaries are distributed to the community for free with a monthly download cap of **50 Terabyte**.
|
||||
|
||||
::: {.callout-note}
|
||||
If you represent a for-profit company and plan to make extensive use of these binaries, please get in touch. We offer the option to obtain a private mirror with unlimited access.
|
||||
:::
|
||||
|
||||
While we strive to build binaries for all CRAN packages, there are various reasons why building binaries for a package might fail.
|
||||
(The most common reason is a missing (exotic) external system dependency which the project links against.)
|
||||
The [R package binary dashboard](https://app.devxy.io/app/r-package-binaries-dashboard) provides global coverage information for each for each operating system and which packages are currently missing.
|
||||
|
||||
## Motivation
|
||||
|
||||
The project was established to offer extensive R CRAN package support for common Linux distributions.
|
||||
Since CRAN does not build binaries for Linux, Posit began providing binaries in 2020.
|
||||
However, their efforts were limited to the amd64 architecture and excluded Alpine Linux, which is among the most popular distributions for container-based workflows.
|
||||
|
||||
This project aims to make R a first-class citizien for CI/CD and containers using a community-driven open-source approach.
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
---
|
||||
title: Infrastructure
|
||||
---
|
||||
|
||||
The infrastructure is composed out of the following main components:
|
||||
|
||||
- A k3s cluster (on Hetzner)
|
||||
- S3 object storage (on Hetzner)
|
||||
- A CDN (Bunny CDN)
|
||||
|
||||
As the resource need for this project is quite variable, running on Kubernetes allows the efficient use of (shared) resources.
|
||||
|
||||
## k3s
|
||||
|
||||
Efficiently building on different architectures requires access to native servers for those architectures.
|
||||
While most cloud providers make this possible, Kubernetes simplifies the process by enabling orchestration of specific builds to designated nodes.
|
||||
This flexibility ensures that builds are executed on the appropriate hardware, optimizing performance and compatibility across diverse environments.
|
||||
|
||||
### Memory Requirements
|
||||
|
||||
Because the memory requirements for each package are vastly different and range between a few hundred MBs to a ~ 12 GB, these must also be reflected in the resource requirements of each pod.
|
||||
Using `requests.memory` of 5Gi and `limits.memory` of 14Gi has been proven to work reliable for scheduling and individual resource needs.
|
||||
|
||||
### Matrix builds
|
||||
|
||||
While daily package updates can be handled with a single process per OS/version, building binaries for all CRAN packages requires a different orchestration strategy.
|
||||
On average, each CRAN package has six versions (calculated by dividing the total number of binaries by the OS/versions built and the number of unique packages), making some level of parallelization necessary.
|
||||
|
||||
Initially, parallelizing at the level of package versions was implemented.
|
||||
However, this approach led to occasional conflicts when dependencies were installed into a shared package cache.
|
||||
It also introduced unpredictable memory requirements within workflows: some packages caused memory usage to spike beyond 30 GB, depending on the number of parallel workers.
|
||||
These spikes not only caused individual processes to crash but also demanded significantly higher overall resource limits.
|
||||
|
||||
A more robust solution was found by processing individual packages sequentially within each matrix job.
|
||||
|
||||
To build all versions of all packages, CRAN packages are divided into subsets, each comprising 1/10 or fewer of the total packages, and these subsets are processed in parallel.
|
||||
The total time required depends on factors such as the distribution (e.g., distributions with newer C compilers like Alpine tend to be faster) and the number of parallel workers.
|
||||
This approach typically takes anywhere from a few days to two weeks.
|
||||
|
||||
## Storage: S3
|
||||
|
||||
Binaries need to be stored somewhere, and what better option than S3? S3 is significantly more cost-effective than traditional cloud disk storage and offers the added benefit of being accessible via a public API.
|
||||
Beyond AWS, the original provider and inventor of S3, there are numerous alternatives with better price-to-storage ratios and lower transfer costs.
|
||||
|
||||
The timing was perfect when Hetzner introduced their own S3-compatible object storage, coinciding with the start of this project’s build processing.
|
||||
This solution brings multiple advantages: lower overall storage costs, free internal traffic between Hetzner servers and their S3 storage, and the proximity of storage to the build servers, which minimizes upload latency.
|
||||
|
||||
::: {.callout-note}
|
||||
While uploading packages to S3 is not complicated, there hasn't been any way to create the required PACKAGES index files for binaries stored there.
|
||||
This is why forked versions of {cranlike} and {desc} have been created.
|
||||
:::
|
||||
|
||||
## Content Delivery Network (CDN)
|
||||
|
||||
Storing binaries in S3 works well for distribution, but it’s not inherently very fast. Adding a CDN in front of S3 enables caching and allows assets to be distributed via servers located in various regions worldwide. This significantly reduces download latency, making downloads feel much faster.
|
||||
|
||||
All packages are delivered through a CDN, which includes three dedicated static caches strategically placed in Germany, the USA, and Asia.
|
||||
|
||||
With a CDN in place, downloads are optimized to feel “fast” from virtually anywhere, with only minor variations depending on the user’s location.
|
||||
|
||||
The CDN determines when an asset is added to its permanent cache and how often it is revalidated against the S3 source. Since package binaries are one-time builds that typically remain unchanged unless a forced rebuild occurs, relying heavily on a permanent cache is highly efficient in this context.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 49 KiB |
BIN
docs/logo.png
BIN
docs/logo.png
Binary file not shown.
|
Before Width: | Height: | Size: 24 KiB |
|
|
@ -1,32 +0,0 @@
|
|||
An R development environment is a shared Linux VM or Kubernetes Cluster that runs an Integrated Development Environment (IDE), e.g. Posit Workbench or RStudio Server, and other related applications, such as Posit Connect or ShinyProxy.
|
||||
In such an environment, multiple users can share the available resources and configurations can be set centrally for all users[^1].
|
||||
|
||||
Using package binaries in shared development environments is a crucial component for the overall success and efficiency of each individual user.
|
||||
|
||||
## Fast Package Downloads
|
||||
|
||||
Besides using R package binaries in the first place, a fast download speed further improves the quality of such an environment.
|
||||
The binaries of this project are 5-10 times faster in download compared to other available binaries*.
|
||||
|
||||
::: {.callout-note}
|
||||
The exact timing depends on the location from where the download is initiated.
|
||||
The closer it is to static cache node of the distributing CDN, the faster.
|
||||
:::
|
||||
|
||||
## Reduced Costs Through arm64 Servers
|
||||
|
||||
Besides accelerating downloads, this project enabled using `arm64` machines for such environments.
|
||||
These server types are substantially cheaper than their `amd64` counterparts, allowing for significant cost savings without any negative drawbacks.
|
||||
|
||||
The following example shows the on-demand prices of a 2 CPU 16 GB VM on Azure (2025/01):
|
||||
|
||||
- amd64: 0.1640/h (Standard_E2ds_v6, eastus)
|
||||
- arm64: 0.1150/h (Standard_E2ps_v6, eastus)
|
||||
|
||||
-> 29.88% savings
|
||||
|
||||
::: {.callout-note}
|
||||
Servers with the same specifications but from different generations can have significant price variations. For this comparison, an effort was made to use the latest configurations of both architectures to ensure a reasonably fair evaluation.
|
||||
:::
|
||||
|
||||
[^1]: For more information about managed R environments, see ["Posit Software Installation for Enterprises" offer from devXY](https://www.devxy.io/posit-enterprise/).
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
/* css styles */
|
||||
|
||||
div.sourceCode {
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
p code:not(.sourceCode),
|
||||
li code:not(.sourceCode),
|
||||
kbd,
|
||||
pre:not(.sourceCode),
|
||||
samp {
|
||||
background-color: #f6f6f6 !important;
|
||||
padding: 0.25em;
|
||||
border-radius: 6px !important;
|
||||
}
|
||||
|
||||
/* sidebar */
|
||||
.sidebar-navigation li a {
|
||||
line-height: 1.2rem;
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.navbar {
|
||||
border-bottom: 1px solid #e5e5e6;
|
||||
max-height: 50px;
|
||||
|
||||
}
|
||||
|
||||
.navbar-logo {
|
||||
max-height: 30px;
|
||||
}
|
||||
Loading…
Reference in a new issue