docs: add 'infrastructure' section

.gitignore

@@ -43,9 +43,6 @@ vignettes/*.pdf
 # R Environment Variables
 .Renviron
 
-# pkgdown site
-docs/
-
 # translation temp files
 po/*~
 

docs/_brand.yaml

@@ -22,13 +22,12 @@ defaults:
 typography:
   fonts:
     - family: Jura
-      source: google
+      source: bunny
-    - family: Inter Variable
+    - family: Inter
-      source: google
+      source: bunny
-    - family: Montserrat Alternates
+    - family: Montserrat
-      source: google
+      source: bunny
-  base: 
+  base: Inter
-    family: Inter Variable
   headings: Montserrat
   link:
     decoration: underline

docs/_quarto.yml

@@ -35,10 +35,10 @@ website:
             file: r-dev-environments/index.qmd
           - text: "Containers"
             file: containers/index.qmd
-      # - section: "**Technical Background**"
+      - section: "**Technical Background**"
-      #   contents:
+        contents:
-      #     - text: "Infrastructure"
+          - text: "Infrastructure"
-      #       file: 031-infrastructure.qmd
+            file: infrastructure/index.qmd
       #     - text: "Continuous Updates"
       #       file: 032-continuous-updates.qmd
       # - section: "**Miscellaneous**"

docs/index.qmd

@@ -5,7 +5,7 @@ title: "CRAN R Package Binaries"
 This project provides R package binaries for various Linux distributions and architectures.
 While it provides binaries for [CRAN packages](https://cran.r-project.org/), it is not affiliated with CRAN in any way.
 
-Binaries are distributed to the community for free with a monthly download cap of **50 Terrabyte**.
+Binaries are distributed to the community for free with a monthly download cap of **50 Terabyte**.
 
 ::: {.callout-note}
 If you represent a for-profit company and plan to make extensive use of these binaries, please get in touch. We offer the option to obtain a private mirror with unlimited access.

docs/infrastructure/index.qmd

@@ -1 +1,61 @@
-WIP
+---
+title: Infrastructure
+---
+
+The infrastructure is composed out of the following main components:
+
+- A k3s cluster (on Hetzner)
+- S3 object storage (on Hetzner)
+- A CDN (Bunny CDN)
+
+As the resource need for this project is quite variable, running on Kubernetes allows the efficient use of (shared) resources.
+
+## k3s
+
+Efficiently building on different architectures requires access to native servers for those architectures.
+While most cloud providers make this possible, Kubernetes simplifies the process by enabling orchestration of specific builds to designated nodes.
+This flexibility ensures that builds are executed on the appropriate hardware, optimizing performance and compatibility across diverse environments.
+
+### Memory Requirements
+
+Because the memory requirements for each package are vastly different and range between a few hundred MBs to a ~ 12 GB, these must also be reflected in the resource requirements of each pod.
+Using `requests.memory` of 5Gi and `limits.memory` of 14Gi has been proven to work reliable for scheduling and individual resource needs.
+
+### Matrix builds
+
+While daily package updates can be handled with a single process per OS/version, building binaries for all CRAN packages requires a different orchestration strategy.
+On average, each CRAN package has six versions (calculated by dividing the total number of binaries by the OS/versions built and the number of unique packages), making some level of parallelization necessary.
+
+Initially, parallelizing at the level of package versions was implemented.
+However, this approach led to occasional conflicts when dependencies were installed into a shared package cache.
+It also introduced unpredictable memory requirements within workflows: some packages caused memory usage to spike beyond 30 GB, depending on the number of parallel workers.
+These spikes not only caused individual processes to crash but also demanded significantly higher overall resource limits.
+
+A more robust solution was found by processing individual packages sequentially within each matrix job.
+
+To build all versions of all packages, CRAN packages are divided into subsets, each comprising 1/10 or fewer of the total packages, and these subsets are processed in parallel.
+The total time required depends on factors such as the distribution (e.g., distributions with newer C compilers like Alpine tend to be faster) and the number of parallel workers.
+This approach typically takes anywhere from a few days to two weeks.
+
+## Storage: S3
+
+Binaries need to be stored somewhere, and what better option than S3? S3 is significantly more cost-effective than traditional cloud disk storage and offers the added benefit of being accessible via a public API.
+Beyond AWS, the original provider and inventor of S3, there are numerous alternatives with better price-to-storage ratios and lower transfer costs.
+
+The timing was perfect when Hetzner introduced their own S3-compatible object storage, coinciding with the start of this project’s build processing.
+This solution brings multiple advantages: lower overall storage costs, free internal traffic between Hetzner servers and their S3 storage, and the proximity of storage to the build servers, which minimizes upload latency.
+
+::: {.callout-note}
+While uploading packages to S3 is not complicated, there hasn't been any way to create the required PACKAGES index files for binaries stored there.
+This is why forked versions of {cranlike} and {desc} have been created.
+:::
+
+## Content Delivery Network (CDN)
+
+Storing binaries in S3 works well for distribution, but it’s not inherently very fast. Adding a CDN in front of S3 enables caching and allows assets to be distributed via servers located in various regions worldwide. This significantly reduces download latency, making downloads feel much faster.
+
+All packages are delivered through a CDN, which includes three dedicated static caches strategically placed in Germany, the USA, and Asia.
+
+With a CDN in place, downloads are optimized to feel “fast” from virtually anywhere, with only minor variations depending on the user’s location.
+
+The CDN determines when an asset is added to its permanent cache and how often it is revalidated against the S3 source. Since package binaries are one-time builds that typically remain unchanged unless a forced rebuild occurs, relying heavily on a permanent cache is highly efficient in this context.