docs: add 'infrastructure' section

docs/infrastructure/index.qmd

@@ -1 +1,61 @@
-WIP
+---
+title: Infrastructure
+---
+
+The infrastructure is composed out of the following main components:
+
+- A k3s cluster (on Hetzner)
+- S3 object storage (on Hetzner)
+- A CDN (Bunny CDN)
+
+As the resource need for this project is quite variable, running on Kubernetes allows the efficient use of (shared) resources.
+
+## k3s
+
+Efficiently building on different architectures requires access to native servers for those architectures.
+While most cloud providers make this possible, Kubernetes simplifies the process by enabling orchestration of specific builds to designated nodes.
+This flexibility ensures that builds are executed on the appropriate hardware, optimizing performance and compatibility across diverse environments.
+
+### Memory Requirements
+
+Because the memory requirements for each package are vastly different and range between a few hundred MBs to a ~ 12 GB, these must also be reflected in the resource requirements of each pod.
+Using `requests.memory` of 5Gi and `limits.memory` of 14Gi has been proven to work reliable for scheduling and individual resource needs.
+
+### Matrix builds
+
+While daily package updates can be handled with a single process per OS/version, building binaries for all CRAN packages requires a different orchestration strategy.
+On average, each CRAN package has six versions (calculated by dividing the total number of binaries by the OS/versions built and the number of unique packages), making some level of parallelization necessary.
+
+Initially, parallelizing at the level of package versions was implemented.
+However, this approach led to occasional conflicts when dependencies were installed into a shared package cache.
+It also introduced unpredictable memory requirements within workflows: some packages caused memory usage to spike beyond 30 GB, depending on the number of parallel workers.
+These spikes not only caused individual processes to crash but also demanded significantly higher overall resource limits.
+
+A more robust solution was found by processing individual packages sequentially within each matrix job.
+
+To build all versions of all packages, CRAN packages are divided into subsets, each comprising 1/10 or fewer of the total packages, and these subsets are processed in parallel.
+The total time required depends on factors such as the distribution (e.g., distributions with newer C compilers like Alpine tend to be faster) and the number of parallel workers.
+This approach typically takes anywhere from a few days to two weeks.
+
+## Storage: S3
+
+Binaries need to be stored somewhere, and what better option than S3? S3 is significantly more cost-effective than traditional cloud disk storage and offers the added benefit of being accessible via a public API.
+Beyond AWS, the original provider and inventor of S3, there are numerous alternatives with better price-to-storage ratios and lower transfer costs.
+
+The timing was perfect when Hetzner introduced their own S3-compatible object storage, coinciding with the start of this project’s build processing.
+This solution brings multiple advantages: lower overall storage costs, free internal traffic between Hetzner servers and their S3 storage, and the proximity of storage to the build servers, which minimizes upload latency.
+
+::: {.callout-note}
+While uploading packages to S3 is not complicated, there hasn't been any way to create the required PACKAGES index files for binaries stored there.
+This is why forked versions of {cranlike} and {desc} have been created.
+:::
+
+## Content Delivery Network (CDN)
+
+Storing binaries in S3 works well for distribution, but it’s not inherently very fast. Adding a CDN in front of S3 enables caching and allows assets to be distributed via servers located in various regions worldwide. This significantly reduces download latency, making downloads feel much faster.
+
+All packages are delivered through a CDN, which includes three dedicated static caches strategically placed in Germany, the USA, and Asia.
+
+With a CDN in place, downloads are optimized to feel “fast” from virtually anywhere, with only minor variations depending on the user’s location.
+
+The CDN determines when an asset is added to its permanent cache and how often it is revalidated against the S3 source. Since package binaries are one-time builds that typically remain unchanged unless a forced rebuild occurs, relying heavily on a permanent cache is highly efficient in this context.