diff --git a/.crow/backfill-flat-safety.yaml b/.crow/backfill-flat-safety.yaml index 0db0735..cfb2be8 100644 --- a/.crow/backfill-flat-safety.yaml +++ b/.crow/backfill-flat-safety.yaml @@ -140,12 +140,11 @@ steps: from_secret: PGPASS REPO_RO_TOKEN: from_secret: REPO_RO_TOKEN - # uvr resolves bincraft's dependencies through the GitHub API, including - # the pat-s/cranlike@s3 fork. Unauthenticated it is rate limited, and the - # failure is not loud: uvr warns, falls back to the registry copy of - # cranlike, and the index write then dies on `unused argument (built = - # built)` because upstream has no such parameter. Intermittent by nature, - # since it depends on the rate-limit window. + # bincraft reads the DESCRIPTION of every new or replaced tarball from + # the CRAN mirror on raw.githubusercontent.com, and authenticates with + # this token when it is set. A slot whose published index cannot be read + # is re-read in full, ~25k requests, which the anonymous rate limit would + # cut short. GITHUB_PAT: from_secret: GITHUB_PAT GIT_USER: pat-s diff --git a/.crow/reindex.yaml b/.crow/reindex.yaml index 803d9a5..7db77fa 100644 --- a/.crow/reindex.yaml +++ b/.crow/reindex.yaml @@ -141,12 +141,11 @@ steps: from_secret: PGPASS REPO_RO_TOKEN: from_secret: REPO_RO_TOKEN - # uvr resolves bincraft's dependencies through the GitHub API, including - # the pat-s/cranlike@s3 fork. Unauthenticated it is rate limited, and the - # failure is not loud: uvr warns, falls back to the registry copy of - # cranlike, and the index write then dies on `unused argument (built = - # built)` because upstream has no such parameter. Intermittent by nature, - # since it depends on the rate-limit window. + # bincraft reads the DESCRIPTION of every new or replaced tarball from + # the CRAN mirror on raw.githubusercontent.com, and authenticates with + # this token when it is set. A slot whose published index cannot be read + # is re-read in full, ~25k requests, which the anonymous rate limit would + # cut short. GITHUB_PAT: from_secret: GITHUB_PAT GIT_USER: pat-s diff --git a/.crow/weekly-rebuild-reindex.yaml b/.crow/weekly-rebuild-reindex.yaml index 698f503..909722f 100644 --- a/.crow/weekly-rebuild-reindex.yaml +++ b/.crow/weekly-rebuild-reindex.yaml @@ -4,8 +4,8 @@ # objects in place, so the slot's index still advertises the old MD5 and, for # anything that had been served from source, no Built stamp. Re-indexing from # inside a shard would mean three concurrent `upload_package_index()` calls on -# one prefix: `cranlike::update_PACKAGES()` lists the live bucket, so an early -# lister that uploads last publishes an index missing its siblings' work. +# one prefix: each lists the live bucket, so an early lister that uploads last +# publishes an index missing its siblings' work. # # So it happens exactly once per slot, here, after every shard has finished. # `runs_on: [success, failure]` keeps that true when a shard fails; only an @@ -144,12 +144,11 @@ steps: from_secret: PGPASS REPO_RO_TOKEN: from_secret: REPO_RO_TOKEN - # uvr resolves bincraft's dependencies through the GitHub API, including - # the pat-s/cranlike@s3 fork. Unauthenticated it is rate limited, and the - # failure is not loud: uvr warns, falls back to the registry copy of - # cranlike, and the index write then dies on `unused argument (built = - # built)` because upstream has no such parameter. Intermittent by nature, - # since it depends on the rate-limit window. + # bincraft reads the DESCRIPTION of every new or replaced tarball from + # the CRAN mirror on raw.githubusercontent.com, and authenticates with + # this token when it is set. A slot whose published index cannot be read + # is re-read in full, ~25k requests, which the anonymous rate limit would + # cut short. GITHUB_PAT: from_secret: GITHUB_PAT GIT_USER: pat-s diff --git a/README.md b/README.md index 99d1143..9dceb42 100644 --- a/README.md +++ b/README.md @@ -103,11 +103,13 @@ Preview the selection without building or accessing storage or build metadata: ### Creating/Updating the PACKAGES Index Files -Currently, the {cranlike} and {desc} packages only work with files on a local file system. +Tools such as `tools::write_PACKAGES()` and {cranlike} only index files on a local file system. This is infeasible if the goal is to store binaries in S3. Storing binaries permanently on a disk-based file system would incur significantly higher costs, especially when operating in the cloud. -Hence, [{cranlike}](https://devxy.gitlab.io/r-package-binaries/bincraft/) and [{desc}](https://github.com/pat-s/desc/tree/description-from-remote) were forked to handle files in S3 (through {s3fs}). +Hence, `bincraft::upload_package_index()` builds the index from a listing of the slot in S3 (through {s3fs}). +Each record is read from the package's CRAN `DESCRIPTION` on the GitHub mirror of CRAN, falling back to the `DESCRIPTION` inside the tarball for a package the mirror does not have. +The previously published `PACKAGES.rds` serves as a cache, so only new or replaced tarballs are read. ### Resources