Compare commits

...
Sign in to create a new pull request.
Author SHA1 Message Date
56d26ef59c
docs: add CLAUDE.md with B2 storage gotchas and CI workflow conventions for agents
All checks were successful
ci/crow/cron/build/16 Pipeline was successful
ci/crow/cron/build/14 Pipeline was successful
ci/crow/cron/build/6 Pipeline was successful
ci/crow/cron/build/3 Pipeline was successful
ci/crow/cron/build/15 Pipeline was successful
ci/crow/cron/build/12 Pipeline was successful
ci/crow/cron/build/9 Pipeline was successful
ci/crow/cron/build/18 Pipeline was successful
ci/crow/cron/build/17 Pipeline was successful
ci/crow/cron/build/7 Pipeline was successful
ci/crow/cron/build/11 Pipeline was successful
ci/crow/cron/build/1 Pipeline was successful
ci/crow/cron/build/5 Pipeline was successful
ci/crow/cron/build/2 Pipeline was successful
ci/crow/cron/build/10 Pipeline was successful
ci/crow/cron/build/4 Pipeline was successful
ci/crow/cron/build/8 Pipeline was successful
ci/crow/cron/build/13 Pipeline was successful
2026-07-02 08:43:35 +02:00
8eacc0ab47
fix(ci): use bare shell vars in existing-version check (crow ${VAR} substitution)
All checks were successful
ci/crow/cron/build/6 Pipeline was successful
ci/crow/cron/build/10 Pipeline was successful
ci/crow/cron/build/16 Pipeline was successful
ci/crow/cron/build/8 Pipeline was successful
ci/crow/cron/build/5 Pipeline was successful
ci/crow/cron/build/18 Pipeline was successful
ci/crow/cron/build/14 Pipeline was successful
ci/crow/cron/build/17 Pipeline was successful
ci/crow/cron/build/9 Pipeline was successful
ci/crow/cron/build/13 Pipeline was successful
ci/crow/cron/build/4 Pipeline was successful
ci/crow/cron/build/15 Pipeline was successful
ci/crow/cron/build/7 Pipeline was successful
ci/crow/cron/build/3 Pipeline was successful
ci/crow/cron/build/2 Pipeline was successful
ci/crow/cron/build/12 Pipeline was successful
ci/crow/cron/build/11 Pipeline was successful
ci/crow/cron/build/1 Pipeline was successful
The merged HEAD-probe check still rebuilt everything: crow runs its own
${VAR} substitution over the commands before the shell executes, so the
shell variables ${BASE} and ${VERSION} were blanked to empty (only matrix
vars resolve at that stage). Every probed URL was therefore malformed and
returned non-200, leaving r-versions-existing.txt empty.

Use bare $name for shell variables and build the filename with printf, which
avoids the ${VERSION}_1 brace requirement entirely. Matrix vars (${PLATFORM},
${PLATFORM_ID}, ${ARCH_ID}) keep braces since crow resolves those correctly.
Verified URL construction against the live bucket for apk, deb and rpm.
2026-06-29 14:15:55 +02:00
ca336c2720 fix(ci): detect existing R versions via public object HEAD probe (#8)
Some checks failed
ci/crow/cron/build/1 Pipeline failed
ci/crow/cron/build/3 Pipeline was canceled
ci/crow/cron/build/4 Pipeline was canceled
ci/crow/cron/build/12 Pipeline was canceled
ci/crow/cron/build/5 Pipeline was canceled
ci/crow/cron/build/14 Pipeline was canceled
ci/crow/cron/build/6 Pipeline was canceled
ci/crow/cron/build/2 Pipeline was canceled
ci/crow/cron/build/13 Pipeline was canceled
ci/crow/cron/build/17 Pipeline was canceled
ci/crow/cron/build/18 Pipeline was canceled
ci/crow/cron/build/9 Pipeline was canceled
ci/crow/cron/build/16 Pipeline was canceled
ci/crow/cron/build/10 Pipeline was canceled
ci/crow/cron/build/11 Pipeline was canceled
ci/crow/cron/build/7 Pipeline was canceled
ci/crow/cron/build/8 Pipeline was canceled
ci/crow/cron/build/15 Pipeline was canceled
The existing-version check listed the bucket with an anonymous curl, but
Backblaze B2 requires authentication for the list-bucket API, so it returned
AccessDenied and r-versions-existing.txt was always empty -> every version
was rebuilt despite already existing in s3.

Signing the request is not workable here: the B2 credentials are scoped to
the plugin-s3 image in crow and are not injected into this plain alpine step
(curl saw --user ":").

Individual artifacts are public-read, so probe each expected object URL with
an anonymous HEAD instead of listing the bucket. The filename mirrors what
nfpm produces (version/release pinned to 1): r-<ver>_1_<arch>.{apk,deb} for
alpine/ubuntu and R-<ver>-1-1.<arch>.rpm for el*. Verified against the live
bucket for all three packager families.

Reviewed-on: #8
2026-06-29 12:11:03 +00:00
2 changed files with 30 additions and 16 deletions

View file

@ -158,28 +158,29 @@ steps:
- name: Check which R versions already exist in s3
image: reg.devxy.io/docker.io/library/alpine:3.23
privileged: true
environment:
AWS_ACCESS_KEY_ID:
from_secret: B2_S3_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY:
from_secret: B2_S3_SECRET_KEY
commands:
- ip link set dev eth0 mtu 1280 2>/dev/null || true
- for i in 1 2 3 4 5; do apk add -q --no-cache curl && break; sleep 5; done
- |
# Backblaze B2 requires authentication for the list-bucket API (anonymous
# GET works only for individual public-read objects), so the request must be
# SigV4-signed with the same credentials used for the upload step. Without
# this the listing returns AccessDenied, r-versions-existing.txt stays empty,
# and every version is rebuilt even though it already exists.
LISTING=$(curl -s --aws-sigv4 "aws:amz:eu-central-003:s3" \
--user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
"https://s3.eu-central-003.backblazeb2.com/devxy-r-builds?prefix=${PLATFORM_ID}/" | \
grep -oE '<Key>[^<]+</Key>' | sed 's/<[^>]*>//g')
# Backblaze B2's list-bucket API requires authentication, but the upload
# credentials are scoped to the plugin-s3 image and aren't available here.
# Individual objects are public-read, so probe each expected artifact URL
# with an anonymous HEAD instead of listing the bucket. The filename mirrors
# what nfpm produces in builder/package.${PLATFORM} (version/release pinned
# to 1): r-<ver>_1_<arch>.{apk,deb} for alpine/ubuntu, R-<ver>-1-1.<arch>.rpm
# for el*. Shell variables must use bare $name, not ${name}: crow performs
# its own ${VAR} substitution on these commands before the shell runs, so
# ${VAR} for a shell var would be blanked out (only matrix vars resolve there).
case "${PLATFORM}" in
alpine-*) FMT="r-%s_1_%s.apk" ;;
ubuntu-*) FMT="r-%s_1_%s.deb" ;;
*) FMT="R-%s-1-1.%s.rpm" ;;
esac
BASE="https://s3.eu-central-003.backblazeb2.com/devxy-r-builds/${PLATFORM_ID}"
: > r-versions-existing.txt
for VERSION in $(cat r-versions-to-build.txt); do
if printf '%s\n' "$LISTING" | \
awk -v arch="${ARCH_ID}" -v ver="$VERSION" 'index($0, arch) && index($0, ver) { found=1 } END { exit !found }'; then
URL="$BASE/$(printf "$FMT" "$VERSION" "${ARCH_ID}")"
if [ "$(curl -s -o /dev/null -w '%{http_code}' -I "$URL")" = "200" ]; then
echo "$VERSION" >> r-versions-existing.txt
fi
done

13
CLAUDE.md Normal file
View file

@ -0,0 +1,13 @@
# CLAUDE.md
CI pipelines (Crow, in `.crow/`) that build R binaries and upload them to Backblaze B2.
## Conventions
- **PRs:** the remote is Forgejo on `git.devxy.io`; use `fj -H git.devxy.io` (not `gh`).
- **Storage:** Backblaze B2 bucket `devxy-r-builds` (endpoint configured in `.crow/build.yaml`).
## Gotchas
- **B2 requires authentication for the list-bucket API.** Anonymous GET only works for individual public-read objects — an empty bucket listing usually means missing credentials, not an empty bucket. This silently broke rebuild dedup once.
- When a pipeline fails, fetch the Crow logs yourself instead of asking the user to paste them, and monitor reruns in the background until green.