fix(ci): authenticate B2 list-bucket request in existing-version check
Backblaze B2 requires authentication for the list-bucket API; anonymous access only works for individual public-read objects. The unauthenticated curl returned AccessDenied, so r-version-s3.txt was always empty and every R version was rebuilt despite already existing in s3. Sign the listing request with --aws-sigv4 using the same B2 credentials as the upload step. Also fix the broken version extraction: the old substr($NF, 3, 5) ran against the prefixed key and yielded garbage, so the Build step's grep could never match. Print the actual version on a match and use a literal index() instead of a regex comparison.
This commit is contained in:
parent
4c63350f7d
commit
a939b47199
1 changed files with 14 additions and 2 deletions
|
|
@ -139,15 +139,27 @@ steps:
|
||||||
- name: Check if files for R version already exist in s3
|
- name: Check if files for R version already exist in s3
|
||||||
image: reg.devxy.io/docker.io/library/alpine:3.23
|
image: reg.devxy.io/docker.io/library/alpine:3.23
|
||||||
privileged: true
|
privileged: true
|
||||||
|
environment:
|
||||||
|
AWS_ACCESS_KEY_ID:
|
||||||
|
from_secret: B2_S3_ACCESS_KEY_ID
|
||||||
|
AWS_SECRET_ACCESS_KEY:
|
||||||
|
from_secret: B2_S3_SECRET_KEY
|
||||||
commands:
|
commands:
|
||||||
- ip link set dev eth0 mtu 1280 2>/dev/null || true
|
- ip link set dev eth0 mtu 1280 2>/dev/null || true
|
||||||
- for i in 1 2 3 4 5; do apk add -q --no-cache curl && break; sleep 5; done
|
- for i in 1 2 3 4 5; do apk add -q --no-cache curl && break; sleep 5; done
|
||||||
- |
|
- |
|
||||||
|
# Backblaze B2 requires authentication for the list-bucket API (anonymous
|
||||||
|
# GET works only for individual public-read objects), so the request must be
|
||||||
|
# SigV4-signed with the same credentials used for the upload step. Without
|
||||||
|
# this the listing returns AccessDenied, r-version-s3.txt stays empty, and
|
||||||
|
# every version is rebuilt even though it already exists.
|
||||||
VERSION=$(cat r-version-to-build.txt)
|
VERSION=$(cat r-version-to-build.txt)
|
||||||
curl -s "https://s3.eu-central-003.backblazeb2.com/devxy-r-builds?prefix=${PLATFORM_ID}/" | \
|
curl -s --aws-sigv4 "aws:amz:eu-central-003:s3" \
|
||||||
|
--user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
|
||||||
|
"https://s3.eu-central-003.backblazeb2.com/devxy-r-builds?prefix=${PLATFORM_ID}/" | \
|
||||||
grep -oE '<Key>[^<]+</Key>' | sed 's/<[^>]*>//g' | \
|
grep -oE '<Key>[^<]+</Key>' | sed 's/<[^>]*>//g' | \
|
||||||
awk -v arch="${ARCH_ID}" -v ver="${VERSION}" \
|
awk -v arch="${ARCH_ID}" -v ver="${VERSION}" \
|
||||||
'$0 ~ arch && $0 ~ ver {print substr($NF, 3, 5)}' > r-version-s3.txt
|
'index($0, arch) && index($0, ver) { found = 1 } END { if (found) print ver }' > r-version-s3.txt
|
||||||
- cat r-version-s3.txt
|
- cat r-version-s3.txt
|
||||||
backend_options:
|
backend_options:
|
||||||
kubernetes:
|
kubernetes:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue