mdm-ordinis/docs-internal/ops/nexus-alpine-mirror.md

Nexus Alpine Mirror — Operations Runbook

Owner: DevOps (cluster.265 infra) Status: Active issue — APKINDEX рассинхронизирован с file storage на v3.23 Related: ordinis ci/apk fixes chain (!227, !229, !230)

---

Problem

Nexus mirror на repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/:

Что	Состояние
APKINDEX.tar.gz	Claims ca-certificates-20251003-r0, libexpat-2.7.3-r0
Filesystem	Только ca-certificates-20260413-r0.apk, libexpat-2.7.5-r0.apk
Result	apk add → HTTP 404 на 2 packages → fail

Проверить актуальность:

curl -s https://repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/APKINDEX.tar.gz \
  | tar xz -C /tmp/ \
  && grep -A1 "^P:ca-certificates$" /tmp/APKINDEX | head -2

# Сравнить с file inventory:
curl -s "https://repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/" \
  | grep "ca-certificates-" | head -3

v3.20 mirror — consistent (APKINDEX и files matching). v3.21+ — нужно проверить per-version.

---

Impact

• node:22-alpine (latest) uses Alpine v3.23 → broken apk add в CI
• Releases pipeline ломается на 2 stages (release job → semantic-release)
• propose-prod-image-bump (alpine:3.20 image) — работает благодаря consistent v3.20 mirror

Workaround (active в проде)

ordinis .gitlab-ci.yml pins release job:

release:
  image: repo.nstart.cloud/library/node:22-alpine3.20  # explicit v3.20 pin

Все Alpine jobs делают sed swap к internal mirror + apk update:

- sed -i 's|https://dl-cdn.alpinelinux.org/alpine|https://repo.nstart.cloud/repository/alpine|g' /etc/apk/repositories
- apk update
- apk add --no-cache <packages>

---

Fix (нужен DevOps action)

Option 1 (recommended): Force Nexus APKINDEX rebuild

Nexus admin UI / API → repository/alpine/v3.23/main/x86_64/APKINDEX.tar.gz:

# Trigger rebuild через Nexus API (требует admin token)
curl -X POST -u admin:<token> \
  "https://repo.nstart.cloud/service/rest/v1/repositories/alpine/proxy/<repo>/invalidate-cache"

Или через UI: Repositories → alpine-proxy → Invalidate Cache.

После invalidate — следующий apk request триггерит fresh fetch upstream APKINDEX.

Option 2: Schedule periodic refresh

Nexus settings → Repositories → alpine → Connector → Maximum component age. Set to 1 hour (default может быть 24h+).

Trade-off: чаще updates → больше upstream traffic. Для CI критично — рекомендую 1h для APKINDEX, 7d для package files (immutable).

Option 3 (last resort): Self-host Alpine mirror

Если upstream Alpine periodically переписывает APKINDEX faster than Nexus refresh интервал — поднять dedicated mirror через rsync от tier-1 Alpine mirror каждые 30 мин.

---

Verification после fix

1. Test mirror consistency:

   APK_CC=$(curl -s https://repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/APKINDEX.tar.gz \
     | tar xz -O APKINDEX | grep -A1 '^P:ca-certificates$' | tail -1 | sed 's/^V://')
   FILE_CC=$(curl -s https://repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/ \
     | grep -o 'ca-certificates-[0-9]*-r[0-9]*' | head -1 | sed 's/ca-certificates-//')
   [ "$APK_CC" = "$FILE_CC" ] && echo "OK: APKINDEX и files matching ($APK_CC)" \
                              || echo "DRIFT: APKINDEX=$APK_CC files=$FILE_CC"
   

2. Revert node:22-alpine3.20 pin в .gitlab-ci.yml обратно на node:22-alpine:

   - image: repo.nstart.cloud/library/node:22-alpine3.20
   + image: repo.nstart.cloud/library/node:22-alpine
   

3. Run release pipeline — должна complete без 404 в apk add.

---

Если воспроизводится снова

Это второе incident за месяц (см. CI saga 2026-05-15 — !227 → !229 → !230). Нужен monitoring:

# Prometheus alert proposal
- alert: NexusAlpineMirrorDrift
  expr: |
    apk_index_version{repo="alpine-v3.23"} != apk_package_version{repo="alpine-v3.23", package="ca-certificates"}
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Nexus Alpine mirror v3.23 APKINDEX out of sync"
    runbook: "docs-internal/ops/nexus-alpine-mirror.md"

(Метрики писать через cron job + node_exporter textfile collector — Nexus сам не отдаёт мониторинг APKINDEX consistency.)