# Nexus Alpine Mirror — Operations Runbook

**Owner:** DevOps (cluster.265 infra)
**Status:** Active issue — APKINDEX рассинхронизирован с file storage на v3.23
**Related:** ordinis ci/apk fixes chain (!227, !229, !230)

---

## Problem

Nexus mirror на `repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/`:

| Что | Состояние |
|---|---|
| APKINDEX.tar.gz | Claims `ca-certificates-20251003-r0`, `libexpat-2.7.3-r0` |
| Filesystem | Только `ca-certificates-20260413-r0.apk`, `libexpat-2.7.5-r0.apk` |
| Result | `apk add` → HTTP 404 на 2 packages → fail |

Проверить актуальность:
```bash
curl -s https://repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/APKINDEX.tar.gz \
  | tar xz -C /tmp/ \
  && grep -A1 "^P:ca-certificates$" /tmp/APKINDEX | head -2

# Сравнить с file inventory:
curl -s "https://repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/" \
  | grep "ca-certificates-" | head -3
```

v3.20 mirror — **consistent** (APKINDEX и files matching). v3.21+ — нужно проверить per-version.

---

## Impact

- `node:22-alpine` (latest) uses Alpine v3.23 → broken `apk add` в CI
- Releases pipeline ломается на 2 stages (release job → semantic-release)
- `propose-prod-image-bump` (alpine:3.20 image) — **работает** благодаря consistent v3.20 mirror

## Workaround (active в проде)

ordinis `.gitlab-ci.yml` pins `release` job:
```yaml
release:
  image: repo.nstart.cloud/library/node:22-alpine3.20  # explicit v3.20 pin
```

Все Alpine jobs делают sed swap к internal mirror + `apk update`:
```yaml
- sed -i 's|https://dl-cdn.alpinelinux.org/alpine|https://repo.nstart.cloud/repository/alpine|g' /etc/apk/repositories
- apk update
- apk add --no-cache <packages>
```

---

## Fix (нужен DevOps action)

### Option 1 (recommended): Force Nexus APKINDEX rebuild

Nexus admin UI / API → `repository/alpine/v3.23/main/x86_64/APKINDEX.tar.gz`:
```bash
# Trigger rebuild через Nexus API (требует admin token)
curl -X POST -u admin:<token> \
  "https://repo.nstart.cloud/service/rest/v1/repositories/alpine/proxy/<repo>/invalidate-cache"
```

Или через UI: `Repositories → alpine-proxy → Invalidate Cache`.

После invalidate — следующий apk request триггерит fresh fetch upstream APKINDEX.

### Option 2: Schedule periodic refresh

Nexus settings → `Repositories → alpine → Connector → Maximum component age`. Set to `1 hour` (default может быть 24h+).

Trade-off: чаще updates → больше upstream traffic. Для CI критично — рекомендую `1h` для APKINDEX, `7d` для package files (immutable).

### Option 3 (last resort): Self-host Alpine mirror

Если upstream Alpine periodically переписывает APKINDEX faster than Nexus refresh интервал — поднять dedicated mirror через `rsync` от tier-1 Alpine mirror каждые 30 мин.

---

## Verification после fix

1. Test mirror consistency:
   ```bash
   APK_CC=$(curl -s https://repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/APKINDEX.tar.gz \
     | tar xz -O APKINDEX | grep -A1 '^P:ca-certificates$' | tail -1 | sed 's/^V://')
   FILE_CC=$(curl -s https://repo.nstart.cloud/repository/alpine/v3.23/main/x86_64/ \
     | grep -o 'ca-certificates-[0-9]*-r[0-9]*' | head -1 | sed 's/ca-certificates-//')
   [ "$APK_CC" = "$FILE_CC" ] && echo "OK: APKINDEX и files matching ($APK_CC)" \
                              || echo "DRIFT: APKINDEX=$APK_CC files=$FILE_CC"
   ```

2. Revert `node:22-alpine3.20` pin в `.gitlab-ci.yml` обратно на `node:22-alpine`:
   ```diff
   - image: repo.nstart.cloud/library/node:22-alpine3.20
   + image: repo.nstart.cloud/library/node:22-alpine
   ```

3. Run release pipeline — должна complete без 404 в apk add.

---

## Если воспроизводится снова

Это второе incident за месяц (см. CI saga 2026-05-15 — `!227 → !229 → !230`). Нужен monitoring:

```yaml
# Prometheus alert proposal
- alert: NexusAlpineMirrorDrift
  expr: |
    apk_index_version{repo="alpine-v3.23"} != apk_package_version{repo="alpine-v3.23", package="ca-certificates"}
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Nexus Alpine mirror v3.23 APKINDEX out of sync"
    runbook: "docs-internal/ops/nexus-alpine-mirror.md"
```

(Метрики писать через cron job + node_exporter textfile collector — Nexus сам не отдаёт мониторинг APKINDEX consistency.)