Zimin A.N.
bcfb07f547
Чёткое разделение audiences:
- docs/ — consumer-facing only, Diplodoc-built. Public contract для
интеграторов (Альтум, BI, third-party).
- docs-internal/ — operator runbooks, status snapshots, legacy tech pages.
НЕ в Diplodoc build, НЕ публикуются.
Изменения:
- Move docs/{ops,status,tech} → docs-internal/{ops,status,tech}
- docs/toc.yaml: убраны секции Operations, Архитектура (legacy), Status
- docs/index.md: убран tab "оператор / on-call", focus на integrators
- docs/README.md: rewritten — scope только integrator guide, editorial
guidelines (no leak internal architecture, stable API contract)
- .yfm: убран явный ignore status/* (теперь не нужно — папка переехала)
- docs-internal/README.md: index для внутренней документации с rationale
разделения
Scrubbing implementation details из Diplodoc страниц:
- events.md: убраны Strimzi/cluster.142 references, OutboxPoller детали,
internal alert names (OrdinisOutboxLagHigh, OrdinisOutboxDLQNonEmpty),
metric names (nsi_outbox_*). Заменены на consumer-observable contract.
- webhooks.md: убраны metric names (nsi_webhook_ssrf_rejected_total),
alert table (OrdinisWebhookHighFailureRate и т.д.). Retry policy
переписана на user-side perspective.
- errors.md: убран alert name OrdinisReadApiErrorBudgetBurnFast, Tempo
endpoint URL, internal connection details.
- x-references.md: OrphanReferenceScanner → general "Ordinis периодически
детектирует". Убраны metric/alert names. Cascade roadmap без link на
internal design doc.
- auth.md: убран file path ScopeContext.java.
- bundle-cuod.md: убран Maven module path.
- caching.md: cascade design link заменён на text reference.
CI:
- .gitlab-ci.yml: новый job build-docs (stage: build) — Node 20 alpine,
pnpm install --frozen-lockfile + pnpm build, artifact docs/_dist на
неделю. Triggers: auto на docs/**, либо manual web. Готов к follow-up
pages-deploy job когда host решён.
- Новый pattern .docs-changes для rules.
Build verified clean: 13 HTML pages, 6.9M, no broken links/refs.
175 lines
12 KiB
Markdown
175 lines
12 KiB
Markdown
# Состояние разработки НСИ (Ordinis ЦУОД ОДХ)
|
||
|
||
**Версия:** v1.0.1 (post-cutover hardening: SPOF closed, monitoring stack ready, image arch incident resolved)
|
||
**Дата:** 2026-05-07 (evening update)
|
||
**Owner:** zimin.an@nstart.space
|
||
**Окружения:**
|
||
- **prod** (192.168.100.142 / cluster.142, ✅ LIVE с Альтум traffic, blue/green cutover завершён)
|
||
- **staging** (192.168.100.161 / cluster.265, ✅ STABLE — теперь dev environment, не consumer-facing)
|
||
**Предыдущий snapshot:** [2026-05-06-state.md](./2026-05-06-state.md) (v0.9, prod hibernated)
|
||
|
||
---
|
||
|
||
## 0. Что изменилось с v0.9
|
||
|
||
### Production stand-up на cluster.142 (Phase 1-5 done)
|
||
|
||
**Phase 1 — Postgres prod stand-up:**
|
||
- Discovery: cluster.142 НЕ hibernated (state.md устарел — k8s API v1.33.7 жив, ordinis-prod ns с Helm rev 17 deployed 4 дня назад, всё scaled to 0)
|
||
- CNPG cluster `ordinis-postgres` healthy, 50Gi local-path PVC
|
||
- Custom image `postgis-barman:18.3-3.6.2-1` (PostGIS 18 + barman-cli-cloud) для CNPG backups — официальный `cnpg/postgis:18-standard-trixie` НЕ имеет barman tools
|
||
- barmanObjectStore → `s3://altum-backups/ordinis-postgres` через `geoportal-prod-minio:9000` (re-use bucket с prefix isolation)
|
||
- ScheduledBackup: daily 02:30 UTC + weekly Sun 04:00 UTC
|
||
- Initial backup landed, PITR window started
|
||
|
||
**Phase 2 — Vault re-init:**
|
||
- Original Vault sealed после restart 17h, unseal key потерян
|
||
- Re-init с wipe `data-vault-0` PVC → init -key-shares=1 -key-threshold=1
|
||
- Setup playbook: `kubernetes` auth, policy `ordinis-app`, 4 roles (per-SA binding)
|
||
- 6 secret paths populated из existing k8s secrets (CNPG postgres, Strimzi KafkaUsers, Redis)
|
||
- Scripts: `ordinis-infra/scripts/vault/{setup,populate}.sh` + README
|
||
|
||
**Phase 3 — Kafka cluster recovery:**
|
||
- `ordinis-kafka` Strimzi cluster CR не было в namespace (только orphaned KafkaUsers + 1 service)
|
||
- Apply `ordinis-infra/k8s/kafka/cluster-prod.yaml` (KafkaNodePool dual + Kafka KRaft + 3 KafkaTopics)
|
||
- 1 broker, single-instance, SCRAM-SHA-512 auth, 20Gi local-path PVC
|
||
- Strimzi operator уже watches ordinis-prod (STRIMZI_NAMESPACE=kafka-test,ordinis-prod)
|
||
- Cluster Ready через 90s после apply
|
||
|
||
**Phase 4 — App deploy:**
|
||
- `config-server`, `ordinis-redis-0` scaled up (были 0/0)
|
||
- Helm upgrade `ordinis` rev 17 → 22 на `image.tag=main-4252ad93`
|
||
- Replicas bumped: writer 1→2, readApi 1→2 (HA для consumer-facing)
|
||
- 9 pods Running, /actuator/health → 200, /api/v1/dictionaries → 200 + 40 dicts seeded
|
||
|
||
**Phase 5 — Альтум cutover:**
|
||
- Blue/green phased shift через Альтум config (10% → 50% → 100%)
|
||
- staging-as-prod decommissioned для production traffic
|
||
- staging cluster.265 теперь чисто dev environment
|
||
|
||
### 0.1. Post-cutover hardening (evening 2026-05-07)
|
||
|
||
**Артефакты добавлены:**
|
||
- `k8s/benchmark/k6-job-prod.yaml` — load test для prod (Service port :80 fix)
|
||
- `k8s/benchmark/synthetic-smoke-prod.yaml` — one-shot smoke (auth + 3 endpoints, exit code = pass/fail)
|
||
- `k8s/benchmark/synthetic-smoke-cronjob-prod.yaml` — continuous monitoring (*/30 min)
|
||
- `k8s/kafka/topic-orphan-fix.sh` — janitorial script для KafkaTopic READY=False (✅ applied 2026-05-07 evening — все 3 topics unmanaged, alerts gone)
|
||
- prometheus-rules group `ordinis.synthetic` (2 alerts: SmokeFailed/Stale, conditional на `prometheusRule.synthetic.enabled`)
|
||
- `.gitlab-ci.yml` job `build-postgis-barman` (kaniko + REGISTRY_USER/REGISTRY auth, currently 401 — known issue, manual push fallback)
|
||
|
||
**Инцидент: postgis-barman image swap** (08:30-22:00 MSK):
|
||
1. Custom image `:18.3-3.6.2-1` запушен в `repo.nstart.cloud` с mac M-series **без `--platform linux/amd64`** → arm64 layers попали в registry
|
||
2. Catalog patched на repo.nstart.cloud → CNPG re-roll → `exec format error` → CrashLoopBackOff
|
||
3. Rollback catalog на staging URL → recovered
|
||
4. Re-push amd64 → но **kubelet cache на srvaltum01** удерживал arm64 layer под тем же тегом → retry catalog swap снова exec format error
|
||
5. Bump rev `:1` → `:2` → kubelet вынужден pull → cache miss → новый amd64 layer → success
|
||
6. Pod stuck Terminating ~14 мин (CNPG `terminationGracePeriodSeconds: 1800`) → `kubectl delete pod --force --grace-period=0` для recovery acceleration
|
||
|
||
**Lessons learned (закреплены в `docker/postgis-barman/README.md`):**
|
||
- `docker buildx --platform linux/amd64 --push` обязательно когда build с M-series mac
|
||
- Bump rev tag при cache invalidation needs (тот же тег + новые слои = kubelet skip)
|
||
- `terminationGracePeriodSeconds=1800` означает что graceful pod delete может висеть до 30 минут
|
||
- Verify image arch через registry API (`config.architecture`) ДО catalog swap
|
||
|
||
**Production verify (post-incident):**
|
||
- Cluster phase: "Cluster in healthy state"
|
||
- status.image: `repo.nstart.cloud/terravault/postgis-barman:18.3-3.6.2-2`
|
||
- imageID digest: `sha256:1b4b074d...` (verified amd64 manifest)
|
||
- All 11 pods Running, postgres restartCount=0, lastSuccessfulBackup preserved
|
||
|
||
**SPOF на staging registry — closed.** Если `registry.k8s.265.nstart.local` упадёт, prod не пострадает.
|
||
|
||
### Что закоммичено
|
||
|
||
**ordinis-infra (4 commits сегодня):**
|
||
- `cluster-prod.yaml` aligned с geoportal pattern (local-path, internal MinIO, single-instance) — было overcomplicated с Longhorn/Yandex S3
|
||
- `values-prod.yaml`: writer=2, readApi=2, host `ordinis.k8s.264.nstart.cloud`, `selfsigned-cluster-issuer`
|
||
- `prometheus-rules.yaml`: 4 CNPG HA alerts (conditional `cnpgHa.enabled` flag, off для single-instance)
|
||
- `k8s/kafka/cluster-prod.yaml` (NEW): Kafka CR + NodePool + 3 Topics для recovery
|
||
- `scripts/vault/{setup,populate}.sh + README.md` (NEW): playbook bootstrap
|
||
|
||
**ordinis (8 commits сегодня — задеплоены в prod):**
|
||
- `c7a917d` test(webhook): unit-test WebhookTestPingService — 7 кейсов
|
||
- `ac93151` test(webhook): re-enable WebhookE2ETest через no-op scheduler + manual ticks
|
||
- `92507ab` feat(audit): URL persistence фильтров + chip-list активных
|
||
- `5941207` feat(records): bulk close — multi-select toolbar + per-record транзакция
|
||
- `640a633` test(records): unit-test BulkRecordService — 10 кейсов
|
||
- `66a4f13` test(geo): e2e auto-derive geometry из data.lat/lon — 6 кейсов
|
||
- `4252ad9` feat(records): bulk export CSV выбранных + pagination
|
||
- + 23 новых теста (7 webhook + 10 bulk + 6 e2e)
|
||
|
||
---
|
||
|
||
## 1. Перечень требований
|
||
|
||
### 1.1. Реализовано (production-ready, LIVE на prod cluster.142)
|
||
|
||
Та же таблица что v0.9 + новые пункты:
|
||
|
||
| # | Требование | Статус |
|
||
|----|---------------------------------------------------------------|--------|
|
||
| 1-32 | (см. v0.9, всё уже было ready) | ✅ |
|
||
| 33 | Production deployment на real cluster (cluster.142) | ✅ NEW |
|
||
| 34 | Bulk close записей с multi-select UI | ✅ NEW |
|
||
| 35 | Bulk CSV export выбранных записей | ✅ NEW |
|
||
| 36 | Audit log URL-persistent filters + chip-list | ✅ NEW |
|
||
| 37 | Pagination для records list (50/page client-side) | ✅ NEW |
|
||
| 38 | WebhookE2ETest stable на CI (no-op scheduler + manual ticks) | ✅ NEW |
|
||
| 39 | Auto-derive POINT geometry из data.lat/lon (3 пути backend + Liquibase backfill 0015) | ✅ |
|
||
| 40 | CNPG continuous WAL archive + scheduled backups в external S3 | ✅ NEW |
|
||
|
||
### 1.2. В процессе / nice-to-have
|
||
|
||
- ~~KafkaTopics READY=False~~ ✅ **APPLIED 2026-05-07 evening** через `k8s/kafka/topic-orphan-fix.sh`. Все 3 topics (.public/.internal/.restricted) теперь annotate'нуты `strimzi.io/managed=false` — Strimzi user-operator больше не пытается reconcile (ошибка "Decreasing partitions not supported" гасла). Topics в брокере продолжают жить, projection-writer connected. Alerts на READY=False больше не срабатывают.
|
||
- Vault SASL passwords могут быть stale если Strimzi user-operator regenerated после Kafka cluster recreate — re-run `scripts/vault/populate.sh`
|
||
- ~~`postgis-barman` image SPOF на staging registry~~ ✅ **CLOSED 2026-05-07 evening:** image mirrored в `repo.nstart.cloud/terravault/postgis-barman:18.3-3.6.2-2`, prod cluster.142 pull'ит оттуда. См. секцию 0.1 ниже про инцидент.
|
||
- k6 benchmark на prod не прогнан (smoke 50 RPS / ramp 1k / sustain 5m готов в `k8s/benchmark/k6-job-prod.yaml`)
|
||
- Synthetic smoke + alerts ready (`k8s/benchmark/synthetic-smoke-cronjob-prod.yaml` + prometheus-rules group `ordinis.synthetic`). **Pre-flight partial:** `k6-tests` ConfigMap создан в `ordinis-prod` (test script). Осталось: создать `k6-keycloak-creds` Secret (требует Keycloak SA `ordinis-bench` client_secret — на стороне Keycloak admin), apply CronJob, helm upgrade с values-prod для активации alerts
|
||
- Real ЦУОД data integration (отдельный epic после prod soak)
|
||
|
||
### 1.3. Roadmap (отложено)
|
||
|
||
- **Approval workflow v2** (draft → review → publish) — design doc сохранён в `~/.gstack/projects/claude/zimin-main-design-approval-workflow-v2-20260507-121632.md`. Defer'нут до prod LIVE + observed compliance needs. Re-evaluate через 1-2 месяца после Альтум stable.
|
||
- **True HA Postgres** (3-instance + Longhorn + external Yandex S3) — отдельный future epic для multi-node cluster, не для cluster.142
|
||
- **Dictionary relationships v2** — combine lineage / dependency graph + cross-dictionary FK cascade rules в один epic. Design DRAFT в `~/.gstack/projects/claude/zimin-main-design-dict-relationships-v2-20260507-230237.md` (3-5 нед CC, 4 phases, 4 D-questions для plan-eng-review). Pre-code assignment: Альтум consumer behavior на cascade events.
|
||
- Data quality rules engine
|
||
- Federation MDM-кластеров (master/replica)
|
||
- Schema registry (Apicurio/Confluent) для Kafka
|
||
|
||
---
|
||
|
||
## 2. Архитектура (без изменений с v0.9, кроме prod)
|
||
|
||
```
|
||
┌─────────────────────┐ ┌──────────────────────┐ ┌─────────────┐
|
||
│ Альтум (real users) │ ──→│ ordinis.k8s.264 │ ──→│ Postgres │
|
||
│ │ │ (cluster.142 prod) │ │ (CNPG 1× │
|
||
│ │ │ • read-api ×2 │ │ + WAL S3 │
|
||
│ │ │ • app ×2 (writer) │ │ + daily │
|
||
│ │ │ • projection-writer │ │ backups) │
|
||
│ │ │ • admin-ui ×1 │ │ │
|
||
│ │ │ • Kafka 1× + 3 topics │ │
|
||
│ │ │ • Redis 1× │ │ │
|
||
│ │ │ • Vault 1× │ │ │
|
||
│ │ └──────────────────────┘ └─────────────┘
|
||
└─────────────────────┘
|
||
┌──────────────────────┐
|
||
│ staging cluster.265 │
|
||
│ (DEV ENVIRONMENT) │
|
||
└──────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Operational notes
|
||
|
||
**Backup verify:** `barman-cloud-backup-list --endpoint-url http://geoportal-prod-minio.geoportal-production:9000 --cloud-provider aws-s3 s3://altum-backups/ordinis-postgres ordinis-postgres`
|
||
|
||
**Vault sealed после restart:** `kubectl exec -n vault vault-0 -- vault operator unseal <KEY>` (key хранится у operator'а в 1Password / GitLab vars).
|
||
|
||
**Rollback план:** DNS TTL 60s — при критичных issues Альтум переключает обратно на staging через config flag.
|
||
|
||
---
|
||
|
||
**Email:** zimin.an@nstart.space (НЕ gmail)
|