feat(topbar): «Что нового» button + 2026-05-12 state.md update

2026-05-12 17:05:24 +00:00
parent 58dd871808
commit a34c6d4c82
21 changed files with 1853 additions and 8 deletions
@@ -0,0 +1,415 @@
+# Design: Redis Projection FK Resolution
+
+**Author:** zimin.an
+**Date:** 2026-05-12
+**Status:** PROPOSED v2.1 (round 2 `/plan-eng-review` fixes applied)
+**Supersedes:**
+- v1 (2026-05-12 first draft, before eng review found `LineageIndexService` reuse opportunity)
+- v2 (2026-05-12 после round 1 review — major scope reduction + critical safety mitigations)
+**Sprint estimate:** **4-5 рабочих дней** (~30.5h CC+gstack)
+**Blockers:** none, рекомендация defer'a сохраняется — implement только когда конкретный dict упрётся в latency SLA
+
+---
+
+## TL;DR
+
+Сейчас Redis projection (writer module) пишет per-locale flattened JSON записи в Redis для dict'ей с `redis_projection_enabled=true`. **FK ссылки** (`x-references: "dict.field"`) хранятся как raw FK value (`satellite_type: "OPERATIONAL"`). Read-api делает N+1 lookup чтобы получить human label (`Действующий`).
+
+**Предложение:** при write resolve'ить FK label'ы и сохранять рядом с FK value в `_resolved` ключе. Per-FK opt-in через `x-resolve-label: true`. **Cascade invalidation reusing existing `LineageIndexService`** (no new Redis reverse index).
+
+**Win:** один Redis GET вместо N+1, latency 1ms вместо N×1ms (для записей с 5-10 FK).
+
+**Risk:** консистентность projection requires careful invalidation strategy, иначе projection stale до next write referencing'а dict'и. **Mitigations** добавлены в этой ревизии (см. § Critical safety).
+
+---
+
+## Изменения от v1 (после eng review)
+
+| # | v1 | v2 (now) | Why |
+|---|---|---|---|
+| 1 | Новый Redis reverse index `fk:<refDict>:<bk>` | **Reuse `LineageIndexService.findRecordDependents()`** | DRY: PG index уже existing, single source of truth |
+| 2 | "Subscribe to OutboxEvents" | **Extend `RecordEventListener` в projection-writer** | Корректное terminology — flow идёт через Kafka topics |
+| 3 | Fan-out 1000 records — pipeline и ОК | **Batch cap 500 + queue depth metric + alert** | Realistic worst case 50k (country dict) saturate'нет consumer |
+| 4 | «Eventual consistency few seconds» | **Explicit SLO: P95 < 60s для <10k, P95 < 300s для >10k** + `X-Projection-Updated-At` header в read-api | UX bug когда админ видит mixed state |
+| 5 | Cycle detection: «only direct FK» (in prose) | **Explicit constraint + test: nested FK resolution forbidden** | Защита от future «улучшений» |
+| 6 | Storage: 10k × 1KB = +300MB | **Fixed formula: × locale count × FK count = realistic +450MB-1.5GB** | Per-locale multiplier compound'ил, не учтён |
+| 7 | `@Cacheable` FkResolver на 5 min | **No Spring cache** — projection-writer уже триггерит cascade, кэш дал бы stale при write | Two caching layers were unsafe |
+| 8 | No test plan | **15-test plan with regression test, race test, E2E через Kafka, perf test** | Был только перечень "Integration tests" |
+| 9 | No rollback strategy | **Toggle off → eventual cleanup on next upsert** (не background job) | Меньше moving parts |
+| 10 | No kill-switch | **Global feature flag `ordinis.projection.fk-resolution.enabled` + per-dict + per-FK** | 3 уровня контроля для prod safety |
+
+**Round 2 fixes (v2 → v2.1):**
+
+| # | v2 | v2.1 (now) | Why |
+|---|---|---|---|
+| 11 | Kill switch precedence implicit | **Explicit precedence table** (write requires all 3; read requires global+dict only) | Operator clarity, lazy cleanup на per-FK toggle off |
+| 12 | `CascadeInvalidator` triggers via schema scan per event | **Step 1.5: fkTargetDicts cache + SchemaPublished invalidation** | High-RPS dicts не платят schema lookup на каждый message |
+| 13 | Bundle isolation lost в Open Question #3 | **Step 1.6: SchemaValidator reject cross-bundle x-references** | Multi-tenancy invariant защищён valid'ом |
+| 14 | `_meta.updatedAt` ambiguous (cascade vs direct) | **Explicit max(direct, cascade) policy** | Silent UX bug: header lying about staleness |
+| 15 | Test plan 15 cases | **20 cases** (+ kill switch matrix, cache invalidation, cross-bundle, `_meta.updatedAt` timing, storage cap precision) | Coverage diagram gaps closed |
+
+---
+
+## Текущее состояние
+
+### Что есть
+
+```
+                         RecordCreated/Updated event
+                                    │
+                                    ▼
+                         Kafka topic (3 scopes)
+                                    │
+                                    ▼
+   ordinis-projection-writer/RecordEventListener
+                                    │
+                                    ▼
+                      ProjectionWriter.upsert()
+                                    │
+                ┌───────────────────┼───────────────────┐
+                ▼                   ▼                   ▼
+    flatten(per-locale)      raw key SET         dictionaryIndex SET ADD
+                │
+                ▼
+       Redis SET key=record(...,locale)
+```
+
+`flatten()` обрабатывает **только** `x-localized` поля. FK поля проходят как есть.
+
+### Что хочется
+
+При write проекции добавляется `_resolved` ключ с pre-fetched label'ами:
+
+```json
+{
+  "businessKey": "ISS",
+  "satellite_type": "OPERATIONAL",
+  "country": "RU",
+  "_resolved": {
+    "satellite_type": "Действующий",
+    "country": "Россия"
+  }
+}
+```
+
+Read-api сразу отдаёт нужный label без второго round-trip.
+
+---
+
+## Архитектура v2
+
+### Component diagram
+
+```
+   Kafka event: RecordCreated{dict=spacecraft, bk=ISS}
+                                    │
+                                    ▼
+                      RecordEventListener (existing)
+                                    │
+                                    ▼
+                      ProjectionWriter.upsert()
+                                    │
+                ┌───────────────────┼───────────────────┐
+                ▼                   ▼                   ▼
+        flatten(...)         FkResolver        raw key SET (как сейчас)
+              │                  │
+              │                  │ для каждого x-resolve-label поля:
+              │                  │   - PG read replica lookup
+              │                  │   - inject _resolved.<field>
+              │                  ▼
+              │           ┌──────────────┐
+              │           │   PG read    │
+              │           │   replica    │ (no cache layer!)
+              │           └──────────────┘
+              ▼
+       Redis SET key=record(...,locale)
+
+
+   Kafka event: RecordUpdated{dict=satellite_types, bk=OPERATIONAL}
+                                    │
+                                    ▼
+                      RecordEventListener (existing)
+                                    │
+                                    ▼
+                  CascadeInvalidator (NEW component)
+                                    │
+                                    ▼
+        LineageIndexService.findRecordDependents()  ← REUSE
+                                    │
+                                    ▼
+              paged list of (source_dict, source_bk)
+                                    │
+                                    ▼
+                  enqueue invalidation batch (cap 500)
+                                    │
+                                    ▼
+              for each batch: re-fetch from PG → ProjectionWriter.upsert()
+                                    │
+                                    ▼
+                Redis pipelined batch write (Spring executePipelined)
+```
+
+### Phase A: write-side FK resolver (~2 дня)
+
+`FkResolver` service в `ordinis-projection-writer`:
+
+```java
+@Component
+public class FkResolver {
+  private final DictionaryRecordRepository recordRepo;   // PG read replica
+  private final DictionaryDefinitionRepository defRepo;
+
+  /**
+   * Resolve FK value → human label per locale.
+   * Performance: single PG read query per (refDict, fkValue) — sub-ms на read replica.
+   * No caching by design (см. eng review #2 — stale read window).
+   */
+  public Optional<String> resolveLabel(
+      String refDict,        // "satellite_types"
+      String refField,       // "code" — default businessKey
+      String fkValue,        // "OPERATIONAL"
+      String locale          // "ru"
+  ) { ... }
+
+  /** Batch resolve для N FK fields одной записи (single PG query через IN clause). */
+  public Map<String, String> resolveBatch(
+      List<FkRequest> requests,
+      String locale
+  ) { ... }
+}
+```
+
+Update `ProjectionWriter.flatten()`:
+- Walk schema.properties, для каждого FK поля с `x-resolve-label: true`:
+  - Call `resolveBatch` (single PG query для всех FK одной записи)
+  - Inject `_resolved.<field>` JSON
+- **Hard constraint:** не рекурсировать в nested objects — direct fields only
+
+### Phase B: cascade invalidation (~2 дня) — **reuse LineageIndexService**
+
+New `CascadeInvalidator` в `ordinis-projection-writer`:
+
+```java
+@Component
+public class CascadeInvalidator {
+  private final LineageIndexService lineageIndex;       // REUSE
+  private final ProjectionWriter writer;
+  private final DictionaryRecordRepository recordRepo;
+  private static final int BATCH_SIZE_CAP = 500;        // hard limit
+
+  /** Triggered RecordEventListener'ом когда updated dict — потенциальный FK target. */
+  public void onSourceRecordUpdate(String refDict, String businessKey) {
+    int page = 0;
+    while (true) {
+      Page<RecordDependent> deps = lineageIndex.findRecordDependents(
+          refDict, businessKey, allScopes(), PageRequest.of(page, BATCH_SIZE_CAP));
+      if (deps.isEmpty()) break;
+
+      // Re-fetch dependents from PG + rewrite projection (pipelined)
+      writer.batchRewriteProjections(deps.getContent());
+
+      cascadeInvalidationCounter.increment(deps.getNumberOfElements());
+      if (!deps.hasNext()) break;
+      page++;
+
+      // Backpressure если queue depth высокий
+      if (cascadeQueueDepthGauge.get() > QUEUE_DEPTH_ALERT) {
+        Thread.sleep(THROTTLE_MS);
+      }
+    }
+  }
+}
+```
+
+Hook в существующий `RecordEventListener`:
+- На `RecordUpdated{dict=X}` event — после обычной upsert(), check'аем X **через cached `fkTargetDicts` set** (step 1.5). Schema-level dependency map immutable между SchemaPublished events; cache invalidate'ится по этому event'у. Это избегает `LineageIndexService.findSchemaDependents()` query на каждом message — критично для high-RPS dicts.
+- Если X ∈ fkTargetDicts → вызываем `CascadeInvalidator.onSourceRecordUpdate(X, businessKey)`
+
+### Phase C: backfill + ops (~1 день)
+
+CLI endpoint `POST /api/v1/admin/projections/{dict}/backfill`:
+- Paginated iterate (batch=500, sleep=100ms между batch'ами)
+- Per-page commit chunk — recoverable если interrupted
+- Lock через advisory lock в PG (one backfill per dict at a time)
+- Progress reporting через outbox event stream
+
+### Phase D: observability + flags
+
+3-tier kill switch:
+1. **Global** — `ordinis.projection.fk-resolution.enabled` (env var)
+2. **Per-dict** — `DictionaryDefinition.fkResolutionEnabled` (existing field паттерн, новая колонка)
+3. **Per-FK** — schema annotation `x-resolve-label: true` (default false)
+
+**Precedence (explicit per round 2 review):**
+
+| Layer | Write path (resolve & inject `_resolved`) | Read-api (return `_resolved` к caller'у) |
+|---|---|---|
+| Global=false | ❌ skip | ❌ ignore `_resolved` even if present (stale residue from before toggle) |
+| Global=true, Dict=false | ❌ skip | ❌ ignore (per-dict opt-out wins over residual data) |
+| Global=true, Dict=true, FK=false | ❌ skip для этого поля | ✅ return whatever's already в `_resolved` (no harm — поле там не появится) |
+| Global=true, Dict=true, FK=true | ✅ resolve & write | ✅ return `_resolved.<field>` |
+
+Logic: **write requires all three true**; **read requires global + dict** (FK granularity не нужна на read — если backend перестал писать поле, `_resolved.<field>` natural выпадает при следующем upsert). Это значит turning off per-FK toggle = lazy cleanup. Turning off per-dict = immediate hide. Turning off global = panic kill для всего projection FK behavior.
+
+**Metrics (Prometheus):**
+- `ordinis_projection_fk_resolved_total{dict, fk_field}`
+- `ordinis_projection_fk_resolve_miss_total{dict, fk_field, reason="not_accessible|not_found|locale_missing"}`
+- `ordinis_projection_cascade_invalidation_total{trigger_dict}`
+- `ordinis_projection_cascade_queue_depth` (gauge)
+- `ordinis_projection_fk_resolve_duration_seconds` (histogram)
+- `ordinis_projection_storage_resolved_bytes_total{dict}` (gauge)
+
+**SLO targets:**
+- P95 FK resolution latency: < 5ms (PG read replica baseline)
+- P95 cascade invalidation:
+  - < 60s для cascade size ≤ 10k dependents
+  - < 300s для cascade size 10k-50k
+  - > 50k cascade → page on-call (likely operator error)
+- Projection eventual consistency window: max 300s
+
+**Read-api integration:**
+- Response header `X-Projection-Updated-At: 2026-05-12T17:30:00Z` (last upsert time per record)
+- `staleness=` query param для force-fresh read через PG fallback (debugging)
+
+**`_meta.updatedAt` policy (per round 2 review — silent UX bug fix):**
+
+Stored в projection JSON как `_meta.updatedAt` (sibling `_resolved`):
+- **Updated on direct upsert** (RecordCreated/Updated event для этой записи)
+- **Updated on cascade rewrite** (CascadeInvalidator пишет проекцию после изменения FK target)
+- Read-api header = `max(direct_upsert_time, cascade_rewrite_time)` = the **`_meta.updatedAt` value** as-stored (cascade overwrites if newer)
+
+Why max'om: иначе админ видит «projection updated 5 минут назад» а на самом деле cascade обновил labels 3 секунды назад — silent UX bug когда юзер ждёт что прочтёт fresh данные. Stored timestamp всегда **точка последней мутации projection'а**, не direct write.
+
+Storage cost: 24 bytes per record × 3 locales = 72 bytes overhead. Negligible.
+
+---
+
+## Critical safety (added in v2)
+
+### Race conditions: concurrent target update during dependent write
+
+**Scenario:** Запись A пишется в момент когда B (FK target) сам обновляется.
+
+**Mitigation:** `FkResolver` читает с PG **after** projection write — если B меняется между resolution и Redis SET, мы пишем stale label. Затем cascade от B's update подхватит A и rewrite projection → eventual consistency.
+
+**Test:** `ConcurrentFkUpdateTest` — два потока, поток1 пишет A, поток2 update B, проверяем что через ≤30s projection A содержит latest label B.
+
+### Kafka consumer lag во время массивного cascade
+
+**Scenario:** Update country=RU → 50k spacecrafts инвалидируются.
+
+**Mitigation:**
+1. Batch cap **500** dependents за раз
+2. `cascadeQueueDepthGauge` exposed → alert thresholds (warn @ 1000, crit @ 5000)
+3. Throttle между batch'ами если queue depth превышает
+4. Async dispatch — cascade work не в main listener thread, отдельный `@Async` executor с bounded queue
+
+**Test:** `CascadeBackpressureTest` — load 10k dependents, verify Kafka consumer lag не превышает 60s.
+
+### Redis memory pressure
+
+**Scenario:** Enable flag для крупного dict'a → проекции вырастут 2-3×.
+
+**Mitigation:**
+1. Metric `ordinis_projection_storage_resolved_bytes_total{dict}` + alert на 80% Redis memory cap
+2. Per-record max size check — если `_resolved` blob > 50KB, skip resolution + log warning (не падать)
+3. Operational runbook: «disable per-dict flag → wait for natural eviction (24h TTL?) → cleanup»
+
+### Rollback path
+
+Если flag toggled off:
+- Existing `_resolved` keys остаются stale, но **harmless** — read-api ignore'ит `_resolved` когда flag=false
+- При next upsert каждой записи `_resolved` automatically dropped (write replaces full value)
+- Если нужен immediate cleanup — CLI `POST /admin/projections/{dict}/strip-resolved` (~1 час имплементации)
+
+---
+
+## Open questions
+
+1. **PG read replica lag.** `FkResolver` читает с replica для performance, но replica может отставать на seconds под write load. Если target dict обновлён 100ms назад, FkResolver получит stale label, cascade подхватит ~30s. Acceptable. Документировать.
+
+2. **Scope-hide handling.** Если FK target dict не доступен caller'у — projection write идёт как **system user** (не user-scope). Resolution всё равно происходит, но read-api позже filter'нет `_resolved` если caller scope не допускает target dict. Implementation: `FkResolver` использует privileged read, read-api проверяет access.
+
+3. **Bundle isolation.** В multi-bundle setup'е (cuod, altum, etc.) — FK может пересекать bundle boundaries? Recommendation: запретить, validation в schema editor (`x-references` поле target dict должен быть в том же bundle).
+
+4. **Storage cap.** Hard limit на per-record `_resolved` blob — 50KB? Это значит до ~25 FK fields × 3 locales × ~600 chars label = 45KB. Достаточно для всех realistic case'ов.
+
+---
+
+## Альтернатива: skip FK resolution, use read-api JOIN (unchanged from v1)
+
+Вместо pre-resolve'инга при write — read-api делает JOIN на читающей стороне (через PG или client-side). Pro: simpler, no cascade. Con: N+1 на каждое чтение, ровно то что мы хотим избежать.
+
+Если RPS не упёрся в predisposed, можно отложить весь FK projection. Прагматичное правило: **включать FK resolution только когда конкретный dict упёрся в latency SLA**. Default off навсегда.
+
+---
+
+## Implementation plan (revised)
+
+| Step | Effort (CC+gstack) | Notes |
+|---|---|---|
+| 1. Schema annotation `x-resolve-label` + JSON Schema validator | 1h | Update SchemaValidator, fail-fast on non-FK field |
+| 1.5. **fkTargetDicts cache** + SchemaPublished invalidation | 1h | Pre-compute set of dicts that являются FK target (via `LineageIndexService.findSchemaDependents` inversion). Avoid schema scan на каждый RecordUpdated event в `RecordEventListener` |
+| 1.6. **Bundle isolation enforcement** в `SchemaValidator` | 0.5h | Reject `x-references: "<targetDict>.<field>"` если target_dict в другом bundle. Single-bundle invariant — иначе multi-tenancy ломается |
+| 2. Per-dict `fk_resolution_enabled` column + migration | 1h | Liquibase, Dictionary CRUD. Default `false` для всех existing dicts (backward-compat). |
+| 3. `FkResolver` service + batch resolve method | 3h | Single PG query per record (IN clause) |
+| 4. Update `ProjectionWriter.flatten()` для `_resolved` injection | 2h | Walk schema, hard constraint on nested |
+| 5. `CascadeInvalidator` reusing `LineageIndexService` | 4h | Batch cap, queue depth gauge, async |
+| 6. `RecordEventListener` hook into `CascadeInvalidator` | 1h | Detect target dict from schemas index |
+| 7. Backfill CLI endpoint + advisory lock | 2h | Paginated, idempotent |
+| 8. Metrics + SLO alerting config (Grafana panels) | 2h | 6 new metrics, 3 alert rules |
+| 9. Read-api `X-Projection-Updated-At` header | 1h | Track via `_meta.updatedAt` в projection JSON |
+| 10. Integration tests (15 cases per coverage diagram) | 8h | testcontainers Postgres+Redis+Kafka |
+| 11. Frontend: schema editor checkbox per FK field | 2h | `DictionaryEditorDialog`, validation |
+| 12. Docs (ops runbook, schema annotation guide) | 2h | docs/ops/projection-fk-resolution.md |
+| **Total** | **~30.5h (4-5d)** | within revised budget (+1.5h после round 2 review) |
+
+---
+
+## Test plan (added in v2 per eng review)
+
+Minimum 15 integration tests, testcontainers Postgres + Redis + Kafka:
+
+| # | Test | Type | Critical? |
+|---|---|---|---|
+| 1 | Happy path: 3 FK fields, all resolve | unit | — |
+| 2 | One FK miss (orphan) → omitted from `_resolved` | unit | — |
+| 3 | Multi-locale resolution per locale array | unit | — |
+| 4 | Locale fallback (ru missing → defaultLocale en) | unit | — |
+| 5 | Schema validation: `x-resolve-label` on non-FK → error | unit | — |
+| 6 | **Regression**: write WITHOUT flag works как раньше | unit | **YES** |
+| 7 | Cascade fires on RecordUpdated of FK target | integration | — |
+| 8 | Cascade batch cap 500 enforced | integration | — |
+| 9 | Cascade queue depth metric increments | integration | — |
+| 10 | Concurrent update race → eventual consistency ≤30s | integration | **CRITICAL** |
+| 11 | Backpressure: 10k dependents → consumer lag <60s | integration | **CRITICAL** |
+| 12 | E2E через Kafka real flow: PUT → cascade → Redis read | E2E | — |
+| 13 | Backfill CLI idempotent (re-run same result) | integration | — |
+| 14 | Toggle off → next upsert removes `_resolved` | integration | — |
+| 15 | Storage cap (50KB) — large record skips resolution + warns | integration | — |
+| 16 | **Kill switch precedence matrix** — все 8 комбинаций global/dict/FK booleans (truth table, parameterized test): write + read behavior matches table в § Phase D | integration | **YES** |
+| 17 | **fkTargetDicts cache invalidation** — SchemaPublished event дропает schema-level dependency cache, next RecordUpdated пересчитывает | integration | — |
+| 18 | **Cross-bundle FK rejected** — `SchemaValidator` reject'нет `x-references` указывающий на dict из другого bundle | unit | — |
+| 19 | **`_meta.updatedAt` cascade vs direct timing** — cascade rewrite обновляет timestamp; direct upsert после cascade перезаписывает; header возвращает max | integration | — |
+| 20 | **Storage cap precision** — 50KB cap применяется per-record (NOT per-field); record с 49KB raw + 5KB `_resolved` = total 54KB → resolved skipped, raw written | integration | — |
+
+---
+
+## Recommendation (unchanged from v1)
+
+**Defer until after v2.12.0 prod stable + 2 weeks dogfooding.** Текущий read path handles RPS, no urgency. Activate per-FK flag только когда first dict bumps into latency SLA. Build full pipeline только if 3+ dict'ам нужно.
+
+**Next step (если decide go):** этот revised doc → `/plan-eng-review` round 2 → if CLEAR → CEO approval → sprint allocation.
+
+---
+
+## GSTACK REVIEW REPORT
+
+| Review | Trigger | Why | Runs | Status | Findings |
+|--------|---------|-----|------|--------|----------|
+| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 2 (v1 + v2) | 🟢 CLEAR | v1: 11 issues + 3 critical → all closed v2. v2 round 2: 5 minor + 1 silent UX bug → all closed v2.1 |
+| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | — | — |
+| Design Review | `/plan-design-review` | UI/UX gaps | 0 | n/a | Skip — backend feature |
+
+**UNRESOLVED:** 0
+**VERDICT:** 🟢 **CLEAR after v2.1 fixes.** Ready для implementation **когда** придёт время. Defer recommendation сохраняется: implement только после v2.12.0 prod stable + 2 weeks dogfooding + первый dict упрётся в latency SLA.