Files
mdm-ordinis/docs/design/redis-fk-projection.md
T
2026-05-12 14:44:55 +00:00

416 lines
24 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Design: Redis Projection FK Resolution
**Author:** zimin.an
**Date:** 2026-05-12
**Status:** PROPOSED v2.1 (round 2 `/plan-eng-review` fixes applied)
**Supersedes:**
- v1 (2026-05-12 first draft, before eng review found `LineageIndexService` reuse opportunity)
- v2 (2026-05-12 после round 1 review — major scope reduction + critical safety mitigations)
**Sprint estimate:** **4-5 рабочих дней** (~30.5h CC+gstack)
**Blockers:** none, рекомендация defer'a сохраняется — implement только когда конкретный dict упрётся в latency SLA
---
## TL;DR
Сейчас Redis projection (writer module) пишет per-locale flattened JSON записи в Redis для dict'ей с `redis_projection_enabled=true`. **FK ссылки** (`x-references: "dict.field"`) хранятся как raw FK value (`satellite_type: "OPERATIONAL"`). Read-api делает N+1 lookup чтобы получить human label (`Действующий`).
**Предложение:** при write resolve'ить FK label'ы и сохранять рядом с FK value в `_resolved` ключе. Per-FK opt-in через `x-resolve-label: true`. **Cascade invalidation reusing existing `LineageIndexService`** (no new Redis reverse index).
**Win:** один Redis GET вместо N+1, latency 1ms вместо N×1ms (для записей с 5-10 FK).
**Risk:** консистентность projection requires careful invalidation strategy, иначе projection stale до next write referencing'а dict'и. **Mitigations** добавлены в этой ревизии (см. § Critical safety).
---
## Изменения от v1 (после eng review)
| # | v1 | v2 (now) | Why |
|---|---|---|---|
| 1 | Новый Redis reverse index `fk:<refDict>:<bk>` | **Reuse `LineageIndexService.findRecordDependents()`** | DRY: PG index уже existing, single source of truth |
| 2 | "Subscribe to OutboxEvents" | **Extend `RecordEventListener` в projection-writer** | Корректное terminology — flow идёт через Kafka topics |
| 3 | Fan-out 1000 records — pipeline и ОК | **Batch cap 500 + queue depth metric + alert** | Realistic worst case 50k (country dict) saturate'нет consumer |
| 4 | «Eventual consistency few seconds» | **Explicit SLO: P95 < 60s для <10k, P95 < 300s для >10k** + `X-Projection-Updated-At` header в read-api | UX bug когда админ видит mixed state |
| 5 | Cycle detection: «only direct FK» (in prose) | **Explicit constraint + test: nested FK resolution forbidden** | Защита от future «улучшений» |
| 6 | Storage: 10k × 1KB = +300MB | **Fixed formula: × locale count × FK count = realistic +450MB-1.5GB** | Per-locale multiplier compound'ил, не учтён |
| 7 | `@Cacheable` FkResolver на 5 min | **No Spring cache** — projection-writer уже триггерит cascade, кэш дал бы stale при write | Two caching layers were unsafe |
| 8 | No test plan | **15-test plan with regression test, race test, E2E через Kafka, perf test** | Был только перечень "Integration tests" |
| 9 | No rollback strategy | **Toggle off → eventual cleanup on next upsert** (не background job) | Меньше moving parts |
| 10 | No kill-switch | **Global feature flag `ordinis.projection.fk-resolution.enabled` + per-dict + per-FK** | 3 уровня контроля для prod safety |
**Round 2 fixes (v2 → v2.1):**
| # | v2 | v2.1 (now) | Why |
|---|---|---|---|
| 11 | Kill switch precedence implicit | **Explicit precedence table** (write requires all 3; read requires global+dict only) | Operator clarity, lazy cleanup на per-FK toggle off |
| 12 | `CascadeInvalidator` triggers via schema scan per event | **Step 1.5: fkTargetDicts cache + SchemaPublished invalidation** | High-RPS dicts не платят schema lookup на каждый message |
| 13 | Bundle isolation lost в Open Question #3 | **Step 1.6: SchemaValidator reject cross-bundle x-references** | Multi-tenancy invariant защищён valid'ом |
| 14 | `_meta.updatedAt` ambiguous (cascade vs direct) | **Explicit max(direct, cascade) policy** | Silent UX bug: header lying about staleness |
| 15 | Test plan 15 cases | **20 cases** (+ kill switch matrix, cache invalidation, cross-bundle, `_meta.updatedAt` timing, storage cap precision) | Coverage diagram gaps closed |
---
## Текущее состояние
### Что есть
```
RecordCreated/Updated event
Kafka topic (3 scopes)
ordinis-projection-writer/RecordEventListener
ProjectionWriter.upsert()
┌───────────────────┼───────────────────┐
▼ ▼ ▼
flatten(per-locale) raw key SET dictionaryIndex SET ADD
Redis SET key=record(...,locale)
```
`flatten()` обрабатывает **только** `x-localized` поля. FK поля проходят как есть.
### Что хочется
При write проекции добавляется `_resolved` ключ с pre-fetched label'ами:
```json
{
"businessKey": "ISS",
"satellite_type": "OPERATIONAL",
"country": "RU",
"_resolved": {
"satellite_type": "Действующий",
"country": "Россия"
}
}
```
Read-api сразу отдаёт нужный label без второго round-trip.
---
## Архитектура v2
### Component diagram
```
Kafka event: RecordCreated{dict=spacecraft, bk=ISS}
RecordEventListener (existing)
ProjectionWriter.upsert()
┌───────────────────┼───────────────────┐
▼ ▼ ▼
flatten(...) FkResolver raw key SET (как сейчас)
│ │
│ │ для каждого x-resolve-label поля:
│ │ - PG read replica lookup
│ │ - inject _resolved.<field>
│ ▼
│ ┌──────────────┐
│ │ PG read │
│ │ replica │ (no cache layer!)
│ └──────────────┘
Redis SET key=record(...,locale)
Kafka event: RecordUpdated{dict=satellite_types, bk=OPERATIONAL}
RecordEventListener (existing)
CascadeInvalidator (NEW component)
LineageIndexService.findRecordDependents() ← REUSE
paged list of (source_dict, source_bk)
enqueue invalidation batch (cap 500)
for each batch: re-fetch from PG → ProjectionWriter.upsert()
Redis pipelined batch write (Spring executePipelined)
```
### Phase A: write-side FK resolver (~2 дня)
`FkResolver` service в `ordinis-projection-writer`:
```java
@Component
public class FkResolver {
private final DictionaryRecordRepository recordRepo; // PG read replica
private final DictionaryDefinitionRepository defRepo;
/**
* Resolve FK value → human label per locale.
* Performance: single PG read query per (refDict, fkValue) — sub-ms на read replica.
* No caching by design (см. eng review #2 — stale read window).
*/
public Optional<String> resolveLabel(
String refDict, // "satellite_types"
String refField, // "code" — default businessKey
String fkValue, // "OPERATIONAL"
String locale // "ru"
) { ... }
/** Batch resolve для N FK fields одной записи (single PG query через IN clause). */
public Map<String, String> resolveBatch(
List<FkRequest> requests,
String locale
) { ... }
}
```
Update `ProjectionWriter.flatten()`:
- Walk schema.properties, для каждого FK поля с `x-resolve-label: true`:
- Call `resolveBatch` (single PG query для всех FK одной записи)
- Inject `_resolved.<field>` JSON
- **Hard constraint:** не рекурсировать в nested objects — direct fields only
### Phase B: cascade invalidation (~2 дня) — **reuse LineageIndexService**
New `CascadeInvalidator` в `ordinis-projection-writer`:
```java
@Component
public class CascadeInvalidator {
private final LineageIndexService lineageIndex; // REUSE
private final ProjectionWriter writer;
private final DictionaryRecordRepository recordRepo;
private static final int BATCH_SIZE_CAP = 500; // hard limit
/** Triggered RecordEventListener'ом когда updated dict — потенциальный FK target. */
public void onSourceRecordUpdate(String refDict, String businessKey) {
int page = 0;
while (true) {
Page<RecordDependent> deps = lineageIndex.findRecordDependents(
refDict, businessKey, allScopes(), PageRequest.of(page, BATCH_SIZE_CAP));
if (deps.isEmpty()) break;
// Re-fetch dependents from PG + rewrite projection (pipelined)
writer.batchRewriteProjections(deps.getContent());
cascadeInvalidationCounter.increment(deps.getNumberOfElements());
if (!deps.hasNext()) break;
page++;
// Backpressure если queue depth высокий
if (cascadeQueueDepthGauge.get() > QUEUE_DEPTH_ALERT) {
Thread.sleep(THROTTLE_MS);
}
}
}
}
```
Hook в существующий `RecordEventListener`:
- На `RecordUpdated{dict=X}` event — после обычной upsert(), check'аем X **через cached `fkTargetDicts` set** (step 1.5). Schema-level dependency map immutable между SchemaPublished events; cache invalidate'ится по этому event'у. Это избегает `LineageIndexService.findSchemaDependents()` query на каждом message — критично для high-RPS dicts.
- Если X ∈ fkTargetDicts → вызываем `CascadeInvalidator.onSourceRecordUpdate(X, businessKey)`
### Phase C: backfill + ops (~1 день)
CLI endpoint `POST /api/v1/admin/projections/{dict}/backfill`:
- Paginated iterate (batch=500, sleep=100ms между batch'ами)
- Per-page commit chunk — recoverable если interrupted
- Lock через advisory lock в PG (one backfill per dict at a time)
- Progress reporting через outbox event stream
### Phase D: observability + flags
3-tier kill switch:
1. **Global**`ordinis.projection.fk-resolution.enabled` (env var)
2. **Per-dict**`DictionaryDefinition.fkResolutionEnabled` (existing field паттерн, новая колонка)
3. **Per-FK** — schema annotation `x-resolve-label: true` (default false)
**Precedence (explicit per round 2 review):**
| Layer | Write path (resolve & inject `_resolved`) | Read-api (return `_resolved` к caller'у) |
|---|---|---|
| Global=false | ❌ skip | ❌ ignore `_resolved` even if present (stale residue from before toggle) |
| Global=true, Dict=false | ❌ skip | ❌ ignore (per-dict opt-out wins over residual data) |
| Global=true, Dict=true, FK=false | ❌ skip для этого поля | ✅ return whatever's already в `_resolved` (no harm — поле там не появится) |
| Global=true, Dict=true, FK=true | ✅ resolve & write | ✅ return `_resolved.<field>` |
Logic: **write requires all three true**; **read requires global + dict** (FK granularity не нужна на read — если backend перестал писать поле, `_resolved.<field>` natural выпадает при следующем upsert). Это значит turning off per-FK toggle = lazy cleanup. Turning off per-dict = immediate hide. Turning off global = panic kill для всего projection FK behavior.
**Metrics (Prometheus):**
- `ordinis_projection_fk_resolved_total{dict, fk_field}`
- `ordinis_projection_fk_resolve_miss_total{dict, fk_field, reason="not_accessible|not_found|locale_missing"}`
- `ordinis_projection_cascade_invalidation_total{trigger_dict}`
- `ordinis_projection_cascade_queue_depth` (gauge)
- `ordinis_projection_fk_resolve_duration_seconds` (histogram)
- `ordinis_projection_storage_resolved_bytes_total{dict}` (gauge)
**SLO targets:**
- P95 FK resolution latency: < 5ms (PG read replica baseline)
- P95 cascade invalidation:
- < 60s для cascade size ≤ 10k dependents
- < 300s для cascade size 10k-50k
- > 50k cascade → page on-call (likely operator error)
- Projection eventual consistency window: max 300s
**Read-api integration:**
- Response header `X-Projection-Updated-At: 2026-05-12T17:30:00Z` (last upsert time per record)
- `staleness=` query param для force-fresh read через PG fallback (debugging)
**`_meta.updatedAt` policy (per round 2 review — silent UX bug fix):**
Stored в projection JSON как `_meta.updatedAt` (sibling `_resolved`):
- **Updated on direct upsert** (RecordCreated/Updated event для этой записи)
- **Updated on cascade rewrite** (CascadeInvalidator пишет проекцию после изменения FK target)
- Read-api header = `max(direct_upsert_time, cascade_rewrite_time)` = the **`_meta.updatedAt` value** as-stored (cascade overwrites if newer)
Why max'om: иначе админ видит «projection updated 5 минут назад» а на самом деле cascade обновил labels 3 секунды назад — silent UX bug когда юзер ждёт что прочтёт fresh данные. Stored timestamp всегда **точка последней мутации projection'а**, не direct write.
Storage cost: 24 bytes per record × 3 locales = 72 bytes overhead. Negligible.
---
## Critical safety (added in v2)
### Race conditions: concurrent target update during dependent write
**Scenario:** Запись A пишется в момент когда B (FK target) сам обновляется.
**Mitigation:** `FkResolver` читает с PG **after** projection write — если B меняется между resolution и Redis SET, мы пишем stale label. Затем cascade от B's update подхватит A и rewrite projection → eventual consistency.
**Test:** `ConcurrentFkUpdateTest` — два потока, поток1 пишет A, поток2 update B, проверяем что через ≤30s projection A содержит latest label B.
### Kafka consumer lag во время массивного cascade
**Scenario:** Update country=RU → 50k spacecrafts инвалидируются.
**Mitigation:**
1. Batch cap **500** dependents за раз
2. `cascadeQueueDepthGauge` exposed → alert thresholds (warn @ 1000, crit @ 5000)
3. Throttle между batch'ами если queue depth превышает
4. Async dispatch — cascade work не в main listener thread, отдельный `@Async` executor с bounded queue
**Test:** `CascadeBackpressureTest` — load 10k dependents, verify Kafka consumer lag не превышает 60s.
### Redis memory pressure
**Scenario:** Enable flag для крупного dict'a → проекции вырастут 2-3×.
**Mitigation:**
1. Metric `ordinis_projection_storage_resolved_bytes_total{dict}` + alert на 80% Redis memory cap
2. Per-record max size check — если `_resolved` blob > 50KB, skip resolution + log warning (не падать)
3. Operational runbook: «disable per-dict flag → wait for natural eviction (24h TTL?) → cleanup»
### Rollback path
Если flag toggled off:
- Existing `_resolved` keys остаются stale, но **harmless** — read-api ignore'ит `_resolved` когда flag=false
- При next upsert каждой записи `_resolved` automatically dropped (write replaces full value)
- Если нужен immediate cleanup — CLI `POST /admin/projections/{dict}/strip-resolved` (~1 час имплементации)
---
## Open questions
1. **PG read replica lag.** `FkResolver` читает с replica для performance, но replica может отставать на seconds под write load. Если target dict обновлён 100ms назад, FkResolver получит stale label, cascade подхватит ~30s. Acceptable. Документировать.
2. **Scope-hide handling.** Если FK target dict не доступен caller'у — projection write идёт как **system user** (не user-scope). Resolution всё равно происходит, но read-api позже filter'нет `_resolved` если caller scope не допускает target dict. Implementation: `FkResolver` использует privileged read, read-api проверяет access.
3. **Bundle isolation.** В multi-bundle setup'е (cuod, altum, etc.) — FK может пересекать bundle boundaries? Recommendation: запретить, validation в schema editor (`x-references` поле target dict должен быть в том же bundle).
4. **Storage cap.** Hard limit на per-record `_resolved` blob — 50KB? Это значит до ~25 FK fields × 3 locales × ~600 chars label = 45KB. Достаточно для всех realistic case'ов.
---
## Альтернатива: skip FK resolution, use read-api JOIN (unchanged from v1)
Вместо pre-resolve'инга при write — read-api делает JOIN на читающей стороне (через PG или client-side). Pro: simpler, no cascade. Con: N+1 на каждое чтение, ровно то что мы хотим избежать.
Если RPS не упёрся в predisposed, можно отложить весь FK projection. Прагматичное правило: **включать FK resolution только когда конкретный dict упёрся в latency SLA**. Default off навсегда.
---
## Implementation plan (revised)
| Step | Effort (CC+gstack) | Notes |
|---|---|---|
| 1. Schema annotation `x-resolve-label` + JSON Schema validator | 1h | Update SchemaValidator, fail-fast on non-FK field |
| 1.5. **fkTargetDicts cache** + SchemaPublished invalidation | 1h | Pre-compute set of dicts that являются FK target (via `LineageIndexService.findSchemaDependents` inversion). Avoid schema scan на каждый RecordUpdated event в `RecordEventListener` |
| 1.6. **Bundle isolation enforcement** в `SchemaValidator` | 0.5h | Reject `x-references: "<targetDict>.<field>"` если target_dict в другом bundle. Single-bundle invariant — иначе multi-tenancy ломается |
| 2. Per-dict `fk_resolution_enabled` column + migration | 1h | Liquibase, Dictionary CRUD. Default `false` для всех existing dicts (backward-compat). |
| 3. `FkResolver` service + batch resolve method | 3h | Single PG query per record (IN clause) |
| 4. Update `ProjectionWriter.flatten()` для `_resolved` injection | 2h | Walk schema, hard constraint on nested |
| 5. `CascadeInvalidator` reusing `LineageIndexService` | 4h | Batch cap, queue depth gauge, async |
| 6. `RecordEventListener` hook into `CascadeInvalidator` | 1h | Detect target dict from schemas index |
| 7. Backfill CLI endpoint + advisory lock | 2h | Paginated, idempotent |
| 8. Metrics + SLO alerting config (Grafana panels) | 2h | 6 new metrics, 3 alert rules |
| 9. Read-api `X-Projection-Updated-At` header | 1h | Track via `_meta.updatedAt` в projection JSON |
| 10. Integration tests (15 cases per coverage diagram) | 8h | testcontainers Postgres+Redis+Kafka |
| 11. Frontend: schema editor checkbox per FK field | 2h | `DictionaryEditorDialog`, validation |
| 12. Docs (ops runbook, schema annotation guide) | 2h | docs/ops/projection-fk-resolution.md |
| **Total** | **~30.5h (4-5d)** | within revised budget (+1.5h после round 2 review) |
---
## Test plan (added in v2 per eng review)
Minimum 15 integration tests, testcontainers Postgres + Redis + Kafka:
| # | Test | Type | Critical? |
|---|---|---|---|
| 1 | Happy path: 3 FK fields, all resolve | unit | — |
| 2 | One FK miss (orphan) → omitted from `_resolved` | unit | — |
| 3 | Multi-locale resolution per locale array | unit | — |
| 4 | Locale fallback (ru missing → defaultLocale en) | unit | — |
| 5 | Schema validation: `x-resolve-label` on non-FK → error | unit | — |
| 6 | **Regression**: write WITHOUT flag works как раньше | unit | **YES** |
| 7 | Cascade fires on RecordUpdated of FK target | integration | — |
| 8 | Cascade batch cap 500 enforced | integration | — |
| 9 | Cascade queue depth metric increments | integration | — |
| 10 | Concurrent update race → eventual consistency ≤30s | integration | **CRITICAL** |
| 11 | Backpressure: 10k dependents → consumer lag <60s | integration | **CRITICAL** |
| 12 | E2E через Kafka real flow: PUT → cascade → Redis read | E2E | — |
| 13 | Backfill CLI idempotent (re-run same result) | integration | — |
| 14 | Toggle off → next upsert removes `_resolved` | integration | — |
| 15 | Storage cap (50KB) — large record skips resolution + warns | integration | — |
| 16 | **Kill switch precedence matrix** — все 8 комбинаций global/dict/FK booleans (truth table, parameterized test): write + read behavior matches table в § Phase D | integration | **YES** |
| 17 | **fkTargetDicts cache invalidation** — SchemaPublished event дропает schema-level dependency cache, next RecordUpdated пересчитывает | integration | — |
| 18 | **Cross-bundle FK rejected**`SchemaValidator` reject'нет `x-references` указывающий на dict из другого bundle | unit | — |
| 19 | **`_meta.updatedAt` cascade vs direct timing** — cascade rewrite обновляет timestamp; direct upsert после cascade перезаписывает; header возвращает max | integration | — |
| 20 | **Storage cap precision** — 50KB cap применяется per-record (NOT per-field); record с 49KB raw + 5KB `_resolved` = total 54KB → resolved skipped, raw written | integration | — |
---
## Recommendation (unchanged from v1)
**Defer until after v2.12.0 prod stable + 2 weeks dogfooding.** Текущий read path handles RPS, no urgency. Activate per-FK flag только когда first dict bumps into latency SLA. Build full pipeline только if 3+ dict'ам нужно.
**Next step (если decide go):** этот revised doc → `/plan-eng-review` round 2 → if CLEAR → CEO approval → sprint allocation.
---
## GSTACK REVIEW REPORT
| Review | Trigger | Why | Runs | Status | Findings |
|--------|---------|-----|------|--------|----------|
| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 2 (v1 + v2) | 🟢 CLEAR | v1: 11 issues + 3 critical → all closed v2. v2 round 2: 5 minor + 1 silent UX bug → all closed v2.1 |
| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | — | — |
| Design Review | `/plan-design-review` | UI/UX gaps | 0 | n/a | Skip — backend feature |
**UNRESOLVED:** 0
**VERDICT:** 🟢 **CLEAR after v2.1 fixes.** Ready для implementation **когда** придёт время. Defer recommendation сохраняется: implement только после v2.12.0 prod stable + 2 weeks dogfooding + первый dict упрётся в latency SLA.