# Design: Redis Projection FK Resolution

**Author:** zimin.an
**Date:** 2026-05-12
**Status:** PROPOSED v2.1 (round 2 `/plan-eng-review` fixes applied)
**Supersedes:**
- v1 (2026-05-12 first draft, before eng review found `LineageIndexService` reuse opportunity)
- v2 (2026-05-12 после round 1 review — major scope reduction + critical safety mitigations)
**Sprint estimate:** **4-5 рабочих дней** (~30.5h CC+gstack)
**Blockers:** none, рекомендация defer'a сохраняется — implement только когда конкретный dict упрётся в latency SLA

---

## TL;DR

Сейчас Redis projection (writer module) пишет per-locale flattened JSON записи в Redis для dict'ей с `redis_projection_enabled=true`. **FK ссылки** (`x-references: "dict.field"`) хранятся как raw FK value (`satellite_type: "OPERATIONAL"`). Read-api делает N+1 lookup чтобы получить human label (`Действующий`).

**Предложение:** при write resolve'ить FK label'ы и сохранять рядом с FK value в `_resolved` ключе. Per-FK opt-in через `x-resolve-label: true`. **Cascade invalidation reusing existing `LineageIndexService`** (no new Redis reverse index).

**Win:** один Redis GET вместо N+1, latency 1ms вместо N×1ms (для записей с 5-10 FK).

**Risk:** консистентность projection requires careful invalidation strategy, иначе projection stale до next write referencing'а dict'и. **Mitigations** добавлены в этой ревизии (см. § Critical safety).

---

## Изменения от v1 (после eng review)

| # | v1 | v2 (now) | Why |
|---|---|---|---|
| 1 | Новый Redis reverse index `fk:<refDict>:<bk>` | **Reuse `LineageIndexService.findRecordDependents()`** | DRY: PG index уже existing, single source of truth |
| 2 | "Subscribe to OutboxEvents" | **Extend `RecordEventListener` в projection-writer** | Корректное terminology — flow идёт через Kafka topics |
| 3 | Fan-out 1000 records — pipeline и ОК | **Batch cap 500 + queue depth metric + alert** | Realistic worst case 50k (country dict) saturate'нет consumer |
| 4 | «Eventual consistency few seconds» | **Explicit SLO: P95 < 60s для <10k, P95 < 300s для >10k** + `X-Projection-Updated-At` header в read-api | UX bug когда админ видит mixed state |
| 5 | Cycle detection: «only direct FK» (in prose) | **Explicit constraint + test: nested FK resolution forbidden** | Защита от future «улучшений» |
| 6 | Storage: 10k × 1KB = +300MB | **Fixed formula: × locale count × FK count = realistic +450MB-1.5GB** | Per-locale multiplier compound'ил, не учтён |
| 7 | `@Cacheable` FkResolver на 5 min | **No Spring cache** — projection-writer уже триггерит cascade, кэш дал бы stale при write | Two caching layers were unsafe |
| 8 | No test plan | **15-test plan with regression test, race test, E2E через Kafka, perf test** | Был только перечень "Integration tests" |
| 9 | No rollback strategy | **Toggle off → eventual cleanup on next upsert** (не background job) | Меньше moving parts |
| 10 | No kill-switch | **Global feature flag `ordinis.projection.fk-resolution.enabled` + per-dict + per-FK** | 3 уровня контроля для prod safety |

**Round 2 fixes (v2 → v2.1):**

| # | v2 | v2.1 (now) | Why |
|---|---|---|---|
| 11 | Kill switch precedence implicit | **Explicit precedence table** (write requires all 3; read requires global+dict only) | Operator clarity, lazy cleanup на per-FK toggle off |
| 12 | `CascadeInvalidator` triggers via schema scan per event | **Step 1.5: fkTargetDicts cache + SchemaPublished invalidation** | High-RPS dicts не платят schema lookup на каждый message |
| 13 | Bundle isolation lost в Open Question #3 | **Step 1.6: SchemaValidator reject cross-bundle x-references** | Multi-tenancy invariant защищён valid'ом |
| 14 | `_meta.updatedAt` ambiguous (cascade vs direct) | **Explicit max(direct, cascade) policy** | Silent UX bug: header lying about staleness |
| 15 | Test plan 15 cases | **20 cases** (+ kill switch matrix, cache invalidation, cross-bundle, `_meta.updatedAt` timing, storage cap precision) | Coverage diagram gaps closed |

---

## Текущее состояние

### Что есть

```
                         RecordCreated/Updated event
                                    │
                                    ▼
                         Kafka topic (3 scopes)
                                    │
                                    ▼
   ordinis-projection-writer/RecordEventListener
                                    │
                                    ▼
                      ProjectionWriter.upsert()
                                    │
                ┌───────────────────┼───────────────────┐
                ▼                   ▼                   ▼
    flatten(per-locale)      raw key SET         dictionaryIndex SET ADD
                │
                ▼
       Redis SET key=record(...,locale)
```

`flatten()` обрабатывает **только** `x-localized` поля. FK поля проходят как есть.

### Что хочется

При write проекции добавляется `_resolved` ключ с pre-fetched label'ами:

```json
{
  "businessKey": "ISS",
  "satellite_type": "OPERATIONAL",
  "country": "RU",
  "_resolved": {
    "satellite_type": "Действующий",
    "country": "Россия"
  }
}
```

Read-api сразу отдаёт нужный label без второго round-trip.

---

## Архитектура v2

### Component diagram

```
   Kafka event: RecordCreated{dict=spacecraft, bk=ISS}
                                    │
                                    ▼
                      RecordEventListener (existing)
                                    │
                                    ▼
                      ProjectionWriter.upsert()
                                    │
                ┌───────────────────┼───────────────────┐
                ▼                   ▼                   ▼
        flatten(...)         FkResolver        raw key SET (как сейчас)
              │                  │
              │                  │ для каждого x-resolve-label поля:
              │                  │   - PG read replica lookup
              │                  │   - inject _resolved.<field>
              │                  ▼
              │           ┌──────────────┐
              │           │   PG read    │
              │           │   replica    │ (no cache layer!)
              │           └──────────────┘
              ▼
       Redis SET key=record(...,locale)


   Kafka event: RecordUpdated{dict=satellite_types, bk=OPERATIONAL}
                                    │
                                    ▼
                      RecordEventListener (existing)
                                    │
                                    ▼
                  CascadeInvalidator (NEW component)
                                    │
                                    ▼
        LineageIndexService.findRecordDependents()  ← REUSE
                                    │
                                    ▼
              paged list of (source_dict, source_bk)
                                    │
                                    ▼
                  enqueue invalidation batch (cap 500)
                                    │
                                    ▼
              for each batch: re-fetch from PG → ProjectionWriter.upsert()
                                    │
                                    ▼
                Redis pipelined batch write (Spring executePipelined)
```

### Phase A: write-side FK resolver (~2 дня)

`FkResolver` service в `ordinis-projection-writer`:

```java
@Component
public class FkResolver {
  private final DictionaryRecordRepository recordRepo;   // PG read replica
  private final DictionaryDefinitionRepository defRepo;

  /**
   * Resolve FK value → human label per locale.
   * Performance: single PG read query per (refDict, fkValue) — sub-ms на read replica.
   * No caching by design (см. eng review #2 — stale read window).
   */
  public Optional<String> resolveLabel(
      String refDict,        // "satellite_types"
      String refField,       // "code" — default businessKey
      String fkValue,        // "OPERATIONAL"
      String locale          // "ru"
  ) { ... }

  /** Batch resolve для N FK fields одной записи (single PG query через IN clause). */
  public Map<String, String> resolveBatch(
      List<FkRequest> requests,
      String locale
  ) { ... }
}
```

Update `ProjectionWriter.flatten()`:
- Walk schema.properties, для каждого FK поля с `x-resolve-label: true`:
  - Call `resolveBatch` (single PG query для всех FK одной записи)
  - Inject `_resolved.<field>` JSON
- **Hard constraint:** не рекурсировать в nested objects — direct fields only

### Phase B: cascade invalidation (~2 дня) — **reuse LineageIndexService**

New `CascadeInvalidator` в `ordinis-projection-writer`:

```java
@Component
public class CascadeInvalidator {
  private final LineageIndexService lineageIndex;       // REUSE
  private final ProjectionWriter writer;
  private final DictionaryRecordRepository recordRepo;
  private static final int BATCH_SIZE_CAP = 500;        // hard limit

  /** Triggered RecordEventListener'ом когда updated dict — потенциальный FK target. */
  public void onSourceRecordUpdate(String refDict, String businessKey) {
    int page = 0;
    while (true) {
      Page<RecordDependent> deps = lineageIndex.findRecordDependents(
          refDict, businessKey, allScopes(), PageRequest.of(page, BATCH_SIZE_CAP));
      if (deps.isEmpty()) break;

      // Re-fetch dependents from PG + rewrite projection (pipelined)
      writer.batchRewriteProjections(deps.getContent());

      cascadeInvalidationCounter.increment(deps.getNumberOfElements());
      if (!deps.hasNext()) break;
      page++;

      // Backpressure если queue depth высокий
      if (cascadeQueueDepthGauge.get() > QUEUE_DEPTH_ALERT) {
        Thread.sleep(THROTTLE_MS);
      }
    }
  }
}
```

Hook в существующий `RecordEventListener`:
- На `RecordUpdated{dict=X}` event — после обычной upsert(), check'аем X **через cached `fkTargetDicts` set** (step 1.5). Schema-level dependency map immutable между SchemaPublished events; cache invalidate'ится по этому event'у. Это избегает `LineageIndexService.findSchemaDependents()` query на каждом message — критично для high-RPS dicts.
- Если X ∈ fkTargetDicts → вызываем `CascadeInvalidator.onSourceRecordUpdate(X, businessKey)`

### Phase C: backfill + ops (~1 день)

CLI endpoint `POST /api/v1/admin/projections/{dict}/backfill`:
- Paginated iterate (batch=500, sleep=100ms между batch'ами)
- Per-page commit chunk — recoverable если interrupted
- Lock через advisory lock в PG (one backfill per dict at a time)
- Progress reporting через outbox event stream

### Phase D: observability + flags

3-tier kill switch:
1. **Global** — `ordinis.projection.fk-resolution.enabled` (env var)
2. **Per-dict** — `DictionaryDefinition.fkResolutionEnabled` (existing field паттерн, новая колонка)
3. **Per-FK** — schema annotation `x-resolve-label: true` (default false)

**Precedence (explicit per round 2 review):**

| Layer | Write path (resolve & inject `_resolved`) | Read-api (return `_resolved` к caller'у) |
|---|---|---|
| Global=false | ❌ skip | ❌ ignore `_resolved` even if present (stale residue from before toggle) |
| Global=true, Dict=false | ❌ skip | ❌ ignore (per-dict opt-out wins over residual data) |
| Global=true, Dict=true, FK=false | ❌ skip для этого поля | ✅ return whatever's already в `_resolved` (no harm — поле там не появится) |
| Global=true, Dict=true, FK=true | ✅ resolve & write | ✅ return `_resolved.<field>` |

Logic: **write requires all three true**; **read requires global + dict** (FK granularity не нужна на read — если backend перестал писать поле, `_resolved.<field>` natural выпадает при следующем upsert). Это значит turning off per-FK toggle = lazy cleanup. Turning off per-dict = immediate hide. Turning off global = panic kill для всего projection FK behavior.

**Metrics (Prometheus):**
- `ordinis_projection_fk_resolved_total{dict, fk_field}`
- `ordinis_projection_fk_resolve_miss_total{dict, fk_field, reason="not_accessible|not_found|locale_missing"}`
- `ordinis_projection_cascade_invalidation_total{trigger_dict}`
- `ordinis_projection_cascade_queue_depth` (gauge)
- `ordinis_projection_fk_resolve_duration_seconds` (histogram)
- `ordinis_projection_storage_resolved_bytes_total{dict}` (gauge)

**SLO targets:**
- P95 FK resolution latency: < 5ms (PG read replica baseline)
- P95 cascade invalidation:
  - < 60s для cascade size ≤ 10k dependents
  - < 300s для cascade size 10k-50k
  - > 50k cascade → page on-call (likely operator error)
- Projection eventual consistency window: max 300s

**Read-api integration:**
- Response header `X-Projection-Updated-At: 2026-05-12T17:30:00Z` (last upsert time per record)
- `staleness=` query param для force-fresh read через PG fallback (debugging)

**`_meta.updatedAt` policy (per round 2 review — silent UX bug fix):**

Stored в projection JSON как `_meta.updatedAt` (sibling `_resolved`):
- **Updated on direct upsert** (RecordCreated/Updated event для этой записи)
- **Updated on cascade rewrite** (CascadeInvalidator пишет проекцию после изменения FK target)
- Read-api header = `max(direct_upsert_time, cascade_rewrite_time)` = the **`_meta.updatedAt` value** as-stored (cascade overwrites if newer)

Why max'om: иначе админ видит «projection updated 5 минут назад» а на самом деле cascade обновил labels 3 секунды назад — silent UX bug когда юзер ждёт что прочтёт fresh данные. Stored timestamp всегда **точка последней мутации projection'а**, не direct write.

Storage cost: 24 bytes per record × 3 locales = 72 bytes overhead. Negligible.

---

## Critical safety (added in v2)

### Race conditions: concurrent target update during dependent write

**Scenario:** Запись A пишется в момент когда B (FK target) сам обновляется.

**Mitigation:** `FkResolver` читает с PG **after** projection write — если B меняется между resolution и Redis SET, мы пишем stale label. Затем cascade от B's update подхватит A и rewrite projection → eventual consistency.

**Test:** `ConcurrentFkUpdateTest` — два потока, поток1 пишет A, поток2 update B, проверяем что через ≤30s projection A содержит latest label B.

### Kafka consumer lag во время массивного cascade

**Scenario:** Update country=RU → 50k spacecrafts инвалидируются.

**Mitigation:**
1. Batch cap **500** dependents за раз
2. `cascadeQueueDepthGauge` exposed → alert thresholds (warn @ 1000, crit @ 5000)
3. Throttle между batch'ами если queue depth превышает
4. Async dispatch — cascade work не в main listener thread, отдельный `@Async` executor с bounded queue

**Test:** `CascadeBackpressureTest` — load 10k dependents, verify Kafka consumer lag не превышает 60s.

### Redis memory pressure

**Scenario:** Enable flag для крупного dict'a → проекции вырастут 2-3×.

**Mitigation:**
1. Metric `ordinis_projection_storage_resolved_bytes_total{dict}` + alert на 80% Redis memory cap
2. Per-record max size check — если `_resolved` blob > 50KB, skip resolution + log warning (не падать)
3. Operational runbook: «disable per-dict flag → wait for natural eviction (24h TTL?) → cleanup»

### Rollback path

Если flag toggled off:
- Existing `_resolved` keys остаются stale, но **harmless** — read-api ignore'ит `_resolved` когда flag=false
- При next upsert каждой записи `_resolved` automatically dropped (write replaces full value)
- Если нужен immediate cleanup — CLI `POST /admin/projections/{dict}/strip-resolved` (~1 час имплементации)

---

## Open questions

1. **PG read replica lag.** `FkResolver` читает с replica для performance, но replica может отставать на seconds под write load. Если target dict обновлён 100ms назад, FkResolver получит stale label, cascade подхватит ~30s. Acceptable. Документировать.

2. **Scope-hide handling.** Если FK target dict не доступен caller'у — projection write идёт как **system user** (не user-scope). Resolution всё равно происходит, но read-api позже filter'нет `_resolved` если caller scope не допускает target dict. Implementation: `FkResolver` использует privileged read, read-api проверяет access.

3. **Bundle isolation.** В multi-bundle setup'е (cuod, altum, etc.) — FK может пересекать bundle boundaries? Recommendation: запретить, validation в schema editor (`x-references` поле target dict должен быть в том же bundle).

4. **Storage cap.** Hard limit на per-record `_resolved` blob — 50KB? Это значит до ~25 FK fields × 3 locales × ~600 chars label = 45KB. Достаточно для всех realistic case'ов.

---

## Альтернатива: skip FK resolution, use read-api JOIN (unchanged from v1)

Вместо pre-resolve'инга при write — read-api делает JOIN на читающей стороне (через PG или client-side). Pro: simpler, no cascade. Con: N+1 на каждое чтение, ровно то что мы хотим избежать.

Если RPS не упёрся в predisposed, можно отложить весь FK projection. Прагматичное правило: **включать FK resolution только когда конкретный dict упёрся в latency SLA**. Default off навсегда.

---

## Implementation plan (revised)

| Step | Effort (CC+gstack) | Notes |
|---|---|---|
| 1. Schema annotation `x-resolve-label` + JSON Schema validator | 1h | Update SchemaValidator, fail-fast on non-FK field |
| 1.5. **fkTargetDicts cache** + SchemaPublished invalidation | 1h | Pre-compute set of dicts that являются FK target (via `LineageIndexService.findSchemaDependents` inversion). Avoid schema scan на каждый RecordUpdated event в `RecordEventListener` |
| 1.6. **Bundle isolation enforcement** в `SchemaValidator` | 0.5h | Reject `x-references: "<targetDict>.<field>"` если target_dict в другом bundle. Single-bundle invariant — иначе multi-tenancy ломается |
| 2. Per-dict `fk_resolution_enabled` column + migration | 1h | Liquibase, Dictionary CRUD. Default `false` для всех existing dicts (backward-compat). |
| 3. `FkResolver` service + batch resolve method | 3h | Single PG query per record (IN clause) |
| 4. Update `ProjectionWriter.flatten()` для `_resolved` injection | 2h | Walk schema, hard constraint on nested |
| 5. `CascadeInvalidator` reusing `LineageIndexService` | 4h | Batch cap, queue depth gauge, async |
| 6. `RecordEventListener` hook into `CascadeInvalidator` | 1h | Detect target dict from schemas index |
| 7. Backfill CLI endpoint + advisory lock | 2h | Paginated, idempotent |
| 8. Metrics + SLO alerting config (Grafana panels) | 2h | 6 new metrics, 3 alert rules |
| 9. Read-api `X-Projection-Updated-At` header | 1h | Track via `_meta.updatedAt` в projection JSON |
| 10. Integration tests (15 cases per coverage diagram) | 8h | testcontainers Postgres+Redis+Kafka |
| 11. Frontend: schema editor checkbox per FK field | 2h | `DictionaryEditorDialog`, validation |
| 12. Docs (ops runbook, schema annotation guide) | 2h | docs/ops/projection-fk-resolution.md |
| **Total** | **~30.5h (4-5d)** | within revised budget (+1.5h после round 2 review) |

---

## Test plan (added in v2 per eng review)

Minimum 15 integration tests, testcontainers Postgres + Redis + Kafka:

| # | Test | Type | Critical? |
|---|---|---|---|
| 1 | Happy path: 3 FK fields, all resolve | unit | — |
| 2 | One FK miss (orphan) → omitted from `_resolved` | unit | — |
| 3 | Multi-locale resolution per locale array | unit | — |
| 4 | Locale fallback (ru missing → defaultLocale en) | unit | — |
| 5 | Schema validation: `x-resolve-label` on non-FK → error | unit | — |
| 6 | **Regression**: write WITHOUT flag works как раньше | unit | **YES** |
| 7 | Cascade fires on RecordUpdated of FK target | integration | — |
| 8 | Cascade batch cap 500 enforced | integration | — |
| 9 | Cascade queue depth metric increments | integration | — |
| 10 | Concurrent update race → eventual consistency ≤30s | integration | **CRITICAL** |
| 11 | Backpressure: 10k dependents → consumer lag <60s | integration | **CRITICAL** |
| 12 | E2E через Kafka real flow: PUT → cascade → Redis read | E2E | — |
| 13 | Backfill CLI idempotent (re-run same result) | integration | — |
| 14 | Toggle off → next upsert removes `_resolved` | integration | — |
| 15 | Storage cap (50KB) — large record skips resolution + warns | integration | — |
| 16 | **Kill switch precedence matrix** — все 8 комбинаций global/dict/FK booleans (truth table, parameterized test): write + read behavior matches table в § Phase D | integration | **YES** |
| 17 | **fkTargetDicts cache invalidation** — SchemaPublished event дропает schema-level dependency cache, next RecordUpdated пересчитывает | integration | — |
| 18 | **Cross-bundle FK rejected** — `SchemaValidator` reject'нет `x-references` указывающий на dict из другого bundle | unit | — |
| 19 | **`_meta.updatedAt` cascade vs direct timing** — cascade rewrite обновляет timestamp; direct upsert после cascade перезаписывает; header возвращает max | integration | — |
| 20 | **Storage cap precision** — 50KB cap применяется per-record (NOT per-field); record с 49KB raw + 5KB `_resolved` = total 54KB → resolved skipped, raw written | integration | — |

---

## Recommendation (unchanged from v1)

**Defer until after v2.12.0 prod stable + 2 weeks dogfooding.** Текущий read path handles RPS, no urgency. Activate per-FK flag только когда first dict bumps into latency SLA. Build full pipeline только if 3+ dict'ам нужно.

**Next step (если decide go):** этот revised doc → `/plan-eng-review` round 2 → if CLEAR → CEO approval → sprint allocation.

---

## GSTACK REVIEW REPORT

| Review | Trigger | Why | Runs | Status | Findings |
|--------|---------|-----|------|--------|----------|
| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 2 (v1 + v2) | 🟢 CLEAR | v1: 11 issues + 3 critical → all closed v2. v2 round 2: 5 minor + 1 silent UX bug → all closed v2.1 |
| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | — | — |
| Design Review | `/plan-design-review` | UI/UX gaps | 0 | n/a | Skip — backend feature |

**UNRESOLVED:** 0
**VERDICT:** 🟢 **CLEAR after v2.1 fixes.** Ready для implementation **когда** придёт время. Defer recommendation сохраняется: implement только после v2.12.0 prod stable + 2 weeks dogfooding + первый dict упрётся в latency SLA.