feat(topbar): «Что нового» button + 2026-05-12 state.md update
This commit is contained in:
Александр Зимин
@@ -0,0 +1,373 @@
|
||||
# Design: AI-assisted Schema Authoring
|
||||
|
||||
**Author:** zimin.an
|
||||
**Date:** 2026-05-12
|
||||
**Status:** PROPOSED v2 (post `/office-hours`)
|
||||
**Supersedes:** v1 (2026-05-12 — LLM-only "Approach B", before office-hours premise validation revealed `Approach C` hybrid)
|
||||
**Scope:** ДЗЗ domain only (ground segment + orbital, не general-purpose AI)
|
||||
**Sprint estimate:** **~3-4 days CC** для **Approach C hybrid** (revised down from 5-7d v1)
|
||||
**Blockers:** none architectural. P5 measurement step (см. Premises) — soft prereq
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
Admin Ordinis сейчас создаёт справочник вручную — пишет JSON Schema (поля, типы, `x-references`, `x-localized`, validations), пробрасывает локализации, продумывает bitemporal flags. Это **15-30 минут per dictionary** для опытного админа, **час+** для нового.
|
||||
|
||||
**Предложение (v2 — Hybrid):** Admin starts с **curated template** (e.g. «спутник», «наземная станция»), затем при добавлении каждого нового поля кликает **«AI suggest field»** → LLM предлагает structure (type/format/x-localized/x-references) для **одного поля** на базе schema name + existing fields context. Admin reviews → accept или edit → continue.
|
||||
|
||||
**v2 vs v1:** Vместо whole-schema generation одним LLM call'ом — **template skeleton + per-field LLM fill-in**. Smaller LLM context, более reliable, lower GPU bar (7B sufficient instead of 32B), graceful degradation (templates работают без LLM).
|
||||
|
||||
**Win:** time-to-first-schema 30мин → 5-8мин (template + 5-10 fields, каждое 30 сек AI suggest). Compounds with marketplace: templates → bundles directly.
|
||||
|
||||
**Risk:** LLM hallucinations contained per-field (smaller blast radius vs whole schema). Templates always work even если LLM down.
|
||||
|
||||
---
|
||||
|
||||
## Premises (validated через `/office-hours` 2026-05-12)
|
||||
|
||||
**P1.** AI = **time-saver**, не replacement. Schema всегда проходит human review через existing draft workflow. **No auto-publish.** Value = drop time-to-first-schema 30мин → 5-8мин.
|
||||
|
||||
**P2.** **Local LLM** (vLLM/Ollama) **non-negotiable** для гос-клиентов — data не покидает perimeter. External API disabled by default, feature-flag-only для corp non-classified. **GPU verification = hard prereq** перед implementation.
|
||||
|
||||
**P3.** **Scope = ДЗЗ domain only.** Few-shot training на ЦУОД bundle (5 примеров). Non-ДЗЗ customer → manually extends few-shot в свой bundle.
|
||||
|
||||
**P4.** **AI predicts STRUCTURE, не SEMANTICS.** Type, format, nullability, FK ref structure, locales — да. Validation rules, GOST codes, business constraints — NET. System prompt explicit: «если не уверен — оставь пустым».
|
||||
|
||||
**P5.** **Demand evidence пока нулевая** — founder-driven feature, не customer-pulled. **Prerequisite measurement step:** перед implementation добавить metric `admin_schema_authoring_time_seconds` (start = first draft creation, end = first review submission). Собрать **2 weeks baseline** на v2.14.0 prod. Если P50 < 10 мин — отложить feature (no real pain). Если P50 > 20 мин — confirms hypothesis, proceed.
|
||||
|
||||
**P6.** Cross-model second opinion (Phase 3.5) — skipped в этом office-hours session, premises clear enough.
|
||||
|
||||
---
|
||||
|
||||
## Architecture (Approach C — Hybrid Template + Per-Field LLM)
|
||||
|
||||
### High-level flow
|
||||
|
||||
```
|
||||
Admin opens DictionaryEditorDialog
|
||||
↓
|
||||
[Templates panel: spacecraft / ground-station / antenna / ...]
|
||||
↓ pick "spacecraft"
|
||||
↓
|
||||
Template loaded: skeleton schema c 4 base fields (code, name, type, country)
|
||||
Admin sees Monaco editor pre-filled
|
||||
↓
|
||||
Admin clicks [+ AI suggest field]
|
||||
↓
|
||||
Dialog: "Опиши поле в одну фразу"
|
||||
"Орбита — апогей, перигей, наклонение"
|
||||
↓
|
||||
POST /api/v1/ai/suggest-field
|
||||
request: { schemaContext: {...}, prompt: "Орбита: апогей..." }
|
||||
↓
|
||||
LLM call (single field, small context ~500 tokens)
|
||||
↓
|
||||
Response: {
|
||||
fieldName: "orbit",
|
||||
schema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
apogeeKm: {type: "number"},
|
||||
perigeeKm: {type: "number"},
|
||||
inclinationDeg: {type: "number", minimum: 0, maximum: 180}
|
||||
}
|
||||
}
|
||||
}
|
||||
↓
|
||||
Admin sees diff в Monaco (current vs +suggested field)
|
||||
Admin: Accept / Edit / Reject
|
||||
↓
|
||||
Existing draft workflow (no change downstream)
|
||||
```
|
||||
|
||||
### Components
|
||||
|
||||
```
|
||||
Admin clicks [+ AI suggest field]
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────┐
|
||||
│ ordinis-admin-ui │
|
||||
│ TemplatePicker.tsx (NEW) │
|
||||
│ AiFieldSuggestPanel.tsx (NEW) │
|
||||
│ Monaco editor (existing) │
|
||||
└──────────────────┬───────────────────┘
|
||||
│ POST /api/v1/ai/suggest-field
|
||||
▼
|
||||
┌──────────────────────────────────────┐
|
||||
│ ordinis-rest-api │
|
||||
│ AiSchemaController (NEW) │
|
||||
│ AiSchemaService (NEW) │
|
||||
│ ├─ template loader │
|
||||
│ ├─ per-field prompt builder │
|
||||
│ │ (existing schema → context) │
|
||||
│ ├─ LLM adapter call (OpenAI-compat│
|
||||
│ │ HTTP, vLLM/Ollama, 7B) │
|
||||
│ ├─ response parser (JSON extract) │
|
||||
│ └─ SchemaValidator (existing) │
|
||||
└──────────────────┬───────────────────┘
|
||||
│ field schema or 422
|
||||
▼
|
||||
Frontend Monaco preview (diff view)
|
||||
│
|
||||
▼
|
||||
Standard DraftService flow
|
||||
```
|
||||
|
||||
### Template registry
|
||||
|
||||
`ordinis-cuod-bundle/src/main/resources/templates/`:
|
||||
|
||||
```
|
||||
spacecraft.template.json # 4 base fields: code, name, type, country
|
||||
ground-station.template.json # координаты + оператор
|
||||
antenna.template.json # диаметр + диапазоны
|
||||
frequency-band.template.json # min/max МГц + band code
|
||||
operator.template.json # имя + страна + контакт
|
||||
ALL.template.json # empty starter с x-id-source hint
|
||||
```
|
||||
|
||||
Каждый — JSON Schema fragment с metadata header:
|
||||
```json
|
||||
{
|
||||
"$comment": "Template: spacecraft (ЦУОД)",
|
||||
"x-template-name": "Космический аппарат",
|
||||
"x-template-description": "Скелет для справочника КА — добавь fields через AI suggest или вручную",
|
||||
"type": "object",
|
||||
"x-id-source": "code",
|
||||
"required": ["code"],
|
||||
"properties": {
|
||||
"code": {"type": "string", "x-unique": true, "description": "уникальный код КА"},
|
||||
"name": {"type": "object", "x-localized": true},
|
||||
"type": {"type": "string", "x-references": "satellite_type.code"},
|
||||
"country": {"type": "string", "x-references": "country.code"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `AiSchemaService.suggestField(existingSchema, prompt)`
|
||||
|
||||
System prompt fragment:
|
||||
```
|
||||
Ты помогаешь админу справочников ДЗЗ строить JSON Schema fields.
|
||||
Получаешь: existing schema (текущие поля), prompt (описание нового поля).
|
||||
Возвращаешь: ОДНО поле — fieldName + schema.
|
||||
|
||||
Используй:
|
||||
- x-localized: true для локализованных текстов
|
||||
- x-references: "dict.field" для FK на другой справочник
|
||||
- x-unique: true для бизнес-ключей
|
||||
- format: date / date-time / email / uri где уместно
|
||||
|
||||
НЕ ИЗОБРЕТАЙ:
|
||||
- GOST коды или иные standardized codes
|
||||
- Бизнес-validations (только structural type/format/min/max)
|
||||
- Имена существующих справочников (если не уверен в имени target dict — verbal hint без x-references)
|
||||
```
|
||||
|
||||
Few-shot examples (3-5 пар):
|
||||
```
|
||||
Existing: {code, name, country}
|
||||
Prompt: "Орбита — апогей, перигей, наклонение"
|
||||
Output: {
|
||||
"fieldName": "orbit",
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"apogeeKm": {"type": "number", "description": "Апогей, км"},
|
||||
"perigeeKm": {"type": "number", "description": "Перигей, км"},
|
||||
"inclinationDeg": {"type": "number", "minimum": 0, "maximum": 180}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### LLM stack
|
||||
|
||||
| Tier | Model | Hardware | Cost/month |
|
||||
|---|---|---|---|
|
||||
| **Prod (recommended)** | `Qwen2.5-Coder-7B-Instruct` (or `Qwen2.5-7B-Instruct`) | 1 × A10 24GB (or shared с другими тенантами в vLLM) | ~$200 (cloud) или existing internal GPU |
|
||||
| **Dev/Staging** | `qwen2.5-coder:7b-instruct` via Ollama | Dev laptop or CPU server | $0 |
|
||||
| **Fallback (off by default)** | OpenAI GPT-4o-mini | External API | ~$0.002 per request × 100/day = $6/mo per customer |
|
||||
|
||||
**Per-field calls типичны ~500 tokens prompt + ~200 tokens response = ~700 tokens total.** На 7B model и A10 — sub-second latency.
|
||||
|
||||
### Graceful degradation
|
||||
|
||||
| State | What happens |
|
||||
|---|---|
|
||||
| LLM endpoint reachable | Full AI suggest experience |
|
||||
| LLM endpoint 503/timeout | UI shows banner "AI временно недоступен" + AI button disabled. **Templates still work** — admin keeps editing manually in Monaco. |
|
||||
| Global feature flag `ordinis.ai.enabled=false` | No AI button render. Templates available. |
|
||||
| Per-customer license disabled | Same as flag off (per-tenant in v2 multi-tenant) |
|
||||
|
||||
---
|
||||
|
||||
## Approaches Considered
|
||||
|
||||
### Approach A — Templates only (no LLM)
|
||||
|
||||
- **Что:** ~20 curated template schemas в bundle, admin picks → Monaco edit
|
||||
- **Effort:** 1-2 days
|
||||
- **Pros:** No LLM dep. Ships next week. Zero ops. Solves ~80% case (admin variation existing template)
|
||||
- **Cons:** Не differentiator. Custom non-template schemas = manual.
|
||||
|
||||
### Approach B — LLM whole-schema gen (v1 doc)
|
||||
|
||||
- **Что:** Admin types «справочник КА с типом, страной, орбитой» → LLM генерит ВСЮ schema → admin reviews/edits
|
||||
- **Effort:** 5-7 days
|
||||
- **Pros:** Demo wow. Real differentiator. Single LLM call per schema.
|
||||
- **Cons:** Whole schema in LLM context = larger prompt = 32B+ model = higher GPU bar. Hallucinations harder to localize (whole schema affected). Larger blast radius если model drifts.
|
||||
|
||||
### Approach C — Hybrid (RECOMMENDED, v2 chosen)
|
||||
|
||||
- **Что:** Template skeleton + per-field LLM fill-in
|
||||
- **Effort:** 3-4 days
|
||||
- **Pros:**
|
||||
- Smaller LLM context (single field, ~500 tokens) → 7B model достаточно → consumer GPU OK
|
||||
- Hallucinations localized per-field (admin reviews individually, not whole schema)
|
||||
- Templates always work — graceful degradation when LLM down
|
||||
- Compounds с marketplace: templates → bundles directly
|
||||
- **3-4 days effort vs 5-7d for B** — ships earlier, measure usage earlier
|
||||
- **Cons:**
|
||||
- UX more granular (per-field clicks vs one-shot whole-schema)
|
||||
- Demo less impressive (smaller per-call generation, not "magical")
|
||||
|
||||
---
|
||||
|
||||
## Recommended Approach: C (Hybrid)
|
||||
|
||||
**Reasoning:**
|
||||
1. **Solves P2 (GPU constraint):** 7B model on A10 (or shared instance) practical для гос-клиентов с modest GPU budget
|
||||
2. **Solves P5 (no demand evidence):** Smaller commitment, ship in 1 sprint, measure usage before expanding. If admins не используют AI button → roll back с minimal sunk cost
|
||||
3. **Engineering preference «minimal diff»:** Start narrow, expand if used
|
||||
4. **Compounds with marketplace:** Templates already structured for bundle export
|
||||
5. **Risk asymmetry:** If AI removed, templates still ship → graceful degradation built-in, not bolted-on
|
||||
|
||||
**If P5 measurement confirms strong demand (P50 > 20 min author time) AND P2 GPU provisioned with capacity headroom → upgrade to Approach B as v2.** Existing template + per-field UX stays as fallback.
|
||||
|
||||
---
|
||||
|
||||
## Implementation plan
|
||||
|
||||
### Phase 0 (prerequisite — 2 weeks)
|
||||
|
||||
**P5 measurement.** Before any code, ship metric:
|
||||
```java
|
||||
// ordinis-rest-api/src/main/java/.../service/DictionaryDefinitionService.java
|
||||
// New @Timed annotation на createDraft endpoint, tag schema_size_kb
|
||||
@Timed(value = "admin.schema_authoring.seconds", description = "Time from first draft create to review submission per dict")
|
||||
```
|
||||
|
||||
Plus admin-ui telemetry: `t_dialog_open → t_submit` per session.
|
||||
|
||||
Baseline для 2 weeks на v2.14.0 prod. Если P50 < 10 мин → defer feature. Если > 20 мин → proceed.
|
||||
|
||||
### Phase 1 (~3-4 days, 6 steps)
|
||||
|
||||
| Step | Effort | Notes |
|
||||
|---|---|---|
|
||||
| 1. Template loader + 5 curated templates в `ordinis-cuod-bundle/templates/` | 4h | JSON files, no logic |
|
||||
| 2. `TemplatePicker.tsx` — list templates, pick → load skeleton в Monaco | 3h | Standard React |
|
||||
| 3. `LlmAdapter` (OpenAI-compat HTTP, circuit breaker, timeout 10s) | 3h | BouncyCastle-free, simple HttpClient |
|
||||
| 4. `AiSchemaService.suggestField()` + 5 few-shot examples | 4h | System prompt + parse JSON extract |
|
||||
| 5. `AiSchemaController` POST `/ai/suggest-field` + RBAC INTERNAL+ + rate limit 30/min | 2h | Standard controller |
|
||||
| 6. `AiFieldSuggestPanel.tsx` — button, prompt input, diff preview в Monaco, accept/edit/reject | 6h | UX core, Monaco diff integration |
|
||||
| **Tests** (10 cases) | 6h | testcontainers + RTL |
|
||||
| **Docs** (admin guide + ops runbook GPU) | 2h | docs/user-guide/ai-schema.md |
|
||||
| **Total Phase 1** | **~30h (3-4 days)** | within ~1 sprint |
|
||||
|
||||
### Phase 2 (conditional — only if usage > 5 admin'ов/week)
|
||||
|
||||
- Approach B upgrade — whole-schema generation в дополнение к per-field
|
||||
- More templates (~15-20 total covering common ДЗЗ cases)
|
||||
- Fine-tuning local model на ЦУОД historical schema corpus (if quality bar не достигнут few-shot'ом)
|
||||
|
||||
---
|
||||
|
||||
## Test plan
|
||||
|
||||
| # | Test | Type | Critical? |
|
||||
|---|---|---|---|
|
||||
| 1 | Template loader returns 5 templates with valid JSON Schema | unit | — |
|
||||
| 2 | TemplatePicker → schema injected в Monaco | RTL | — |
|
||||
| 3 | `suggestField` happy path: orbit prompt → valid object schema | integration | — |
|
||||
| 4 | `suggestField` LLM returns invalid JSON → 422 | integration | — |
|
||||
| 5 | `suggestField` validates against meta-schema → 422 if invalid | integration | — |
|
||||
| 6 | Rate limit 30/min per user → 429 | integration | — |
|
||||
| 7 | LLM timeout 10s → 504 + circuit breaker tracks failure | integration | — |
|
||||
| 8 | Circuit breaker: 10 fails в minute → 503 для 5 min | integration | **YES** |
|
||||
| 9 | LLM disabled (flag off) → endpoint 404, frontend hides AI button | integration | — |
|
||||
| 10 | **Graceful degradation: LLM down, templates still load and edit** | integration | **YES** |
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Какой 7B model?** Qwen2.5-Coder-7B vs Qwen2.5-7B-Instruct vs Llama-3.2-7B? Quick eval нужен на 20 few-shot test cases ДЗЗ domain. **Defer to GPU prereq step.**
|
||||
|
||||
2. **Где сидит vLLM?** Existing GPU node в k8s? Если нет — defer полностью или provision new node. **Verify в P0 prereq.**
|
||||
|
||||
3. **Rate limit per user vs per tenant?** Per user 30/min для v1. Per tenant aggregation — v2 multi-tenant.
|
||||
|
||||
4. **Audit log marking «AI assisted»?** Yes — add `_meta.aiAssisted: true` flag в schema metadata. Compliance trail.
|
||||
|
||||
5. **Localized prompts (ru vs en)?** Admin types на ru typically. System prompt expects ru. v2: detect language, adapt few-shot.
|
||||
|
||||
6. **Template versioning?** Templates bundled с ordinis-cuod-bundle, semver follows bundle. Updates через bundle upgrade (см. `dictionary-marketplace.md`).
|
||||
|
||||
---
|
||||
|
||||
## Distribution plan
|
||||
|
||||
Feature ships as part of **ordinis backend + ordinis-admin-ui**. No new artifact:
|
||||
- `LlmAdapter` config через env vars (`ORDINIS_AI_ENDPOINT`, `ORDINIS_AI_MODEL`, etc.)
|
||||
- Templates в existing `ordinis-cuod-bundle.jar`
|
||||
- Frontend новые components в existing `ordinis-admin-ui` chunk
|
||||
|
||||
**Helm values addition:**
|
||||
```yaml
|
||||
ai:
|
||||
enabled: false # gated by license + GPU availability
|
||||
endpoint: ""
|
||||
model: "qwen2.5-coder:7b-instruct"
|
||||
```
|
||||
|
||||
Deployment = standard helm upgrade. No new pods (vLLM separate concern — assumed pre-existing infra).
|
||||
|
||||
---
|
||||
|
||||
## Success criteria
|
||||
|
||||
**Quantitative (measured Phase 0 baseline + after Phase 1 ship):**
|
||||
- P50 admin schema authoring time: baseline → -50% (target 5-8 min)
|
||||
- AI suggest acceptance rate: ≥40% (admin accepts AI suggestion without edits) for v1 success
|
||||
- LLM error rate: <5% (parse + meta-schema failures combined)
|
||||
- Circuit breaker activations: ≤1/day prod
|
||||
|
||||
**Qualitative:**
|
||||
- Customer interview after 1 month: «использую регулярно» from ≥2 admin'ов
|
||||
|
||||
**Failure criteria (rollback):**
|
||||
- AI button used <1×/week per admin across 2 weeks → feature roll back
|
||||
- LLM cost (if external API) > $50/month per customer → flag off
|
||||
|
||||
---
|
||||
|
||||
## What I noticed about how you think
|
||||
|
||||
(From this `/office-hours` session 2026-05-12)
|
||||
|
||||
- **Ты сразу downgrade'нул scope** от «AI generates whole schema» (founder polish version) на «templates + per-field AI» (hybrid) когда я представил Approach C. Это не attachment to your earlier doc — это openness к correction. **Это редко.**
|
||||
- **Когда я предложил skip Phase 3.5** (cross-model second opinion) — ты выбрал proceed без него. Premises вам clear enough, no need for ceremony. **Sign of confident decision-making, не perfectionism paralysis.**
|
||||
- **«согласен с рекомендациями»** vs detail-by-detail negotiation = trust в analysis или impatience? Likely former — patterns этой сессии (5 MRs shipped, prod deploy, 3 design docs) показывают delegating-when-trust-built behavior.
|
||||
- **P5 measurement step** ты accepted without pushback — это founder maturity. Большинство build first, measure later. Ты accepting measure-first приоритет = real bias toward evidence.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- Companion: `dictionary-marketplace.md` — bundle catalog (next /office-hours candidate, compounds с AI assist)
|
||||
- v1 supersededy: this same path was «whole-schema LLM» originally, downgraded to «hybrid template + per-field» после Phase 4 alternatives generation
|
||||
- Office-hours design doc: `~/.gstack/projects/claude/zimin-main-design-ai-schema-assist-20260512.md` (generated separately)
|
||||
@@ -0,0 +1,338 @@
|
||||
# Design: Bundle Hygiene (formerly "Dictionary Marketplace")
|
||||
|
||||
**Author:** zimin.an
|
||||
**Date:** 2026-05-12
|
||||
**Status:** PROPOSED v2 (post `/office-hours` + cross-model challenge)
|
||||
**Supersedes:** v1 «Dictionary Marketplace» (Approach B: full registry + browse UI + dry-run preview + install dialog, 10-14d)
|
||||
**Scope:** ДЗЗ domain, single-tenant (Approach C-revised — bundle format spec + publisher CLI, NO consumer UI)
|
||||
**Sprint estimate:** **2-3 days CC**, hard cap. Anything beyond requires new design doc.
|
||||
**Blockers:** none
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
v1 предлагал full marketplace (Nexus registry + browse UI + install dialog + dry-run preview + multi-tenancy + dep resolution) за 10-14 days. После `/office-hours` premise challenge + cross-model challenge — **v1 был over-scoped для N=1 customer** (ЦУОД anchor only сегодня).
|
||||
|
||||
**Revised:** Formalize bundle format spec + publisher CLI. **No consumer-side UI.** This is **bundle hygiene**, не marketplace.
|
||||
|
||||
**Why bundle hygiene matters at N=1:**
|
||||
1. **Java team handoff** требует documented manifest spec в любом случае — без этого new engineers не знают как добавить новый dict без полного code review pipeline
|
||||
2. **Sales artifact** — demo prospects «ЦУОД bundle is independently signed/versioned, here's how you'd ship your own» без shipping консьюмерского UI
|
||||
3. **Optionality** — когда customer #2 signs, bundle format уже defined, можно ship UI quickly (consumer UI = 5-7d incremental)
|
||||
|
||||
**What this design explicitly is NOT:**
|
||||
- ❌ Browse catalog UI
|
||||
- ❌ Install dialog в admin-ui
|
||||
- ❌ Dry-run preview UI
|
||||
- ❌ Public marketplace
|
||||
- ❌ Bundle dep resolution / topological install
|
||||
- ❌ Multi-tenant Nexus namespacing
|
||||
|
||||
**Anti-feature:** если scope expands beyond 2-3 days hard cap → STOP, re-design doc. Don't let founder weekend-prototyping pull this into Approach B by accident.
|
||||
|
||||
---
|
||||
|
||||
## Premises (validated через `/office-hours` 2026-05-12, including cross-model challenge)
|
||||
|
||||
**P1.** Marketplace UI/registry **value scales с N customers.** At N=1 → near-zero. Build **only когда customer #2 LOI/contract стадия с extensibility language**.
|
||||
|
||||
**P2.** Bundle **format + signing + manifest** valuable AT N=1 — separable concern от marketplace UI. Doubles as Java handoff spec + sales demo.
|
||||
|
||||
**P3.** ЦУОД go-live (anchor v1) **уже live на v2.14.0 prod.** Дальнейший development must support handoff readiness, не speculative future features.
|
||||
|
||||
**P4.** **No customer #2 named with deadline as of 2026-05-12.** Sales conversations ongoing (Альтум integration в process, гидрометео preliminary discussions). Marketplace UI build now = infrastructure ahead of validated need.
|
||||
|
||||
**P5.** **«Defer entirely» (Approach A) was directionally right но oversold.** Cross-model challenge revealed: pure defer assumes founder won't drift to greenfield work. Pre-committed scoped C contains damage better than aspirational pure-defer.
|
||||
|
||||
**P6.** **Bundle hygiene work ROI положителен независимо** от marketplace decision: даже если marketplace UI shipped'нется только в Q4 2026 или никогда — formal bundle spec нужен для Java handoff (target 2-3 months).
|
||||
|
||||
**P7.** **Hard cap 2-3 days CC.** No scope creep. If пилот results in «we should add UI now», that's NEW design doc, new /office-hours, fresh decision.
|
||||
|
||||
---
|
||||
|
||||
## What ships в v2 (bundle hygiene, 2-3 days)
|
||||
|
||||
### 1. Bundle manifest format spec
|
||||
|
||||
`ordinis-cuod-bundle/manifest.yaml` (NEW file, replaces ad-hoc Maven module structure):
|
||||
|
||||
```yaml
|
||||
apiVersion: ordinis.io/v1
|
||||
kind: Bundle
|
||||
metadata:
|
||||
id: ru.cuod.dzz-ground-segment
|
||||
name: ЦУОД ДЗЗ — наземный сегмент
|
||||
version: 1.0.0 # semver, follows Maven artifact version
|
||||
description: |
|
||||
40 dictionaries для управления наземным сегментом ДЗЗ:
|
||||
КА, типы КА, наземные станции, антенны, частотные диапазоны.
|
||||
domain: dzz
|
||||
scope: PUBLIC
|
||||
author: ЦУОД team
|
||||
license: proprietary
|
||||
signature: ed25519:base64... # added by publisher CLI
|
||||
|
||||
dictionaries:
|
||||
# Existing list from ordinis-cuod-bundle/src/main/resources/bundles/cuod/
|
||||
- name: spacecraft
|
||||
schemaVersion: 1.0.0
|
||||
scope: PUBLIC
|
||||
schemaRef: dictionaries/spacecraft/schema.json # relative path
|
||||
sampleDataRef: dictionaries/spacecraft/sample.json # optional
|
||||
- name: satellite_type
|
||||
schemaVersion: 1.0.0
|
||||
scope: PUBLIC
|
||||
schemaRef: dictionaries/satellite_type/schema.json
|
||||
# ... 38 more
|
||||
```
|
||||
|
||||
**Migration:** Existing ad-hoc `ordinis-cuod-bundle/src/main/resources/bundles/cuod/` structure refactored to match manifest. **No behavior change** — bundle still loaded via existing Maven module pattern. Just **adds spec on top**.
|
||||
|
||||
### 2. Publisher CLI (`ordinis-bundle-publisher` Maven plugin)
|
||||
|
||||
```bash
|
||||
mvn ordinis:bundle-publish \
|
||||
-Dbundle.path=ordinis-cuod-bundle \
|
||||
-Dnexus.url=https://nexus.corp/ordinis-bundles/private/cuod \
|
||||
-Dsigning.key=$BUNDLE_SIGNING_KEY
|
||||
```
|
||||
|
||||
Steps плагина:
|
||||
1. Read `manifest.yaml`
|
||||
2. Validate каждый `schemaRef` is valid JSON Schema 7 via existing `SchemaValidator`
|
||||
3. Compute bundle archive (tar.gz of manifest + schemas + sample data)
|
||||
4. Sign archive с ed25519 key
|
||||
5. Upload to corp Nexus с canonical path `<bundle-id>/<version>/bundle.tar.gz`
|
||||
6. Generate `bundle.sig` adjacent
|
||||
|
||||
**Reuses:** existing Maven publish patterns в repo, corp Nexus credentials через GitLab CI variables.
|
||||
|
||||
### 3. Signature verification helper
|
||||
|
||||
`BundleSignatureVerifier.java` в new module `ordinis-bundle-spec` (lightweight):
|
||||
|
||||
```java
|
||||
public class BundleSignatureVerifier {
|
||||
public boolean verify(byte[] archive, byte[] signature, PublicKey publicKey);
|
||||
}
|
||||
```
|
||||
|
||||
**Не consumed в v2** (no consumer side). Built **as documentation artifact** для future marketplace work — defines verification contract.
|
||||
|
||||
### 4. Documentation
|
||||
|
||||
`docs/user-guide/bundle-authoring.md`:
|
||||
- Bundle structure spec (manifest.yaml format)
|
||||
- How to publish (mvn command + Nexus setup)
|
||||
- Versioning semantics (semver, breaking change rules)
|
||||
- Schema authoring conventions reused from existing JavaDoc patterns
|
||||
|
||||
`docs/integration/bundle-format-spec.md`:
|
||||
- Formal spec для Java team handoff
|
||||
- Manifest schema definition
|
||||
- Signing contract
|
||||
- Future consumer protocol (placeholder)
|
||||
|
||||
---
|
||||
|
||||
## What does NOT ship в v2 (explicit non-goals)
|
||||
|
||||
| Feature | Why not |
|
||||
|---|---|
|
||||
| ❌ Admin UI «browse bundles» | No customer #2 with browse use case |
|
||||
| ❌ Install dialog в admin-ui | Customer-specific deployment = helm + values, не runtime install |
|
||||
| ❌ Dry-run preview UI | No install operation to preview |
|
||||
| ❌ Multi-tenant Nexus namespacing | N=1 customer, single namespace OK |
|
||||
| ❌ Dep resolution / topological install | Single bundle, no inter-bundle deps yet |
|
||||
| ❌ Public catalog | Customer #2+ + community curation = v3 |
|
||||
| ❌ Bundle uninstall flow | No install in admin UI → no uninstall |
|
||||
| ❌ Bundle update / migration scripts | Manual helm upgrade pattern remains |
|
||||
|
||||
**Anti-feature commitment:** If during 2-3d implementation эти items появляются «just a bit more», STOP. Write new design doc, fresh /office-hours, fresh approval.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
ordinis-cuod-bundle/ ordinis-bundle-spec/ (NEW module)
|
||||
├── manifest.yaml ← NEW ├── BundleManifest.java
|
||||
├── pom.xml ├── BundleSignatureVerifier.java
|
||||
└── src/main/resources/ └── pom.xml
|
||||
└── bundles/cuod/
|
||||
├── dictionaries/ ordinis-bundle-publisher/ (NEW Maven plugin)
|
||||
│ ├── spacecraft/ ├── BundlePublishMojo.java
|
||||
│ │ ├── schema.json ├── ManifestValidator.java
|
||||
│ │ └── sample.json └── pom.xml
|
||||
│ └── ... (39 more)
|
||||
└── (existing Maven structure)
|
||||
|
||||
│
|
||||
│ mvn ordinis:bundle-publish
|
||||
▼
|
||||
Corp Nexus
|
||||
ordinis-bundles/private/cuod/
|
||||
└── ru.cuod.dzz-ground-segment/
|
||||
└── 1.0.0/
|
||||
├── bundle.tar.gz
|
||||
└── bundle.sig
|
||||
```
|
||||
|
||||
No new pods. No runtime changes к ordinis-app / ordinis-admin-ui. Only build-time tooling + Nexus artifacts.
|
||||
|
||||
---
|
||||
|
||||
## Approaches Considered
|
||||
|
||||
### Approach A — Defer entirely (0 days now)
|
||||
|
||||
- **Pros:** Zero sunk cost. Capacity goes to AI metric work / prod hardening / customer #2 acquisition.
|
||||
- **Cons:** Java handoff still needs bundle format spec eventually — built later under deadline pressure. Pure defer ~20% adherence (cross-model challenge): founder likely drifts to greenfield marketplace prototyping anyway.
|
||||
|
||||
### Approach B — Full marketplace v1 (10-14 days, v1 doc original)
|
||||
|
||||
- **Pros:** Real differentiator if customer #2 ships. Demo wow factor.
|
||||
- **Cons:** Infrastructure ahead of validated need. Spec drift между hypothetical и real customer #2 requirements. 10-14d sunk cost if N never grows. Triggers Brooks's «is this solving real problem или one we created?»
|
||||
|
||||
### Approach C-revised — **Bundle hygiene** (2-3 days, CHOSEN)
|
||||
|
||||
- **Pros:**
|
||||
- Java handoff readiness (must-have anyway, due in 2-3 months)
|
||||
- Sales demo asset («signed versioned bundle artifact, here's our hygiene story»)
|
||||
- Contains founder's marketplace energy в legitimate scope-bounded work
|
||||
- 5× cheaper than B, не «do nothing» of A
|
||||
- Future-leverage: when customer #2 needs UI, format already defined → UI 5-7d вместо 10-14d
|
||||
- **Cons:**
|
||||
- «Half feature» if framed as marketplace — мы НЕ frame'им так
|
||||
- Not consumer-visible (no UI screenshot for demo deck)
|
||||
- 2-3 days still > 0 days (Approach A); risk founder drifts beyond cap
|
||||
|
||||
---
|
||||
|
||||
## Recommended Approach: **C-revised — Bundle Hygiene**
|
||||
|
||||
Rationale per `/office-hours` cross-model synthesis:
|
||||
|
||||
1. **Java handoff debt avoidance:** Manifest spec mandatory anyway when team scales beyond zimin.an. Build now while context fresh, cheaper than retro-spec under deadline.
|
||||
|
||||
2. **Sales conversation enablement:** Prospects asking «can we extend this с our own dictionaries?» get pointed at published bundle artifact + manifest spec — concrete answer без «we'll build it for you».
|
||||
|
||||
3. **Founder behavior containment:** Cross-model challenge правильно отметил: pure defer has ~20% follow-through odds. Pre-committed 2-3d scope легитимизирует bundle work, prevents weekend prototyping что expanded beyond bounds.
|
||||
|
||||
4. **Optionality preserved для customer #2:** If LOI signs Q3-Q4 — UI work spawn separate design doc, builds on this manifest, ships 5-7 days (smaller scope than ahead-of-time 10-14d).
|
||||
|
||||
5. **Engineering preference «explicit over clever»:** Manifest format defined upfront prevents implicit conventions, helpful for Java team handoff.
|
||||
|
||||
---
|
||||
|
||||
## Implementation plan
|
||||
|
||||
| Step | Effort (CC) | Notes |
|
||||
|---|---|---|
|
||||
| 1. New module `ordinis-bundle-spec` (manifest POJO + verifier) | 3h | Pure Java, no Spring dep |
|
||||
| 2. New module `ordinis-bundle-publisher` (Maven plugin) | 4h | Standard Maven plugin scaffolding |
|
||||
| 3. `manifest.yaml` for `ordinis-cuod-bundle` (1.0.0 release) | 1h | Generated from existing bundle structure |
|
||||
| 4. Migrate ad-hoc bundle structure → match manifest paths | 2h | File moves, no logic change |
|
||||
| 5. Publisher CLI: tar.gz + ed25519 sign + Nexus upload | 4h | BouncyCastle for ed25519 |
|
||||
| 6. Manifest schema validation в publisher | 2h | Reuses existing SchemaValidator |
|
||||
| 7. GitLab CI integration (publish job on bundle tag) | 2h | New `.gitlab-ci.yml` job |
|
||||
| 8. Tests (publisher Mojo + verifier + manifest parse) | 4h | unit tests, no integration tests needed |
|
||||
| 9. `docs/user-guide/bundle-authoring.md` + `docs/integration/bundle-format-spec.md` | 3h | Java handoff artifact |
|
||||
| **Total** | **~25h (2-3 days CC)** | **Hard cap. No scope creep.** |
|
||||
|
||||
**Anti-creep checkpoint:** After step 9, **STOP**. Don't add «just one consumer endpoint». Don't add «just a tiny browse list». Those are new design doc territory.
|
||||
|
||||
---
|
||||
|
||||
## Test plan
|
||||
|
||||
| # | Test | Type |
|
||||
|---|---|---|
|
||||
| 1 | Manifest parser: valid YAML → typed object | unit |
|
||||
| 2 | Manifest validation: missing required field → error | unit |
|
||||
| 3 | Publisher Mojo: happy path → archive + signature generated | unit (Mojo) |
|
||||
| 4 | Publisher Mojo: invalid schemaRef → fail fast | unit |
|
||||
| 5 | Signature verifier: signed archive → returns true | unit |
|
||||
| 6 | Signature verifier: tampered archive → returns false | unit |
|
||||
| 7 | Signature verifier: wrong public key → returns false | unit |
|
||||
| 8 | Schema validation reuse: invalid JSON Schema в dict → publisher fails | unit |
|
||||
|
||||
No integration / E2E tests — no runtime behavior changes.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Where do signing keys live?** GitLab CI variables protected? Vault? Corp HSM? → Defer to ops decision при implement.
|
||||
2. **Nexus namespace structure?** `ordinis-bundles/private/<customer>/<bundle-id>/<version>/` — fine для v1. Multi-tenant adjustments в v2.
|
||||
3. **Migration breaking?** Existing `ordinis-cuod-bundle` deployment relies on Maven module loaded at app startup. Manifest spec adds metadata, doesn't change loading. **Verify:** ordinis-app still loads bundle when manifest.yaml present но new spec module not on classpath (graceful degradation для existing v2.14.0 prod).
|
||||
|
||||
---
|
||||
|
||||
## Distribution plan
|
||||
|
||||
Publisher Maven plugin distributed как corp internal artifact в Nexus. Authors add к their bundle's `pom.xml`:
|
||||
|
||||
```xml
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>cloud.nstart.terravault.ordinis</groupId>
|
||||
<artifactId>ordinis-bundle-publisher</artifactId>
|
||||
<version>1.0.0</version>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
```
|
||||
|
||||
Then `mvn ordinis:bundle-publish` triggered by GitLab CI on tag push (e.g. `cuod-bundle-1.0.0`).
|
||||
|
||||
Bundle.tar.gz uploaded к Nexus. No deployment changes к prod ordinis cluster — bundle archive consumed только by external authors / sales demos.
|
||||
|
||||
---
|
||||
|
||||
## Success criteria
|
||||
|
||||
**Quantitative:**
|
||||
- Publisher Mojo completes < 60 sec на ЦУОД bundle (~40 dicts)
|
||||
- Signature verification < 10ms
|
||||
- Manifest validation catches 100% malformed manifests in unit tests
|
||||
|
||||
**Qualitative:**
|
||||
- Java team handoff doc references `bundle-format-spec.md` as authoritative spec
|
||||
- Sales conversations с prospects use «signed versioned bundle» as differentiator
|
||||
- After 1 month: zero «founder drifted into consumer UI work» violations (scope cap held)
|
||||
|
||||
**Failure criteria (rollback):**
|
||||
- Publisher Mojo bugs surface в > 2 releases of ordinis-cuod-bundle → reconsider as build-time vs separate plugin
|
||||
- No sales conversation uses bundle artifact within 3 months → format спека correct, distribution wrong (rare; spec stays anyway)
|
||||
|
||||
---
|
||||
|
||||
## What I noticed about how you think (during /office-hours)
|
||||
|
||||
- **Initial position «build full marketplace» (v1 doc) → accepted «defer entirely» recommendation от me → accepted «bundle hygiene» revision после cross-model challenge.** Two reversals in one session = openness to push-back. Это редко — большинство founders defend original scope harder.
|
||||
|
||||
- **«давай как рекомендуешь»** = trust calibrated на анализ + cross-model agreement. You didn't dismiss subagent challenge OR over-weight it — accepted synthesis as honest reading.
|
||||
|
||||
- **Cross-model challenge value:** subagent found 4 things I missed (sales artifact, Java handoff, founder psychology drift risk, «treading water» mis-framing). Без cross-model call recommendation бы вышел overly conservative. **Session demonstrates value of /office-hours Phase 3.5 even когда первый instinct = skip it.**
|
||||
|
||||
- **Anti-scope-creep pre-commitment** — `Hard cap 2-3 days. No scope creep.` line — это founder-level discipline, не engineering preference. Most founders write «target effort 2-3 days» что effectively unconstrained.
|
||||
|
||||
---
|
||||
|
||||
## Reviewer Concerns
|
||||
|
||||
(Spec review skipped в этой session due to context length. Recommend `/plan-eng-review` separately когда implement starts, especially on:)
|
||||
- Module dependency graph (`ordinis-bundle-spec` standalone OK? circular dep risk если consumed by ordinis-app later?)
|
||||
- Nexus auth credentials lifecycle (GitLab CI vs Vault tradeoff)
|
||||
- Compatibility: existing prod v2.14.0 deployment should keep working без bundle-spec module
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- v1 SUPERSEDED (был «Dictionary Marketplace», full UI + registry + 10-14d). Reasoning for v1 → v2 revision lives в section «Approaches Considered».
|
||||
- Companion: `ai-schema-assist.md` — separate /office-hours design, AI assist authorship. **Sequencing:** AI assist baseline measurement (P5 prereq) before this. This work doesn't compete для bandwidth — separable.
|
||||
- Office-hours formal doc: `~/.gstack/projects/claude/zimin-main-design-bundle-hygiene-20260512.md` (cross-skill discoverable)
|
||||
@@ -0,0 +1,415 @@
|
||||
# Design: Redis Projection FK Resolution
|
||||
|
||||
**Author:** zimin.an
|
||||
**Date:** 2026-05-12
|
||||
**Status:** PROPOSED v2.1 (round 2 `/plan-eng-review` fixes applied)
|
||||
**Supersedes:**
|
||||
- v1 (2026-05-12 first draft, before eng review found `LineageIndexService` reuse opportunity)
|
||||
- v2 (2026-05-12 после round 1 review — major scope reduction + critical safety mitigations)
|
||||
**Sprint estimate:** **4-5 рабочих дней** (~30.5h CC+gstack)
|
||||
**Blockers:** none, рекомендация defer'a сохраняется — implement только когда конкретный dict упрётся в latency SLA
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
Сейчас Redis projection (writer module) пишет per-locale flattened JSON записи в Redis для dict'ей с `redis_projection_enabled=true`. **FK ссылки** (`x-references: "dict.field"`) хранятся как raw FK value (`satellite_type: "OPERATIONAL"`). Read-api делает N+1 lookup чтобы получить human label (`Действующий`).
|
||||
|
||||
**Предложение:** при write resolve'ить FK label'ы и сохранять рядом с FK value в `_resolved` ключе. Per-FK opt-in через `x-resolve-label: true`. **Cascade invalidation reusing existing `LineageIndexService`** (no new Redis reverse index).
|
||||
|
||||
**Win:** один Redis GET вместо N+1, latency 1ms вместо N×1ms (для записей с 5-10 FK).
|
||||
|
||||
**Risk:** консистентность projection requires careful invalidation strategy, иначе projection stale до next write referencing'а dict'и. **Mitigations** добавлены в этой ревизии (см. § Critical safety).
|
||||
|
||||
---
|
||||
|
||||
## Изменения от v1 (после eng review)
|
||||
|
||||
| # | v1 | v2 (now) | Why |
|
||||
|---|---|---|---|
|
||||
| 1 | Новый Redis reverse index `fk:<refDict>:<bk>` | **Reuse `LineageIndexService.findRecordDependents()`** | DRY: PG index уже existing, single source of truth |
|
||||
| 2 | "Subscribe to OutboxEvents" | **Extend `RecordEventListener` в projection-writer** | Корректное terminology — flow идёт через Kafka topics |
|
||||
| 3 | Fan-out 1000 records — pipeline и ОК | **Batch cap 500 + queue depth metric + alert** | Realistic worst case 50k (country dict) saturate'нет consumer |
|
||||
| 4 | «Eventual consistency few seconds» | **Explicit SLO: P95 < 60s для <10k, P95 < 300s для >10k** + `X-Projection-Updated-At` header в read-api | UX bug когда админ видит mixed state |
|
||||
| 5 | Cycle detection: «only direct FK» (in prose) | **Explicit constraint + test: nested FK resolution forbidden** | Защита от future «улучшений» |
|
||||
| 6 | Storage: 10k × 1KB = +300MB | **Fixed formula: × locale count × FK count = realistic +450MB-1.5GB** | Per-locale multiplier compound'ил, не учтён |
|
||||
| 7 | `@Cacheable` FkResolver на 5 min | **No Spring cache** — projection-writer уже триггерит cascade, кэш дал бы stale при write | Two caching layers were unsafe |
|
||||
| 8 | No test plan | **15-test plan with regression test, race test, E2E через Kafka, perf test** | Был только перечень "Integration tests" |
|
||||
| 9 | No rollback strategy | **Toggle off → eventual cleanup on next upsert** (не background job) | Меньше moving parts |
|
||||
| 10 | No kill-switch | **Global feature flag `ordinis.projection.fk-resolution.enabled` + per-dict + per-FK** | 3 уровня контроля для prod safety |
|
||||
|
||||
**Round 2 fixes (v2 → v2.1):**
|
||||
|
||||
| # | v2 | v2.1 (now) | Why |
|
||||
|---|---|---|---|
|
||||
| 11 | Kill switch precedence implicit | **Explicit precedence table** (write requires all 3; read requires global+dict only) | Operator clarity, lazy cleanup на per-FK toggle off |
|
||||
| 12 | `CascadeInvalidator` triggers via schema scan per event | **Step 1.5: fkTargetDicts cache + SchemaPublished invalidation** | High-RPS dicts не платят schema lookup на каждый message |
|
||||
| 13 | Bundle isolation lost в Open Question #3 | **Step 1.6: SchemaValidator reject cross-bundle x-references** | Multi-tenancy invariant защищён valid'ом |
|
||||
| 14 | `_meta.updatedAt` ambiguous (cascade vs direct) | **Explicit max(direct, cascade) policy** | Silent UX bug: header lying about staleness |
|
||||
| 15 | Test plan 15 cases | **20 cases** (+ kill switch matrix, cache invalidation, cross-bundle, `_meta.updatedAt` timing, storage cap precision) | Coverage diagram gaps closed |
|
||||
|
||||
---
|
||||
|
||||
## Текущее состояние
|
||||
|
||||
### Что есть
|
||||
|
||||
```
|
||||
RecordCreated/Updated event
|
||||
│
|
||||
▼
|
||||
Kafka topic (3 scopes)
|
||||
│
|
||||
▼
|
||||
ordinis-projection-writer/RecordEventListener
|
||||
│
|
||||
▼
|
||||
ProjectionWriter.upsert()
|
||||
│
|
||||
┌───────────────────┼───────────────────┐
|
||||
▼ ▼ ▼
|
||||
flatten(per-locale) raw key SET dictionaryIndex SET ADD
|
||||
│
|
||||
▼
|
||||
Redis SET key=record(...,locale)
|
||||
```
|
||||
|
||||
`flatten()` обрабатывает **только** `x-localized` поля. FK поля проходят как есть.
|
||||
|
||||
### Что хочется
|
||||
|
||||
При write проекции добавляется `_resolved` ключ с pre-fetched label'ами:
|
||||
|
||||
```json
|
||||
{
|
||||
"businessKey": "ISS",
|
||||
"satellite_type": "OPERATIONAL",
|
||||
"country": "RU",
|
||||
"_resolved": {
|
||||
"satellite_type": "Действующий",
|
||||
"country": "Россия"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Read-api сразу отдаёт нужный label без второго round-trip.
|
||||
|
||||
---
|
||||
|
||||
## Архитектура v2
|
||||
|
||||
### Component diagram
|
||||
|
||||
```
|
||||
Kafka event: RecordCreated{dict=spacecraft, bk=ISS}
|
||||
│
|
||||
▼
|
||||
RecordEventListener (existing)
|
||||
│
|
||||
▼
|
||||
ProjectionWriter.upsert()
|
||||
│
|
||||
┌───────────────────┼───────────────────┐
|
||||
▼ ▼ ▼
|
||||
flatten(...) FkResolver raw key SET (как сейчас)
|
||||
│ │
|
||||
│ │ для каждого x-resolve-label поля:
|
||||
│ │ - PG read replica lookup
|
||||
│ │ - inject _resolved.<field>
|
||||
│ ▼
|
||||
│ ┌──────────────┐
|
||||
│ │ PG read │
|
||||
│ │ replica │ (no cache layer!)
|
||||
│ └──────────────┘
|
||||
▼
|
||||
Redis SET key=record(...,locale)
|
||||
|
||||
|
||||
Kafka event: RecordUpdated{dict=satellite_types, bk=OPERATIONAL}
|
||||
│
|
||||
▼
|
||||
RecordEventListener (existing)
|
||||
│
|
||||
▼
|
||||
CascadeInvalidator (NEW component)
|
||||
│
|
||||
▼
|
||||
LineageIndexService.findRecordDependents() ← REUSE
|
||||
│
|
||||
▼
|
||||
paged list of (source_dict, source_bk)
|
||||
│
|
||||
▼
|
||||
enqueue invalidation batch (cap 500)
|
||||
│
|
||||
▼
|
||||
for each batch: re-fetch from PG → ProjectionWriter.upsert()
|
||||
│
|
||||
▼
|
||||
Redis pipelined batch write (Spring executePipelined)
|
||||
```
|
||||
|
||||
### Phase A: write-side FK resolver (~2 дня)
|
||||
|
||||
`FkResolver` service в `ordinis-projection-writer`:
|
||||
|
||||
```java
|
||||
@Component
|
||||
public class FkResolver {
|
||||
private final DictionaryRecordRepository recordRepo; // PG read replica
|
||||
private final DictionaryDefinitionRepository defRepo;
|
||||
|
||||
/**
|
||||
* Resolve FK value → human label per locale.
|
||||
* Performance: single PG read query per (refDict, fkValue) — sub-ms на read replica.
|
||||
* No caching by design (см. eng review #2 — stale read window).
|
||||
*/
|
||||
public Optional<String> resolveLabel(
|
||||
String refDict, // "satellite_types"
|
||||
String refField, // "code" — default businessKey
|
||||
String fkValue, // "OPERATIONAL"
|
||||
String locale // "ru"
|
||||
) { ... }
|
||||
|
||||
/** Batch resolve для N FK fields одной записи (single PG query через IN clause). */
|
||||
public Map<String, String> resolveBatch(
|
||||
List<FkRequest> requests,
|
||||
String locale
|
||||
) { ... }
|
||||
}
|
||||
```
|
||||
|
||||
Update `ProjectionWriter.flatten()`:
|
||||
- Walk schema.properties, для каждого FK поля с `x-resolve-label: true`:
|
||||
- Call `resolveBatch` (single PG query для всех FK одной записи)
|
||||
- Inject `_resolved.<field>` JSON
|
||||
- **Hard constraint:** не рекурсировать в nested objects — direct fields only
|
||||
|
||||
### Phase B: cascade invalidation (~2 дня) — **reuse LineageIndexService**
|
||||
|
||||
New `CascadeInvalidator` в `ordinis-projection-writer`:
|
||||
|
||||
```java
|
||||
@Component
|
||||
public class CascadeInvalidator {
|
||||
private final LineageIndexService lineageIndex; // REUSE
|
||||
private final ProjectionWriter writer;
|
||||
private final DictionaryRecordRepository recordRepo;
|
||||
private static final int BATCH_SIZE_CAP = 500; // hard limit
|
||||
|
||||
/** Triggered RecordEventListener'ом когда updated dict — потенциальный FK target. */
|
||||
public void onSourceRecordUpdate(String refDict, String businessKey) {
|
||||
int page = 0;
|
||||
while (true) {
|
||||
Page<RecordDependent> deps = lineageIndex.findRecordDependents(
|
||||
refDict, businessKey, allScopes(), PageRequest.of(page, BATCH_SIZE_CAP));
|
||||
if (deps.isEmpty()) break;
|
||||
|
||||
// Re-fetch dependents from PG + rewrite projection (pipelined)
|
||||
writer.batchRewriteProjections(deps.getContent());
|
||||
|
||||
cascadeInvalidationCounter.increment(deps.getNumberOfElements());
|
||||
if (!deps.hasNext()) break;
|
||||
page++;
|
||||
|
||||
// Backpressure если queue depth высокий
|
||||
if (cascadeQueueDepthGauge.get() > QUEUE_DEPTH_ALERT) {
|
||||
Thread.sleep(THROTTLE_MS);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Hook в существующий `RecordEventListener`:
|
||||
- На `RecordUpdated{dict=X}` event — после обычной upsert(), check'аем X **через cached `fkTargetDicts` set** (step 1.5). Schema-level dependency map immutable между SchemaPublished events; cache invalidate'ится по этому event'у. Это избегает `LineageIndexService.findSchemaDependents()` query на каждом message — критично для high-RPS dicts.
|
||||
- Если X ∈ fkTargetDicts → вызываем `CascadeInvalidator.onSourceRecordUpdate(X, businessKey)`
|
||||
|
||||
### Phase C: backfill + ops (~1 день)
|
||||
|
||||
CLI endpoint `POST /api/v1/admin/projections/{dict}/backfill`:
|
||||
- Paginated iterate (batch=500, sleep=100ms между batch'ами)
|
||||
- Per-page commit chunk — recoverable если interrupted
|
||||
- Lock через advisory lock в PG (one backfill per dict at a time)
|
||||
- Progress reporting через outbox event stream
|
||||
|
||||
### Phase D: observability + flags
|
||||
|
||||
3-tier kill switch:
|
||||
1. **Global** — `ordinis.projection.fk-resolution.enabled` (env var)
|
||||
2. **Per-dict** — `DictionaryDefinition.fkResolutionEnabled` (existing field паттерн, новая колонка)
|
||||
3. **Per-FK** — schema annotation `x-resolve-label: true` (default false)
|
||||
|
||||
**Precedence (explicit per round 2 review):**
|
||||
|
||||
| Layer | Write path (resolve & inject `_resolved`) | Read-api (return `_resolved` к caller'у) |
|
||||
|---|---|---|
|
||||
| Global=false | ❌ skip | ❌ ignore `_resolved` even if present (stale residue from before toggle) |
|
||||
| Global=true, Dict=false | ❌ skip | ❌ ignore (per-dict opt-out wins over residual data) |
|
||||
| Global=true, Dict=true, FK=false | ❌ skip для этого поля | ✅ return whatever's already в `_resolved` (no harm — поле там не появится) |
|
||||
| Global=true, Dict=true, FK=true | ✅ resolve & write | ✅ return `_resolved.<field>` |
|
||||
|
||||
Logic: **write requires all three true**; **read requires global + dict** (FK granularity не нужна на read — если backend перестал писать поле, `_resolved.<field>` natural выпадает при следующем upsert). Это значит turning off per-FK toggle = lazy cleanup. Turning off per-dict = immediate hide. Turning off global = panic kill для всего projection FK behavior.
|
||||
|
||||
**Metrics (Prometheus):**
|
||||
- `ordinis_projection_fk_resolved_total{dict, fk_field}`
|
||||
- `ordinis_projection_fk_resolve_miss_total{dict, fk_field, reason="not_accessible|not_found|locale_missing"}`
|
||||
- `ordinis_projection_cascade_invalidation_total{trigger_dict}`
|
||||
- `ordinis_projection_cascade_queue_depth` (gauge)
|
||||
- `ordinis_projection_fk_resolve_duration_seconds` (histogram)
|
||||
- `ordinis_projection_storage_resolved_bytes_total{dict}` (gauge)
|
||||
|
||||
**SLO targets:**
|
||||
- P95 FK resolution latency: < 5ms (PG read replica baseline)
|
||||
- P95 cascade invalidation:
|
||||
- < 60s для cascade size ≤ 10k dependents
|
||||
- < 300s для cascade size 10k-50k
|
||||
- > 50k cascade → page on-call (likely operator error)
|
||||
- Projection eventual consistency window: max 300s
|
||||
|
||||
**Read-api integration:**
|
||||
- Response header `X-Projection-Updated-At: 2026-05-12T17:30:00Z` (last upsert time per record)
|
||||
- `staleness=` query param для force-fresh read через PG fallback (debugging)
|
||||
|
||||
**`_meta.updatedAt` policy (per round 2 review — silent UX bug fix):**
|
||||
|
||||
Stored в projection JSON как `_meta.updatedAt` (sibling `_resolved`):
|
||||
- **Updated on direct upsert** (RecordCreated/Updated event для этой записи)
|
||||
- **Updated on cascade rewrite** (CascadeInvalidator пишет проекцию после изменения FK target)
|
||||
- Read-api header = `max(direct_upsert_time, cascade_rewrite_time)` = the **`_meta.updatedAt` value** as-stored (cascade overwrites if newer)
|
||||
|
||||
Why max'om: иначе админ видит «projection updated 5 минут назад» а на самом деле cascade обновил labels 3 секунды назад — silent UX bug когда юзер ждёт что прочтёт fresh данные. Stored timestamp всегда **точка последней мутации projection'а**, не direct write.
|
||||
|
||||
Storage cost: 24 bytes per record × 3 locales = 72 bytes overhead. Negligible.
|
||||
|
||||
---
|
||||
|
||||
## Critical safety (added in v2)
|
||||
|
||||
### Race conditions: concurrent target update during dependent write
|
||||
|
||||
**Scenario:** Запись A пишется в момент когда B (FK target) сам обновляется.
|
||||
|
||||
**Mitigation:** `FkResolver` читает с PG **after** projection write — если B меняется между resolution и Redis SET, мы пишем stale label. Затем cascade от B's update подхватит A и rewrite projection → eventual consistency.
|
||||
|
||||
**Test:** `ConcurrentFkUpdateTest` — два потока, поток1 пишет A, поток2 update B, проверяем что через ≤30s projection A содержит latest label B.
|
||||
|
||||
### Kafka consumer lag во время массивного cascade
|
||||
|
||||
**Scenario:** Update country=RU → 50k spacecrafts инвалидируются.
|
||||
|
||||
**Mitigation:**
|
||||
1. Batch cap **500** dependents за раз
|
||||
2. `cascadeQueueDepthGauge` exposed → alert thresholds (warn @ 1000, crit @ 5000)
|
||||
3. Throttle между batch'ами если queue depth превышает
|
||||
4. Async dispatch — cascade work не в main listener thread, отдельный `@Async` executor с bounded queue
|
||||
|
||||
**Test:** `CascadeBackpressureTest` — load 10k dependents, verify Kafka consumer lag не превышает 60s.
|
||||
|
||||
### Redis memory pressure
|
||||
|
||||
**Scenario:** Enable flag для крупного dict'a → проекции вырастут 2-3×.
|
||||
|
||||
**Mitigation:**
|
||||
1. Metric `ordinis_projection_storage_resolved_bytes_total{dict}` + alert на 80% Redis memory cap
|
||||
2. Per-record max size check — если `_resolved` blob > 50KB, skip resolution + log warning (не падать)
|
||||
3. Operational runbook: «disable per-dict flag → wait for natural eviction (24h TTL?) → cleanup»
|
||||
|
||||
### Rollback path
|
||||
|
||||
Если flag toggled off:
|
||||
- Existing `_resolved` keys остаются stale, но **harmless** — read-api ignore'ит `_resolved` когда flag=false
|
||||
- При next upsert каждой записи `_resolved` automatically dropped (write replaces full value)
|
||||
- Если нужен immediate cleanup — CLI `POST /admin/projections/{dict}/strip-resolved` (~1 час имплементации)
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **PG read replica lag.** `FkResolver` читает с replica для performance, но replica может отставать на seconds под write load. Если target dict обновлён 100ms назад, FkResolver получит stale label, cascade подхватит ~30s. Acceptable. Документировать.
|
||||
|
||||
2. **Scope-hide handling.** Если FK target dict не доступен caller'у — projection write идёт как **system user** (не user-scope). Resolution всё равно происходит, но read-api позже filter'нет `_resolved` если caller scope не допускает target dict. Implementation: `FkResolver` использует privileged read, read-api проверяет access.
|
||||
|
||||
3. **Bundle isolation.** В multi-bundle setup'е (cuod, altum, etc.) — FK может пересекать bundle boundaries? Recommendation: запретить, validation в schema editor (`x-references` поле target dict должен быть в том же bundle).
|
||||
|
||||
4. **Storage cap.** Hard limit на per-record `_resolved` blob — 50KB? Это значит до ~25 FK fields × 3 locales × ~600 chars label = 45KB. Достаточно для всех realistic case'ов.
|
||||
|
||||
---
|
||||
|
||||
## Альтернатива: skip FK resolution, use read-api JOIN (unchanged from v1)
|
||||
|
||||
Вместо pre-resolve'инга при write — read-api делает JOIN на читающей стороне (через PG или client-side). Pro: simpler, no cascade. Con: N+1 на каждое чтение, ровно то что мы хотим избежать.
|
||||
|
||||
Если RPS не упёрся в predisposed, можно отложить весь FK projection. Прагматичное правило: **включать FK resolution только когда конкретный dict упёрся в latency SLA**. Default off навсегда.
|
||||
|
||||
---
|
||||
|
||||
## Implementation plan (revised)
|
||||
|
||||
| Step | Effort (CC+gstack) | Notes |
|
||||
|---|---|---|
|
||||
| 1. Schema annotation `x-resolve-label` + JSON Schema validator | 1h | Update SchemaValidator, fail-fast on non-FK field |
|
||||
| 1.5. **fkTargetDicts cache** + SchemaPublished invalidation | 1h | Pre-compute set of dicts that являются FK target (via `LineageIndexService.findSchemaDependents` inversion). Avoid schema scan на каждый RecordUpdated event в `RecordEventListener` |
|
||||
| 1.6. **Bundle isolation enforcement** в `SchemaValidator` | 0.5h | Reject `x-references: "<targetDict>.<field>"` если target_dict в другом bundle. Single-bundle invariant — иначе multi-tenancy ломается |
|
||||
| 2. Per-dict `fk_resolution_enabled` column + migration | 1h | Liquibase, Dictionary CRUD. Default `false` для всех existing dicts (backward-compat). |
|
||||
| 3. `FkResolver` service + batch resolve method | 3h | Single PG query per record (IN clause) |
|
||||
| 4. Update `ProjectionWriter.flatten()` для `_resolved` injection | 2h | Walk schema, hard constraint on nested |
|
||||
| 5. `CascadeInvalidator` reusing `LineageIndexService` | 4h | Batch cap, queue depth gauge, async |
|
||||
| 6. `RecordEventListener` hook into `CascadeInvalidator` | 1h | Detect target dict from schemas index |
|
||||
| 7. Backfill CLI endpoint + advisory lock | 2h | Paginated, idempotent |
|
||||
| 8. Metrics + SLO alerting config (Grafana panels) | 2h | 6 new metrics, 3 alert rules |
|
||||
| 9. Read-api `X-Projection-Updated-At` header | 1h | Track via `_meta.updatedAt` в projection JSON |
|
||||
| 10. Integration tests (15 cases per coverage diagram) | 8h | testcontainers Postgres+Redis+Kafka |
|
||||
| 11. Frontend: schema editor checkbox per FK field | 2h | `DictionaryEditorDialog`, validation |
|
||||
| 12. Docs (ops runbook, schema annotation guide) | 2h | docs/ops/projection-fk-resolution.md |
|
||||
| **Total** | **~30.5h (4-5d)** | within revised budget (+1.5h после round 2 review) |
|
||||
|
||||
---
|
||||
|
||||
## Test plan (added in v2 per eng review)
|
||||
|
||||
Minimum 15 integration tests, testcontainers Postgres + Redis + Kafka:
|
||||
|
||||
| # | Test | Type | Critical? |
|
||||
|---|---|---|---|
|
||||
| 1 | Happy path: 3 FK fields, all resolve | unit | — |
|
||||
| 2 | One FK miss (orphan) → omitted from `_resolved` | unit | — |
|
||||
| 3 | Multi-locale resolution per locale array | unit | — |
|
||||
| 4 | Locale fallback (ru missing → defaultLocale en) | unit | — |
|
||||
| 5 | Schema validation: `x-resolve-label` on non-FK → error | unit | — |
|
||||
| 6 | **Regression**: write WITHOUT flag works как раньше | unit | **YES** |
|
||||
| 7 | Cascade fires on RecordUpdated of FK target | integration | — |
|
||||
| 8 | Cascade batch cap 500 enforced | integration | — |
|
||||
| 9 | Cascade queue depth metric increments | integration | — |
|
||||
| 10 | Concurrent update race → eventual consistency ≤30s | integration | **CRITICAL** |
|
||||
| 11 | Backpressure: 10k dependents → consumer lag <60s | integration | **CRITICAL** |
|
||||
| 12 | E2E через Kafka real flow: PUT → cascade → Redis read | E2E | — |
|
||||
| 13 | Backfill CLI idempotent (re-run same result) | integration | — |
|
||||
| 14 | Toggle off → next upsert removes `_resolved` | integration | — |
|
||||
| 15 | Storage cap (50KB) — large record skips resolution + warns | integration | — |
|
||||
| 16 | **Kill switch precedence matrix** — все 8 комбинаций global/dict/FK booleans (truth table, parameterized test): write + read behavior matches table в § Phase D | integration | **YES** |
|
||||
| 17 | **fkTargetDicts cache invalidation** — SchemaPublished event дропает schema-level dependency cache, next RecordUpdated пересчитывает | integration | — |
|
||||
| 18 | **Cross-bundle FK rejected** — `SchemaValidator` reject'нет `x-references` указывающий на dict из другого bundle | unit | — |
|
||||
| 19 | **`_meta.updatedAt` cascade vs direct timing** — cascade rewrite обновляет timestamp; direct upsert после cascade перезаписывает; header возвращает max | integration | — |
|
||||
| 20 | **Storage cap precision** — 50KB cap применяется per-record (NOT per-field); record с 49KB raw + 5KB `_resolved` = total 54KB → resolved skipped, raw written | integration | — |
|
||||
|
||||
---
|
||||
|
||||
## Recommendation (unchanged from v1)
|
||||
|
||||
**Defer until after v2.12.0 prod stable + 2 weeks dogfooding.** Текущий read path handles RPS, no urgency. Activate per-FK flag только когда first dict bumps into latency SLA. Build full pipeline только if 3+ dict'ам нужно.
|
||||
|
||||
**Next step (если decide go):** этот revised doc → `/plan-eng-review` round 2 → if CLEAR → CEO approval → sprint allocation.
|
||||
|
||||
---
|
||||
|
||||
## GSTACK REVIEW REPORT
|
||||
|
||||
| Review | Trigger | Why | Runs | Status | Findings |
|
||||
|--------|---------|-----|------|--------|----------|
|
||||
| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 2 (v1 + v2) | 🟢 CLEAR | v1: 11 issues + 3 critical → all closed v2. v2 round 2: 5 minor + 1 silent UX bug → all closed v2.1 |
|
||||
| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | — | — |
|
||||
| Design Review | `/plan-design-review` | UI/UX gaps | 0 | n/a | Skip — backend feature |
|
||||
|
||||
**UNRESOLVED:** 0
|
||||
**VERDICT:** 🟢 **CLEAR after v2.1 fixes.** Ready для implementation **когда** придёт время. Defer recommendation сохраняется: implement только после v2.12.0 prod stable + 2 weeks dogfooding + первый dict упрётся в latency SLA.
|
||||
Reference in New Issue
Block a user