From a3eb5f0c86196acf3fbbad0e5ac74eb058eaa7e3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=90=D0=BB=D0=B5=D0=BA=D1=81=D0=B0=D0=BD=D0=B4=D1=80=20?= =?UTF-8?q?=D0=97=D0=B8=D0=BC=D0=B8=D0=BD?= Date: Tue, 12 May 2026 17:03:52 +0000 Subject: [PATCH] =?UTF-8?q?docs(design):=20ai-schema-assist=20v2=20?= =?UTF-8?q?=E2=80=94=20/office-hours,=20Approach=20C=20hybrid?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/ai-schema-assist.md | 454 +++++++++++++++++++------------- 1 file changed, 274 insertions(+), 180 deletions(-) diff --git a/docs/design/ai-schema-assist.md b/docs/design/ai-schema-assist.md index ab32f32..cc8890c 100644 --- a/docs/design/ai-schema-assist.md +++ b/docs/design/ai-schema-assist.md @@ -2,278 +2,372 @@ **Author:** zimin.an **Date:** 2026-05-12 -**Status:** PROPOSED v1 — needs `/office-hours` for premise validation + `/plan-eng-review` +**Status:** PROPOSED v2 (post `/office-hours`) +**Supersedes:** v1 (2026-05-12 — LLM-only "Approach B", before office-hours premise validation revealed `Approach C` hybrid) **Scope:** ДЗЗ domain only (ground segment + orbital, не general-purpose AI) -**Sprint estimate:** ~5-7 days CC (1 sprint), or ~3-4 days если skip GUI polish -**Blockers:** none, но recommendation defer'a до v2.14.0 prod stable +**Sprint estimate:** **~3-4 days CC** для **Approach C hybrid** (revised down from 5-7d v1) +**Blockers:** none architectural. P5 measurement step (см. Premises) — soft prereq --- ## TL;DR -Admin Ordinis сейчас создаёт новый справочник вручную: пишет JSON Schema (поля, типы, `x-references`, `x-localized`, validations), пробрасывает локализации, продумывает bitemporal flags. Это **15-30 минут per dictionary** для опытного админа, **час+** для нового. Из плана v2 mention'ится «smart suggestions» как roadmap item — никогда не shipped. +Admin Ordinis сейчас создаёт справочник вручную — пишет JSON Schema (поля, типы, `x-references`, `x-localized`, validations), пробрасывает локализации, продумывает bitemporal flags. Это **15-30 минут per dictionary** для опытного админа, **час+** для нового. -**Предложение:** LLM-assisted authoring. Admin описывает справочник на русском в одну фразу («справочник КА с кодом, типом, страной, активностью, орбитой») → backend строит JSON Schema draft через LLM с ДЗЗ-glossary few-shot promt → admin видит preview, accept/edit/reject → schema идёт в нормальный draft → review workflow. +**Предложение (v2 — Hybrid):** Admin starts с **curated template** (e.g. «спутник», «наземная станция»), затем при добавлении каждого нового поля кликает **«AI suggest field»** → LLM предлагает structure (type/format/x-localized/x-references) для **одного поля** на базе schema name + existing fields context. Admin reviews → accept или edit → continue. -**Дифференциатор:** local LLM (vLLM/Ollama), self-hosted, **никаких данных не уходит наружу**. Critical для гос-клиентов. +**v2 vs v1:** Vместо whole-schema generation одним LLM call'ом — **template skeleton + per-field LLM fill-in**. Smaller LLM context, более reliable, lower GPU bar (7B sufficient instead of 32B), graceful degradation (templates работают без LLM). -**Win:** time-to-first-schema 30мин → 2-3мин, новый admin onboarding hour → 5мин. Demo wow-эффект для sales. +**Win:** time-to-first-schema 30мин → 5-8мин (template + 5-10 fields, каждое 30 сек AI suggest). Compounds with marketplace: templates → bundles directly. -**Risk:** LLM hallucinates fields, `x-references`, валидации. Mitigation: каждое suggestion **обязательно проходит human review через existing draft workflow** — никакого auto-publish. +**Risk:** LLM hallucinations contained per-field (smaller blast radius vs whole schema). Templates always work even если LLM down. --- -## Current state +## Premises (validated через `/office-hours` 2026-05-12) + +**P1.** AI = **time-saver**, не replacement. Schema всегда проходит human review через existing draft workflow. **No auto-publish.** Value = drop time-to-first-schema 30мин → 5-8мин. + +**P2.** **Local LLM** (vLLM/Ollama) **non-negotiable** для гос-клиентов — data не покидает perimeter. External API disabled by default, feature-flag-only для corp non-classified. **GPU verification = hard prereq** перед implementation. + +**P3.** **Scope = ДЗЗ domain only.** Few-shot training на ЦУОД bundle (5 примеров). Non-ДЗЗ customer → manually extends few-shot в свой bundle. + +**P4.** **AI predicts STRUCTURE, не SEMANTICS.** Type, format, nullability, FK ref structure, locales — да. Validation rules, GOST codes, business constraints — NET. System prompt explicit: «если не уверен — оставь пустым». + +**P5.** **Demand evidence пока нулевая** — founder-driven feature, не customer-pulled. **Prerequisite measurement step:** перед implementation добавить metric `admin_schema_authoring_time_seconds` (start = first draft creation, end = first review submission). Собрать **2 weeks baseline** на v2.14.0 prod. Если P50 < 10 мин — отложить feature (no real pain). Если P50 > 20 мин — confirms hypothesis, proceed. + +**P6.** Cross-model second opinion (Phase 3.5) — skipped в этом office-hours session, premises clear enough. + +--- + +## Architecture (Approach C — Hybrid Template + Per-Field LLM) + +### High-level flow ``` Admin opens DictionaryEditorDialog ↓ -Manually types JSON Schema in Monaco editor +[Templates panel: spacecraft / ground-station / antenna / ...] + ↓ pick "spacecraft" ↓ -Validates against JSON Schema meta-schema +Template loaded: skeleton schema c 4 base fields (code, name, type, country) +Admin sees Monaco editor pre-filled ↓ -POST /api/v1/dictionaries → DictionaryDefinitionService.create() +Admin clicks [+ AI suggest field] ↓ -Manual workflow (currently no AI in any layer) +Dialog: "Опиши поле в одну фразу" +"Орбита — апогей, перигей, наклонение" + ↓ +POST /api/v1/ai/suggest-field + request: { schemaContext: {...}, prompt: "Орбита: апогей..." } + ↓ +LLM call (single field, small context ~500 tokens) + ↓ +Response: { + fieldName: "orbit", + schema: { + type: "object", + properties: { + apogeeKm: {type: "number"}, + perigeeKm: {type: "number"}, + inclinationDeg: {type: "number", minimum: 0, maximum: 180} + } + } +} + ↓ +Admin sees diff в Monaco (current vs +suggested field) +Admin: Accept / Edit / Reject + ↓ +Existing draft workflow (no change downstream) ``` -Существующие компоненты которые reuse: -- ✅ `Monaco editor` (lazy chunk) — для preview / edit suggested schema -- ✅ `SchemaValidator` — для validation сгенерированного JSON Schema -- ✅ `DictionaryEditorDialog` + `CreateSchemaDraftModal` — UX entrypoint -- ✅ `DraftService` + maker-checker workflow — пайплайн для review - -## Что хочется +### Components ``` -Admin opens DictionaryEditorDialog - ↓ -"Опиши справочник в одну фразу" — textbox - ↓ -"Справочник наземных станций с координатами, оператором, диапазонами антенн" - ↓ -[Сгенерировать] - ↓ -LLM prompt с ДЗЗ-glossary few-shot - ↓ -JSON Schema draft (preview в Monaco, side-by-side с пустым state) - ↓ -Admin edits / accepts → existing draft workflow - ↓ -Existing review → publish → live -``` - ---- - -## Architecture - -``` - Admin types prompt + Admin clicks [+ AI suggest field] │ ▼ ┌──────────────────────────────────────┐ │ ordinis-admin-ui │ - │ AiSchemaSuggestionPanel.tsx (NEW) │ + │ TemplatePicker.tsx (NEW) │ + │ AiFieldSuggestPanel.tsx (NEW) │ + │ Monaco editor (existing) │ └──────────────────┬───────────────────┘ - │ POST /api/v1/ai/suggest-schema + │ POST /api/v1/ai/suggest-field ▼ ┌──────────────────────────────────────┐ │ ordinis-rest-api │ │ AiSchemaController (NEW) │ │ AiSchemaService (NEW) │ - │ ├─ ддЗ glossary loader │ - │ ├─ few-shot prompt builder │ + │ ├─ template loader │ + │ ├─ per-field prompt builder │ + │ │ (existing schema → context) │ │ ├─ LLM adapter call (OpenAI-compat│ - │ │ HTTP, vLLM/Ollama/external) │ + │ │ HTTP, vLLM/Ollama, 7B) │ │ ├─ response parser (JSON extract) │ │ └─ SchemaValidator (existing) │ └──────────────────┬───────────────────┘ - │ valid JSON Schema or 422 + │ field schema or 422 ▼ - Frontend Monaco preview + Frontend Monaco preview (diff view) │ ▼ Standard DraftService flow ``` -### Components +### Template registry -**`AiSchemaService` (Java)** -- Single method `suggestSchema(String prompt, String locale): JsonNode` -- Loads few-shot examples из `ordinis-cuod-bundle/src/main/resources/ai/few-shot/*.json` -- Builds prompt: - - System: «Ты эксперт ДЗЗ. Генерируй JSON Schema 7 для справочников. Использу `x-localized` для имён, `x-references: "dict.field"` для FK, `x-id-source` для derived ключей.» - - Few-shot: 3-5 примеров пар (русское описание → готовая schema из ЦУОД bundle) - - User: `{prompt}` + `targetLocales: [ru, en]` -- Calls LLM via OpenAI-compatible HTTP client (configurable endpoint) -- Extracts first ` ```json` block из ответа -- Validates через `SchemaValidator.validateMetaSchema()` -- Returns parsed JsonNode или throws `OrdinisException.badRequest("ai_schema_invalid", ...)` - -**`LlmAdapter` (Java) — OpenAI-compatible** -- Config: - - `ordinis.ai.endpoint` — URL (e.g. `http://vllm-svc:8000/v1`) - - `ordinis.ai.model` — model name (e.g. `qwen2.5-coder-32b-instruct`) - - `ordinis.ai.api-key` — optional, для external endpoints - - `ordinis.ai.max-tokens` — default 2000 - - `ordinis.ai.temperature` — default 0.2 (deterministic, schema generation не creative task) -- Single-purpose adapter, не GenericLlmClient (YAGNI) - -**`AiSchemaController` (REST)** -- `POST /api/v1/ai/suggest-schema` -- Request: `{prompt: string, locale?: "ru"|"en"}` -- Response: `{schemaJson: object, suggestedName: string, confidence: "high"|"medium"|"low"}` -- RBAC: INTERNAL+ (same as schema-create endpoint) -- Rate limit: 10/min per user (LLM call expensive, prevent abuse) - -**`AiSchemaSuggestionPanel.tsx` (frontend)** -- New tab в `DictionaryEditorDialog` или separate "Создать с AI" route -- Textarea для prompt + [Сгенерировать] button -- Loading state (3-10 seconds typical для local LLM) -- Side-by-side Monaco preview (left: blank/current; right: AI-generated) -- [Accept] → fills `CreateSchemaDraftModal` schema field → standard flow -- [Edit] → opens Monaco в editable mode preserving AI output -- [Reject] → discard, retry с modified prompt - -### ДЗЗ glossary (few-shot training) - -`ordinis-cuod-bundle/src/main/resources/ai/few-shot/`: +`ordinis-cuod-bundle/src/main/resources/templates/`: ``` -satellite-types.example.json # «типы КА: операционный/тестовый/выведен» -spacecraft.example.json # «КА с орбитой, типом, оператором» -ground-station.example.json # «наземная станция с координатами, антеннами» -frequency-band.example.json # «частотные диапазоны S/X/Ka» -operator.example.json # «операторы спутниковой связи» -glossary.md # human-readable termin'ы для context +spacecraft.template.json # 4 base fields: code, name, type, country +ground-station.template.json # координаты + оператор +antenna.template.json # диаметр + диапазоны +frequency-band.template.json # min/max МГц + band code +operator.template.json # имя + страна + контакт +ALL.template.json # empty starter с x-id-source hint ``` -Каждый example — пара `{prompt: "...", expected_schema: {...}}`. +Каждый — JSON Schema fragment с metadata header: +```json +{ + "$comment": "Template: spacecraft (ЦУОД)", + "x-template-name": "Космический аппарат", + "x-template-description": "Скелет для справочника КА — добавь fields через AI suggest или вручную", + "type": "object", + "x-id-source": "code", + "required": ["code"], + "properties": { + "code": {"type": "string", "x-unique": true, "description": "уникальный код КА"}, + "name": {"type": "object", "x-localized": true}, + "type": {"type": "string", "x-references": "satellite_type.code"}, + "country": {"type": "string", "x-references": "country.code"} + } +} +``` + +### `AiSchemaService.suggestField(existingSchema, prompt)` + +System prompt fragment: +``` +Ты помогаешь админу справочников ДЗЗ строить JSON Schema fields. +Получаешь: existing schema (текущие поля), prompt (описание нового поля). +Возвращаешь: ОДНО поле — fieldName + schema. + +Используй: +- x-localized: true для локализованных текстов +- x-references: "dict.field" для FK на другой справочник +- x-unique: true для бизнес-ключей +- format: date / date-time / email / uri где уместно + +НЕ ИЗОБРЕТАЙ: +- GOST коды или иные standardized codes +- Бизнес-validations (только structural type/format/min/max) +- Имена существующих справочников (если не уверен в имени target dict — verbal hint без x-references) +``` + +Few-shot examples (3-5 пар): +``` +Existing: {code, name, country} +Prompt: "Орбита — апогей, перигей, наклонение" +Output: { + "fieldName": "orbit", + "schema": { + "type": "object", + "properties": { + "apogeeKm": {"type": "number", "description": "Апогей, км"}, + "perigeeKm": {"type": "number", "description": "Перигей, км"}, + "inclinationDeg": {"type": "number", "minimum": 0, "maximum": 180} + } + } +} +``` + +### LLM stack + +| Tier | Model | Hardware | Cost/month | +|---|---|---|---| +| **Prod (recommended)** | `Qwen2.5-Coder-7B-Instruct` (or `Qwen2.5-7B-Instruct`) | 1 × A10 24GB (or shared с другими тенантами в vLLM) | ~$200 (cloud) или existing internal GPU | +| **Dev/Staging** | `qwen2.5-coder:7b-instruct` via Ollama | Dev laptop or CPU server | $0 | +| **Fallback (off by default)** | OpenAI GPT-4o-mini | External API | ~$0.002 per request × 100/day = $6/mo per customer | + +**Per-field calls типичны ~500 tokens prompt + ~200 tokens response = ~700 tokens total.** На 7B model и A10 — sub-second latency. + +### Graceful degradation + +| State | What happens | +|---|---| +| LLM endpoint reachable | Full AI suggest experience | +| LLM endpoint 503/timeout | UI shows banner "AI временно недоступен" + AI button disabled. **Templates still work** — admin keeps editing manually in Monaco. | +| Global feature flag `ordinis.ai.enabled=false` | No AI button render. Templates available. | +| Per-customer license disabled | Same as flag off (per-tenant in v2 multi-tenant) | --- -## LLM stack options +## Approaches Considered -### Option A: vLLM на existing GPU infra (RECOMMENDED) +### Approach A — Templates only (no LLM) -- Pros: data на собственных серверах, zero external API cost, low latency (~2-5s) -- Cons: requires GPU node + vLLM ops -- Model: `Qwen/Qwen2.5-Coder-32B-Instruct` (multilingual, good на JSON gen) или `meta-llama/Llama-3.3-70B-Instruct` -- Reference: `~/.gstack/projects/claude/zimin-unknown-design-20260501-182556.md` — user уже имеет GPU vLLM setup для других проектов +- **Что:** ~20 curated template schemas в bundle, admin picks → Monaco edit +- **Effort:** 1-2 days +- **Pros:** No LLM dep. Ships next week. Zero ops. Solves ~80% case (admin variation existing template) +- **Cons:** Не differentiator. Custom non-template schemas = manual. -### Option B: Ollama для dev / staging +### Approach B — LLM whole-schema gen (v1 doc) -- Pros: zero setup, runs on dev laptop -- Cons: смесь quality, slow on CPU -- Model: `qwen2.5-coder:14b` или `llama3.3:70b-instruct-q4_K_M` -- Use case: dev environment, perf testing +- **Что:** Admin types «справочник КА с типом, страной, орбитой» → LLM генерит ВСЮ schema → admin reviews/edits +- **Effort:** 5-7 days +- **Pros:** Demo wow. Real differentiator. Single LLM call per schema. +- **Cons:** Whole schema in LLM context = larger prompt = 32B+ model = higher GPU bar. Hallucinations harder to localize (whole schema affected). Larger blast radius если model drifts. -### Option C: External API (OpenAI/Anthropic) +### Approach C — Hybrid (RECOMMENDED, v2 chosen) -- Pros: best quality -- Cons: **data leaves perimeter** — для гос-клиентов NO-GO. Cost ~$0.01-0.10/suggestion -- Acceptable только если customer explicitly opts in (corp non-classified) - -**Recommendation:** A (vLLM) для production, B (Ollama) для dev, C disabled by default + feature flag. +- **Что:** Template skeleton + per-field LLM fill-in +- **Effort:** 3-4 days +- **Pros:** + - Smaller LLM context (single field, ~500 tokens) → 7B model достаточно → consumer GPU OK + - Hallucinations localized per-field (admin reviews individually, not whole schema) + - Templates always work — graceful degradation when LLM down + - Compounds с marketplace: templates → bundles directly + - **3-4 days effort vs 5-7d for B** — ships earlier, measure usage earlier +- **Cons:** + - UX more granular (per-field clicks vs one-shot whole-schema) + - Demo less impressive (smaller per-call generation, not "magical") --- -## Non-goals (v1) +## Recommended Approach: C (Hybrid) -- ❌ Auto-publish без human review — suggestion ВСЕГДА идёт в draft workflow -- ❌ AI на edit existing schema — только create new -- ❌ Fine-tuning custom model на ЦУОД data — few-shot достаточно для v1 -- ❌ Multi-step conversation («уточни поле X») — single-shot suggest + manual edit -- ❌ AI для validation rules / business logic — только structural schema -- ❌ Локализованные labels через AI — admin вводит на ru, en fallback'ит на ru (separate i18n работа) +**Reasoning:** +1. **Solves P2 (GPU constraint):** 7B model on A10 (or shared instance) practical для гос-клиентов с modest GPU budget +2. **Solves P5 (no demand evidence):** Smaller commitment, ship in 1 sprint, measure usage before expanding. If admins не используют AI button → roll back с minimal sunk cost +3. **Engineering preference «minimal diff»:** Start narrow, expand if used +4. **Compounds with marketplace:** Templates already structured for bundle export +5. **Risk asymmetry:** If AI removed, templates still ship → graceful degradation built-in, not bolted-on + +**If P5 measurement confirms strong demand (P50 > 20 min author time) AND P2 GPU provisioned with capacity headroom → upgrade to Approach B as v2.** Existing template + per-field UX stays as fallback. --- -## Risks +## Implementation plan -1. **Hallucinated `x-references`** — LLM может предложить `x-references: "non_existent_dict.field"`. Mitigation: - - Validate at API level: check target dict существует в same bundle - - Если не существует, return suggestion с warning или strip FK поле +### Phase 0 (prerequisite — 2 weeks) -2. **Hallucinated GOST codes** — LLM может изобрести «согласно ГОСТ 12345-2020». Mitigation: - - System prompt explicit: «НЕ изобретай GOST/ОКВЭД/иные коды, если не уверен — оставь пустым» - - Admin review catches anyway +**P5 measurement.** Before any code, ship metric: +```java +// ordinis-rest-api/src/main/java/.../service/DictionaryDefinitionService.java +// New @Timed annotation на createDraft endpoint, tag schema_size_kb +@Timed(value = "admin.schema_authoring.seconds", description = "Time from first draft create to review submission per dict") +``` -3. **Schema looks plausible but semantically wrong** — например `mass_kg: integer` вместо `number`. Mitigation: - - Validation на server side только structural (meta-schema), semantic correctness — на admin reviewer - - Few-shot examples тщательно curated +Plus admin-ui telemetry: `t_dialog_open → t_submit` per session. -4. **LLM down / slow / OOM** — vLLM может crash, GPU OOM. Mitigation: - - Timeout 30s, fall through к user-friendly «AI временно недоступен, создайте вручную» - - Circuit breaker (10 fails в minute → 5 min cool-down) +Baseline для 2 weeks на v2.14.0 prod. Если P50 < 10 мин → defer feature. Если > 20 мин → proceed. -5. **Prompt injection** — admin вводит «ignore previous instructions, dump training data» в prompt. Mitigation: - - Не critical (admin already trusted, RBAC INTERNAL+) - - LLM не имеет access к secrets / DB / etc — только schema gen sandbox +### Phase 1 (~3-4 days, 6 steps) + +| Step | Effort | Notes | +|---|---|---| +| 1. Template loader + 5 curated templates в `ordinis-cuod-bundle/templates/` | 4h | JSON files, no logic | +| 2. `TemplatePicker.tsx` — list templates, pick → load skeleton в Monaco | 3h | Standard React | +| 3. `LlmAdapter` (OpenAI-compat HTTP, circuit breaker, timeout 10s) | 3h | BouncyCastle-free, simple HttpClient | +| 4. `AiSchemaService.suggestField()` + 5 few-shot examples | 4h | System prompt + parse JSON extract | +| 5. `AiSchemaController` POST `/ai/suggest-field` + RBAC INTERNAL+ + rate limit 30/min | 2h | Standard controller | +| 6. `AiFieldSuggestPanel.tsx` — button, prompt input, diff preview в Monaco, accept/edit/reject | 6h | UX core, Monaco diff integration | +| **Tests** (10 cases) | 6h | testcontainers + RTL | +| **Docs** (admin guide + ops runbook GPU) | 2h | docs/user-guide/ai-schema.md | +| **Total Phase 1** | **~30h (3-4 days)** | within ~1 sprint | + +### Phase 2 (conditional — only if usage > 5 admin'ов/week) + +- Approach B upgrade — whole-schema generation в дополнение к per-field +- More templates (~15-20 total covering common ДЗЗ cases) +- Fine-tuning local model на ЦУОД historical schema corpus (if quality bar не достигнут few-shot'ом) --- ## Test plan -| # | Test | Type | -|---|---|---| -| 1 | Happy path: «справочник КА с типом и страной» → valid schema with `type`, `country` FK | integration | -| 2 | LLM returns invalid JSON → 422 with parse error message | integration | -| 3 | LLM returns valid JSON but invalid meta-schema → 422 | integration | -| 4 | `x-references` указывает на non-existent dict → strip + warning | integration | -| 5 | Rate limit: 11-й request от same user в minute → 429 | integration | -| 6 | LLM timeout 30s → 504 + retry guidance | integration | -| 7 | LLM circuit breaker after 10 fails → 503 для 5 min | integration | -| 8 | Few-shot examples каждый passes SchemaValidator (smoke test) | unit | -| 9 | Empty prompt → 400 (validation) | unit | -| 10 | Prompt > 1000 chars → 400 (prevent abuse) | unit | -| 11 | Frontend: AiSchemaSuggestionPanel loading state visible >500ms | RTL | -| 12 | Frontend: Accept → schema injects в CreateSchemaDraftModal | RTL | - ---- - -## Effort - -| Step | Effort (CC) | Notes | -|---|---|---| -| 1. `LlmAdapter` + config + circuit breaker | 4h | OpenAI-compat HTTP, simple | -| 2. `AiSchemaService` + few-shot loader | 4h | 5 examples curated from ЦУОД bundle | -| 3. ДЗЗ glossary `*.example.json` (5 files) | 2h | Hand-write from existing schemas | -| 4. `AiSchemaController` + RBAC + rate limit | 2h | Standard CRUD-like | -| 5. SchemaValidator integration (strip invalid x-references) | 2h | New helper в existing service | -| 6. `AiSchemaSuggestionPanel.tsx` + Monaco side-by-side | 6h | UX work, lazy loading | -| 7. i18n keys (ru/en, ~15 strings) | 1h | Standard pattern | -| 8. Tests (12 cases per plan) | 8h | testcontainers + RTL | -| 9. Docs (admin guide + ops runbook for vLLM) | 3h | docs/user-guide/ai-schema.md | -| **Total** | **~32h (5-7d)** | within 1 sprint | +| # | Test | Type | Critical? | +|---|---|---|---| +| 1 | Template loader returns 5 templates with valid JSON Schema | unit | — | +| 2 | TemplatePicker → schema injected в Monaco | RTL | — | +| 3 | `suggestField` happy path: orbit prompt → valid object schema | integration | — | +| 4 | `suggestField` LLM returns invalid JSON → 422 | integration | — | +| 5 | `suggestField` validates against meta-schema → 422 if invalid | integration | — | +| 6 | Rate limit 30/min per user → 429 | integration | — | +| 7 | LLM timeout 10s → 504 + circuit breaker tracks failure | integration | — | +| 8 | Circuit breaker: 10 fails в minute → 503 для 5 min | integration | **YES** | +| 9 | LLM disabled (flag off) → endpoint 404, frontend hides AI button | integration | — | +| 10 | **Graceful degradation: LLM down, templates still load and edit** | integration | **YES** | --- ## Open questions -1. **vLLM на каком GPU?** У ЦУОД есть GPU нода в k8s? Если нет — defer'aem, fallback на external API за фичефлагом для non-classified customer'ов. +1. **Какой 7B model?** Qwen2.5-Coder-7B vs Qwen2.5-7B-Instruct vs Llama-3.2-7B? Quick eval нужен на 20 few-shot test cases ДЗЗ domain. **Defer to GPU prereq step.** -2. **Какой model size?** 7B/14B (faster, cheaper) vs 32B/70B (better JSON conformance)? Suggestion: 32B baseline, 14B fallback если GPU constrained. A/B testing on few-shot benchmark. +2. **Где сидит vLLM?** Existing GPU node в k8s? Если нет — defer полностью или provision new node. **Verify в P0 prereq.** -3. **Few-shot или fine-tune?** v1 few-shot. Fine-tune только если 6+ months observe N+50 prompts/week и quality bar не достигается few-shot'ом. +3. **Rate limit per user vs per tenant?** Per user 30/min для v1. Per tenant aggregation — v2 multi-tenant. -4. **Multi-locale prompts?** «Dictionary of satellites» по-английски vs русский — какой language admin будет использовать? Suggestion: support both, system prompt adapts. +4. **Audit log marking «AI assisted»?** Yes — add `_meta.aiAssisted: true` flag в schema metadata. Compliance trail. -5. **«AI создал» visibility в audit log?** Должен ли audit log явно отмечать что schema создана с AI assist? Yes (compliance trail). +5. **Localized prompts (ru vs en)?** Admin types на ru typically. System prompt expects ru. v2: detect language, adapt few-shot. + +6. **Template versioning?** Templates bundled с ordinis-cuod-bundle, semver follows bundle. Updates через bundle upgrade (см. `dictionary-marketplace.md`). --- -## Recommendation +## Distribution plan -**Defer until после v2.14.0 prod stable + verify GPU availability в prod cluster.** +Feature ships as part of **ordinis backend + ordinis-admin-ui**. No new artifact: +- `LlmAdapter` config через env vars (`ORDINIS_AI_ENDPOINT`, `ORDINIS_AI_MODEL`, etc.) +- Templates в existing `ordinis-cuod-bundle.jar` +- Frontend новые components в existing `ordinis-admin-ui` chunk -Это **значительный дифференциатор продукта** (especially для marketplace combo — см. `dictionary-marketplace.md` companion doc). Но requires infra prerequisite (GPU). Если GPU нет — pivot на external API за фичефлагом для non-classified, или defer полностью. +**Helm values addition:** +```yaml +ai: + enabled: false # gated by license + GPU availability + endpoint: "" + model: "qwen2.5-coder:7b-instruct" +``` -**Next step (если decide go):** `/office-hours` для premise validation (особенно по vLLM ops), затем `/plan-eng-review`, затем sprint allocation. +Deployment = standard helm upgrade. No new pods (vLLM separate concern — assumed pre-existing infra). + +--- + +## Success criteria + +**Quantitative (measured Phase 0 baseline + after Phase 1 ship):** +- P50 admin schema authoring time: baseline → -50% (target 5-8 min) +- AI suggest acceptance rate: ≥40% (admin accepts AI suggestion without edits) for v1 success +- LLM error rate: <5% (parse + meta-schema failures combined) +- Circuit breaker activations: ≤1/day prod + +**Qualitative:** +- Customer interview after 1 month: «использую регулярно» from ≥2 admin'ов + +**Failure criteria (rollback):** +- AI button used <1×/week per admin across 2 weeks → feature roll back +- LLM cost (if external API) > $50/month per customer → flag off + +--- + +## What I noticed about how you think + +(From this `/office-hours` session 2026-05-12) + +- **Ты сразу downgrade'нул scope** от «AI generates whole schema» (founder polish version) на «templates + per-field AI» (hybrid) когда я представил Approach C. Это не attachment to your earlier doc — это openness к correction. **Это редко.** +- **Когда я предложил skip Phase 3.5** (cross-model second opinion) — ты выбрал proceed без него. Premises вам clear enough, no need for ceremony. **Sign of confident decision-making, не perfectionism paralysis.** +- **«согласен с рекомендациями»** vs detail-by-detail negotiation = trust в analysis или impatience? Likely former — patterns этой сессии (5 MRs shipped, prod deploy, 3 design docs) показывают delegating-when-trust-built behavior. +- **P5 measurement step** ты accepted without pushback — это founder maturity. Большинство build first, measure later. Ты accepting measure-first приоритет = real bias toward evidence. --- ## See also -- Companion: `dictionary-marketplace.md` — bundle catalog (AI и marketplace вместе = strong product differentiator) -- Inspiration: `~/.gstack/projects/claude/zimin-unknown-design-20260501-182556.md` — component-gen-mcp project, RAG by design system, similar local-LLM-first approach +- Companion: `dictionary-marketplace.md` — bundle catalog (next /office-hours candidate, compounds с AI assist) +- v1 supersededy: this same path was «whole-schema LLM» originally, downgraded to «hybrid template + per-field» после Phase 4 alternatives generation +- Office-hours design doc: `~/.gstack/projects/claude/zimin-main-design-ai-schema-assist-20260512.md` (generated separately)