docs(design): ai-schema-assist v2 — /office-hours, Approach C hybrid
This commit is contained in:
Александр Зимин
+274
-180
@@ -2,278 +2,372 @@
|
|||||||
|
|
||||||
**Author:** zimin.an
|
**Author:** zimin.an
|
||||||
**Date:** 2026-05-12
|
**Date:** 2026-05-12
|
||||||
**Status:** PROPOSED v1 — needs `/office-hours` for premise validation + `/plan-eng-review`
|
**Status:** PROPOSED v2 (post `/office-hours`)
|
||||||
|
**Supersedes:** v1 (2026-05-12 — LLM-only "Approach B", before office-hours premise validation revealed `Approach C` hybrid)
|
||||||
**Scope:** ДЗЗ domain only (ground segment + orbital, не general-purpose AI)
|
**Scope:** ДЗЗ domain only (ground segment + orbital, не general-purpose AI)
|
||||||
**Sprint estimate:** ~5-7 days CC (1 sprint), or ~3-4 days если skip GUI polish
|
**Sprint estimate:** **~3-4 days CC** для **Approach C hybrid** (revised down from 5-7d v1)
|
||||||
**Blockers:** none, но recommendation defer'a до v2.14.0 prod stable
|
**Blockers:** none architectural. P5 measurement step (см. Premises) — soft prereq
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## TL;DR
|
## TL;DR
|
||||||
|
|
||||||
Admin Ordinis сейчас создаёт новый справочник вручную: пишет JSON Schema (поля, типы, `x-references`, `x-localized`, validations), пробрасывает локализации, продумывает bitemporal flags. Это **15-30 минут per dictionary** для опытного админа, **час+** для нового. Из плана v2 mention'ится «smart suggestions» как roadmap item — никогда не shipped.
|
Admin Ordinis сейчас создаёт справочник вручную — пишет JSON Schema (поля, типы, `x-references`, `x-localized`, validations), пробрасывает локализации, продумывает bitemporal flags. Это **15-30 минут per dictionary** для опытного админа, **час+** для нового.
|
||||||
|
|
||||||
**Предложение:** LLM-assisted authoring. Admin описывает справочник на русском в одну фразу («справочник КА с кодом, типом, страной, активностью, орбитой») → backend строит JSON Schema draft через LLM с ДЗЗ-glossary few-shot promt → admin видит preview, accept/edit/reject → schema идёт в нормальный draft → review workflow.
|
**Предложение (v2 — Hybrid):** Admin starts с **curated template** (e.g. «спутник», «наземная станция»), затем при добавлении каждого нового поля кликает **«AI suggest field»** → LLM предлагает structure (type/format/x-localized/x-references) для **одного поля** на базе schema name + existing fields context. Admin reviews → accept или edit → continue.
|
||||||
|
|
||||||
**Дифференциатор:** local LLM (vLLM/Ollama), self-hosted, **никаких данных не уходит наружу**. Critical для гос-клиентов.
|
**v2 vs v1:** Vместо whole-schema generation одним LLM call'ом — **template skeleton + per-field LLM fill-in**. Smaller LLM context, более reliable, lower GPU bar (7B sufficient instead of 32B), graceful degradation (templates работают без LLM).
|
||||||
|
|
||||||
**Win:** time-to-first-schema 30мин → 2-3мин, новый admin onboarding hour → 5мин. Demo wow-эффект для sales.
|
**Win:** time-to-first-schema 30мин → 5-8мин (template + 5-10 fields, каждое 30 сек AI suggest). Compounds with marketplace: templates → bundles directly.
|
||||||
|
|
||||||
**Risk:** LLM hallucinates fields, `x-references`, валидации. Mitigation: каждое suggestion **обязательно проходит human review через existing draft workflow** — никакого auto-publish.
|
**Risk:** LLM hallucinations contained per-field (smaller blast radius vs whole schema). Templates always work even если LLM down.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Current state
|
## Premises (validated через `/office-hours` 2026-05-12)
|
||||||
|
|
||||||
|
**P1.** AI = **time-saver**, не replacement. Schema всегда проходит human review через existing draft workflow. **No auto-publish.** Value = drop time-to-first-schema 30мин → 5-8мин.
|
||||||
|
|
||||||
|
**P2.** **Local LLM** (vLLM/Ollama) **non-negotiable** для гос-клиентов — data не покидает perimeter. External API disabled by default, feature-flag-only для corp non-classified. **GPU verification = hard prereq** перед implementation.
|
||||||
|
|
||||||
|
**P3.** **Scope = ДЗЗ domain only.** Few-shot training на ЦУОД bundle (5 примеров). Non-ДЗЗ customer → manually extends few-shot в свой bundle.
|
||||||
|
|
||||||
|
**P4.** **AI predicts STRUCTURE, не SEMANTICS.** Type, format, nullability, FK ref structure, locales — да. Validation rules, GOST codes, business constraints — NET. System prompt explicit: «если не уверен — оставь пустым».
|
||||||
|
|
||||||
|
**P5.** **Demand evidence пока нулевая** — founder-driven feature, не customer-pulled. **Prerequisite measurement step:** перед implementation добавить metric `admin_schema_authoring_time_seconds` (start = first draft creation, end = first review submission). Собрать **2 weeks baseline** на v2.14.0 prod. Если P50 < 10 мин — отложить feature (no real pain). Если P50 > 20 мин — confirms hypothesis, proceed.
|
||||||
|
|
||||||
|
**P6.** Cross-model second opinion (Phase 3.5) — skipped в этом office-hours session, premises clear enough.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture (Approach C — Hybrid Template + Per-Field LLM)
|
||||||
|
|
||||||
|
### High-level flow
|
||||||
|
|
||||||
```
|
```
|
||||||
Admin opens DictionaryEditorDialog
|
Admin opens DictionaryEditorDialog
|
||||||
↓
|
↓
|
||||||
Manually types JSON Schema in Monaco editor
|
[Templates panel: spacecraft / ground-station / antenna / ...]
|
||||||
|
↓ pick "spacecraft"
|
||||||
↓
|
↓
|
||||||
Validates against JSON Schema meta-schema
|
Template loaded: skeleton schema c 4 base fields (code, name, type, country)
|
||||||
|
Admin sees Monaco editor pre-filled
|
||||||
↓
|
↓
|
||||||
POST /api/v1/dictionaries → DictionaryDefinitionService.create()
|
Admin clicks [+ AI suggest field]
|
||||||
↓
|
↓
|
||||||
Manual workflow (currently no AI in any layer)
|
Dialog: "Опиши поле в одну фразу"
|
||||||
|
"Орбита — апогей, перигей, наклонение"
|
||||||
|
↓
|
||||||
|
POST /api/v1/ai/suggest-field
|
||||||
|
request: { schemaContext: {...}, prompt: "Орбита: апогей..." }
|
||||||
|
↓
|
||||||
|
LLM call (single field, small context ~500 tokens)
|
||||||
|
↓
|
||||||
|
Response: {
|
||||||
|
fieldName: "orbit",
|
||||||
|
schema: {
|
||||||
|
type: "object",
|
||||||
|
properties: {
|
||||||
|
apogeeKm: {type: "number"},
|
||||||
|
perigeeKm: {type: "number"},
|
||||||
|
inclinationDeg: {type: "number", minimum: 0, maximum: 180}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
↓
|
||||||
|
Admin sees diff в Monaco (current vs +suggested field)
|
||||||
|
Admin: Accept / Edit / Reject
|
||||||
|
↓
|
||||||
|
Existing draft workflow (no change downstream)
|
||||||
```
|
```
|
||||||
|
|
||||||
Существующие компоненты которые reuse:
|
### Components
|
||||||
- ✅ `Monaco editor` (lazy chunk) — для preview / edit suggested schema
|
|
||||||
- ✅ `SchemaValidator` — для validation сгенерированного JSON Schema
|
|
||||||
- ✅ `DictionaryEditorDialog` + `CreateSchemaDraftModal` — UX entrypoint
|
|
||||||
- ✅ `DraftService` + maker-checker workflow — пайплайн для review
|
|
||||||
|
|
||||||
## Что хочется
|
|
||||||
|
|
||||||
```
|
```
|
||||||
Admin opens DictionaryEditorDialog
|
Admin clicks [+ AI suggest field]
|
||||||
↓
|
|
||||||
"Опиши справочник в одну фразу" — textbox
|
|
||||||
↓
|
|
||||||
"Справочник наземных станций с координатами, оператором, диапазонами антенн"
|
|
||||||
↓
|
|
||||||
[Сгенерировать]
|
|
||||||
↓
|
|
||||||
LLM prompt с ДЗЗ-glossary few-shot
|
|
||||||
↓
|
|
||||||
JSON Schema draft (preview в Monaco, side-by-side с пустым state)
|
|
||||||
↓
|
|
||||||
Admin edits / accepts → existing draft workflow
|
|
||||||
↓
|
|
||||||
Existing review → publish → live
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
Admin types prompt
|
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
┌──────────────────────────────────────┐
|
┌──────────────────────────────────────┐
|
||||||
│ ordinis-admin-ui │
|
│ ordinis-admin-ui │
|
||||||
│ AiSchemaSuggestionPanel.tsx (NEW) │
|
│ TemplatePicker.tsx (NEW) │
|
||||||
|
│ AiFieldSuggestPanel.tsx (NEW) │
|
||||||
|
│ Monaco editor (existing) │
|
||||||
└──────────────────┬───────────────────┘
|
└──────────────────┬───────────────────┘
|
||||||
│ POST /api/v1/ai/suggest-schema
|
│ POST /api/v1/ai/suggest-field
|
||||||
▼
|
▼
|
||||||
┌──────────────────────────────────────┐
|
┌──────────────────────────────────────┐
|
||||||
│ ordinis-rest-api │
|
│ ordinis-rest-api │
|
||||||
│ AiSchemaController (NEW) │
|
│ AiSchemaController (NEW) │
|
||||||
│ AiSchemaService (NEW) │
|
│ AiSchemaService (NEW) │
|
||||||
│ ├─ ддЗ glossary loader │
|
│ ├─ template loader │
|
||||||
│ ├─ few-shot prompt builder │
|
│ ├─ per-field prompt builder │
|
||||||
|
│ │ (existing schema → context) │
|
||||||
│ ├─ LLM adapter call (OpenAI-compat│
|
│ ├─ LLM adapter call (OpenAI-compat│
|
||||||
│ │ HTTP, vLLM/Ollama/external) │
|
│ │ HTTP, vLLM/Ollama, 7B) │
|
||||||
│ ├─ response parser (JSON extract) │
|
│ ├─ response parser (JSON extract) │
|
||||||
│ └─ SchemaValidator (existing) │
|
│ └─ SchemaValidator (existing) │
|
||||||
└──────────────────┬───────────────────┘
|
└──────────────────┬───────────────────┘
|
||||||
│ valid JSON Schema or 422
|
│ field schema or 422
|
||||||
▼
|
▼
|
||||||
Frontend Monaco preview
|
Frontend Monaco preview (diff view)
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
Standard DraftService flow
|
Standard DraftService flow
|
||||||
```
|
```
|
||||||
|
|
||||||
### Components
|
### Template registry
|
||||||
|
|
||||||
**`AiSchemaService` (Java)**
|
`ordinis-cuod-bundle/src/main/resources/templates/`:
|
||||||
- Single method `suggestSchema(String prompt, String locale): JsonNode`
|
|
||||||
- Loads few-shot examples из `ordinis-cuod-bundle/src/main/resources/ai/few-shot/*.json`
|
|
||||||
- Builds prompt:
|
|
||||||
- System: «Ты эксперт ДЗЗ. Генерируй JSON Schema 7 для справочников. Использу `x-localized` для имён, `x-references: "dict.field"` для FK, `x-id-source` для derived ключей.»
|
|
||||||
- Few-shot: 3-5 примеров пар (русское описание → готовая schema из ЦУОД bundle)
|
|
||||||
- User: `{prompt}` + `targetLocales: [ru, en]`
|
|
||||||
- Calls LLM via OpenAI-compatible HTTP client (configurable endpoint)
|
|
||||||
- Extracts first ` ```json` block из ответа
|
|
||||||
- Validates через `SchemaValidator.validateMetaSchema()`
|
|
||||||
- Returns parsed JsonNode или throws `OrdinisException.badRequest("ai_schema_invalid", ...)`
|
|
||||||
|
|
||||||
**`LlmAdapter` (Java) — OpenAI-compatible**
|
|
||||||
- Config:
|
|
||||||
- `ordinis.ai.endpoint` — URL (e.g. `http://vllm-svc:8000/v1`)
|
|
||||||
- `ordinis.ai.model` — model name (e.g. `qwen2.5-coder-32b-instruct`)
|
|
||||||
- `ordinis.ai.api-key` — optional, для external endpoints
|
|
||||||
- `ordinis.ai.max-tokens` — default 2000
|
|
||||||
- `ordinis.ai.temperature` — default 0.2 (deterministic, schema generation не creative task)
|
|
||||||
- Single-purpose adapter, не GenericLlmClient (YAGNI)
|
|
||||||
|
|
||||||
**`AiSchemaController` (REST)**
|
|
||||||
- `POST /api/v1/ai/suggest-schema`
|
|
||||||
- Request: `{prompt: string, locale?: "ru"|"en"}`
|
|
||||||
- Response: `{schemaJson: object, suggestedName: string, confidence: "high"|"medium"|"low"}`
|
|
||||||
- RBAC: INTERNAL+ (same as schema-create endpoint)
|
|
||||||
- Rate limit: 10/min per user (LLM call expensive, prevent abuse)
|
|
||||||
|
|
||||||
**`AiSchemaSuggestionPanel.tsx` (frontend)**
|
|
||||||
- New tab в `DictionaryEditorDialog` или separate "Создать с AI" route
|
|
||||||
- Textarea для prompt + [Сгенерировать] button
|
|
||||||
- Loading state (3-10 seconds typical для local LLM)
|
|
||||||
- Side-by-side Monaco preview (left: blank/current; right: AI-generated)
|
|
||||||
- [Accept] → fills `CreateSchemaDraftModal` schema field → standard flow
|
|
||||||
- [Edit] → opens Monaco в editable mode preserving AI output
|
|
||||||
- [Reject] → discard, retry с modified prompt
|
|
||||||
|
|
||||||
### ДЗЗ glossary (few-shot training)
|
|
||||||
|
|
||||||
`ordinis-cuod-bundle/src/main/resources/ai/few-shot/`:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
satellite-types.example.json # «типы КА: операционный/тестовый/выведен»
|
spacecraft.template.json # 4 base fields: code, name, type, country
|
||||||
spacecraft.example.json # «КА с орбитой, типом, оператором»
|
ground-station.template.json # координаты + оператор
|
||||||
ground-station.example.json # «наземная станция с координатами, антеннами»
|
antenna.template.json # диаметр + диапазоны
|
||||||
frequency-band.example.json # «частотные диапазоны S/X/Ka»
|
frequency-band.template.json # min/max МГц + band code
|
||||||
operator.example.json # «операторы спутниковой связи»
|
operator.template.json # имя + страна + контакт
|
||||||
glossary.md # human-readable termin'ы для context
|
ALL.template.json # empty starter с x-id-source hint
|
||||||
```
|
```
|
||||||
|
|
||||||
Каждый example — пара `{prompt: "...", expected_schema: {...}}`.
|
Каждый — JSON Schema fragment с metadata header:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"$comment": "Template: spacecraft (ЦУОД)",
|
||||||
|
"x-template-name": "Космический аппарат",
|
||||||
|
"x-template-description": "Скелет для справочника КА — добавь fields через AI suggest или вручную",
|
||||||
|
"type": "object",
|
||||||
|
"x-id-source": "code",
|
||||||
|
"required": ["code"],
|
||||||
|
"properties": {
|
||||||
|
"code": {"type": "string", "x-unique": true, "description": "уникальный код КА"},
|
||||||
|
"name": {"type": "object", "x-localized": true},
|
||||||
|
"type": {"type": "string", "x-references": "satellite_type.code"},
|
||||||
|
"country": {"type": "string", "x-references": "country.code"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `AiSchemaService.suggestField(existingSchema, prompt)`
|
||||||
|
|
||||||
|
System prompt fragment:
|
||||||
|
```
|
||||||
|
Ты помогаешь админу справочников ДЗЗ строить JSON Schema fields.
|
||||||
|
Получаешь: existing schema (текущие поля), prompt (описание нового поля).
|
||||||
|
Возвращаешь: ОДНО поле — fieldName + schema.
|
||||||
|
|
||||||
|
Используй:
|
||||||
|
- x-localized: true для локализованных текстов
|
||||||
|
- x-references: "dict.field" для FK на другой справочник
|
||||||
|
- x-unique: true для бизнес-ключей
|
||||||
|
- format: date / date-time / email / uri где уместно
|
||||||
|
|
||||||
|
НЕ ИЗОБРЕТАЙ:
|
||||||
|
- GOST коды или иные standardized codes
|
||||||
|
- Бизнес-validations (только structural type/format/min/max)
|
||||||
|
- Имена существующих справочников (если не уверен в имени target dict — verbal hint без x-references)
|
||||||
|
```
|
||||||
|
|
||||||
|
Few-shot examples (3-5 пар):
|
||||||
|
```
|
||||||
|
Existing: {code, name, country}
|
||||||
|
Prompt: "Орбита — апогей, перигей, наклонение"
|
||||||
|
Output: {
|
||||||
|
"fieldName": "orbit",
|
||||||
|
"schema": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"apogeeKm": {"type": "number", "description": "Апогей, км"},
|
||||||
|
"perigeeKm": {"type": "number", "description": "Перигей, км"},
|
||||||
|
"inclinationDeg": {"type": "number", "minimum": 0, "maximum": 180}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### LLM stack
|
||||||
|
|
||||||
|
| Tier | Model | Hardware | Cost/month |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **Prod (recommended)** | `Qwen2.5-Coder-7B-Instruct` (or `Qwen2.5-7B-Instruct`) | 1 × A10 24GB (or shared с другими тенантами в vLLM) | ~$200 (cloud) или existing internal GPU |
|
||||||
|
| **Dev/Staging** | `qwen2.5-coder:7b-instruct` via Ollama | Dev laptop or CPU server | $0 |
|
||||||
|
| **Fallback (off by default)** | OpenAI GPT-4o-mini | External API | ~$0.002 per request × 100/day = $6/mo per customer |
|
||||||
|
|
||||||
|
**Per-field calls типичны ~500 tokens prompt + ~200 tokens response = ~700 tokens total.** На 7B model и A10 — sub-second latency.
|
||||||
|
|
||||||
|
### Graceful degradation
|
||||||
|
|
||||||
|
| State | What happens |
|
||||||
|
|---|---|
|
||||||
|
| LLM endpoint reachable | Full AI suggest experience |
|
||||||
|
| LLM endpoint 503/timeout | UI shows banner "AI временно недоступен" + AI button disabled. **Templates still work** — admin keeps editing manually in Monaco. |
|
||||||
|
| Global feature flag `ordinis.ai.enabled=false` | No AI button render. Templates available. |
|
||||||
|
| Per-customer license disabled | Same as flag off (per-tenant in v2 multi-tenant) |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## LLM stack options
|
## Approaches Considered
|
||||||
|
|
||||||
### Option A: vLLM на existing GPU infra (RECOMMENDED)
|
### Approach A — Templates only (no LLM)
|
||||||
|
|
||||||
- Pros: data на собственных серверах, zero external API cost, low latency (~2-5s)
|
- **Что:** ~20 curated template schemas в bundle, admin picks → Monaco edit
|
||||||
- Cons: requires GPU node + vLLM ops
|
- **Effort:** 1-2 days
|
||||||
- Model: `Qwen/Qwen2.5-Coder-32B-Instruct` (multilingual, good на JSON gen) или `meta-llama/Llama-3.3-70B-Instruct`
|
- **Pros:** No LLM dep. Ships next week. Zero ops. Solves ~80% case (admin variation existing template)
|
||||||
- Reference: `~/.gstack/projects/claude/zimin-unknown-design-20260501-182556.md` — user уже имеет GPU vLLM setup для других проектов
|
- **Cons:** Не differentiator. Custom non-template schemas = manual.
|
||||||
|
|
||||||
### Option B: Ollama для dev / staging
|
### Approach B — LLM whole-schema gen (v1 doc)
|
||||||
|
|
||||||
- Pros: zero setup, runs on dev laptop
|
- **Что:** Admin types «справочник КА с типом, страной, орбитой» → LLM генерит ВСЮ schema → admin reviews/edits
|
||||||
- Cons: смесь quality, slow on CPU
|
- **Effort:** 5-7 days
|
||||||
- Model: `qwen2.5-coder:14b` или `llama3.3:70b-instruct-q4_K_M`
|
- **Pros:** Demo wow. Real differentiator. Single LLM call per schema.
|
||||||
- Use case: dev environment, perf testing
|
- **Cons:** Whole schema in LLM context = larger prompt = 32B+ model = higher GPU bar. Hallucinations harder to localize (whole schema affected). Larger blast radius если model drifts.
|
||||||
|
|
||||||
### Option C: External API (OpenAI/Anthropic)
|
### Approach C — Hybrid (RECOMMENDED, v2 chosen)
|
||||||
|
|
||||||
- Pros: best quality
|
- **Что:** Template skeleton + per-field LLM fill-in
|
||||||
- Cons: **data leaves perimeter** — для гос-клиентов NO-GO. Cost ~$0.01-0.10/suggestion
|
- **Effort:** 3-4 days
|
||||||
- Acceptable только если customer explicitly opts in (corp non-classified)
|
- **Pros:**
|
||||||
|
- Smaller LLM context (single field, ~500 tokens) → 7B model достаточно → consumer GPU OK
|
||||||
**Recommendation:** A (vLLM) для production, B (Ollama) для dev, C disabled by default + feature flag.
|
- Hallucinations localized per-field (admin reviews individually, not whole schema)
|
||||||
|
- Templates always work — graceful degradation when LLM down
|
||||||
|
- Compounds с marketplace: templates → bundles directly
|
||||||
|
- **3-4 days effort vs 5-7d for B** — ships earlier, measure usage earlier
|
||||||
|
- **Cons:**
|
||||||
|
- UX more granular (per-field clicks vs one-shot whole-schema)
|
||||||
|
- Demo less impressive (smaller per-call generation, not "magical")
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Non-goals (v1)
|
## Recommended Approach: C (Hybrid)
|
||||||
|
|
||||||
- ❌ Auto-publish без human review — suggestion ВСЕГДА идёт в draft workflow
|
**Reasoning:**
|
||||||
- ❌ AI на edit existing schema — только create new
|
1. **Solves P2 (GPU constraint):** 7B model on A10 (or shared instance) practical для гос-клиентов с modest GPU budget
|
||||||
- ❌ Fine-tuning custom model на ЦУОД data — few-shot достаточно для v1
|
2. **Solves P5 (no demand evidence):** Smaller commitment, ship in 1 sprint, measure usage before expanding. If admins не используют AI button → roll back с minimal sunk cost
|
||||||
- ❌ Multi-step conversation («уточни поле X») — single-shot suggest + manual edit
|
3. **Engineering preference «minimal diff»:** Start narrow, expand if used
|
||||||
- ❌ AI для validation rules / business logic — только structural schema
|
4. **Compounds with marketplace:** Templates already structured for bundle export
|
||||||
- ❌ Локализованные labels через AI — admin вводит на ru, en fallback'ит на ru (separate i18n работа)
|
5. **Risk asymmetry:** If AI removed, templates still ship → graceful degradation built-in, not bolted-on
|
||||||
|
|
||||||
|
**If P5 measurement confirms strong demand (P50 > 20 min author time) AND P2 GPU provisioned with capacity headroom → upgrade to Approach B as v2.** Existing template + per-field UX stays as fallback.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Risks
|
## Implementation plan
|
||||||
|
|
||||||
1. **Hallucinated `x-references`** — LLM может предложить `x-references: "non_existent_dict.field"`. Mitigation:
|
### Phase 0 (prerequisite — 2 weeks)
|
||||||
- Validate at API level: check target dict существует в same bundle
|
|
||||||
- Если не существует, return suggestion с warning или strip FK поле
|
|
||||||
|
|
||||||
2. **Hallucinated GOST codes** — LLM может изобрести «согласно ГОСТ 12345-2020». Mitigation:
|
**P5 measurement.** Before any code, ship metric:
|
||||||
- System prompt explicit: «НЕ изобретай GOST/ОКВЭД/иные коды, если не уверен — оставь пустым»
|
```java
|
||||||
- Admin review catches anyway
|
// ordinis-rest-api/src/main/java/.../service/DictionaryDefinitionService.java
|
||||||
|
// New @Timed annotation на createDraft endpoint, tag schema_size_kb
|
||||||
|
@Timed(value = "admin.schema_authoring.seconds", description = "Time from first draft create to review submission per dict")
|
||||||
|
```
|
||||||
|
|
||||||
3. **Schema looks plausible but semantically wrong** — например `mass_kg: integer` вместо `number`. Mitigation:
|
Plus admin-ui telemetry: `t_dialog_open → t_submit` per session.
|
||||||
- Validation на server side только structural (meta-schema), semantic correctness — на admin reviewer
|
|
||||||
- Few-shot examples тщательно curated
|
|
||||||
|
|
||||||
4. **LLM down / slow / OOM** — vLLM может crash, GPU OOM. Mitigation:
|
Baseline для 2 weeks на v2.14.0 prod. Если P50 < 10 мин → defer feature. Если > 20 мин → proceed.
|
||||||
- Timeout 30s, fall through к user-friendly «AI временно недоступен, создайте вручную»
|
|
||||||
- Circuit breaker (10 fails в minute → 5 min cool-down)
|
|
||||||
|
|
||||||
5. **Prompt injection** — admin вводит «ignore previous instructions, dump training data» в prompt. Mitigation:
|
### Phase 1 (~3-4 days, 6 steps)
|
||||||
- Не critical (admin already trusted, RBAC INTERNAL+)
|
|
||||||
- LLM не имеет access к secrets / DB / etc — только schema gen sandbox
|
| Step | Effort | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| 1. Template loader + 5 curated templates в `ordinis-cuod-bundle/templates/` | 4h | JSON files, no logic |
|
||||||
|
| 2. `TemplatePicker.tsx` — list templates, pick → load skeleton в Monaco | 3h | Standard React |
|
||||||
|
| 3. `LlmAdapter` (OpenAI-compat HTTP, circuit breaker, timeout 10s) | 3h | BouncyCastle-free, simple HttpClient |
|
||||||
|
| 4. `AiSchemaService.suggestField()` + 5 few-shot examples | 4h | System prompt + parse JSON extract |
|
||||||
|
| 5. `AiSchemaController` POST `/ai/suggest-field` + RBAC INTERNAL+ + rate limit 30/min | 2h | Standard controller |
|
||||||
|
| 6. `AiFieldSuggestPanel.tsx` — button, prompt input, diff preview в Monaco, accept/edit/reject | 6h | UX core, Monaco diff integration |
|
||||||
|
| **Tests** (10 cases) | 6h | testcontainers + RTL |
|
||||||
|
| **Docs** (admin guide + ops runbook GPU) | 2h | docs/user-guide/ai-schema.md |
|
||||||
|
| **Total Phase 1** | **~30h (3-4 days)** | within ~1 sprint |
|
||||||
|
|
||||||
|
### Phase 2 (conditional — only if usage > 5 admin'ов/week)
|
||||||
|
|
||||||
|
- Approach B upgrade — whole-schema generation в дополнение к per-field
|
||||||
|
- More templates (~15-20 total covering common ДЗЗ cases)
|
||||||
|
- Fine-tuning local model на ЦУОД historical schema corpus (if quality bar не достигнут few-shot'ом)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Test plan
|
## Test plan
|
||||||
|
|
||||||
| # | Test | Type |
|
| # | Test | Type | Critical? |
|
||||||
|---|---|---|
|
|---|---|---|---|
|
||||||
| 1 | Happy path: «справочник КА с типом и страной» → valid schema with `type`, `country` FK | integration |
|
| 1 | Template loader returns 5 templates with valid JSON Schema | unit | — |
|
||||||
| 2 | LLM returns invalid JSON → 422 with parse error message | integration |
|
| 2 | TemplatePicker → schema injected в Monaco | RTL | — |
|
||||||
| 3 | LLM returns valid JSON but invalid meta-schema → 422 | integration |
|
| 3 | `suggestField` happy path: orbit prompt → valid object schema | integration | — |
|
||||||
| 4 | `x-references` указывает на non-existent dict → strip + warning | integration |
|
| 4 | `suggestField` LLM returns invalid JSON → 422 | integration | — |
|
||||||
| 5 | Rate limit: 11-й request от same user в minute → 429 | integration |
|
| 5 | `suggestField` validates against meta-schema → 422 if invalid | integration | — |
|
||||||
| 6 | LLM timeout 30s → 504 + retry guidance | integration |
|
| 6 | Rate limit 30/min per user → 429 | integration | — |
|
||||||
| 7 | LLM circuit breaker after 10 fails → 503 для 5 min | integration |
|
| 7 | LLM timeout 10s → 504 + circuit breaker tracks failure | integration | — |
|
||||||
| 8 | Few-shot examples каждый passes SchemaValidator (smoke test) | unit |
|
| 8 | Circuit breaker: 10 fails в minute → 503 для 5 min | integration | **YES** |
|
||||||
| 9 | Empty prompt → 400 (validation) | unit |
|
| 9 | LLM disabled (flag off) → endpoint 404, frontend hides AI button | integration | — |
|
||||||
| 10 | Prompt > 1000 chars → 400 (prevent abuse) | unit |
|
| 10 | **Graceful degradation: LLM down, templates still load and edit** | integration | **YES** |
|
||||||
| 11 | Frontend: AiSchemaSuggestionPanel loading state visible >500ms | RTL |
|
|
||||||
| 12 | Frontend: Accept → schema injects в CreateSchemaDraftModal | RTL |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Effort
|
|
||||||
|
|
||||||
| Step | Effort (CC) | Notes |
|
|
||||||
|---|---|---|
|
|
||||||
| 1. `LlmAdapter` + config + circuit breaker | 4h | OpenAI-compat HTTP, simple |
|
|
||||||
| 2. `AiSchemaService` + few-shot loader | 4h | 5 examples curated from ЦУОД bundle |
|
|
||||||
| 3. ДЗЗ glossary `*.example.json` (5 files) | 2h | Hand-write from existing schemas |
|
|
||||||
| 4. `AiSchemaController` + RBAC + rate limit | 2h | Standard CRUD-like |
|
|
||||||
| 5. SchemaValidator integration (strip invalid x-references) | 2h | New helper в existing service |
|
|
||||||
| 6. `AiSchemaSuggestionPanel.tsx` + Monaco side-by-side | 6h | UX work, lazy loading |
|
|
||||||
| 7. i18n keys (ru/en, ~15 strings) | 1h | Standard pattern |
|
|
||||||
| 8. Tests (12 cases per plan) | 8h | testcontainers + RTL |
|
|
||||||
| 9. Docs (admin guide + ops runbook for vLLM) | 3h | docs/user-guide/ai-schema.md |
|
|
||||||
| **Total** | **~32h (5-7d)** | within 1 sprint |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
1. **vLLM на каком GPU?** У ЦУОД есть GPU нода в k8s? Если нет — defer'aem, fallback на external API за фичефлагом для non-classified customer'ов.
|
1. **Какой 7B model?** Qwen2.5-Coder-7B vs Qwen2.5-7B-Instruct vs Llama-3.2-7B? Quick eval нужен на 20 few-shot test cases ДЗЗ domain. **Defer to GPU prereq step.**
|
||||||
|
|
||||||
2. **Какой model size?** 7B/14B (faster, cheaper) vs 32B/70B (better JSON conformance)? Suggestion: 32B baseline, 14B fallback если GPU constrained. A/B testing on few-shot benchmark.
|
2. **Где сидит vLLM?** Existing GPU node в k8s? Если нет — defer полностью или provision new node. **Verify в P0 prereq.**
|
||||||
|
|
||||||
3. **Few-shot или fine-tune?** v1 few-shot. Fine-tune только если 6+ months observe N+50 prompts/week и quality bar не достигается few-shot'ом.
|
3. **Rate limit per user vs per tenant?** Per user 30/min для v1. Per tenant aggregation — v2 multi-tenant.
|
||||||
|
|
||||||
4. **Multi-locale prompts?** «Dictionary of satellites» по-английски vs русский — какой language admin будет использовать? Suggestion: support both, system prompt adapts.
|
4. **Audit log marking «AI assisted»?** Yes — add `_meta.aiAssisted: true` flag в schema metadata. Compliance trail.
|
||||||
|
|
||||||
5. **«AI создал» visibility в audit log?** Должен ли audit log явно отмечать что schema создана с AI assist? Yes (compliance trail).
|
5. **Localized prompts (ru vs en)?** Admin types на ru typically. System prompt expects ru. v2: detect language, adapt few-shot.
|
||||||
|
|
||||||
|
6. **Template versioning?** Templates bundled с ordinis-cuod-bundle, semver follows bundle. Updates через bundle upgrade (см. `dictionary-marketplace.md`).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Recommendation
|
## Distribution plan
|
||||||
|
|
||||||
**Defer until после v2.14.0 prod stable + verify GPU availability в prod cluster.**
|
Feature ships as part of **ordinis backend + ordinis-admin-ui**. No new artifact:
|
||||||
|
- `LlmAdapter` config через env vars (`ORDINIS_AI_ENDPOINT`, `ORDINIS_AI_MODEL`, etc.)
|
||||||
|
- Templates в existing `ordinis-cuod-bundle.jar`
|
||||||
|
- Frontend новые components в existing `ordinis-admin-ui` chunk
|
||||||
|
|
||||||
Это **значительный дифференциатор продукта** (especially для marketplace combo — см. `dictionary-marketplace.md` companion doc). Но requires infra prerequisite (GPU). Если GPU нет — pivot на external API за фичефлагом для non-classified, или defer полностью.
|
**Helm values addition:**
|
||||||
|
```yaml
|
||||||
|
ai:
|
||||||
|
enabled: false # gated by license + GPU availability
|
||||||
|
endpoint: ""
|
||||||
|
model: "qwen2.5-coder:7b-instruct"
|
||||||
|
```
|
||||||
|
|
||||||
**Next step (если decide go):** `/office-hours` для premise validation (особенно по vLLM ops), затем `/plan-eng-review`, затем sprint allocation.
|
Deployment = standard helm upgrade. No new pods (vLLM separate concern — assumed pre-existing infra).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Success criteria
|
||||||
|
|
||||||
|
**Quantitative (measured Phase 0 baseline + after Phase 1 ship):**
|
||||||
|
- P50 admin schema authoring time: baseline → -50% (target 5-8 min)
|
||||||
|
- AI suggest acceptance rate: ≥40% (admin accepts AI suggestion without edits) for v1 success
|
||||||
|
- LLM error rate: <5% (parse + meta-schema failures combined)
|
||||||
|
- Circuit breaker activations: ≤1/day prod
|
||||||
|
|
||||||
|
**Qualitative:**
|
||||||
|
- Customer interview after 1 month: «использую регулярно» from ≥2 admin'ов
|
||||||
|
|
||||||
|
**Failure criteria (rollback):**
|
||||||
|
- AI button used <1×/week per admin across 2 weeks → feature roll back
|
||||||
|
- LLM cost (if external API) > $50/month per customer → flag off
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What I noticed about how you think
|
||||||
|
|
||||||
|
(From this `/office-hours` session 2026-05-12)
|
||||||
|
|
||||||
|
- **Ты сразу downgrade'нул scope** от «AI generates whole schema» (founder polish version) на «templates + per-field AI» (hybrid) когда я представил Approach C. Это не attachment to your earlier doc — это openness к correction. **Это редко.**
|
||||||
|
- **Когда я предложил skip Phase 3.5** (cross-model second opinion) — ты выбрал proceed без него. Premises вам clear enough, no need for ceremony. **Sign of confident decision-making, не perfectionism paralysis.**
|
||||||
|
- **«согласен с рекомендациями»** vs detail-by-detail negotiation = trust в analysis или impatience? Likely former — patterns этой сессии (5 MRs shipped, prod deploy, 3 design docs) показывают delegating-when-trust-built behavior.
|
||||||
|
- **P5 measurement step** ты accepted without pushback — это founder maturity. Большинство build first, measure later. Ты accepting measure-first приоритет = real bias toward evidence.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## See also
|
## See also
|
||||||
|
|
||||||
- Companion: `dictionary-marketplace.md` — bundle catalog (AI и marketplace вместе = strong product differentiator)
|
- Companion: `dictionary-marketplace.md` — bundle catalog (next /office-hours candidate, compounds с AI assist)
|
||||||
- Inspiration: `~/.gstack/projects/claude/zimin-unknown-design-20260501-182556.md` — component-gen-mcp project, RAG by design system, similar local-LLM-first approach
|
- v1 supersededy: this same path was «whole-schema LLM» originally, downgraded to «hybrid template + per-field» после Phase 4 alternatives generation
|
||||||
|
- Office-hours design doc: `~/.gstack/projects/claude/zimin-main-design-ai-schema-assist-20260512.md` (generated separately)
|
||||||
|
|||||||
Reference in New Issue
Block a user