feat(lineage): Phase 2 — materialized view + throttled refresh + UI staleness
dict-relationships-v2 epic, Phase 2. Precomputed reverse FK index +
throttled refresh + feature flag + UI staleness badge. Migration катится
всегда; query path активируется через ordinis.lineage.mv.enabled=true.
Backend:
- Liquibase 0016: CREATE MATERIALIZED VIEW record_dependents_index +
UNIQUE INDEX(source_record_id, source_field) (REQUIRED для REFRESH
CONCURRENTLY) + composite indices для record-level и schema-level lookup.
Plus lineage_mv_meta table — single-row tracking (last_refreshed_at,
duration_ms, row_count, status, error).
- DependentsQueryMv (interface impl, @Primary + @ConditionalOnProperty)
— один SQL query через MV вместо N JSONB lookups (по одному per
source dict).
- MaterializedViewRefreshService:
* @Scheduled refresh каждые ordinis.lineage.mv.refresh-interval-sec
(default 60). Storm prevention: AtomicReference guard для in-flight,
cooldown через ordinis.lineage.mv.min-interval-sec (default 30).
* REFRESH CONCURRENTLY с fallback на plain REFRESH когда MV пустой
(initial state). Не блокирует concurrent reads.
* MvGateway interface (extracted из-за JDK 25 + Mockito incompat для
concrete JdbcTemplate). Production: JdbcMvGateway (nested static).
- LineageIndexService теперь возвращает LineageMeta в RecordDependentsPage
— mvEnabled, lastRefreshedAt, lastStatus, lastDurationMs. Optional<MV>
inject — works без MV (mvEnabled=false, всё null).
Tests:
- MaterializedViewRefreshServiceTest (9 tests):
* scheduledRefresh first time → REFRESH CONCURRENTLY
* skip when within cooldown
* refreshNow throws RefreshThrottledException when throttled
* refresh succeeds после cooldown
* fallback на plain REFRESH когда CONCURRENTLY fails
* STORM PREVENTION (R2): 100 concurrent refresh attempts → ровно 1
actual refresh (via cooldown + in-flight guard)
* lastRefresh returns meta correctly
* updates lineage_mv_meta after refresh (status=ok)
* updates with error если оба refresh'а fail (status=error)
Admin UI:
- RecordDependentsPage type расширен LineageMeta { mvEnabled,
lastRefreshedAt, lastStatus, lastDurationMs }.
- RecordDependentsPanel показывает staleness badge "обновлено N мин
назад" если MV active + last refresh > 2 min ago. Tooltip объясняет
что closed-between-refreshes records отфильтрованы query-side.
- i18n RU/EN: lineage.refs.{stale, staleHint}.
Verify:
- mvn -pl ordinis-rest-api -am test: 98/98 PASS (+9 new MV tests).
- mvn verify -pl '!ordinis-app': все модули SUCCESS (9.3s).
- pnpm tsc --noEmit: clean.
- pnpm test (vitest): 89/89 PASS.
Deploy strategy:
- Migration катится автоматом (initial REFRESH < 5 min на 5k records).
- Query path остаётся JSONB direct (Phase 1) до явного flip flag'а
ordinis.lineage.mv.enabled=true. Это позволит верифицировать MV
корректность на staging без риска для prod.
- После flip: refresh каждые 60 sec, stale window <= 2 min обычно.
This commit is contained in:
@@ -276,6 +276,16 @@ export type PerSourceSummary = {
|
||||
count: number
|
||||
}
|
||||
|
||||
export type LineageMeta = {
|
||||
/** True если backend использует materialized view; false — JSONB direct (fresh). */
|
||||
mvEnabled: boolean
|
||||
/** ISO timestamp последнего успешного MV refresh; null если MV выключена / never. */
|
||||
lastRefreshedAt: string | null
|
||||
/** "ok" | "error" | "never" — null если mvEnabled=false. */
|
||||
lastStatus: string | null
|
||||
lastDurationMs: number | null
|
||||
}
|
||||
|
||||
export type RecordDependentsPage = {
|
||||
targetDict: string
|
||||
targetBusinessKey: string
|
||||
@@ -284,6 +294,8 @@ export type RecordDependentsPage = {
|
||||
total: number
|
||||
page: number
|
||||
size: number
|
||||
/** Phase 2 dict-relationships-v2: backing data source metadata + staleness. */
|
||||
lineageMeta?: LineageMeta | null
|
||||
}
|
||||
|
||||
export type WebhookTestPingResult = {
|
||||
|
||||
@@ -52,15 +52,38 @@ export const RecordDependentsPanel = ({ dictionaryName, businessKey }: Props) =>
|
||||
|
||||
const totalPages = Math.max(1, Math.ceil(data.total / size))
|
||||
|
||||
// Staleness: show если MV active + last refresh > 2 min ago.
|
||||
// <2 min — fresh enough, не отвлекаем UI'ем.
|
||||
const stale =
|
||||
data.lineageMeta?.mvEnabled && data.lineageMeta.lastRefreshedAt
|
||||
? Math.max(
|
||||
0,
|
||||
Math.floor(
|
||||
(Date.now() - new Date(data.lineageMeta.lastRefreshedAt).getTime()) /
|
||||
60000,
|
||||
),
|
||||
)
|
||||
: null
|
||||
|
||||
return (
|
||||
<section className="space-y-3">
|
||||
<div className="flex items-center justify-between">
|
||||
<h3 className="text-sm font-primary text-carbon">
|
||||
{t('lineage.refs.title')}
|
||||
</h3>
|
||||
<span className="text-2xs text-carbon/60">
|
||||
{t('lineage.refs.total', { count: data.total })}
|
||||
</span>
|
||||
<div className="flex items-center gap-2">
|
||||
{stale !== null && stale >= 2 && (
|
||||
<span
|
||||
className="text-2xs text-carbon/60 px-2 py-0.5 rounded-sm border border-regolith"
|
||||
title={t('lineage.refs.staleHint')}
|
||||
>
|
||||
{t('lineage.refs.stale', { minutes: stale })}
|
||||
</span>
|
||||
)}
|
||||
<span className="text-2xs text-carbon/60">
|
||||
{t('lineage.refs.total', { count: data.total })}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Per-source summary chips */}
|
||||
|
||||
@@ -249,6 +249,8 @@ i18n
|
||||
'lineage.refs.field': 'поле {{field}}',
|
||||
'lineage.refs.created': 'создана {{when}}',
|
||||
'lineage.refs.openSourceRecord': 'Открыть запись {{key}} в источнике',
|
||||
'lineage.refs.stale': 'обновлено {{minutes}} мин назад',
|
||||
'lineage.refs.staleHint': 'Данные из materialized view; обновляются раз в минуту. Закрытые между обновлениями записи отфильтрованы по valid_to.',
|
||||
'lineage.onClose.BLOCK': 'block',
|
||||
'lineage.onClose.WARN': 'warn',
|
||||
'lineage.onClose.CASCADE': 'cascade',
|
||||
@@ -552,6 +554,8 @@ i18n
|
||||
'lineage.refs.field': 'field {{field}}',
|
||||
'lineage.refs.created': 'created {{when}}',
|
||||
'lineage.refs.openSourceRecord': 'Open referencing record {{key}}',
|
||||
'lineage.refs.stale': 'refreshed {{minutes}} min ago',
|
||||
'lineage.refs.staleHint': 'Data from materialized view, refreshed once per minute. Records closed between refreshes are filtered by valid_to.',
|
||||
'lineage.onClose.BLOCK': 'block',
|
||||
'lineage.onClose.WARN': 'warn',
|
||||
'lineage.onClose.CASCADE': 'cascade',
|
||||
|
||||
Reference in New Issue
Block a user