feat(lineage): Phase 2 — materialized view + throttled refresh + UI staleness
dict-relationships-v2 epic, Phase 2. Precomputed reverse FK index +
throttled refresh + feature flag + UI staleness badge. Migration катится
всегда; query path активируется через ordinis.lineage.mv.enabled=true.
Backend:
- Liquibase 0016: CREATE MATERIALIZED VIEW record_dependents_index +
UNIQUE INDEX(source_record_id, source_field) (REQUIRED для REFRESH
CONCURRENTLY) + composite indices для record-level и schema-level lookup.
Plus lineage_mv_meta table — single-row tracking (last_refreshed_at,
duration_ms, row_count, status, error).
- DependentsQueryMv (interface impl, @Primary + @ConditionalOnProperty)
— один SQL query через MV вместо N JSONB lookups (по одному per
source dict).
- MaterializedViewRefreshService:
* @Scheduled refresh каждые ordinis.lineage.mv.refresh-interval-sec
(default 60). Storm prevention: AtomicReference guard для in-flight,
cooldown через ordinis.lineage.mv.min-interval-sec (default 30).
* REFRESH CONCURRENTLY с fallback на plain REFRESH когда MV пустой
(initial state). Не блокирует concurrent reads.
* MvGateway interface (extracted из-за JDK 25 + Mockito incompat для
concrete JdbcTemplate). Production: JdbcMvGateway (nested static).
- LineageIndexService теперь возвращает LineageMeta в RecordDependentsPage
— mvEnabled, lastRefreshedAt, lastStatus, lastDurationMs. Optional<MV>
inject — works без MV (mvEnabled=false, всё null).
Tests:
- MaterializedViewRefreshServiceTest (9 tests):
* scheduledRefresh first time → REFRESH CONCURRENTLY
* skip when within cooldown
* refreshNow throws RefreshThrottledException when throttled
* refresh succeeds после cooldown
* fallback на plain REFRESH когда CONCURRENTLY fails
* STORM PREVENTION (R2): 100 concurrent refresh attempts → ровно 1
actual refresh (via cooldown + in-flight guard)
* lastRefresh returns meta correctly
* updates lineage_mv_meta after refresh (status=ok)
* updates with error если оба refresh'а fail (status=error)
Admin UI:
- RecordDependentsPage type расширен LineageMeta { mvEnabled,
lastRefreshedAt, lastStatus, lastDurationMs }.
- RecordDependentsPanel показывает staleness badge "обновлено N мин
назад" если MV active + last refresh > 2 min ago. Tooltip объясняет
что closed-between-refreshes records отфильтрованы query-side.
- i18n RU/EN: lineage.refs.{stale, staleHint}.
Verify:
- mvn -pl ordinis-rest-api -am test: 98/98 PASS (+9 new MV tests).
- mvn verify -pl '!ordinis-app': все модули SUCCESS (9.3s).
- pnpm tsc --noEmit: clean.
- pnpm test (vitest): 89/89 PASS.
Deploy strategy:
- Migration катится автоматом (initial REFRESH < 5 min на 5k records).
- Query path остаётся JSONB direct (Phase 1) до явного flip flag'а
ordinis.lineage.mv.enabled=true. Это позволит верифицировать MV
корректность на staging без риска для prod.
- После flip: refresh каждые 60 sec, stale window <= 2 min обычно.
This commit is contained in:
+160
@@ -0,0 +1,160 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<databaseChangeLog
|
||||
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
|
||||
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.27.xsd">
|
||||
|
||||
<!--
|
||||
dict-relationships-v2 epic, Phase 2: precomputed reverse FK index.
|
||||
|
||||
Materialized view `record_dependents_index` сворачивает все active FK
|
||||
relationships (source_record + source_field → target_dict + target_value)
|
||||
в один таблично-индексированный snapshot. Даёт O(log n) lookup для
|
||||
record-level dependents query — без JSONB предикатов в hot path.
|
||||
|
||||
Включение per request: feature flag `ordinis.lineage.mv.enabled`
|
||||
(default false). Migration катится всегда — flag решает, использует
|
||||
ли service path MV vs прямой JSONB query (Phase 1 implementation).
|
||||
|
||||
Refresh: REFRESH MATERIALIZED VIEW CONCURRENTLY (требует UNIQUE INDEX).
|
||||
Throttled на app-side через `MaterializedViewRefreshService` — default
|
||||
1 refresh / 60 sec, чтобы избежать storm при bulk close (R2 eng review).
|
||||
|
||||
Active record check сделан query-side (на valid_from/valid_to колонках),
|
||||
чтобы записи закрытые между refresh'ами не появились в результатах
|
||||
(stale window — "data may be stale: refreshed N min ago").
|
||||
|
||||
Initial population SLO: <5 min на 5k records (тест plan target). На
|
||||
pure JSONB indexing (jsonb_each + ?-marker) — не ожидаем больше.
|
||||
-->
|
||||
<changeSet id="0016-record-dependents-index-create" author="ordinis">
|
||||
<comment>MV record_dependents_index: precomputed reverse FK</comment>
|
||||
<preConditions onFail="MARK_RAN">
|
||||
<not>
|
||||
<sqlCheck expectedResult="1">
|
||||
SELECT COUNT(*) FROM pg_matviews WHERE matviewname = 'record_dependents_index'
|
||||
</sqlCheck>
|
||||
</not>
|
||||
</preConditions>
|
||||
<sql splitStatements="false"><![CDATA[
|
||||
CREATE MATERIALIZED VIEW record_dependents_index AS
|
||||
SELECT
|
||||
r.id AS source_record_id,
|
||||
r.dictionary_id AS source_dict_id,
|
||||
src_def.name AS source_dict_name,
|
||||
src_def.display_name AS source_display_name,
|
||||
src_def.scope AS source_scope,
|
||||
r.business_key AS source_business_key,
|
||||
r.data_scope AS source_data_scope,
|
||||
r.valid_from,
|
||||
r.valid_to,
|
||||
r.created_at,
|
||||
prop.key AS source_field,
|
||||
split_part(prop.value->>'x-references', '.', 1) AS target_dict_name,
|
||||
split_part(prop.value->>'x-references', '.', 2) AS target_field,
|
||||
UPPER(COALESCE(prop.value->>'x-references-on-close', 'BLOCK')) AS on_close,
|
||||
r.data ->> prop.key AS target_business_key_value
|
||||
FROM dictionary_records r
|
||||
JOIN dictionary_definitions src_def ON src_def.id = r.dictionary_id
|
||||
JOIN LATERAL jsonb_each(src_def.schema_json -> 'properties') AS prop(key, value) ON true
|
||||
WHERE prop.value ? 'x-references'
|
||||
AND prop.value->>'x-references' ~ '^[a-zA-Z_][a-zA-Z0-9_]*\.[a-zA-Z_][a-zA-Z0-9_]*$'
|
||||
AND r.data ? prop.key
|
||||
AND jsonb_typeof(r.data -> prop.key) = 'string'
|
||||
WITH NO DATA;
|
||||
]]></sql>
|
||||
<rollback>
|
||||
<sql>DROP MATERIALIZED VIEW IF EXISTS record_dependents_index;</sql>
|
||||
</rollback>
|
||||
</changeSet>
|
||||
|
||||
<!--
|
||||
UNIQUE INDEX (source_record_id, source_field) — REQUIRED для REFRESH
|
||||
CONCURRENTLY. Composite потому что одна запись может иметь несколько
|
||||
FK fields в schema (spacecraft.satTypeCode + spacecraft.operatorId).
|
||||
-->
|
||||
<changeSet id="0016-record-dependents-index-unique" author="ordinis">
|
||||
<comment>UNIQUE INDEX для REFRESH CONCURRENTLY</comment>
|
||||
<sql>
|
||||
CREATE UNIQUE INDEX uniq_record_dependents_index_source
|
||||
ON record_dependents_index (source_record_id, source_field);
|
||||
</sql>
|
||||
<rollback>
|
||||
<sql>DROP INDEX IF EXISTS uniq_record_dependents_index_source;</sql>
|
||||
</rollback>
|
||||
</changeSet>
|
||||
|
||||
<!--
|
||||
Main lookup index: (target_dict_name, target_business_key_value).
|
||||
Includes valid_from/valid_to для query-side active filter без
|
||||
дополнительного table scan.
|
||||
-->
|
||||
<changeSet id="0016-record-dependents-index-target-lookup" author="ordinis">
|
||||
<comment>Composite index для record-level dependents lookup</comment>
|
||||
<sql>
|
||||
CREATE INDEX idx_record_dependents_target
|
||||
ON record_dependents_index
|
||||
(target_dict_name, target_business_key_value, valid_from, valid_to);
|
||||
</sql>
|
||||
<rollback>
|
||||
<sql>DROP INDEX IF EXISTS idx_record_dependents_target;</sql>
|
||||
</rollback>
|
||||
</changeSet>
|
||||
|
||||
<!--
|
||||
Schema-level lookup: (target_dict_name) для "какие словари ссылаются".
|
||||
Не нужен composite — только префикс target_dict_name + DISTINCT
|
||||
source_dict_name fingerprint берётся в SQL.
|
||||
-->
|
||||
<changeSet id="0016-record-dependents-index-schema-lookup" author="ordinis">
|
||||
<comment>Index для schema-level reverse FK list</comment>
|
||||
<sql>
|
||||
CREATE INDEX idx_record_dependents_schema
|
||||
ON record_dependents_index (target_dict_name);
|
||||
</sql>
|
||||
<rollback>
|
||||
<sql>DROP INDEX IF EXISTS idx_record_dependents_schema;</sql>
|
||||
</rollback>
|
||||
</changeSet>
|
||||
|
||||
<!--
|
||||
Initial population. Не блокирует existing reads/writes (REFRESH без
|
||||
CONCURRENTLY на пустой view — fast path). На 5k records ожидание <5 min.
|
||||
-->
|
||||
<changeSet id="0016-record-dependents-index-populate" author="ordinis">
|
||||
<comment>Initial REFRESH (no CONCURRENTLY на первый populate)</comment>
|
||||
<sql>REFRESH MATERIALIZED VIEW record_dependents_index;</sql>
|
||||
<rollback>
|
||||
<!-- No-op: rollback drops view целиком. -->
|
||||
</rollback>
|
||||
</changeSet>
|
||||
|
||||
<!--
|
||||
Refresh metadata table — single-row для tracking last refresh timestamp,
|
||||
duration, row count. Используется service для exposure через API
|
||||
(stale window UX), и admin endpoint для health check.
|
||||
-->
|
||||
<changeSet id="0016-record-dependents-refresh-meta" author="ordinis">
|
||||
<comment>Metadata табличка для last MV refresh tracking</comment>
|
||||
<createTable tableName="lineage_mv_meta">
|
||||
<column name="id" type="SMALLINT" defaultValueNumeric="1">
|
||||
<constraints primaryKey="true" nullable="false"/>
|
||||
</column>
|
||||
<column name="last_refreshed_at" type="TIMESTAMPTZ"/>
|
||||
<column name="last_duration_ms" type="BIGINT"/>
|
||||
<column name="last_row_count" type="BIGINT"/>
|
||||
<column name="last_status" type="VARCHAR(32)" defaultValue="never"/>
|
||||
<column name="last_error" type="TEXT"/>
|
||||
</createTable>
|
||||
<sql>
|
||||
ALTER TABLE lineage_mv_meta
|
||||
ADD CONSTRAINT lineage_mv_meta_singleton CHECK (id = 1);
|
||||
</sql>
|
||||
<insert tableName="lineage_mv_meta">
|
||||
<column name="id" valueNumeric="1"/>
|
||||
<column name="last_status" value="never"/>
|
||||
</insert>
|
||||
</changeSet>
|
||||
|
||||
</databaseChangeLog>
|
||||
@@ -25,5 +25,6 @@
|
||||
<include file="changes/0013-webhook-subscriptions.xml" relativeToChangelogFile="true"/>
|
||||
<include file="changes/0014-webhook-deliveries.xml" relativeToChangelogFile="true"/>
|
||||
<include file="changes/0015-backfill-latlon-geometry.xml" relativeToChangelogFile="true"/>
|
||||
<include file="changes/0016-record-dependents-index.xml" relativeToChangelogFile="true"/>
|
||||
|
||||
</databaseChangeLog>
|
||||
|
||||
Reference in New Issue
Block a user