feat(lineage): Phase 2 — materialized view + throttled refresh + UI staleness

dict-relationships-v2 epic, Phase 2. Precomputed reverse FK index +
throttled refresh + feature flag + UI staleness badge. Migration катится
всегда; query path активируется через ordinis.lineage.mv.enabled=true.

Backend:
- Liquibase 0016: CREATE MATERIALIZED VIEW record_dependents_index +
  UNIQUE INDEX(source_record_id, source_field) (REQUIRED для REFRESH
  CONCURRENTLY) + composite indices для record-level и schema-level lookup.
  Plus lineage_mv_meta table — single-row tracking (last_refreshed_at,
  duration_ms, row_count, status, error).
- DependentsQueryMv (interface impl, @Primary + @ConditionalOnProperty)
  — один SQL query через MV вместо N JSONB lookups (по одному per
  source dict).
- MaterializedViewRefreshService:
  * @Scheduled refresh каждые ordinis.lineage.mv.refresh-interval-sec
    (default 60). Storm prevention: AtomicReference guard для in-flight,
    cooldown через ordinis.lineage.mv.min-interval-sec (default 30).
  * REFRESH CONCURRENTLY с fallback на plain REFRESH когда MV пустой
    (initial state). Не блокирует concurrent reads.
  * MvGateway interface (extracted из-за JDK 25 + Mockito incompat для
    concrete JdbcTemplate). Production: JdbcMvGateway (nested static).
- LineageIndexService теперь возвращает LineageMeta в RecordDependentsPage
  — mvEnabled, lastRefreshedAt, lastStatus, lastDurationMs. Optional<MV>
  inject — works без MV (mvEnabled=false, всё null).

Tests:
- MaterializedViewRefreshServiceTest (9 tests):
  * scheduledRefresh first time → REFRESH CONCURRENTLY
  * skip when within cooldown
  * refreshNow throws RefreshThrottledException when throttled
  * refresh succeeds после cooldown
  * fallback на plain REFRESH когда CONCURRENTLY fails
  * STORM PREVENTION (R2): 100 concurrent refresh attempts → ровно 1
    actual refresh (via cooldown + in-flight guard)
  * lastRefresh returns meta correctly
  * updates lineage_mv_meta after refresh (status=ok)
  * updates with error если оба refresh'а fail (status=error)

Admin UI:
- RecordDependentsPage type расширен LineageMeta { mvEnabled,
  lastRefreshedAt, lastStatus, lastDurationMs }.
- RecordDependentsPanel показывает staleness badge "обновлено N мин
  назад" если MV active + last refresh > 2 min ago. Tooltip объясняет
  что closed-between-refreshes records отфильтрованы query-side.
- i18n RU/EN: lineage.refs.{stale, staleHint}.

Verify:
- mvn -pl ordinis-rest-api -am test: 98/98 PASS (+9 new MV tests).
- mvn verify -pl '!ordinis-app': все модули SUCCESS (9.3s).
- pnpm tsc --noEmit: clean.
- pnpm test (vitest): 89/89 PASS.

Deploy strategy:
- Migration катится автоматом (initial REFRESH < 5 min на 5k records).
- Query path остаётся JSONB direct (Phase 1) до явного flip flag'а
  ordinis.lineage.mv.enabled=true. Это позволит верифицировать MV
  корректность на staging без риска для prod.
- После flip: refresh каждые 60 sec, stale window <= 2 min обычно.
This commit is contained in:
Zimin A.N.
2026-05-08 11:05:33 +03:00
parent 6a365fcef7
commit c06ae263e0
9 changed files with 813 additions and 10 deletions
@@ -0,0 +1,110 @@
package cloud.nstart.terravault.ordinis.domain.record;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Primary;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
/**
* MV-backed реализация {@link DependentsQuery}. Активируется через property
* {@code ordinis.lineage.mv.enabled=true} (default false). Когда выключена —
* Spring инжектит {@code DependentsQueryJdbc} (без MV, прямой JSONB query).
*
* <p>Производительность: один SQL по composite index
* {@code (target_dict_name, target_business_key_value)} вместо N JSONB
* lookups (по одному на каждый source dict).
*
* <p>Stale window: MV refresh throttled на 1/min (см. {@code MaterializedViewRefreshService}).
* Записи закрытые между refresh'ами всё равно отфильтрованы query-side через
* {@code valid_from/valid_to} columns в MV — actual valid_to update сделан
* через {@code DictionaryRecord} table refresh, MV отстаёт только по новым
* referencing records которые ещё не подхвачены MV.
*
* <p>Active filter в WHERE clause использует {@code valid_from <= ? AND valid_to > ?}
* (тот же at) что и в JdbcImpl — consistent semantics.
*/
@Component
@Primary
@ConditionalOnProperty(name = "ordinis.lineage.mv.enabled", havingValue = "true", matchIfMissing = false)
public class DependentsQueryMv implements DependentsQuery {
private static final Logger log = LoggerFactory.getLogger(DependentsQueryMv.class);
private final JdbcTemplate jdbc;
@Autowired
public DependentsQueryMv(JdbcTemplate jdbc) {
this.jdbc = jdbc;
log.info("DependentsQuery: MV-backed implementation активирована "
+ "(ordinis.lineage.mv.enabled=true). Query path использует "
+ "record_dependents_index materialized view.");
}
@Override
public List<DependentRow> findActive(
UUID sourceDictionaryId,
String sourceField,
String targetValue,
OffsetDateTime at,
int limit,
int offset) {
if (limit <= 0 || limit > 500) {
throw new IllegalArgumentException("limit must be 1..500, got " + limit);
}
if (offset < 0) {
throw new IllegalArgumentException("offset must be >= 0, got " + offset);
}
// MV path: source_dict_id + source_field + target_value уникально определяет ряды.
String sql = """
SELECT source_record_id, source_dict_id, source_business_key,
valid_from, valid_to, created_at
FROM record_dependents_index
WHERE source_dict_id = ?
AND source_field = ?
AND target_business_key_value = ?
AND valid_from <= ?
AND valid_to > ?
ORDER BY created_at DESC
LIMIT ? OFFSET ?
""";
return jdbc.query(
sql,
(rs, n) -> new DependentRow(
(UUID) rs.getObject("source_record_id"),
(UUID) rs.getObject("source_dict_id"),
rs.getString("source_business_key"),
rs.getObject("valid_from", OffsetDateTime.class),
rs.getObject("valid_to", OffsetDateTime.class),
rs.getObject("created_at", OffsetDateTime.class)),
sourceDictionaryId, sourceField, targetValue, at, at, limit, offset);
}
@Override
public long countActive(
UUID sourceDictionaryId,
String sourceField,
String targetValue,
OffsetDateTime at) {
String sql = """
SELECT COUNT(*) FROM record_dependents_index
WHERE source_dict_id = ?
AND source_field = ?
AND target_business_key_value = ?
AND valid_from <= ?
AND valid_to > ?
""";
Long count = jdbc.queryForObject(sql, Long.class,
sourceDictionaryId, sourceField, targetValue, at, at);
return count == null ? 0 : count;
}
}