Commit Graph

21 Commits

Author SHA1 Message Date
Zimin A.N. 7abff8f73f fix(security): CSO findings — non-root containers + .env gitignore + actuator hardening
Три прямых фикса по /cso security audit на v2.45.2:

## Finding #1 (HIGH): Backend containers run as root

ordinis-app / ordinis-read-api / ordinis-projection-writer Dockerfile'ы
запускали JVM от root (eclipse-temurin:21-jre default'но не задаёт USER).
Compromise через Spring CVE / Jackson deserialization → root-в-контейнере
= escalation surface на ноду через любой pod misconfig.

Добавлено: `RUN groupadd -r -g 10001 app && useradd -r -u 10001 -g app
-s /sbin/nologin app` + `USER app`. UID 10001 — outside system (0-999)
и distribution reserved (1000-9999). `--chown=app:app` на COPY чтобы
JVM могла прочитать jar.

ordinis-migrations (USER liquibase) и docs (nginx default) уже non-root.

## Finding #2 (HIGH): .env / .env.local not in .gitignore

`.gitignore` блокировал *.pem, *.key, .gstack/, .claude/ — но не .env.
Развработчик с realный dotenv для local dev мог случайно `git add .` и
закоммитить creds. Сейчас нет committed .env (verified) но защита нулевая.

Добавлено: `.env`, `.env.local`, `.env.*.local`, `*.env` + whitelist
`!.env.example`, `!.env.template`, `!.env.sample` для шаблонов.

Существующий tracked `ordinis-admin-ui/.env.example` остался tracked
(verified `git ls-files | grep .env.example`).

## Finding #3 (MEDIUM): Spring Actuator permitAll() — defense-in-depth gap

SecurityConfig had `requestMatchers("/actuator/**").permitAll()` →
любой pod в кластере мог `curl ordinis-app:8080/actuator/env` и
получить ORDINIS_KEYCLOAK_ADMIN_CLIENT_SECRET, DB creds и пр.

VERIFIED: nginx ingress на проде НЕ проксирует /actuator/* (SPA fallback
отдаёт index.html), так что не exposed на интернет. Но pod-to-pod в
кластере = открыто.

Defense in two layers:

1. SecurityConfig: split actuator routes —
     /actuator/health/**, /actuator/info, /actuator/prometheus → permitAll
     /actuator/**                                              → denyAll
   Probes/Prometheus scrape работают, остальное блок.

2. application.yml для всех трёх Spring apps: narrowed default exposure
   to "health,info,prometheus". Поддерживается env override
   MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE для staging diagnostic
   (включить env/configprops/loggers только когда нужно дебажить
   autoconfig).

## Тесты

- mvn package — все 15 модулей SUCCESS
- ordinis-rest-api + ordinis-auth tests — SUCCESS

## Manual verify post-deploy

После пророллится v2.45.3:
- staging+prod liveness/readiness probes должны остаться зелёными
  (/actuator/health/{liveness,readiness} в permitAll list).
- Prometheus scrape /actuator/prometheus → 200.
- ssh pod && curl localhost:8080/actuator/env → должен вернуть
  401/403 (или 404 если endpoint вообще выключен через exposure).

## Откат

Если probes сломаются (маловероятно — health endpoints whitelisted) —
revert этого MR. Image rollback к v2.45.2 через helm.
2026-05-27 19:02:34 +03:00
Zimin A.N. e4cd7b9709 feat(ai): Phase 1 step 3-6 — LLM adapter + per-field AI suggest endpoint + UI
Approach C upgrade на templates foundation (!233). Admin может попросить AI
предложить новое поле — system prompt force'ит structural JSON Schema
fragment, admin review через accept/edit/reject. Bitemporal draft workflow
unchanged downstream.

## Backend
- `LlmAdapter` — OpenAI-compatible /v1/chat/completions client (Ollama,
  vLLM, OpenAI). Bean conditional на ordinis.ai.enabled=true.
- `AiSchemaService` — system prompt + 3 few-shot examples, JSON output
  parsing (с markdown fence stripping), validation (fieldName snake_case).
  In-memory circuit breaker: 10 fails в 60s → open 5 min.
- `AiSchemaController`:
  - GET /api/v1/ai/info → 200 (enabled) / 404 (disabled). Frontend probe.
  - POST /api/v1/ai/suggest-field → suggestion. Per-IP rate limit 30/min.
- All bean-conditional: ordinis.ai.enabled=false → controllers нет →
  endpoints 404 → frontend hides AI buttons automatically.

## Frontend
- `useAiFeatureAvailable` query — probes /ai/info, cache 5 min
- `useAiSuggestField` mutation — 30s timeout (LLM может быть slow)
- `AiFieldSuggestPanel` component — inline expandable: prompt → preview →
  accept/retry/reject. Error handling per HTTP status (503/429/422/502/404).
- DictionaryEditorDialog: AI toggle visible на schema tab когда
  aiAvailable=true. Accept wraps suggestion в mini-schema → parseSchemaJson
  (reuse existing kind-inference) → push в properties (overwrite если
  duplicate fieldName).
- i18n RU + EN

## Config
- application.yml: ordinis.ai.{enabled,endpoint,model,bearer-token,
  timeout-seconds}
- Helm writer.yaml: ORDINIS_AI_* env vars, optional secretRef для bearer
- values.yaml: ai.{enabled,endpoint,model,timeoutSeconds,bearerTokenSecretRef}
  defaults disabled

Pair с infra MR — values-staging enables AI с vortex.nstart.cloud Ollama
serving qwen2.5:7b.
2026-05-16 13:55:16 +03:00
Александр Зимин a1989da1be fix(actuator): disable Redis health indicator in writer 2026-05-15 15:20:54 +00:00
Александр Зимин fdf321316f fix(actuator): expose loggers endpoint + force release trigger 2026-05-15 09:35:14 +00:00
Александр Зимин 20d6c919cd chore(actuator): expose conditions/beans/configprops для autoconfig debug 2026-05-15 09:14:11 +00:00
Александр Зимин 9050e2427e feat(notifications): TODO 7 — draft decision toast + email + bell badge 2026-05-14 14:40:35 +00:00
Andrei Zimin 83caf9c3c8 feat(user-display): KeycloakAdminUserResolver — Phase 2 on-demand lookup
Implements the Phase 2 hook from the previous commit. With this in
place, UserDisplayService.find chains:

  hot cache → DB (user_display_cache) → Keycloak Admin API

so any realm user resolves on first display, even if they never made
an authenticated request to ordinis. The resolved entry is persisted
via UserDisplayService.upsert with source=KEYCLOAK_ON_DEMAND, so the
next render of the same sub hits the DB tier (~ms) instead of going
back to Keycloak.

Implementation notes:

- Spring RestClient (modern blocking HTTP, replaces RestTemplate).
- client_credentials grant against
  {issuer}/realms/{realm}/protocol/openid-connect/token. Token cached
  in-memory minus a 30s safety margin from `expires_in`.
- 401 on user lookup wipes the cached token and surfaces a
  RuntimeException, so the next resolve refetches.
- 404 on user lookup → Optional.empty (definitive miss, sub not in
  realm). UserDisplayService logs and the caller falls back to short
  UUID. Negative cache TTL not implemented — added complexity not
  justified for the current realm size (~50 users).
- Bean activated only when ordinis.keycloak.admin.enabled=true AND
  the env vars are set. Default disabled — UserDisplayService treats
  KeycloakUserResolver as @Autowired(required = false), so a
  deployment without Keycloak admin creds keeps the previous behavior
  (DB cache only) instead of crashing on startup.

Application config:
- ORDINIS_KEYCLOAK_ADMIN_ENABLED      (false by default)
- ORDINIS_KEYCLOAK_ADMIN_URL          e.g. https://auth.nstart.space
- ORDINIS_KEYCLOAK_ADMIN_REALM        defaults to nstart
- ORDINIS_KEYCLOAK_ADMIN_CLIENT_ID    via vault (see docs-internal/keycloak-admin-setup.md)
- ORDINIS_KEYCLOAK_ADMIN_CLIENT_SECRET via vault

Setup runbook (docs-internal/keycloak-admin-setup.md) covers:
- creating the Keycloak service-account client + assigning view-users
- writing the secret into both vault instances (different per env)
- wiring vault.hashicorp.com agent-inject annotations into the chart
- verification steps + common failure modes

Phase 3 (scheduled bulk sync) is documented at the end of the runbook
but deferred — depends on observed cache miss rate after this lands.
2026-05-14 14:47:49 +03:00
Александр Зимин e2080d8991 fix(docs,security): Swagger UI — staging для exploration, prod env-flagged 2026-05-12 17:43:12 +00:00
Zimin A.N. b601c4636f feat(swagger): OAuth2/Keycloak PKCE flow в Swagger UI + docs links
OpenApiConfig:
- Добавлена OAuth2 security scheme (authorization_code flow с Keycloak
  realm `nstart`). Auth/token URLs из ordinis.openapi.oauth.issuer-uri
  configuration property (default: auth.nstart.space/realms/nstart).
- BearerAuth scheme сохранён secondary — для pre-issued tokens из CI/scripts.
- Servers list переупорядочен — prod первый, staging второй.
- Description обновлён с pointer на /docs/ для full integration guide.

application.yml:
- springdoc.swagger-ui.oauth: client-id (env override), use-pkce-with-
  authorization-code-grant=true. Public client `ordinis-swagger`
  настраивается отдельно в Keycloak — see keycloak-setup-swagger-client.md
  в ordinis-infra repo.
- ORDINIS_OPENAPI_OAUTH_ISSUER_URI env var для override realm на разных
  окружениях.

docs/integration/api/read-endpoints.md:
- Swagger UI section обновлён: акцент на Authorize button + Keycloak
  PKCE login (no client_secret needed), tip про bearerAuth fallback.

docs/index.md:
- Добавлен tab "Я хочу пощупать API" с link на Swagger UI.

После deploy:
- https://ordinis.k8s.264.nstart.cloud/swagger-ui.html  — Swagger UI
- https://ordinis.k8s.264.nstart.cloud/v3/api-docs      — OpenAPI spec
Authorize → выбрать oauth2 → Keycloak login (PKCE) → Try it out с JWT.
2026-05-08 00:08:29 +03:00
Zimin A.N. 3a28d741e9 feat(metrics): включить percentiles-histogram для http.server.requests
Spring Boot Micrometer по дефолту эмитит только _count/_sum/_max
для http.server.requests — без _bucket нельзя посчитать
histogram_quantile() в Prometheus, p50/p95/p99 latency дашборды
получают "No data".

Включён percentiles-histogram + кастомные SLO bucket boundaries
от 5ms до 5s — покрывают realistic range read-api JSON endpoints.
Server-side histograms точнее client-side percentiles (по которым
нельзя aggregate across instances).

Применить: build + push image + helm upgrade. После рестарта
ordinis-app/read-api/projection-writer Grafana дашборды покажут
latency графики.
2026-05-06 11:10:33 +03:00
Zimin A.N. b58f6f400d fix(otel): отключить metrics/logs export — Tempo не принимает их
Tempo на staging (tempo.monitoring:4318) — single-process, принимает
только traces. Spring Boot OTel autoconfig по умолчанию отправляет всё
по тому же endpoint → 404 на каждой попытке (раз в секунду в логах
writer/read-api/projection-writer).

Override через ENV — можно переключить когда поднимем полный stack
(Mimir для metrics, Loki для logs).

Метрики Prometheus продолжают работать через actuator/prometheus
(scrape снаружи), это отдельный path.
2026-05-05 17:33:49 +03:00
Zimin A.N. e182b30c58 feat(outbox): attempt cap → DLQ + admin endpoint + 6 unit-тестов
После N (default 1000) неудачных попыток публикации событие маркируется
dlq_at и исключается из polling. Без cap poller бесконечно ретраил мёртвые
сообщения (orphan topic, ACL deny), забивая логи и lag-метрику.

- 0011-outbox-dlq.xml: dlq_at column + перестроенный idx_outbox_unpublished
  (исключает DLQ) + idx_outbox_dlq для admin-листинга.
- OutboxPoller: ordinis.outbox.attempt-max (default 1000), markDlq() при
  достижении cap, новый counter ordinis_outbox_dlq_total.
- OutboxLagGauge: ordinis_outbox_dlq_size gauge — алерт на > 0.
- OutboxEventRepository: countPending() (без DLQ), findByDlqAtIsNotNull(Pageable).
- OutboxAdminController: GET /admin/outbox/{stats,dlq} для UI.
- 6 unit-тестов через mock-maker-subclass (JDK 25 ломает Mockito inline).
2026-05-04 15:43:56 +03:00
Александр Зимин 6ee7a70ad1 feat(auth): JWT scope authorization (@nstart/auth контракт)
Новый модуль ordinis-auth:
- OrdinisAuthProperties — public-key, issuer, requireAuthentication, allowQueryScope
- ScopeContext — резолвит current scopes из JWT claim 'scopes' (приоритет),
  потом legacy ?as_scope=, потом anonymous=PUBLIC
- SecurityConfig — Spring Security OAuth2 Resource Server + RSA public-key
  decoder. Если public-key пуст — JWT validation отключён (permissive
  fallback); rollout без auth-сервера не ломает API.

Read controller теперь делегирует scope resolution в ScopeContext вместо
прямого парсинга ?as_scope=. Backward compat: legacy query параметр
работает пока ordinis.auth.allow-query-scope=true (по умолчанию true в
dev/staging, false в prod после интеграции с @nstart/auth).

Vault: public-key читается из secret/ordinis/cuod/jwt-public-key (key=key).
Spring Cloud Vault flatten'ит в property 'key', resolved в
ordinis.auth.public-key=${key:}.
2026-05-04 00:28:27 +03:00
Александр Зимин b5b5bc512b fix(otel): use OTLP HTTP/protobuf to tempo:4318 instead of gRPC:4317
Stack trace showed io.opentelemetry.exporter.internal.http.HttpExporter
sending to :4317 — Tempo on 4317 expects gRPC frames, gets HTTP, sends
RST. Switch endpoint to :4318 (Tempo OTLP HTTP receiver) and protocol to
http/protobuf so the SDK and Tempo speak the same protocol.

The gRPC config never took effect — Spring Boot SDK on micrometer-tracing-
otlp pulls in only the HTTP exporter by default; gRPC needs the explicit
opentelemetry-exporter-otlp jar.
2026-05-03 22:19:32 +03:00
Александр Зимин 1c4e8e5be2 fix(app): switch hibernate ddl-auto to none in k8s profile
Hibernate validate mode threw 'wrong column type [supported_locales]:
found [_text (Types#ARRAY)] but expecting [text[] (Types#OTHER)]' against
the Liquibase-managed schema. This is a well-known hypersistence-utils +
PostgreSQL TEXT[] friction with Hibernate validate.

Schema is owned by Liquibase migrations; validate adds no safety here. Set
to 'none' (matches read-api and projection-writer).
2026-05-03 20:33:21 +03:00
Александр Зимин d49dd6be06 fix(k8s): add datasource/kafka/redis config to k8s profile
K8s profile relied on shared/dev defaults for spring.datasource.url, but those
live inside the dev profile block — invisible when k8s profile is active.
Result: 'APPLICATION FAILED TO START — Failed to configure a DataSource'.

Move full datasource (and kafka/redis for projection-writer) wiring into the
k8s profile section. ENV vars come from Vault Agent injector
(ORDINIS_PG_USER/PASSWORD, ORDINIS_KAFKA_USER/PASSWORD, ORDINIS_REDIS_PASSWORD)
and chart-supplied ENV (ORDINIS_PG_HOST, ORDINIS_KAFKA_BOOTSTRAP, etc).
2026-05-03 20:24:53 +03:00
Александр Зимин 413bd3449d chore: update default config-server fallback to ordinis-staging
application.yml fallback URL was http://config-server.ordinis-dev:8888 — namespace
removed. Default now points to staging; SPRING_CLOUD_CONFIG_URI env still overrides
via Helm chart for any deployment.
2026-05-03 20:11:51 +03:00
Zimin A.N. be77c30687 feat(validation,rest-api,outbox): write-side end-to-end working
ordinis-validation:
- ValidationError, ValidationResult, ValidationException
- RecordValidator: networknt JSON Schema (Draft-7) + custom rule для
  x-localized полей. Walk schema, для каждого x-localized:true property
  проверяет: значение это object с locale keys (BCP 47 pattern xx-XX),
  все ключи в supported_locales, default_locale обязательно присутствует,
  значения non-empty strings.

ordinis-outbox:
- KafkaTopicResolver: ordinis.<bundle>.events.<scope>
- OutboxRecorder: запись события в outbox в той же транзакции (MANDATORY
  propagation), с tracer для trace_id/span_id из MDC если OTel доступен
- OutboxPoller: @Scheduled, batch_size + poll_interval из config, метрики
  ordinis_outbox_published_total / publish_errors_total / kafka_publish_duration_seconds
- OutboxLagGauge: ordinis_outbox_pending_events (gauge для алертов)
- ConditionalOnProperty ordinis.outbox.enabled — отключаем в read-api/projection-writer

ordinis-rest-api:
- DTOs: CreateDictionaryRequest, DictionaryResponse, CreateRecordRequest,
  RecordResponse (с WKT serialization geometry)
- DictionaryDefinitionService: CRUD definitions, outbox events DefinitionCreated/Updated
- DictionaryRecordService: bitemporal write-side, SERIALIZABLE isolation,
  bound check FAR_FUTURE valid_to. Update = close current + create new
  (никогда in-place). Outbox event RecordCreated/Updated/Deleted.
- DictionaryDefinitionController: GET list/by-name, POST create, PUT update
- DictionaryRecordController: GET active/history, POST create, PUT update,
  DELETE (close)
- OrdinisException + ApiErrorResponse + GlobalExceptionHandler
  (422 validation, 404 not found, 409 conflict, 500 internal)

ordinis-app:
- @EnableJpaRepositories + @EntityScan для multi-package сканирования
- application.yml profile-based: dev (PG localhost через port-forward,
  ddl-auto=update, OTel disabled), k8s (Cloud Config + Vault Kubernetes auth)
- spring.kafka producer config с SASL SCRAM-SHA-512

ordinis-domain:
- JpaAuditingConfig: добавлен DateTimeProvider возвращающий OffsetDateTime
  (Spring Data Auditing по дефолту его не поддерживает, без него падает
  IllegalArgumentException 'Cannot convert LocalDateTime to OffsetDateTime')
- @ConditionalOnBean(DataSource) убран — race с lifecycle ломал auditing

E2E test (port-forward к cluster PG+Kafka):
- POST dictionary 'ground_station' → 201, JSONB schema сохранена,
  supported_locales = {ru-RU, en-US}, default_locale = ru-RU
- POST record MOSCOW-1 multi-locale name + geometry POINT(37.618 55.751) → 201
- POST record с en-US без ru-RU → 422 с двумя ошибками:
  * networknt 'required property ru-RU not found'
  * custom 'x-localized.missing_default Default locale ru-RU is required'
- outbox_events: 2 row (DefinitionCreated + RecordCreated), topic
  ordinis.cuod.events.public, ключи правильные
- revinfo + dictionary_records_aud: 1 row (Envers tx-time history активен)
2026-05-03 13:04:36 +03:00
Zimin A.N. b778832f49 feat(domain,events): bitemporal entities + i18n-aware definitions + sealed event API
ordinis-domain:
- DataScope enum (PUBLIC/INTERNAL/RESTRICTED) с Kafka topic suffix + visibility logic
- DictionaryDefinition entity: scope, schema_json (JsonNode JSONB), schema_version,
  bundle, supported_locales (text[]), default_locale + Spring Data Auditing
- DictionaryRecord entity: bitemporal (Envers tx-time + valid_from/valid_to),
  data JSONB, geometry (JTS via hibernate-spatial), data_scope. EXCLUSION
  CONSTRAINT enforced в БД (Liquibase 0003).
- AuditLog entity для compliance (отдельно от Loki)
- OutboxEvent entity: aggregate_*, kafka_topic/kafka_key, attempt_count,
  markPublished/markFailed, partial index idx_outbox_unpublished
- IdempotencyKey entity для exactly-once семантики write API
- Repositories: findActiveAt, findActiveByDictionaryAndScopes, findUnpublished
- JpaAuditingConfig: ConditionalOnBean(DataSource) — dev profile без БД
  не активирует auditing, metamodel не строится

ordinis-events-api:
- Mirror DataScope enum (events не depend on domain)
- EventEnvelope<T> record: eventId, occurredAt, schemaVersion (semver),
  bundle, dictionaryName, scope, traceId, spanId, payload
- Sealed DictionaryRecordEvent + records Created/Updated/Deleted
- Sealed DictionaryDefinitionEvent + records Created/Updated
- Jackson polymorphism через @JsonTypeInfo discriminator
- Multi-locale payload: события возят ВСЕ локали (consumer сам сплющивает)

i18n design:
- В JSONB поле кодируется как { 'ru-RU': '...', 'en-US': '...' }
- JSON Schema справочника помечает локализуемые поля extension x-localized:true
- supported_locales + default_locale в dictionary_definitions
- Read API будет negotiate через Accept-Language (RFC 7231) и сплющивать

Validation:
- mvn install all 11 modules: BUILD SUCCESS
- ordinis-app boot dev profile: /actuator/health UP, / отвечает
2026-05-03 12:10:36 +03:00
Zimin A.N. d1c8d5755e fix(scaffolding): make ordinis-app boot in dev profile
- Remove spring-cloud-starter-bootstrap (deprecated в SC 2024.x; используем
  spring.config.import вместо bootstrap context)
- Profile-based config: dev отключает Cloud Config + Vault + OTel + JPA
  auto-config для standalone smoke testing; k8s profile подключает реальную
  инфру (vault + config server)
- Dockerfile: COPY всех модулей parent pom их видит при reactor
- Уберём spring-boot-maven-plugin из read-api/projection-writer пока нет main

Validated:
- mvn install all 11 modules: BUILD SUCCESS
- Spring Boot стартует за 2.7s в dev profile
- /actuator/health UP, /actuator/prometheus отдаёт http_server_requests + jvm
- HelloController отвечает на GET /
2026-05-03 01:03:03 +03:00
Zimin A.N. a28fd0b352 feat: Maven multimodule scaffolding for Ordinis MDM service
- 10 модулей (domain, validation, events-api, outbox, rest-api, app, read-api, projection-writer, cuod-bundle, migrations)
- Parent pom + BOM с Spring Boot 3.4.2, Spring Cloud 2024.0.0, Vault Config 4.2.0
- ordinis-app: Spring Boot main с Actuator/Prometheus/Logstash JSON encoder/OpenTelemetry/Cloud Config/Vault skeleton
- ordinis-migrations: Liquibase changelogs (dictionary_definitions/records, audit_log, outbox_events, idempotency_keys, EXCLUSION CONSTRAINT через btree_gist)
- Multi-stage Dockerfiles для app и migrations
2026-05-03 00:41:53 +03:00