feat(user-display): KeycloakAdminUserResolver — Phase 2 on-demand lookup

Implements the Phase 2 hook from the previous commit. With this in
place, UserDisplayService.find chains:

  hot cache → DB (user_display_cache) → Keycloak Admin API

so any realm user resolves on first display, even if they never made
an authenticated request to ordinis. The resolved entry is persisted
via UserDisplayService.upsert with source=KEYCLOAK_ON_DEMAND, so the
next render of the same sub hits the DB tier (~ms) instead of going
back to Keycloak.

Implementation notes:

- Spring RestClient (modern blocking HTTP, replaces RestTemplate).
- client_credentials grant against
  {issuer}/realms/{realm}/protocol/openid-connect/token. Token cached
  in-memory minus a 30s safety margin from `expires_in`.
- 401 on user lookup wipes the cached token and surfaces a
  RuntimeException, so the next resolve refetches.
- 404 on user lookup → Optional.empty (definitive miss, sub not in
  realm). UserDisplayService logs and the caller falls back to short
  UUID. Negative cache TTL not implemented — added complexity not
  justified for the current realm size (~50 users).
- Bean activated only when ordinis.keycloak.admin.enabled=true AND
  the env vars are set. Default disabled — UserDisplayService treats
  KeycloakUserResolver as @Autowired(required = false), so a
  deployment without Keycloak admin creds keeps the previous behavior
  (DB cache only) instead of crashing on startup.

Application config:
- ORDINIS_KEYCLOAK_ADMIN_ENABLED      (false by default)
- ORDINIS_KEYCLOAK_ADMIN_URL          e.g. https://auth.nstart.space
- ORDINIS_KEYCLOAK_ADMIN_REALM        defaults to nstart
- ORDINIS_KEYCLOAK_ADMIN_CLIENT_ID    via vault (see docs-internal/keycloak-admin-setup.md)
- ORDINIS_KEYCLOAK_ADMIN_CLIENT_SECRET via vault

Setup runbook (docs-internal/keycloak-admin-setup.md) covers:
- creating the Keycloak service-account client + assigning view-users
- writing the secret into both vault instances (different per env)
- wiring vault.hashicorp.com agent-inject annotations into the chart
- verification steps + common failure modes

Phase 3 (scheduled bulk sync) is documented at the end of the runbook
but deferred — depends on observed cache miss rate after this lands.

docs-internal/keycloak-admin-setup.md

@@ -0,0 +1,136 @@
+# Keycloak Admin client setup for UserDisplayService Phase 2
+
+`KeycloakAdminUserResolver` resolves a JWT `sub` against the Keycloak Admin
+REST API on cache miss, so users that never hit ordinis still get a
+display name in audit / reviews / changelog / events / history. Without
+this, only users who personally made an authenticated request show up
+with a real name (the "ток мой UUID отображается" pain).
+
+This is a per-environment one-time setup. Disabled by default — flip on
+once the Keycloak client + vault secret exist.
+
+## Step 1: Create service-account client in Keycloak
+
+Realm: `nstart`. Path: Clients → Create client.
+
+| Field | Value |
+|---|---|
+| Client type | OpenID Connect |
+| Client ID | `ordinis-admin-client` |
+| Client authentication | **On** (confidential) |
+| Authorization | Off |
+| Standard flow | Off |
+| Direct access grants | Off |
+| Service accounts roles | **On** |
+
+Save. On the created client → **Service accounts roles** tab → **Assign
+role** → Filter by clients → `realm-management` → check **`view-users`**
+→ Assign.
+
+(`view-users` is the minimum scope needed for `GET /admin/realms/{realm}/users/{id}`.
+Don't assign `manage-users` — read-only is enough.)
+
+Then **Credentials** tab → copy the `Client secret` (long random string).
+
+## Step 2: Store secret in Vault
+
+The cluster-domain matters here — staging is `265.local`, prod is
+`altum.local`. Both have separate Vault instances.
+
+Staging (cluster.265, namespace `config`):
+```bash
+kubectl -n config exec -it vault-0 -- sh -c '
+  vault login <root-or-admin-token>
+  vault kv put secret/ordinis/cuod/keycloak-admin \
+    client_id=ordinis-admin-client \
+    client_secret=<paste-from-keycloak>
+'
+```
+
+Prod (altum, namespace `vault`):
+```bash
+ssh root@192.168.100.142 'kubectl -n vault exec -it vault-0 -- sh -c "
+  vault login <prod-root-token>
+  vault kv put secret/ordinis/cuod/keycloak-admin \
+    client_id=ordinis-admin-client \
+    client_secret=<paste-from-keycloak>
+"'
+```
+
+⚠️ Different secret per environment — staging and prod have separate
+Keycloak clients (and certainly separate secrets).
+
+## Step 3: Wire into ordinis-app deployment
+
+Add to `ordinis-app` helm chart annotations (vault agent injector
+template) — same pattern as `secret-postgres` / `secret-kafka` already
+present in the chart:
+
+```yaml
+vault.hashicorp.com/agent-inject-secret-keycloak-admin: secret/data/ordinis/cuod/keycloak-admin
+vault.hashicorp.com/agent-inject-template-keycloak-admin: |
+  {{- with secret "secret/data/ordinis/cuod/keycloak-admin" -}}
+  export ORDINIS_KEYCLOAK_ADMIN_CLIENT_ID="{{ .Data.data.client_id }}"
+  export ORDINIS_KEYCLOAK_ADMIN_CLIENT_SECRET="{{ .Data.data.client_secret }}"
+  {{- end }}
+```
+
+Add env vars to the deployment:
+
+```yaml
+env:
+  - name: ORDINIS_KEYCLOAK_ADMIN_ENABLED
+    value: "true"
+  - name: ORDINIS_KEYCLOAK_ADMIN_URL
+    value: "https://auth.nstart.space"   # both envs use the same Keycloak
+  - name: ORDINIS_KEYCLOAK_ADMIN_REALM
+    value: "nstart"
+  # CLIENT_ID + CLIENT_SECRET arrive via the vault-injected
+  # /vault/secrets/keycloak-admin file sourced before java starts.
+```
+
+## Step 4: Verify
+
+After redeploy, check the pod log on startup:
+
+```
+KeycloakAdminUserResolver active — base=https://auth.nstart.space realm=nstart client=ordinis-admin-client
+```
+
+End-to-end:
+
+```bash
+# Find a sub that's NOT in the JVM cache (e.g. a user who never logged in).
+# Hit the display endpoint as an admin user:
+curl -s -H "Authorization: Bearer $TOKEN" \
+  https://ordinis.k8s.265.nstart.cloud/api/v1/admin/users/<uuid>/display | jq
+# Expect 200 with username/name/email even on cold cache, instead of 404.
+```
+
+Then check the DB — the row was persisted from the on-demand resolve:
+
+```sql
+SELECT sub, preferred_username, source, synced_at
+  FROM user_display_cache
+ WHERE sub = '<uuid>';
+-- source = KEYCLOAK_ON_DEMAND
+```
+
+## Step 5: Phase 3 (later — scheduled bulk sync)
+
+When ready to refill the table proactively (avoids the per-user
+on-demand latency for fresh subs), add a Spring `@Scheduled` task that
+calls `GET /admin/realms/{realm}/users` paginated and writes via
+`UserDisplayService.upsertFromKeycloak`. Recommended every 6h. Not
+included in this MR — depends on observed cache miss rate.
+
+## Failure modes
+
+- **Service-account client missing `view-users`** → 403 on lookup,
+  resolver throws RuntimeException, UserDisplayService logs warn,
+  caller falls back to short UUID. No outage, just no resolve.
+- **Keycloak unreachable** → connect timeout / 502. Same fallback as
+  above. Cached entries (DB tier) keep working.
+- **client_secret rotated in Keycloak but vault not updated** → 401 on
+  token request → resolver wipes cached token and surfaces
+  RuntimeException. Update vault secret and the next attempt succeeds.

ordinis-app/src/main/resources/application.yml

@@ -211,3 +211,16 @@ ordinis:
     issuer: ${jwt-issuer:}
     require-authentication: ${ORDINIS_AUTH_REQUIRED:false}
     allow-query-scope: ${ORDINIS_AUTH_ALLOW_QUERY_SCOPE:true}
+
+  # Keycloak Admin REST integration for UserDisplayService Phase 2 — on-demand
+  # sub→user lookup when JWT capture cache + DB cache both miss. See
+  # KeycloakAdminUserResolver.java + 0023-user-display-cache.xml migration.
+  # Disabled by default; flip on per-environment when the service-account
+  # client + vault secret are provisioned.
+  keycloak:
+    admin:
+      enabled: ${ORDINIS_KEYCLOAK_ADMIN_ENABLED:false}
+      url: ${ORDINIS_KEYCLOAK_ADMIN_URL:}
+      realm: ${ORDINIS_KEYCLOAK_ADMIN_REALM:nstart}
+      client-id: ${ORDINIS_KEYCLOAK_ADMIN_CLIENT_ID:}
+      client-secret: ${ORDINIS_KEYCLOAK_ADMIN_CLIENT_SECRET:}

ordinis-rest-api/src/main/java/cloud/nstart/terravault/ordinis/restapi/service/KeycloakAdminUserResolver.java

@@ -0,0 +1,171 @@
+package cloud.nstart.terravault.ordinis.restapi.service;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.http.HttpHeaders;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.MediaType;
+import org.springframework.stereotype.Service;
+import org.springframework.util.LinkedMultiValueMap;
+import org.springframework.util.MultiValueMap;
+import org.springframework.web.client.HttpClientErrorException;
+import org.springframework.web.client.RestClient;
+import org.springframework.web.util.UriComponentsBuilder;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Optional;
+
+/**
+ * Phase 2: resolve {@code sub} via Keycloak Admin REST API. Activated when
+ * {@code ordinis.keycloak.admin.enabled=true} (and required env vars set).
+ *
+ * <p>Auth flow: client_credentials grant against
+ * {@code {issuer}/protocol/openid-connect/token}. The service-account
+ * client must have realm role {@code view-users} (composite of
+ * {@code realm-management/view-users}).
+ *
+ * <p>Token caching: the access_token from Keycloak typically lives
+ * 60-300s. We cache it minus 30s safety margin and refresh on miss.
+ *
+ * <p>Lookup uses {@code GET /admin/realms/{realm}/users/{sub}} which
+ * returns 404 cleanly when the sub is unknown. Network/5xx are surfaced
+ * as RuntimeException so {@link UserDisplayService} can log and skip.
+ */
+@Service
+@ConditionalOnProperty(name = "ordinis.keycloak.admin.enabled", havingValue = "true")
+public class KeycloakAdminUserResolver implements KeycloakUserResolver {
+
+  private static final Logger log = LoggerFactory.getLogger(KeycloakAdminUserResolver.class);
+  private static final Duration TOKEN_REFRESH_MARGIN = Duration.ofSeconds(30);
+
+  private final String adminBaseUrl;
+  private final String realm;
+  private final String clientId;
+  private final String clientSecret;
+  private final RestClient http;
+
+  // Token cache — Keycloak access_token + expiry. Access is single-threaded
+  // in practice (one resolver call at a time); volatile is enough.
+  private volatile String cachedToken;
+  private volatile Instant cachedTokenExpiresAt = Instant.EPOCH;
+
+  public KeycloakAdminUserResolver(
+      @Value("${ordinis.keycloak.admin.url}") String adminBaseUrl,
+      @Value("${ordinis.keycloak.admin.realm}") String realm,
+      @Value("${ordinis.keycloak.admin.client-id}") String clientId,
+      @Value("${ordinis.keycloak.admin.client-secret}") String clientSecret) {
+    this.adminBaseUrl = trimTrailingSlash(adminBaseUrl);
+    this.realm = realm;
+    this.clientId = clientId;
+    this.clientSecret = clientSecret;
+    this.http = RestClient.builder()
+        .baseUrl(this.adminBaseUrl)
+        .build();
+    log.info("KeycloakAdminUserResolver active — base={} realm={} client={}",
+        this.adminBaseUrl, realm, clientId);
+  }
+
+  @Override
+  public Optional<KeycloakUser> findById(String sub) {
+    if (sub == null || sub.isBlank()) return Optional.empty();
+    String token = currentToken();
+    String url = UriComponentsBuilder.fromHttpUrl(adminBaseUrl)
+        .pathSegment("admin", "realms", realm, "users", sub)
+        .toUriString();
+    try {
+      KeycloakUserDto dto = http.get()
+          .uri(url)
+          .header(HttpHeaders.AUTHORIZATION, "Bearer " + token)
+          .retrieve()
+          .body(KeycloakUserDto.class);
+      if (dto == null || dto.id == null) return Optional.empty();
+      String name = composeName(dto.firstName, dto.lastName);
+      return Optional.of(new KeycloakUser(
+          dto.id,
+          dto.username,
+          name,
+          dto.email));
+    } catch (HttpClientErrorException.NotFound e) {
+      // Sub doesn't exist in this realm — definitive miss. Don't cache the
+      // negative; UserDisplayService.find returns Optional.empty and UserCell
+      // falls back to short UUID. Negative cache TTL would help latency on
+      // bots/scanners but adds complexity not justified yet.
+      return Optional.empty();
+    } catch (HttpClientErrorException.Unauthorized e) {
+      // Cached token rejected — wipe and retry once. If still 401, surface.
+      cachedToken = null;
+      cachedTokenExpiresAt = Instant.EPOCH;
+      throw new RuntimeException("Keycloak admin auth rejected (token wiped, will retry)", e);
+    }
+  }
+
+  // --- token management ---
+
+  private synchronized String currentToken() {
+    if (cachedToken != null && Instant.now().isBefore(cachedTokenExpiresAt)) {
+      return cachedToken;
+    }
+    return fetchAndCacheToken();
+  }
+
+  private String fetchAndCacheToken() {
+    String tokenUrl = UriComponentsBuilder.fromHttpUrl(adminBaseUrl)
+        .pathSegment("realms", realm, "protocol", "openid-connect", "token")
+        .toUriString();
+    MultiValueMap<String, String> form = new LinkedMultiValueMap<>();
+    form.add("grant_type", "client_credentials");
+    form.add("client_id", clientId);
+    form.add("client_secret", clientSecret);
+    TokenResponse tr = http.post()
+        .uri(tokenUrl)
+        .contentType(MediaType.APPLICATION_FORM_URLENCODED)
+        .body(form)
+        .retrieve()
+        .onStatus(s -> s.is4xxClientError() || s.is5xxServerError(), (req, res) -> {
+          throw new RuntimeException("Keycloak admin token request failed: HTTP "
+              + res.getStatusCode() + " — check client_id/secret + realm");
+        })
+        .body(TokenResponse.class);
+    if (tr == null || tr.accessToken == null) {
+      throw new RuntimeException("Keycloak admin token response missing access_token");
+    }
+    cachedToken = tr.accessToken;
+    int expiresIn = tr.expiresIn != null ? tr.expiresIn : 60;
+    cachedTokenExpiresAt = Instant.now()
+        .plusSeconds(expiresIn)
+        .minus(TOKEN_REFRESH_MARGIN);
+    return cachedToken;
+  }
+
+  // --- helpers ---
+
+  private static String trimTrailingSlash(String s) {
+    return (s != null && s.endsWith("/")) ? s.substring(0, s.length() - 1) : s;
+  }
+
+  private static String composeName(String first, String last) {
+    String f = (first == null || first.isBlank()) ? "" : first.trim();
+    String l = (last == null || last.isBlank()) ? "" : last.trim();
+    String joined = (f + " " + l).trim();
+    return joined.isEmpty() ? null : joined;
+  }
+
+  // --- DTOs (Keycloak 26 admin API) ---
+
+  @JsonIgnoreProperties(ignoreUnknown = true)
+  private record TokenResponse(
+      @com.fasterxml.jackson.annotation.JsonProperty("access_token") String accessToken,
+      @com.fasterxml.jackson.annotation.JsonProperty("expires_in") Integer expiresIn) {}
+
+  @JsonIgnoreProperties(ignoreUnknown = true)
+  private record KeycloakUserDto(
+      String id,
+      String username,
+      String firstName,
+      String lastName,
+      String email) {}
+}