Keycloak Runbook

Keycloak Runbook

Keycloak is complex. This guide helps troubleshoot common issues.

Authentication & JWT

Keycloak has no dedicated host port or subdomain of its own, unlike most other services — it is reached exclusively through Traefik’s PathPrefix(/kc) rule on the shared web entrypoint (port 80, or TRAEFIK_WEB_HOST_PORT if that’s overridden in .env). Don’t invent a :8180 or similar port for it.

”invalid algorithm” in middleware

Cause: Strapi admin sends HS256 JWT, but middleware expects RS256 (from Keycloak).

Solution: KC middleware decodes header and skips non-RS256 tokens:

sencai.space/src/middlewares/keycloak-jwt.ts

const decoded = jwt.decode(token);

if (decoded?.header?.alg !== 'RS256') {
  return next(); // Skip JWKS validation
}

“Token expired”

Cause: KC JWT has 12-hour TTL. Refresh token expired.

Solution: Frontend calls /auth/refresh endpoint:

async function refreshTokenIfNeeded() {
  const decoded = jwtDecode(token.value);
  const expiresIn = decoded.exp * 1000 - Date.now();

  if (expiresIn < 5 * 60 * 1000) { // Less than 5 minutes
    await $fetch('/api/auth/refresh', { method: 'POST' });
  }
}

“audience mismatch” — 100% login failure after adding audience to jwt.verify()

Cause: Passing audience: 'sencai-frontend' into jsonwebtoken’s jwt.verify() options seems like the obvious way to pin the middleware to the frontend client, but it broke every single login. jsonwebtoken’s built-in audience check only compares against the aud claim — and Keycloak’s default aud on sencai-frontend tokens is the built-in account client, not sencai-frontend. The client that actually requested the token is recorded separately, in the azp claim.

Solution: Don’t use jsonwebtoken’s built-in audience option. Decode first, then verify the client manually against both azp and aud:

// sencai.space/src/middlewares/keycloak-jwt.ts (~lines 206-226)
// jsonwebtoken's built-in `audience` option only checks the `aud` claim, but
// the real requesting client lives in `azp`. Enforcing `audience: 'sencai-frontend'`
// here rejected every valid token (100% login failure). The azp/aud check is
// done manually after jwt.verify() instead.
const azpMatches = decoded.azp === expectedClientId;
const audMatches = Array.isArray(decoded.aud)
  ? decoded.aud.includes(expectedClientId)
  : decoded.aud === expectedClientId;

if (!azpMatches && !audMatches) {
  logger.error('❌ Keycloak JWT audience/azp mismatch:', { azp: decoded.azp, aud: decoded.aud });
  reject(new Error(`JWT audience/azp mismatch: expected client "${expectedClientId}"`));
}

This fix still carries the inline comment in keycloak-jwt.ts explaining why — don’t remove the manual azp/aud check or re-add audience to the jwt.verify() options.

User synchronization

”Loop in user-kc-sync”

Cause: User update in Strapi → webhook → auth-service-consumer → KC sync → Strapi update → infinite loop.

Solution: Use runWithoutKcSync() AsyncLocalStorage guard:

src/lifecycles/user-kc-sync.ts

export function runWithoutKcSync<T>(fn: () => Promise<T>): Promise<T> {
  return asyncLocalStorage.run({ skipKcSync: true }, fn);
}

“Username is email instead of KC UUID”

Cause: User created before first KC login has username=email.

Solution: KC JWT middleware automatically updates username to KC sub:

if (decoded.preferred_username !== strapiUser.username) {
  await strapi.entityService.update(
    'plugin::users-permissions.user',
    strapiUser.id,
    { data: { username: decoded.sub } }
  );
}

Keycloak setup

”KC returns 503”

Cause: Keycloak is slow to start (normal).

Solution: Wait 30–90 seconds. Health check:

curl -f http://localhost/kc/health || exit 1

“KC /kc endpoint 404”

Cause: KC_HTTP_RELATIVE_PATH not set to /kc.

Solution: In docker-compose.auth.yml:

environment:
  KC_HTTP_RELATIVE_PATH: "/kc"
  KEYCLOAK_URL: "http://keycloak:8080/kc"

“Can’t get admin token”

Cause: Wrong grant type or realm.

Solution: Use password grant on master realm with admin-cli:

POST /realms/master/protocol/openid-connect/token
client_id=admin-cli&
username=admin&
password=<password>&
grant_type=password

User management

”User can’t verify email”

Cause: VERIFY_EMAIL required action not active.

Solution: In Strapi, when user confirms email:

if (event.params.data.emailVerified) {
  await removeKcRequiredAction(user.id, 'VERIFY_EMAIL');
}

“Can’t complete profile (UPDATE_PROFILE loop)”

Cause: User has UPDATE_PROFILE required action.

Solution: When creating KC user, set firstName/lastName (fallback to '-'):

const kcUser = {
  username: registration.email,
  firstName: registration.name || '-',
  lastName: registration.surname || '-',
  requiredActions: ['VERIFY_EMAIL'], // Not UPDATE_PROFILE
};

Realm configuration

”Wrong realm”

Examples:

# WRONG — uses master realm
POST http://keycloak:8080/realms/master/...

# RIGHT — for user token (sencai realm)
POST http://keycloak:8080/realms/sencai/...

# RIGHT — for admin API (master realm)
POST http://keycloak:8080/realms/master/...

Client configuration

Client	Realm	Use case
sencai-frontend	sencai	Browser login (public client)
auth-service-consumer	sencai	Backend KC-admin-API calls from the sync consumer (confidential, client-secret)
admin-cli	master	Admin API (password grant, master realm)

There is no sencai-backend client in the realm import — don’t reference it. The realm import lives at orchestration/config/keycloak/data/import/sencai-realm.json.

”Client not found”

Solution: Verify in Keycloak Admin Console:

1. Go to http://localhost/kc/admin
2. Select realm sencai
3. Check Clients → sencai-frontend and auth-service-consumer exist

SMTP & email

”Email not sending”

Solution: Configure SMTP in Keycloak:

1. Realm sencai
2. Realm settings → Email
3. Fill in SMTP: host, port, username, password
4. Click Save

Frontend integration

”Can’t exchange auth code”

Cause: Redirect URI not configured in KC client.

Solution: In Keycloak Admin, the real sencai-frontend client (from orchestration/config/keycloak/data/import/sencai-realm.json) is configured as:

1. Open sencai-frontend client

2. Valid redirect URIs:

   • http://localhost/* (Traefik-routed — this is the primary local URL)
   • http://localhost:3000/* (direct Nuxt dev server, bypassing Traefik)
   • https://app.sencai.space/* (prod)

3. Web Origins: set to the wildcard +, which trusts any origin implied by the configured redirect URIs — it is not a hand-maintained list of individual origins.

”CORS error on KC token endpoint”

Solution: First check whether Web Origins still holds the wildcard +. If someone replaced + with an explicit list of origins, that list is the likely cause — check it for a missing origin before adding new ones. Only fall back to hand-listing origins if there’s a specific reason + can’t be used.

Debugging

View KC logs

docker logs -f sencai-keycloak-1 | grep -i error

Verify JWT token

# Decode token (no verification)
echo "token123" | jq -R 'split(".") | .[1] | @base64d | fromjson'

KC health

curl http://localhost/kc/health/ready
# 200 = ready
# 503 = starting up

Sync consumer issues

These are auth-service-consumer-specific failures (see auth-service-consumer/CLAUDE.md), distinct from the Strapi-side middleware issues above.

KC 401 on the admin token endpoint

Cause: KEYCLOAK_ADMIN_PASSWORD set for auth-service-consumer doesn’t match the actual Keycloak admin password.

Solution: Make sure KEYCLOAK_ADMIN_PASSWORD is identical between the keycloak service and auth-service-consumer in the compose env (docker-compose.auth.yml / .env).

”User not found for update”

Cause: A Keycloak event arrived at the consumer before the corresponding Strapi user existed yet (race between registration and KC event processing).

Solution: The consumer retries and also falls back to looking the user up by email, not just by KC sub, before giving up.

Persistent failures after retries

Cause: Processing keeps failing after MAX_RETRY_ATTEMPTS (default 5, RETRY_DELAY_MS default 12000 ms).

Solution: The message lands in the user.fallback RabbitMQ queue (durable: true, 14-day TTL). Inspect it there and check consumer logs for the underlying error.

KC → Strapi sync is pull-based only

There is currently no instant push from Keycloak to Strapi. If a KC user is edited directly in the Keycloak Admin Console, Strapi does not find out immediately — sync only happens:

• On login, via the keycloak-jwt middleware (pulls KC user data and reconciles it into Strapi), or
• On demand, via POST /api/auth/sync-from-kc (authenticated endpoint, same logic invoked manually from the frontend).

A Keycloak Event Listener SPI plugin (Java) for instant KC→Strapi push is planned but not implemented — it’s tracked as deferred work in auth-service-consumer/CLAUDE.md. Don’t assume edits made directly in the Admin Console propagate to Strapi until the next login or an explicit sync call.

Checklist

1. KC running? → curl http://localhost/kc/health
2. JDBC connection? → Check docker logs sencai-keycloak-db-1
3. Correct realm? → Verify in Admin Console
4. Client exists? → Check Clients (sencai-frontend, auth-service-consumer)
5. JWT format? → Decode and verify iss, sub, exp, azp
6. Email SMTP? → Check Realm settings → Email
7. Sync loop? → Verify runWithoutKcSync() in lifecycle
8. Sync consumer healthy? → Check auth-service-consumer logs and the user.fallback queue

See Root CLAUDE.md for full Keycloak architecture and auth-service-consumer/ for sync implementation.