Local Observability Stack Runbook

Local Observability Stack Runbook

The core monitoring stack (Prometheus/Grafana/Loki/Tempo/Alertmanager/Uptime Kuma), Vault, GlitchTip, and PostHog all run as part of the default run-local-dev.sh profile set (monitoring, vault, glitchtip, posthog profiles). The monitoring stack itself is auto-provisioned (dashboards, data sources, scrape configs are all pre-baked) and needs no manual setup beyond the one-time volume permissions below — Uptime Kuma, Vault, GlitchTip, and PostHog need a one-time first-run step because they manage their own users/projects/keys. Actual secret values (tokens, DSNs, passwords) live only in orchestration/.env (gitignored) — never in this doc or in git.

The root CLAUDE.md “Default profily” table is stale — it lists a shorter profile set than what actually ships. The authoritative current list is the DEFAULT_PROFILES variable in orchestration/run-local.sh (and orchestration/run-local-dev.sh, which mirrors it), which today also includes audit, agent, vault, billing, watchdog, glitchtip, posthog, opsloop, llm, backup, cell, and bootstrap in addition to the profiles the CLAUDE.md table names. Check that file directly rather than trusting the table when you need to know what starts by default.

Core monitoring stack

The monitoring profile brings up Prometheus, Grafana, Loki, Tempo, Alertmanager, and Uptime Kuma (plus exporters/OTEL collector). All of these are pre-provisioned — no manual dashboard/data-source setup needed — except Uptime Kuma, which needs a first-run admin account.

Service	URL	Notes
Grafana	http://grafana.sencai.localhost	Default login admin/admin, overridable via GRAFANA_ADMIN_USER/GRAFANA_ADMIN_PASSWORD in orchestration/.env
Prometheus	http://prometheus.sencai.localhost	Metrics + query UI
Alertmanager	http://alertmanager.sencai.localhost	Alert routing UI
Uptime Kuma (status page)	http://status.sencai.localhost	First-run: first visit prompts you to create the admin account — same first-run-setup pattern as GlitchTip/PostHog below

Loki quirk: grafana/loki:3.6.x ships as a distroless image with no shell, wget, or curl, so the usual HTTP-based Docker healthcheck can’t run against it; the compose config instead runs loki -version as a process-liveness check. Separately, Loki’s /ready endpoint returns 503 permanently in this single-node, in-memory-ring setup — that’s expected behavior for this deployment shape, not a sign that Loki is broken; Loki still ingests and serves queries normally. See orchestration/CLAUDE.md for the full gotcha writeup.

One-time volume ownership fix: Docker creates bind-mount directories as root, but these containers run as non-root users. After the first start of the monitoring profile, run:

sudo chown -R 65534:65534 ~/sencai_volumes/prometheus ~/sencai_volumes/alertmanager
sudo chown -R 10001:10001 ~/sencai_volumes/loki ~/sencai_volumes/tempo
sudo chown -R 472:472 ~/sencai_volumes/grafana

(Adjust the base path if you’ve overridden LOCAL_VOLUME_PATH/VOLUME_PATH in .env.) See orchestration/CLAUDE.md for the same note in context with the other volume-ownership fixes (e.g. Forum’s pict-rs).

Vault

Local dev defaults to Vault running sealed, with file-backed storage — VAULT_DEV_MODE=false is the default in orchestration/.env.example. On a fresh volume the container starts sealed with no known root token; you must run the standard init/unseal flow once:

docker exec -it sencai-vault vault operator init
# save the printed unseal keys + initial root token OUTSIDE the repo
docker exec -it sencai-vault vault operator unseal   # repeat with 3 of the 5 keys

See orchestration/config/vault/vault-config.hcl for the storage/listener config and orchestration/scripts/vault-init.sh for the follow-up KV/AppRole bootstrap (it expects VAULT_TOKEN set to the root token from the init step above).

Dev mode is opt-in only, for a throwaway local machine: set VAULT_DEV_MODE=true and a VAULT_DEV_ROOT_TOKEN in orchestration/.env to get in-memory storage, auto-unseal on every start, and a fixed root token — nothing survives a container recreate in this mode. Do not use dev mode as a template for staging/production, and do not assume it’s what a fresh local checkout gives you by default; the sealed/file-storage path is what actually ships unless you’ve explicitly flipped the flag.

If Vault ever shows unhealthy despite the container running, check docker logs sencai-vault first — if you’re in the default sealed mode, this is often just the unseal step not having been run yet.

GlitchTip (self-hosted Sentry-compatible error tracking)

1. First visit to http://glitchtip.sencai.localhost/ creates the first user as an org owner. The initial “select platform” step is cosmetic — it only affects the onboarding code snippet shown, not functionality — pick anything (e.g. node) if unsure.
2. GlitchTip’s REST API (Sentry-API-compatible) requires a personal API token, created under Settings → Auth Tokens — the org-creation wizard doesn’t produce one automatically.
3. Every service configured with SENTRY_DSN (see docker-compose.core.yml, docker-compose.cloud.yml, etc.) needs a GlitchTip project + key. Today the whole platform shares one project (all backend services publish to the same DSN) since that’s how SENTRY_DSN is already wired as a single shared env var across ~11 services — splitting into per-service projects would require introducing per-service env var names in each compose file first. Not every microservice has SENTRY_DSN wired yet — currently it’s set on api-manager, backend, frontend, webhook-publisher, auth-service-consumer, billing-adapter, git-connector, notification-service, cloud-connector/cloud-connector-worker, and opsloop-consumer. Wiring it into a new service means adding SENTRY_DSN=${SENTRY_DSN:-} to that service’s compose env block and calling Sentry.init() conditionally on it being set (see F3.SECURITY.02 in PHASE-3-PLAN.md).
4. Server-side vs. client-side DSN gotcha: *.sencai.localhost hostnames resolve to loopback inside containers (RFC 6761 — same class of issue as the earlier HSTS/Chromium investigation), not to Traefik. So:
   • SENTRY_DSN (consumed by backend services running inside Docker) must point at the internal service name: http://<key>@glitchtip-web:8000/<project-id>.
   • NUXT_PUBLIC_SENTRY_DSN (bundled into the browser) must use the public Traefik-routed hostname: http://<key>@glitchtip.sencai.localhost/<project-id>.
   • Getting these backwards means either “no errors ever show up” (server-side pointed at the browser hostname, connection refused to container’s own loopback) or a DSN that only works from a browser tab (client-side pointed at the internal name, which a browser can’t resolve at all).
5. After wiring the DSN(s) into orchestration/.env, recreate (not just restart) the affected services — env vars are baked in at container creation:

   ./run-local-dev.sh up -d api-manager cloud-connector cloud-connector-worker \
     billing-adapter git-connector notification-service opsloop-consumer \
     auth-service-consumer webhook-publisher backend frontend

6. Verify end-to-end with a throwaway event (delete the resulting issue afterward):

   docker exec sencai-api-manager-1 node -e "
     const S = require('@sentry/node');
     S.init({ dsn: process.env.SENTRY_DSN });
     S.captureException(new Error('smoke test'));
     S.flush(5000).then(() => process.exit(0));
   "

PostHog (self-hosted product analytics)

1. First visit to http://posthog.sencai.localhost/ walks through org + project creation and a “select launch mode” step (live implementation vs. just experimenting) — either is fine for local dev, it only tweaks onboarding tips.
2. The project’s public API key (phc_..., shown on the project settings page) is a write-only capture key — safe to embed client-side, and it’s exactly what NUXT_PUBLIC_POSTHOG_KEY expects (see plugins/posthog.client.ts in the frontend, consent-gated behind the cookie banner).
3. Unlike GlitchTip’s DSN, NUXT_PUBLIC_POSTHOG_HOST stays as the public http://posthog.sencai.localhost even though it’s also client-side-only usage here — there is no server-side PostHog capture path in this codebase today, so the container-vs-browser DNS split doesn’t come up.
4. Deeper PostHog-side configuration (dashboards, insights, cohorts, feature flags) needs a personal API key (Settings → Personal API Keys), separate from the project capture key — the capture key can only write events in, not read/configure anything.

Lago (billing)

Lago is the self-hosted usage-based billing/metering system backing billing-adapter (port 3600, ADR-002) — it runs as getlago/api + getlago/front (both pinned to v1.30.0) and is reachable at http://lago.sencai.localhost: the API/GraphQL surface is routed under /api, /rails, and /graphql on that host, while the bare host serves the Lago UI. It’s a Rails app with has_secure_password (bcrypt, password_digest column) — there is no self-service “forgot password” flow in this self-hosted setup. To reset a user’s password directly:

docker exec sencai-lago-api bundle exec rails runner "
  u = User.find_by(email: 'user@example.com')
  u.update!(password: 'NEW_PASSWORD', password_confirmation: 'NEW_PASSWORD')
"

Verify via a login GraphQL mutation before handing the new password back:

curl -s -X POST -H 'Host: lago.sencai.localhost' -H 'Content-Type: application/json' \
  -d '{"query":"mutation { loginUser(input: { email: \"user@example.com\", password: \"NEW_PASSWORD\" }) { token } }"}' \
  http://localhost/graphql