OPSLOOP Intelligence Loop — Developer Reference

OPSLOOP Intelligence Loop — Developer Reference

OPSLOOP is the incident-processing pipeline that turns Alertmanager alerts (and other platform events) into incident records, enriches them with context, ranks likely root causes, and — for the modules that are enabled — triggers remediation. Most of its capabilities beyond the core alert-to-incident pipeline are opt-in, gated behind environment flags.

Components

Component	Location	Role
opsloop-consumer	opsloop-consumer/	Fastify HTTP server + RabbitMQ consumer — alert dedup/correlation, context collection, RCA, remediation, forecasting
llm-service	llm-service/	Fastify facade over Anthropic (default) / any OpenAI-compatible endpoint (fallback), port 4430
Lumen AI panel	sencai.space-frontend/	Frontend intelligence panel (stores/lumen.ts, lib/lumen/)
Strapi content-type	sencai.space/src/api/incident/	Single incident CT — alert correlation + RCA context live as JSON fields on the incident itself

There is no opsloop-incident, opsloop-context-bundle, or opsloop-runbook-suggestion content-type. Everything is stored on the one incident collection type: context is the context_bundle JSON field, ranked root-cause evidence is the rca_candidates JSON field, patched onto the same record as the pipeline progresses.

llm-service

A Fastify facade (port 4430) that routes LLM calls to a configured backend and enforces per-org authorization, rate limiting, and budget caps. It is provider-agnostic:

• Default provider: Anthropic (@anthropic-ai/sdk), model claude-haiku-4-5-20251001 (LLM_DEFAULT_MODEL)
• Fallback provider: LLM_PROVIDER=openai_compat — works against any OpenAI-compatible endpoint (OPENAI_COMPAT_BASE_URL), including self-hosted options like Ollama, vLLM, or LocalAI

Endpoints (no /api/ prefix, no /embed endpoint):

POST /chat
Body: { messages: [{ role: 'user'|'assistant', content: string }], model?: string, org_id?: number }
Response: { content: string, model: string, usage: { input_tokens, output_tokens, cost_usd } }

POST /complete
Body: { prompt: string, model?: string, org_id?: number }
Response: { content: string, model: string, usage: { input_tokens, output_tokens, cost_usd } }

Authentication is mandatory on both endpoints — a Keycloak Bearer JWT is required, verified via JWKS (RS256 only). org_id used for rate limiting, usage accounting, and budget enforcement is always resolved from the caller’s real organisation-member Strapi record (looked up by the verified Keycloak sub), never trusted from the request body. A body-supplied org_id is cross-checked against the caller’s actual memberships and rejected on mismatch (403 llm.auth.org_mismatch).

Budget enforcement: each organisation has a monthly spend cap, LLM_BUDGET_USD_PER_ORG_MONTH (default $10). Requests are rejected with 402 (llm.budget.exceeded) before the provider is called once the cap is hit.

Rate limiting: per-org daily request counters, tier-aware — free vs pro plan (RATE_LIMIT_FREE_PER_DAY default 100, RATE_LIMIT_PRO_PER_DAY default 5000).

Both the budget tracker and the rate limiter are in-memory only — state is lost on service restart, and there is no Redis-backed persistence yet.

Lumen AI Panel

Lumen is the frontend intelligence surface in sencai.space-frontend (stores/lumen.ts, components/layout/Lumen*.vue, collapsible panel).

Privacy filter: before any context is handed to an LLM adapter, sencai.space-frontend/lib/lumen/privacy-filter.ts (scrubContext()) runs an allowlist-based scrub of the context-bus snapshot: unknown item kinds are dropped entirely, only allowlisted keys survive per item kind, any key matching a sensitive-name pattern (secret, password, token, credential, api[-_]?key, auth, …) is redacted, and values that look like JWTs, Stripe/OpenAI-style API keys, AWS access-key IDs, or credential-bearing connection strings are dropped outright. This filter lives client-side in the frontend, not in llm-service.

Alert → Incident Pipeline

The primary, always-on pipeline:

Alertmanager → POST /webhook/alertmanager (HMAC-verified via ALERTMANAGER_WEBHOOK_SECRET)
  → processWebhookBatch()
  → buildDedupKey() — SHA-256 of sorted alert labels, excluding `instance`
  → in-memory dedup window (5 min, CORRELATION_WINDOW_MS) + Strapi fallback lookup
  → existing incident within window → alert_count incremented
  → otherwise → createIncident() → Strapi POST /api/incidents
  → collectContext() [async] → context_bundle patched onto the incident
  → scoreAndPatchRcaCandidates() [async] → rca-engine.ts heuristic scoring → rca_candidates patched
  → handleIncidentAutoRemediation() [async, fire-and-forget] → remediation-rule matching + cooldown

RabbitMQ exchanges and queues

Exchange	Type	Queue	Routing key	Source / purpose
alertmanager.events	topic	opsloop.alerts	alert.firing	Primary alert dedup/incident pipeline (also reachable via HMAC-verified POST /webhook/alertmanager)
platform.events	fanout	opsloop.alert-correlator	monitoring.alert.*, agent.alert.*, cloud-instance.error (filtered in-consumer)	Alert correlator
platform.events	fanout	rca-generator.dispatch	incident.resolved (filtered in-consumer)	AI-assisted RCA generator
platform.events	fanout	code-review.dispatch	code.review.# (filtered in-consumer)	Code review dispatch handler
opsloop.dlx	topic	opsloop.alerts.dead	#	Shared dead-letter queue for all four queues above

platform.events is declared as a fanout exchange — every bound queue receives every message, and each consumer filters by routing-key pattern itself (amqp-topic-match.ts) rather than relying on broker-side topic routing.

Prefetch is 10 across the consumer. On processing failure, messages are nacked without requeue straight to the dead-letter queue — there is currently no retry counter or exponential backoff; a failed message is dead-lettered on the first error.

Modules

Beyond the always-on dedup/context/RCA-heuristic/auto-remediation path above, opsloop-consumer ships roughly ten additional capabilities. Most are opt-in via environment flag and default to false; two (the heuristic RCA engine and auto-remediation) are always active because they have no independent flag of their own — their safety gate lives at the remediation-rule level (confidence_threshold/cooldown_minutes), not a kill switch.

Module	File	Description	Enabled via
Alert correlator	services/alert-correlator.service.ts	Cross-service correlation of platform.events alerts into incidents (org:severity:resource-type key, 5-min sliding window)	OPSLOOP_DEDUP_ENABLED=true
Context collector (enriched)	services/context-collector.service.ts	Collects a change timeline around the incident window — Gitea commits, IAM audit events, drift, provisioning — tenant-scoped to orgId	CONTEXT_COLLECTION_ENABLED=true
RCA engine	rca-engine.ts	Heuristic scoring of root-cause candidates (deploy_recent, iam_change, drift_event, config_change, resource_exhaustion). Never calls an LLM and never issues a single authoritative verdict — only ranks evidence for the operator	Always active (invoked from the context collector)
RCA generator (AI-assisted)	rca-generator.ts	On incident.resolved, assembles an evidence bundle and calls llm-service (model claude-haiku-4-5-20251001); falls back to a rule-based summary if the LLM call fails or is unavailable. Distinct from the always-on RCA engine above	INCIDENT_RCA_ENABLED=true
RCA ranker	services/rca-ranker.service.ts	Evidence-weighted confidence scoring plus Jaccard-similarity dedup of RCA candidates	Invoked from the RCA generator pipeline
Auto-remediation	auto-remediation.ts	After incident creation: matches a remediation-rule (regex/keyword on dedup_key/title), applies confidence threshold + cooldown, then auto-triggers a runbook (auto_approve=true) or opens a change request for human approval	Always active — no independent flag
SRE runbook auto-trigger	services/auto-trigger.service.ts	Alternative/complementary path to auto-remediation — severity gate + cooldown, emits ops.auto_trigger.fired/skipped	AUTO_TRIGGER_ENABLED=true
Capacity forecast	capacity-forecast.service.ts	Weekly cron (Sunday 05:00 UTC) — linear regression over agent-metric-snapshot (cpu/memory/disk), threshold-based recommended_size, upserts a lumen-recommendation (right-sizing)	CAPACITY_FORECAST_ENABLED=true
Workload profiler	services/workload-profiler.service.ts	Daily cron (04:00 UTC) — hour-of-day CPU/memory bucketing into profile_type (cpu_bound/memory_bound/idle/io_bound/mixed), LLM-assisted migration recommendation with rule-based fallback	WORKLOAD_PROFILER_ENABLED=true
Pattern correlator	pattern-correlator.ts	Runs every 15 minutes — cross-service anomaly detection: cascade failure (≥3 incidents / 10 min), recurring alert (same dedup_key ≥3× / 24h), upserts correlation-findings	CORRELATION_ENABLED=true
Code review handler	code-review-handler.ts	On code.review.requested: fetches the raw diff from the Gitea Apps API (max 2 MB), writes diff_patch to Strapi, triggers POST /api/code-review-requests/:id/process	Always active (registered unconditionally in consumer.ts)

Runbook Suggestions and Auto-Remediation

Auto-remediation and the SRE auto-trigger path do not produce a free-form {cmd, timeout} proposal. They operate against the real runbook Strapi content-type schema:

{
  "name": "Restart nginx on node-abc",
  "trigger_type": "alert",
  "trigger_condition": { "metric": "cpu_percent", "operator": ">", "threshold": 90 },
  "actions": [
    { "type": "restart_service", "params": { "service": "nginx" } }
  ],
  "confirmation_required": true,
  "cooldown_minutes": 60
}

actions[].type is one of restart_service, clear_disk_space, kill_process, run_approved_script. Runbooks flagged confirmation_required: true (the schema default) are not executed directly — they are submitted as an approval-request (POST /api/approval-requests, action_type: "runbook.execute") for a human to approve, with a 4-hour TTL (expires_at) and a stored blast_radius/rollback_plan. There is no “Action Gateway” or “cloud-approval-request” content-type in this flow — approval-request is the actual mechanism.

Escalation (PagerDuty / Opsgenie)

Escalation to an external paging provider does not live in opsloop-consumer. It is implemented in cloud-connector/src/services/escalation.service.ts, reading policy configuration from the Strapi escalation-policy content-type (provider: pagerduty or opsgenie, per-org routing_key / encrypted API key, gated by ESCALATION_ENABLED).

escalateIncident() in that service posts to the PagerDuty Events v2 API or the Opsgenie Alerts API depending on the policy’s configured provider. As of this writing the function exists and is fully implemented but is not called from any incident-creation code path in the repository — only startEscalationConsumer() runs at worker startup, and it does no more than log readiness. Confirm the current wiring status with the service owner before relying on live PagerDuty/Opsgenie escalation in production.

Synthetic Monitoring

Synthetic checks are run by cloud-connector (src/services/synthetic-check.service.ts, gated by SYNTHETIC_CHECK_ENABLED), not by Alertmanager. A per-minute cron re-runs any synthetic-check whose interval has elapsed, then posts the outcome to Strapi:

cloud-connector cron (every minute)
  → GET /api/synthetic-checks (enabled checks)
  → per check: HTTP request, compare status/body against expectations
  → POST /api/synthetic-results (X-Service-Secret)
  → PUT /api/synthetic-checks/:id (last_check_at, last_status)

There is currently no lifecycle hook, RabbitMQ publish, or anomaly.events-style routing from synthetic-result creation into the incident pipeline — a failing synthetic check is recorded in Strapi but does not automatically open an incident or notify opsloop-consumer. Treat any synthetic-check-to-incident automation as not yet implemented; confirm with the service owner before documenting or relying on one.

Observability

• Health: GET /health
• Metrics: GET /metrics — Prometheus text format (text/plain; version=0.0.4), including httpRequestsTotal, httpRequestDurationMs, alertsProcessedTotal, incidentsCreatedTotal
• Error tracking: @sentry/node, pointed at a self-hosted GlitchTip instance (Sentry-ingestion-protocol compatible, not Sentry SaaS — profile glitchtip in orchestration), consistent with the platform’s no-paid-SaaS policy. beforeSend PII scrubbing happens in src/sentry-pii-scrub.ts. An empty SENTRY_DSN is a safe no-op — no GlitchTip instance is required for local development.
• Audit: non-HTTP mutations (dedup, context collection, RCA scoring/generation, remediation, forecasting) publish directly via the vendored @sencai/audit package (e.g. incident.deduplicated, incident.context.collected, incident.rca.candidates_updated, rca.generated, ops.remediation.triggered/completed, ops.auto_trigger.fired/skipped, capacity.forecast.generated, workload.profile.generated). Incident creation via the Strapi REST API additionally passes through the global audit-logger middleware automatically.

Adding a New Intelligence Source

1. Publish to the appropriate exchange — alertmanager.events (routing key alert.firing) for alert-shaped sources, or platform.events (fanout; pick a routing key convention and filter for it in-consumer via amqp-topic-match.ts) for everything else.
2. Add a consumer/handler module in opsloop-consumer/src/ (or src/services/) following the existing pattern (assert queue with x-dead-letter-exchange: opsloop.dlx, bind, filter by routing key).
3. Wire it up in consumer.ts (or index.ts for HTTP/cron-based modules) and gate it behind a new environment flag, defaulting to false, unless it is a low-risk read path.
4. If the source should feed root-cause analysis, extend context-collector.ts / context-collector.service.ts to include the new evidence type, and add a corresponding candidate type to rca-engine.ts if appropriate.