Architecture

Platform Architecture

Sencai is an EU-sovereign cloud management platform — an event-driven, multi-tenant SaaS that gives teams a single pane of glass over their cloud and on-prem infrastructure. The architecture is microservice-based, orchestrated with Docker Compose, and built for async operations via RabbitMQ.

---

Mission

“A DevOps colleague with hands on cloud and on-prem” — automate the boring parts, surface the important parts, and enforce policy automatically.

---

Service Map

Service	Purpose	Port	Language
sencai.space	Strapi v5 backend — REST API, content-types, lifecycle hooks	1337	Node.js / TS
sencai-backend	Go strangler-fig replacement backend — shadow-only, no live traffic (see below)	8080 (internal)	Go
sencai.space-frontend	Platform UI — auth, dashboard, gravity UI	3000	Nuxt 3 / Vue 3
sencai-admin	Internal admin dashboard (KC-auth)	3100	Nuxt 3
sencai-audit	Audit event viewer SPA (read-only)	3300	Nuxt 3
sencai-web	Marketing site (Astro 5 SSG, 15 locales)	—	Astro
api-manager	Encrypted service-to-service token management	3200	Fastify / TS
auth-service-consumer	RabbitMQ → Keycloak sync + audit events	—	Node.js / TS
webhook-publisher	Strapi webhooks → RabbitMQ bridge	1347	Node.js (JS)
git-connector	RabbitMQ → Gitea sync (Toolbox + Apps)	3002	Node.js / TS
cloud-connector	Multi-cloud provisioning worker (AWS/GCP/Azure/Hetzner/DO)	4321	Node.js / TS
opsloop-consumer	Intelligence loop — incident correlation, runbook suggestions	3005	Node.js / TS
ssh-proxy-service	Browser terminal via WebSocket (SSH proxy)	3400	Node.js / TS
graphql-gateway	Unified GraphQL API gateway	4000	Node.js / TS
billing-adapter	Lago usage-based billing + Stripe payments (ADR-002)	3600	Node.js / TS
sencai-watchdog	Dunning microservice — payment-failure suspend/archive cron	3700	Node.js / TS
llm-service	Multi-provider LLM facade (Anthropic-first, OpenAI-compatible fallback)	4430	Node.js / TS
workspace-connector	Google Workspace Directory API sync	3500	Node.js / TS
notification-service	Email + Slack consumer (notification.events, billing.events)	—	Node.js / TS
event-store-consumer	platform.events fanout consumer → Strapi event ingest	—	Node.js / TS
backup-scheduler	Scheduled MySQL → S3/GCS backup + restore drill cron	—	Node.js / TS
agent-gateway	Fleet C2 gateway — enrollment + WSS heartbeat/command channel	4400	Go
sencai-agent	On-host fleet agent binary (monitoring, patch, inventory, runbooks)	— (client)	Go
k8s-agent	Kubernetes DaemonSet Fleet agent	—	Go
gamification-consumer	Sole writer of the user-rank/gamification-event content-types (F4.GAM) — computes XP/rank/badge changes server-side	3800	Node.js / TS
forum-connector	JIT provisioning bridge between Keycloak and sencai-forum (Lemmy)	4181	Fastify / TS
sencai-forum	Community forum (Lemmy 0.19 + oauth2-proxy for KC SSO)	8536	— (upstream)
auth-service	Keycloak configuration + custom themes	8080	— (upstream)
sencai-mq	RabbitMQ broker	5673	— (upstream)

Note

infra-gcp (CDKTF IaC for the platform’s own multi-region GCP infrastructure) is not a Docker Compose service — it is deployed independently of the orchestration stack.

---

High-Level Architecture

┌──────────────────────────────────────────────────────────────────┐
│                      User & Marketing Layer                       │
├──────────────────────────────────────────────────────────────────┤
│  sencai.space-frontend (Nuxt 3)  │  sencai-web (Astro SSG)      │
│  sencai-admin (Nuxt 3)           │  sencai-audit (Nuxt 3 SPA)   │
└───────────────────────┬──────────────────────────────────────────┘
                        │ HTTPS
┌───────────────────────▼──────────────────────────────────────────┐
│                    Traefik (Edge Proxy)                           │
│  Routes: /api/* → Strapi  |  /api/token → API Manager           │
│          /kc/* → Keycloak |  /api/auth/* → Frontend server route │
└──────┬───────────────┬───────────────────┬───────────────────────┘
       │               │                   │
  ┌────▼────┐    ┌─────▼──────┐    ┌──────▼──────┐
  │  Strapi │    │ API Manager│    │  Keycloak   │
  │  (1337) │    │   (3200)   │    │   (8080)    │
  │  MySQL  │    │ AES-256-GCM│    │   MySQL     │
  └────┬────┘    └────────────┘    └─────────────┘
       │
       │ webhooks
┌──────▼──────────────────────────────────────────────────────────┐
│                  Webhook Publisher (1347)                        │
│              Strapi events → RabbitMQ routing                    │
└──────┬──────────────────────────────────────────────────────────┘
       │ AMQP
┌──────▼──────────────────────────────────────────────────────────┐
│                 RabbitMQ (5673 AMQP / 15673 UI)                  │
│  Exchanges: user  |  organisation  |  git.events                │
│             cloud.events  |  audit.events  |  notification.events│
│             billing.events  |  platform.events  |  alertmanager.events │
└──┬───────────┬──────────────┬────────────────┬───────────────────┘
   │           │              │                │
   ▼           ▼              ▼                ▼
Auth        Git          Cloud           OpsLoop /
Consumer  Connector    Connector        Notification /
(KC sync) (Gitea)    (AWS/GCP/...)   Event-Store Consumers
                           │
                    ┌──────▼──────┐
                    │ Provider SDK │
                    │ AWS/GCP/AZ  │
                    │ Hetzner/DO  │
                    └─────────────┘

Fleet plane (separate from the RabbitMQ bus):

sencai-agent (Go, on host) ──WSS+mTLS──▶ agent-gateway (Go, :4400) ──REST──▶ Strapi

---

Event-Driven Architecture

Most cross-service communication goes through RabbitMQ. Business events avoid direct service-to-service HTTP calls where possible — this keeps services decoupled and makes the system resilient to temporary failures. (Fleet enrollment/heartbeat is the deliberate exception — see Fleet Agent Enrollment — it runs over a dedicated WSS+mTLS channel between sencai-agent and agent-gateway, not RabbitMQ.)

RabbitMQ Exchanges

Routing keys follow <entity>.<action> on the topic exchanges. user and organisation are direct exchanges; platform.events is fanout.

Exchange	Type	Routing Key Examples	Consumers
user	direct	create, update	auth-service-consumer
organisation	direct	create, member.added, member.removed	git-connector, opsloop-consumer
git.events	topic	git.repo.created, git.key.rotated	git-connector
cloud.events	topic	cloud-instance.provision, cloud-instance.stop, cloud-instance.terminate	cloud-connector worker
audit.events	topic	audit.write	auth-service-consumer / audit-consumer (persists hash-chained entries)
notification.events	topic	billing.dunning.grace_period, feedback.submitted, dpa.requested, …	notification-service (email + Slack)
billing.events	topic	billing.subscription_activated, billing.payment_succeeded, billing.payment_failed, …	notification-service
platform.events	fanout	(all platform events, no routing key)	event-store-consumer → POST /api/platform-events/ingest
alertmanager.events	topic	monitoring alerts	opsloop-consumer
gamification.events	topic	xp.granted, badge.check, streak.tick, easter-egg.roll	gamification-consumer (re-publishes rank.up/badge.awarded/easter-egg.triggered for the frontend notification bridge)

Message Envelope (cloud.events example)

{
  "event": "cloud-instance.provision",
  "data": {
    "jobId": "uuid-v4",
    "type": "cloud-instance.provision",
    "cloudInstanceId": "strapi-document-id",
    "credential_id": "byoc-credential-document-id",
    "payload": {
      "provider": "hetzner",
      "region": "nbg1",
      "config": { "serverType": "cx22", "image": "ubuntu-24.04" }
    }
  }
}

Failure Handling

• Fallback queues: 14-day TTL for undeliverable messages
• Retry policy: max 5 attempts with exponential backoff
• Dead-letter exchanges capture permanently failed messages for inspection (e.g. notification.events.dlq, platform.events.dlq)

---

Key Data Flows

User Registration

POST /api/registrations (Strapi)
  → Strapi afterCreate lifecycle → webhook-publisher
  → RabbitMQ: user exchange / create
  → auth-service-consumer → KC Admin API (create user, VERIFY_EMAIL action)
  → KC sends verification email

Admin sets Registration.emailVerified = true in Strapi UI
  → Strapi lifecycle → webhook-publisher
  → auth-service-consumer removes VERIFY_EMAIL from KC user
  → User can log in

First Login (Profile Completion)

Login → Keycloak token → /callback.vue
  → GET /api/users?filters[email]=... (Strapi)
  → if !name OR !surname → redirect /auth/complete-profile
  → Form submit → PUT /api/users/me/profile (Strapi)
  → Strapi User afterUpdate lifecycle → KC sync (via webhook-publisher)
  → redirect /gravity/dashboard

Cloud Instance Provisioning (ADR-001 canonical path)

Frontend → POST /api/cloud-instances (Strapi)
  → afterCreate lifecycle generates jobId + canonical message
  → webhook-publisher → RabbitMQ cloud.events (cloud-instance.provision)
  → cloud-connector worker picks up message
  → provider SDK/REST (Hetzner API, AWS EC2, GCP Compute, etc.)
  → POST /api/cloud-instances/:documentId/status-callback (X-Service-Secret)
  → Strapi updates instance status: "ready"
  → Frontend polls/receives update

Note

Cloud provisioning uses provider SDKs directly — not CDKTF/Terraform for runtime operations. CDKTF is reserved for the platform’s own infrastructure baseline (infra-gcp/), not customer cloud-instance provisioning.

Fleet Agent Enrollment

Server admin generates an org-scoped enrollment token in Strapi
  → sencai-agent (Go binary) calls POST /enroll on agent-gateway (port 4400)
     body: { enrollment_token, hostname, os, arch, version }
     (bootstrap call, no mTLS required yet)
  → agent-gateway verifies the token against Strapi (fail-closed on Strapi outage),
     checks the org's agent-count limit, then issues a per-agent mTLS leaf cert
  → agent-gateway activates the agent in Strapi (status=active, cert fingerprint, enrolled_at)
  → agent stores the signed cert + CA bundle, opens GET /ws over WSS+mTLS
  → agent sends a heartbeat every 30s; agent-gateway throttles writes to
     Strapi's last_heartbeat_at (max one write per ~25s) and pushes
     capability grant/revoke messages down the same socket

ssh-proxy-service is unrelated to fleet enrollment — it only powers the browser terminal (see below).

Browser Terminal (SSH Proxy)

User clicks "Connect" on enrolled server in UI
  → Frontend opens WebSocket to ssh-proxy-service (wss://)
  → ssh-proxy-service authenticates user JWT
  → Opens SSH connection to target server via the fleet agent
  → Bidirectional terminal stream over WebSocket
  → All keystrokes logged to audit trail

OPSLOOP Intelligence

Incident event arrives (alert, anomaly, error spike)
  → opsloop-consumer receives alertmanager.events / cloud.events
  → llm-service (multi-provider LLM facade, Anthropic-first)
  → Correlates with recent runbooks, CIS scan results, inventory
  → Generates: incident summary, root cause candidates, suggested runbook
  → Creates OPSLOOP bundle, pushes notification to frontend

API Token Management

Microservice needs a token for another service
  → GET /api/token/:service (API Manager, port 3200, X-Service-Secret)
  → API Manager decrypts the token from the /data volume (AES-256-GCM)
  → If missing or older than 7 days → fetch/rotate via Strapi admin API
  → New token encrypted (AES-256-GCM) and persisted
  → Auto-rotation every 7 days

Git Synchronization

Strapi webhook (user/org action)
  → webhook-publisher
  → RabbitMQ: git.events
  → git-connector (port 3002)
  → Gitea Toolbox API + Gitea Apps API
  → Confirmation callback back into Strapi

Billing / Usage Metering (ADR-002)

Platform events (provisioning, usage)
  → billing-adapter (port 3600) aggregates usage events
  → Lago (usage metering) → Stripe (invoicing/payments)
  → subscription-tier / quota enforcement in Strapi
  → payment failures escalate through sencai-watchdog's dunning ladder
     (notification.events, billing.dunning.* routing keys)

Notification + Event Store

*.events (notification.events, billing.events)
  → notification-service → email + Slack delivery

*.events (fanout via platform.events)
  → event-store-consumer
  → POST /api/platform-events/ingest (Strapi)

---

Database Topology

Service	Engine	Docker Host	Dev Port	Notes
Strapi (backend)	MySQL 8	backend-db	3306	Knex auto-migrations
Keycloak	MySQL 8	keycloak-db	3307	KC-managed schema
Gitea (both instances)	MySQL 8	git-service-mysql	3308	Shared DB
Cloud Connector	PostgreSQL 15	cloud-connector-db	5434	TypeORM sync
Forum (Lemmy)	PostgreSQL 15	forum-db	5435	Lemmy-managed schema
sencai-backend (Go)	MySQL 8	go-backend-db	3309	Split from backend-db 2026-07-06; no automated migration tool, schema evolves via reviewed SQL applied by hand

Note

Most of the newer microservices — agent-gateway, billing-adapter, sencai-watchdog, llm-service, opsloop-consumer, notification-service, event-store-consumer, workspace-connector — are stateless. They persist data via Strapi/MySQL over REST rather than running their own database.

Multi-region HA: Active-passive PostgreSQL replication with automated failover. Target RTO < 15 minutes, RPO < 5 minutes.

---

Go Backend Migration (Strangler-Fig)

sencai-backend is a Go rewrite of the Strapi backend, ported content-type by content-type behind the identical HTTP contract. It runs against its own database (go-backend-db, above), kept current with Strapi’s backend-db through a one-directional, event-triggered sync mechanism (internal/dbsync) rather than a shared database. It is shadow-only — not wired into default orchestration profiles, and it routes no real customer traffic; Strapi remains the platform’s only live backend. Full detail: Go Backend Migration — Developer Reference.

---

Authentication & Identity

Keycloak (Primary IdP)

• Realm: sencai
• Client IDs: sencai-backend, sencai-frontend
• URL behind Traefik: /kc prefix (KC_HTTP_RELATIVE_PATH=/kc)
• Internal URL: http://keycloak:8080/kc
• JWT access tokens: 12-hour expiry, with a proactive refresh-token flow (implemented, WAVE-5) — the frontend refreshes ~60s before expiry, with single-flight retry and logout only on a genuine 401 from the refresh endpoint
• TOTP/2FA: Speakeasy library

Enterprise SSO

• EntraId / Azure AD: OIDC federation into Keycloak realm
• Google Workspace SSO: OAuth2 via Keycloak social login
• SCIM provisioning: SCIM 2.0 endpoint for automated user lifecycle from IdP

Strapi ↔ KC Sync

Two-way sync, loop-guarded with AsyncLocalStorage:

Strapi → KC (lifecycle hooks):
  plugin::users-permissions.user afterCreate/afterUpdate
  → webhook-publisher → RabbitMQ → auth-service-consumer → KC Admin API

KC → Strapi (on login):
  keycloak-jwt middleware validates RS256 JWT
  → if stale profile: fetchKcUserById() → runWithoutKcSync(syncKcUserToStrapi)

Synchronized fields: email, username (= KC sub UUID), firstName/lastName, emailVerified, blocked/enabled, jobTitle, country.

---

Security Model

Transport

• External traffic: TLS via Traefik (Let’s Encrypt or custom cert)
• Internal Docker network: Plain TCP (no TLS required — isolated bridge network)
• Fleet agent ↔ agent-gateway: mTLS over WebSocket (WSS) — the initial /enroll bootstrap call is unauthenticated by design (token-based), but the resulting /ws control channel requires the per-agent client certificate issued during enrollment

Secrets at Rest

• API tokens (API Manager): AES-256-GCM, key from API_MANAGER_ENCRYPTION_KEY
• BYOC credentials: AES-256-GCM per-organization, stored in Strapi, never logged
• Vault integration: HashiCorp Vault (vault profile) available for secret injection into services and CI jobs

Audit Trail

• Every mutating action publishes to audit.events exchange (routing key audit.write)
• @sencai/audit package: publishAuditEvent(action, payload) with AsyncLocalStorage context
• Immutable hash chain: each entry includes SHA-256 of the previous entry
• On-demand anchoring: POST /api/audit-logs/anchor (admin-only, X-Service-Secret) signs the latest entry and stores it in the audit-anchor content-type (anchor_hash, signature, public_key_fingerprint, anchored_entry_id, entries_count) — there is no fixed hourly schedule; anchoring is triggered explicitly
• Verification: GET /api/audit-logs/verify-chain recomputes the full hash chain; GET /api/audit-logs/verify?from=&to=&organisation= verifies a scoped range
• ESLint gate: eslint-plugin-sencai/require-audit-log enforces audit coverage on mutating functions

Agent Autonomy & Policy Enforcement

There is no standalone “Action Gateway” service, no OPA/Rego/Cedar policy bundle format, and no api::capability or cloud-approval-request content-type. The real mechanism:

• agent-policy content-type — OPA-inspired but native Strapi data: autonomy_level (L0 read-only → L1 propose-only → L2 execute-with-approval → L3 pre-blessed automation), freeze_windows, max_blast_radius, and an optional allowlist of action types. Deny-by-default: no matching active policy blocks the action.
• approval-request and elevation-request content-types hold the actual human-in-the-loop records for actions that need sign-off or temporary capability elevation.
• Enforcement point: sencai.space/src/middlewares/autonomy-gate.ts — a Strapi middleware that blocks agent-initiated mutating requests based on the org’s autonomy_level and any active freeze_windows before the request reaches the controller.

Service-to-Service Auth

• X-Service-Secret header for internal REST calls (API Manager, Strapi webhooks, agent-gateway)
• Rate limiting on API Manager: 100 req / 15 min per IP
• Helmet + CORS on all Fastify/Express services

---

Multi-Tenancy Model

Organization (tenant)
  ├── Members (role: Owner / Admin / Member / Viewer)
  ├── Cloud Instances (scoped per org)
  ├── BYOC Credentials (encrypted per org)
  ├── Fleet Agents (enrolled per org)
  ├── Agent Policies (autonomy_level, freeze_windows, per org)
  └── Billing Subscription (Lago + Stripe per org)

Cross-tenant operations are allowed only for MSP/agency roles (B2B2B track). All API endpoints filter by organizationId from the KC JWT claims. Data isolation is enforced at the Strapi policy layer.

---

Monorepo Structure

code/
├── orchestration/               # Docker Compose entry point + .env
├── sencai.space/                # Strapi v5 backend
├── sencai.space-frontend/       # Platform Nuxt UI
├── sencai-admin/                # Admin Nuxt UI
├── sencai-audit/                # Audit viewer SPA
├── sencai-web/                  # Astro marketing site
├── api-manager/                 # Fastify token service
├── auth-service/                # Keycloak config + themes
├── auth-service-consumer/       # RabbitMQ → KC sync
├── webhook-publisher/           # Strapi → RabbitMQ bridge
├── git-connector/               # RabbitMQ → Gitea sync
├── cloud-connector/             # Multi-cloud provisioning worker
├── opsloop-consumer/            # AI intelligence loop consumer
├── ssh-proxy-service/           # Browser terminal WebSocket service
├── graphql-gateway/             # GraphQL API gateway
├── billing-adapter/             # Lago + Stripe billing bridge
├── sencai-watchdog/             # Dunning microservice (payment-failure cron)
├── llm-service/                 # Multi-provider LLM facade
├── workspace-connector/         # Google Workspace Directory sync
├── notification-service/        # Email + Slack consumer
├── event-store-consumer/        # platform.events fanout → Strapi ingest
├── backup-scheduler/            # MySQL → S3/GCS backup cron
├── agent-gateway/                # Fleet C2 gateway (Go)
├── sencai-agent/                 # On-host fleet agent binary (Go)
├── k8s-agent/                   # Kubernetes Fleet agent (Go)
├── infra-gcp/                    # CDKTF IaC — platform's own GCP infra (not a compose service)
├── gamification-consumer/        # XP/rank/badge RabbitMQ consumer (F4.GAM)
├── forum-connector/               # KC ↔ Lemmy JIT provisioning bridge
├── sencai-mq/                    # RabbitMQ config
├── git-service/                  # Gitea config (Toolbox + Apps)
├── sencai-forum/                 # Lemmy community forum
├── bootstrap/                    # DB bootstrap scripts
├── sencai-e2e/                   # Playwright E2E test suite
├── scripts/                      # Cross-repo helper scripts
├── packages/
│   └── audit/                   # @sencai/audit shared package
├── tooling/
│   └── eslint-plugin-sencai/    # require-audit-log ESLint rule
└── sencai-docs/             # This documentation site

---

Startup Order

Services declare Docker health checks; depends_on: { condition: service_healthy } enforces ordering:

1. Databases & secrets — MySQL (x3), PostgreSQL (x2), and Vault (if the vault profile is enabled) start early, since several services fetch secrets from Vault at boot
2. Messaging — RabbitMQ (sencai-mq)
3. Proxy — Traefik
4. Core — Strapi, Keycloak (30–90s startup), API Manager
5. Consumers / standalone services — auth-service-consumer, git-connector, cloud-connector worker, opsloop-consumer, notification-service, event-store-consumer, agent-gateway, billing-adapter, gamification-consumer, forum-connector
6. Frontends — sencai.space-frontend, sencai-admin, sencai-audit
7. Optional — Forum, docs, monitoring

---

Scaling Considerations

Stateless (horizontal scale-out):

• Strapi (shared MySQL), API Manager, Webhook Publisher, Git Connector, Cloud Connector workers, OpsLoop Consumer, agent-gateway, billing-adapter, sencai-watchdog, llm-service, notification-service, event-store-consumer, workspace-connector, gamification-consumer, forum-connector

Stateful (single-instance or cluster mode):

• RabbitMQ (cluster supported but complex), Keycloak (cluster with shared DB), Gitea (one instance per purpose)

Databases:

• MySQL: replica read pools recommended for Strapi at scale
• PostgreSQL: active-passive streaming replication (Cloud Connector), multi-AZ for HA