Fleet Agent — Developer Reference

Fleet Agent — Developer Reference

The Sencai fleet layer consists of a lightweight Go binary, sencai-agent, that runs on managed hosts (VMs/bare-metal), and a Go control-plane service, agent-gateway, that enrolls agents and bridges their WebSocket-over-TLS (WSS) connection to Strapi. A separate, unrelated component, k8s-agent, handles Kubernetes clusters and does not go through agent-gateway — see Kubernetes fleet agent below.

ssh-proxy-service (port 3400) is not part of this architecture — it only proxies browser-based SSH terminal sessions to cloud instances and has no relation to sencai-agent or agent-gateway.

Architecture

Managed Host / VM
├── sencai-agent (Go binary)
│   ├── mTLS client → agent-gateway (WSS, port 4400)
│   ├── patch management (internal/patch)
│   ├── software inventory + EOL detection (internal/inventory)
│   ├── CIS/Lynis hardening scan (internal/cis)
│   ├── log shipping to Loki (internal/logshipper)
│   ├── Prometheus Pushgateway export, opt-in (internal/pushgateway)
│   └── runbook executor (internal/runbook)
│
agent-gateway (Go, port 4400)
├── POST /enroll        — bootstrap, no auth, verifies token against Strapi
├── GET  /ws             — mTLS, heartbeat + server-push channel
├── GET  /health          — liveness probe
├── POST /bulk-action     — X-Service-Secret, server-side callers only
├── GET  /fleet-status    — X-Service-Secret, server-side callers only
├── POST /dispatch-runbook — X-Service-Secret, server-side callers only
└── POST /dispatch-exec   — X-Service-Secret, server-side callers only

Source: sencai-agent/ (agent binary), agent-gateway/ (control-plane service). Both are Go repos, independent of k8s-agent/.

Enrollment flow

1. An enrollment token is generated for a sencai-agent record in Strapi (status pending, org-scoped).
2. The agent sends POST <gateway>/enroll with {enrollment_token, hostname, os, arch, version}. This endpoint requires no client certificate — it is the bootstrap step.
3. agent-gateway verifies the token against Strapi (the matching sencai-agent record must be status: pending), then generates the keypair and certificate itself: a self-signed CA (4096-bit RSA, 10-year validity, persisted under /etc/agent-gateway/) issues a per-agent leaf certificate (2048-bit RSA, 1-year validity, CommonName set to the Strapi documentId of the agent). The agent does not generate its own keypair.
4. The gateway activates the agent in Strapi (status: active, cert_fingerprint stored) and returns {agent_id, cert, key, ca_cert, capabilities}.
5. The agent writes the certificate, key, and CA bundle to /etc/sencai-agent/certs/ and its configuration to /etc/sencai-agent/config.yaml.
6. sencai-agent start opens the mTLS WSS connection to /ws and begins the heartbeat loop.

sencai-agent enroll --gateway https://agent-gateway:4400 --token <token>
sencai-agent start

Installation

curl -fsSL https://agent.sencai.space/install.sh \
  | SENCAI_GATEWAY_URL=https://agent-gateway:4400 SENCAI_TOKEN=<token> bash

Signed .deb/.rpm packages (built with nfpm, GPG-signed) are also published to tagged GitHub releases of sencai-agent; on Debian/Ubuntu hosts the installer offers the package path before falling back to the raw binary.

Heartbeat

Once connected over the mTLS GET /ws channel, the agent sends a heartbeat every 30 seconds carrying cpu_percent, memory_used_bytes, and disk_used_bytes. Heartbeat writes to Strapi (last_heartbeat_at) are rate-limited on the gateway side to protect against connection storms.

{
  "type": "heartbeat",
  "agent_id": "...",
  "timestamp": 1718900000,
  "cpu_percent": 12.5,
  "memory_used_bytes": 1073741824,
  "disk_used_bytes": 21474836480
}

agent-gateway ports and endpoints

agent-gateway listens on a single port, 4400 (HTTPS/WSS). There is no separate metrics port.

Endpoint	Auth	Purpose
POST /enroll	none (bootstrap)	Exchange enrollment token for mTLS certificate
GET /ws	mTLS client cert	Heartbeat channel + server-push (capabilities_update, revoke, execute_runbook)
GET /health	none	Liveness probe
POST /bulk-action	X-Service-Secret	Server-side only — bulk fleet operations
GET /fleet-status	X-Service-Secret	Server-side only — aggregate fleet status
POST /dispatch-runbook	X-Service-Secret	Server-side only — triggers runbook execution on an agent
POST /dispatch-exec	X-Service-Secret	Server-side only — triggers a whitelisted exec action

The X-Service-Secret-protected endpoints are called only by other backend services (never directly by agents or browsers).

Software inventory

There is no osquery integration anywhere in the fleet stack. Inventory is collected by internal/inventory in sencai-agent, gated behind the inventory:scan capability:

• Reads installed packages via dpkg-query (Debian/Ubuntu) or rpm -qa (RHEL/Fedora/CentOS).
• Cross-references package versions against the endoflife.date API to flag packages that are approaching or past end-of-life, classified as critical, warning, or info urgency.
• Runs on a staggered schedule: starts 10 minutes after agent boot, then repeats every 24 hours.

CIS / Lynis hardening scan

internal/cis runs:

lynis audit system --quiet --no-log --no-colors --cronjob

This requires the lynis binary to already be present on the host — if it is not installed, the scan goroutine exits silently without error (no scan result is produced). It starts 15 minutes after agent boot and then repeats every 24 hours. There is no fixed daily schedule (e.g. 02:00) and no /api/fleet/nodes/:nodeId/compliance-report endpoint — results are parsed from the Lynis stdout output (hardening score, test count, warnings) and handled internally by the agent/gateway pipeline.

Patch management

internal/patch detects the host package manager (apt, dnf, or zypper), lists available updates (Scan(), gated by capability patch_management:scan), and can install them (Apply(), gated by capability patch_management:apply). InMaintenanceWindow() parses a simplified cron-like expression (hour + day of week) to restrict Apply() to configured maintenance windows. The scan starts 5 minutes after boot and repeats every 24 hours.

Log shipping to Loki

internal/logshipper reads the systemd journal (journalctl --output=short-iso) and log files matched by configurable glob patterns (defaults: /var/log/syslog, /var/log/messages, /var/log/*.log). Before shipping, it redacts PII from each line — email addresses, IPv4 addresses, MAC addresses, credential-looking key/value pairs, and payment card numbers — then pushes batches to Loki’s push API (tagged with X-Scope-OrgID for tenant isolation). Gated behind capabilities log_shipping:system / log_shipping:app, disabled by default until log_shipping_enabled and loki_url are set in the agent config.

Prometheus Pushgateway export (opt-in)

internal/pushgateway pushes three gauge metrics (sencai_agent_cpu_percent, sencai_agent_memory_used_bytes, sencai_agent_disk_used_bytes) to a Prometheus Pushgateway on every heartbeat. It is entirely opt-in: setting the environment variable SENCAI_PUSHGATEWAY_URL is the only way to enable it. Without it, the agent behaves exactly as before, with no extra network traffic.

Runbook execution

Runbook execution is a fixed whitelist, not a generic YAML/shell-command format. The only allowed actions are:

• restart_service
• clear_disk_space
• kill_process
• run_approved_script — restricted to scripts under /etc/sencai-agent/scripts/

Dispatch flow:

1. A caller (server-side only) sends POST /dispatch-runbook to agent-gateway with the X-Service-Secret header.
2. agent-gateway pushes a WS message to the target agent: {type: "execute_runbook", execution_id, actions, dry_run}.
3. The agent (internal/runbook) enforces a rate limit of 5 executions per hour per action, tracked in memory and reset on agent restart.
4. The result is returned as a WS message {type: "runbook_result", ...} with a per-action result and an overall success flag.

There is no HTTP POST /runbook-result endpoint — the result travels back over the same WS connection.

Quarantine

Quarantine is server-initiated and OS-level — the agent does not self-quarantine. When a quarantine is triggered from the Fleet dashboard (backed by the Strapi quarantine-policy / quarantine-event content types), agent-gateway’s internal/quarantine handler runs, on the target host (Linux only, requires passwordless sudo):

sudo iptables -I OUTPUT -j DROP -m comment --comment sencai-quarantine-<agentID>

Releasing quarantine removes the tagged rule (matched by the sencai-quarantine-<agentID> comment). There is no “Action Gateway approval” step in this flow — no standalone Action Gateway service exists; see dev/gateway.md for what does.

Self-update

sencai-agent update checks for a new signed release and applies it:

• Downloaded binaries and checksums are signed with cosign sign-blob (keyless, via GitHub OIDC); the agent verifies the SHA-256 checksum before replacing itself.
• The binary swap is atomic (os.Rename), with the previous binary kept as a .bak file for rollback if the new version fails a health check.
• Rollout is staged in rings (10% / 50% / 100%) assigned deterministically from the agent’s agent_id, so a bad release affects only a fraction of the fleet before wider rollout.

Supply-chain signing

• Binaries: cross-compiled for linux/amd64 and linux/arm64, signed with cosign sign-blob (keyless, GitHub OIDC-based), alongside an SPDX SBOM generated by anchore/sbom-action.
• Packages: .deb/.rpm packages are built with nfpm and GPG-signed on tagged releases.

cosign verify-blob \
  --certificate sencai-agent-linux-amd64.pem \
  --signature sencai-agent-linux-amd64.sig \
  sencai-agent-linux-amd64

Kubernetes fleet agent

k8s-agent is a separate, deployment-only repository — it contains no Go source. It ships a Helm chart (helm/sencai-agent/) plus a flat manifest, both of which deploy the same prebuilt image, ghcr.io/sencai/sencai-agent:dev-latest, published from the sencai-agent repo’s release pipeline.

It is not a “DaemonSet variant of the same binary” going through agent-gateway — Kubernetes clusters use an entirely separate path. The in-cluster agent reports read-only cluster metrics (node/pod/namespace counts, cluster version) every 30 seconds via:

POST /api/k8s-agents/<id>/heartbeat
X-K8s-Agent-Token: <token>

directly to Strapi. It never talks to agent-gateway. Configuration (sencai.apiUrl, sencai.registrationToken, sencai.organisationId) is supplied through Helm values.yaml or the flat manifest’s ConfigMap/Secret.

Troubleshooting

Agent not appearing in Fleet dashboard after enrollment

Confirm the gateway URL is reachable from the host and that the sencai-agent Strapi record is still pending (an already-used or expired token returns 401 from /enroll). Review agent logs:

journalctl -u sencai-agent -n 50

mTLS handshake failure / certificate issues

Re-run enrollment to obtain a fresh certificate:

sencai-agent enroll --gateway <url> --token <new-token>

The sencai-agent CLI currently only exposes --gateway and --token on enroll — there is no documented --force flag; confirm with the service owner before relying on one.

Lynis scan never produces a result

The lynis binary must already be installed on the host — internal/cis does not install it, and silently skips the scan if it’s missing.