Run Thanos: monitor everything, know first, fix fast.

Deploy the platform

Docker Compose is the supported path. You need Docker 24+ / Compose v2 and a TLS proxy in front for production.

1. Prepare secrets. Copy the env templates and fill them: deploy/backend.env and deploy/postgres.env. Generate strong keys:

   # 32-byte hex for encryption + agent pepper, 64-byte for JWT
   openssl rand -hex 16   # ENCRYPTION_KEY, AGENT_TOKEN_PEPPER
   openssl rand -hex 32   # JWT_SECRET

   Production startup refuses to boot on weak or missing secrets.
2. Bring the stack up.

   docker compose -f docker-compose.prod.yml up -d --build

   Migrations apply automatically on backend startup.
3. Reach it. The frontend (UI + /api proxy) is on http://localhost:8088; the backend that agents connect to is on http://localhost:8090.
4. Serve under your domain. Front the frontend container with a TLS proxy (nginx/Caddy/traefik) and set both CORS_ORIGINS and WEBSOCKET_ALLOWED_ORIGINS to your origin. In production the backend refuses to start with * or empty WebSocket origins.

Health endpoints: GET /api/v1/live (liveness), /api/v1/ready (readiness incl. dependencies), /api/v1/health (per-service detail), /api/v1/metrics (Prometheus).

Accounts & sign-in

1. Create the first account. Open the app and choose Sign up. Enter name, email, and a password of at least 8 characters. You're signed in and land on the dashboard.
2. Sign in later with email + password. Sessions persist across reloads.
3. Google sign-in (optional). The Google button only appears when a client ID is configured, so an unconfigured deploy never shows a broken button. To enable it, see Configuration reference → Google sign-in.

The dashboard

The landing page summarizes your world: counts of endpoints by status, open incidents, and the most recent activity. Use it as the morning glance; drill into Incidents or Endpoints for detail. Status is encoded as a pill everywhere: UP DEGRADED DOWN.

Endpoint monitoring

Monitors HTTP / HTTPS / TCP targets with configurable checks, SSL tracking, and multi-region rollup.

1. Go to Endpoints → Add Endpoint.
2. Enter a Name and the URL (e.g. https://api.example.com/health). Pick the Protocol.
3. Set the Check interval (seconds between probes) and Timeout.
4. Optional: set an Expected status code and/or Expected body substring, custom headers, request method and body, and accepted status codes (e.g. 2xx, 3xx).
5. Optional Authentication: none, basic, or bearer token — credentials are encrypted at rest.
6. Choose the Check mode (see below) and, for agent mode, the agent that will run it.
7. Save. The endpoint begins checking on its interval; its status appears within one cycle.

An endpoint that starts failing accrues consecutive failures; once past the threshold it flips DOWN and opens an incident. When it recovers, the incident resolves and a recovery notification is sent through the same channels.

Muting & maintenance

On an endpoint's edit page, the Alert Suppression card lets you mute all alerts, set a maintenance window (Muted until), or suppress specific incident types (e.g. only SSL_EXPIRY). Muted endpoints are still checked; they just don't dispatch alerts.

Multi-region & availability policy

Attach regional targets so several agents check the same endpoint from different regions. The Availability policy then decides the rolled-up status: any (one region UP → UP), all (every region must be UP), or quorum (at least K regions UP). A mix of UP and failing regions reads as DEGRADED.

Check modes: central vs agent

SSL / certificate tracking

For HTTPS endpoints, Thanos reads the certificate from the TLS handshake on every check and records the issuer and expiry. When the certificate nears expiry it opens an SSL_EXPIRY incident. Configure the warning windows per endpoint under Cert expiry thresholds (days), or fall back to the global default.

Latency & anomaly detection

Two independent ways to catch an endpoint that's up but slow.

Latency threshold (deterministic)

On the endpoint's edit page set a Latency threshold (ms). When sustained average latency crosses it, Thanos opens a DEGRADED incident — no baseline required, so it works on brand-new endpoints. Leave it empty to disable.

Anomaly detection (statistical)

Enable Anomaly detection to catch deviations from the endpoint's own learned baseline. It compares the current 5-minute bucket to a seasonal (hour-of-day) baseline and fires when the z-score exceeds your threshold for two consecutive ticks.

• Z-score threshold — standard deviations from the mean before alerting (default 3.0).
• Baseline window — hours of history used to learn normal (default 24).

Endpoint groups

Groups organize endpoints for status pages and SLOs. Create one under EP Groups, then assign a group to an endpoint from its edit page. A group is the unit an SLO target is measured over and a status page is built from.

Agents & regions

An agent runs near your infrastructure to check endpoints from a region and collect server metrics. Setting one up is two steps: register it, then run the binary with its token.

1. Register the agent. Go to Agents → Add Agent, give it a name and one or more regions (e.g. sgp, eu). Save.
2. Mint a token. Open the agent and create a token. Copy it now — it's shown once. Optionally scope it read-only or to specific CIDRs.
3. Run the agent. On the host, run the binary (or container) pointed at your backend:

   docker run -d --name thanos-agent \
     -e AGENT_API_URL=https://your-domain \
     -e AGENT_TOKEN=<token-from-ui> \
     -e AGENT_REGION=sgp \
     thanos-agent

   Or as a native binary (better for host-level server metrics):

   AGENT_API_URL=https://your-domain \
   AGENT_TOKEN=<token> \
   AGENT_REGION=sgp \
   AGENT_STATE_DIR=/var/lib/thanos-agent \
   ./thanos-agent

4. Confirm it's online. Within a minute the agent shows Online in Agents with a fresh heartbeat, IP, and version.

Assign agent-mode endpoints (and servers) to the agent; it fetches its work list, probes on schedule, and streams heartbeats back. The rebalancer spreads endpoints across healthy agents and can hand a region's work to a fallback agent if the primary dies.

eBPF service metrics

Optional. Capture RED metrics (request rate, errors, duration) plus TCP and resource-pressure metrics straight from the kernel — no code changes to the monitored apps.

A separate node-agent loads eBPF programs on a Linux host and exposes the collected metrics as Prometheus text. Your Thanos agent forward-scrapes that endpoint and ships the samples to the backend, where they're stored in a queryable time-series store.

1. Run the node-agent on the host (needs a recent kernel with BTF and CAP_BPF/root):

   ./thanos-node-agent --listen=127.0.0.1:19100

   It attaches probes and serves metrics at /metrics — HTTP durations, TCP connects/retransmits, CPU/IO/memory pressure, and more.
2. Point a Thanos agent at it by adding two variables and restarting the agent:

   AGENT_EBPF_METRICS_URL=http://127.0.0.1:19100/metrics
   AGENT_EBPF_INTERVAL=15s

   The agent forward-scrapes on the interval and submits the exposition to the backend.
3. Verify. Samples begin landing in the metric store; the container-level RED metrics become available for the service map and analysis.

Server monitoring

Track host CPU / memory / disk and up-down status over SSH or via an agent.

1. Go to Servers → Add Server. Enter a name and hostname/IP.
2. Provide access: an SSH username with a saved key or password (or an assigned agent). For a private host, route through a bastion.
3. Pick a Health check method — ssh, http, telnet, or none. When you assign an agent, the form defaults the method to SSH so the server is actually monitored.
4. Save. Status flips to Online once the first check succeeds.

Bulk import / export

Import many servers from CSV (username + password creds; connectivity is probed, capped at 500 rows, private-range SSRF-guarded). Export writes a CSV; SSH passwords are omitted unless you pass ?includeCredentials=true, and every cell is neutralized against spreadsheet formula injection.

SSH keys & bastions

SSH Keys are reusable private keys stored encrypted at rest (with optional passphrase). Reference a saved key from a server or bastion instead of pasting it each time; a key that's in use can't be deleted.

Bastion Hosts are jump hosts for reaching servers on private networks. Add a bastion once, then select it on any server that lives behind it — Thanos dials the two hops with host-key pinning on both.

Web terminal

Open an interactive SSH session to any server from the browser under Terminal. Sessions are authenticated, ownership-checked before dialing, and host-key pinned. Open multiple tabs; they persist in the session and surface a green SSH badge on the servers list. Run a saved script on a server: Thanos uploads it over SSH to /tmp and executes it, capturing the output.

Kubernetes

Discover cluster resources, visualize them as a graph, and get advisor findings.

1. Go to Kubernetes → Add Cluster. Provide access — paste a kubeconfig, or client-certificate data, and the API server URL.
2. Assign the cluster to an agent that can reach the API server, and enable Discovery.
3. Save. The cluster shows Connected. On the next discovery tick the agent enumerates nodes, pods, deployments, services, ingresses and more, and syncs them back.

Resource graph

Open K8s Graph (or View Graph on a cluster) and pick a namespace to see an interactive node-link graph — ownerRef, Service-selector, Ingress-backend, and volume/PVC↔PV edges across the discovered kinds.

Advisor

K8s Advisor runs checks against a connected cluster and reports issues (or confirms all checks passed). Trigger a fresh analysis with Analyse Now.

Incidents

An incident is one open problem. It opens automatically when detection crosses a threshold, or you can create one manually against an endpoint or server.

Lifecycle

1. Open. Detection fires; the incident lists the endpoint, type, severity, and failing regions.
2. Acknowledge. On the incident page, click Acknowledge to signal you're on it (stops escalation from advancing).
3. Resolve. When fixed, click Mark resolved. Recovery from monitoring auto-resolves it and clears the dedup window.

Grouping & flapping

A burst of related failures is grouped under one parent incident so you get one alert, not a storm. An endpoint that rapidly flips up/down is classified FLAPPING, which suppresses per-transition noise; it auto-resolves after a stable cooldown.

Notification channels

Where alerts go. Set up a channel once, then route rules to it.

1. Go to Channels → Add Channel, choose the type, and paste the webhook URL / credentials.
2. Save, then click Test on the channel row — Thanos sends a sample alert through the real sender so you can confirm delivery before a real incident depends on it.

Alert rules

A rule decides which channel an incident is dispatched to, and at what severity. Rules apply globally across your endpoints; an endpoint can suppress rules it doesn't want.

1. Go to Alert Rules → Add Rule. Name it and choose the notification channel it routes to.
2. Set the severity and any per-channel routing. Save.

A rule may only reference your own channels; the dispatcher additionally refuses to deliver through a channel owned by another tenant.

Escalation & on-call

Make sure an unacknowledged incident reaches someone — climbing a ladder of steps on a schedule.

Escalation policies

1. Go to Escalation Policies → New Policy. Add ordered steps. Each step has a delay (minutes after the incident opened) and an action: notify a channel, a specific user, or the current on-call.
2. Attach the policy to an endpoint (from the endpoint's edit page).

The engine polls open incidents every 60 seconds. Step 0 with delay 0 fires on the first poll after the incident opens; a step with delay N fires ~N minutes in — unless the incident is acknowledged first. A claim-before-send guard means each step fires exactly once, even across replicas.

On-call schedules

Under On-Call Schedules build a weekly rotation of shifts. An escalation step that targets on-call resolves the person currently on shift and notifies them. The calendar view shows the rotation at a glance.

Scripts

A script is reusable shell (or other) code you can run on the backend host or on a server over SSH, and link into runbook steps.

1. Go to Scripts → Add Script. Name it, choose the language, paste the content, set a timeout.
2. Run / Logs to execute it. In the Run modal choose the target — the backend host or a specific server (SSH) — and Thanos captures and persists the output.

Runbooks

A runbook is an ordered list of steps (each backed by a script) that runs to diagnose or fix a problem — manually or automatically.

1. Go to Runbooks → Add Runbook. Name it.
2. Add steps: pick a script for each, order them, and optionally mark a step require approval so execution pauses there until a human approves.
3. Run it: on the runbooks list click the ▶ action. In the Run modal choose the target, optionally link an incident, and toggle Dry run to log "would execute" without running anything — validate the runbook safely first.

Safety guardrails

Auto-remediation

Let a runbook run automatically when a matching incident opens — safely, on the affected server.

1. Open the runbook and enable Auto-trigger. Choose the incident type that triggers it (e.g. DOWN, DEGRADED).
2. Set the Remediation target. Pick Server (SSH) and select the server the fix should run on. (Backend host is deny-by-default in production, so a server target is what actually works there.) If you leave it on Backend, the incident's own server is used when it has one.
3. Save. When a matching incident opens, Thanos resolves the target, runs the runbook over SSH, and — if the fix works — the endpoint recovers and the incident resolves.

Target resolution order: the runbook's explicit server → the incident's own server → the backend host (fail-closed in production). Guardrails that ride along: the server must belong to you (re-verified at trigger time), the circuit breaker is keyed on the target so a shared server can't be hammered, and a single-flight lock prevents duplicate runs for one incident.

Root-cause analysis

Open an incident and Thanos assembles an RCA: hypotheses with supporting evidence, a dependency chain from the entity graph (deepest root wins), and signals pulled from your log/metric stores (Prometheus, Sentry, OpenSearch, SigNoz). When a failure cascades, the incident is attributed to its cause with a root-cause badge. RCA is display-only — it explains, it never acts on its own.

Analytics & SLI

The Analytics view reports coverage-aware service-level indicators: availability decomposed into good / bad / observed / expected events with an explicit NODATA state, plus latency SLIs (p95 per bucket over pure-UP samples). Coverage below 50% reads as UNKNOWN rather than a fabricated number — the platform would rather say "not enough data" than guess.

SLO targets

Set an objective for a group and track its error budget and burn rate.

1. Go to SLO and create a target for an endpoint group.
2. Pick the SLI kind (availability or latency), the objective (e.g. 99.9%), the window (7/30/90 days), and — for latency — the threshold. Optionally route burn alerts to a channel.

The budget read-model decomposes the objective into an expected denominator and remaining budget; coverage below 50% reports UNKNOWN. The burn engine watches the 6-hour burn rate on a 5-minute tick and, with two-tick hysteresis, opens a SLO_BURN_TICKET (fast burn) or critical incident (budget nearly exhausted).

SLA contracts

An SLA wraps an existing SLO target in a customer contract: a frozen target percentage, a calendar period, and a credit schedule. Downtime is derived from the incident lifecycle (not raw region events); append-only exclusion windows and downtime intervals are intersected for the period. Reports are frozen and immutable once generated, with append-only amendments; when coverage is under 50%, a report reads INSUFFICIENT_COVERAGE rather than inventing a number.

Status pages

A public page that shows the health of the monitors you choose.

1. Go to Status Pages → Add Status Page. Name it, set a slug, and choose Public or private.
2. Add the endpoint groups to display.
3. Share /public/status/<slug>. It exposes only published data — no user IDs or emails, and incidents on operator-hidden groups are filtered out.

Observability integrations

Connect external systems as metric and log sources for server metrics and RCA. Under Observability → Add Integration configure a Prometheus, Sentry, OpenSearch, or SigNoz connection, then Test connection. SigNoz is the metrics source for server/node CPU-memory enrichment and the K8s advisor; Sentry and the others feed RCA evidence collection. Credentials are encrypted and every outbound call is SSRF-guarded.

Settings & RBAC

Under Settings set platform defaults — default check interval, accepted statuses, and more. Admins manage users and roles under Admin → Users; role-based checks gate privileged actions (a role-denied action returns not-found for anti-enumeration). AI Providers optionally enables display-only AI incident summaries — the AI layer is off unless configured and can never execute anything.

Configuration reference

Key backend environment variables. Set them in deploy/backend.env or a compose override.

Google sign-in

1. In Google Cloud Console add your deploy's exact origin to the OAuth client's Authorized JavaScript origins.
2. Bake the (public) client ID into the frontend at build time:

   VITE_GOOGLE_CLIENT_ID=<id>.apps.googleusercontent.com \
     docker compose -f docker-compose.prod.yml up -d --build frontend

3. Set GOOGLE_CLIENT_ID on the backend for token verification. Only the client ID is needed — no secret is stored.

Scaling, TLS & backup

Scale the backend

Engines are leader-locked (Postgres advisory locks) and heartbeat consumption is shared (NATS durable pull), so multiple replicas are safe:

docker compose -f docker-compose.prod.yml up -d --scale backend=2

With more than one replica, front them with the frontend container or an external load balancer (one host port can't map to two containers).

TLS & WebSockets

Terminate TLS at your proxy and forward to the frontend container with WebSocket upgrade headers and a long read timeout (the SSH terminal and heartbeats use long-lived WS). Raise client_max_body_size to at least 64m.

Backup & restore

1. Back up TimescaleDB (all platform state) daily and store deploy/*.env secrets in your secret manager:

   docker compose -f docker-compose.prod.yml exec timescaledb \
     pg_dump -U postgres -Fc thanos > thanos-$(date +%F).dump

2. Restore into a running stack and restart the backend:

   docker compose -f docker-compose.prod.yml exec -T timescaledb \
     pg_restore -U postgres -d thanos --clean --if-exists < thanos-YYYY-MM-DD.dump

Redis and NATS are disposable (dedup cache and in-flight heartbeats); losing them costs at most one dedup window. Migrations are forward-only and auto-apply on upgrade — take a DB backup before upgrading.