Set up alerts

The alert engine runs inside the binary and is on by default. It evaluates persisted rules every FANOUT_ALERTS_EVALUATION_INTERVAL (30s) against service rollups, and delivers by webhook.

Rules are managed over HTTP. There is no browser page for them yet — see capabilities.

Endpoint	Does
GET /api/alerts	Current alert state
GET /api/alerts/summary	Counts by state
GET /api/alerting/rules	List rules
POST /api/alerting/rules	Create and compile a rule
PUT /api/alerting/rules/:id	Replace a rule; recompile it when enabled

Reading takes telemetry:read; creating and replacing takes alerts:manage, which starts at the operator role. See roles.

What a rule contains

An expression, an optional service to scope it to, timing, and delivery:

Field	Means
name	Required rule name
enabled	Set true to evaluate the rule; omitted means disabled
expression	The condition, in CEL
service	Evaluate only for this service; omit for all
for_seconds	How long the condition must hold before firing
cooldown_s	Minimum quiet period after resolving before firing again
repeat_interval_s	How often to re-deliver while still firing
webhook_url	Where to deliver
webhook_headers	Extra headers, for a shared secret or routing key
webhook_template	Custom payload body; omit for the default
notify_on_resolve	Also deliver when the alert clears

Nonempty expressions supplied when creating or replacing a rule are type-checked against the declared variables and must return a boolean. An unknown variable or a type mistake is rejected at save time. Supply a complete rule body when replacing it; omitted fields take their zero values.

Variables available to an expression

An expression reads one service’s rollup for the current window. The full list — error_rate, p95, throughput, the baseline deltas and the rest — is generated from the evaluation environment itself, so it cannot drift from what the evaluator actually binds.

So a rule body reads like the thing it means:

error_rate > 0.02 && throughput > 10

The throughput clause matters more than it looks. Here throughput is the total span count in the current rollup window, not requests per second. Without a traffic floor, a single failure on a quiet service can trigger the rule.

The lifecycle

An alert moves through three states:

• pending — the condition is true but has not held for for_seconds yet. With for_seconds: 0 a rule skips this and fires immediately.
• firing — it has held. The webhook is delivered, and re-delivered every repeat_interval_s while it stays true.
• resolved — the expression went false. If notify_on_resolve is set, that is delivered too.

cooldown_s then holds the rule quiet for that long after resolving, which is what stops a metric hovering at the threshold from delivering repeatedly.

Each alert records its last delivery status and timestamp, so a webhook endpoint that started rejecting is visible in GET /api/alerts rather than silent.

The namespace field on a rule is not applied

A rule carries a namespace selector and it is stored, but evaluation reads instance-wide service rollups and does not filter on it. A rule you believe is scoped to one namespace is evaluating across all of them.

Scope with service instead, which is applied. This is a real gap rather than a subtlety — it is marked here because the field’s presence implies otherwise.

Why alerts lag the raw data

Evaluation reads rollups, not raw telemetry, and rollups deliberately trail the newest data by a safety margin because telemetry arrives out of order. An alert therefore fires seconds after the condition is visible in a raw query. That is a correctness choice — how storage works explains the margin — and it means for_seconds should not be set below the rollup interval expecting finer resolution than the data has.