Troubleshoot

The process will not start

Fanout validates its whole configuration before opening any listener, so a configuration refusal names the setting. Other startup errors can report storage, listener, or identity-provider failures; read the error before changing settings.

Unknown FANOUT_ setting names fail startup, except platform-injected service discovery names. Check spelling against the configuration reference.

An unresolved compaction blocks startup

One startup refusal is not about configuration. If a compaction was interrupted and Fanout can neither finish nor undo it, recovery fails closed and the error names telemetry/COMPACTION.json.

The marker identifies the authoritative batch set during a swap. The replacement is staged under telemetry/compaction/ or published as telemetry/parquet/batches/<output id>.batch. Retired inputs are named <id>.retired-<output id> under telemetry/parquet/batches/. A partial publish can remove some retired inputs; verify every input before attempting a rollback.

Clear the underlying cause, such as a full disk, and start Fanout again. Recovery runs automatically at startup.

If recovery keeps failing, roll the compaction back by hand. With the process stopped, use telemetry/parquet/batches/ as the batch directory:

1. Read telemetry/COMPACTION.json. It is JSON, with an output object and an inputs list of batch ids.
2. Back up telemetry/, then check every id in inputs. Each must exist in exactly one form: <id>.batch (never retired) or <id>.retired-<output id> (retired, restorable). If an id exists in both forms or neither form, stop here. Both forms make the authoritative copy ambiguous; neither means its rows may survive only inside the replacement. Reconcile the batch from the backup, or complete the compaction instead of rolling it back.
3. Rename each <id>.retired-<output id> back to <id>.batch.
4. Delete telemetry/compaction/<output id> if it exists.
5. Delete telemetry/parquet/batches/<output id>.batch if it exists. Leaving a published replacement beside the restored inputs duplicates its rows; a corrupt replacement continues to block startup.
6. Delete telemetry/COMPACTION.json.
7. Start Fanout.

Verify inputs before deleting the replacement

Restore the retired inputs before removing the compaction marker. The marker protects those directories from startup cleanup. Keep the replacement until every input is restored; it may hold the only surviving copy of some rows.

An unreadable telemetry batch blocks startup

Fanout also refuses to start when an authoritative .batch directory has invalid metadata, a corrupt Parquet file, a missing signal file, or a damaged trace index. First stop Fanout and back up telemetry/, then verify the full authoritative set:

# Use the same environment and data directory as the server.
fanout repair verify

The command is read-only and reports confirmed corrupt batches. Unsupported formats and operational errors stop verification; they do not authorize quarantine. Restore each reported corrupt batch from backup when possible. If one cannot be recovered and discarding only its telemetry is preferable to leaving the instance offline, set it aside explicitly:

fanout repair quarantine --batch <id>

Quarantine is an atomic rename beside the authoritative batch set, not a deletion. It refuses a batch that validates successfully and refuses any input or output protected by a live compaction marker. Preserve the reported quarantine directory for later recovery, run repair verify again, and only then restart Fanout.

Quarantine makes those rows unavailable

The command is intentionally batch-specific and never runs during startup. Every row in the quarantined directory disappears from queries until the batch is restored, so do not use it as a substitute for fixing a transient disk or permission error.

An exporter is rejected

Work down this list in order:

1. Has setup finished? Before the first administrator exists there is no ingest token to check against, and ingest rejects everything. A collector started ahead of setup fails until you complete it.
2. Is it the right credential, in the right header? The ingest token is fo_-prefixed and goes in Authorization: Bearer. Browser sessions and MCP tokens are not ingest credentials.
3. Is the listener reachable? Both OTLP transports use FANOUT_ADDR (127.0.0.1:7520 by default; the image sets :7520). Check container port mappings, bind address and firewall rules.
4. Right protocol? Both use port 7520. gRPC requires HTTP/2 through any proxy. Use grpc or http/protobuf consistently with the exporter configuration; HTTP/1.1 proxy forwarding cannot carry gRPC.

For OTLP/HTTP, set the exporter protocol to http/protobuf and give it the base endpoint, http://fanout:7520. Exporters derive /v1/traces, /v1/metrics and /v1/logs from it themselves.

Someone cannot sign in and SMTP is not configured

SMTP is optional. Creating a local user without it succeeds and reports invite_delivery: "not_configured" with login_link_required: true. An operator with shell access to the same configuration and data directory then mints a link directly:

fanout login-link [email protected]

The link expires after 15 minutes, works once, and is recorded in the authentication audit history. It is available in local auth mode only.

A configured SMTP relay that fails delivery returns an error. See authentication operations for invitation responses and account recovery.

Preserve the account

Deleting a user row cascades to sessions, OAuth grants and owned dashboards. Use a login link for an active local account.

Telemetry arrives but queries look empty

Check the namespace. Fanout reads service.namespace from the OTLP resource and falls back to FANOUT_DEFAULT_NAMESPACE, so telemetry exported without one lands in default.

An unscoped query spans every namespace, so this is only a problem when something has scoped the query, through a dashboard filter or an MCP call passing namespace. Clear the scope before concluding the data is missing.

The alert engine and overview read analytical rollups, which follow durable publication. Raw queries can see committed rows before the next rollup pass. See rollup timing.

What to watch in production

The endpoint reference lists health probes, scrape credentials and signals to monitor.