How storage works

Fanout retains telemetry as ordinary Parquet files in self-contained atomic batch directories. Persistent sidecars index traces, and DuckDB handles SQL, broad scans, and rebuildable analytical rollups in the same process. Application state—users, sessions, dashboards, alert rules, and agent history—lives in SQLite.

Component	Purpose
Atomic Parquet batches	Authoritative retained spans, logs, and metrics
Trace sidecar	Persistent row-range index for targeted trace reads from Parquet
DuckDB	SQL, log filtering, broad analytical reads, and rebuildable rollups
SQLite	Transactional application and identity state

Parquet is authoritative; indexes and DuckDB rollups are derived state.

Writes are durable immediately

Concurrent small OTLP requests may share one group-commit batch. Up to four commit workers encode independent batches in parallel. Each worker writes all present signals and metadata into a staging directory, adds trace.fidx when spans are present, fsyncs them, and publishes the complete directory with one atomic rename. Fanout returns success only after every batch belonging to the request is published.

A crash before the rename leaves only an unacknowledged staging directory, which startup removes. A crash after the rename leaves a complete batch that startup discovers directly from the filesystem. The atomic directory publish is both the durable commit and the unit startup discovers; acknowledgement occurs only after publication and parent-directory fsync complete.

One authoritative copy for logs and metrics

Every signal is written once, to typed Zstandard-compressed Parquet. DuckDB applies filters, ordering, LIMIT, and aggregation inside its vectorized scan. Trace lookup binary-searches a fixed-width sidecar on disk, seeks directly to the matching Parquet row ranges, and verifies the complete trace ID after hashing. The sidecar is an index, not another telemetry representation.

Dashboard rollups are rebuildable DuckDB caches. SQLite remains independent and stores only transactional product state.

Small files are bounded by compaction

Atomic ingestion creates immutable batch directories. Maintenance drains every eligible compaction group, choosing inputs by both count and a 25-million-row ceiling. It combines same-day, same-generation files, builds a replacement trace index, and atomically swaps the replacement for its inputs. A durable compaction marker makes an interrupted swap resumable. Active DuckDB readers pin immutable files while retention or compaction removes them, so reads never lose an open input file.

The expensive merge runs before readers are excluded. Only the bounded directory swap takes the exclusive reader gate; new ingest batches continue encoding while that work is prepared.

Retention is based on ingestion time rather than event time. A service with a bad clock therefore cannot pin a batch on disk by emitting a far-future event.

Rollups lag deliberately

Overview and alert queries read rebuildable DuckDB rollups. Their watermark trails committed telemetry by a fixed safety window so out-of-order events from bounded commit retries are not skipped. Raw committed telemetry is queryable before it appears in a rollup.