How storage works

Fanout retains telemetry as ordinary Parquet files in self-contained atomic batch directories. Persistent sidecars index traces, and DuckDB handles SQL, broad scans, and rebuildable analytical rollups in the same process. Application state—users, sessions, dashboards, alert rules, and agent history—lives in SQLite.

Component	Purpose
Atomic Parquet batches	Authoritative retained spans, logs, and metrics
Trace sidecar	Persistent row-range index for targeted trace reads from Parquet
DuckDB	SQL, log filtering, broad analytical reads, and rebuildable rollups
SQLite	Transactional application and identity state

Parquet is authoritative; indexes and DuckDB rollups are derived state.

Writes are durable immediately

Concurrent small OTLP requests may share one group-commit batch. Up to four commit workers encode independent batches in parallel. Each worker writes all present signals and metadata into a staging directory, adds trace.fidx when spans are present, fsyncs them, and publishes the complete directory with one atomic rename. Fanout returns success only after every batch belonging to the request is published.

A crash before the rename leaves only an unacknowledged staging directory, which startup removes. A crash after the rename leaves a complete batch that startup discovers directly from the filesystem. The atomic directory publish is both the durable commit and the unit startup discovers; acknowledgement occurs only after publication and parent-directory fsync complete.

One authoritative copy for logs and metrics

Every signal is written once, to typed Zstandard-compressed Parquet. DuckDB applies filters, ordering, LIMIT, and aggregation inside its vectorized scan. Trace lookup binary-searches a fixed-width sidecar on disk, seeks directly to the matching Parquet row ranges, and verifies the complete trace ID after hashing. The sidecar is an index, not another telemetry representation.

Dashboard rollups are rebuildable DuckDB caches. SQLite stores only transactional product state and never holds telemetry rows. Ingest authentication still reads the ingest token hash from it on every export, so an unreadable control database rejects OTLP requests.

Attributes retain their types

Attribute and resource values support up to 128 nested arrays or objects. An export that exceeds this limit is rejected in full with OTLP INVALID_ARGUMENT (HTTP 400); accepted values retain their original types.

Span, log, and metric attributes and resource attributes use Parquet VARIANT columns. Frequently queried resource and messaging fields also have typed Parquet columns inside the VARIANT encoding. The span read view exposes messaging_system and messaging_destination directly from those shredded fields, so messaging rollups can avoid decoding the whole attributes object. Values with different types remain in the residual encoding; an integer does not become a string just because another row uses one.

Integers, booleans, binary values, nulls, arrays, and nested objects survive ingestion, indexed reads, and compaction. Event timestamps retain nanosecond precision as UTC instants (TIMESTAMPTZ_NS). Keep timestamp columns unchanged in filters so Parquet row-group statistics can prune the scan. For datetime functions that require a timestamp without a timezone, cast their argument to TIMESTAMP_NS; the JSON result boundary handles returned timestamps automatically. Attribute names such as service.name are literal keys. In SQL, use attr(attributes, 'service.name') and an explicit cast when a scalar type is needed, for example TRY_CAST(attr(attributes, 'retry.count') AS BIGINT). General attribute extracts through views retain their types, but this engine preview does not propagate their extraction across view projections. Prefer the existing scalar service, namespace, version, environment, and messaging columns for those hot queries.

The JSON SQL response converts nested values only when returning results. Integer tokens retain their full precision; clients that parse JSON numbers as floating-point values should cast large integers to strings in their query. Non-finite floats are returned as "NaN", "Infinity", or "-Infinity". Binary values stay binary in storage; cast them explicitly to the desired string encoding when querying them for a JSON client.

Engine and format are pinned

Fanout embeds DuckDB v2.0.0-alpha43763, a preview engine, with its extensions compiled into the binary. It downloads no extensions at runtime. Native source builds use scripts/with-duckdb.sh, which checks the platform archive and uses the matching engine headers; the ordinary just build and test recipes invoke it automatically. DuckDB 2’s asynchronous I/O uses its default memory-governed read-ahead rather than an unlimited queue. Rooted service traversal uses keyed recursion with explicit hop, accumulated-node, and execution-time limits.

Offline verification decodes every Parquet column with the native engine and separately checks physical schemas, span ordering, row counts, and trace-index ranges. It reuses one bounded in-memory engine and creates no spill files. Discarded hashes force decoding; they do not establish a stored checksum. Verification failures such as memory exhaustion or cancellation never authorize quarantine.

Telemetry batches use format 4. An unsupported batch format makes startup fail before cleanup or schema rewriting, leaving the files intact.

Small files are bounded by compaction

Atomic ingestion creates immutable batch directories. Maintenance drains every eligible compaction group, choosing inputs by both count and a 25-million-row ceiling. It combines same-day, same-generation files, builds a replacement trace index, and atomically swaps the replacement for its inputs. A durable compaction marker makes an interrupted swap resumable. Active DuckDB readers pin immutable files while retention or compaction removes them, so reads never lose an open input file.

The expensive merge runs before readers are excluded. Only the bounded directory swap takes the exclusive reader gate; new ingest batches continue encoding while that work is prepared.

Retention is based on ingestion time rather than event time. A service with a bad clock therefore cannot pin a batch on disk by emitting a far-future event.

Dashboard reads pin completed batches

Dashboard log, metric, and span scans select immutable files from their actual minimum and maximum event timestamps. Bounds come from every Parquet row-group footer, including negative timestamps. Files without complete statistics remain eligible. Late-arriving telemetry is selected by its event time, independently of its ingestion timestamp. Arbitrary SQL retains access to all active files.

Endpoint panels and log counts scan footer-pruned Parquet for the exact requested window. Text searches match the redacted body. Notable-trace selection keeps one indexed candidate per trace, updated from new batches. A candidate records whether its namespace and service are unambiguous; mixed scopes use exact per-batch contributions for the requested scope. Error priority, duration, and trace-ID tie ordering are preserved. Boundary trace parts are recomputed from Parquet.

These caches acknowledge complete batch IDs rather than an ingestion watermark. A batch’s trace bounds, candidate updates, and acknowledgement commit in one transaction. A read pins active files and reads cache rows and acknowledgements from one DuckDB transaction. Newly published uncached files are read directly, so cache backfill does not delay visibility. Retired batch contributions are excluded by active ID even before cleanup; they cannot invalidate unrelated traces. Compaction derives the output trace contributions from complete cached inputs. If an input is not yet cached, the output remains directly queryable until its own backfill.

The private cache stores per-batch trace parts and one incremental candidate per trace. It does not copy every span or log timestamp or maintain four global trace indexes. Complete cached compaction replacements preserve candidate bounds without recomputation. Retention and incomplete replacements recompute only affected trace IDs from active contributions. A semantic version covers the cache schema and aggregation rules; a version change atomically discards all private read tables and rebuilds from authoritative format-4 Parquet. The background worker processes bounded groups; fanout_read_cache_pending_batches reports its trace backlog independently of the service and edge rollups’ safety window. Completed-batch and analytical writers use separate gates and two connections outside the read pool. They modify disjoint tables; maintenance holds both gates for checkpointing. Trace reads avoid materializing every retained candidate into a temporary relation for each request. A snapshot-local ambiguity probe keeps unused batch-parts aggregation out of single-scope read plans.

Narrow file selections use explicit lists from footer bounds. Broad selections use a glob with a captured-batch-ID filter, which excludes files published after snapshot capture. Only signals referenced by a query are bound.

FANOUT_ROLLUP_SKIP_TO_LATEST skips historical service and edge aggregation. Completed-batch trace caches still rebuild retained history in bounded passes; uncached files remain immediately queryable.

Rollups lag deliberately

Overview and alert queries read rebuildable DuckDB rollups. Their watermark trails committed telemetry by a fixed safety window so out-of-order events from bounded commit retries are not skipped. Raw committed telemetry is queryable before it appears in a rollup.

Service dependency traversal reads the complete edge rollup within the selected window, including edges outside the topology view’s displayed page. It follows upstream or downstream edges from a named service, retains the minimum hop count per namespace and service, and bounds depth, accumulated nodes, and execution time. A truncated result states which limit was reached. Rollup lag still applies; dependency edges describe observed calls, not proof of causality.