Streaming OLAP: The Post-Kafka Stack for Real-Time Analytics

What "streaming OLAP" was, 2018 to 2024

For six years the answer to "we need real-time analytics" was a four-box whiteboard diagram. A Kafka topic ingested events from the source system. A Flink job consumed that topic, doing windowed aggregations, joins, and schema enrichment. An OLAP sink — ClickHouse, Pinot, or Druid — caught what Flink wrote. A dashboard or API layer read from the sink at sub-second latency.

The architecture was correct. It was also expensive in ways nobody priced honestly. Three teams — Kafka platform, Flink platform, database. Three sets of state to reconcile (broker offsets, Flink checkpoints, sink commit cursors). Three schema languages to keep in sync. When latency regressed, debugging meant tracing one event across three operational systems whose logs lived in three different observability stacks.

The part everyone undercounted: the architecture isn't expensive because any single box is hard. It's expensive because the seams between boxes are where the bugs live. A Flink job that drops one in ten thousand events is invisible at the box level and catastrophic at the SLA level. The 2018-2024 era shipped this stack anyway because the alternative was a nightly batch job. The option set was empty.

The three forks in 2026

By mid-2026 the option set isn't empty anymore. Three credible architectures have emerged, each collapsing a different seam.

Fork 1 — Single-engine streaming SQL

The most aggressive collapse deletes two of the three boxes. RisingWave, Materialize, and the newer Apache Fluss are variations on the same bet: build a single streaming database that ingests from sources, maintains materialized views incrementally, and serves queries over the Postgres wire protocol. Source → materialized view → query. One system, one schema language.

RisingWave's pitch is the cleanest. You write CREATE MATERIALIZED VIEW ... AS SELECT ... over a Kafka source or a Postgres CDC stream, and the engine maintains the view incrementally — new records flow through, the view updates in milliseconds, and any client speaking Postgres can SELECT * FROM the_view and get a current answer (RisingWave docs). The Flink job, the ClickHouse sink, and the dashboard query layer collapse into one statement.

Materialize ships the same idea on a different runtime — Timely Dataflow and Differential Dataflow, the academic work out of Microsoft Research that pioneered general-purpose incremental view maintenance. It's the strongest answer for workloads with complex joins and recursive CTEs because differential dataflow handles those natively.

Apache Fluss is the newest entrant and the most interesting structurally. Where RisingWave and Materialize are compute-plus-storage databases, Fluss is a streaming storage layer — a Kafka replacement designed for columnar real-time analytics rather than row-oriented log replay. Fluss 0.9 shipped in March 2026 with Arrow IPC columnar storage, server-side column pruning that yields a 10× read throughput improvement when pruning 90% of columns, and an Aggregation Merge Engine that pushes real-time aggregation down into storage. The thesis: Kafka was always the wrong shape for analytics — it stored rows when analytical engines want columns.

Fork 2 — Table-format-as-stream

The second fork goes the opposite direction. Instead of replacing the storage layer with a streaming-native one, it teaches the existing lakehouse storage layer — Apache Iceberg — to behave like a stream.

Iceberg supports incremental reads natively: every commit creates a snapshot, and any consumer can read only the files added between two snapshots (Apache Iceberg in 2026). The same Parquet files that a batch Spark job scans for last quarter's revenue can be read incrementally by a Flink or RisingWave job for last-second revenue. Stream and batch share storage.

The catalog made this practical in 2026. Apache Polaris — donated by Snowflake to the Apache Software Foundation, now at version 1.3 (release notes) — implements the Iceberg REST Catalog API as a vendor-neutral service. Snowflake's Open Catalog is a managed Polaris deployment, and as of February 2026 external query engines can query Snowflake-managed Iceberg tables via Horizon Catalog at GA. Trino, Flink, Spark, StarRocks, Doris, PyIceberg — all reading the same tables through the same catalog.

Honest accounting: this fork wins when you already have a batch lakehouse and need to add a streaming layer without forking storage. It loses when end-to-end latency needs to be sub-second — Iceberg's micro-batch commit cadence puts a floor measured in seconds, not milliseconds.

Fork 3 — OLAP-eats-streaming

The third fork is the most obvious in retrospect: the OLAP database eats the streaming layer entirely. If your use case is "I have a Postgres database and I want dashboards on it with seconds of lag," you don't need Kafka, you don't need Flink, you don't need a sink. You need an OLAP database that ingests from Postgres directly.

That's what shipped in 2026. ClickHouse's ClickPipes Postgres CDC connector reached general availability with replication latencies as low as 10 seconds, 10× faster initial loads through parallel snapshotting, and automatic schema-change replication. In January 2026 ClickHouse launched a fully managed Postgres service natively integrated with ClickHouse — one-click CDC sync from transactional Postgres to analytical ClickHouse with up to 100× faster analytics. OLTP and OLAP share a vendor; the seam collapses into a single product surface.

StarRocks ships the same idea via Flink CDC + Stream Load into Primary Key tables, which handle record-level updates and deletes at millisecond query latency. Flink CDC 3.0 even synchronizes schemas across sharded MySQL databases automatically (StarRocks docs). Apache Doris pairs with the Flink Doris Connector for the same shape (Doris engineering blog). The pattern: source DB → CDC connector → OLAP DB. Kafka and Flink either disappear (ClickPipes) or shrink to a thin transport layer the OLAP vendor owns.

Where Kafka still wins

Reading the three forks you might conclude Kafka is dying. It isn't. Kafka is becoming infrastructure plumbing again, the way TCP became plumbing once HTTP existed. You don't write your application against TCP; you write it against HTTP, and HTTP runs on TCP somewhere underneath. The same shift is happening to Kafka.

Three workloads still need Kafka at the application layer in 2026:

• Multi-consumer fan-out across org boundaries. One event stream consumed by a dozen teams who all need replay, ordering, and weeks of durable retention. None of the streaming SQL engines or CDC connectors target this shape — they assume one producer and a small number of well-known consumers.
• Decoupled producer/consumer evolution. If the producer team ships a new event schema quarterly and consumer teams roll out at different speeds, you want an immutable log with schema-registry guardrails sitting between them. Kafka's home turf.
• Durable replay across team boundaries. When a downstream bug means re-processing the last three weeks of events with a corrected transformation, the log has to still be there. Streaming SQL engines materialize the answer and discard the input. CDC connectors replicate but don't generally retain history. Kafka does.

For the rest — the single-team real-time-analytics use case that drove most Kafka adoption between 2018 and 2024 — one of the three forks above is a better answer. Kafka is often still in the picture, but as a transport layer someone else manages, not as a system your team operates.

The latency-cost-complexity triangle

Each of the three forks wins one axis cleanly and loses one. The chart below is the honest version of the comparison — not the marketing version.

Read it honestly. OLAP-eats-streaming wins latency because there's no streaming layer to traverse — CDC lands records into the analytical engine and queries hit them seconds later. Table-format-as-stream wins cost because storage is shared with batch — you don't pay for two copies of the data. Single-engine streaming SQL wins complexity because there's one system to operate, one schema language, one set of failure modes. No fork wins all three. The right call depends on which axis your organization is actually bottlenecked on.

What to build in 2026

The decision framework, sharpened:

• Your data lives in Postgres or MySQL and you need sub-second analytics dashboards: OLAP-eats-streaming. ClickHouse + ClickPipes Postgres CDC is the lowest-friction path; StarRocks Primary Key tables with Flink CDC if you need stronger upsert semantics. Skip Kafka entirely.
• You have rich event semantics (multi-stream joins, windowed aggregations, complex CDC patterns) and your transformation logic is best expressed in SQL: single-engine streaming SQL. RisingWave for high-throughput append-mostly workloads; Materialize for complex joins and recursive CTEs.
• You already have a Spark/Iceberg batch pipeline and want to add streaming without forking storage: table-format-as-stream. Iceberg incremental reads via a Polaris-compatible catalog; let Flink or RisingWave write into Iceberg, let Trino and DuckDB read from it. Stream and batch share files.
• You have a multi-team event-distribution problem with a dozen independent consumers, durable replay, and decoupled producer/consumer release cadences: Kafka remains the right substrate. Pick one of the above for the analytics layer on top of it.

The wrong move in 2026 is to default to the 2020 stack — Kafka + Flink + ClickHouse — because that's the architecture the team learned five years ago. Every fork above is meaningfully simpler for the workload it's targeted at, and "meaningfully simpler" compounds into faster delivery, lower bug rates, and fewer 2 AM pages.

The Flink question

You'll notice Flink keeps appearing as a component inside other products — embedded inside StarRocks's loader, running under Confluent Cloud for Apache Flink (now GA across all three clouds with materialized tables and a dbt integration), writing into Iceberg in the table-format-as-stream fork. Flink isn't dying. It's becoming a substrate engine the way Postgres became a substrate engine for a dozen different SaaS products.

The "every platform team writes Flink jobs" era is over. Flink isn't gone — it's more pervasive than ever, just invisible behind someone else's product surface. Successful infrastructure becomes plumbing. For teams choosing what to build in mid-2026: don't pick a stack because it's what the industry built in 2020. Pick the fork that matches your actual bottleneck. Latency, cost, complexity — you can win one, often two, never all three. The honest accounting is what makes the decision tractable.