Category

Data Engineering

Pipelines, warehouses, GPU-accelerated query engines, and big-data systems.

30 posts

AllAI & Machine LearningData EngineeringEdge AIOpinionResearch
sqljev: TypeSafe Jev's jev() for SQL Server, Postgres, Snowflake, BigQuery and DuckDB, on Jev or Open-Weight Laya
Data Engineering10 min read

sqljev: TypeSafe Jev's jev() for SQL Server, Postgres, Snowflake, BigQuery and DuckDB, on Jev or Open-Weight Laya

SQL cannot say 'the customer threatens to cancel'. sqljev adds jev(), jev_prob() and jev_choice() to SQL Server, PostgreSQL, MySQL, Snowflake, Databricks, BigQuery, Redshift and DuckDB, and answers them with Laya, an open-weight decision model that runs on your hardware, returns calibrated probabilities instead of text, and can be fine-tuned on your own tables. 140,000 decisions in 271 seconds on one laptop GPU; re-running all 13 queries, 0.9 seconds. Apache 2.0, version 0.1.0.

65 views
Read
pankhllm: The LLM Gateway That Learns to Skip the LLM, Without Replacing the Stack You Already Run
Data Engineering10 min read

pankhllm: The LLM Gateway That Learns to Skip the LLM, Without Replacing the Stack You Already Run

Most of an agent's LLM calls are not writing anything. They are decisions: which tool, which skill, which parameters, made thousands of times a day by a model paid in seconds and tokens. pankhllm sits where your app already calls an LLM, learns those decisions from its own traffic, and starts making them in 0.2 ms on a CPU with a 262 KB model. What it is not sure about still goes to your LLM. On the same 14 questions: a 12B planner 1,743 ms, Laya 49 ms, pankhllm's own model 4 ms, all 14 correct. Here is what it is, what it is not, and where it stops.

76 views
Read
Your Apple Silicon GPU Loses to One CPU Core Until a Million Rows. I Measured 111 Operations, Then Rebuilt ArrowMetal 0.2.0 Around the Answer
Data Engineering19 min read

Your Apple Silicon GPU Loses to One CPU Core Until a Million Rows. I Measured 111 Operations, Then Rebuilt ArrowMetal 0.2.0 Around the Answer

Every GPU data library benchmarks itself at 50 million rows. Your dataframe has 80,000. On an Apple M4 Max, summing 1,000 integers takes the GPU 112 microseconds and Polars less than one: the GPU is more than 100 times behind. I built one of these libraries, so I measured the row count where the GPU overtakes the fastest CPU code for 111 operations: the median needs 10,000,000 rows against a multi-core library, about a million against one core, and sixteen never get there. So ArrowMetal 0.2.0 refuses the GPU below the line, byte-identical, and around that router it grew GPU readers for CSV, JSON, nested Parquet, Delta Lake and Iceberg, a Polars engine and a DuckDB optimizer extension.

77 views
Read
DuckDB on the Apple Silicon GPU: Plain SQL, 19 of 22 TPC-H Queries on the Mac's Own GPU, and a Rule That Says Never Slower
Data Engineering15 min read

DuckDB on the Apple Silicon GPU: Plain SQL, 19 of 22 TPC-H Queries on the Mac's Own GPU, and a Rule That Says Never Slower

DuckDB has no GPU backend of its own, and the GPU engines built for it need an NVIDIA card. gpudb 0.7 is my Apache-2.0 DuckDB extension for the GPU already inside your Mac, and for CUDA too. You write plain DuckDB SQL; the GPU takes a statement only where it has been measured faster than DuckDB on your own machine. On an Apple M4 Max, 19 of 22 TPC-H SF10 queries run on the Metal GPU at 1.06x to 48x with zero rows differing. The one row below parity is printed, not dropped.

92 views
Read
Apple's GPU Has No 64-bit Floats. I Made It Sort 50 Million Doubles Anyway
Data Engineering15 min read

Apple's GPU Has No 64-bit Floats. I Made It Sort 50 Million Doubles Anyway

The Metal Shading Language has no double type, and float64 is the default number in Python, pandas and Apache Arrow. Building ArrowMetal meant getting past three walls: a GPU with no 64-bit floats, a missing 64-bit atomic add, and a Swift compiler bug that reports errors nobody threw. Here is how each one was solved, what it cost, and why a GPU that cannot add two doubles sorts 50,000,000 of them in 32 ms.

139 views
Read
Polars vs DuckDB vs ArrowMetal GPU on Apple Silicon: Sort and Group-By Benchmarks
Data Engineering12 min read

Polars vs DuckDB vs ArrowMetal GPU on Apple Silicon: Sort and Group-By Benchmarks

Polars, DuckDB and ArrowMetal on an Apple M4 Max: sort and group-by benchmarks at 10M and 50M rows, wall time next to CPU time, and the rows where the CPU is still ahead.

171 views
Read
Apache Arrow Compute on the Apple Silicon GPU: The First Arrow Project I Created That Does It, With 173 Operations Measured Against Polars, pyarrow and pandas
Data Engineering11 min read

Apache Arrow Compute on the Apple Silicon GPU: The First Arrow Project I Created That Does It, With 173 Operations Measured Against Polars, pyarrow and pandas

I built ArrowMetal, the first Apache Arrow project I could find that runs compute on the Apple silicon GPU. Apple silicon has one memory for CPU and GPU, and an Arrow buffer in shared Metal memory is already a GPU buffer; no Arrow project used that. ArrowMetal does: 307 of Arrow's 307 compute functions, seven languages, take at 24.2x pyarrow on an M4 Max, and 339 benchmark rows against the fastest CPU idiom of Polars, pyarrow, pandas and numpy, including the 77 where the CPU is still ahead.

228 views
Read
“PostgreSQL-compatible” Is Not PostgreSQL: What Arrow's Native ADBC Driver Does on 14 Wire-Compatible Databases
Data Engineering12 min read

“PostgreSQL-compatible” Is Not PostgreSQL: What Arrow's Native ADBC Driver Does on 14 Wire-Compatible Databases

I ran the native PostgreSQL and MySQL ADBC drivers against 28 databases that speak their protocols. Half stopped. Then I found a bug in my own driver.

123 views
Read
Apache Arrow ADBC Just Listed My ODBC Bridge on Its Official Integrations Page — Seven Days After v0.1.0
Data Engineering6 min read

Apache Arrow ADBC Just Listed My ODBC Bridge on Its Official Integrations Page — Seven Days After v0.1.0

The Apache Arrow ADBC documentation now lists adbcBridge on its Tools & Integrations page — seven days after I released v0.1.0. I filed the listing request on August 29; on August 31 a project member invited a PR, and it was merged six hours after the invitation. What the entry says, how the week that earned it went, and what it changes for anyone with an ODBC-only database.

147 views
Read
Your ODBC Driver Says SQL_SUCCESS and Lies: 24 Bugs Found in 13 Database Projects (adbcBridge Part 2)
Data Engineering15 min read

Your ODBC Driver Says SQL_SUCCESS and Lies: 24 Bugs Found in 13 Database Projects (adbcBridge Part 2)

Part 2 of the adbcBridge story. Running one workload through 46 databases on three operating systems turned up 24 defects that belong to other projects — twelve of them return wrong or lost data under SQL_SUCCESS. Every one is filed upstream with a reproduction that needs no adbcBridge in the stack. Here is the ledger, what the bugs have in common, and what happened when the maintainers read them: within the first week, two fixes landed upstream and four more fix PRs went up.

143 views
Read
I Built an Apache Arrow ADBC Driver for Every ODBC Database: 46 Databases, 5 Languages, Every Number Measured
Data Engineering21 min read

I Built an Apache Arrow ADBC Driver for Every ODBC Database: 46 Databases, 5 Languages, Every Number Measured

Native Apache Arrow ADBC drivers exist for a handful of databases. The other few hundred ship an ODBC driver and nothing else. adbcBridge is one plain-C11 shared library that turns every ODBC driver on your machine into an Arrow-native ADBC driver — columnar record batches out, bulk ingest in — from Python, Rust, Go, Java and C#. Today it is public: 46 databases verified on Linux, 41 on macOS, 45 on Windows, five languages measured against all of them, and every figure named with the laptop and the load it was taken under.

330 views
Read
The First SQL Engine for Apple Silicon GPUs Is Now a DuckDB Community Extension
Data Engineering8 min read

The First SQL Engine for Apple Silicon GPUs Is Now a DuckDB Community Extension

In May 2026 I shipped gpudb v0.1 — the first SQL execution engine targeting Apple Silicon GPUs, built as a DuckDB extension with a CUDA backend on Linux. Three releases later, the project crossed two lines at once. v0.3.0's streaming-aggregate rewrite reached parity with native DuckDB on end-to-end TPC-H queries — the worst cell improved roughly 100×, from 11.05 s to 0.109 s. And gpudb became an official DuckDB Community Extension: INSTALL gpudb FROM community now works in any DuckDB ≥ 1.5.5, signed, no flags. This is the full arc — what v0.1 proved, what v0.2 honestly lost, what v0.3 fixed, and why the next GPU frontier is joins.

208 views
Read
The Agent-Written Data Pipeline: The Review Bottleneck Nobody Priced In
Data Engineering10 min read

The Agent-Written Data Pipeline: The Review Bottleneck Nobody Priced In

AI agents can now write dbt models, SQL transforms, and backfills that pass CI and ship. The catch: a wrong number doesn't crash, it quietly poisons every dashboard downstream. The hard part moved from authoring to verification.

96 views
Read
We Published Our 110× Loss. One Release Later, It Was Gone.
Data Engineering9 min read

We Published Our 110× Loss. One Release Later, It Was Gone.

A reviewer on gpudb's DuckDB community-extensions PR asked the question every GPU project dreads: forget the kernel benchmarks — what does a user actually see end-to-end? We ran it honestly. Native DuckDB won every query shape, by 3× to 109×, against our own extension. We published those numbers in our own release notes — and the act of writing them down produced the structural diagnosis that closed the entire gap in the very next release. The fix was the opposite of what a GPU database is supposed to do: delete the GPU from the hot path. This is the full story, with every number.

83 views
Read
The Inference Hardware Wars: Why Your Token Bill Is Decided in a Fab, Not a Prompt
Data Engineering10 min read

The Inference Hardware Wars: Why Your Token Bill Is Decided in a Fab, Not a Prompt

The token-price crash everyone cheers isn't software magic. It's a hardware war: Cerebras and Groq attacking on speed, NVIDIA's Rubin counterpunching on cost-per-token. The winner of that fight, not your prompt, sets your inference bill and your latency floor.

99 views
Read
samkhya v1.1: Never Regress — Putting a Model in Your Query Optimizer Without Letting It Wreck the Plan
Data Engineering10 min read

samkhya v1.1: Never Regress — Putting a Model in Your Query Optimizer Without Letting It Wreck the Plan

samkhya is a Rust SDK that lets a model — a gradient-boosted tree, TabPFN-2.5, even an LLM — correct the row-count estimates your query optimizer runs on, under a provable ceiling that a hallucinating model can never breach. This is the deep dive: the never-regress clamp, the portable Iceberg sidecar, the three swappable backends, and the honest benchmark I pre-registered and then failed — reported as such.

128 views
Read
My DuckDB Extension PR Sat Red for Eight Weeks. The Bug Wasn't What the CI Said.
Data Engineering5 min read

My DuckDB Extension PR Sat Red for Eight Weeks. The Bug Wasn't What the CI Said.

A DuckDB community-extension PR sat red for eight weeks over a job named linux_arm64. The real bug was three words in the log — and a fleet of AI agents took it green on all four platforms in about seventy minutes.

75 views
Read
Streaming OLAP: The Post-Kafka Stack for Real-Time Analytics
Data Engineering10 min read

Streaming OLAP: The Post-Kafka Stack for Real-Time Analytics

The Kafka + Flink + ClickHouse/Pinot/Druid stack we built between 2018 and 2024 is fragmenting into three forks: single-engine streaming SQL, table-format-as-stream, and OLAP databases that eat the streaming layer entirely. Kafka isn't dying — it's becoming plumbing.

132 views
Read
What I Learned Writing GPU Kernels for SQL Aggregates
Data Engineering6 min read

What I Learned Writing GPU Kernels for SQL Aggregates

Three months, two abandoned designs, one breakthrough. The one-paragraph version: Apple Silicon GPUs don't have 64-bit atomic_fetch_add until very recent OS versions, and that single missing instruction shapes every other architectural decision in a Metal SQL aggregate engine.

83 views
Read
Multi-Aggregate Fusion: One Read, Four Answers
Data Engineering5 min read

Multi-Aggregate Fusion: One Read, Four Answers

Every analytical engine treats SELECT SUM(x), MIN(x), MAX(x), COUNT(x) FROM t as four passes of the column. Fuse them into a single kernel and the speedup ratio against four-pass code becomes 9x to 25x. Here's why the technique works, and the data shape where it doesn't.

93 views
Read
Apple Silicon's Unified Memory Is the Quiet Revolution in Analytical Compute
Data Engineering6 min read

Apple Silicon's Unified Memory Is the Quiet Revolution in Analytical Compute

M3 Ultra ships 512 GB of memory at 819 GB/s, addressable by the GPU with zero PCIe transfer cost. Every GPU database project from the past decade was architected around the assumption that memory bandwidth came at PCIe-tax prices. That assumption is now wrong on a fifth of the developer laptops in the world.

141 views
Read
samkhya v1.0: Plug Claude, GPT-4o-mini, or Local Ollama Into Your SQL Query Optimizer
Data Engineering16 min read

samkhya v1.0: Plug Claude, GPT-4o-mini, or Local Ollama Into Your SQL Query Optimizer

samkhya v1.0 ships an LLM-pluggable corrector backend for embedded analytical engines — DataFusion, DuckDB, Polars, Postgres, Iceberg, gpudb. Plug Claude, GPT-4o-mini, or local Ollama into the cardinality-estimation slot via a simple HTTP wire contract (Python FastAPI and Node TypeScript reference servers ship in the box). Every LLM output is clamped from above by a provable pessimistic ceiling (LpJoinBound — 40.95× tighter than the 2008 AGM bound) so the LLM can never make your plan worse than the engine's native estimate. Transport-floor latency measured at P95 0.07–0.11 ms; live-LLM end-to-end cells honestly marked PROJECTED pending API budget.

121 views
Read
Why I built a GPU SQL engine in 2026 — when every other one died
Data Engineering27 min read

Why I built a GPU SQL engine in 2026 — when every other one died

Every standalone GPU database built between 2013 and 2024 was acqui-hired or pivoted. So why ship gpudb in 2026? Because nobody had wired Apple Silicon's unified memory into a SQL engine — and DuckDB hands you a hundred-thousand-user distribution channel without writing a database from scratch.

703 views
Read
Databricks vs Snowflake vs The New Wave: The Data Engineering Paradigm Shift
Data Engineering5 min read

Databricks vs Snowflake vs The New Wave: The Data Engineering Paradigm Shift

Snowflake just posted $4.68B in FY26 revenue at 29% growth. Databricks crossed $5.4B ARR in February at 65% growth. And neither chart explains why the most interesting data infrastructure being shipped in 2026 is single-process, embeddable, and runs on a laptop.

238 views
Read
Iceberg's Puffin Sidecars: Portable Stats for the Open Lakehouse
Data Engineering10 min read

Iceberg's Puffin Sidecars: Portable Stats for the Open Lakehouse

Apache Iceberg's Puffin file format is the most strategically important subsystem nobody is talking about. It is the mechanism by which an open lakehouse can carry warehouse-grade statistics across vendors — write the sketch once in Trino, read it tomorrow in Snowflake, plan a join correctly on the first cold query.

103 views
Read
DuckDB Ate the Modern Data Stack
Data Engineering6 min read

DuckDB Ate the Modern Data Stack

An embedded analytical engine with no servers, no cluster, no migration cost just quietly displaced Spark for small data and Snowflake XS for medium data. MotherDuck closed Series B at a $400M post-money. Here's the part everyone undercounts.

98 views
Read
Iceberg, Delta, Hudi: Pick One in 2026 and Move On
Data Engineering5 min read

Iceberg, Delta, Hudi: Pick One in 2026 and Move On

The table-format wars are functionally over. Iceberg won on interop. Delta won on installed base. Hudi won on streaming upserts. The decision tree for a new project in 2026 is shorter than the comparison-blog industry wants you to believe.

85 views
Read
Polars vs DuckDB in 2026: When To Pick Which
Data Engineering9 min read

Polars vs DuckDB in 2026: When To Pick Which

Polars ate Pandas. DuckDB ate everything below the warehouse. The 2023 expectation was a cage match between two in-process analytical engines — the 2026 reality is they ate different cake, and the decision is mostly about whether your team thinks in DataFrames or SQL.

495 views
Read
Vector Indexes in OLAP Engines: 2025 Is Where Search Ate Analytics
Data Engineering10 min read

Vector Indexes in OLAP Engines: 2025 Is Where Search Ate Analytics

DuckDB, ClickHouse, Snowflake, BigQuery, Postgres — by late 2025 every serious analytical engine ships a native vector index. That wasn't an AI-hype reflex. It was the realization that embedding search is just a column scan with a different distance function, and the warehouse-plus-vector-DB split was operational waste for the 90% case.

73 views
Read
Apache Arrow IPC vs JSON: The Numbers Behind the Switch
Data Engineering10 min read

Apache Arrow IPC vs JSON: The Numbers Behind the Switch

Most data-API traffic in 2025 still moves as JSON because humans need to read it. But for any system actually shipping columnar batches between services — analytical pipelines, feature stores, embedding services, MCP-style tool calls — Arrow IPC is 3-30× faster end-to-end. Honest accounting of when the switch pays off and when JSON is still correct.

205 views
Read