I Built an Apache Arrow ADBC Driver for Every ODBC Database: 46 Databases, 5 Languages, Every Number Measured

Here is a number that has bothered me for a while: Apache Arrow's ADBC — the database API whose result sets are columnar Arrow batches instead of row-at-a-time buffers — has native drivers for roughly six databases. PostgreSQL, SQLite, DuckDB, Snowflake, BigQuery, Flight SQL. Everything else your company actually runs — Db2, Oracle, Teradata, SQL Server, SAP HANA, Informix, Vertica, and the long tail of a few hundred engines — ships an ODBC driver and nothing else. If you wanted Arrow out of them, you wrote the row-to-column conversion yourself, per language, per driver, and you found the driver's quirks the hard way.

So I built the piece in between. adbcBridge is an ADBC driver for any ODBC data source: one plain-C11 shared library that turns every ODBC driver on your machine into an Arrow-native ADBC driver — columnar record batches out, bulk ingest in — from Python, Rust, Go, Java, C# and anything else that speaks the ADBC driver manager. Today, v0.1.0 is public: on GitHub under Apache-2.0, on PyPI, crates.io and NuGet, with docs at adbcbridge.org.

This post is the launch, but I want it to read like the benchmark files do: every number below names the machine it was taken on and the load that machine was under, and the one place where the bridge cannot win gets the same space as the places it does.

What it does, in one paragraph

A SELECT becomes Arrow record batches: columns are bound once with SQLBindCol into rowsets of up to 8 MiB and copied column-at-a-time, UTF-16 becomes UTF-8, decimals become decimal128, and long or unbounded columns are chunked through SQLGetData. Parameters, prepared statements, transactions, rows_affected, and the catalog calls — GetInfo, GetObjects, GetTableTypes, GetTableSchema — all map onto the ODBC calls underneath, with structured errors carrying the SQLSTATE and native code. Bulk ingest goes in as parameter arrays or a multi-row INSERT, probed once per connection and fanned out over up to 64 connections if you ask. ExecutePartitions splits one query across N connections. It speaks the ADBC 1.0.0 and 1.1.0 ABI, installs as a driver manifest so every binding loads it as driver="odbc", and where a native ADBC driver is installed — PostgreSQL, SQLite, DuckDB, Flight SQL — it hands the whole connection over so you get native speed from the same install.

Who this is for

Three kinds of team hit the same wall from different sides, and adbcBridge is the same answer to all three.

• The data engineer with the estate nobody is migrating. Db2 on the mainframe, Oracle under the ERP, SQL Server under everything else, Teradata for the warehouse that predates the warehouse. Every one of them has an ODBC driver already installed and configured. None of them will ever get a native ADBC driver. Today the path into DuckDB, Polars, pandas or a Parquet lake is pyodbc → list of tuples → DataFrame, row by row, in Python only.
• The polyglot shop. A Rust or Go service that needs the same customer table the Python notebook reads, and a C# reporting job that reads it again. Three languages, three ODBC wrappers, three sets of driver quirks discovered three times. One driver library with one quirk table, loaded by name from all of them, is the point of this project.
• The tool builder. If you ship a BI tool, an ETL engine, a notebook product or a semantic layer, "which databases do you support" is a list you maintain one vendor integration at a time. ADBC is one C ABI. adbcBridge makes that ABI reach 46 verified engines and a few hundred reachable ones, with the columnar conversion done once, in C, and every driver quirk handled below your code.

The positioning is deliberately narrow: Arrow-native access to the databases that Arrow's native drivers will never cover, from every language, with the numbers to prove it. Where a native driver exists, adbcBridge steps aside and lets it do the work.

The Monday-morning use case

Here is the job that made me build it. A nightly extract pulls the orders table out of Db2 and the customer dimension out of SQL Server, joins them against last quarter's Parquet in a data lake, and writes the result back out for the dashboard. The old version was three pyodbc cursors, three fetchall() calls, three DataFrame constructions, and a pandas merge that owned all the memory. The new version is this:

import adbc_driver_manager.dbapi as dbapi
import duckdb

def arrow(uri, sql):
    with dbapi.connect(driver="odbc", db_kwargs={"uri": uri}) as conn, conn.cursor() as cur:
        cur.execute(sql)
        return cur.fetch_arrow_table()          # Arrow batches straight from the ODBC block cursor

orders    = arrow("DSN=db2prod",  "SELECT order_id, cust_id, amount, placed_on FROM orders WHERE placed_on >= CURRENT DATE - 1 DAY")
customers = arrow("DSN=mssqlcrm", "SELECT cust_id, segment, region FROM dim_customer")

con = duckdb.connect()
con.register("orders", orders)                  # zero-copy: DuckDB reads the Arrow buffers in place
con.register("customers", customers)
con.sql("""
  COPY (
    SELECT o.placed_on, c.region, c.segment, SUM(o.amount) AS revenue
    FROM orders o JOIN customers c USING (cust_id)
    GROUP BY ALL
  ) TO 's3://lake/daily/revenue.parquet'
""")

No pandas hop, no per-cell Python objects, no second copy for the join. The Db2 and SQL Server rows arrive as Arrow record batches — Db2 was measured at 1.4–1.6 M rows/s and SQL Server at 0.76 M rows/s on the Linux host, so a 20-million-row extract is tens of seconds of transfer, not minutes — and DuckDB, Polars and pandas all take an Arrow table without copying it. The same two SELECTs run unchanged from the Go service that serves the dashboard and the Rust job that validates it, through the same libadbc_driver_odbc.so, because the driver is one file and the language packages only find and load it. If the orders table were in PostgreSQL instead, the same code would be handed to the native driver automatically — or, for the big read, split across eight connections with one option.

Measured, not claimed

Every figure in the repository was measured on one of four ordinary laptops, never a cloud instance, never a quiesced benchmark host — and the files say which. The Linux reference host is an i9-13900HK with 31 GiB, typically carrying ~23 GiB of other work and a 1-minute load of 2–5 during runs. The macOS host is an M4 Max. Two Windows laptops did the Windows campaign, the second an i9-13900HK under Docker Desktop on WSL2. Each benchmark file opens with the host state of its runs, and every table carries the spread, not only the median. When a number below has a qualifier attached, the qualifier is the point.

The read path: within 7% of the ODBC floor

The first question for any ODBC-to-Arrow bridge is how much it costs on top of ODBC itself. The answer, on this stack, is almost nothing.

pyodbc's fetchall() alone accounts for 0.95 s of its 1.16 s: the bridge beats pyodbc's row materialisation before any columnar conversion is even charged to it. And that floor is instructive in a second way — it is also the ceiling for a single connection. Nothing an ODBC bridge does can read faster than SQLFetch hands over rows. Which raises the obvious question.

Can it beat a native ADBC driver? Only by doing work the native driver doesn't

Against the native adbc_driver_postgresql, adbcBridge on one connection is 0.40× native on 1 M rows and 0.36× on 10 M. That is the ODBC boundary — the native driver talks the wire protocol and builds Arrow directly — and I am stating it up front because the next figure only makes sense next to it.

ADBC's partition contract exists for exactly this situation: ExecutePartitions hands back N opaque descriptors, and ReadPartition turns any one of them back into a stream on any connection, in any order, in any process. adbcBridge implements both. On PostgreSQL it slices the heap by ctid — a tuple id of (block, offset) that compares lexicographically, so the table's blocks are a total order that cuts the heap with no index, no key column and no sort, and PostgreSQL 14+ runs each slice as a TID Range Scan. Where there is no heap (CockroachDB, YugabyteDB, a declaratively partitioned parent) it falls back to the leading primary-key column, and on YugabyteDB's hash-partitioned keys to yb_hash_code(), because a plain key range there is a storage filter over a sequential scan and would be N times slower than not splitting.

Two honest footnotes on that chart. The speed-up flattens after N=8 (1.07× from 8 to 16 on 10 M rows) because by then the client is burning ~10 cores decoding and the server is the one waiting. And the split strategy matters more than the fan-out: on CockroachDB, the key-range split the driver picks is 9.9× a single partition, where a LIMIT/OFFSET split is 3.4× and a modulo split 2.7×, because those two make every slice scan the whole table.

Ingest: fast on every driver, and the one place it cannot win

adbc_ingest builds its own statement, so it can pack K rows into one INSERT … VALUES (…),(…),… inside a single transaction. K is probed against the driver — SQLite's 999-variable limit, ClickHouse preparing 500 row-groups and then refusing to execute them, Oracle rejecting the multi-row form outright (it gets INSERT ALL … SELECT 1 FROM dual instead), Firebird having no multi-row VALUES at all (it gets a UNION ALL of typed one-row selects) — all discovered at run time and remembered on the connection. It works on every driver that can bind a parameter, including the eight whose parameter arrays are unusable.

Against PostgreSQL specifically the driver goes one step further and sends a whole column of a batch as a single array parameter — INSERT INTO t SELECT * FROM unnest(?::bigint[], ?::float8[], ?::text[], ?::date[]) — one parameter per column however many rows the statement carries. On a million four-column rows that takes single-connection ingest from 2.75 s to 1.40 s, and sixteen-connection ingest from 0.45 s to 0.34 s, with about a third less CPU on both sides. It is keyed on the server's version() banner being real PostgreSQL (TimescaleDB and Citus qualify; CockroachDB, YugabyteDB, CrateDB, Materialize and the other forks do not), and the server is asked to prove it expands the form correctly — NULLs, empties, embedded braces and quotes — before it is ever used. A wrongly quoted array literal would be a data-corruption bug rather than a slow one, so this one is deliberately narrow.

Even at its best, though, adbcBridge's ingest does not beat the native PostgreSQL driver: 0.73–1.02× over repeated runs. At a million rows and sixteen connections the two are within noise of each other; at ten million the bridge stays about 1.4× behind. The reason is not the statement shape, and it is worth showing because it is the one thing no ODBC bridge can fix.

46 databases, three operating systems, every non-pass explained

"Any ODBC data source" is a statement about reachability. What I can actually stand behind is the compatibility matrix: the same workload — types, NULLs, Unicode including emoji in both parameters and statement text, parameters, bulk ingest, batched reads, GetObjects, error mapping — run through one Python test file against a real server or file, for 46 databases, on three operating systems.

The list runs from the obvious — PostgreSQL, MySQL, MariaDB, SQL Server, Oracle 23ai, IBM Db2, SQLite, DuckDB — through the distributed engines (CockroachDB, YugabyteDB, TiDB, OceanBase, Citus, Cloudberry), the warehouses (ClickHouse, StarRocks, Doris, MonetDB, Vertica, Databend), the time-series stores (TimescaleDB, QuestDB, GreptimeDB, TDengine, InfluxDB 3), and the ones you would not expect to see behind an ODBC driver at all: Google Cloud Spanner through PGAdapter, MongoDB through its BI Connector, Microsoft Access files, Dremio and Arrow Flight SQL servers, OpenSearch, Apache Ignite, and Informix over DRDA.

Nearly every one of those drivers needed at least one workaround, and the workarounds are the part of the project I am quietly proudest of. adbcBridge detects the backing driver at connect time and sets 64 quirk flags that the reader and the parameter binder consult: Db2's 32-bit SQLLEN on a 64-bit build; Firebird and Ignite sizing SQL_C_WCHAR in 4-byte wchar_t; SQL Server's SQLGetTypeInfo naming the deprecated TEXT type, which the server will not sort, group or even compare, so ingest DDL spells an Arrow string NVARCHAR(MAX) instead; Db2's LONG VARCHAR, which writes ~700× slower than a VARCHAR; MatrixOne describing every TEXT column as five characters wide however long its values are (3k rows/s before the fix, 2.05 M after); the Flight SQL driver whose SQLColumns segfaults on its first fetch. None of it is configuration you touch. Every quirk is keyed on what the driver or server says it is, and the quirks reference documents each one with the SQLSTATE that revealed it.

Five languages, one binary

The driver is a plain C shared library, so every binding dlopens the same file and calls AdbcDriverInit. The point of running one workload from Python, Rust, C#, Java and Go against all 46 databases was to check that the bindings' differences are the bindings' own, not the driver's — and they are.

Each language gets a package that finds and loads that library: a Python wheel with the library bundled (pip install adbcbridge), a Rust crate whose bundled feature compiles the C driver from the sources it carries, a NuGet package with runtimes/<rid>/native/ assets, a jar with the natives inside, and a Go module over drivermgr. Python, Rust, C# and Go are on their registries as of this week; the jar is a release asset until it reaches Maven Central.

Native delegation: getting out of the way

Replacing native ADBC drivers where they exist is an explicit non-goal. So when AdbcDatabaseInit recognises a target a native driver handles — a postgresql://, sqlite:, duckdb: or grpc:// URI, or an ODBC connection string naming psqlodbcw.so — adbcBridge loads that driver and forwards every call to it. The result set is the native driver's own Arrow stream, handed back untouched: one function-pointer hop per ADBC call and nothing per row, and delegated fetches measure the same as calling the native driver directly (0.20 s for the million PostgreSQL rows against 0.21 s native). It is a best-effort optimisation: if the native driver is not installed, auto falls back to ODBC and records why. It is also careful — rebuilding a libpq URI from an ODBC string is only safe if every keyword is accounted for, so sslmode=verify-full is forwarded, driver-only keywords are ignored, and anything unknown stops delegation rather than silently changing your connection's security.

Giving back

Driving 46 databases through one driver on three operating systems turns up defects that belong to other projects, and I have filed them with reproductions that need no adbcBridge in the stack. unixODBC #239: the driver manager overwrites its own stack and heap, and the process aborts, on the first SQL_ERROR from a driver whose SQLWCHAR is 4 bytes — a 40-line fake driver reproduces it, and the maintainer committed a check the same day. Virtuoso #1469: the Homebrew macOS driver is built to iODBC's 4-byte width and nothing says so. Arrow Flight SQL ODBC #16: the Apple Silicon build is iODBC-width and undocumented, and LogEnabled=true makes SQLAllocHandle fail. A dozen more findings — segfaults in MySQL Connector/ODBC's SQLColumns on a NULL precision, astral characters coming back as three question marks on Windows, a Go ODBC package that access-violates on its first diagnostic — are recorded in docs/UPSTREAM.md with the conditions that reproduce them.

What adbcBridge does not claim

• It is not an Apache project. It is an independent Apache-2.0 driver that implements the Apache Arrow ADBC standard. The ADBC Driver Foundry validation suite and a Foundry listing are on the roadmap, not done.
• A single connection does not beat a native driver, and never will over ODBC: 0.36–0.40× native on PostgreSQL. The win comes from partitioning, and it comes with the host-load qualifiers above.
• Bulk ingest does not beat native COPY — 0.73–1.02×, for the WAL reason shown. Delegate for that.
• The Windows build lacks prefetch and parallel ingest. Both are pthreads and compiled out on _WIN32 until a Win32 thread shim lands, so Windows numbers measure a different code path.
• Prefetch is worth very little even where it exists — 1–10% — because most ODBC drivers have already buffered the whole result set client-side by the time SQLFetch is called. It is off by default and the docs say so.
• Teradata, SAP HANA, Snowflake, Databricks and Hive are reachable, not verified. Their ODBC drivers sit behind vendor logins; the matrix rows are marked "help wanted".

Try it

You need an ODBC driver for your database installed (sqliteodbc, psqlodbc, msodbcsql, the Db2 clidriver, …) and, on Linux or macOS, a driver manager. From source, one script builds the library and installs it into ~/.local with a driver manifest, no root:

./install.sh
pip install adbc-driver-manager pyarrow

import adbc_driver_manager.dbapi as dbapi

conn = dbapi.connect(driver="odbc", db_kwargs={"uri": "Driver=SQLite3;Database=my.db;"})
with conn.cursor() as cur:
    cur.execute("SELECT 42 AS answer")
    print(cur.fetch_arrow_table())

Or skip the build and take the package for your language — the Python wheel carries the library:

pip install adbcbridge                                  # Python — wheel bundles the driver
cargo add adbcbridge@0.1.0                              # Rust — features = ["bundled"] compiles it
dotnet add package AdbcBridge --version 0.1.0           # C# — native lib inside the package
go get github.com/singhpratech/adbcbridge/go@v0.1.0     # Go — cgo, points at the installed library

import adbcbridge

conn = adbcbridge.connect(uri="Driver=SQLite3;Database=first.db;")
with conn.cursor() as cur:
    cur.execute("SELECT 42 AS answer")
    print(cur.fetch_arrow_table())

For a large PostgreSQL read, ask for partitions and give each one its own connection:

cur.adbc_statement.set_options(**{"adbc.odbc.partitions": "8"})
cur.adbc_statement.set_sql_query("SELECT id, val, txt, dt FROM bench")
descriptors, schema, _ = cur.adbc_statement.execute_partitions()
# read each descriptor on its own connection, in a thread pool, and concat_tables the pieces

Per-OS install pages, six language guides, the type mapping, the options reference and the connection strings for all 46 databases are at adbcbridge.org; the benchmark files with the exact commands that produced every table are under bench/; the per-cell matrix is docs/COMPATIBILITY.md.

What comes next

The roadmap is short and specific: the Win32 thread shim that restores prefetch and parallel ingest on Windows; a driver bootstrap so install.sh fetches the open-licence ODBC drivers a first run needs; the ADBC Driver Foundry validation suite; Maven Central. Then a JDBC bridge on the same model — a JVM loaded in-process so Python, Go, Rust and C# can reach JDBC-only drivers without a Java application — and, later, OLE DB for the few Windows-only sources left.

If you have a database with an ODBC driver that is not in the list, or a binding you want measured, the matrix is one Python file and a docker-compose service per database. Bring the database. I would like to see the 46 become 60.

adbcBridge is at github.com/singhpratech/adbcbridge, Apache-2.0. It implements the Apache Arrow ADBC standard and is not affiliated with the Apache Software Foundation. Related reading on this site: Apache Arrow IPC vs JSON, why I built gpudb, and Polars vs DuckDB.