Polars vs DuckDB vs ArrowMetal GPU on Apple Silicon: Sort and Group-By Benchmarks

Three ways to sort fifty million numbers on the same Mac, measured the same way. Polars and DuckDB are the two engines people choose between on a laptop, and I wrote when to pick which in January. The third entrant is the machine's own GPU, reached through ArrowMetal, the Apache Arrow compute library I built for Apple silicon, listed on Apache Arrow's Powered By page. Same Apple M4 Max, same generated data, same best-of-five protocol, wall time and CPU time side by side.

The headline, at 50,000,000 rows on an M4 Max: a float64 sort takes 32 ms on the GPU, 133 ms in Polars on 9.7 cores and 294 ms in DuckDB on 13 cores. A 1,000-group sum takes 4.9 ms on the GPU, 9.0 ms in DuckDB on 15 cores, 18.5 ms in pyarrow's Acero and 82 ms in Polars. DuckDB is the fastest CPU engine on low-cardinality group-by and multi-key sort, which narrows the GPU's lead there to about 2x; on every other sort and from 100,000 groups upward the GPU stays 3x or more ahead of whichever CPU engine is best, and it does it on 1 to 2 ms of CPU time where the engines spend hundreds or thousands.

How the three were measured

Everything runs from one harness, Benchmarks/full_matrix.py, on an Apple M4 Max with 16 CPU cores and 64 GB of unified memory. Each row is the best wall time of up to five calls after one warm-up, with the process CPU time of that same call recorded beside it across all threads. That second number tells you how many cores the wall time was bought with, and what is left for the rest of your program.

Every CPU library is measured twice and the faster idiom is the one in the table: Polars eager and lazy, pyarrow's plain call and its threaded Acero path, all with 16 threads available (Polars 1.44.1, pyarrow 25.0.1). ArrowMetal 0.1.0 is called from Python on Arrow arrays whose buffers already sit in Metal shared memory, so there is no upload in the timing and nothing to download afterwards. The full 339-row matrix with pandas and numpy columns is in the repository.

DuckDB 1.5.5 was run on 12 September 2026, five days later, on the same machine, through a companion script that reuses the matrix's generators, timing loop and byte counts (Benchmarks/duckdb_matrix.py, written up in docs/DUCKDB.md), default 16 threads, insertion order preserved. Each operation was timed four ways with the same SQL: over the registered Arrow table in one chunk and in 16 record batches, and over a native DuckDB table with the result fetched as Arrow or kept inside DuckDB. The cell is the fastest of the four, which for every sort is the last. That flatters DuckDB and I want it stated: delivering a wide sorted result back to Arrow is expensive, so the 50,000,000-row float64 sort is 294 ms in-engine and 1,308 ms delivered as an Arrow table. For a 1,000-row group-by result the two are the same. DuckDB has no argsort, so those rows carry an explicit int64 row-index column and read 8 bytes per row more than the kernel.

Sorting 50 million rows: Polars vs DuckDB vs the GPU

sort	Polars ms (CPU-ms, cores)	DuckDB ms (CPU-ms, cores)	pyarrow ms (CPU-ms, cores)	ArrowMetal ms (CPU-ms)	GPU ratio
sort float64 50,000,000 rows	133 (1,284, 9.7c)	294 (3,866, 13.2c)	6,044 (6,044, 1c)	32.12 (1.25)	4.1x
argsort int64 50,000,000 rows	302 (3,574, 11.8c)	321 (4,340, 13.5c)	5,197 (5,196, 1c)	39.15 (0.86)	7.7x
argsort float64 50,000,000 rows	420 (5,531, 13.2c)	344 (4,349, 12.6c)	5,885 (5,885, 1c)	39.91 (0.83)	8.6x
lexsort, 2 int32 keys 50,000,000 rows	903 (9,948, 11.0c)	265 (3,532, 13.3c)	7,774 (7,773, 1c)	37.59 (2.42)	7.0x
sort float64 10,000,000 rows	26.16 (250, 9.6c)	54.6 (718, 13.1c)	1,006 (1,006, 1c)	6.85 (1.17)	3.8x
sort utf8 10,000,000 rows	133 (1,326, 10.0c)	79.4 (1,063, 13.4c)	1,865 (1,865, 1c)	22.96 (7.13)	3.5x

Read the Polars column as two numbers. The float64 sort at 50,000,000 rows is 133 ms of wall time and 1,284 ms of CPU time, nearly ten cores held for the duration. The GPU sort is 32 ms of wall and about a millisecond of CPU, because the host thread issues the dispatch and waits. On a machine that is also running your notebook, that is the difference you feel.

DuckDB's sort is the mirror image of Polars'. On a plain float64 sort it is the slower of the two, 294 ms in-engine against 133 ms. Give it two keys and it becomes the fastest CPU engine in the table: 265 ms for the int32 lexsort where Polars lazy needs 903 ms and pyarrow 7.8 s. It also takes the float64 argsort, 344 ms against 420 ms. Both pay in CPU time: 1,284 CPU-ms for Polars' float64 sort, 3,866 for DuckDB's.

The GPU's ratio over the best CPU engine widens with the work the sort has to do: 4.1x Polars on a plain float64 sort, 7.7x Polars on int64 argsort, 8.6x DuckDB on float64 argsort, 7.0x DuckDB on the two-key lexsort. Sorting strings at 10,000,000 rows is 3.5x DuckDB, and it is the one sort row where ArrowMetal's own CPU time is not negligible (7.13 ms), because variable-length keys need host-side preparation before the radix passes.

Group-by: DuckDB is the CPU engine to beat at low cardinality

group-by, int32 key	Polars ms (CPU-ms, cores)	DuckDB ms (CPU-ms, cores)	pyarrow ms (CPU-ms, cores)	ArrowMetal ms (CPU-ms)	GPU ratio
sum, 1,000 groups 50,000,000 rows	81.93 (1,186, 14.5c)	9.00 (135, 15.0c)	18.45 (250, 13.6c)	4.89 (1.23)	1.8x
count, 1,000 groups 50,000,000 rows	89.01 (810, 9.1c)	8.35 (126, 15.1c)	17.38 (235, 13.5c)	4.23 (1.23)	2.0x
sum, 100,000 groups 50,000,000 rows	99.43 (1,461, 14.7c)	75.9 (1,130, 14.9c)	46.39 (542, 11.7c)	7.41 (1.25)	6.3x
count, 100,000 groups 50,000,000 rows	121 (1,060, 8.8c)	64.9 (974, 15.0c)	38.29 (421, 11.0c)	4.92 (1.26)	7.8x
sum, 10,000,000 groups 50,000,000 rows	322 (4,585, 14.3c)	149 (2,210, 14.9c)	1,303 (3,682, 2.8c)	46.12 (2.16)	3.2x
sum, two int32 keys, ~1,024 groups 50,000,000 rows	164 (2,216, 13.5c)	13.9 (199, 14.3c)	23.9 (322, 13.5c)	5.91 (2)	2.3x
sum, 1,000 groups 10,000,000 rows	20.24 (242, 12.0c)	2.09 (28, 13.4c)	4.61 (50.9, 11.0c)	1.73 (1.03)	1.2x

This is the table that changed most when DuckDB joined. Summing 50,000,000 rows into 1,000 groups takes 9.0 ms in DuckDB on 15 cores: twice as fast as pyarrow's threaded Acero (18.45 ms), nine times faster than Polars (81.93 ms), and within 1.8x of the GPU's 4.89 ms. Count into 1,000 groups is 8.35 ms against 4.23 ms; a sum over two int32 keys is 13.9 ms against 5.91 ms. These are the narrowest ratios in the post: if low-cardinality aggregation is your whole workload, DuckDB on the CPU is already close.

Cardinality changes the picture. At 100,000 groups the fastest CPU engine is Acero again (46.4 ms; DuckDB 75.9 ms; Polars 99.4 ms) and the GPU is 6.3x at 7.41 ms. At 10,000,000 groups, where the key is nearly unique, DuckDB in-engine is the fastest CPU at 149 ms, ahead of Polars lazy at 322 ms and pyarrow at 1.3 s, and the GPU is 3.2x at 46.12 ms. Polars' group-by trails on every cardinality except the highest; I report it as measured, group_by().agg() on an in-memory frame, eager and lazy both tried. What does not narrow is the CPU time: DuckDB's 9.0 ms for the 1,000-group sum is 135 CPU-ms on about 15 cores; the GPU's 4.89 ms is 1.23 CPU-ms.

Does the GPU help at 10 million rows, or only at 50?

At 10,000,000 rows the sorts hold their shape: the float64 sort is 6.85 ms against 26.16 ms for Polars (3.8x) and 54.6 ms for DuckDB; the lexsort is 7.29 ms against 49.9 ms for DuckDB (6.8x). The low-cardinality group-by does not: the 1,000-group sum is 1.73 ms on the GPU and 2.09 ms in DuckDB, 1.2x, a tie for practical purposes. At 100,000 groups the GPU is back to 8.4x over Acero.

Go smaller and the fixed cost of a dispatch takes over. The project's to-improve list publishes the same 1,000-group sum at small sizes: 0.20x at 1,000 rows, where pandas finishes in 0.08 ms and the GPU cannot start in under 0.40 ms; 0.51x at 100,000 rows; 0.58x at 1,000,000 rows. The crossover for group-by on this machine is somewhere between one and ten million rows. Sort was only measured at 10,000,000 and 50,000,000, so I will not put a crossover number on it.

Where the CPU is still ahead

Three kinds of rows at 50,000,000 rows belong to the CPU today, and they sit in the matrix next to the wins.

• Short chains that Polars lazy fuses. Group-by after filter is 0.72x (6.73 ms on the GPU, 4.85 ms for Polars lazy); filter on two columns then sum is 0.83x; a filter that keeps 90% of rows is 0.87x. Polars runs those as one fused pass; ArrowMetal 0.1.0 runs two or three dispatches. Pipelining the dispatch floor is the next item on the project's list.
• Single-pass operations already at memory bandwidth. A sum over int64 with 10% nulls is 1.11 ms on the GPU and 2.30 ms for Polars lazy, 2.1x, and there is little room left on either side.
• Anything under about a million rows. If your frames are that size, none of this applies to you and Polars or DuckDB is the right call.

The matrix has 339 rows: 145 at or above 3x, 102 between 1x and 3x, and 77 where the fastest CPU idiom is ahead, each filed under its cause. Read that page rather than this one if you are deciding for a particular operation.

Which one should you use?

The advice from my January Polars vs DuckDB post still stands for the CPU half, with one number added. Polars when the work is a dataframe pipeline in one process; it is the faster of the two on a plain sort. DuckDB when the work is SQL or larger than memory, and now also for low-cardinality aggregation, where it is the fastest CPU engine here by a wide margin and within 2x of the GPU. The GPU is a third option, not a replacement, under three conditions: the data is already Apache Arrow, the operation is a sort, a gather or a group-by with many groups over ten million rows or more, and the machine is an Apple silicon Mac. For a 1,000-group sum, use whichever engine your data is already in.

The three are not in competition because of the Arrow C Data interface: a Polars frame, a DuckDB result and an ArrowMetal array are the same bytes in the same unified memory. Keep the pipeline in Polars or DuckDB and hand the one 50,000,000-row sort or group-by to the GPU.

How to use ArrowMetal: install, first sort, group-by, Polars and DuckDB

Everything below ran on the same M4 Max on 12 September 2026 against the PyPI wheel (ArrowMetal 0.1.0, pyarrow 25.0.1, DuckDB 1.5.5, Polars 1.44.2); the comments are the actual outputs. macOS 14 or later on Apple silicon, Python 3.10 or later; pyarrow comes with the wheel.

pip install arrowmetal          # macOS 14+ on Apple silicon, Python 3.10+; pyarrow comes with it
pip install polars duckdb       # optional: the two bridges used below

import pyarrow as pa, arrowmetal as am
am.device_name()                                 # 'Apple M4 Max'

col = am.array(pa.array([1, None, 3, 40]))       # small arrays copy in; big aligned buffers are borrowed
col.sort().to_arrow()                            # [1, 3, 40, null]   GPU radix sort, stable, nulls last
col.argsort().to_arrow()                         # [0, 2, 3, 1]       int32 indices (array_sort_indices)

keys = am.array(pa.array([0, 1, 0, 2], pa.int32()))
keys.group_by(3).sum(col).to_arrow()             # [4, null, 40]      dense int32 keys, 3 groups

# Polars: a Series crosses via Arrow (zero-copy where buffers allow) and comes back
import polars as pl
s = pl.Series("x", [3.5, None, 1.25, 40.0])
am.to_polars(am.from_polars(s).sort(), "x")      # [1.25, 3.5, 40.0, null]

# DuckDB scans and joins; the GPU does the group-by; the result is a DuckDB relation
import duckdb
con = duckdb.connect()
con.sql("create table t as select (i*7919) % 1000 as k, (i % 97)::double as v "
        "from range(1000000) r(i)")
rel = am.duckdb_gpu_query(con, "select k, v from t",
    then=lambda c: {"k":     am.group_by([c["k"]]).keys()[0],
                    "total": am.group_by([c["k"]]).sum(c["v"])})
rel.order("total desc").limit(2).fetchall()      # [(120, 48111.0), (263, 48111.0)]

One thing before you time it yourself. The tables above measure the kernel with the column already in Metal memory. A single cold query that includes the crossing from DuckDB is much closer to parity: the project's DuckDB notes put a 100,000-key group-by at 16x DuckDB resident and 1.1x cold, and a sort at 2.2x resident and 0.7x cold. Keep columns resident with am.from_duckdb and wrap a chain in with am.batch(): to pay one round trip.

FAQ: Polars, DuckDB and the Apple silicon GPU

Is DuckDB faster than Polars for sorting?

Not for a plain sort: on an Apple M4 Max, 50,000,000 float64 values sort in 133 ms with Polars and 294 ms in DuckDB 1.5.5 in-engine. DuckDB wins the two-key sort (265 ms against 903 ms) and its 1,000-group group-by is nine times faster than Polars (9.0 ms against 82 ms). The GPU does the float64 sort in 32 ms.

Does the GPU help at 10 million rows, or only at 50 million?

At 10,000,000 rows the float64 sort is 3.8x Polars, but the 1,000-group sum is 1.2x DuckDB, close to a tie. Below about a million rows the CPU is ahead on group-by (0.20x at 1,000 rows, 0.58x at 1,000,000) because a GPU dispatch has a fixed floor.

When is the CPU still faster than ArrowMetal?

On small inputs, on single-pass operations already near memory bandwidth, and on short chains where Polars lazy fuses the steps: group-by after filter is 0.72x, filter plus sum 0.83x. All 77 such rows are published in the project's to-improve list.

Can I use ArrowMetal from Polars or DuckDB?

Yes, through the Apache Arrow C Data interface: am.from_polars, am.to_polars, am.from_duckdb and am.duckdb_gpu_query. The crossing is zero-copy when the producer's buffers are page aligned and arrive as one chunk.

Read next

• Polars vs DuckDB in 2026: when to pick which
• ArrowMetal: Apache Arrow compute on the Apple silicon GPU, the launch post with all 339 rows
• gpudb: a SQL engine on Apple silicon GPUs, now a DuckDB community extension