A Polars GPU Engine for Apple Silicon, Now on M1 Macs Too: ArrowMetal 0.6.1 Runs Polars 2.1 and Ships Its First Outside Contribution

On the evening of 2 November, issue #1 was opened on ArrowMetal. The reporter was on an M1 Max, and every sort of 64-bit keys failed to build its GPU pipeline. They did not stop at the report: they found the exact construct the M1's Metal compiler aborts on, wrote a twelve-line kernel that reproduces it, found the form that compiles, ran the suites, and sent the pull request thirty-six minutes later. It is merged this morning as ArrowMetal 0.6.1, the project's first outside contribution. The same release brings the Polars engine to polars 2.1.0 and, as a side effect nobody planned, turns GitHub's CI runner into a real GPU test gate.

Tests and the runner: ArrowMetal v0.6.1, 8 November 2026. Polars rows: Apple M4 Max, polars 2.1.0, 50,000,000 rows, best of 5, AC power, 6 November 2026.

ArrowMetal is my Apache-2.0 library that runs Apache Arrow compute on the Apple silicon GPU through Metal: Arrow buffers stay in unified memory, the GPU reads them in place, and results come back as Arrow. Its Polars GPU engine is the Mac counterpart of the NVIDIA-only one Polars ships: collect(engine=MetalEngine()), the same frames, no CUDA. It also has a DataFusion optimizer rule, DuckDB extensions and seven language bindings. Until this week every GPU number it published came from one machine, my M4 Max.

Issue #1: an M1 Max, a twelve-line reproducer, and the fix

The report, from GitHub user d-polikhranidi, is the kind a maintainer hopes for and rarely gets. On their M1 Max every sort of Int64, Double, timestamp, utf8 and binary keys failed with Metal pipeline creation failed: radix_histogram, and so did two group-by paths. They traced it to Apple's shader compiler aborting on one shape of code, a 64-bit value assembled with shifts from two threadgroup atomic_uint loads, showed which form compiles instead, fixed the three kernels that did it, ran the suites on the M1 Max (160 tests, 0 failures) and sorted a 11.9-million-row CSV to check the order on the CPU.

// 0.6.0 — the join the M1 Max's GPU compiler aborts on: two threadgroup atomics, shifted together
ulong v = (ulong)atomic_load_explicit(&lo, memory_order_relaxed)
        | ((ulong)atomic_load_explicit(&hi, memory_order_relaxed) << 32);
// -> Metal pipeline creation failed: radix_histogram on Apple M1 Max:
//    Compilation failed due to an interrupted connection: XPC_ERROR_CONNECTION_INTERRUPTED

// 0.6.1 (#2) — the same 64 bits, and it compiles on the M1 Max, the M4 Max and GitHub's runner
ulong v = as_type<ulong>(uint2(atomic_load_explicit(&lo, memory_order_relaxed),
                               atomic_load_explicit(&hi, memory_order_relaxed)));

I added the fourth kernel with the same shape, gs_b_priv, which happened to pass on their machine but had no business relying on it, and checked the two forms against each other on the M4 Max over 65,536 word pairs, edge words first, then random ones: identical bit for bit. The table has the whole of it.

Why had the suite never seen it? Because the kernels were written and tested where they ran, on an M4 Max whose compiler accepts all three forms. That is a reasonable way to build a GPU library alone, and exactly the gap a second machine closes. ArrowMetal is now verified at both ends of the Apple silicon line, by two people, and the upstream tracker records the compiler behaviour as an Apple Metal finding, worked around in 0.6.1, with the reproducer. One other M1 difference stays on the list: a float32 logb at one row, 231 ulp off, the same before and after the fix.

The side effect: GitHub's runner now builds every kernel

Since October, GitHub's macos-15 runner had refused to build the 64-bit-key radix_histogram in every form I tried; its virtual GPU reported Compilation failed and nothing else. So the pull-request conformance job, 2,000 cases sampled from the differential matrix, ran few of its GPU cases: on 6 November, with 0.6.0's kernels, 496 passed and 1,491 were skipped on pipeline failures. The runner and the M1 Max had been refusing the same construct all along.

With 0.6.1's kernels, the run of 8 November built every kernel its 2,000 cases asked for: 1,868 pass, 81 documented divergences, 0 unclassified, 51 not implemented for the type, 0 pipeline failures. One contributor's fix made the CI gate real; TESTING.md section 7 now describes a job that tests the GPU rather than one that mostly skips it.

No speed paid on M4 Max

The as_type join is the same 64 bits as the shift join on a little-endian GPU, so the compiler should emit the same code; the question was whether it does. I ran the matrix's sort, group-by and window families at 50,000,000 rows on the M4 Max in one session: the 0.6.0 wheel's kernels, then 0.6.1's twice, 63 rows each.

The median of 0.6.1's time over 0.6.0's is 1.003 in the first run and 0.991 in the second; the differences on any one row are the few percent two back-to-back runs of the same kernel always show, and every answer agreed with the CPU libraries in all three runs. The fix is free.

Polars 2.1.0 on the Apple silicon GPU

Polars 2.1.0 shipped on 25 October with a new plan IR major, (16, 0). MetalEngine fails closed on an IR major it has not been tested on, so on 2.1.0 it left every plan to Polars until the IR was checked. Checked, it is IR 15's node kinds plus two additions: a truncating division, which stays with Polars, and a predicate on the join node. Polars 2.1 lets a filter comparing columns from both sides, join(...).filter(pl.col("x") < pl.col("y")), ride inside the join as a fused predicate. An engine that read only the join keys would hand back every matching pair and silently drop the filter. The engine runs such a node as the join, then the filter; any other join kind with a predicate stays with Polars. Here it is from a fresh install of the 0.6.1 wheel, with the engine's own report:

$ pip install "arrowmetal[polars]"       # 0.6.1 from PyPI; the extra now installs polars>=1.44,<2.2 -> 2.1.0
$ python -m arrowmetal.polars_engine check
polars 2.1.0: a tested release
IR version (16, 0): a tested major (tested: (14, 7), (15, 2), (16, 0))
crossover table: arrowmetal._engine_crossovers_pl2 (polars 2.0.0)

# 50,000,000 rows (k, x) joined to 1,000,000 rows (k, y), then a filter over BOTH sides —
# Polars 2.1 folds the filter INTO the join; the engine runs the join, then the filter
q = left.lazy().join(right.lazy(), on="k").filter(pl.col("x") < pl.col("y")).select(pl.len())
print(engine.explain(q))

    INNER JOIN:  FUSED PREDICATE: col("x") < col("y")          <- Polars 2.1's own plan
MetalEngine report for explain (polars 2.1.0, IR (16, 0))
  metal:  SimpleProjection#7 [SimpleProjection > Join > DataFrameScan > DataFrameScan] over 51,000,000 rows
          rule: 51,000,000 input rows is at or above the 1,942,068-row crossover for join:inner
  polars: Select#4: rule: aggregate:count was not measured ahead of Polars up to 50,000,000 input rows

# best of 5: polars 2.1.0 collect() 70.18 ms; collect(engine=MetalEngine()) 27.82 ms; equal: True; 25,012,296 rows kept

Then the numbers. Polars 2.x uses the crossover table fitted on 2.0.0, and the default benchmark at 50,000,000 rows on the M4 Max says it holds on 2.1.0: the default takes the same 52 of 107 queries, each 1.31x to 10.31x faster than Polars 2.1.0's faster engine, none behind, every frame equal to Polars'.

Read the CPU column too. The first row is 2,406 CPU-milliseconds on sixteen cores against 10 on the GPU side, for the same answer in a tenth of the wall time; while the GPU does the group-by, the cores are free for the rest of the program. Four queries the table leaves to Polars are now 1.18x to 1.24x faster on the GPU on 2.1.0: headroom for the next fit. The engine conformance grid also ran on 2.0.0 and 2.1.0 for the first time, 15,376 cases, 0 unclassified on all three Polars versions (FINDINGS Round 24).

Also this week: 0.6.0

Two days earlier, 0.6.0 shipped something a reader prompted: under the Polars post on Medium, someone asked how the engine's group-count probe behaves on heavily skewed keys. I measured it. On keys concentrated on a few values Polars' streaming engine, with its hash table in cache, is the right tool, so the probe now reads the concentration off its sample and leaves such a group-by to the CPU. On polars 1.44.1 the default picks the faster engine in 68 of 72 skewed cases (was 59) and never runs the GPU where it is slower (was 13); uniform keys are unchanged (Round 23).

Install

• Python: pip install "arrowmetal[polars]" installs 0.6.1 and a Polars from 1.44 to 2.1; python -m arrowmetal.polars_engine check prints the Polars, the IR and the table in force.
• Rust: arrowmetal = "0.6.1" on crates.io; for DataFusion 55.1, datafusion-arrowmetal = "0.6.1".
• macOS 14 or later on Apple silicon, verified this week on an M1 Max and an M4 Max; python -m arrowmetal.bench measures yours in under 30 seconds, and a results CSV from any other chip is welcome on the tracker.

My thanks to d-polikhranidi, whose first issue on this project came with its own fix, its own tests and a reproducer Apple could act on. A library that runs on one person's machine is a prototype; one that runs on a machine its author has never touched is software. 0.6.1 is where ArrowMetal became the second thing.

FAQ: a Polars GPU engine on Apple silicon, M1 support and Polars 2.1

Is there a Polars GPU engine for Mac and Apple silicon?

Yes. Polars' own GPU engine (engine="gpu") is NVIDIA cuDF and runs on CUDA only. ArrowMetal's MetalEngine is a Polars engine for the Apple silicon GPU through Metal: pip install "arrowmetal[polars]", then collect(engine=am.MetalEngine()). It takes the plans its measured crossover table says the GPU wins and leaves the rest to Polars, so every frame equals Polars' own.

Does ArrowMetal run on M1 Macs now?

Yes. 0.6.1's 64-bit sorts and group-bys build their GPU pipelines on an M1 Max, where 0.6.0's did not, and the sort, group-by, top-k and window suites pass there, 160 of 160. That is one M1 Max, verified by the contributor, plus the M4 Max every published number comes from. macOS 14 or later on Apple silicon, as before.

What is a fused-predicate join and why does it matter?

Polars 2.1 lets a filter comparing columns from both sides of a join ride inside the join node. An engine that read only the keys would return every matching pair and drop the filter. ArrowMetal's engine runs the join, then the filter; the run in this post keeps 25,012,296 of 50,000,000 rows, equal to Polars.

Read next

• Polars 2.0 is faster. With ArrowMetal on the Apple silicon GPU, 52 of 107 queries go 1.35x to 7.94x faster still
• Apple's GPU has no 64-bit floats. I made it sort 50 million doubles anyway
• Apache Arrow and Apache DataFusion now list ArrowMetal
• ArrowMetal: Apache Arrow compute on the Apple silicon GPU through Metal