Apache Arrow and Apache DataFusion Now List ArrowMetal: Arrow Compute and a DataFusion Optimizer Rule on the Apple Silicon GPU

ArrowMetal is now on two Apache project pages, and I am going to enjoy this one. Since 8 September it has been on Apache Arrow's Powered By page, alongside the libraries I learned Arrow from. Since today, 9 October, it is on Apache DataFusion's integrations list in the user guide, as datafusion-arrowmetal. Both went in the way every entry on those lists does: a pull request, read by the project's committers, merged. Here is what the two listings mean, and what the listed thing does today, because 0.5.0 shipped yesterday and both entries describe it.

Apple M4 Max, macOS 27; arrowmetal 0.5.0 from PyPI with polars 2.0.0, best of 5, 9 October 2026; the grid from docs/DATAFUSION.md, DataFusion 55.1.0, 1 October 2026.

ArrowMetal is my Apache-2.0 Arrow compute library for the Apple silicon GPU: Arrow buffers stay in unified memory, the GPU reads them in place, and the results come back as Arrow. It has a Polars engine, a DataFusion optimizer rule, DuckDB extensions and seven language bindings. The launch post covers the design; this one is about the listings.

Two listings, and what they mean

Arrow's Powered By page is the Arrow website's list of projects and products using Apache Arrow; DataFusion's is the section of its user guide for community projects that extend DataFusion. Getting on either means a committer read a description of your work, judged it accurate and on topic, and put their name on the merge. For Arrow that took one morning; for DataFusion, six days, one committer's careful review and a maintainer's merge, with a thank-you at the end. Those are the two communities ArrowMetal is built on, and being read and accepted by them is the part I will remember.

It matters in a practical way too: these are the pages people read when they ask “what runs on Arrow?” or “what extends DataFusion?”, and an accurate line there reaches the right readers. Both descriptions name what the software does and nothing it does not; that is the bar this post holds to. In fairness to both projects, a listing is not an endorsement or a code review, and ArrowMetal remains independent; the table has the texts and dates.

The Arrow side: what the entry describes

The Powered By entry says the kernels run on Arrow buffers in unified memory without a copy either way, and that data crosses through Arrow's C Data, C Stream and C Device Data interfaces. That is the architecture: a pyarrow array, a Polars Series or a DataFusion RecordBatch is handed over as Arrow, the GPU works on the same bytes, and Arrow comes back. Here is the README's first thing to run, from the 0.5.0 wheel on PyPI, then a 50,000,000-row sort through the Polars engine on Polars 2.0.0, with the engine's report.

$ pip install "arrowmetal[polars]"            # 0.5.0 from PyPI; polars 2.0.0 comes with the extra

import pyarrow as pa, arrowmetal as am          # the README's first thing to run, as installed
col = am.array(pa.array([1, None, 3, 40]))      # a pyarrow array crosses in through the C Data interface
big = col.filter_where(">", 2)                  # GPU
print(big.sum(), big.to_arrow())                # 43  [3, 40]
keys = am.array(pa.array([0, 1, 0, 2], pa.int32()))
print(keys.group_by(3).sum(col).to_arrow())     # [4, null, 40]

import polars as pl                             # 50,000,000 permuted int64 keys, a Float64 payload
engine = am.MetalEngine()                       # the measured defaults; on 2.0.0, the 2.x table
print(engine.explain(df.lazy().sort("k")))

MetalEngine report for explain (polars 2.0.0, IR (15, 2))
  metal:  Sort#1 [Sort > DataFrameScan] over 50,000,000 rows
          rule: 50,000,000 input rows is at or above the 10,000,000-row crossover for sort
                (engine table, Benchmarks/results/polars_engine_crossover_2026-10-08-polars2.csv)

# best of 5: polars 2.0.0 collect() 242.37 ms; collect(engine=engine) 43.16 ms; frames equal

am.array(pa.array(...)) is the C Data interface at work; nothing is converted. The report names the rule, the row count and the crossover file, so the decision is checkable. The 43 against 242 milliseconds is one M4 Max sorting 50,000,000 permuted int64 keys with a Float64 payload, best of 5, against Polars 2.0.0's own collect(), frames equal. The engine takes the sort from 10,000,000 rows up, where the crossover table puts it; below that, Polars sorts.

The DataFusion side: what the entry describes

The integrations entry calls the crate a physical optimizer rule that runs sorts and measured group-bys on the Apple silicon GPU through Metal. Such a rule is a hook DataFusion gives to anyone: it sees the physical plan and may replace nodes. This one replaces a full ORDER BY, and the aggregate shapes its measured table takes, with a node that runs the plan on the GPU and hands back the RecordBatches DataFusion expected. The SQL and every other node are untouched. The crate's quickstart, built from crates.io 0.5.0 against DataFusion 55.1.0 on this M4 Max: the sort it takes, the two it leaves.

# Cargo.toml: datafusion = "=55.1.0", datafusion-arrowmetal = "0.5.0"
let rule = ArrowMetalRule::new(ArrowMetalConfig::default());   // full sorts from 250,000 rows
let ctx = session_context(SessionConfig::new().with_target_partitions(4), rule.clone());
ctx.register_table("sales", mem_table)?;                       // 1,000,000 rows: region, amount, name

== SELECT name, region, amount FROM sales ORDER BY amount DESC NULLS LAST, name
ProjectionExec: expr=[name@2 as name, region@0 as region, amount@1 as amount]
  MetalExec: sort=[amount DESC NULLS LAST, name ASC NULLS LAST]
    DataSourceExec: partitions=1, partition_sizes=[1]
TAKEN  SortExec -- input rows 1000000 (exact) vs min_rows 250000
1000000 rows; the first 3: n001395 | 0 | 1429.4285714285713 ...

== SELECT name, amount FROM sales ORDER BY amount DESC NULLS LAST LIMIT 3
LEFT   SortExec: TopK(fetch=3) -- top-k (sort with fetch 3) disabled in config

== SELECT region, count(*) AS n, avg(amount) AS mean FROM sales GROUP BY region ORDER BY region
LEFT   AggregateExec: gby=[region], aggr=[count(Int64(1)), avg(sales.amount)]
       -- count + sum_avg_f64 over 1 i64 key: the measured table takes it at no
          group count and size (results/datafusion_groupby_sweep_2026-10-01.csv)
13 rows; the first 3: 0 | 76924 | 714.7095888611585 ...

The numbers behind “measured” are in the DataFusion post and docs/DATAFUSION.md: on the M4 Max, full sorts from 250,000 to 50,000,000 rows ran 6.9x to 28.8x faster than DataFusion alone across four key types, 50,000,000 rows sorted in 126 CPU-milliseconds against 6,104 on 16 partitions, and 13,632 query pairs, rule off and on, matched with 0 mismatches. Measured on 0.4.0 in late September; 0.5.0 changes the aggregate side, below, not the sort numbers.

What 0.5.0 changed since those two posts

Yesterday's release is the one both entries now describe, and it is the best one yet. Four lines from the release notes matter here.

The Polars line closes a caveat from Wednesday's post, which said the 2.0.0 support was on main, unreleased. As of 0.5.0 it is the wheel on PyPI, which is why the card above needed only a fresh venv.

Install

• Python: pip install "arrowmetal[polars]" installs 0.5.0 and a Polars from 1.44 to 2.0; python -m arrowmetal.polars_engine check prints the Polars, the IR and the table in force. Without the extra, pyarrow is the only dependency.
• Rust: datafusion = "=55.1.0" and datafusion-arrowmetal = "0.5.0" in Cargo.toml; the crate links the Metal library through ARROWMETAL_LIB or the copy the Python wheel installs. docs.rs and docs/RUST.md have the finding rules.
• macOS 14 or later on Apple silicon only. Every number here is one M4 Max on AC power; python -m arrowmetal.bench measures yours in under 30 s.

ArrowMetal is an independent Apache-2.0 project; Apache Arrow and Apache DataFusion are trademarks of the Apache Software Foundation, whose projects I use and build on every day. My thanks to the committers on both sides who read the entries and let them in. Two gates, one small craft, a harbour I am glad to be moored in.

FAQ: the listings, Apache Arrow, Apache DataFusion and the Apple silicon GPU

Is ArrowMetal an Apache project?

No, and proudly independent: an Apache-2.0 project that implements Apache Arrow and extends Apache DataFusion. Both listings are entries on community-maintained lists, read and merged by the projects' committers; they are a welcome, not an endorsement.

Does Apache DataFusion ship the GPU rule?

No. datafusion-arrowmetal is a separate crate on crates.io, pinned to DataFusion 55.1.0. You register its rule on a SessionContext; the SQL and every other plan node stay DataFusion's.

Which Macs does it run on?

Apple silicon on macOS 14 or later. The Python wheel carries the Metal library; the Rust crate finds it through ARROWMETAL_LIB or the wheel's copy. The build script stops on any other target.

Are the answers the same as the CPU's?

The differential grid runs 13,632 DataFusion query pairs, rule off and on, with 0 mismatches; every Polars benchmark query is checked equal to Polars' own frame. Float64 runs through a software IEEE-754 path, bit-exact with the CPU.

Read next

• DataFusion on the Apple silicon GPU: one optimizer rule, the same SQL, sorts 6.9x to 28.8x faster
• Polars 2.0 is faster. With ArrowMetal on the Apple silicon GPU, 52 of 107 queries go 1.35x to 7.94x faster still
• ArrowMetal: Apache Arrow compute on the Apple silicon GPU through Metal