Spark GPU Acceleration on Apple Silicon Macs: A Spark SQL Extension Runs GROUP BY 2.3x to 2.6x Faster Through ArrowMetal
Apache Spark can now use the Apple silicon GPU. A Spark SQL extension runs the filters, projections and partial GROUP BY above a Parquet scan as one GPU stage through ArrowMetal, with the SQL unchanged: 2.3x to 2.6x on 1,000,000 groups at 50,000,000 rows on an M4 Max, 1,300 random queries with 0 different answers, and everything it does not translate stays on Spark.
Spark GPU acceleration has meant one thing: an NVIDIA card. The RAPIDS Accelerator for Apache Spark is CUDA, and a Mac has no CUDA, so the GPU in every Apple silicon laptop has sat idle while Spark ran on the cores beside it. As of this week ArrowMetal has a Spark SQL extension that changes that. Set one configuration key, and the filters, projections and partial GROUP BY directly above a Parquet scan run on the Apple silicon GPU as one stage, translated with Spark's own semantics. The SQL does not change. On an M4 Max at 50,000,000 rows, a 1,000,000-group GROUP BY runs 2.3x to 2.6x faster than Spark's own operators, 1,300 random queries came back with the same rows as plain Spark, and everything the extension cannot translate exactly stays where it was.
Apple M4 Max, 16 CPU cores; Spark 4.3.0 in local mode; median of five runs a side after two warm-ups, sides alternating, each side at its better partition count; 22 November 2026.
ArrowMetal is my Apache-2.0 library that runs Apache Arrow compute on the Apple silicon GPU through Metal: Arrow buffers stay in unified memory, the GPU reads them in place, and results come back as Arrow. It already sits inside DataFusion as an optimizer rule, inside Polars as an engine and inside DuckDB as an extension. Spark is the fourth engine, and the first one that lives on the JVM. That is the part worth writing about, because the obvious way to do it would have been wrong.
What the Spark extension does
Spark plans a GROUP BY over Parquet as a scan, then the filters and projections, then a partial (map-side) HashAggregate per partition, an exchange, and a final aggregate. The extension replaces the middle of that: the whole chain from the scan's column batches up to and including the partial aggregate becomes one ArrowMetal plan per partition, a MetalStage in the physical plan. Spark still reads the Parquet, still shuffles, still finishes the aggregate. The GPU does the part that touches every row.
# build the extension from a checkout (JDK 22 or newer; a Python with pyspark installed)
JAVA_HOME=<JDK 22+> PYTHON=<python with pyspark> spark/build.sh # -> spark/build/arrowmetal-spark.jar
export ARROWMETAL_LIB=$PWD/.build/release/libArrowMetalC.dylib
# enable it: the SQL does not change
pyspark --jars spark/build/arrowmetal-spark.jar \
--conf spark.sql.extensions=org.arrowmetal.spark.ArrowMetalExtensions \
--conf spark.driver.extraJavaOptions=--enable-native-access=ALL-UNNAMED
# what the stage takes is in the plan
>>> spark.sql("SELECT k_long, SUM(v_double) FROM t GROUP BY k_long").explain()
== Physical Plan ==
AdaptiveSparkPlan ...
+- HashAggregate(keys=[k_long], functions=[sum(v_double)]) <- Spark's final (reduce-side) aggregate
+- Exchange hashpartitioning(k_long, 200)
+- MetalStage [filter/project/partial-aggregate on the GPU] <- ArrowMetal, one plan per partition
+- FileScan parquet t[k_long, v_double]
One translator, SparkToArrowMetal.scala, writes Spark's expressions into ArrowMetal's lazy plan. Where Arrow's semantics differ from Spark's, it writes the expression Spark's way or refuses it. Spark's float comparisons put NaN equal to NaN and above every other value, so every float comparison is written that way, with a null input giving null. AND and OR are three-valued. IF and CASE WHEN take the ELSE branch on a null condition. x IN (...) is null when x is null. And integer arithmetic under ANSI mode, Spark 4's default, stays on Spark entirely: Spark raises on overflow and the GPU wraps, and a sum that is wrong only on the day a column overflows is the worst kind of fast.
| runs on the GPU | stays on Spark | |
|---|---|---|
| Column types | byte, short, int, long, float, double, boolean, string, date, timestamp | decimal, binary, nested types |
| Predicates | = <> < <= > >= on numbers, dates, timestamps, booleans; string = literal; AND OR NOT; IS [NOT] NULL; IN lists; LIKE 'abc%', '%abc%'; startswith, contains | string ordering, LIKE '%abc', patterns with _ |
| Expressions | + - * and unary minus on floats (on integers only without ANSI); / by a non-zero literal; abs; widening casts; coalesce; IF; CASE WHEN | integer arithmetic under ANSI, division by a column, narrowing casts, string-valued expressions |
| Partial aggregates | SUM of integers and floats (longs only without ANSI), COUNT(*), COUNT(col), MIN/MAX of integers and dates, AVG; any number of non-string keys, or one string key, or none | MIN/MAX of floats, DISTINCT, FILTER, several keys with a string among them |
| Sort (opt-in) | one key of type long, int, double, float, date, timestamp; either direction, nulls first or last | several keys, other key types |
A query that uses something in the right-hand column is not refused; the stage simply ends below that operator and Spark takes over, so the extension can be left on for a whole workload. In the fuzz run below, 1,081 of 1,300 random queries had a GPU stage in their executed plan, and the other 219 ran on Spark and were compared the same way.
How Java reaches the GPU with no glue
ArrowMetal has a C ABI, and every binding so far, Python, Rust, Go, TypeScript, R, calls it. The JVM way to do that for twenty years was JNI: a compiled shim per platform, build tooling, and a crash surface of its own. The extension has none. ArrowMetalNative.java calls the dylib directly through Java's foreign function and memory API, final since JDK 22: am_import, am_plan_run, am_argsort_ex2, am_export. The Arrow C Data Interface structs are laid out in off-heap memory by hand, 72 and 80 bytes, and their release callbacks are upcall stubs. Spark 4.3.0 runs on JDK 27 with --enable-native-access=ALL-UNNAMED, and that flag is the whole of the build complexity.
The column batches Spark's Parquet reader produces are copied column by column into off-heap Arrow columns, in chunks of 16,000,000 rows or 1 GiB, whichever comes first. That chunking is a design choice with a receipt: gathered on the Java heap, a 100,000-group query over 50,000,000 rows that plain Spark completes at its default 1 GB driver heap ran out of memory with the plugin. Off-heap in chunks, it completes with the plugin and without it (spark_heap_2026-11-22.txt). A string column that is only carried to the output never goes to the GPU at all; it stays on the JVM and is joined back by row number.
One more thing the first version got wrong, for anyone building a columnar Spark operator: Spark inserts ColumnarToRowExec between a columnar scan and every operator that does not declare columnar input. An operator substituted in preColumnarTransitions that calls executeColumnar() on its child gets the converter instead of the scan and fails with "column support mismatch". Substituting in postColumnarTransitions, on the pattern HashAggregateExec(ColumnarToRowExec(scan)), takes the scan's batches directly. FINDINGS Round 26 has the rest of the list.
The same answer, 1,300 times
A GPU stage that is fast and occasionally different is a liability, so the first measurement is not speed. spark/fuzz.py writes a 1,000,000-row Parquet table built to hurt: nulls, NaN, -0.0 next to 0.0, both infinities, the int32 extremes, non-ASCII strings, and byte, short, date and timestamp columns. It generates random queries over it, runs each with and without the plugin, half under ANSI, and compares the rows as multisets, double sums and averages to 1e-9 relative because the summation order differs, everything else exactly, with -0.0 kept apart from 0.0.
$ python spark/fuzz.py --queries 500 --seed 12
ArrowMetal Spark plugin fuzz: Apple M4 Max, macOS 27.2.1; Spark 4.3.0, Java 27.0.1.1, ArrowMetal 0.6.1; 500 queries, seed 12, 1,000,000 rows
500 queries: 490 with identical rows, 10 where both raise, 0 different, 0 harness errors;
426 ran a MetalStage on the GPU (global 115/126, group 192/242, select 119/132)
$ python spark/check.py check # 25 sorted key columns against Spark's own sort
k_double asc nulls first ORDER BY (global) identical
k_double desc nulls last sortWithinPartitions identical
k_date asc nulls first ORDER BY (global) identical
...
MetalSortExec in the plan: True; 25 of 25 identical to Spark's sort
Three runs on the published code, 300 + 500 + 500 queries: 1,277 with identical rows, 23 where both sides raise the same way, 0 different (spark_fuzz_2026-11-22_published.txt). Separately, spark/check.py compares 25 sorted key columns, in order, with Spark's own sort, five key types, both directions, both null placements, global ORDER BY included: 25 of 25 identical. Three GROUP BY queries over 20,000,000 Parquet rows, compared group by group: 3 of 3.
The fuzz earned its keep before it passed. An earlier build failed 76 of 300 queries, every one on a byte or short column: imported to the GPU, never read back. Its first 300 queries found 20 errors from two limits of the plan engine on strings. Those are the bugs that make a release note, and they were found by a script instead of a user.
What was measured, and what is off by default
The timing protocol is the one I would want to read: two warm-up runs, then five timed runs a side with the side that goes first alternating, median reported, and each side at the better of the partition counts tried, because Spark at 16 tasks and the plugin at 6 is not the same comparison as both at 16. The first recordings of the day were best-of-two with Spark first on a cold JVM, and their largest row said 3.38x; the fair protocol says 2.3x to 2.6x, and that is the number in the README.
| query, 50,000,000 Parquet rows | Spark 4.3.0 | with ArrowMetal | ratio | default |
|---|---|---|---|---|
SUM, COUNT, MIN, MAX, 1,000,000 groups | 2.35 s | 1.01 s | 2.3x | on |
SUM(double), 1,000,000 groups | 1.40 s | 0.53 s | 2.6x | on |
| revenue by part, 100,000 groups | 0.88 s | 0.54 s | 1.6x | on |
SUM, COUNT, MIN, MAX, 1,001 groups | 0.55 s | 0.49 s | 1.1x | on |
SUM(double), 1,001 groups | 0.31 s | 0.29 s | 1.1x | on |
| pricing summary, 4 groups (TPC-H Q1 shape) | 0.39 s | 0.40 s | 1.0x | on |
filtered revenue, no GROUP BY (TPC-H Q6 shape) | 0.13 s | 0.23 s | 0.6x | opt-in |
sortWithinPartitions, long key | 0.58 s | 0.45 s | 1.3x | opt-in |
sortWithinPartitions, double key | 0.57 s | 0.69 s | 0.8x | opt-in |
ORDER BY, long key | 2.59 s | 2.69 s | 1.0x | opt-in |
ORDER BY, double key | 2.64 s | 2.98 s | 0.9x | opt-in |
Read it in two halves. The top six rows are what the extension does out of the box: the win grows with the number of groups, because Spark's own hash aggregate slows as its table outgrows cache, from 0.55 s at 1,001 groups to 2.35 s at 1,000,000 on the same file, while the GPU stage goes from 0.49 s to 1.01 s. On a handful of groups the two are level, and the plugin stays on because it costs nothing.
The bottom five rows are off by default, and each is off because of the number next to it. An aggregate with no GROUP BY over 50,000,000 rows takes Spark 0.13 s; copying the columns off-heap costs more than the arithmetic saves, so it measures 0.6x and is opt-in. The sort takes rows rather than columns, because Spark's SortExec hands it rows, so every key is read row by row into an Arrow column: 0.8x to 1.3x at Spark's parallelism, opt-in. When a single task sorts all 50,000,000 rows, the GPU sort is 1.3x to 1.5x ahead on both key types tried, which is the shape the one-partition rows of the sort file record. Both switches are one configuration key each, and docs/SPARK.md keeps the numbers that argue for leaving them off.
Where this stands
It is on main today, in spark/, about 1,200 lines of Scala and Java, built from a checkout against Spark 4.3.0. It is not in the 0.6.1 wheel or crate, and it will be in the next release. It runs in local mode on one Mac; a cluster of Macs is a thing people build, and the extension does not care which executor it is on, but I have not measured that and will not describe it until I have. Parquet scans only: ORC has no test. Decimals, binary and nested types stay on Spark. And every number here is from one M4 Max; the M1 Max that verified 0.6.1 has not run Spark yet.
What I wanted from this week was narrower than "Spark on the GPU", and I think it landed: a Spark operator that is exactly Spark's answer, measured honestly against Spark at its best, with the slow cases named and switched off rather than hidden. The RAPIDS Accelerator exists because Spark's heaviest stage is a map-side aggregate over a columnar scan and a GPU is good at exactly that. The same stage exists on every Mac. Now it runs there.
FAQ: Spark GPU acceleration on Apple silicon
Can Apache Spark use the GPU on a Mac?
Yes, as of this week. ArrowMetal's Spark SQL extension runs the filters, projections and partial GROUP BY directly above a Parquet scan on the Apple silicon GPU through Metal, with the SQL unchanged. Until now Spark GPU acceleration meant the RAPIDS Accelerator, which runs on NVIDIA CUDA GPUs only; there was nothing for Apple silicon.
Do I have to change my Spark SQL or DataFrame code?
No. The extension is set with spark.sql.extensions and a JVM flag that allows native access. It replaces operators in the physical plan where it has an exact translation, and whatever it does not translate stays on Spark's own operators, so every query completes with Spark's answer.
Is the Spark extension in the ArrowMetal 0.6.1 release?
Not yet. It is on main, built from a checkout with spark/build.sh against Spark 4.3.0, and needs a JDK 22 or newer because it calls the C ABI through Java's foreign function API. It will ship in the next release; the measured numbers in this post are from the published commit.
How fast is Spark GROUP BY on the Apple silicon GPU?
On an M4 Max at 50,000,000 Parquet rows, with each side at its better partition count, a 1,000,000-group GROUP BY runs 2.3x to 2.6x faster than Spark's own operators, 100,000 groups 1.6x, and a few groups about level. Sorts measured 0.8x to 1.3x at Spark's parallelism and aggregates without GROUP BY 0.6x, so both are off by default.
Read next
- DataFusion on the Apple silicon GPU: one optimizer rule, the same SQL, sorts 6.9x to 28.8x faster
- DuckDB GPU acceleration after 2.0: where it wins, where it does not
- Polars 2.0 is faster. With ArrowMetal on the Apple silicon GPU, 52 of 107 queries go faster still
- A Polars GPU engine for Apple silicon, now on M1 Macs too: ArrowMetal 0.6.1
- Apple's GPU has no 64-bit floats. I made it sort 50 million doubles anyway
References & Citations
- ArrowMetal (2026). README, Apache Spark in full and docs/SPARK.md at commit 89eef11 (main, 22 November 2026): build and enable, the settings table, what it takes, how a column reaches the GPU, correctness, timings. CHANGELOG, Unreleased.
- ArrowMetal (2026). docs/FINDINGS.md Round 26: the foreign function API without JNI, the
postColumnarTransitionsplacement, Spark's float order, the partition-count rule, the fair timings and what they changed, byte and short columns, the heap. - Timings: Benchmarks/results/spark_groupby_2026-11-22_16m.txt, spark_stage_2026-11-22_16m.txt, spark_sort_2026-11-22_final.txt and spark_heap_2026-11-22.txt: Apple M4 Max (16 CPU cores, macOS 27.2.1), Spark 4.3.0 local mode, OpenJDK 27.0.1.1, ArrowMetal 0.6.1 dylib, 50,000,000 rows, median of 5 runs a side after 2 warm-ups, sides alternating, 22 November 2026.
- Correctness: Benchmarks/results/spark_fuzz_2026-11-22_published.txt (three runs, 300 + 500 + 500 random queries over 1,000,000 rows, seeds 11 to 13) from spark/fuzz.py; the 25 of 25 sort columns and 3 of 3 group-by checks from spark/check.py. The code card shows the harness's own output lines, trimmed for width; the
explain()card is schematic. - Source: SparkToArrowMetal.scala, MetalStageExec.scala, ArrowMetalNative.java.
- OpenJDK (2023). JEP 454: Foreign Function & Memory API, final in JDK 22. NVIDIA. RAPIDS Accelerator for Apache Spark, the CUDA path this post contrasts with. Apache Spark, Apache Arrow and Apache DataFusion are trademarks of the Apache Software Foundation.
Subscribe to new posts from theaivibe.org
Related Posts

Why Your Streaming Pipeline Lies About Latency: Event Time vs. Processing Time
Your streaming pipeline's latency metric is probably measuring when data arrived at the broker, not when the event actually happened. That distinction quietly breaks more anal
A Polars GPU Engine for Apple Silicon, Now on M1 Macs Too: ArrowMetal 0.6.1 Runs Polars 2.1 and Ships Its First Outside Contribution
ArrowMetal's Polars GPU engine for Apple silicon now runs on M1 Macs and on polars 2.1.0: 52 of 107 queries at 50,000,000 rows, 1.31x to 10.31x faster, none behind. The M1 fix is the project's first outside contribution, issue #1 and pull request #2, with a 12-line reproducer; GitHub's runner now builds every GPU kernel.

DuckDB GPU Acceleration After 2.0: Where It Wins, Where It Doesn't, and Why the Mac Is Different
DuckDB 2.0 turned DuckDB into a server, AWS is buying the company behind it, and the GPU question is back. DuckDB 2.0 also changed the query format my GPU extension reads, and gpudb 0.7.1 adapts to it: on an Apple M4 Max, 18 of 22 TPC-H SF10 queries run on the GPU under DuckDB 2.0, 3.7x faster end to end, with every answer identical to DuckDB's. Here is where a GPU helps DuckDB, how gpudb decides, and why the Mac is the interesting case.