Skip to content
Back to Blog

Spark GPU Acceleration on Apple Silicon Macs: A Spark SQL Extension Runs GROUP BY 2.3x to 2.6x Faster Through ArrowMetal

Prateek SinghNovember 22, 202613 min read5 views
Spark GPU Acceleration on Apple Silicon Macs: A Spark SQL Extension Runs GROUP BY 2.3x to 2.6x Faster Through ArrowMetal

Apache Spark can now use the Apple silicon GPU. A Spark SQL extension runs the filters, projections and partial GROUP BY above a Parquet scan as one GPU stage through ArrowMetal, with the SQL unchanged: 2.3x to 2.6x on 1,000,000 groups at 50,000,000 rows on an M4 Max, 1,300 random queries with 0 different answers, and everything it does not translate stays on Spark.

Spark GPU acceleration has meant one thing: an NVIDIA card. The RAPIDS Accelerator for Apache Spark is CUDA, and a Mac has no CUDA, so the GPU in every Apple silicon laptop has sat idle while Spark ran on the cores beside it. As of this week ArrowMetal has a Spark SQL extension that changes that. Set one configuration key, and the filters, projections and partial GROUP BY directly above a Parquet scan run on the Apple silicon GPU as one stage, translated with Spark's own semantics. The SQL does not change. On an M4 Max at 50,000,000 rows, a 1,000,000-group GROUP BY runs 2.3x to 2.6x faster than Spark's own operators, 1,300 random queries came back with the same rows as plain Spark, and everything the extension cannot translate exactly stays where it was.

GROUP BY over Parquet, 1,000,000 groups
2.3x to 2.6x
faster than Spark's own operators at 50,000,000 rows; 1.6x on 100,000 groups; level on a handful
random queries, with and without the plugin
1,300, 0 different
1,277 with identical rows, 23 raising on both sides; nulls, NaN, signed zeros, infinities, int32 extremes, non-ASCII strings
lines of glue between Java and the GPU
0
Java's foreign function API calls ArrowMetal's C ABI directly: no JNI, no compiled shim
engines ArrowMetal now sits inside
4
DataFusion, Polars, DuckDB and, as of this week, Apache Spark; the SQL is unchanged in all four

Apple M4 Max, 16 CPU cores; Spark 4.3.0 in local mode; median of five runs a side after two warm-ups, sides alternating, each side at its better partition count; 22 November 2026.

ArrowMetal is my Apache-2.0 library that runs Apache Arrow compute on the Apple silicon GPU through Metal: Arrow buffers stay in unified memory, the GPU reads them in place, and results come back as Arrow. It already sits inside DataFusion as an optimizer rule, inside Polars as an engine and inside DuckDB as an extension. Spark is the fourth engine, and the first one that lives on the JVM. That is the part worth writing about, because the obvious way to do it would have been wrong.

What the Spark extension does

Spark plans a GROUP BY over Parquet as a scan, then the filters and projections, then a partial (map-side) HashAggregate per partition, an exchange, and a final aggregate. The extension replaces the middle of that: the whole chain from the scan's column batches up to and including the partial aggregate becomes one ArrowMetal plan per partition, a MetalStage in the physical plan. Spark still reads the Parquet, still shuffles, still finishes the aggregate. The GPU does the part that touches every row.

# build the extension from a checkout (JDK 22 or newer; a Python with pyspark installed)
JAVA_HOME=<JDK 22+> PYTHON=<python with pyspark> spark/build.sh      # -> spark/build/arrowmetal-spark.jar
export ARROWMETAL_LIB=$PWD/.build/release/libArrowMetalC.dylib

# enable it: the SQL does not change
pyspark --jars spark/build/arrowmetal-spark.jar \
  --conf spark.sql.extensions=org.arrowmetal.spark.ArrowMetalExtensions \
  --conf spark.driver.extraJavaOptions=--enable-native-access=ALL-UNNAMED

# what the stage takes is in the plan
>>> spark.sql("SELECT k_long, SUM(v_double) FROM t GROUP BY k_long").explain()
== Physical Plan ==
AdaptiveSparkPlan ...
+- HashAggregate(keys=[k_long], functions=[sum(v_double)])        <- Spark's final (reduce-side) aggregate
   +- Exchange hashpartitioning(k_long, 200)
      +- MetalStage [filter/project/partial-aggregate on the GPU]  <- ArrowMetal, one plan per partition
         +- FileScan parquet t[k_long, v_double]

One translator, SparkToArrowMetal.scala, writes Spark's expressions into ArrowMetal's lazy plan. Where Arrow's semantics differ from Spark's, it writes the expression Spark's way or refuses it. Spark's float comparisons put NaN equal to NaN and above every other value, so every float comparison is written that way, with a null input giving null. AND and OR are three-valued. IF and CASE WHEN take the ELSE branch on a null condition. x IN (...) is null when x is null. And integer arithmetic under ANSI mode, Spark 4's default, stays on Spark entirely: Spark raises on overflow and the GPU wraps, and a sum that is wrong only on the day a column overflows is the worst kind of fast.

runs on the GPUstays on Spark
Column typesbyte, short, int, long, float, double, boolean, string, date, timestampdecimal, binary, nested types
Predicates= <> < <= > >= on numbers, dates, timestamps, booleans; string = literal; AND OR NOT; IS [NOT] NULL; IN lists; LIKE 'abc%', '%abc%'; startswith, containsstring ordering, LIKE '%abc', patterns with _
Expressions+ - * and unary minus on floats (on integers only without ANSI); / by a non-zero literal; abs; widening casts; coalesce; IF; CASE WHENinteger arithmetic under ANSI, division by a column, narrowing casts, string-valued expressions
Partial aggregatesSUM of integers and floats (longs only without ANSI), COUNT(*), COUNT(col), MIN/MAX of integers and dates, AVG; any number of non-string keys, or one string key, or noneMIN/MAX of floats, DISTINCT, FILTER, several keys with a string among them
Sort (opt-in)one key of type long, int, double, float, date, timestamp; either direction, nulls first or lastseveral keys, other key types
Condensed from docs/SPARK.md, What it takes. Where Arrow's semantics differ from Spark's, the expression is written Spark's way or refused; see the text.

A query that uses something in the right-hand column is not refused; the stage simply ends below that operator and Spark takes over, so the extension can be left on for a whole workload. In the fuzz run below, 1,081 of 1,300 random queries had a GPU stage in their executed plan, and the other 219 ran on Spark and were compared the same way.

How Java reaches the GPU with no glue

ArrowMetal has a C ABI, and every binding so far, Python, Rust, Go, TypeScript, R, calls it. The JVM way to do that for twenty years was JNI: a compiled shim per platform, build tooling, and a crash surface of its own. The extension has none. ArrowMetalNative.java calls the dylib directly through Java's foreign function and memory API, final since JDK 22: am_import, am_plan_run, am_argsort_ex2, am_export. The Arrow C Data Interface structs are laid out in off-heap memory by hand, 72 and 80 bytes, and their release callbacks are upcall stubs. Spark 4.3.0 runs on JDK 27 with --enable-native-access=ALL-UNNAMED, and that flag is the whole of the build complexity.

The column batches Spark's Parquet reader produces are copied column by column into off-heap Arrow columns, in chunks of 16,000,000 rows or 1 GiB, whichever comes first. That chunking is a design choice with a receipt: gathered on the Java heap, a 100,000-group query over 50,000,000 rows that plain Spark completes at its default 1 GB driver heap ran out of memory with the plugin. Off-heap in chunks, it completes with the plugin and without it (spark_heap_2026-11-22.txt). A string column that is only carried to the output never goes to the GPU at all; it stays on the JVM and is joined back by row number.

One more thing the first version got wrong, for anyone building a columnar Spark operator: Spark inserts ColumnarToRowExec between a columnar scan and every operator that does not declare columnar input. An operator substituted in preColumnarTransitions that calls executeColumnar() on its child gets the converter instead of the scan and fails with "column support mismatch". Substituting in postColumnarTransitions, on the pattern HashAggregateExec(ColumnarToRowExec(scan)), takes the scan's batches directly. FINDINGS Round 26 has the rest of the list.

The same answer, 1,300 times

A GPU stage that is fast and occasionally different is a liability, so the first measurement is not speed. spark/fuzz.py writes a 1,000,000-row Parquet table built to hurt: nulls, NaN, -0.0 next to 0.0, both infinities, the int32 extremes, non-ASCII strings, and byte, short, date and timestamp columns. It generates random queries over it, runs each with and without the plugin, half under ANSI, and compares the rows as multisets, double sums and averages to 1e-9 relative because the summation order differs, everything else exactly, with -0.0 kept apart from 0.0.

$ python spark/fuzz.py --queries 500 --seed 12
ArrowMetal Spark plugin fuzz: Apple M4 Max, macOS 27.2.1; Spark 4.3.0, Java 27.0.1.1, ArrowMetal 0.6.1; 500 queries, seed 12, 1,000,000 rows
500 queries: 490 with identical rows, 10 where both raise, 0 different, 0 harness errors;
426 ran a MetalStage on the GPU (global 115/126, group 192/242, select 119/132)

$ python spark/check.py check          # 25 sorted key columns against Spark's own sort
  k_double  asc  nulls first  ORDER BY (global)     identical
  k_double  desc nulls last   sortWithinPartitions  identical
  k_date    asc  nulls first  ORDER BY (global)     identical
  ...
MetalSortExec in the plan: True; 25 of 25 identical to Spark's sort

Three runs on the published code, 300 + 500 + 500 queries: 1,277 with identical rows, 23 where both sides raise the same way, 0 different (spark_fuzz_2026-11-22_published.txt). Separately, spark/check.py compares 25 sorted key columns, in order, with Spark's own sort, five key types, both directions, both null placements, global ORDER BY included: 25 of 25 identical. Three GROUP BY queries over 20,000,000 Parquet rows, compared group by group: 3 of 3.

The fuzz earned its keep before it passed. An earlier build failed 76 of 300 queries, every one on a byte or short column: imported to the GPU, never read back. Its first 300 queries found 20 errors from two limits of the plan engine on strings. Those are the bugs that make a release note, and they were found by a script instead of a user.

What was measured, and what is off by default

The timing protocol is the one I would want to read: two warm-up runs, then five timed runs a side with the side that goes first alternating, median reported, and each side at the better of the partition counts tried, because Spark at 16 tasks and the plugin at 6 is not the same comparison as both at 16. The first recordings of the day were best-of-two with Spark first on a cold JVM, and their largest row said 3.38x; the fair protocol says 2.3x to 2.6x, and that is the number in the README.

query, 50,000,000 Parquet rowsSpark 4.3.0with ArrowMetalratiodefault
SUM, COUNT, MIN, MAX, 1,000,000 groups2.35 s1.01 s2.3xon
SUM(double), 1,000,000 groups1.40 s0.53 s2.6xon
revenue by part, 100,000 groups0.88 s0.54 s1.6xon
SUM, COUNT, MIN, MAX, 1,001 groups0.55 s0.49 s1.1xon
SUM(double), 1,001 groups0.31 s0.29 s1.1xon
pricing summary, 4 groups (TPC-H Q1 shape)0.39 s0.40 s1.0xon
filtered revenue, no GROUP BY (TPC-H Q6 shape)0.13 s0.23 s0.6xopt-in
sortWithinPartitions, long key0.58 s0.45 s1.3xopt-in
sortWithinPartitions, double key0.57 s0.69 s0.8xopt-in
ORDER BY, long key2.59 s2.69 s1.0xopt-in
ORDER BY, double key2.64 s2.98 s0.9xopt-in
Apple M4 Max (16 CPU cores, macOS 27.2.1), Spark 4.3.0 in local mode, OpenJDK 27, 50,000,000 rows read from Parquet; median of five runs a side after two warm-ups, the side that goes first alternating, written to Spark's noop sink. Each side at the better of the partition counts tried (16 and 6; 16, 4 and 1 for the sorts). From spark_groupby_2026-11-22_16m.txt, spark_stage_2026-11-22_16m.txt and spark_sort_2026-11-22_final.txt. Both answers compared on every row.

Read it in two halves. The top six rows are what the extension does out of the box: the win grows with the number of groups, because Spark's own hash aggregate slows as its table outgrows cache, from 0.55 s at 1,001 groups to 2.35 s at 1,000,000 on the same file, while the GPU stage goes from 0.49 s to 1.01 s. On a handful of groups the two are level, and the plugin stays on because it costs nothing.

The bottom five rows are off by default, and each is off because of the number next to it. An aggregate with no GROUP BY over 50,000,000 rows takes Spark 0.13 s; copying the columns off-heap costs more than the arithmetic saves, so it measures 0.6x and is opt-in. The sort takes rows rather than columns, because Spark's SortExec hands it rows, so every key is read row by row into an Arrow column: 0.8x to 1.3x at Spark's parallelism, opt-in. When a single task sorts all 50,000,000 rows, the GPU sort is 1.3x to 1.5x ahead on both key types tried, which is the shape the one-partition rows of the sort file record. Both switches are one configuration key each, and docs/SPARK.md keeps the numbers that argue for leaving them off.

Where this stands

It is on main today, in spark/, about 1,200 lines of Scala and Java, built from a checkout against Spark 4.3.0. It is not in the 0.6.1 wheel or crate, and it will be in the next release. It runs in local mode on one Mac; a cluster of Macs is a thing people build, and the extension does not care which executor it is on, but I have not measured that and will not describe it until I have. Parquet scans only: ORC has no test. Decimals, binary and nested types stay on Spark. And every number here is from one M4 Max; the M1 Max that verified 0.6.1 has not run Spark yet.

What I wanted from this week was narrower than "Spark on the GPU", and I think it landed: a Spark operator that is exactly Spark's answer, measured honestly against Spark at its best, with the slow cases named and switched off rather than hidden. The RAPIDS Accelerator exists because Spark's heaviest stage is a map-side aggregate over a columnar scan and a GPU is good at exactly that. The same stage exists on every Mac. Now it runs there.

FAQ: Spark GPU acceleration on Apple silicon

Can Apache Spark use the GPU on a Mac?

Yes, as of this week. ArrowMetal's Spark SQL extension runs the filters, projections and partial GROUP BY directly above a Parquet scan on the Apple silicon GPU through Metal, with the SQL unchanged. Until now Spark GPU acceleration meant the RAPIDS Accelerator, which runs on NVIDIA CUDA GPUs only; there was nothing for Apple silicon.

Do I have to change my Spark SQL or DataFrame code?

No. The extension is set with spark.sql.extensions and a JVM flag that allows native access. It replaces operators in the physical plan where it has an exact translation, and whatever it does not translate stays on Spark's own operators, so every query completes with Spark's answer.

Is the Spark extension in the ArrowMetal 0.6.1 release?

Not yet. It is on main, built from a checkout with spark/build.sh against Spark 4.3.0, and needs a JDK 22 or newer because it calls the C ABI through Java's foreign function API. It will ship in the next release; the measured numbers in this post are from the published commit.

How fast is Spark GROUP BY on the Apple silicon GPU?

On an M4 Max at 50,000,000 Parquet rows, with each side at its better partition count, a 1,000,000-group GROUP BY runs 2.3x to 2.6x faster than Spark's own operators, 100,000 groups 1.6x, and a few groups about level. Sorts measured 0.8x to 1.3x at Spark's parallelism and aggregates without GROUP BY 0.6x, so both are off by default.

References & Citations

Subscribe to new posts from theaivibe.org

No spam — just new posts. One-click unsubscribe.
Share this article

Related Posts