Data Engineering13 min read
Spark GPU Acceleration on Apple Silicon Macs: A Spark SQL Extension Runs GROUP BY 2.3x to 2.6x Faster Through ArrowMetal
Apache Spark can now use the Apple silicon GPU. A Spark SQL extension runs the filters, projections and partial GROUP BY above a Parquet scan as one GPU stage through ArrowMetal, with the SQL unchanged: 2.3x to 2.6x on 1,000,000 groups at 50,000,000 rows on an M4 Max, 1,300 random queries with 0 different answers, and everything it does not translate stays on Spark.
6 views
Read