llama.cpp Teaches an NPU to Share the Load as Small AI Engines Keep Multiplying

llama.cpp learns to split work across multiple NPUs

The llama.cpp build tagged b10643 adds Hexagon backend support for multi-NPU devices (IQ9, IQ10) alongside a fully asynchronous backend, per pull request #26501. That build lands in the days following the August 26, 2026 b10636 release, part of the project's ongoing Hexagon push documented on newreleases.io.

It's a pre-release nightly, not a tagged stable version, and no independent benchmarks accompany it yet — this is plumbing, not a performance claim.

Still, matters for anyone running Snapdragon-class silicon with more than one NPU core: async dispatch across multiple Hexagon units is exactly the kind of work needed before phones and AI PCs with multi-die NPUs can actually use all their silicon for local inference instead of leaving half of it idle.

ONNX Runtime's browser backend gets generative-model muscle

Microsoft shipped ONNX Runtime WebGPU Plugin EP v0.3.0 on August 24, 2026, adding quantized KV cache support, sliding-window GQA for Gemma 4, PagedAttention, and Intel subgroup-matrix kernels, plus deferred shader compilation to cut cold-start time.

These are vendor release notes, not third-party numbers, and the WebGPU execution provider is still labeled a plugin rather than a stable core backend — browser and driver support will vary.

The point of a browser backend is inference that never leaves the tab: a quantized Gemma model running through WebGPU means no server round trip and no install, which matters for the edge case people actually hit most — a website, not a downloaded app.

A solo-built Rust engine beats llama.cpp on small models

Developer Antonello Fratepietro published benchmarks for Ferrox v0.9.1, his from-scratch Rust GGUF inference engine, run head-to-head against llama.cpp on the same Metal machine, same GGUF files. SmolLM2-135M hit 321 tok/s on Ferrox versus 214 on llama.cpp — a 50% edge — while the gap narrows toward parity as models get bigger, with Llama-3.2-3B landing at a 0.96x ratio.

These are single-machine, self-reported numbers from a young project, not an audited benchmark suite, and the advantage shrinks fast once fixed per-token overhead stops dominating.

Still, it's a useful data point: fused, purpose-built Metal kernels can beat a general-purpose engine specifically where tiny models live, which is most of what actually runs on a phone or a MacBook's efficiency cores.

A two-board offline translator built on Gemma

Maker Jdaie Lin simplified Google's open Gemma Translator reference design into a two-board build — a Raspberry Pi 5 and a Whisplay HAT+ — for fully offline speech translation, documented on Hackster on August 28, 2026.

It's a build write-up, not a vendor-audited product; latency and accuracy figures aren't independently verified beyond what's in the post.

The value is in the reduction: taking a reference stack meant for more hardware and getting it running on commodity SBC parts is exactly how offline voice pipelines spread from demos into things people actually carry.

An 80GB model, crawling on a mid-range phone

A poster on r/LocalLLaMA reported running Qwen 3.8 Flash Next, an 80GB-class MoE model, on a 12GB mid-range Android phone at 3.5 tokens per second, using low quantization on the dense part of the network and streaming the rest from storage.

This is a self-reported forum result with no shared methodology or repo, and 3.5 tok/s is far too slow for real-time chat — this is a proof of concept, not a usable app.

Even so, it marks how far selective quantization has pushed the size ceiling for phone-class inference: a model many times larger than the phone's own RAM, technically running, entirely on-device.

Runtimes keep splintering in useful directions this week — an NPU backend learning to share load, a browser engine gaining generative-model tricks, and a lone Rust project beating the incumbent at its own game on small models.