Edge Dispatch: Raspberry Pi's GPU Joins the AI Party While a 14MB Model Learns to Call Tools

Raspberry Pi 5 gets GPU-accelerated on-device AI via LiteRT

Google AI Edge and Raspberry Pi published a joint announcement showing the Raspberry Pi 5 running language, vision, and speech models entirely locally using LiteRT, Google's on-device inference runtime, paired with the Gemma family of open models. The demo centers on the Reachy Mini robot, where object detection, speech recognition, and a Gemma 4 E2B language model run together in real time with no cloud call, according to igalia.com.

The notable part for hardware watchers is the GPU path. Igalia engineer José María Casanova enabled experimental WebGPU support in Mesa's V3DV Vulkan driver so LiteRT's WebGPU backend, via Google's Dawn implementation, can target the Pi 5's VideoCore VII GPU. In the Reachy Mini demo, object detection runs on the GPU while Gemma's language inference runs on the CPU, so the two workloads stop fighting over the same CPU cycles. It's turned on with a `V3D_WEBGPU_OVERRIDE=1` flag in the updated Raspberry Pi OS Mesa package, and Igalia is explicit that this is experimental, not production-ready.

This matters for the Pi ecosystem specifically because the VideoCore GPU has sat mostly idle for AI workloads while the CPU did everything. Splitting perception and language generation across CPU and GPU is a real architectural change, not a benchmark trick, though it's still early days for how much headroom that GPU actually offers versus, say, a Coral or Hailo accelerator card.

Needle 2: a 14MB binary that does tool-calling, nothing else

Cactus Compute released Needle 2, a 45-million-parameter model built specifically for tool calling, device actions, and structured extraction. The whole thing ships as a single 14MB binary running a full session in about 28MB of RAM, with weights trained natively at 2-bit precision using the company's Cactus Quants rather than quantized after the fact, per marktechpost.com.

Cactus reports decode speeds of 500 tokens/sec on a Raspberry Pi 5, 400 to 1,500 tokens/sec on a Meta Quest 3S and Apple Vision Pro, and 300 to 700 tokens/sec on sub-$200 phones — all self-reported figures, and all on hardware with no GPU or NPU involved. On benchmarks, Needle 2 leads Seal-Tools in-domain and out-of-domain but trails on the broader BFCL v4 test, which fits the model's stated design: it knows how to map a messy sentence onto a function signature, not how to hold a conversation or write an essay.

The interesting bet here is narrowness as a feature. Cactus says its Pebble hardware already runs Needle locally in the Index 01 app for offline voice actions. For edge devices, a model this small that reliably picks the right function call could matter more than a chatty 1B model that occasionally hallucinates an API parameter.

FastFlowLM graduates into AMD's ROCm, brings a robot model with it

FastFlowLM, the independent NPU runtime AMD acquired in July, shipped its 1.0 release this week as an official project under the ROCm organization on GitHub, according to hwbusters.com. The runtime itself is a 17MB install that runs inference entirely on XDNA2 NPUs — Strix, Strix Halo, Kraken, and Gorgon Point — leaving the CPU and integrated GPU untouched, with an Ollama-style CLI (`flm run`, `flm serve`) that speaks the OpenAI API.

The 1.0 release adds SmolVLA, a vision-language-action model, pulling robotics workloads into a lineup that was previously just LLMs and vision-language models. AMD claims better than tenfold power efficiency over unspecified alternatives — a vendor claim worth treating as such — and context windows up to 256k tokens on models like Qwen3-4B-Thinking-2507. Licensing stays permissive: orchestration code is MIT, and the hand-tuned NPU kernels are free for commercial use.

For anyone with a Ryzen AI laptop, the practical note is a driver requirement: FastFlowLM wants NPU driver 32.0.203.304 or newer, and plenty of machines still ship with older versions in the box. The real story is AMD formally standardizing on an outside project as the supported way to reach silicon that's shipped on every Ryzen AI chip since Strix Point and been mostly idle until now.

Three different answers to the same question — how much intelligence fits in how little hardware — from a Pi's idle GPU to a 14MB tool-caller to a laptop NPU finally getting official software.