Edge Dispatch: Ryzen AI's NPU Runtime Goes Official While a Raspberry Pi Learns to See and Speak with a Tiny LLM

FastFlowLM hits 1.0 as an official AMD ROCm project — and gains a robotics model

FastFlowLM, the Ollama-style runtime for AMD's Ryzen AI NPUs, shipped its first general release as an official ROCm project on GitHub, according to a writeup by hwbusters.com. The runtime itself is 17MB, installs in about twenty seconds, and mirrors Ollama's workflow: flm run llama3.2:1b pulls a model, flm serve opens a local OpenAI-compatible endpoint. It targets XDNA2 NPUs only — Strix, Strix Halo, Kraken, and Gorgon Point — and runs inference on the NPU alone, leaving CPU and iGPU idle.

AMD's own claim of a tenfold power-efficiency edge over alternatives is self-reported and unverified here, and the runtime demands a fairly recent NPU driver (32.0.203.304 or newer), so plenty of Ryzen AI laptops sitting on stock drivers won't see any of this without an update. There's also something a little odd about a chipmaker's official NPU story being an acquired community project, though folding it into ROCm at least gives Ryzen AI one supported runtime that also covers Radeon GPUs.

The release adds SmolVLA, a vision-language-action model, to a lineup that had been LLMs and vision-language models only. That matters for the edge robotics side of this beat: a small VLA policy running on NPU silicon that isn't the CPU or GPU is the kind of always-on, low-power perception loop that could make sense on a thin laptop or a small robot controller rather than requiring a discrete accelerator.

Gemma 4 E2B drives a Raspberry Pi 5 robot's eyes, ears, and voice — fully offline

Google's developer blog published a detailed walkthrough of running Gemma on a Raspberry Pi 5 through LiteRT-LM, its on-device inference stack, to power the Reachy Mini robot's perception and speech entirely without a network connection, per developers.googleblog.com. Google's own numbers, which should be read as vendor benchmarks, show Gemma 4 E2B hitting 99 tokens/sec prefill and 9 tokens/sec decode on the Pi 5, with peak memory around 1,432MB — and an end-to-end voice output speed of roughly 27.3 characters per second, about twice normal human speaking pace.

Nine tokens per second of decode is not fast for general chat, but Gemma's tokenizer packs about 4.2 characters per token, which is why the voice pipeline still feels responsive. The architecture splits vision and language work concurrently across the Pi's CPU and GPU, which is a real distinction worth naming: this is on-device inference driving a physical robot's reactions, not a cloud call routed through a local microphone.

What matters here is the hardware floor. An $80 board running a real vision-plus-voice loop with no cloud dependency, using an openly available model and runtime, is a concrete data point for what a small robotics hobbyist or product team can actually ship today rather than promise.

Needle 2's 45M-parameter tool-caller: narrow by design, and honest about its limits

Cactus Compute's Needle 2, a 45M-parameter model built specifically for tool calling and structured extraction, continues making rounds this week, with marktechpost.com detailing how the whole model ships as a 14MB binary running a full session in roughly 28MB of RAM. Weights are trained at CQ2-bit from the start rather than quantized after the fact, and the model runs inside Cactus's own C++ engine with no separate runtime to install.

Self-reported decode throughput is 500 tokens/sec on a Raspberry Pi 5, 400 to 1,500 tokens/sec on headsets like Meta Quest 3S and Apple Vision Pro, and 300 to 700 tokens/sec on sub-$200 phones. The model leads the Seal-Tools benchmark in and out of domain but trails on the broader BFCL v4 leaderboard (42.6 overall) — a limitation the team states plainly rather than burying.

The underlying bet is that mapping a messy sentence onto a typed function call doesn't need world knowledge, so 45M parameters is enough on hardware with no GPU and no NPU. A 256-token sliding window with pinned tool sinks keeps RAM use flat instead of growing with conversation length, and built-in confidence scoring turns escalation to a cloud model into an explicit design choice rather than a silent fallback — a useful pattern for anyone building offline-first mobile agents.

Two things repeated this week: a chipmaker finally shipping usable software for NPU silicon it built years ago, and small boards proving they can run a full sense-and-speak loop with no network cable. Read the vendor numbers as vendor numbers, but the underlying hardware claims check out.