The Edge AI Compute Ladder: What Actually Runs on Every Board From $5 to $600

Every week someone asks a version of the same question: what can I actually run on this board? The answers online are a mess — listicles that put Gemma 3 270M on a microcontroller, vendor pages that call a cloud API "on-device AI," benchmark numbers copied between blogs until nobody remembers who measured them. So here is a reference instead, built rung by rung, with a price and an honest ceiling attached to each tier.

The framing comes from Derek Molloy of Dublin City University, whose July 10, 2026 essay From TinyML to Tiny Language Models lays out edge AI as a power-budget ladder: plain microcontrollers at 1–50mW doing keyword spotting, then microcontrollers with a small NPU, then single-board Linux computers running roughly 1-billion-parameter models on CPU, then those same boards with a dedicated accelerator. Price tracks power closely enough to climb the same ladder with a wallet instead of a wattmeter, which is what I'll do here.

Rung 1 — plain microcontrollers, $5 to $10

The bottom rung is a Raspberry Pi Pico 2 at $5 or an ESP32-S3 dev board at roughly $8. No operating system, no NPU — Espressif has never shipped a chip with a neural accelerator, and what gets marketed as AI acceleration on the S3 and P4 is a set of SIMD vector instructions. Three numbers define everything here: SRAM in hundreds of kilobytes, PSRAM in single-digit megabytes, flash in low tens of megabytes.

What runs beautifully at this tier is keyword spotting. microWakeWord compiles INT8 models down to LiteRT for Microcontrollers — Google's rebrand of TensorFlow Lite Micro — and detects wake words entirely on an ESP32-S3. It has shipped inside ESPHome and Home Assistant since the Voice Chapter 6 release of February 2024, which makes it the least exciting thing on this page and also the most deployed. Rung one is not a research curiosity; it is production hardware in people's kitchens.

Language models at this tier are real but narrow. On July 25, 2026, a developer going by slvDev surfaced esp32-ai: a 28.9-million-parameter TinyStories model generating at about 9.9 tokens per second on an $8 ESP32-S3 with 512KB of SRAM, 8MB of PSRAM and 16MB of flash. The 4-bit file is 14.9MB, and the trick is per-layer embeddings borrowed from Google's Gemma 3n — a dense reasoning core of roughly 560 thousand parameters lives in RAM while the 25-million-parameter embedding table stays in flash, touched at about 450 bytes per generated token. On the RP2350 side, earlephilhower's PicoLlama does the equivalent job: Karpathy's llama2.c TinyStories checkpoints as an Arduino library, requiring an RP2350 with PSRAM and a model small enough to live in onboard flash.

The ceiling matters as much as the achievement. These models write short children's stories and nothing else — CNX Software called esp32-ai "more an engineering feat than a useful LLM project," and its own author says it cannot answer questions or follow instructions. Four-bit quantization is not an optimization at this rung, it is the entry fee. I unpacked the full ESP32 story, including the four very different things people mean by "LLM on ESP32," in running an LLM on an $8 microcontroller.

Rung 2 — microcontrollers with an NPU, $16 to $90

Bolt a neural accelerator onto a Cortex-M and the vision workloads that crawled on rung one start running continuously. Seeed's Grove Vision AI V2, at $15.99, pairs a Cortex-M55 with an Arm Ethos-U55 microNPU — rated in the 0.1 to 0.6 TOPS range, which looks laughable next to a datacenter figure and is entirely sufficient for always-on object detection at a few hundred milliwatts. Further up sits the 01Studio CanMV K230, a RISC-V board with a 6 TOPS NPU at roughly $49 to $88.

The genuinely new thing on this rung is Alif Semiconductor's Ensemble E4/E6/E8 family, shipping since August 2025, which integrates the Arm Ethos-U85 — a 400 MHz NPU with a 128-bit local memory bus above 12 GB/s and, crucially, native support for transformer operators. This is the first microcontroller-class silicon marketed explicitly for small language models, and Arm has ported ViT-Tiny and TinyLlama-1.1B to it as demonstrations.

Read that carefully, because it is the most over-claimed spot on the ladder. A demonstrated silicon-IP port is not a shipping product with numbers behind it. No consumer dev board at this tier has published tokens-per-second for a language model in the wild, and Alif's marketing figure of 36 milliwatts while running a small language model is a vendor claim with unspecified model and workload, with no independent measurement I could find. The era of language models on microcontroller NPUs has been announced. It has not been benchmarked.

Rung 3 — Pi-class Linux boards, $15 to $80

Step from a microcontroller to a Linux single-board computer and the entire small-model ecosystem opens at once, because you get an MMU, swap space, and llama.cpp. The bottom of this rung is the $15 Raspberry Pi Zero 2 W with 512MB of RAM, where SmolLM2-135M is the standard community pick. One widely-circulated guide clocks SmolLM2-135M-Instruct at Q4_K_M around 9.5 tokens per second there — treat that as a single community benchmark rather than a settled figure, since it has not been independently reproduced and it gets routinely conflated with a far slower 0.45 tok/s number measured on the single-core original Pi Zero. Projects like ravijo's pi-llm document what genuinely fits in 512MB once swap and aggressive quantization are in play.

The famous small models live here, not below

Now the correction this article mostly exists to make. Several 2026 listicles claim Qwen3-0.6B or Gemma 3 270M "run on microcontrollers." That is false as stated. Those models want hundreds of megabytes of RAM; they need a Pi-class Linux board. Gemma 3 270M splits into 170M embedding parameters against a 256k vocabulary plus 100M transformer parameters, ships INT4 quantization-aware-training checkpoints, and is positioned by Google itself for phones and Pi-class devices — and Google's claim that it burned 0.75% of a Pixel 9 Pro battery over 25 conversations is an internal measurement with undefined conversation length, not a reproduced result. Qwen3-0.6B, the smallest dense Qwen3, is Apache 2.0 and runs through Ollama or llama.cpp on the same hardware. Liquid AI's LFM2.5-350M, pretrained to 28 trillion tokens and shipping official GGUF builds under 500MB on disk, targets tool use and function calling at this exact tier.

The upper edge of the rung is set by memory arithmetic. SmolLM2-1.7B at Q4_K_M needs about 1.1GB for weights plus roughly 500MB of inference overhead, making a 2GB board — not a 512MB one — its practical floor. Below that, expect single digits: Qwen2.5-0.5B on a 1GB Raspberry Pi 2 has been measured at about 1 to 2 tokens per second. For the fully-open reference point at the top of the small-model range, SmolLM3-3B (released July 8, 2025) is Apache 2.0 with its full training recipe and 100+ intermediate checkpoints public, 128k context via YARN, six languages.

Rung 4 — Linux board plus accelerator, $70 and up

The Raspberry Pi AI Kit puts a Hailo-8L — 13 TOPS at INT8 — on an M.2 HAT+ for $70, requiring a Pi 5 host of roughly $80 underneath it. Molloy's ladder places the Hailo-10H one step beyond that, running 2-to-4-billion-parameter models at conversational speed, and spells out the arithmetic that makes it possible: 4-bit quantization takes a roughly 4-billion-parameter model from about 8GB down to about 2GB.

At the top of the consumer ladder sits Sipeed's K3, a RISC-V SBC announced in May 2026 with a 60 TOPS NPU and up to 32GB of LPDDR5, priced between $299 and $639. Sipeed claims 30-billion-parameter-class models at 10 to 15 tokens per second. Those are vendor launch claims. No independent benchmark exists yet, so until third-party reviews land, read every K3 number as marketing. Same discipline as rung two: announced silicon earns attention, not belief.

Sidebar — an offline voice assistant, rung by rung

The clearest way to feel the ladder is to build one thing that spans it. A fully offline voice assistant has four parts, and they do not all live on the same rung.

Wake word sits on rung one. microWakeWord's INT8 models run on an ESP32-S3; openWakeWord, the larger framework, wants Pi-class hardware — that gap is precisely why microWakeWord exists. Speech recognition starts at 27 million parameters with Moonshine from Useful Sensors, which per its paper cuts word error rate roughly 48% against a same-size Whisper Tiny and scales compute with actual clip length instead of padding everything to Whisper's fixed 30-second window; Arabic, Chinese, Japanese, Korean, Ukrainian and Vietnamese flavors arrived through February 2026. Its "up to 5x faster than Whisper" headline is the developers' own benchmark, and depends heavily on clip length.

Text to speech is the rung-three anchor. Piper runs in real time on Pi-class CPUs across 30+ languages and is Home Assistant's default TTS — but it is not microcontroller-class, which is why ESP32 voice satellites stream synthesized audio down from a hub rather than generating it. Note the relocation, since many tutorials still point at the wrong repository: the original rhasspy/piper was archived on October 6, 2025, and development now lives at the Open Home Foundation's piper1-gpl, at v1.6.0 as of July 2026. The language model in the middle is what sets your rung: hearing a wake word and producing a transcript takes $8 of hardware; answering the question takes $15 to $70 more.

The ladder keeps sliding downward

The useful thing about laying the rungs side by side is watching techniques fall through them. Per-layer embeddings were designed for Gemma 3n, a phone-class model; within a year they were the reason 28.9 million parameters fit on an $8 microcontroller. Ternary quantization in the BitNet style was a research direction aimed at large models; it now underpins a 180.9-million-parameter mixture-of-experts running on an ESP32-P4. Transformer operators were something you assumed a GPU had; Ethos-U85 put them inside a microcontroller NPU. Every one of those started at least two rungs above where it now lives.

So the buying advice is simple, with a shelf life built in. Pick the rung that does the job you need today — wake words and vision are solved at $5 to $16, a small assistant that reasons needs the Linux boards, anything conversational at a few billion parameters needs an accelerator. Then assume the technique currently holding your model up gets smuggled one rung down before you finish the project. That has been the pattern for three straight years running.