Running an LLM on an $8 Microcontroller: What's Real in 2026

On July 25, 2026, a repository called esp32-ai hit the front page of Hacker News. The claim sounded like a typo: a 28.9-million-parameter language model running entirely on an ESP32-S3 — a microcontroller that costs about eight dollars, has 512KB of SRAM, and is more commonly found blinking LEDs in smart plugs. Not streaming from a server. Not offloading to a phone. Generating text on the chip, at 9.88 tokens per second, which happens to be roughly the speed you read.

Within a week, Tom's Hardware, The Register, Adafruit, and CNX Software had all covered it. The previous record for a language model on an ESP32 was about 260 thousand parameters. This was a hundred times larger, on the same class of hardware, at interactive speed. Something real changed — and it wasn't the silicon.

The trick is memory, not compute

The ESP32-S3 in question has three tiers of storage: 512KB of fast SRAM, 8MB of slower PSRAM, and 16MB of flash. A 28.9M-parameter model quantized to 4 bits needs about 14.9MB — it fits in flash, but flash is far too slow to stream weights through on every token. That's where the developer, who goes by slvDev, borrowed an idea from Google's Gemma 3n: per-layer embeddings.

The insight is that most of a small model's parameters live in its embedding tables, and embeddings are lookups, not matrix math. So the architecture splits the model in two. A dense reasoning core of roughly 560 thousand parameters — the part that actually does computation every step — lives in RAM. The other 25 million parameters sit in flash as a memory-mapped lookup table, and generating a token touches only about six rows of it: roughly 450 bytes of flash reads per token. The chip never needs the whole model at once; it needs a sliver, and it knows exactly which sliver.

SRAM holds activations and normalization weights, PSRAM holds the core and the output head, flash holds the embedding table. It's a beautiful piece of systems engineering — the breakthrough is a memory hierarchy, not a faster processor. The lineage matters too: nearly every microcontroller LLM port traces back to Andrej Karpathy's llama2.c, a single-file C inference engine, and to DaveBben's esp32-llm, which proved in 2024 that a 260K-parameter model could run at 19 tokens per second on the same chip family. From 260K to 28.9M in two years, powered by one architectural idea.

The one almost nobody has noticed yet

Here's the part that hasn't been on any front page. On August 5, 2026, a developer going by cyfrit posted a project called p-for-llm to Hacker News. It got two points and vanished. It shouldn't have.

p-for-llm runs a 180.9-million-parameter mixture-of-experts model on an ESP32-P4 — 12 layers, 29 experts per layer with top-1 routing, ternary quantization in the BitNet style, and a vocabulary pruned down from Qwen. The board costs six to ten dollars. It generates at roughly 9 tokens per second, follows simple ChatML-formatted instructions, and even has an early tool-calling demo, which the author honestly describes as frequently going off the rails. The honest asterisk: the weights are pushed to the board over USB at startup — about 44MiB spread across flash and PSRAM — with SD-card autonomy listed as future work. The author trained it on about 12 billion tokens using a single consumer GPU.

If esp32-ai proved a microcontroller can hold a language model, p-for-llm is the first hint a microcontroller might hold a useful one. And for completeness, there's a third frontier: esp32s3-distributed-ai shards a 56M-parameter model across three ESP32-S3 boards communicating over ESP-NOW radio in 250-byte packets. It's rough — about thirty coherent words demonstrated so far — but it's a genuine distributed-inference cluster you can build for the price of a pizza.

The four tiers of "LLM on ESP32"

Search for "ChatGPT on ESP32" and you'll drown in demos. Almost none of them are what they appear to be. As of August 2026, every project in this space falls into one of four tiers, and the difference is everything:

Tier 1 — True on-device. The model's weights and computation both live on the microcontroller. esp32-ai is the flagship example. This is the hard thing.

Tier 2 — On-device compute, streamed weights. The chip does the math, but weights arrive from a host — p-for-llm's USB loading, or the wifi-llm experiment that streams a 16MB model layer-by-layer over WiFi into 300KB of usable RAM. Real inference, with an umbilical cord.

Tier 3 — Distributed microcontrollers. Multiple boards each hold a shard and pass activations over radio. Early, fascinating, mostly a research toy today.

Tier 4 — A microphone with cloud attached. The ESP32 records audio, ships it to OpenAI or Gemini or Qwen, and plays back the answer. This describes nearly every polished "AI assistant on ESP32" video you've seen — and notably, it describes Espressif's own official ESP LLM solution, which is explicitly cloud inference. Nothing wrong with it. Just don't confuse it with tiers 1 through 3.

One more myth worth killing: no ESP32 chip has an NPU. The AI acceleration on the S3 and P4 is a set of SIMD vector instructions, and Espressif's newest chip — the ESP32-S31, in mass production since July 2026 — doubles down on radios and SIMD, not neural hardware. Espressif's silicon roadmap is betting that microcontrollers will be LLM clients. The hobbyists keep proving they can be LLM hosts. That tension is one of the most interesting stories in embedded computing right now.

So what can you actually build?

Honesty time, because the coverage mostly skipped it. The 28.9M model on the S3 was trained on TinyStories: it writes short, mostly coherent children's stories, and that is all it does. It cannot answer questions, follow instructions, or look up facts — its own creator says so plainly. The bottleneck isn't the memory trick; it's that a half-million-parameter reasoning core can only reason so much. One viral post compared it to a quarter of OpenAI's first GPT by parameter count, which is true and also misleading — parameters aren't capability.

What today's tier-1 hardware genuinely supports: offline story generators, interactive badges and toys, novelty typewriters, ambient text art — anything where charming beats correct. The binding constraints are PSRAM (8MB on a standard S3, 32MB on the P4 boards) and flash, and 4-bit or ternary quantization is mandatory. Meanwhile the supporting cast is quietly excellent: wake-word detection on ESP32-S3 is production-grade (it ships in Home Assistant), and 27M-parameter speech recognition models now beat same-size Whisper variants. The pieces of a fully offline voice pipeline are converging on hardware that costs less than lunch.

And the trajectory is the point. Every technique in play — per-layer embeddings, ternary quantization, expert routing, weight tiering across a memory hierarchy — was invented for big models and is being smuggled downward. The gap between "writes cute stories" and "useful assistant" is roughly the gap between 28.9M and a few hundred million parameters, and p-for-llm just showed 180.9M fits.

This is what Edge is for

This post launches Edge, a new section of The AI Vibe covering AI on the smallest machines — a newsletter-style feed of what's real in microcontroller and edge-AI land, plus hands-on bench logs. I have boards on order: the plan is to run these projects myself, measure actual tokens per second instead of quoting READMEs, and publish the results here (and on video). The people building this field — slvDev, cyfrit, DaveBben, wladimiravila, and everyone downstream of Karpathy's llama2.c — are doing some of the most creative systems engineering in AI right now, on hardware everyone else ignores. They deserve the credit, and the scrutiny of replication. Both are coming.