Edge Dispatch: The $8 AI Chip Grows a Coffee Habit: ESP32 Tiny-LLM Trick Gets a Second Model

A second model joins the ESP32 experiment: this one talks espresso

The developer behind the widely-shared demo of a 28.9-million-parameter language model running locally on an $8 ESP32-S3 board has a second act: a model called Barista that answers questions about espresso, running roughly twice as fast as the original TinyStories model on the same chip, according to reporting from The Register.

The honest caveat: Barista is not a general knowledge model. It works within a narrow topic, the same way the original TinyStories model only writes simple children's stories. Neither model answers questions outside its training scope, follows multi-step instructions, or does anything resembling reasoning. This is domain-specific compression, not a shrunk version of a general chatbot.

What matters for tiny hardware is the pattern: once the memory trick works for one narrow-domain model, it is straightforward to retrain and redeploy for other narrow domains on the same $8 board. That is a more realistic path to useful microcontroller AI than waiting for a general-purpose model to shrink enough to fit.

How a 25-million-row table fits on a chip with 512KB of fast memory

The technique underneath both models is Per-Layer Embeddings, borrowed from Google's Gemma 3n and Gemma 4 architecture, applied here to a microcontroller instead of a phone or GPU, as detailed in the project's own writeup on GitHub by martinavila (Slava S., @slvDev) and broken down by CNX Software. The model's embedding table, about 25 million of its 28.9 million parameters, sits in slow 16MB flash storage rather than RAM. Each token only needs a handful of rows from that table, roughly 450 bytes, so the chip's 512KB of SRAM is reserved for the small part of the model that actually computes, while 8MB of PSRAM holds the output head, KV cache, and working memory.

The self-reported result is about 9 to 9.5 tokens per second, generation speed the developer says is faster than most people read. That number comes from the developer's own testing on one board configuration (512KB SRAM, 8MB PSRAM, 16MB flash), not an independent benchmark, and memory tiering like this trades latency and complexity for capacity.

The interesting part isn't the story quality, it's the memory architecture. Fitting a 28.9M-parameter model into a chip that normally handles Wi-Fi and sensor polling, using a technique designed for phone-class hardware, suggests the same trick could extend to other narrow-domain models well beyond what a $10 microcontroller was ever expected to run.

The skeptic's view: engineering feat, not a product

Coverage of the project has split between admiration for the engineering and blunt skepticism about its usefulness. The Register put it plainly: getting a model running on a microcontroller like this gets you "dumb simple pride" more than a usable product, and Tom's Hardware and Geeky Gadgets both note the model can't hold a conversation, write code, or recall facts outside its narrow training set.

That is a fair read, and it is worth repeating clearly: this is genuine on-device inference, nothing goes to a server, but it is inference on a model trained to do one small thing well. It is not evidence that general-purpose LLMs are coming to microcontrollers anytime soon.

Still, the honest version of the story is more interesting than either the hype or the dismissal. A hundred-fold parameter increase over what ran on similar hardware in 2023, achieved by relocating most of a model's weights to flash instead of shrinking the model itself, is a real technique that other embedded developers can reuse for their own narrow tasks, coffee-related or otherwise.

None of this makes an ESP32 into a general assistant, but it does show how far memory-tiering tricks can stretch a chip that costs less than a decent lunch.