A 27B LLM Shrinks to Phone Size as Edge Quantization Keeps Rewriting the Rules

Prism ML squeezes a 27B reasoning model down to 3.9 GB

Prism ML released Bonsai 27B, a ternary and binary quantization of Alibaba's Qwen3.6-27B that shrinks a roughly 54 GB model to as little as 3.9 GB. The team quantized everything — embeddings, attention, MLP projections, output head — with no higher-precision fallback, per intelligentliving.co's write-up. The ternary (1.58-bit) build lands around 7.2 GB and is reported to keep 95% of the FP16 model's score across 15 benchmarks; the pure 1-bit build drops to 3.9 GB, small enough for an iPhone 17 Pro Max via MLX.

These are self-reported numbers from the developers, not an independent audit, and the 26 tok/s figure on an Apple M5 Pro is a single vendor benchmark. Fernando Nogueira's technical breakdown walks through the distillation pipeline in more detail.

Whatever the true accuracy gap turns out to be, the file sizes are real and llama.cpp/MLX support already exists under Apache 2.0. That matters because it moves a 27B-class model from '24 GB GPU territory' into 'flagship phone' territory, a jump few quantization efforts have credibly claimed.

An essay arguing small models already crossed the line

An essay titled 'Small Models Have Arrived' pulled 792 points and 345 comments on Hacker News on August 27, 2026, arguing that sub-10B and even sub-1B models have quietly gotten good enough for a wide swath of everyday tasks — not because they got smarter in the abstract, but because distillation and quantization closed the practical gap for narrow jobs.

The honest caveat is that it's an opinion piece, not a benchmark study, and the HN thread pushed back hard on how 'good enough' is defined and for whom.

It's a useful frame for a week where a 27B model fits a phone, a 250M-class checkpoint runs on a $60 board, and 30-million-parameter models limp along on microcontrollers — the size floor for 'useful' keeps dropping.

Quantization-Aware Healing beats the model it's copying

Researchers describe a technique called Quantization-Aware Healing (QAH) in a new arXiv paper and companion Multiverse Computing blog post. Applied to a GPT-OSS 120B model pruned to 60B parameters and quantized to MXFP4 4-bit, the resulting model, released open-weight as Hypernova-60B, matches or beats its own bfloat16 original on 7 of 9 benchmarks, at roughly a quarter of the weight memory.

That 'beats its teacher' claim comes from the paper's own benchmark selection, and the pipeline is specific to this prune-then-heal recipe, not a drop-in fix for any quantized model. It's still notable: most quantization work is a damage-control story, this one claims occasional net gains.

For edge deployment, a healed 4-bit 60B model is far more plausible to run on a single high-end consumer GPU than the 120B original ever was, without the usual 'quantization tax' on accuracy.

A 2.7-bit format for vision-language models on Arm CPUs

A new paper, 'Llama-Mobile', introduces S3D8, a 2.7-bit-per-parameter format that packs three signed weights into a byte and decodes them to INT8 for inference on Arm CPUs. The researchers compressed Llama 3.2 11B Vision Instruct to 3.7 GB with 8-bit activations, training the low-bit version using data generated by the model itself rather than the original training set.

No public code or checkpoint release is mentioned in the paper, so this is a research result, not yet something you can pull down and run today.

Vision-language models are heavier than text-only LLMs and have mostly stayed cloud-side on phones; a genuinely CPU-friendly sub-3-bit path, if it reproduces, would matter for on-device visual assistants and robotics perception that can't spare a GPU.

A 30-million-parameter LLM, dense, on an $8 chip

Developer JARACH-209 published esp32-30.7M, a 30.72-million-parameter language model running fully dense — every stored parameter multiplied on every token, no mixture-of-experts sparsity — on an ESP32-S3, a chip that costs about $8. The repo reports 0.95 tokens per second, no cloud calls, no SD card, with the vocabulary trimmed down until the model fits in 16 MB of flash, building on Andrej Karpathy's tinyllamas/stories42M lineage.

Under one token per second is a demo speed, not a usable assistant, and the tiny trimmed vocabulary limits what the model can actually say coherently.

It's still a real dense-transformer inference on the cheapest class of Wi-Fi microcontroller, a useful data point for how far parameter count can be pushed before a board simply runs out of room and patience.

Nothing here is a finished product you can trust blindly — vendor benchmarks, single-paper claims, and a novelty microcontroller demo all come with the caveats above. But the direction is consistent: the floor for 'runs on this device' keeps sliding down, one quantization trick at a time.