Ternary Weights and Tiny Tanks: Small-Model Quantization Gets Concrete for Edge AI

Ternary Quantization Scales Up to an 8B Model

A paper posted to arXiv on September 13, 2026 pushes post-training ternarisation — packing weights down toward {-1, 0, +1} — up to Qwen3-8B, a scale most ternary work has skipped. The authors report the ternarized model holds recorded perplexity to measurement precision while shrinking to 8.24 GiB, and that a direct packed-execution kernel runs at 15.52 tokens per second inside a 7.35 GiB footprint.

The honest caveat is right there in the abstract: their packed GEMV kernel is still slower than FP16 cuBLAS. This is a memory-savings result first, a speed result second — useful for squeezing an 8B model onto a 8 GB card or an AI PC's unified memory, not yet a throughput win.

For the edge, memory is often the harder constraint than compute, so a validated ternary baseline at 8B parameters — with reproducible packing — matters more than the current kernel speed suggests. Read the paper on arXiv.

A 9M-Parameter BitNet Model Trained From Scratch, No PTQ Involved

Thox-ai published ThoxMicro-1bit-9M this week, a BitNet b1.58-style language model trained from scratch rather than quantized after the fact. Transformer weights are ternary {-1, 0, +1} with a per-tensor absmean scale; activations stay int8. Trained on TinyStories, the model's own card reports validation loss rising from 1.5476 (fp16 baseline) to 1.7385 in the ternary version, with perplexity going from 4.70 to 5.69 — a real, disclosed accuracy cost, not a free lunch.

The model is explicitly not an assistant: no chat template, no instructions, just children's-story completions. What's notable for the microcontroller crowd is a second artifact, TERN1, a packed ternary format sized for ESP32-S3-class targets — though Thox-ai is careful to note it isn't distributed as a runnable file and no general runtime reads it yet.

It's a research toy, and the license terms around TinyStories-derived weights are still unresolved by the team's own admission. But it's a clean, from-scratch demonstration that ternary training works end to end at this size. See the model card on Hugging Face.

A 14M-Parameter LLM Keeps a Virtual Fish Tank Alive on an $8 Chip

A GitHub project called pocket-tank, published by developer mediacutlet, distills a 26B-parameter teacher down to a 14M-parameter model small enough to run inference directly on an $8 ESP32-S3 board. The repo, which started trending on September 12, 2026, uses the tiny model to drive an on-device virtual aquarium — narration and behavior generated locally rather than fetched from an API.

It's a small, playful demo rather than a benchmark paper, and the readme doesn't publish a formal accuracy comparison against the teacher. But the distillation ratio — roughly 1,800x down — is the point: it's another data point for how far LLM distillation can be pushed onto sub-dollar microcontroller hardware while keeping generation on-device.

See the repo on GitHub.

An Android Overlay Translator Runs OCR and LLM Entirely On-Device

A GitHub project called overlay-translator, by developer ciddwd, has climbed to over 800 stars this week as an Android tool that reads screen text and overlays translations in real time — no root required. It's aimed at games, visual novels, and manga, and it supports both on-device OCR plus an offline local LLM for translation, alongside cloud services and text-to-speech for users who want them.

The project doesn't publish latency or accuracy numbers, so the on-device path should be read as a capability claim rather than a benchmarked one for now. Still, it's a concrete example of a full vision-plus-language pipeline — capture, OCR, translate, speak — running without a network round trip when the offline LLM option is chosen.

See the repo on GitHub.

A 9.8M-Parameter Retrieval Encoder Runs 53x Faster in Pure Rust

Developer motedb published a write-up on September 2026 describing VelaVec, a 9.8M-parameter retrieval encoder distilled from a 33M-parameter teacher (bge-small), then rebuilt as a pure-Rust inference stack using AMX, BNNS, and NEON on Apple's M4 CPU. The self-reported numbers: query encoding at 103 microseconds versus the teacher's 5.5 milliseconds, a 5.4 millisecond cold start versus roughly 29 seconds for the PyTorch original, and a weight footprint of 39 MB versus 130 MB.

These are the builder's own benchmarks on a single laptop chip, not a peer-reviewed study, and the embedding dimension shrank from 384 to 256 alongside the parameter cut — a real capacity trade-off, not a free win. Even so, it's a useful example of what CPU-only, GPU-free retrieval can look like when the runtime is rewritten from scratch instead of just ported.

Read the full write-up on DEV Community.

None of today's numbers are peer-verified beyond their own papers or repos, but the pattern is consistent: quantization research is chasing memory and latency on real chips, not just theoretical bit budgets, and distillation keeps finding smaller and cheaper places to land.