Skip to content
Back to Edge

A 26B Looped-MoE LLM Gets Squeezed to 3.85 GB for Phones as New Papers Cut Decode Costs

Prateek SinghNovember 22, 20264 min read18 views
A 26B Looped-MoE LLM Gets Squeezed to 3.85 GB for Phones as New Papers Cut Decode Costs

LookAtMyQuant packs Inkling-2-Mini into six Apple Silicon builds, while new papers tackle weight decoding, laptop fine-tuning and Cortex-M inference.

LookAtMyQuant fits Inkling-2-Mini into 3.85 to 12.76 GB

On November 22, 2026, a Repocharts write-up described a new release of LookAtMyQuant, an MIT-licensed quantization compiler. It shrinks Thinking Machines Lab's Inkling-2-Mini into six Apple Silicon builds. The model is a 26.4B-parameter mixture-of-experts with about 5.4B active parameters and a looped block of layers. The model's own huggingface.co post lists a 1M-token context, image and text input, and NVFP4 and MXFP8 expert weights.

According to repocharts.com, the builds run from 3.85 GB (1.18 bits per weight, aimed at 8 GB iPhones) to 12.76 GB. The 5.38 GB build is reported at 87.0% on GSM8K and 71.3% on strict HumanEval. The 3.85 GB build recovers to 78.5% and 48.2% after a fine-tuning pass.

The caveat is that every number is self-reported by the author, and the benchmark samples are modest. I did not see an independent run on a phone. Still, sub-2-bit builds of a mixture-of-experts model are the kind of result that decides whether a model fits in an 8 GB handset.

LazyWeights decodes compressed weights only when needed

A November 2026 arXiv paper, LazyWeights: Decode-on-Demand Edge LLMs, keeps weights compressed in memory and decodes them just in time. The abstract says it needs 58 to 64% less time on the "Nano" board the authors test, and runs a streamed Mistral-7B 6.6 times faster.

The authors also report that a Huffman decoder beats an rANS decoder in rate and area at equal throughput. They say the decoder unit runs bit-exact on a Zynq UltraScale+ FPGA. These are the authors' own measurements, and the abstract excerpt I could read does not give the full test setup.

The idea targets the real bottleneck on small boards: memory bandwidth, not arithmetic. A small hardware decoder trades silicon for fewer bytes moved.

Fine-tuning Gemma 4 edge models on a laptop iGPU

A new arXiv paper, Fine-Tuning Gemma 4 Edge Models for MATLAB Signal-Processing Code with SFT and QLoRA on a Laptop iGPU, adapts small Gemma 4 models to write MATLAB code without a discrete GPU. The E4B model with QLoRA reaches 47 of 66 test programs.

The error analysis is the useful part. After reading all 159 failing programs, the authors found the adapters remove most convention errors, such as the sign of a lag, units and array shapes. They fix few errors in building signals and noise. In other words, the adapter teaches house style more than it teaches the underlying engineering.

For tiny-hardware readers, the point is that domain tuning of a 4B-class model is now feasible on an ordinary laptop's integrated graphics. Results come from one task set and are the authors' own.

BeFit moves complexity from weights into activations for microcontrollers

The arXiv paper BeFit targets sensing models on chips with no room to spare. It replaces part of the weight complexity with Bernstein-polynomial activations. The authors report the projected model stays within 1.0 percentage point of the reference accuracy while running 5.9 times faster.

They also test a RISC-V core without a hardware multiplier. There, the instruction-count reduction against a DS-CNN baseline grows from 5.2 times on a Cortex-M4. The abstract excerpt cuts off before the RISC-V figure.

These are paper results, not a shipping toolchain. Still, cores without multipliers are common in the cheapest chips, and this is a direct attempt to make them useful for classification.

MoMoVL compresses on-device mixture-of-experts vision-language models

The arXiv paper MoMoVL argues that quantization methods built for text mixture-of-experts models miss a problem in vision-language ones. The abstract's argument is that the two input types, image and text, behave differently inside the experts. It proposes modality-aware adaptive compression for on-device use.

I only had the abstract opening, so I cannot report its benchmark numbers. It is worth watching because vision-language models of this type are the next thing phones are expected to run locally.

Today's common thread is memory and bytes: compressed weights, fewer bits and cheaper activations. Every figure above is the authors' own, so read the linked pages for the full setups.

References & Citations

Subscribe to new posts from theaivibe.org

No spam — just new posts. One-click unsubscribe.
Share this article

Related Posts