
A 26B Looped-MoE LLM Gets Squeezed to 3.85 GB for Phones as New Papers Cut Decode Costs
LookAtMyQuant packs Inkling-2-Mini into six Apple Silicon builds, while new papers tackle weight decoding, laptop fine-tuning and Cortex-M inference.
Tag
18 posts

LookAtMyQuant packs Inkling-2-Mini into six Apple Silicon builds, while new papers tackle weight decoding, laptop fine-tuning and Cortex-M inference.

Tether's 0.8B tool-calling model, Huawei's W4A4 quantizer, a 4.5M-parameter open TTS and a lattice-quantization experiment, all with the caveats left in.

An independent pruning pipeline shrinks a 26B mixture-of-experts model to 4.75GB while two new papers chase lossless low-bit inference.

From a from-scratch LLM on a $25 microcontroller to an 8B MoE model living in under a gigabyte, the edge keeps shrinking its memory budget.

ML Drift goes open source, ExecuTorch grows up on Cortex-M, a 1.58-bit MoE lands on a 12GB GPU, and two new boards hit the bench.

A from-scratch tiny LLM, a sub-1-bit quantization trick, a MoE model on a phone NPU, and two new chips round out today's edge sweep.

Apple's Neural Engine runs a 27B model at low bit-width, a distilled 14M LLM animates a $8 microcontroller pet, and new research questions KV-cache quant on Raspberry Pi.

Samsung's sub-1-bit quantization paper gets a reality check, plus fresh research on tiny LLMs and ESP32 voice silicon.

Lesson 10: int8 quantization of the Stage C deciders, CPU latency and size, a stdlib server speaking pankhllm's decision shape, and a real run through pankhllm's own binary. The default int8 broke Qwen; the small decider got smaller and slower. And the contract, applied honestly: neither lane earns its place yet.

A BitNet cluster splits an LLM across seven ESP32-S3 boards, a 2-bit tool-calling model lands on GitHub, and a quantization paper gets llama.cpp 15x faster on an M4 Pro.

Liquid AI ships a 2.6B agentic model for edge hardware, a 144M classifier skips text generation entirely, and a new paper shows byte-level LLMs can beat tokenized ones with sc

AMD open-sources fused FlashAttention kernels for XDNA NPUs, Qualcomm ships a Linux preview for Snapdragon X2, and fresh research pushes tiny and trillion-scale models toward

PrismML compresses a 27B model to 5.9GB, Intel's BITCOS beats the 1.585-bit ternary limit, and a 44M-parameter model claims exact arithmetic on a laptop CPU.

A ternary 8B model, a BitNet toy for TinyStories, a fish tank run by a 14M LLM, and a Rust retrieval encoder push quantization research toward real hardware.

Fresh arXiv work rethinks quantization strategy and sustainability, an Apple-adjacent paper shrinks the dictation encoder, and new silicon and Jetson guidance round out the ed

A 1-bit Qwen derivative fits an iPhone, a healing trick beats its own teacher at 4-bit, and a dense LLM limps along on an $8 chip.

A Rockchip-powered duck robot, two offline Raspberry Pi builds, and audits exposing quantization's blind spots and mislabeled GGUF files.

Liquid AI distills instead of just rounding for 4-bit LFM2.5 checkpoints, as fresh research flags what low-bit quantization costs in memory and multilingual accuracy.