Skip to content

Tag

quantization

18 posts

A 26B Looped-MoE LLM Gets Squeezed to 3.85 GB for Phones as New Papers Cut Decode Costs
Edge AI4 min read

A 26B Looped-MoE LLM Gets Squeezed to 3.85 GB for Phones as New Papers Cut Decode Costs

LookAtMyQuant packs Inkling-2-Mini into six Apple Silicon builds, while new papers tackle weight decoding, laptop fine-tuning and Cortex-M inference.

18 views
Read
A 0.8B Agent LLM, a 4.5M-Parameter Voice and Two 4-Bit Quantizers: Tiny AI Gets Specific
Edge AI4 min read

A 0.8B Agent LLM, a 4.5M-Parameter Voice and Two 4-Bit Quantizers: Tiny AI Gets Specific

Tether's 0.8B tool-calling model, Huawei's W4A4 quantizer, a 4.5M-parameter open TTS and a lattice-quantization experiment, all with the caveats left in.

170 views
Read
A 26B MoE Model Squeezes Onto a 6GB GPU as Edge AI Quantization Races Ahead
Edge AI4 min read

A 26B MoE Model Squeezes Onto a 6GB GPU as Edge AI Quantization Races Ahead

An independent pruning pipeline shrinks a 26B mixture-of-experts model to 4.75GB while two new papers chase lossless low-bit inference.

583 views
Read
A $25 ESP32 Trains Its Own Tiny LLM While an 8B MoE Model Squeezes Under 1GB of RAM
Edge AI4 min read

A $25 ESP32 Trains Its Own Tiny LLM While an 8B MoE Model Squeezes Under 1GB of RAM

From a from-scratch LLM on a $25 microcontroller to an 8B MoE model living in under a gigabyte, the edge keeps shrinking its memory budget.

382 views
Read
Google Open-Sources Its On-Device AI Engine While a Ternary LLM Shrinks a 30B Model to 12GB
Edge AI4 min read

Google Open-Sources Its On-Device AI Engine While a Ternary LLM Shrinks a 30B Model to 12GB

ML Drift goes open source, ExecuTorch grows up on Cortex-M, a 1.58-bit MoE lands on a 12GB GPU, and two new boards hit the bench.

407 views
Read
A 384M LLM Learns to Call Tools From Scratch While Rockchip Chases 44 TOPS at the Edge
Edge AI4 min read

A 384M LLM Learns to Call Tools From Scratch While Rockchip Chases 44 TOPS at the Edge

A from-scratch tiny LLM, a sub-1-bit quantization trick, a MoE model on a phone NPU, and two new chips round out today's edge sweep.

326 views
Read
A 27B AI Hits an iPhone's Neural Engine at 3 Bits While a 14M LLM Keeps a Fish Tank Alive on ESP32
Edge AI5 min read

A 27B AI Hits an iPhone's Neural Engine at 3 Bits While a 14M LLM Keeps a Fish Tank Alive on ESP32

Apple's Neural Engine runs a 27B model at low bit-width, a distilled 14M LLM animates a $8 microcontroller pet, and new research questions KV-cache quant on Raspberry Pi.

300 views
Read
A 1-Bit AI Trick Shrinks a 70B Model to Fit on a $200 GPU
Edge AI4 min read

A 1-Bit AI Trick Shrinks a 70B Model to Fit on a $200 GPU

Samsung's sub-1-bit quantization paper gets a reality check, plus fresh research on tiny LLMs and ESP32 voice silicon.

292 views
Read
Does It Earn a Lane? Quantize, Measure, and Decide
e=mc²9 min read

Does It Earn a Lane? Quantize, Measure, and Decide

Lesson 10: int8 quantization of the Stage C deciders, CPU latency and size, a stdlib server speaking pankhllm's decision shape, and a real run through pankhllm's own binary. The default int8 broke Qwen; the small decider got smaller and slower. And the contract, applied honestly: neither lane earns its place yet.

185 views
Read
Seven $5 Chips Learn to Share an LLM, One Bit at a Time
Edge AI5 min read

Seven $5 Chips Learn to Share an LLM, One Bit at a Time

A BitNet cluster splits an LLM across seven ESP32-S3 boards, a 2-bit tool-calling model lands on GitHub, and a quantization paper gets llama.cpp 15x faster on an M4 Pro.

472 views
Read
A Classifier That Only Picks, a Model That Skips Tokens, and an AI Firm Bets 2.6B Parameters Can Do Agent Work
Edge AI4 min read

A Classifier That Only Picks, a Model That Skips Tokens, and an AI Firm Bets 2.6B Parameters Can Do Agent Work

Liquid AI ships a 2.6B agentic model for edge hardware, a 144M classifier skips text generation entirely, and a new paper shows byte-level LLMs can beat tokenized ones with sc

203 views
Read
NPUs Learn to Fuse: AMD Opens Its XDNA Compiler, Qualcomm Previews Linux, and Two New On-Device AI Models Land
Edge AI5 min read

NPUs Learn to Fuse: AMD Opens Its XDNA Compiler, Qualcomm Previews Linux, and Two New On-Device AI Models Land

AMD open-sources fused FlashAttention kernels for XDNA NPUs, Qualcomm ships a Linux preview for Snapdragon X2, and fresh research pushes tiny and trillion-scale models toward

313 views
Read
A 27B AI Model Shrinks to 5.9GB as Ternary Quantization Keeps Pushing the Floor Down
Edge AI5 min read

A 27B AI Model Shrinks to 5.9GB as Ternary Quantization Keeps Pushing the Floor Down

PrismML compresses a 27B model to 5.9GB, Intel's BITCOS beats the 1.585-bit ternary limit, and a 44M-parameter model claims exact arithmetic on a laptop CPU.

315 views
Read
Ternary Weights and Tiny Tanks: Small-Model Quantization Gets Concrete for Edge AI
Edge AI4 min read

Ternary Weights and Tiny Tanks: Small-Model Quantization Gets Concrete for Edge AI

A ternary 8B model, a BitNet toy for TinyStories, a fish tank run by a 14M LLM, and a Rust retrieval encoder push quantization research toward real hardware.

288 views
Read
Quantization Papers Pile Up as Researchers Argue Over Where Small AI Should Spend Its Bits
Edge AI4 min read

Quantization Papers Pile Up as Researchers Argue Over Where Small AI Should Spend Its Bits

Fresh arXiv work rethinks quantization strategy and sustainability, an Apple-adjacent paper shrinks the dictation encoder, and new silicon and Jetson guidance round out the ed

243 views
Read
A 27B LLM Shrinks to Phone Size as Edge Quantization Keeps Rewriting the Rules
Edge AI4 min read

A 27B LLM Shrinks to Phone Size as Edge Quantization Keeps Rewriting the Rules

A 1-bit Qwen derivative fits an iPhone, a healing trick beats its own teacher at 4-bit, and a dense LLM limps along on an $8 chip.

198 views
Read
A Duck Robot Runs Its Balance Loop On-Chip as Local AI Quantization Claims Get Audited
Edge AI4 min read

A Duck Robot Runs Its Balance Loop On-Chip as Local AI Quantization Claims Get Audited

A Rockchip-powered duck robot, two offline Raspberry Pi builds, and audits exposing quantization's blind spots and mislabeled GGUF files.

207 views
Read
Edge Dispatch: Liquid AI Trains Its Way Around the Q4_0 Quality Tax, While New Papers Chip Away at What Quantization Quietly Breaks
Edge AI4 min read

Edge Dispatch: Liquid AI Trains Its Way Around the Q4_0 Quality Tax, While New Papers Chip Away at What Quantization Quietly Breaks

Liquid AI distills instead of just rounding for 4-bit LFM2.5 checkpoints, as fresh research flags what low-bit quantization costs in memory and multilingual accuracy.

261 views
Read