A 26B MoE Model Squeezes Onto a 6GB GPU as Edge AI Quantization Races Ahead

MAPLE Prunes a 26B MoE Model Onto a 6GB Laptop GPU

Independent researcher Aman Sachan released MAPLE, a structural pruning pipeline that cuts Google's Gemma 4 26B-A4B mixture-of-experts model from 9.92GB down to 4.75GB, or about 1.48 effective bits per parameter. The method removes 80 of 128 routed experts per layer, keeping a 48-expert manifold chosen by profiling live forward-pass activations rather than static embeddings.

The resulting GGUF loads entirely into VRAM on a consumer RTX 4050 laptop GPU with 6GB of memory, running through llama.cpp at a self-reported 45.4 tokens per second, according to Sachan's Show HN thread and a write-up from Zubiqo.

The caveat: this is one person's benchmark on one GPU, and the pruned model is specialized for Python and STEM reasoning, not general chat. Still, it is a concrete demonstration that a 26-billion-parameter MoE can be forced onto genuinely small VRAM budgets without retraining from scratch.

ProQuant Claims Near-Lossless 4-Bit Compression for Edge Vision-Language Models

A new arXiv paper, ProQuant: Progressive Quantization-aware Training for Edge MLLMs, posted on November 5, 2026, tackles a specific pain point for multimodal models on phones: low-bit W4A8 quantization usually wrecks small models' ability to connect what they see with what they say.

ProQuant re-labels fine-tuning data to match what the model already knows, then restores capability in stages — image understanding first, language skills second. On a small edge-sized model, the authors report accuracy roughly 6.6 percentage points above existing compressed baselines at the same bit width, landing within about one point of the full-precision original, per a summary from Zotpaper.

These are the authors' own numbers on their own test set, not an independent benchmark, and the paper has not been peer reviewed. But the specific technique — staged recovery of vision before language — is a useful idea for anyone trying to get a camera-plus-LLM pipeline onto phone-class memory.

llama.cpp Adds a Snapdragon-Tuned OpenCL Kernel, No Benchmarks Yet

llama.cpp's release b12200, dated November 8, 2026, folds in PR #31831, which adds an OpenCL q8_0 GEMV kernel and a small-batch mixture-of-experts GEMV path for quantization formats including q5_1 and q4_k. The commit also tunes Adreno GPU kernels and disables flash-attention batching for small validation batches.

The PR lists a Qualcomm engineer as co-author, and the release's build matrix now includes Snapdragon targets spanning CPU, Adreno GPU, and Hexagon NPU, as flagged by a release tracker.

No tokens-per-second figures or chip-specific gains are published in the notes, so it is impossible to say yet how much this helps a given Snapdragon phone or laptop. The direction is clear, though: llama.cpp's Qualcomm path keeps getting narrower, chip-specific attention, one small-batch kernel at a time.

Waveshare's ESP32-S31-Zero Packs 32MB PSRAM Into a Thumbnail Board

On November 7, 2026, CNX Software detailed Waveshare's ESP32-S31-Zero, a development board built around the ESP32-S31 chip that follows the compact "Zero" form factor the company has used for earlier boards like the ESP32-C6-Zero. The board is configurable with up to 32MB of PSRAM and 32MB of SPI flash — generous headroom for a board this small.

That much PSRAM matters for the tiny-LLM crowd: ESP32-class microcontrollers running quantized sub-1B models or wake-word pipelines are usually memory-starved first, not compute-starved. Pricing and availability were not confirmed at the time of the report, which is the main open question before anyone builds a project around it.

Raspberry Pi Brings iroh Mesh Networking to the RP2350

Raspberry Pi's own blog announced on November 6, 2026, that iroh now runs on the RP2350 microcontroller. Iroh is peer-to-peer connection software that lets devices, apps and cloud instances form a secure mesh without a central server coordinating every handshake.

Porting it to a $0.X-class microcontroller means battery-powered RP2350 devices can sync data or coordinate with other nodes directly, which is useful for distributed sensor swarms or offline voice/vision gadgets that occasionally need to talk to each other without routing through a cloud account. This is networking infrastructure, not AI model inference, but it is the kind of plumbing that on-device AI projects increasingly depend on once they leave the bench and need to talk to a phone or a hub.

A quiet day on headline models, but a busy one underneath: pruning, quantization-aware training and chip-specific kernels are all chipping away at the same wall — fitting more capable models into less memory, one benchmark and one pull request at a time.