Edge Runtimes Learn to Run LLMs at 2 Bits While a New Chip Takes Aim at DGX Spark

2-Bit Models Finally Load in Ollama

Until October 16, 2026, Ollama could not load Prism ML's Bonsai 2 models at their smallest quantization. The bundled llama.cpp engine could run the files, but Ollama's own GGUF reader didn't recognize tensor type 42, so loading failed with a "size overflows" error on the 2B through 27B checkpoints. PR #18967 merged on October 15 and shipped in Ollama 0.40.6 the next day, per the writeup from MindPattern. Q2_0 packs 64 weights into 18 bytes, about 2.25 bits per weight.

The same window saw llama.cpp PR #30122 add Q1_0/Q2_0 to the Hexagon NPU backend, passing all 128 tests on a Qualcomm SM8850 device, and PR #30517 rewrite CUDA mat-vec kernels for low-bit types — its own llama-bench tables show a used RTX 3090 jumping from 81.04 to 126.22 t/s on Qwen3.5-9B Q3_K_S, a 1.56x gain, while newer 4090/5090 cards barely move.

Caveat: nobody has shown the Hexagon path running faster than higher-bit quants yet, and the CUDA gain is mostly an Ampere-architecture quirk. Still, this is three separate backends — a phone app, a phone NPU, and an old desktop GPU — all getting cheaper memory footprints for the same weights in the same week.

PyTorch Puts Models on Bare Cortex-M Chips

PyTorch's ExecuTorch project added a Cortex-M backend that runs models directly on Arm Cortex-M CPUs using CMSIS-NN acceleration, covering Scalar, DSP and MVE/Helium tiers from one workflow, according to PyTorch's own blog post and a detailed writeup at Brocker.org. On a single Alif Ensemble E8 Cortex-M55 core at 400 MHz with its NPU left idle, four MLPerf Tiny models ran in 21.5 ms or less, each under 90 KiB of RAM — DS-CNN keyword spotting hit 5.825 ms using just 26 KiB.

A µYOLO person-detection demo on a Renesas Cortex-M85 board ran at 24 ms per frame, about 12.7 FPS, inside 254 KiB of RAM. PyTorch is explicit that these are its own measurements, not formal MLPerf Tiny submissions, so there's no apples-to-apples comparison to the public leaderboard.

Still, the core ExecuTorch runtime stays under 20 KiB of flash, and the backend already validates on the Raspberry Pi Pico 2 — a PyTorch-native path onto chips with no dedicated AI silicon at all.

Google's LiteRT Cuts Phone LLM Memory at Long Context

Google shipped LiteRT 2.3.0 this month with a FlashAttention implementation for Metal that the vendor says gives up to 38% faster decode, up to 2.98x faster prefill, and more than 4 GB less peak RAM at 32K context on Apple Silicon, per Google's own release notes. New fused QKV-RMSNorm-RoPE and SwiGLU GPU kernels add roughly 12% faster decode on top of that.

The release also splits Android GPU and vendor NPU libraries into separate Maven packages, adds Samsung Exynos AI LiteCore NPU support, and ships Intel NPU/GPU weight sharing that roughly halves the size of an AOT-compiled LLM. All of these are Google's self-reported numbers, not independently benchmarked.

For a lane obsessed with how much model fits in how little RAM, a documented 4 GB savings at 32K tokens on a phone-class chip is the kind of concrete number worth tracking as third parties confirm or contest it.

A 16-Core SoC Aims at the DGX Spark's Price Tag

Axera's new AX8860P is a 16-core Cortex-A55 SoC pairing a 128 TOPS (256 TOPS at FP4) NPU with 256-bit LPDDR5/5X memory and PCIe 5.0 connectivity, positioned by the company as a cheaper desktop AI alternative to NVIDIA's DGX Spark, according to CNX Software's October 19, 2026 report.

The TOPS figures come from Axera, not an independent benchmark, and no retail pricing has been confirmed yet for boards built around the chip. Whether it can actually run a usable LLM stack at that memory bandwidth is the real test once dev boards ship.

A $34 Plug-and-Play Camera for Edge Vision Projects

Arduino launched a $34 USB Camera built for plug-and-play computer vision work, with built-in lighting and audio capture, as reported by Hackster on October 18, 2026.

The camera itself carries no NPU — it's a standard UVC sensor module, with any on-device inference left to whatever board it's plugged into. The news here is accessibility: a cheap, standardized input makes it easier to feed TinyML vision pipelines on boards that already do the actual AI work.

A busy runtime week: Ollama, llama.cpp and Google's LiteRT all tightened the bits-per-weight and memory math that decides what fits on constrained hardware, while new silicon and a cheap camera widened the hardware end of the pipeline.