Skip to content
Back to Edge

A 2-Bit LLM Kernel Triples CPU Speed in llama.cpp as Jetson Thor Gets an Industrial Box

Prateek SinghNovember 16, 20263 min read91 views
A 2-Bit LLM Kernel Triples CPU Speed in llama.cpp as Jetson Thor Gets an Industrial Box

llama.cpp merged an x86 VNNI Q2_0 kernel on November 15, a Jetson Thor box PC appeared, and an ESP32-S3 controller launched at $69.

llama.cpp merges an x86 VNNI kernel for 2-bit Q2_0 weights

On November 15, 2026, llama.cpp merged a group-64 Q2_0 kernel for x86 CPUs that use VNNI integer instructions. RuntimeWire credits contributor Saejin Kim-Heinert. It reports speedups of 3.0x to 3.64x on the Bonsai model, and a small perplexity drift that the contributor attributes to compiler behavior.

Caveat: I read the outlet's account and did not open the pull request, so treat the figures as the contributor's self-reported, single-configuration results. The speedup is also relative to the previous CPU path for this format, not to other engines.

It matters because many small boxes have no GPU or NPU. A faster integer path for 2-bit weights is the kind of work that makes a larger model practical on a plain x86 mini PC.

llama.cpp build b12433 fixes an AMD crash with LoRA on MoE experts

Build b12433, published on November 16, 2026 as a pre-release, fixes a CUDA-backend abort on AMD devices. According to the release notes, MUL_MAT_ID with F32, F16 or BF16 weights and at most 8 tokens always took the MMVF path. MMVF reads values in pairs and asserts an even column count, so a rank-1 LoRA on mixture-of-experts weights, where lora_b has one column, aborted. The fix uses MMVF only for even column counts and sends other shapes to the existing fallback. The commit resolves issue #30576 and was marked as assisted by GitHub Copilot.

This is a narrow bug fix, not a performance change. It shows how much of the edge-runtime work is edge cases in kernels, as MoE models and adapters meet many GPU back ends.

AAEON BOXER-8741AI puts a Jetson Thor T5000 in an industrial box

On November 16, 2026, CNX Software covered the AAEON BOXER-8741AI, an industrial edge AI box PC built around the NVIDIA Jetson Thor T5000 module. The listing cites up to 2070 FP4 TFLOPS (sparse) of AI performance, with dual 10GbE and four 5GbE LAN ports.

That figure is a vendor peak number for sparse FP4, not a measured result for any model, and I did not see a price in the pull. For robotics and vision work, the useful part is the networking and enclosure around a module that can run large models locally.

Nabu Casa launches the $69 Home Assistant Connect AUX-2

On November 16, 2026, CNX Software reported that Nabu Casa has launched the Home Assistant Connect AUX-2 at $69. It is an ESP32-S3-based smart home controller with PoE, audio streaming, RS-232 serial control and a Bluetooth proxy.

No model runs on it as described, so this is a hardware note, not an inference story. It matters because ESP32-S3 boards with audio paths are the usual base for local voice assistants, and wired power and Ethernet make them easier to leave running.

A quiet day on the beat: one kernel merge, one bug fix and two hardware launches, each with the caveats noted above. Read the linked sources for the full details.

References & Citations

Subscribe to new posts from theaivibe.org

No spam — just new posts. One-click unsubscribe.
Share this article

Related Posts