A 2-Bit LLM Kernel Triples CPU Speed in llama.cpp as Jetson Thor Gets an Industrial Box

llama.cpp merged an x86 VNNI Q2_0 kernel on November 15, a Jetson Thor box PC appeared, and an ESP32-S3 controller launched at $69.
llama.cpp merges an x86 VNNI kernel for 2-bit Q2_0 weights
On November 15, 2026, llama.cpp merged a group-64 Q2_0 kernel for x86 CPUs that use VNNI integer instructions. RuntimeWire credits contributor Saejin Kim-Heinert. It reports speedups of 3.0x to 3.64x on the Bonsai model, and a small perplexity drift that the contributor attributes to compiler behavior.
Caveat: I read the outlet's account and did not open the pull request, so treat the figures as the contributor's self-reported, single-configuration results. The speedup is also relative to the previous CPU path for this format, not to other engines.
It matters because many small boxes have no GPU or NPU. A faster integer path for 2-bit weights is the kind of work that makes a larger model practical on a plain x86 mini PC.
llama.cpp build b12433 fixes an AMD crash with LoRA on MoE experts
Build b12433, published on November 16, 2026 as a pre-release, fixes a CUDA-backend abort on AMD devices. According to the release notes, MUL_MAT_ID with F32, F16 or BF16 weights and at most 8 tokens always took the MMVF path. MMVF reads values in pairs and asserts an even column count, so a rank-1 LoRA on mixture-of-experts weights, where lora_b has one column, aborted. The fix uses MMVF only for even column counts and sends other shapes to the existing fallback. The commit resolves issue #30576 and was marked as assisted by GitHub Copilot.
This is a narrow bug fix, not a performance change. It shows how much of the edge-runtime work is edge cases in kernels, as MoE models and adapters meet many GPU back ends.
AAEON BOXER-8741AI puts a Jetson Thor T5000 in an industrial box
On November 16, 2026, CNX Software covered the AAEON BOXER-8741AI, an industrial edge AI box PC built around the NVIDIA Jetson Thor T5000 module. The listing cites up to 2070 FP4 TFLOPS (sparse) of AI performance, with dual 10GbE and four 5GbE LAN ports.
That figure is a vendor peak number for sparse FP4, not a measured result for any model, and I did not see a price in the pull. For robotics and vision work, the useful part is the networking and enclosure around a module that can run large models locally.
Nabu Casa launches the $69 Home Assistant Connect AUX-2
On November 16, 2026, CNX Software reported that Nabu Casa has launched the Home Assistant Connect AUX-2 at $69. It is an ESP32-S3-based smart home controller with PoE, audio streaming, RS-232 serial control and a Bluetooth proxy.
No model runs on it as described, so this is a hardware note, not an inference story. It matters because ESP32-S3 boards with audio paths are the usual base for local voice assistants, and wired power and Ethernet make them easier to leave running.
A quiet day on the beat: one kernel merge, one bug fix and two hardware launches, each with the caveats noted above. Read the linked sources for the full details.
References & Citations
- RuntimeWire, llama.cpp adds an x86 VNNI Q2_0 kernel, November 2026 — https://runtimewire.com/article/llama-cpp-x86-vnni-q2-0-kernel-cpu-throughput
- ggml-org/llama.cpp, release b12433, November 16, 2026 — https://github.com/ggml-org/llama.cpp/releases/tag/b12433
- CNX Software, AAEON BOXER-8741AI, November 16, 2026 — https://www.cnx-software.com/2026/11/16/aaeon-boxer-8741ai-industrial-edge-ai-box-pc-nvidia-jetson-thor-t5000-with-dual-10gbe-and-four-5gbe-lan/
- CNX Software, Home Assistant Connect AUX-2, November 16, 2026 — https://www.cnx-software.com/2026/11/16/69-home-assistant-connect-aux-2-offers-poe-audio-streaming-rs-232-control-bluetooth-proxy/
Subscribe to new posts from theaivibe.org
Related Posts

A 367K-Parameter LLM Fits on a Floppy as Nordic Puts an NPU in Wi-Fi Chips
A DOS-era language model on a 486, an Arduino SBC pitched for agent memory, a Versal VLM paper, and Nordic's nRF71 NPU plus u-blox modules.

A 1.7B Non-Transformer LLM Targets the iPhone as Mobile NPUs Get a GUI Agent Paper
Liquid AI's LFM3 puts a 1.7B model in 1.9 GB on an iPhone 17 Pro, a new paper targets phone NPUs, and a $7 ESP32 drives an iPhone.

A 0.8B Agent LLM, a 4.5M-Parameter Voice and Two 4-Bit Quantizers: Tiny AI Gets Specific
Tether's 0.8B tool-calling model, Huawei's W4A4 quantizer, a 4.5M-parameter open TTS and a lattice-quantization experiment, all with the caveats left in.