Windows Learns GGUF as AI Agents Design Their Own Inference Chip

Windows ML Learns to Speak GGUF

On October 7, 2026, Microsoft added experimental llama.cpp support to Windows ML, the OS-level inference layer spanning AMD, Intel, Nvidia and Qualcomm silicon. A new Text Generation API now accepts both GGUF and ONNX models through one call, and Windows ML picks the backend itself — llama.cpp for GGUF, QNN for Qualcomm NPUs, OpenVINO for Intel NPUs, NvTensorRtRtx for Nvidia GPUs, CPU as fallback — according to Microsoft's Foundry on Windows blog.

The pitch is that any Copilot+ PC's idle NPU becomes a practical GGUF target without a developer hand-wiring the backend, as byteiota.com notes. It's still labeled experimental, and Microsoft hasn't published independent throughput numbers for the llama.cpp path.

Folding the most widely used open quantized-model runtime into the native Windows inference stack matters more than another chatbot app: the glut of GGUF models already on Hugging Face gets a first-class, driver-level route onto NPU silicon OEMs are shipping now.

AI Agents Designed Their Own Inference Chip

On October 6, 2026, developer FeSens published openTPU, an Apache-2.0 accelerator stack built largely by AI coding agents: SystemVerilog RTL, a custom instruction set, a bit-exact simulator, a compiler and a profiler in one repo. Tested on an Inspur YPCB-00338 card with a Xilinx Kintex-7 xc7k480t FPGA and dual DDR3 channels, it runs ten real models — Qwen3.5, Gemma 4 E2B, SmolLM3-3B, Phi-4-mini among them — with output matching the simulator bit for bit, per bytechap's writeup.

One bitstream at 133.33 MHz serves every model, and a 4-bit weight format cuts bytes-per-token by roughly a third, lifting decode speed 40–45%, per the project's own figures reported by a technical breakdown on DEV Community; decode ranges from 3.75 tokens/sec on larger models to 85.8 tokens/sec on the smallest.

This is a proof of concept, not shippable silicon — 133 MHz on an eight-year-old FPGA trails any modern NPU badly — but it hit Hacker News' front page because it's a fully open hardware-and-software stack, iterated by agents rather than a chip team, running real weights end to end.

ExecuTorch Becomes a Zephyr Module

On October 9, 2026, the Zephyr Project announced that ExecuTorch is now a Zephyr module — add it to the manifest, configure it in prj.conf, and build it with west like any other Zephyr component, with no separate toolchain required.

Zephyr already runs on hardware from Nordic's nRF chips to STM32 microcontrollers, so wiring PyTorch's edge-export runtime directly into its build system removes a real integration tax: getting ExecuTorch onto an RTOS target previously meant hand-stitching a separate build and link step. The announcement doesn't include new benchmarks, so there's no fresh latency or memory number to report yet.

It's plumbing, not a new model or a speed record. But plumbing decides whether a quantized model exported from PyTorch actually ends up running on the next RTOS-based sensor board instead of staying a demo on a dev machine.

A 27B Model Splits the Difference Between GPU and NPU

A self-reported benchmark posted to r/LocalLLaMA on October 9, 2026 clocked Qwen3.8-27B at 159 tokens/sec on an AMD Radeon AI R9700 workstation GPU and 64 tokens/sec on AMD's Strix Halo APU, per the LemonSeed Studio post. The rig pairs an iPad-based editor/IDE with inference running on the AMD GPU over a Thunderbolt enclosure, using unmodified upstream Linux amdgpu and amdkfd drivers rather than a vendor fork.

These are one user's numbers on one machine, not a controlled lab test, and the R9700 is a discrete workstation card rather than something that fits in a laptop. But the gap — more than double the throughput on discrete silicon versus the integrated NPU/APU combo in Strix Halo — is a useful data point for anyone weighing whether an onboard-NPU 'AI PC' can carry a 27B-class model on its own, or still needs a GPU in the loop.

Apple Silicon Gets a Megakernel Compiler for Local LLMs

LithosAI open-sourced lithos-metal this week, a tool that generates 'megakernels' for Apple's Metal API. The company's own benchmark has it serving Qwen3.8-27B at more than 200 tokens per second per user on an M5 Max chip — a self-reported, vendor number, not independently verified, but notable because it targets interactive use like a local coding assistant rather than batch generation.

Megakernel generation fuses the many small GPU operations a transformer needs into fewer, larger kernels, cutting overhead that normally eats into throughput on consumer GPUs. It's the same idea behind the Uzu engine that dailydoseofds.com traced back to a technique Andrej Karpathy sketched out, now showing up as production tooling.

For Apple Silicon machines with strong unified memory bandwidth, a compiler-level speedup like this moves a 27B dense model from 'runs, slowly' to something close to responsive, with no cloud call involved.

Five different layers moved this week — OS runtime, FPGA silicon, RTOS build system, GPU driver, and Metal compiler — all pointed the same direction: more of the token generation happening on the box in front of you rather than a server somewhere else.