A $300 GPU Streams a 177B AI Model From an SSD While llama.cpp Learns to Skip Ahead

llama.cpp Gets a Faster Way to Skip Ahead

Developer jadidbourbaki published a write-up on September 26, 2026 detailing a faster implementation of prompt-lookup drafting in llama.cpp — a speculative decoding trick that drafts candidate tokens by matching n-grams already in the prompt, instead of running a separate small draft model. The post walks through the matching-and-verification loop and the tuning it needed to actually pay off, and drew a 12-comment discussion on Hacker News.

The honest caveat, laid out in a separate analysis on startupfortune.com: testing several of llama.cpp's speculative modes on Qwen3.6-35B-A3B on an RTX 3090 found none beat the plain baseline for that setup. Prompt-lookup drafting shines specifically when the output repeats chunks of the input — code edits, summarization, structured extraction — not general chat. For edge devices running the same repetitive local tasks, that narrow win is still a real one, since it needs no extra model weights in memory.

A 177B Model Streams Off an SSD Into 16GB of VRAM

A r/LocalLLaMA poster describes an inference engine called InferredThoughts that keeps most of a 177B-parameter Mixture-of-Experts model — Qwen3.8-Flash-Next, quantized to NVFP4 at 119GiB on disk — sitting on an SSD, reading only the experts each token actually needs. On a 16GB RTX 5060 Ti plus 32GB of system RAM, the builder reports 9 to 10 tokens per second — self-reported, no independent benchmark run yet.

A separate thread has someone running the same model family's dense 3.8B-27B variant on an M4 Pro MacBook with 48GB unified memory and calling it noticeably faster than the MoE version. Neither result is a formal benchmark, but the SSD-streaming approach is the more interesting story for the edge beat: it trades VRAM for patience, letting a consumer card that can't hold a fraction of the model still produce usable tokens from it.

Apple's Built-In On-Device LLM Gets a Developer Wrapper

A developer on r/LocalLLaMA posted on September 28, 2026 that Apple Silicon Macs running macOS 26 and up already ship with a small local LLM baked in — no download, no API key, nothing leaving the machine — and built a package to call it easily from Node and Python, outside Apple's native Swift framework.

The model itself isn't new — it's Apple's on-device Foundation Model, sized for tasks like text rewriting and summarization rather than open-ended chat — but making it reachable from ordinary scripting languages lowers the bar for hobby projects that want private, zero-cost inference on hardware people already own. The claim about which macOS build ships this is self-reported by the poster and worth verifying against Apple's own release notes before relying on it.

EWatch Puts an ESP32-S3 on Your Wrist, Drivers Included

Developer Ewan Wills has built EWatch, an open-source ESP32-S3 smartwatch with a 1.69-inch color touchscreen, 350mAh battery, vibration feedback, and USB-C charging, according to LinuxGizmos' report on September 27, 2026. The project ships open drivers and leaves room for DIY assembly rather than locking buyers into a sealed gadget.

There's no on-device model here — it's a wearable platform, not an AI demo — but it's exactly the kind of board that ends up running keyword-spotting or gesture models once someone bolts a microphone or IMU pipeline onto it, and the open driver stack means that someone doesn't have to reverse-engineer the hardware first.

Sony's Virtual Dog Gets a $40 New Home

Adafruit's blog reported on September 25, 2026 that a project called AiboJam brings Sony's old desktop AIBO character to Adafruit's Fruit Jam board, priced at $40, running on CircuitPython with a native C component.

It's a nostalgia build, not an inference story, but it's a useful data point on the beat's other half: a $40 board with a display and enough headroom for CircuitPython and native code is the same class of hardware people are now loading TinyML keyword spotters and small vision models onto. The Fruit Jam's price and openness matter more than the dog does.

Nothing here is a launch-day headline — it's the usual grind of people pushing weights onto smaller boxes, whether that box is a $300 GPU with an SSD strapped to it or a $40 badge with a screen. That grind is the beat.