The headline claim is simple: a 26-billion-parameter model (Gemma 4 26B) running at about 5 tokens per second on a Xeon CPU that's roughly 13 years old, with no graphics card involved. That's slow by GPU standards, but it's a usable speed for batch jobs, background tasks, and interactive use where you don't need instant responses.
Why this matters: it undercuts the assumption that you need expensive, current-generation hardware to run capable local models. CPU inference leans on system RAM instead of scarce VRAM, and old server-class machines are cheap on the secondhand market precisely because they have lots of memory channels and core counts. If you already own aging hardware, the marginal cost of experimenting with a large model is close to zero.
The practical enabler here is quantization. Running a 26B model on a CPU realistically means using a heavily compressed version (typically 4-bit or similar GGUF quants) through a runtime like llama.cpp or Ollama. Quantization shrinks the memory footprint enough to fit in ordinary RAM and reduces the arithmetic per token, which is what makes single-digit tokens-per-second achievable on old silicon.
What you can do with it: reach for CPU-only inference when latency isn't critical—document summarization, offline drafting, code assistance in a home lab, or a private assistant that never sends data to a cloud. Five tokens per second is roughly reading speed for short replies, so it's fine for one user and awkward for anything real-time or multi-user.
To replicate it, start with a quantized GGUF build of your target model, install llama.cpp or Ollama, and tune thread count to match your physical cores. Watch memory bandwidth: on older Xeons that's often the real bottleneck, not raw clock speed, so populating all memory channels and using the smallest quant that preserves quality is where you'll find your gains.
The broader takeaway is that the barrier to running large local models is dropping on the software side, not just the hardware side. Before buying a GPU, it's worth benchmarking what your existing machine can already do—you may find it's good enough for the workloads you actually have.