36.25 tokens per second. That number isn't going to embarrass a modern GPU, but it becomes far more interesting when you learn that the engine behind it is a zero-dependency C99 binary—no Python, no CUDA, no BLAS—running on an off-the-shelf Intel Xeon with just four threads. The project, published on GitHub as project-zero, illustrates how a craft approach to ternary model inference can deliver respectable performance, and more importantly, reveals where the real physical wall—before any compute limit—blocks further progress.

BitNet, the family of models that compresses weights to just 1.58 bits (values -1, 0, +1), has always been a natural fit for CPU inference: the memory savings are extreme, and the required math boils down to integer addition. The problem is that most inference stacks treat even ternary weights as if they were float32, unpacking the bytes into full-precision vectors before multiplying. The creator of project-zero instead wrote custom SIMD routines for AVX2 and AVX-512, using VNNI instructions (vpdpbusds) to accumulate products directly in integer registers. Weights are packed four per byte and never promoted to float. The thread pool, built on C11 atomics with spin-then-yield backoff, eliminates synchronization contention during token generation. The result is a single executable that serves an OpenAI-compatible API endpoint, with no interpreted layers in sight.

All this cleverness, however, slams into a limit the CPU cannot dodge: DRAM bandwidth. At batch size 1, BitNet decode runs at roughly 95% of the system's theoretical memory bandwidth. That means the processor spends more time waiting for data than computing. Making the kernels even faster won't budge end-to-end token latency unless you batch across multiple sequences, thereby amortizing the cost of data transfers.

This is a structural signal that deserves careful reading. First, it proves that for extremely quantized model inference, the critical hardware is not the CPU but the memory subsystem. Choosing a server for on-premise deployment based solely on compute specs is misleading: memory channels, type (DDR5, HBM, potential CXL), and controller architecture matter far more. Second, it reaffirms that weight compression is not a magic wand—even at 1.58 bits, the bottleneck remains moving parameters from RAM to registers. Third, for those running LLMs in air-gapped environments or with data-sovereignty requirements, the combination of a hyper-quantized model and a zero-dependency C engine lowers the barrier to entry, but it demands investment in generous memory configurations.

There's also a methodological lesson. The zero-dependency approach, compiled into a single binary, eliminates entire classes of security and supply-chain issues: no Python stack to audit, no CUDA drivers to update. For those assessing TCO trade-offs for local infrastructure, such a runtime can reduce operational and maintenance costs—provided that performance under multi-user scenarios is validated. That work is still open: the repository asks for data from other CPU architectures, especially AMD Zen and ARM NEON, to see whether bandwidth saturation is a universal wall or whether some platforms can push it further. For on-premise deployers, the invitation is to measure your memory before you buy compute.