Four open decision models on a rented RTX 4090: a test just posted on Reddit put Laya, Liquid AI's d1, Cloudflare's Clef-Flash, and Interfaze's Lev to work reading nine Wikipedia articles on centipedes, 9,534 words in total, flagging every word that names a centipede. The Python client ran on the same box, over localhost, to isolate end-to-end latency. The protocol was deliberately brutal: one /v1/systemone call per word, with the next request sent as soon as the answer came back. This is not a continuous generation benchmark. It is a stress test on per-request latency, serving overhead, and a runtime's ability to handle thousands of micro-inferences without collapsing.

The numbers deliver a ranking but also a misunderstanding. Laya-BF16 on llama.cpp b11495 closed at 3.9 ms per word (p50) and 7,980 words processed in 32 seconds. d1-3B in AD-Q4_K_M quantization, also on llama.cpp, recorded 6.0 ms and 5,306 words. Clef-Flash-9B in Q8_0 dropped to 1,292 words at 24.4 ms. Lev-4B, running BF16 on the PyTorch lev serve server, reached 626 words at 51 ms. In raw speed, Laya and d1 pull away from the others; but the raw number does not tell the whole story.

The accuracy trap

The first problem is metric: reported accuracy ranges from 96.5% to 98.9%, but in a highly imbalanced task like this, answering "no" for the vast majority of words automatically produces high scores. The share of centipede names actually caught is far more informative: Lev catches 83%, Laya 70%, d1 51%, Clef-Flash 36%. The slowest model is also the one that misses the real target least, while the faster models trade speed for more false negatives or false positives. Anyone evaluating a decision model for an operational flow should look at precision and recall on the specific use case, not aggregate accuracy.

The runtime as bottleneck

The second point is structural and concerns the runtime ecosystem. On the same GPU, Laya and d1 use llama.cpp, a C++/CUDA runtime optimized for local inference, while Lev pays a visible cost for the PyTorch path. The gap reaches roughly 13x, and the author notes that on a different RTX 4090 Lev measured 68 ms, so the result is also CPU-sensitive. With identical hardware, the serving engine becomes a first-class performance lever, not an implementation detail. For teams building self-hosted decision pipelines, the model is not the only thing to evaluate: it must be tested inside the runtime that will run in production.

There is also a third-order signal. The availability of small open decision models, combined with lightweight runtimes like llama.cpp, is shifting the center of gravity of local inference toward consumer GPUs. You do not need a datacenter to run high-frequency classification on a single document: a card like the RTX 4090 and a local server are enough. For those with data sovereignty constraints or reluctance to send text to external endpoints, this makes on-premise decision models more concrete, even if the GPU here was rented and therefore not an air-gapped environment. The unresolved problem is the maturity of alternative servers: the fact that --compile never finished warming up for Lev says a lot about the variability of emerging stacks.

In the end, the test paints a more interesting picture than a simple ranking: Laya is the fastest and, according to the author, easy to fine-tune for specific cases; d1 offers a middle ground; Lev shows that target accuracy can come with a non-trivial latency cost. The real message for local deployment is that bottlenecks are shifting from parameter count to the runtime and to the metric used to define success.