Optimizing inference on Large Language Models has always been a race against time. Models are released more frequently, architectures are increasingly diverse, and traffic patterns change; a static compiler tuned for a single model or chip family risks falling behind within a few weeks. Together AI tries to reverse this logic with Together Compiler (TKCC), a continuous compilation system built on agents that, according to data published by the company, outperforms hand-optimized kernels on Llama 1B by up to 1.7x and the best open source engines on the latest frontier models by up to 2.6x in TPS on the InferenceX benchmark.
The core difficulty is not finding a better optimization, but measuring it reliably. Agents working on individual kernels use offline benchmarks; the problem is that in mixture-of-experts models, behavior depends on data and those timings do not reproduce the real forward pass. Conversely, agents targeting end-to-end engines directly suffer from very slow launch times, which slow down every iteration. TKCC breaks this trade-off with a Tiered Persistent Engine that separates engine launch overhead from forward pass computation and with a cost model to select the initial parallelism configurations.
The conceptual leap matters more than any single benchmark. TKCC turns inference optimization from a one-time activity tied to the release of a hand-written kernel into a continuous process that adapts to real workload. This shifts competitive advantage from kernel-writing specialists to those who control the measurement pipeline and the agent state management. Anyone managing self-hosted infrastructure should read a clear signal here: the same VRAM and compute constraints can produce very different throughput depending on compiler quality. A 2.6x improvement on the same hardware base is not a niche detail: it potentially translates into fewer GPUs to serve the same traffic, hence a lower TCO while retaining control over data and latency.
The batch size 1 figure is also relevant. The 1111 TPS on MiniMax M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash indicate that optimization is not only about aggregating requests into large batches, but also about low-latency scenarios where every millisecond counts. For anyone serving frontier models in low-latency environments, this kind of result changes the required capacity calculation.
Generalizing beyond frontier language models, with trillions of tokens served in production, suggests that continuous compilation can become a system component rather than an experiment. For those evaluating on-premise deployments, there are trade-offs between hardware control and the availability of similar optimization loops; AI-RADAR offers analytical frameworks at /llm-onpremise to explore decision criteria. But the broader lesson from TKCC is that inference optimization is becoming a problem of measurement and automation, not brute force on kernels.