Eighty-six and a half billion tokens versus eighteen trillion. That is the first number that stands out in the Apex-2 experiment, a decoder-only MoE model with 3.87 billion total parameters and 1.45 billion active per token, trained from scratch without reusing external weights. The base checkpoint reached HumanEval+ parity with Qwen2.5-1.5B, a model trained on roughly 18 trillion tokens. The scale difference is enormous; the technical signal, on the other hand, is more nuanced.

Code learns fast, knowledge doesn't

The reported benchmarks show a sharp profile: HumanEval 43.9, HumanEval+ 41.5, MBPP 56.3, MBPP+ 48.9. On code, Apex-2 holds up against much more expensive pre-trained models. But GSM8K stops at 32.4, MATH-500 at 21.0, and MMLU at 28.6. The message is that syntactic patterns and coding procedures can be learned with far fewer tokens than factual knowledge and mathematical reasoning. A model can generate correct code on narrow benchmarks without having the knowledge base needed to avoid hallucinations or handle complex math.

That separation has practical implications for evaluating small models in production. A MoE with 1.45 billion active parameters per token has a potentially lower per-step inference cost than a dense model of similar code quality, but the memory needed to hold all 3.87 billion parameters does not disappear. In a self-hosted context, the advantage is not the VRAM occupied by the experts, but the active computation per token. The 4096-token context window remains a limit that rules out workloads on large codebases or extended retrieval. The checkpoint is available on Hugging Face and loads with transformers or vLLM via the Qwen3MoeForCausalLM mapping.

GH200, DiLoCo, and what didn't work

The other system-level detail is training on GH200, scaled from one to two with DiLoCo. It is not a home-lab setup, but it is far from the clusters used for models with hundreds of billions of parameters. Using two GH200 with reduced communication suggests that pre-training a small MoE can be tackled outside large cloud data centers, with implications for data sovereignty and control over the development cycle. There are no energy or TCO figures in the source; reading it as proof of economic convenience would be a mistake. Still, the experimental sequence — one researcher, 86.5 billion tokens, two GH200 — shifts attention from the number of GPUs to data quality and architecture choice.

An equally useful part is what did not work. DPO with 220k pairs and length normalization produced longer answers and hurt code, math, and IFEval. The checkpoint was dropped. That deserves attention because DPO is often presented as a nearly free step for aligning models. In this case it degraded exactly the abilities where Apex-2 was competitive.

For those looking at local and self-hosted models, the Apex-2 case shows that the from-scratch path on controlled hardware is technically possible for specialized models, but the price in general knowledge is high. It is not a replacement for a generalist LLM; it is a signal that the token-count race can be bypassed for some tasks, not all. AI-RADAR offers analytical frameworks on /llm-onpremise for evaluating these deployment trade-offs. The open question remains how much factual knowledge a model really needs when its job is writing code in a narrow business domain.