The striking figure is not the $164 in rented GPU time. It is that six serious defects passed loss-curve inspection without announcing themselves. A researcher working alone, with no PyTorch and no Python in the training path, pretrained an LLM end-to-end in Rust and documented an experience that matters for anyone evaluating non-mainstream frameworks. The result is not a recommendation: it is a measured taxonomy of silent failures in the two leading Rust ML frameworks, Candle and Burn, used as training backends in 2026.
The report describes five Candle defects, including fused kernels that silently produce no gradient, and three Burn defects, including a backward pass running at around 3% of theoretical GPU throughput and a kernel-fusion path that segfaults mid-training at multi-billion-parameter scale. All six passed ordinary loss-curve inspection; none announced itself. In practice, the model appeared to be learning while parts of the network stayed frozen or produced wrong computations.
What made the difference was a verification discipline built around a gradient-flow arbiter: a test that runs one forward/backward pass and checks that every trainable parameter receives a finite, nonzero gradient. The approach generalizes to any framework and caught the six failures. A loss curve can fall even when some branches are not contributing; the aggregate signal hides individual dead spots.
The project also exposed a tokenizer-fertility trap in Bengali script. With naive byte-level tokenization, Bangla collapsed to about 1.4 characters per token against English's 3.9, quietly inverting the corpus language balance; the fix brought the ratio to about 4.1. In multilingual datasets, tokenization is not a detail: it decides how much text each language can transmit within the same context window.
The model, roughly 0.4 billion parameters and Bangla-first, shows a strong signal: a per-token negative log-likelihood of 0.93 against 12.60 for a random-initialized twin. It scores at chance on English commonsense multiple-choice, the expected outcome for a deliberately small, Bangla-weighted budget (about 2 billion tokens, 54.6 hours on one rented H100).
The author's next step is a signal for the ecosystem: training moved to PyTorch, while Rust stayed for on-device serving. In practice, Rust is not yet a competitive place to train an LLM, but it may be a good place to serve one. For teams thinking about self-hosted or on-premise stacks, that does not mean discarding Rust frameworks outright; it means their value today sits on inference and that, before relying on them for training, gradient-flow tests are needed rather than aggregate metrics. The real cost of silent failures does not appear on the GPU bill until it accumulates in models that look valid but are not. For those comparing cloud and on-premise options, AI-RADAR offers analytical frameworks at /llm-onpremise for evaluating these trade-offs.
๐ฌ Comments (0)
๐ Log in or register to comment on articles.
No comments yet. Be the first to comment!