The weights of a trained Transformer don't grow randomly. In a study of training dynamics, weight magnitudes follow a Weibull distribution with shape k≈1.2 stable across layers and architectures, while the scale λ carries most of the training-induced movement. The question becomes: which corpus property determines how much λ grows? The proposed answer is bigram conditional entropy D=H(next|prev), a statistic computed before training.

The researchers used controlled corruption families and measured a learning-rate-conditioned law: λ²−λ₀² = C₀(η)+C₁(η)(H_r−D)^0.59, where H_r is a matched-budget shuffle baseline. The convex exponent 0.59 is not fitted directly to the growth curve; it is inherited from a data-side saturation relation. After removing the two learning-rate-specific coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto (H_r−D)^0.59 with unit slope and R²=0.941. Direct per-learning-rate fits are weaker, with R² around 0.82.

The most interesting part is that D is a pre-training statistic. In an end-to-end self-validation, the researchers recovered held-out within-family weight growth with 5.7% relative error. The result holds at model and per-layer resolutions, and across two tested architectures: the functional form is preserved, only the coefficients change.

The structural message is not just that data matters: a single low-cost statistic can separate part of the variance in weight dynamics before spending compute. For teams working with self-hosted stacks or constrained training budgets, estimating weight-scale growth from a corpus analysis opens the door to more informed data selection. But the coefficients C₀ and C₁ depend on learning rate and architecture: this is not a universal law ready for deployment, but a signal that data predictability can become a control variable alongside parameters and compute.

The failure on code is the most instructive part. Cross-corpus prediction over-predicts weight growth on code. The researchers interpret this as evidence that redundancy is a second axis, within a broader Φ(D,R,A,H) data-to-weight framework. On code, structural repetition is not captured by bigram entropy alone. For teams working on code LLMs or technical data, a metric based only on conditional entropy risks being misleading.

At an industry level, this kind of research shifts incentives: rather than comparing only models and optimizers, training teams can invest in corpus diagnostics. Data curation tooling could integrate entropy and redundancy estimates before training, reducing exploratory runs. The fact that the exponent is inherited from a data-side saturation relation, rather than fitted, is crucial: it lowers the risk of statistical overfitting and gives the law a more solid foundation.

The boundary is clear: within the same corpus family, the prediction holds; as soon as the domain changes, a second axis is needed. It is not a promise of miracles, but a different way of looking at data: not as undifferentiated fuel, but as a space with its own predictive structure.