Training artificial intelligence models is becoming significantly cheaper. Andrej Karpathy estimates a cost reduction of approximately 40% per year for training models like GPT-2. This decrease is attributable to several factors, including advances in hardware, software, and algorithms.

Key Factors in Cost Reduction

  • Flash Attention 3: An optimized implementation of attention that offers an improvement of approximately 9% in throughput (tokens/sec). The unification of APIs for training and inference and the native tensor layout contribute to this efficiency.
  • Sliding window attention: The implementation of SSSL patterns allows saving computational resources without compromising the quality of the model.
  • Muon optimizer: A complete overhaul of the Muon optimizer, with the introduction of Polar Express and NorMuon for variance reduction, and a cautious approach to weight decay with a linear schedule.
  • Per-layer residual scalars: The use of residual scalars for each layer (x = ฮป_resid * x + ฮป_x0 * x0) has shown a consistent improvement across models of different sizes.
  • Value Embeddings at alternating layers: Placing value embeddings in alternating layers has proven more effective than other configurations.
  • BOS-aligned dataloader: The use of a dataloader aligned to the BOS (Beginning of Sequence) token has made mid-training unnecessary.
  • Hyperparameter sweep at scale: Running a wide search for hyperparameters (320 experiments) made it possible to identify optimal values, highlighting how small-scale tuning is not always transferable.
  • Scaling law discovery: The empirical measurement of the optimal ratio between tokens and parameters (approximately 10) is crucial for optimizing the training of neural networks.

For those evaluating on-premise deployments, there are trade-offs to consider carefully. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.