It is no longer enough for a Large Language Model to generate text or code: today we ask it to prepare the learning material for other models. DataPrep-Bench is the first unified benchmark that puts this very ability to the test, treating the entire data preparation pipeline as a task in its own right, evaluable in a rigorous and reproducible way.

The underlying idea is as simple as it is radical: the quality of training data determines the capabilities of an LLM more than any other variable. Yet until now there was no tool to objectively measure how well an LLM, a software agent, or a data-centric workflow could truly extract value from heterogeneous raw sources. DataPrep-Bench fills this gap by splitting the problem into two complementary abilities. On one side, data construction – transforming unstructured sources into supervised examples ready for fine-tuning – and on the other, evaluating the quality of that data before training even begins. In both cases, "quality" does not mean formal cleanliness or absence of duplicates, but predictiveness with respect to downstream performance: a dataset is good if the model trained on it works better.

For teams operating on-premises and investing in fine-tuning models on proprietary data, this distinction is far from academic. Anyone setting up a self-hosted training environment knows that the computational cost of each run is high and getting the dataset composition wrong means wasting GPU hours. Being able to rely on ex ante quality evaluation methods, like the Distributional Alignment Score (DAS) introduced with the benchmark, changes the game. DAS uses Maximum Mean Discrepancy (MMD) to compare the statistical distribution of the candidate dataset against a domain proxy, achieving a correlation with final performance that in four out of six domains surpasses all other tested evaluators. It is the only metric to clear the r > 0.70 threshold simultaneously in math, science, and medicine, proving that distributional similarity is a solid predictor of training value.

On the data construction front, the benchmark adopts a concrete protocol: all competing methods start from the same raw sources and are judged by how much fine-tuning a base model on their outputs – together with Dolly-15k – improves its performance across the six domains considered. In this context, the researchers also released Data-Construction-Skill, a skill-guided agent that raises the bar by nearly 20 absolute points on Llama-3.1-8B in the finance domain, proving competitive with the most advanced agent- or DataFlow-based approaches in knowledge-extraction-intensive domains.

What emerges from DataPrep-Bench is a structural signal: data preparation for training is no longer an ancillary activity that each lab improvises in its own way. It is becoming a subfield of LLM engineering with its own metrics, standardized pipelines, and measurable objectives. For organizations that treat data sovereignty as a foundational requirement – for example in healthcare, legal, or defense – the ability to rigorously evaluate and select datasets before training models on-premises enables them to maintain control over the entire AI lifecycle, reducing dependence on external providers and minimizing the risk of data leakage. The benchmark does not offer magic wands, but a downstream-grounded framework that equates data construction and data evaluation as two sides of the same coin, shifting the focus from raw model power to the discipline of the data that feeds them.