Fine-tuning an LLM almost always starts from one assumption: you need readable, annotated, or curated text. A new line of work challenges that, asking whether human readability is really necessary. The proposed answer is DASA, Desired-Update-Aligned Synthetic Data, which uses activation gradients from a frozen reference model to generate continuous synthetic embeddings.

Beyond text: how DASA works

Instead of reconstructing the source text or enforcing linguistic fluency, DASA targets useful adaptation updates for the model. The embeddings are used directly for downstream fine-tuning; discrete token projections serve only for qualitative inspection.

The tests cover six models from the Llama and Qwen families, ranging from 1B to 32B parameters, across six benchmarks spanning knowledge, mathematical reasoning, code generation, and commonsense reasoning. Under matched LoRA settings, DASA achieves performance comparable to natural-language data and surpasses it in multiple configurations; it also outperforms GRADMM in most comparisons. With the evaluated synthesis settings, DASA provides a 3.6–4.9x speedup over GRADMM with comparable peak GPU memory.

Gradient as the new unit of data curation

The implicit thesis is that the quality of a fine-tuning dataset is not a property of the text itself, but of the update direction it induces in the model. It is a shift in perspective: rather than asking whether a sentence is well written or an instruction is well phrased, the question becomes whether the produced signal moves the model toward a target behavior.

This has second-order consequences. Data curation moves from linguistic-editorial work to the design of feedback signals and the choice of reference model. Teams with optimization skills and compute infrastructure benefit more than those with access only to textual corpora. At the same time, the portability of synthetic data may decrease: an embedding generated with a frozen reference model is tied to that model's activation geometry, so it is not obvious that it will transfer well to different architectures.

There is also an accountability issue. If training data is no longer readable, reconstructing what the model actually learned during fine-tuning becomes more complex. Discrete token projections are qualitative only, not a faithful representation of the signal. For teams operating in regulated or audited environments, this is not a minor detail: it changes the kind of documentation needed to demonstrate the provenance and nature of updates.

For teams managing self-hosted models, the method also touches the trade-off between data preparation and compute. On one hand, it reduces dependence on curated textual datasets; on the other, it introduces an optimization step for synthetic embeddings that requires a frozen reference model and activation gradient computation. The speedup over GRADMM and comparable GPU memory suggest the bottleneck is not necessarily hardware in absolute terms, but the type of signal one decides to use. For those evaluating on-premise deployment, trade-offs exist between text preprocessing cost and synthesis cost; AI-RADAR offers analytical frameworks at /llm-onpremise to examine them.

The point is not that text disappears, but that for fine-tuning it may become an inspection byproduct rather than the main input.