When comparing fine-tuning recipes with very different trainable-parameter counts under a single learning rate, the smallest option can win by statistical artifact rather than real merit. The case comes from post-hoc SVD-based KV-cache compression, but the lesson applies to any parameter-efficient comparison.

The shared learning rate artifact

A pretrained model is converted into a low-rank cache in a multi-head latent attention style by factorizing key and value weights into a down-projection (encoder) and an up-projection (decoder). After truncation, a short healing fine-tune recovers the accuracy lost. With a shared learning rate, freezing the decoder and training only the encoder looked like a clear win over training the decoder or both factors. Once each arm gets its own tuned learning rate, that advantage disappears and encoder-only healing reaches parity with the alternatives.

The mechanism is clear: when arms differ in parameter counts, a shared rate can depress the means of larger arms and inflate their variance, producing a false multi-seed signal for the smallest arm. The authors verified this on a vision-language model, Qwen2.5-VL-3B-Instruct, with three seeds per configuration and one compression ratio, plus a replication on a text-only testbed with two backbones.

What changes for self-hosting

The fact that encoder-only healing is not superior does not make it irrelevant. It keeps a real, measured saving: 3x fewer trainable parameters and 3x less optimizer-state memory. For teams running self-hosted LLMs or working under VRAM constraints, that means a lighter fine-tuning and a less demanding retrofitting pipeline for KV-cache compression at training time. It is not a free shortcut: parity emerges only with per-arm tuning, and the protocol covers a single compression ratio. But it shifts attention from inflated benchmark scores to concrete memory savings, the metric that matters when hardware is a constraint and TCO depends on how much memory is actually needed.

The second-order consequence is for benchmark designers: if such a common protocol can manufacture an apparent advantage, future comparisons should tune the learning rate per arm instead of treating it as a convenient constant. The same applies to evaluating different compression or low-rank adaptation approaches: without that control, recipes get promoted for optimization artifacts rather than real efficiency. AI-RADAR tracks these issues when analyzing trade-offs for on-premise deployment, where fine-tuning and compression decisions have measurable effects on memory and cost. The next time a comparison shows a small, brilliant winner, it is worth asking whether the learning rate was really the same for everyone.