Research on low-rank autoencoders, synthetic embedding datasets and external n-gram memories targets fewer FLOPs and better small-model performance without relying on readable text.
DASA uses activation gradients from a frozen reference model to produce synthetic embeddings, fine-tuning LLMs without readable text. Tests on Llama and Qwen show performance similar to or better than natural language data, with up to 4.9x speedup ov...
A framework splits retrieval failures into two states: unanswerable and distracted. A lightweight module, Sieve, filters noisy documents before invoking the costly LLM, Sage. In tests, accuracy improves by up to 69.4 percentage points and Macro-F1 by...
A new method, DASA, generates synthetic fine-tuning data as continuous embeddings driven by activation gradients from a frozen reference model. In tests on six Llama and Qwen models with LoRA, it matches or surpasses natural-language data and speeds ...
A new approach replaces attention in Transformers with bottleneck autoencoder modules plus iterative refinement at masked positions. In tests with BERT and C4 it reaches a significant share of attention performance with about 1.9× fewer FLOPs and mat...
Qwengram-0.8B freezes Qwen3.5-0.8B and adds a 51-billion-parameter n-gram memory outside the GGUF. Perplexity drops by 5.05%, but the relevant signal is architectural: late injection controlled by a per-token gate, Q8_0 quantized sidecar, and no back...
An experiment keeps Qwen3.5-0.8B frozen, adds a 51-billion-parameter PLE memory and an R=1 reader: validation perplexity drops by 5.05%. The author warns this is not benchmark accuracy gain and releases GGUFs, a modified llama.cpp runtime, and a quan...
