Methods like DASA and external n-gram memory fine-tune or augment models without increasing the active model footprint. Low-rank autoencoders and hardware-aware NAS further reduce FLOPs and memory for small deployments.
DASA uses activation gradients from a frozen reference model to produce synthetic embeddings, fine-tuning LLMs without readable text. Tests on Llama and Qwen show performance similar to or better than natural language data, with up to 4.9x speedup ov...
A new method, DASA, generates synthetic fine-tuning data as continuous embeddings driven by activation gradients from a frozen reference model. In tests on six Llama and Qwen models with LoRA, it matches or surpasses natural-language data and speeds ...
ENAS is a hardware-aware Neural Architecture Search framework for TinyML that does not require GPUs. Tested on Visual Wake Words and Melanoma Cancer across eight microcontrollers with 20 KB to 1 MB SRAM, it speeds up search by up to 2.41x over NanoNA...
A new approach replaces attention in Transformers with bottleneck autoencoder modules plus iterative refinement at masked positions. In tests with BERT and C4 it reaches a significant share of attention performance with about 1.9× fewer FLOPs and mat...
Qwengram-0.8B freezes Qwen3.5-0.8B and adds a 51-billion-parameter n-gram memory outside the GGUF. Perplexity drops by 5.05%, but the relevant signal is architectural: late injection controlled by a per-token gate, Q8_0 quantized sidecar, and no back...
An experiment keeps Qwen3.5-0.8B frozen, adds a 51-billion-parameter PLE memory and an R=1 reader: validation perplexity drops by 5.05%. The author warns this is not benchmark accuracy gain and releases GGUFs, a modified llama.cpp runtime, and a quan...
