Researchers are cutting LLM costs with alternatives to attention, synthetic embedding fine-tuning, external n-gram memory, retrieval filtering and hardware-aware TinyML models.
DASA uses activation gradients from a frozen reference model to produce synthetic embeddings, fine-tuning LLMs without readable text. Tests on Llama and Qwen show performance similar to or better than natural language data, with up to 4.9x speedup ov...
A framework splits retrieval failures into two states: unanswerable and distracted. A lightweight module, Sieve, filters noisy documents before invoking the costly LLM, Sage. In tests, accuracy improves by up to 69.4 percentage points and Macro-F1 by...
A new method, DASA, generates synthetic fine-tuning data as continuous embeddings driven by activation gradients from a frozen reference model. In tests on six Llama and Qwen models with LoRA, it matches or surpasses natural-language data and speeds ...
ENAS is a hardware-aware Neural Architecture Search framework for TinyML that does not require GPUs. Tested on Visual Wake Words and Melanoma Cancer across eight microcontrollers with 20 KB to 1 MB SRAM, it speeds up search by up to 2.41x over NanoNA...
A new approach replaces attention in Transformers with bottleneck autoencoder modules plus iterative refinement at masked positions. In tests with BERT and C4 it reaches a significant share of attention performance with about 1.9× fewer FLOPs and mat...
Qwengram-0.8B freezes Qwen3.5-0.8B and adds a 51-billion-parameter n-gram memory outside the GGUF. Perplexity drops by 5.05%, but the relevant signal is architectural: late injection controlled by a per-token gate, Q8_0 quantized sidecar, and no back...
An experiment keeps Qwen3.5-0.8B frozen, adds a 51-billion-parameter PLE memory and an R=1 reader: validation perplexity drops by 5.05%. The author warns this is not benchmark accuracy gain and releases GGUFs, a modified llama.cpp runtime, and a quan...
A 0.4M-parameter multimodal model combines SMILES transformers and GATv2 graphs via cross-attention, reaching AUC-ROC 0.987 and F1 0.906 on ClinTox. The ChemBERTa-2-backed variant posts a mean AUC-ROC of 0.750 on Tox21, comparable to larger baselines...
