Research on quantization, fine-tuning without backprop, block attention and routing shows progress on efficiency, while coding benchmarks reveal generalization limits.
ECASQ jointly optimizes adaptive quantization and lossless compression by minimizing MSE under an entropy budget and an unbiasedness constraint. The optimal dynamic program runs in O(sd^2) time and O(d^2) space. A GPU-friendly approximate version red...
FPO proposes fine-tuning LLMs without propagating errors between layers or building autograd graphs. The method reduces peak training memory and increases throughput, but concentrates adaptation in the final layers. For self-hosted deployments, the o...
FPO adapts LLMs without a backward pass through the model body, reaching 2.7–3.2x the throughput of standard fine-tuning and about 40% less peak training memory. On OLMo-2-7B, Qwen3-8B, and Falcon3-7B, it improves in-domain perplexity while leaving M...
Fine-tuning on SWE-bench does not transfer capability to suites like Django or LiveCodeBench. The benchmark becomes a training target and stops measuring general ability. For self-hosted LLM operators, an inflated score distorts VRAM, quantization, a...
BCMT separates local token interaction from global context propagation. In tests up to 1024 tokens, it achieves validation performance comparable to Dense Transformers, with higher training throughput and lower memory consumption. The exponential cau...
SELR introduces a single model that reasons in latent space and decodes its own reasoning into human-readable steps. A multi-task objective combines Answer Loss and CoT Loss, removing external decoders and keeping the explanation tied to the actual r...
Optimizing an LLM on popular coding benchmarks does not improve general programming ability. A new Django-based suite shows rankings often fail to generalize, and fine-tuning on SWE-bench yields limited or no gains on other tasks. Evaluation must be ...
A study on Qwen3.6-35B-A3B shows that sensitivity to expert masking in MoE models depends heavily on depth. Early and middle layers are fragile, while late layers tolerate aggressive cuts. On H100 servers, targeted late-layer policies preserve far mo...
