In Transformers, attention is the dominant mechanism for mixing context across tokens. But it is not the only way. A recent line of work replaces attention with fixed MLPs or hypernetwork-generated MLPs, giving up part of the dynamic flexibility to reduce computational complexity. The work described here tries a different route: a low-rank bottleneck autoencoder instead of attention.

This is not just an academic experiment. The choice to replace attention with compression and reconstruction modules has concrete implications for anyone running language models on limited hardware or in on-premise environments, where every saved FLOP matters in TCO calculations.

The architecture uses a stack of autoencoder-based mixing modules. The first operates over local neighborhoods, the second over the full sequence, and the third across attention heads. Each module compresses its input through a bottleneck and then reconstructs it. The bottleneck width is a hyperparameter, not a training effect. That is an important detail: it shifts design from learned dynamics to a configuration chosen upfront, making the model more predictable in terms of compute constraints. For teams planning self-hosted deployments, having an explicit parameter to control mixing capacity is useful because it allows reasoning more directly about the trade-off between quality and resources.

At masked positions, the model does not stop at a single pass. It introduces an iterative procedure with two steps. A pulling step moves the embedding representation toward a weighted average of its neighbors. A correcting step projects the result back onto the manifold learned by the autoencoder. The idea of manifold projection combines local context with a regularity constraint: the corrected embedding is not just a raw average, but a plausible point in the latent space.

In tests with pretraining on C4 and parameter-matched BERT baselines, the architecture reaches a significant portion of attention performance while using about 1.9× fewer FLOPs. It is not an overtaking, but a signal: much can be achieved without paying the full price of attention. Moreover, the model matches parameter-matched BERT and TinyBERT on the rarest-token bucket when using a frequency-aware training schedule that samples rare tokens more often than uniform sampling for masking tasks. This detail is often overlooked: the improvement on rare tokens comes not only from the architecture, but also from how the model is trained.

What does this mean for the on-premise landscape? The news is not that attention can be replaced, but that the design space for more FLOP-efficient models is expanding. Today attention benefits from optimized kernels and a mature hardware ecosystem. An autoencoder-based alternative can reduce theoretical FLOPs, but it does not automatically translate into lower latency if the operators are not equally optimized on existing GPUs. This is a crucial point for TCO evaluation: compute savings must be measured on real silicon, not just on paper. In addition, bottleneck width as a hyperparameter introduces a new tuning dimension: it is no longer enough to choose the number of layers or the hidden size; one must also decide how much to compress the context. For small teams or deployments with data sovereignty constraints, this can be both an opportunity and an added complexity. For those evaluating on-premise deployment, the analytical frameworks on /llm-onpremise help separate theoretical FLOPs gains from real integration costs.

The rare-token result deserves a final consideration. It shows that quality depends not only on the mixing architecture, but also on the training data distribution. Non-uniform sampling can close the gap on difficult classes without increasing FLOPs. For teams building local models on specific domains, this is a reminder: optimizing training can be cheaper than chasing more powerful hardware.