A recent experiment explored the architecture of large language models (LLM), focusing on the effect of repeating layers within the model.

Experiment Details

The experiment, named RYS II, used the Qwen3.5 27B model and tested the hypothesis that LLMs may develop a kind of internal "universal language." Analysis of the latent representations in the middle layers of the model showed greater similarity between identical content in Chinese and English than between different content in the same language. This suggests that the model may abstract concepts to a deeper level, independent of the input language.

Architecture and Results

Repeating blocks in the middle layers of the transformer architecture proved to be the most effective strategy. Several pre-trained models have been made available on Hugging Face, with different configurations. The researcher suggests that fine-tuning the models with repeated layers could lead to state-of-the-art (SOTA) results for models of this size.

Considerations

The original article mentions optimizing VRAM usage through specific formats. For those evaluating on-premise deployments, there are trade-offs between performance, TCO, and memory requirements that AI-RADAR helps to evaluate.