LLMs and Geometric Representations: A New Perspective

A recent research posted on r/LocalLLaMA explores how Large Language Models (LLMs) internally represent information. The findings suggest that, at a deep level, these models may not "think" in terms of language, but rather through a kind of conceptual geometry.

The experiment involved four different models: Qwen3.5-27B, MiniMax M2.5, GLM-4.7, and GPT-OSS-120B. All four were found to exhibit the same behavior: sentences describing the same concept (e.g., photosynthesis) in different languages (English, Chinese, Arabic, Russian, Japanese, Korean, Hindi, and French) are closer to each other in the model's internal space than sentences describing different concepts in the same language.

Multimodal Convergence

Even more interesting is the finding that natural language descriptions, Python functions (with single-letter variables), and LaTeX equations of the same concept (e.g., kinetic energy: ½mv²) converge in the same region of the model's internal space. This suggests that the universal representation is not only language-independent but also input modality-independent.

These results, replicated across dense transformers and MoE (Mixture of Experts) architectures from different organizations, suggest that this is a convergent solution and not a specific artifact of a particular model or training.