LLMs and Cognition: Model Representations Predict Human Reading Times

Understanding how Large Language Models (LLMs) process and represent linguistic information is a rapidly evolving field of research. While it is widely recognized that these models encode a vast array of linguistic data, it remains less clear whether their internal representations also capture cognitive signals related to human processing. Recent research has focused precisely on this aspect, investigating whether LLM representations can predict human reading times, offering new perspectives on the functional alignment between model depth and the temporal stages of human reading.

This study, published on arXiv, utilized a "probing" methodology to analyze the internal representations of LLMs. The probing technique involves training a simple classifier or linear regressor on a specific task, using as input the representations extracted from different layers of a pre-trained model. In this case, researchers employed regularized linear regression to compare the representations from each model layer against established scalar predictors, such as "surprisal," "information value," and "logit-lens surprisal." The analysis was conducted on two eye-tracking corpora, spanning five different languages: English, Greek, Hebrew, Russian, and Turkish, ensuring a robust and multilingual comparative basis.

Key Findings: Early Layers and Reading Measures

The research findings revealed an interesting differentiation in the predictive capability of the model layers. It emerged that representations from the early layers of LLMs outperform "surprisal" in predicting early-pass measures, such as first fixation duration and gaze duration. This concentration of predictive power in the early layers suggests that human-like processing signatures, in these early stages, are captured by low-level structural or lexical representations within the model. This indicates a potential functional alignment between model depth and the temporal stages of human reading processing.

In contrast, for late-pass measures, such as total reading time, scalar "surprisal" maintained predictive superiority, despite being a much more compressed representation. Researchers also observed that the combined use of "surprisal" and early-layer representations led to further improvements in predictive performance. It is important to note that the most effective predictor varied significantly depending on the specific language and eye-tracking measure considered, highlighting the complexity and variability of cognitive processes and their representations within the models.

Implications for LLM Architecture and Deployment

These findings have significant implications for the understanding and development of LLMs. The idea that early layers of a model can capture fundamental aspects of human cognitive processing suggests that the internal hierarchy of LLMs might, in part, mirror the hierarchy of cognitive processes. For CTOs, DevOps leads, and infrastructure architects evaluating LLM deployment, understanding which model layers are responsible for specific capabilities can be crucial. For instance, if an application requires rapid, low-level text comprehension, optimizing for Inference of early layers could offer advantages in terms of latency and Throughput.

In a self-hosted or on-premise deployment context, where control over hardware and software is maximal, this knowledge could guide decisions on model optimization or the choice of specific architectures. The ability to identify which parts of an LLM are most relevant for certain tasks can influence Quantization strategies, pruning, or the selection of smaller, specialized models, reducing the Total Cost of Ownership (TCO) and improving resource efficiency. For those evaluating on-premise deployment, AI-RADAR offers analytical Frameworks on /llm-onpremise to assess trade-offs between performance, costs, and data sovereignty requirements.

Future Perspectives and the Challenge of Variability

The variability of results across different languages and eye-tracking measures underscores the need for further research. Understanding the reasons for these differences could lead to more robust and generalizable models, capable of better adapting to linguistic and cognitive nuances. This study paves the way for deeper investigations into the intersection of artificial intelligence and cognitive sciences, exploring not only "what" LLMs learn, but also "how" they do so, and whether this "how" resembles the way humans process information.

Ultimately, the ability to "read" human cognitive signals in LLM representations not only enriches our understanding of these complex models but also offers tools to design AI systems more aligned with human expectations and processes. For companies seeking to implement advanced AI solutions, this research highlights the importance of a detailed analysis of models' internal capabilities, a key factor for effective and strategically advantageous deployment, whether on-premise or in hybrid environments.