Multiple Updates and Bias in Language Models

Large language models (LLMs) are increasingly used in knowledge-intensive tasks. In these scenarios, it is common for information to be updated multiple times within the context. A new study focuses on how LLMs handle these multiple updates, where different historically valid versions of a fact compete during the retrieval process.

The DKI Framework for Evaluation

The researchers introduced an evaluation framework called Dynamic Knowledge Instance (DKI). This framework models multiple updates of the same fact as a cue associated with a sequence of updated values. Models are evaluated by probing the earliest (initial) and latest (current) states.

Results and Analysis

The results show that retrieval bias increases with the number of updates. Accuracy in the earliest state remains high, while accuracy in the latest state decreases significantly. Diagnostic analyses of attention, hidden-state similarity, and output logits reveal that these signals become less discriminative on errors, providing an unstable basis for identifying the latest update. Heuristic interventions inspired by cognitive psychology produced only modest improvements.

Implications

The study highlights a persistent challenge for LLMs: tracking and following knowledge updates in long contexts. This has important implications for the reliability of models in applications where knowledge is constantly evolving.