The problem of "context rot"

A Reddit user has raised a crucial issue in the development of conversational agents based on large language models (LLMs): the loss of context, or "context rot", in long conversations. The user, who is developing a support agent, found that GPT-4o begins to contradict itself or forget key details after about 15 turns of conversation.

Strategies and limitations

The user has tried several strategies to mitigate the problem, including using a sliding window and context summarization. However, the sliding window risks discarding important information from the beginning of the conversation, while summarization can lead to the loss of crucial nuances. Effective context management remains a significant challenge for applications that require extended and coherent conversations.

For those evaluating on-premise deployments, there are trade-offs to consider. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.