LLMs and the Book Summarization Challenge

The ability to summarize texts is a key function in Natural Language Processing (NLP). With the advent of Large Language Models (LLMs) and the increase in context windows, it is now possible to process entire books in a single prompt. However, for well-known works, LLMs can generate summaries based solely on the knowledge acquired during training.

Comparison between Internal Knowledge and Full Text

A recent study compared the summaries generated by LLMs using two approaches: (i) only the model's internal knowledge and (ii) the full text of the book. The results show that providing the full text generally leads to more detailed summaries. Surprisingly, for some books, summaries based on internal knowledge scored higher.

Implications for Long Text Processing

This raises questions about the actual ability of models to perform effective summaries of long texts. The information learned during the training phase can, under certain circumstances, outperform the performance obtained from direct text analysis. Further research is needed to fully understand the factors influencing the quality of summaries generated by LLMs and to improve their ability to process extended texts.