This is not the usual benchmark. LongNovel starts from a precise observation: as context grows longer, the probability of hallucinations in a summary does not just increase linearly; it changes in nature. A long novel is not a collection of isolated facts like a news article: it is a network of events, dialogues and details that reference each other across distance. That is where Large Language Models show their most subtle limits, the kind that do not emerge from a quick read.

The benchmark, available on GitHub, uses 29 Chinese novels with lengths between 16k and 100k tokens and chapter-level data from BookSum. It is bilingual Chinese-English and covers eight hallucination types. To build the dataset, the team combined arbitration across multiple models with entity-referenced hallucination generation: two choices aimed at avoiding the classic risk of any benchmark, namely fake data that measures nothing. The test set was manually revised.

There is a point here that goes beyond academic research. Teams evaluating models for summarizing long documents on self-hosted stacks know that the critical parameter is not only the context window declared by the vendor, but the fidelity of the output when the document is complex. Summarizing a news item is one thing; asking an LLM to synthesize a legal file, a clinical record or technical documentation spread over hundreds of pages is another. In these scenarios, a hallucination is not a stylistic defect: it can change a decision. The cost of human verification becomes part of the real TCO, often higher than the savings obtained with a generic cloud service.

LongNovel does not measure inference speed or the tokens per second a system can produce. It is not a hardware benchmark. But its value for on-premises deployments lies exactly here: it separates evaluation quality from the race for specs. A local cluster can have GPUs with plenty of VRAM and support very long contexts; if the model attributes an event to the wrong character, raw computing power does not help. The problem is not throughput, it is control.

The multi-scale design also challenges the most common metrics. Many summarization evaluations use automatic similarity scores that reward fluent text but do not check coherence with the entities of the source document. In LongNovel, the reference to entities during hallucination generation shifts attention to a finer level: not 'the summary is readable', but 'the summary is faithful'. For teams that need to document compliance or quality before putting a system into production, this is an operational criterion, not an academic curiosity.

Obvious limits remain: the benchmark covers novels, not contracts or corporate reports; the Chinese base and BookSum data say nothing about behavior in other languages. But the project signals a precise direction: hallucination evaluation is becoming a data engineering problem, with curated test sets and explicit categories, rather than a generic percentage reported in a technical sheet. For teams evaluating these trade-offs, AI-RADAR collects analytical frameworks on /llm-onpremise, not to prescribe a choice but to make constraints explicit.