ReportLogic: A Benchmark for Logical Quality in LLM Reports

A new study introduces ReportLogic, a benchmark designed to evaluate the logical quality of deep research reports generated by large language models (LLMs). The research highlights how users increasingly rely on LLMs to synthesize complex information into structured reports, crucial for understanding and action.

The practical validity of these reports depends on their logical quality: claims and arguments must be explicitly supported and verifiable, not just fluent or informative. ReportLogic addresses this need through a hierarchical taxonomy that evaluates the ability to trace a coherent report structure (Macro-Logic), understand the progression with necessary context (Expositional-Logic), and verify conclusions through explicit evidence (Structural-Logic).

Evaluation and Robustness

A rubric-guided, expert-annotated dataset was created to train an open-source LogicJudge, designed for scalable evaluation. The robustness of the evaluation system was tested through adversarial attacks, revealing that standard LLM evaluators are often influenced by superficial elements such as verbosity, and that reasoning modes can mask incorrect support relations. The results provide useful guidance for developing more robust logic evaluators and improving the logical reliability of LLM-generated reports.

For those evaluating on-premise deployments, there are trade-offs to consider. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.