EduResearchBench: Fine-Grained Evaluation of LLMs in Academic Research

A new benchmark, called EduResearchBench, has been developed to more accurately evaluate the capabilities of large language models (LLMs) in academic writing. This tool arises from the need to overcome the limitations of current benchmarks, which often focus on monolithic evaluations and do not offer a detailed view of model performance in complex research contexts.

EduResearchBench is based on a Hierarchical Atomic Task Decomposition (HATD) framework, which divides a complete research workflow into six specialized modules. These modules cover various areas, including quantitative analysis, qualitative research, and policy research. In total, the framework defines 24 atomic tasks, allowing for automated and fine-grained evaluation of model capabilities.

A key aspect of EduResearchBench is its ability to provide detailed diagnostic feedback on specific model deficiencies. This approach contrasts with holistic evaluation systems, where aggregate scores can mask specific weaknesses. Furthermore, the benchmark includes a curriculum learning strategy that aims to progressively develop model skills, starting from basic skills to methodological reasoning and complex argumentation.

To train a specialized model for academic writing, EduWrite (30B) was created, using 11,000 high-quality instruction pairs derived from 55,000 raw academic samples. Experimental results show that EduWrite significantly outperforms larger general-purpose models (72B) on several key metrics, highlighting the importance of data quality and a hierarchical training approach in vertical domains.