FIRE: A New Benchmark for Finance

FIRE, a comprehensive benchmark designed to evaluate the capabilities of LLMs (Large Language Models) in the financial sector, has been introduced. This tool aims to measure both the theoretical knowledge of finance and the ability to handle real-world business scenarios.

Benchmark Components

FIRE includes a set of questions drawn from recognized financial certification exams, designed to assess the understanding and application of financial knowledge by LLMs. In addition, the benchmark proposes a systematic evaluation matrix that categorizes complex financial domains, ensuring coverage of essential subdomains and business activities. 3,000 questions based on financial scenarios have been collected, including both closed-form questions and open-ended questions evaluated using predefined rubrics.

Evaluation and Results

Comprehensive evaluations of state-of-the-art LLMs have been conducted using the FIRE benchmark, including XuanYuan 4.0, a financial-domain specific model. The results obtained allow for a systematic analysis of the limitations of current LLM capabilities in financial applications. The benchmark and evaluation code have been publicly released to support future research in the field.