In brief
Key figure
Runs
Abstract
FinanceBench is a first-of-its-kind test suite for evaluating the performance of LLMs on open book financial question answering (QA). It comprises 10,231 questions about publicly traded companies, with corresponding answers and evidence strings. The questions in FinanceBench are ecologically valid and cover a diverse set of scenarios. They are intended to be clear-cut and straightforward to answer to serve as a minimum performance standard. We test 16 state of the art model configurations (including GPT-4-Turbo, Llama2 and Claude2, with vector stores and long context prompts) on a sample of 150 cases from FinanceBench, and manually review their answers (n=2,400). The cases are available open-source. We show that existing LLMs have clear limitations for financial QA. Notably, GPT-4-Turbo used with a retrieval system incorrectly answered or refused to answer 81% of questions. While augmentation techniques such as using longer context window to feed in relevant evidence improve performance, they are unrealistic for enterprise settings due to increased latency and cannot support larger financial documents. We find that all models examined exhibit weaknesses, such as hallucinations, that limit their suitability for use by enterprises.
Using FinanceBench
FinanceBench tests whether a model can answer an analyst’s questions about a public company from its filings.
The full set has 10,231 questions about 40 US public companies. Each question comes with an answer and the evidence behind it, taken from 10-Ks, 10-Qs, 8-Ks and earnings reports. Annotators with finance backgrounds, many of them analysts, wrote the questions, and a senior analyst reviewed their work.
Teams that build and train models use it to measure how well a model answers financial questions.
An open sample of 150 questions, with the source filings, is free on GitHub and Hugging Face for noncommercial use. The full set is available to license.
We partner with finance experts to train world-class financial AI. See how we build finance tasks
Questions about FinanceBench
What is FinanceBench?
FinanceBench is a benchmark for financial question answering. It has 10,231 questions about 40 US public companies, each with an answer and the evidence from the company’s filings. Patronus AI published it in 2023.
How do I get the full FinanceBench dataset?
An open sample of 150 questions is free on GitHub and Hugging Face for noncommercial use. The full set is available to license. Contact us to license it or to talk about custom finance work.
What does FinanceBench test in a finance LLM?
It tests whether a model can find the right figure in a long filing and reason from it to a correct answer. In the paper, a leading model with retrieval got 81% of the 150 sample questions wrong or refused to answer them.
