AI Struggles to Reliably Analyze Complex Financial Documents
-
AI models like ChatGPT struggled to accurately analyze SEC filings, only succeeding about 30% of the time without exact guidance of where to look.
-
Financial companies hoped AI could automate analyzing earnings reports and filings but may need to wait as models still struggle with complex math and research.
-
Tests by startup Patronus AI found OpenAI, Meta, and Anthropic models failed to respond or provided incorrect "hallucinated" answers 70% of the time.
-
Models performed better when provided full documents for context but still struggled with lengthy, complex filings full of financial jargon and tables.
-
Accurately automating analysis of financial docs with AI remains difficult as models cannot be fully trusted to make reliable judgments like humans.