Financial answers you can check.
FinRAG MCP gives Claude, Cursor, or any MCP client exact reported figures for any US-listed company and cited passages from SEC filings, plus a tool that checks every figure in the answer. Add it to Claude, no sign-in.
Stated figures grounded, FinanceBench 150Q. Answers written mainly by the gpt-4o-mini fallback; to be re-measured.
SEC filings ingested
Companies
Chunks embedded
Ask an LLM about revenue and you get a confident guess.
Every public company files the real numbers with the SEC every quarter. A plain chat model rarely has them, and rarely says so.
Stale training data
A model's knowledge cutoff is months old. It has no idea what a company reported last quarter.
Invented numbers
Ask for a specific figure and you get one, stated with full confidence and sourced from nothing.
No citation
Even a correct-sounding answer gives you nothing to check it against.
One question, every step shown.
- 01
Claude picks a tool
Your AI assistant decides which of the six tools fits your question.
- 02
Limits and cache
A shared limit keeps the server fair for everyone. If the same call was made in the last day, the saved result comes back right away.
- 03
Ticker and year filters
The server works out which company and which year you mean.
- 04
GAAP expansion
Your wording is translated into the words a filing actually uses.
- 05
BM25 and Pinecone
Two searches run at once, one by exact words and one by meaning, so a miss in one can be caught by the other.
- 06
Full text and rerank
Each hit is replaced by its complete passage, then the 5 best are kept.
- 07
Evidence returned
You get the source passages, each labeled with the filing it came from.
- 08
Claude writes the answer
Your own AI assistant writes the answer, using only what the passages say.
- 09
verify_figures
The dollar amounts and percentages in the answer are checked against the sources. Anything missing is flagged.
- 10
Log
A record of each call is kept for up to 90 days in case something needs checking.
The part that says no.
Edit the number below. If it doesn't match what's in the source chunk, the verifier flags it, live, in your browser. This is a TypeScript port of the production verifier's matching rule.
Retrieved source chunk
From 3M's real 10-K, as extracted by the pipeline. Table cells only, cleaned up for reading here; the verifier below runs against the unmodified chunk.
| Years ended December 31 | |||
| (Millions) | 2018 | 2017 | 2016 |
| Major GAAP Cash Flow Categories | |||
| Net cash provided by operating activities | $6439 | $6240 | $6662 |
| Net cash provided by (used in) investing activities | 222 | (3,086) | (1,403) |
| Net cash used in financing activities | (6,701) | (2,655) | (4,626) |
| Free Cash Flow (non-GAAP measure) | |||
| Net cash provided by operating activities | $6439 | $6240 | $6662 |
| Purchases of property, plant and equipment (PP&E) | (1,577) | (1,373) | (1,420) |
| Free cash flow | $4862 | $4867 | $5242 |
| Net income attributable to 3M | $5349 | $4858 | $5050 |
| Free cash flow conversion |
Generated answer (editable)
3M's capital expenditure in fiscal year 2018 was $ million.
The bug this exists to prevent
An early version of this verifier used substring matching: it checked whether "$1,234" appeared anywhere in the source text. That passed a fabricated $1,234 whenever the source actually said $1,234.56, since the shorter string is a literal substring of the longer one. The fix compares parsed numeric magnitude instead of text, which is what you're running above.
Six tools. Figures and evidence in, your model writes.
Requests and responses below are real calls to the live server, with passage text shortened. Figures work for any US-listed company; passage search covers 71.
search_sec_filings
Free-text search over every filing. Optional ticker and fiscal year filters.
{
"question": "3M capital expenditure",
"ticker": "MMM",
"fiscal_year": 2018
}{
"passages": [
{
"citation": "[MMM 10-K 2019-02-07]",
"text": "Company pension and postretirement contributions Company pension and postretirement expense Income taxes (deferred and accrued income taxes) Net cash provided by operating activities Cash fl ..."
},
"... 4 more"
]
}get_company_financials
Exact reported figures for any US-listed company, read from the XBRL data in its filings, with period, filing link and citation. Q4 is labelled when derived.
{
"ticker": "MMM",
"metrics": [
"capital expenditure"
],
"period": "FY2018"
}{
"results": [
{
"label": "Payments to Acquire Property, Plant, and Equipment",
"display": "$1,577.0 million",
"period": {
"fiscal_year": 2018,
"fiscal_period": "FY",
"start": "2018-01-01",
"end": "2018-12-31"
},
"citation": "[MMM 10-K 2019-02-07]",
"id": "xbrl:MMM:us-gaap:PaymentsToAcquirePropertyPlantAndEquipment:FY2018"
}
]
}compare_companies
The same figures for two to five companies, side by side.
{
"tickers": [
"TSLA",
"F"
],
"metrics": [
"revenue"
],
"period": "FY2024"
}{
"results": {
"TSLA": [
"$97,690.0 million [TSLA 10-K 2025-01-30]"
],
"F": [
"$184,992.0 million [F 10-K 2025-02-06]"
]
}
}get_latest_filing
Any company SEC lists, any form (10-K by default), looked up live from EDGAR, with a flag for whether it is already in the corpus.
{
"ticker": "AAPL",
"filing_type": "10-Q"
}{
"filing": {
"ticker": "AAPL",
"cik": 320193,
"form": "10-Q",
"filed": "2026-07-31",
"accession": "0000320193-26-000020",
"primary_document": "aapl-20260627.htm"
},
"in_corpus": true
}list_covered_companies
Every ticker with first and last filing date per form. Call it to see exact years.
{}{
"total_filings": 1082,
"companies": [
{
"ticker": "AAPL",
"forms": {
"10-K": {
"first": "2024-11-01",
"last": "2025-10-31",
"filings": 2
},
"10-Q": {
"first": "2025-08-01",
"last": "2026-07-31",
"filings": 4
}
}
},
"... 70 more"
]
}verify_figures
Send your draft answer and the passage and figure ids. Each dollar amount, percentage and comma-grouped number comes back grounded or not_found; against a figure, units convert.
{
"answer": "3M spent $1,577 million on capex in 2018.",
"passage_ids": [
"5d44fe78114e18919005a0d70ddd24e4",
"92b3ab0ae49e688caee769614bc6c911",
"..."
]
}{
"grounded": [
"$1,577 million"
],
"not_found": [],
"checked_against": 5
}Measured, not claimed.
End to end, the way a user meets it: a model calls the real tools in a loop, as Claude or Cursor would, and each answer is scored right, wrong or refused against FinanceBench's answer key. gpt-4o-mini plays the client.
| Agent test, 2026-10-30 | Search only | + exact figures (live) |
|---|---|---|
| FinanceBench right (of 110) | 67 | 92 |
| Wrong | 21 | 14 |
| Refused | 22 | 4 |
| Fourth-quarter set right / wrong (of 48) | 13 / 32 | 39 / 7 |
Exact figures come from the XBRL data in each filing, for any US-listed company. Searching passages, the model had to find one table row among 171,790 chunks; most of its fourth-quarter answers reported the full year, since no filing states a Q4 on its own. The figures tool derives Q4 and says how.
The one-shot test isolates retrieval: one search, five passages, one answer. Re-measured on 2026-10-30: the same 150 FinanceBench questions through the live server's retrieval, answers written by gpt-4o-mini, September answers re-scored with the same code. Automated checks, not hand-graded. “Voyage rerank” is the same server with a neural reranker over 40 candidates per search instead of a word-overlap scorer over 10.
| FinanceBench, 150 questions | September | Word-overlap rerank | Voyage rerank | + clean tables (live) |
|---|---|---|---|---|
| Refused to answer | 121 | 99 | 55 | 61 |
| Right number (of 110) | 16 | 26 | 54 | 53 |
| Wrong number (of 110) | 4 | 9 | 14 | 9 |
| Evidence passage in the top 5 (of 127) | not measured | 13.4% | 45.7% | 46.5% |
| Every figure found in its sources | not measured | 96.7% | 84% | 85.3% |
Clean tables raised refusals from 55 to 61 and cut wrong answers from 14 to 9: 17 of the 18 answers that became refusals had been wrong, mostly a full-year figure reported as a quarter. A wrong figure is worse than a refusal, so refusals alone are the wrong scoreboard. The figure check is lower than before because more answers compute ratios or convert units, which it does not do by design. The charts below are the September pipeline.
Where the metric lied
BM25-only scores higher here than the full pipeline, and for a while I published that as an honest loss. The metric is the problem. Numerical accuracy asks what fraction of the numbers an answer states are grounded, and an answer with no numbers in it scores a perfect 1.0, so refusing raises the score. BM25-only refused the most (133 of 150, against 121 for the other two). But that explains only part of its lead. On a rough check of whether the answer states the correct number, the three setups land within noise of each other (52, 48 and 52 of 110, a count that also included stray numbers inside refusals). So this benchmark cannot rank them either way. The lesson is to ask what a do-nothing system would score before trusting a metric.
What we can't explain yet
Answer relevancy, context precision and context recall score low (about 10 to 20%). Part of that is explained: about four in five answers were refusals, and ragas scores a refusal as zero relevancy; about 28 of 150 questions need years the corpus doesn't hold; and in a rough check the right chunk was in the top five only about half the time. Whether the metrics also fit this task badly (terse numeric answers judged against long table chunks) is untested.
Cost and latency.
Measured against the live server on 2026-10-30, 10 searches. A warm search takes at the median, and the slowest of the 10 took . One search costs about , an estimate from Lambda time plus one embedding, not a bill. There is no chat model cost, because the server runs none.
The cache race: 2.6x faster on a repeat call
The first call after a deploy is slow: server-side, because the Lambda loads the 964 MB keyword index. A health check every 5 minutes keeps one instance warm; a second instance, started when calls overlap, pays the same cost once.
Bugs that looked like answers.
Financial RAG fails quietly. A wrong number looks exactly like a right one until you check it. These are real bugs from building this system, not hypotheticals.
Read the full write-up on these bugsDense results silently discarded
- Symptom
- faithfulness scored 0.96 but every context metric scored ~0.0.
- Root cause
- merge_and_dedup concatenated BM25-then-dense and rerank truncated with chunks[:top_k], so the top-5 was always 100% BM25, and every dense result was discarded before reranking ever saw it.
- Fix
- Hybrid search now actually merges both sources before the top-k cut.
The substring verifier
- Symptom
- Fabricated figures passed the grounding check undetected.
- Root cause
- "$1,234" is a literal substring of "$1,234.56", so a naive text-containment check treats a fabricated $1,234 as grounded whenever the source happens to contain a longer number that starts the same way.
- Fix
- Compare parsed numeric magnitude instead of raw text. See the verifier demo above.
uuid4 chunk IDs
- Symptom
- Re-ingesting a filing duplicated every one of its chunks.
- Root cause
- chunk_id was uuid4(), so every re-ingest generated fresh IDs. BM25 and Pinecone IDs never matched, so cross-source dedup silently couldn't work at all.
- Fix
- Chunk IDs are now content-addressed (sha256), so the same chunk always gets the same ID.
The corpus held the wrong years
- Symptom
- 147 of 150 FinanceBench questions were unanswerable.
- Root cause
- list_filings defaulted to "last 2 10-Ks and 4 10-Qs", and in 2026 that means 2025-2026 filings, while FinanceBench asks about 2015-2024. The corpus had the right companies, just the wrong years.
- Fix
- Benchmark companies now ingest {10-K: 8, 10-Q: 12, 8-K: 8}, 28 filings each instead of 6.
Unpinned temperature
- Symptom
- The identical question, through the identical fixed pipeline, was correct once and wrong once.
- Root cause
- A coin flip at each API's default temperature (1.0). The same query could return a correctly cited figure or "the context does not provide this figure" on two separate runs.
- Fix
- All four LLM call sites (Bedrock + OpenAI fallback, rewriter + generator) now pin temperature=0. All four have since been removed from the server.
The XBRL prior-year-comparative trap
- Symptom
- NVDA revenue was reported as $26,044M instead of the correct $44,062M.
- Root cause
- A 10-Q's XBRL tags the prior-year comparative figure under the same fy/fp as the current quarter, so a naive lookup grabbed last year's number for AAPL, NVDA, META, and GOOGL alike.
- Fix
- Ground-truth extraction now takes the maximum end date within the filing, not the first match.
The model on the label wasn't the model writing answers
- Symptom
- The site credited the 91.0% FinanceBench score to Claude Sonnet.
- Root cause
- The account's Bedrock quota was 0. Over 100 days CloudWatch shows 241 successful Bedrock calls against about 20,000 throttles, so the gpt-4o-mini fallback wrote most answers. The answer files never recorded which model wrote them.
- Fix
- Evals now play the client with a named model and record it on every answer. The 91.0% figure stays on this site with that caveat until it is re-measured.
Vector hits arrived cut to 1,000 characters
- Symptom
- Figures that sat late in a chunk never reached the answer when a hit came from Pinecone alone.
- Root cause
- Pinecone metadata stored only the first 1,000 characters of each chunk, but 75% of chunks are longer than that (the average chunk is 3,728 characters).
- Fix
- Each candidate is swapped for its full stored text before reranking. On the same index, evidence in the top 5 rose from 10.2% to 13.4% and the number in the top 5 from 47.8% to 52.2%.
And everything underneath.
Add it to Claude. No sign-in.
The server is public. Streamable HTTP, one URL.
Claude (claude.ai and Claude Desktop)
Customize > Connectors > + Add > Add custom connector. Enter a name, paste the URL, and continue. Choose No sign in if asked. On Team and Enterprise plans an Owner adds it first under Organization settings > Connectors.
https://ecvsxkeqdpyj5hkal7wplm2b4q0gacfh.lambda-url.us-east-1.on.aws/mcp
Claude Code
One command in your terminal.
claude mcp add --transport http finrag https://ecvsxkeqdpyj5hkal7wplm2b4q0gacfh.lambda-url.us-east-1.on.aws/mcp
Cursor and other MCP clients
In Cursor, add this to .cursor/mcp.json in your project or ~/.cursor/mcp.json. Other clients that speak streamable HTTP take the same URL, though their config keys may differ.
{ "mcpServers": { "finrag": { "url": "https://ecvsxkeqdpyj5hkal7wplm2b4q0gacfh.lambda-url.us-east-1.on.aws/mcp" } } }
Deploy your own.
Want your own limits and your own corpus? These are the exact steps. You need an AWS account with the CLI configured, Node 20 or newer, Python 3.12 or newer, an OpenAI API key, a Pinecone account and a Voyage AI key with a payment method on file. Everything deploys to us-east-1.
- 01
Create the Pinecone index
Serverless index named finrag-filings, 1536 dimensions, cosine metric (it must match OpenAI text-embedding-3-small)
- 02
Clone and install
git clone https://github.com/karthikreddyyalala/FinRAG.git && cd FinRAG python3 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt
- 03
Rename the S3 buckets (names are global), then set SEC_USER_AGENT in pipeline/edgar_client.py to your name and email
perl -pi -e 's/finrag-(raw|processed)-filings/yourname-$1-filings/g' $(git grep -lE 'finrag-(raw|processed)-filings' -- '*.py')
- 04
Store your keys in SSM
aws ssm put-parameter --region us-east-1 --name /finrag/openai-api-key --type SecureString --value <your-openai-key> aws ssm put-parameter --region us-east-1 --name /finrag/pinecone-api-key --type SecureString --value <your-pinecone-key>
- 05
Deploy the CDK stacks (alert_email is optional: alarm emails and a $10 monthly budget alert)
npx aws-cdk bootstrap aws://<account-id>/us-east-1 npx aws-cdk deploy --all --require-approval never -c alert_email=you@example.com
- 06
Build the corpus (takes hours, a few dollars of OpenAI embeddings), then seed the weekly refresh
export OPENAI_API_KEY=<your-openai-key> PINECONE_API_KEY=<your-pinecone-key> PYTHONPATH=. python scripts/bootstrap_corpus.py aws s3 sync evals/results/chunk_cache s3://yourname-processed-filings/chunk_cache/
- 07
Connect with the McpEndpoint stack output
claude mcp add --transport http finrag <McpEndpoint, already ending in /mcp>