AI / ML · 02 / 03 · RAG · pgvector · LangGraph
Financial RAG QA
Question answering over 57,638 finance answers where every sentence cites its source, and every design choice was measured before it was kept.
- Python
- PostgreSQL
- pgvector
- PyTorch
- sentence-transformers
- LangGraph
- FastAPI
- Ollama
Why I built this
Companies are putting chatbots on top of their documents, and the costly failure is a confident answer that cites nothing or cites the wrong thing. In finance a made-up number isn't a typo, it's a liability.
Does the standard RAG recipe actually work, or do people just assume it does? I wanted to measure every piece against human relevance labels, keep only what helped, and make the model show where each sentence came from.
What I built
A RAG service over FiQA, a personal-finance Q&A corpus with 648 held-out questions people labelled for relevance. BM25 and dense embeddings in PostgreSQL with pgvector, fused, reranked, and handed to a local LLM that must cite a passage for every sentence.
The measurement changed the design: the off-the-shelf reranker made results worse, so I fine-tuned the embedding model on FiQA's training questions and fused the reranker's order with the retriever's instead of replacing it. A LangGraph agent adds a self-check that drops unsupported sentences.
How it works
Served through FastAPI with streaming answers and as an MCP server, so other AI tools can use it.
How it stays correct
Every retriever variant scored on 648 human-labelled questions, with paired significance tests.
The textbook reranker cut nDCG by 14%; fine-tuning the embedder and fusing ranks added +7% over dense (p < 0.001).
Citations are parsed and checked; unsupported sentences are flagged.
An LLM judge scores whether each sentence is backed by its citation: 69% with BM25, 80% final, 93% with the agent.
Measured
- Retrieval
- nDCG@10 BM25 0.253 → dense 0.406 → final 0.435 (+72% vs BM25)
- Faithfulness
- 150 questions: 69.1% BM25, 80.2% final, 93.4% agent (4× the LLM calls)
- Speed
- Retrieval p95 95 ms; first token p95 2.3 s on a local 8B model
- Tests
- 61 tests, 93% coverage, CI on every push