Financial Information Retrieval

The project tests whether dense semantic retrieval can improve financial question answering when queries and relevant passages use different vocabulary.

This project was completed as part of a four-person university group. My primary contribution focused on evaluation design, experimental methodology, BM25 parameter tuning analysis, results interpretation, and project coordination.

The team built the complete system documented here. The technical sections describe that shared work; the project information above distinguishes my individual contribution.

Three systems are evaluated on the same FiQA-2018 test queries and relevance judgements: a BM25 baseline, an optimised BM25 configuration and an SBERT dense retriever backed by exact FAISS search. Both retrieval branches return the top 100 documents per query and are evaluated independently rather than fused.

The notebook preserves the complete implementation, tuning outputs, model comparisons and an interactive search interface.

• Python
• BM25
• Sentence Transformers
• FAISS
• BEIR
• Gradio

BEIR FiQA-2018 provides conversational financial questions, answer passages and test relevance judgements.

The notebook downloads the public BEIR archive at runtime and loads its test split through GenericDataLoader. Titles and passage text are concatenated before indexing. The dataset is not duplicated in either repository.

A shared corpus feeds two independent retrieval paths: lexical matching with BM25 and semantic matching with SBERT.

Each branch produces query-to-document scores in BEIR format, retrieves its top 100 candidates and passes them into the same evaluation workflow. Keeping the branches independent makes their differences easier to interpret.

A nine-configuration grid search examined term-frequency saturation and document-length normalisation.

The baseline uses k1=0.9 and b=0.4. Tuning evaluates k1 ∈ {0.9, 1.2, 1.5} and b ∈ {0.5, 0.75, 0.9}. The best stored setting is k1=0.9, b=0.75, selected by NDCG@10. It retains the baseline term-frequency setting while applying stronger document-length normalisation.

Sentence embeddings allow the retriever to match semantic meaning even when queries and passages do not share exact terms.

The pretrained multi-qa-mpnet-base-dot-v1 model encodes documents and queries into 768-dimensional, L2-normalised vectors. Document vectors are stored in faiss.IndexFlatIP; with normalised embeddings, inner-product ranking is equivalent to cosine similarity. The index performs exact rather than approximate search.

Every run is converted to BEIR's query-to-document score format and evaluated against the FiQA test qrels with four complementary metrics.

SBERT Dense achieved the strongest stored result on every reported metric.

Final stored comparison on the BEIR FiQA-2018 test split. Higher is better for every metric.

BM25 tuning improves all four metrics, though the gains are modest. Dense retrieval substantially outperforms both lexical variants, consistent with its ability to bridge vocabulary differences between conversational questions and more specialised financial passages.

A Gradio interface turns the evaluated retrievers into an explorable financial search tool.

Users can enter a free-text query, choose between tuned BM25 and SBERT Dense, select a Top-K value from 1 to 20 and inspect ranked document IDs, scores, titles and text snippets. Example financial questions support both retrieval modes.

The temporary gradio.live URL preserved in the notebook is not presented as a permanent deployment; running the interface cell creates a new session.

• How lexical and dense retrieval solve different aspects of the relevance problem.
• Why a single metric cannot fully describe ranking quality, precision and coverage.
• How document normalisation parameters change BM25 behaviour across a real corpus.
• How embedding normalisation connects cosine similarity with FAISS inner-product search.
• Why validation splits, latency benchmarks and significance testing are important next steps.
• How to turn an evaluated notebook workflow into an interface without overstating deployment maturity.