Zero-shot retrieval on five niche technical domains (October 2024 snapshot), ranked by average Recall@50.
| Avg. (5) | LangChain | Yolo v7 & v8 | Laravel 10 & 11 | Angular 16, 17 & 18 | Godot4 | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # | Retriever | Type | Size | Date | α@10 | C@20 | R@50 | α@10 | C@20 | R@50 | α@10 | C@20 | R@50 | α@10 | C@20 | R@50 | α@10 | C@20 | R@50 | α@10 | C@20 | R@50 |
💡 Queries in FreshStack can be pretty long, so use a model that can handle code and long queries.
Add your results to leaderboard_data.json and open a pull request.
We introduce FreshStack, a reusable framework for automatically building information retrieval (IR) evaluation benchmarks from community-asked questions and answers. FreshStack conducts the following steps: (1) automatic corpus collection from code and technical documentation, (2) nugget generation from community-asked questions and answers, and (3) nugget-level support, retrieving documents using a fusion of retrieval techniques and hybrid architectures. We use FreshStack to build five datasets on fast-growing, recent, and niche topics to ensure the tasks are sufficiently challenging. Existing retrieval models, when applied out-of-the-box, significantly underperform oracle approaches on all five topics, denoting plenty of headroom to improve IR quality.
Automatically collect a corpus from code and technical documentation.
Generate nuggets from community-asked questions and answers.
Retrieve documents with a fusion of retrieval techniques and hybrid architectures to support each nugget.
Step-by-step procedure followed in FreshStack for constructing evaluation datasets.
Each point is a model with model parameters in the leaderboard (parameters in millions or billions, parsed from values such as 7B or 434M).
The y-axis is the average across the five FreshStack domains for α@10 (alpha-nDCG@10), C@20 (Coverage@20) and R@50 (Recall@50).
Models without a disclosed size (shown as “-” in the table) are listed below the plot and appear only in the release-date plot.
Marker colors group model families (the family field in leaderboard_data.json). Models released in the last 90 days are outlined and labeled.
Each point is a model with a valid release date in the leaderboard.
The y-axis is the average across the five FreshStack domains for
α@10 (alpha-nDCG@10), C@20 (Coverage@20), and R@50 (Recall@50).
Marker colors group model families (the family field in leaderboard_data.json). Models released in the last 90 days are outlined and labeled.
@inproceedings{thakur2025freshstack,
title={FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents},
author={Nandan Thakur and Jimmy Lin and Sam Havens and Michael Carbin and Omar Khattab and Andrew Drozdov},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2025},
url={https://openreview.net/forum?id=54TTgXlS2U}
}