ClawIndex wins NDCG@10 on all five datasets - 700 queries, zero hallucinations

We benchmarked Reasoning-Native Retrieval at scale: 500 HotpotQA questions and 200 BEIR queries across five datasets. ClawIndex wins NDCG@10 on all five. Every result is pointer-validated back to its source. No vector database. No embeddings.

Why NDCG@10

NDCG@10 (Normalized Discounted Cumulative Gain at rank 10) measures both whether the right documents are retrieved and where they land in the ranked list. A system that retrieves the right answer at position 10 scores lower than one that puts it at position 1. It is the standard metric for retrieval quality in information retrieval research, used by BEIR, MS MARCO, and most published benchmarks. We lead with it because it is the hardest metric to game - you cannot pad your way to a good NDCG score.

For completeness: FAISS edges ClawIndex on MRR@10 in the two BEIR datasets, and all three systems tie on Hit@5 in the comparison subset. Those numbers are in the tables below. We are not hiding them.

Results

HotpotQA - All types (n=500)

HotpotQA - Bridge reasoning only (n=404)

HotpotQA - Comparison only (n=96)

BEIR ArguAna (n=100) and SciDocs (n=100)

The SciDocs NDCG margin (0.975 vs 0.972) is small at n=100 without confidence intervals. We are not treating it as a definitive win - it is directional. FAISS leads on MRR@10 across both BEIR datasets. The tables are complete; read them in full.

Where the gap is largest

Bridge-style multi-hop questions are where the reasoning architecture earns its keep. These are questions where the answer requires connecting two documents that don't obviously belong together from keyword overlap alone. On 404 bridge questions, ClawIndex scores NDCG@10 of 0.920 vs FAISS at 0.837 - a 9.9 point gap. BM25 lands at 0.812.

Comparison questions (n=96) tell a cleaner story: all three systems find the right documents (100% Hit@5), but ranking quality separates them. ClawIndex at 0.992 NDCG@10 vs BM25 at 0.800 is a 19 point gap driven entirely by how well each system orders what it retrieves.

Compared to published systems

One caveat on the PRISM comparison: HotpotQA has two corpus settings - distractor (10-paragraph context window) and fullwiki (full Wikipedia). These produce very different absolute NDCG numbers. Our benchmark uses distractor. We have not confirmed which setting PRISM used. If they ran fullwiki, a direct comparison is not valid. We are treating the +2.7 point gap as directional until we can verify corpus settings.

Provenance validation

ClawIndex uses pointer validation: every retrieved passage is traced back to its source index entry before being returned. If the pointer does not resolve, the passage is dropped and the query falls back to a broader search pass. 51 of the 500 HotpotQA queries triggered a fallback; all 51 returned valid, traced results.

This is not a comparison against FAISS or BM25 - pure vector and keyword retrievers return real indexed documents and cannot hallucinate by construction. Pointer validation is an architectural guarantee for a different use case: when you need to know exactly where an answer came from, not just that it came from somewhere in the index. For legal, compliance, medical, or financial workflows where provenance is auditable, that guarantee is the product.

The cost and latency tradeoff

ClawIndex averages 27 seconds per query. FAISS runs in 7 milliseconds. The reasoning pass is LLM calls, which means there is a dollar cost per query: roughly $0.01 to $0.04 per query depending on document size and model tier. FAISS and BM25 are effectively free per query once indexed.

For real-time search, high-volume pipelines, or cost-sensitive applications: FAISS. For async document processing, contract review, compliance workflows, research pipelines, or anything where a traceable answer matters more than a fast one: ClawIndex. The use cases are different enough that "which is better" is the wrong question.

What comes next

Two things on the roadmap. First, latency: the 27-second average is driven by sequential LLM calls in the reasoning pass. We are exploring parallel index traversal and result caching to bring that under 5 seconds without touching retrieval quality. Second, multi-index federation: the next benchmark will test ClawIndex querying across multiple independent document stores simultaneously, which is the architecture most production deployments actually need.

Results go here when they are ready.