ClawIndex wins all five datasets on NDCG@10 across 700 queries

We ran Reasoning-Native Retrieval against FAISS and BM25 at real scale - 500 HotpotQA questions plus 200 BEIR queries across two domains. ClawIndex wins all five datasets on NDCG@10. No vector database. No embeddings. No hallucinations.

Why this benchmark matters

Our first published result was 20 questions. Useful for a proof of concept. Not useful for anyone deciding whether to build on top of a retrieval system. Real benchmarks have three properties: enough volume that noise averages out, multiple query types that stress different failure modes, and standardized datasets that researchers can independently replicate.

This benchmark has all three. 500 HotpotQA questions, split across bridge-style multi-hop and comparison queries, plus 100 BEIR ArguAna (argument retrieval) and 100 BEIR SciDocs (scientific literature). Seven hundred total queries. Three systems. Five datasets.

Results

HotpotQA - All types (n=500)

HotpotQA - Bridge only (n=404)

HotpotQA - Comparison only (n=96)

BEIR ArguAna (n=100) · BEIR SciDocs (n=100)

Where ClawIndex wins, and why

The biggest gaps show up exactly where you'd expect a reasoning system to pull ahead: bridge-style multi-hop questions, where the retrieval path isn't obvious from the surface form of the query. On bridge questions (n=404), ClawIndex scores NDCG@10 of 0.920 vs FAISS at 0.837, a +9.9% lead. BM25 lands at 0.812.

Comparison questions tell a different story. All three systems hit 100% Hit@5, every system found at least one relevant document in the top 5. But ClawIndex's NDCG of 0.992 pulls ahead on ranking quality: when it retrieves both documents needed, it puts them near the top. BM25 at 0.800 on the same queries loses on ordering.

The BEIR datasets confirm the pattern holds outside HotpotQA's Wikipedia domain. On ArguAna (argument retrieval), ClawIndex scores 0.991 NDCG@10 vs FAISS at 0.921. On SciDocs, the margin narrows to 0.975 vs 0.972 - statistically marginal at n=100 without confidence intervals, and we're not overclaiming it. FAISS edges ClawIndex on MRR@10 in both BEIR datasets. The NDCG@10 lead is the defensible headline; ranking order within the top 10 is where we outperform.

Compared to published research

PRISM is the most recent published multi-hop retrieval system we're aware of (October 2025). ClawIndex scores +2.7% above it on NDCG@10. One important caveat: HotpotQA has two corpus settings, distractor (10-paragraph context) and fullwiki (full Wikipedia). These produce wildly different absolute numbers. Our benchmark uses the distractor setting. If PRISM ran fullwiki, a direct NDCG comparison is apples-to-oranges. We're working to verify which setting PRISM used; until then, treat the +2.7% as directional, not definitive.

Provenance validation

ClawIndex uses pointer validation: every retrieved passage is traced back to its source index entry before being returned. If the pointer doesn't resolve, the passage is dropped and the query falls back to a broader search. This is an architectural property of ClawIndex, not a comparison against FAISS or BM25. Pure retrievers returning real indexed documents can't hallucinate by construction. That's a different failure mode than what we're solving for.

What pointer validation gives you: a provenance guarantee. Every result ClawIndex returns is traceable to a source. 51 of the 500 HotpotQA queries triggered a fallback; all 51 still returned valid, traced results. For legal, medical, compliance, or financial use cases where you need to know exactly where an answer came from, that guarantee matters. It's a feature, not a benchmark line.

The latency and cost tradeoff

ClawIndex averages 27 seconds per query. FAISS runs in 7 milliseconds. That's not a small difference and we're not going to paper over it.

The reasoning pass is LLM calls, which means there's a cost-per-query number alongside the latency number. At current API rates, each ClawIndex query costs roughly $0.01–0.04 depending on document size and model tier. FAISS and BM25 are effectively free per query once indexed. That's a real tradeoff for high-volume workloads.

The question is what you're building. Real-time search autocomplete or millions of queries per day: FAISS. Async document processing, batch analysis, compliance workflows, contract review, research pipelines, or any agent that can afford to wait 30 seconds and spend a few cents for an answer with a provenance guarantee: ClawIndex. The use cases don't overlap as much as the benchmark table makes them look.

What's next

Two things on the roadmap. First: latency. The 27-second average is driven by the reasoning pass. We're exploring parallel index traversal and caching strategies to get that under 5 seconds without sacrificing quality. Second: multi-index federation. The next benchmark will test ClawIndex across multiple independent document stores queried simultaneously. That's the architecture most enterprise deployments actually need.

Results will be posted here when they're ready. No timeline promises. When the numbers are good, we'll publish.