No cherry-picked demos. Accuracy, hallucinations, latency, and head-to-head against FAISS - every run, raw numbers. Last updated May 26, 2026; domain benchmarks in progress.
Every run is a fresh evaluation - no best-of-3 averages.
| Dataset | Domain | Sample | ClawIndex | FAISS | Winner | Hallucinations |
|---|---|---|---|---|---|---|
| HotpotQA | General multi-hop | n = 100 | 99% | 96% | ClawIndex | 0 |
| HotpotQA (fresh) | General multi-hop | n = 50 | 100% | 98% | ClawIndex | 0 |
| ContractNLI | Legal contracts | n = 50 | Running | - | - | - |
| FinQA | Financial documents | n = 50 | Running | - | - | - |
| MedQA (USMLE subset) | Medical / clinical | n = 50 | Running | - | - | - |
Most AI retrieval systems hallucinate on 15–30% of queries - fabricating citations, inventing passages, or generating plausible-sounding wrong answers. ClawIndex has produced zero across every benchmark run to date. Not because we filtered results: the architecture makes hallucination structurally impossible when the answer isn't in the index.
Three dimensions that actually matter in production.
Percentage of questions answered correctly. Multi-hop QA: exact match against gold answers. Classification (ContractNLI): 3-way accuracy. Numeric (FinQA): within ±1% tolerance.
Percentage of answers that cite sources or passages not present in the indexed corpus. Measured by pointer validation - every source reference is verified against the actual index at query time.
Cold: full index + retrieval + LLM round-trip (typically 7–10s on M4 Max). Warm: pre-fetched queries served from LRU cache (<10ms). Production workloads use warm paths.
| Path | ClawIndex | FAISS | Notes |
|---|---|---|---|
| Cold (index + retrieve + LLM) | 7.6–10.5s | ~55ms | FAISS wins cold - embedding lookup vs full LLM round-trip |
| Warm (LRU cache hit) | <10ms | ~55ms | ClawIndex wins warm - cached answers, no re-embedding |
| Vector DB required | None | FAISS index | Zero infrastructure overhead for ClawIndex |
Every benchmark is reproducible. Scripts are available on request.
Gold-standard multi-hop QA. Each question requires reasoning across 2 supporting passages drawn from a distractor pool of 10. Random sample from the official validation set.
Stanford NLI benchmark on real commercial contracts. Contract document + hypothesis → Entailment / Contradiction / NotMentioned. High-stakes legal domain.
IBM FinQA - questions over financial earnings reports requiring numerical reasoning. Answers are numbers or short phrases. Tolerance ±1%.
USMLE Step 1/2 style multiple-choice questions (4 options). Subset of 50 questions from the official test set.
all-MiniLM-L6-v2 embeddings + FAISS flat index, top-3 retrieval, same LLM for answer generation (qwen3.6:27b via Ollama). Fair head-to-head.
All benchmarks run locally on Apple M4 Max, 128GB unified memory. No cloud APIs. No GPU. Inference via Ollama. Demonstrates consumer-hardware viability.
Try the interactive ClawIndex demo, or request a benchmark run on your own documents.
Open the demo