depiedencap-rag/docs/rag/REFERENCES.en.md

References — proven RAG techniques (reading notes)

📘 Cette page est également disponible en français 🇫🇷

Goal: what industry does, and what applies to our case (French-language forum, ~425k posts, megathreads, local 30B LLM).

1. Contextual Retrieval — Anthropic (Sept. 2024)

Source: anthropic.com/engineering/contextual-retrieval · cookbook: claude-cookbooks/capabilities/contextual-embeddings

• Problem: a chunk taken out of its document loses context → the retriever cannot judge relevance.
• Method: before embedding each chunk, an LLM generates 1-2 sentences that situate it ("this chunk comes from X and discusses Y"), prefixed to the text. Same for the BM25 index.
• Reported results: −35 % top-20 retrieval failures (contextualised embeddings alone) → −49 % with contextual BM25 → −67 % adding a cross-encoder reranker.
• Cost: ~$1.02 / Mtok of generated context (with prompt caching).

Here: the LLM version (C4) costs 1 call × ~350k posts = expensive for a lab. The free version (C1-C3: title + OP + parent) captures the same idea — situate the chunk. Still to measure whether the LLM blurb adds anything beyond that.

2. "Retrieve small, synthesize big" — sentence-window / parent-document

Sources: LlamaIndex SentenceWindowRetriever · LangChain ParentDocumentRetriever · small-to-big survey

• Principle: decouple the retrieval unit from the generation unit. Embed small (sentence/message → precision); on a match, give the LLM something larger (neighbourhood window / parent).
• Sentence-window: embed 1 sentence, read sentence ± N neighbours (window_size=3 by default).
• Parent-document: embed child chunks, return the parent (document/section) via a parent_id stored as metadata.
• Auto-merging: hierarchical variant — promote to the parent when a ratio of siblings is reached.

Here: our natural post→topic structure is exactly parent/child. Retrieve at post (small), evidence = post + ±N neighbourhood + title/OP (large). That is our config D.

3. Hybrid search + Reciprocal Rank Fusion (RRF)

Sources: Cormack, Clarke & Buettcher, SIGIR 2009 ("RRF outperforms Condorcet and individual rank learning methods") · Elastic/etc. guides.

• Why hybrid: BM25 excels on exact match (proper names, brands, rare terms); dense excels on paraphrase/concept. Neither wins everywhere — WANDS benchmark: hybrid +7.4 % NDCG vs the best alone.
• RRF: score(d) = Σ 1/(k + rank_i(d)) over each list, k=60 standard. Ignores raw scores (unbounded BM25 vs incommensurable cosine [-1,1]) — fuses by rank, not by score.
• No tuning needed to beat a weighted average.

Here: current fusion is a custom score (base rank + bonuses). RRF is the standard — a config to A/B. Discourse Search plays the BM25 role (Postgres full-text).

4. Cross-encoder reranking

• A bi-encoder (bge-m3) compares q and d separately (fast, indexable).
• A cross-encoder takes (query, doc) together → fine relevance score, but expensive → apply it only to the fused top ~50-150.
• Models: bge-reranker-v2-m3 (BGE family, multilingual, ~2 GB), Cohere Rerank (API). At Anthropic: rerank moves from −49 % to −67 %.

Here: optional step E. A bge-reranker-v2-m3 runs on mini-PC or laptop CPU/iGPU. Test only after D — it is a multiplier, not the foundation.

5. Evaluation — RAGAS and per-layer metrics

Source: RAGAS (EACL 2024 demo) — ragas.io / explodinggradients

The structuring principle: separate retrieval metrics from generation metrics — a faithful answer on the wrong context is still a failure.

Metric	Layer	Measures
Context recall	retrieval	are the expected docs in the top-k?
Context precision	retrieval	are the retrieved docs relevant?
Faithfulness	generation	are the answer's claims supported by the context?
Answer relevance	generation	does the answer address the question?

• LLM-as-judge is standard for faithfulness/relevance; context recall can be non-LLM if we have golden docs.
• Typical eval-set size: 50-200 Q for ongoing tracking; 15-20 well-chosen questions are enough to iterate fast.

Here: custom set in EVAL.en.md — goldens = known forum threads/messages. Faithfulness is partly guaranteed by the sanitizer (URL whitelist); what we mainly measure is context recall (does the right thread/message come back?) and citation relevance. Hit@5 / MRR are defined in EVAL.en.md §2.

6. What we already do — and the literature backs

Homegrown practice	Literature status
Server-side retrieval, LLM tools []	Equivalent to "evidence injection"; limits tool-call failures
[[n]] → server URL whitelist	Anti-hallucination: the LLM never generates an identifier
HyDE off	Defensible: HyDE costs 1 LLM call and can drift; re-evaluate later if needed
Abstain if 0 evidence	Standard — the bot must not answer without context
temp 0	Standard for fidelity

7. Known techniques, not taken as default

Technique	Why not now
HyDE / multi-query expansion	LLM cost per query; re-evaluate if recall is insufficient
Matryoshka (reduced dims)	bge-m3 is not Matryoshka — fixed 1024 dims
Semantic chunking (split by meaning)	Our "chunk" = the post, a natural boundary
GraphRAG / knowledge graph	Oversized for a forum
Embedding fine-tuning	Possible someday (MTEB/custom), not before the baseline

Links

• Anthropic Contextual Retrieval: https://www.anthropic.com/engineering/contextual-retrieval
• Cormack et al. 2009 (RRF): SIGIR — "Reciprocal Rank Fusion outperforms Condorcet…"
• RAGAS: https://arxiv.org/abs/2309.15217
• LlamaIndex sentence-window: docs.llamaindex.ai (SentenceWindowRetriever pack)