depiedencap-rag/docs/rag/EVAL.en.md

RAG evaluation — protocol and question set

📘 Cette page est également disponible en français 🇫🇷

Status: active (13/09/2026). Set expanded to 101 questions sourced from the local DB (goldens verified as existing in the DB; relevance of canonical threads still to be validated by a forum human). Principle: measure by layer (retrieval ≠ generation), see REFERENCES.en.md §5.

Public-facing intent: this rag/ folder (protocol, timestamped results, comparison table) is a deliverable meant to be shown publicly later (Forgejo). Consequences for how we work:

• each run produces a timestamped JSON committed in results/;
• the §4 table is filled for every tested config, with qualitative notes;
• commits tell the reasoning (why a config, what the measurement showed) — not just the mechanics;
• one factor at a time (§3), so the table reads as a demonstration.

1. Eval-set structure

101 questions (initial 15–20 target exceeded — a larger corpus gives less noisy signals), each with golden documents (threads/messages that should come back). Format:

- id: calceophilie
  question: "C'est quoi la calcéophilie ?"
  expected_topic_ids: [34091]       # /t/du-terme-calceophile/34091
  expected_post_ids: []             # empty = topic grain is enough
  difficulty: facile
  tags: [définition]

- id: <ex-megathread>
  question: "<question whose answer is a precise message in a long thread>"
  expected_topics: [<topic_id>]
  expected_posts: [<post_id>]       # the precise message — the real post-grain test
  difficulty: megathread
  tags: [contexte, anaphore]

Actual coverage (101 questions, questions.yml sections A–J):

Question type	n	What it tests
Definition / manufacturing technique	7	semantic retrieval on concepts (constructions, gemming…)
Brand opinions	24	brand→canonical thread, megathreads up to 2.7k posts
Care / repair	21	fine grain (wax, glazing, fading…)
Buying / second-hand / budget	8	transactional intents
Sizing / width	7	sizing — answers scattered across posts
Shoe types	12	product taxonomy + colour megathreads
Forum life / rules	10	non-"shoes" categories, sensitive topics
Answer inside a megathread (post grain)	2	the core of the workstream — expected_post_ids
Off-topic	6	correct abstention
Anaphora / conversational context	—	out of scope of the single-turn harness — manual bot test
LLM unavailable	—	manual test (humorous message)

2. Metrics

The two README numbers (Hit@5 = 0.97, MRR = 0.89) measure retrieval only (does the right thread/post come back?), not the quality of the generated sentence. They are computed on the golden set (questions.yml): each question has one or more expected_topic_ids (and sometimes expected_post_ids).

Hit@5 (recall in the top-5)

For one question, retrieval succeeds if at least one golden document appears in the first 5 results. Hit@5 = that success averaged over the set (0 or 1 per question).

• 0.97 (config E, n=796): 97 questions out of 100 see the right thread in the top-5.
• A Hit@5 of 1.0 with the right thread always in 5th place would still be "perfect" as recall, and bad as ranking → hence MRR beside it.
• Variants in the tables: Hit@k topic (the golden thread is in the top-k); Hit@k post (the golden message is in the top-k, post grain only). The headline figure is scale Hit@5 topic (batch K). Early passes also used k ∈ {3, 6} on curated batch A–J.

MRR — Mean Reciprocal Rank (ranking of the first hit)

For each question: 1 / rank of the first relevant result (1st → 1, 2nd → 0.5, 3rd → ≈ 0.33, 5th → 0.2). If the golden is missing from the returned list: 0. MRR = mean of those scores.

• 0.89 (config E): the first relevant hit is in practice 1st or 2nd, rarely 4th/5th.
• Unlike Hit@5, a golden moving from #1 to #5 lowers MRR (1 → 0.2) even if it stays "in the top-5".

In one line: Hit@5 = did we miss it? · MRR = did we put it first? The topic → post jump (0.68 → 0.94 Hit@5) is the main gain; rerank (0.94 → 0.97 Hit@5, 0.84 → 0.89 MRR) mainly improves order.

Retrieval (automatic) — recap

Metric	Definition
Hit@5 topic	at least one expected_topic in the top-5 (README figure)
Hit@k topic	same for a given k (batches A–J: also k ∈ {3, 6})
Hit@k post	the expected_post in the top-k posts (post grain only)
MRR	mean of 1/rank of the first relevant hit (0 if missing)
Pollution	evidence slots occupied by the same topic (measures aggregation)

Generation (semi-auto)

Metric	How
Real cited links	% [[n]] → Post.exists? — should be 100 % (sanitizer)
Relevant citation	does the cited message carry the answer? (human, 0/1)
Faithfulness	claims supported by the evidence (LLM-judge or review)
Correct abstention	off-topic → abstain, no invention
CTA present	"To go further" footer with an invitation to post

3. Protocol

1. Baseline A (live config, topic grain): run the set on the local forum → fill the table. That is the bar to beat.
2. Each config (B, C, D, E…): one factor changed vs the previous config, same set, same metrics.
3. Results → table below + qualitative notes (striking examples).
4. A question that regresses = we understand it before concluding.

4. Results table (to fill)

Config	Hit@5 topic	Hit@k post	Real links	Abstention OK	Notes
A (live, topics)	39/95 (kw 17/95, sem 32/95)	n/a			13/09 v2, goldens corrected — hybrid=41%
B (native posts)	88/95	exact post #1 ×2 (carmina-sizing, glaçage-sèche-cheveux)			13/09 v2 — 425k posts embedded
C (enriched posts)	0.85 (n=796)	—			14/09 — measured, rejected: enriching the embedded text (OP + parent excerpt) degrades vs B (0.94) — dilution of post content, §4quater
D (+ neighbourhood)					generation side (LLM context), not Hit@k
E (+ rerank)	97% (n=796)	p✓ kept			13/09 — CPU cross-encoder, Hit@5=0.97/MRR=0.89 (rrf rerank×hybrid) — detail §4ter
F (RRF fusions)	87-88/95	p✓ ×2 kept at #1			13/09 v2 — posts×hybrid MRR 0.80
G (size routing)	84-88/95	p✓ ×2			13/09 — rejected: degrades vs F, RRF already routes implicitly

Size-routing findings (13/09, thresholds 25/50/100/250 posts): filtering topics > N posts out of the topic channel before fusion degrades monotonically (MRR 0.68@25 → 0.76@250 < 0.80 with no filter). Long threads almost never get a "topic" vote anyway (diluted embedding = invisible), so there is no noise to remove; cutting medium threads removes useful votes. RRF already implements the spirit of routing: a megathread only comes back via its posts, a short thread benefits from both channels. Variant documented then dropped — negative result, kept for the demonstration.

Baseline A→B findings (13/09, n=95, goldens v2): the native posts channel alone does Hit@5 0.93 / MRR 0.77 vs topic hybrid 0.41. Golden correction by pooling (13/09): the initial "failures" were bringing back valid threads missing from the expected list — widened for corthay-avis, noeuds-lacets, presentation-nouveau-membre, bienvenue-nouveau; richelieu-bout-rapporte rephrased (a member names the brand). Method lesson: the initial 0.87→0.93 gap comes from golden completeness, not retrieval — keeping goldens current is the most fragile part of the benchmark.

Fusion findings (13/09): RRF posts×{semantic, hybrid} does not gain recall (posts is enough) but improves MRR (0.77→0.80) and rescues posts failures (loding-pointure #23→#1, grande-pointure #11→#1 — sizing diluted in posts benefits from the semantic topic signal). RRF posts×keyword degrades (BM25 noise: MRR 0.74). Posts-channel pollution: top-5 = 2.8 distinct topics on average, dominant topic 63 % of slots.

Baseline A findings (12/09): the Carmina megathread (2690 posts) absent from semantic top-20 and keyword top-25 → topic-embedding dilution is real and measured. The title gate (relevance_score requires a distinctive word in the title) also rejects relevant threads whose title does not contain the keyword — keep that in mind when reading post-grain metrics.

4bis. Scale confirmation (812 questions, 13/09)

Batch A-J (101 curated questions, goldens verified by pooling) was completed by a scale-generated batch (section K of questions.yml, +701 questions + 10 off-topic, sourced by mining all categories/tags in the DB — different methodology, documented at the top of the section, golden = single source topic without multi-thread union). Goal: check that curated-batch results are not a small-sample artefact.

Channel	Hit@5 (n=796)	MRR
semantic (topic)	0.55	0.43
keyword (topic)	0.41	0.38
native posts	0.94	0.84
hybrid (then-current prod)	0.68	0.63
rrf posts×hybrid	0.95	0.88
routed_250	0.95	0.86
enriched posts (C)	0.85	0.80
rrf enriched×hybrid	0.92	0.85
rerank enriched	0.91	0.86

Confirmed: the native-posts / RRF-fusion advantage holds at scale, slightly improving (less statistical noise). Notable gap vs the curated batch: prod hybrid scores 0.68 here vs 0.41 on A-J — the generated batch contains many highly distinctive titles (named brands/products) that BM25 matches easily, while A-J deliberately tests rephrasings and harder megathreads. The two measurements are complementary: A-J documents fine cases, K measures general robustness. Size routing (rejected on A-J) is confirmed rejected here too (routed_250 < rrf_post_hyb with no filter).

4ter. Config E — cross-encoder rerank (13/09, n=796)

Model: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 (multilingual, light, CPU — loaded once via sentence-transformers, --rerank in run_eval.py). Re-sorts the posts-channel top-25 ({title}. {first 300 characters of the post} vs question) before topic dedup.

Channel	Hit@5	MRR
native posts	0.94	0.84
rrf posts×hybrid	0.95	0.88
rerank	0.97	0.88
rrf rerank×hybrid	0.97	0.89

Rerank mainly gains recall (0.94→0.97 Hit@5): it promotes relevant posts poorly ranked by cosine distance alone, already in the top-25 but not in the top-5. Net but incremental gain versus the initial topic→post jump (still the dominant factor of the workstream). eval_retrieval.rb now exposes text (title + 300c excerpt) per post so the host can rerank without an extra step inside the container.

4quater. Config C (enriched posts) — measured, rejected (14/09, n=796)

Implemented for the vitrine (comparable metrics on the whole dataset): full strategy_id=2 backfill on 425 505 posts (backfill_enriched.rb, ~3h GPU). Embedded text = native context (title+cat+tags) + 500c excerpt of the thread's first message + 400c excerpt of the parent post (reply_to_post_number), truncated at max_seq−2 like the native strategy. Hypothesis: disambiguate short replies ("same in 8.5") that the bare post does not contextualise.

Channel	Hit@5	MRR
native posts (s1)	0.94	0.84
enriched posts (s2)	0.85	0.80
rrf posts×hybrid	0.95	0.88
rrf enriched×hybrid	0.92	0.85
rerank (s1)	0.97	0.88
rerank enriched (s2)	0.91	0.86

Negative result, uniform across all variants: enrichment costs ~9 Hit@5 points and ~4 MRR points on the channel alone, and the degradation propagates to fusions and rerank. Interpretation: the OP/parent excerpt consumes token budget and dilutes the post's own content in the vector — contextual noise hurts more than disambiguation helps. Typical case: scaled-faq-les-diff-rents-montages — the golden (36778) is #1 in s1 but drops out of the top-25 in s2: the shared OP context homogenises vectors of posts in the same thread and flips toward a neighbouring thread (36856) that occupies the first 5 slots at ~equal distance.

Decision: rejected on measurement. The bare post (title+cat+tags+body) is the right granularity. Context belongs at generation time (config D — neighbourhood injected into the LLM prompt, without touching embeddings), not in the indexed vector.

Dataset note: residual failures (~3 % after rerank) remain dominated by generic titles in batch K ("Présentation": 1671 close threads in the DB, "Question": 211, "ceinture": 123) whose single-topic golden is structurally ambiguous — a measurement limit, not a retrieval one.

4quinquies. FULL eval — real generation (14/09, n=17→20)

First end-to-end test with the real prod LLM (qwen3:30b-a3b-q6k, vLLM mini PC :32134). eval_full.rb emits per question the prod evidence (instructions_block/for_question as-is) + the native-posts top-25; run_full_eval.py rebuilds the "new" block (posts+rerank, same template), calls the LLM with system = persona + block, and collects the answer + [[n]] markers for validation. Readable digest in results/*-full-*.yml.

Lessons (17 questions: 14 in-domain, 3 off-topic)

	prod (topic grain)	new (posts+rerank)
in-domain	answers 12/14; wrong abstention ×2 (montage-norvegien-def, carmina-sizing-megathread — empty evidence)	answers 14/14
off-topic	abstains ×3 cleanly	hallucinates ×3 with citations (invented recipe, "42 °C in Paris", "DPEC investment committee")
invalid citations	2 (scaled-aston-martin cites 23 with only 1 source)	0
LLM latency	~3-10 s/answer (mini PC)	same

Central finding: the prod title gate was serving as abstention by accident — off-topic → no distinctive word in titles → empty evidence → abstention. Removing it gains in-domain recall (the 2 wrong abstentions are fixed) but loses out-of-domain protection: the posts channel always returns 25 candidates and the LLM uses them, even off-brief.

CE score alone does not split cleanly (calibration 14/09): top-1 CE on 796 in-domain questions: min −4.74, p25 +0.42, 4/796 < −4; on 16 off-topic: −5.9 → +2.77. The overlap comes from Le Comptoir — the forum talks food, wine, cinema: "bœuf bourguignon" has legitimate tangential threads (off-topic cinema = +2.77). A −4 threshold cuts extreme cases with no measured in-domain false negative, but lets the grey zone through; that is as much a product decision (should the bot answer Comptoir questions?) as a technical knob.

To decide for the final abstention design (options, not yet settled): calibrated CE threshold, minimum number of sources above a score, second lexical signal, or categories to exclude from retrieval. The full eval did its job: reveal the gap Hit@k does not measure — 0.97 recall ≠ safe answers.

4.4 Measured solution: graduated zone (weak evidence) — 14/09 run weak-zone

The binary gate (full evidence / empty) is replaced by three zones on top-1 CE score: < −4 → empty evidence (abstention); [−4, +1[ → WEAK_EVIDENCE block: sources are provided but with the instruction to own the weak relevance ("honestly say no excerpt really answers, point to the closest thread with caution, invite to post"); ≥ +1 → normal evidence.

Result on the 17 representative cases (--variants new, --weak-threshold 1.0):

• the 3 off-topic hallucinations disappear: cooking → "no excerpt provides a recipe"; weather → "no precise information"; finance → "no recommendation available". No more asserted "42 °C" or "DPEC investment committee".
• no in-domain regression: calceophilie (CE −2.32, weak zone!), bannissement (−1.87), carmina-megathread (−0.91), presentation (−0.21) all answer correctly — graduated framing does not turn legitimate low-CE questions into abstentions.
• confident cases (CE ≥ +1, including Comptoir: aston-martin +3.8, finsbury +3.4) answer normally.

Key lesson: overlapping CE distributions (in-domain min −4.7 vs off-topic max +2.8) make any hard threshold imperfect — but the graduated zone does not need a clean split: ambiguous cases are handled honestly rather than cut. That is the measured answer to the "post grain without OOD protection" problem in §4.3.

Watch the quality/safety trade-off — top-1 CE distribution recomputed on the 796 in-domain questions (offline, no LLM):

Weak threshold	% legitimate questions in the "cautious" zone
+2.0	33 %
+1.0	25 % (first-test value — too high)
0.0	17 %
−1.0	10 %
−4.0 (abstention)	0 %

The 3 measured off-topic cases (−0.74, −0.46, −3.60) all fall in [−4, 0) → threshold 0.0 captures the same cases while almost halving exposure of legitimate questions (25 % → 17 %). The cost of the WEAK block is stylistic (more cautious answer), not an abstention — legitimate low-CE questions still answer correctly (calceophilie −2.3). Trade-off to confirm with a scale A/B run.

File: results/20260914-115544-weak-zone.*. See also SAFETY.en.md (refusal/abuse axis — complementary: the weak block handles weak relevance, the refusal clause handles out-of-role requests).

5. Harness

rag/eval/:

• questions.yml — question set + expected_topic_ids (golden). The initial seed is to validate/enrich (some expected items are guesses — check that the listed thread is the canonical one).
• safety.yml — refusal/injection probes (safety axis, see SAFETY.en.md).
• input_gate.yml — input-gate regex rules (intent, not topic).
• prompt_clauses/ — system-prompt variants testable without touching the DB (--system-append).
• eval_retrieval.rb — run inside the container (rails runner): replays Retrieval without cache, splits semantic / keyword / hybrid.
• run_eval.py — host side: copies the ruby, runs, computes Hit@k/MRR per channel, saves results/<date>-<label>.json.
• eval_full.rb + run_full_eval.py — end-to-end eval: prod retrieval vs posts+rerank, real LLM (mini PC), citation validation, gate, weak zone, evidence injection.

python3 -m venv rag/eval/.venv && rag/eval/.venv/bin/pip install pyyaml
rag/eval/.venv/bin/python rag/eval/run_eval.py --label baseline-topic
# options : --k 6  --container app  --questions rag/eval/questions.yml

# for --rerank (config E, CPU cross-encoder, ~1.5 GB RAM at load):
rag/eval/.venv/bin/pip install sentence-transformers
rag/eval/.venv/bin/python rag/eval/run_eval.py --label with-rerank --rerank

# end-to-end generation eval (real mini-PC LLM, ~10 s/question):
rag/eval/.venv/bin/python rag/eval/run_full_eval.py \
    --questions rag/eval/safety.yml --label safety-v3 \
    --input-gate rag/eval/input_gate.yml \
    --system-append rag/eval/prompt_clauses/refusal.txt \
    --weak-threshold 0.0
# options : --n 20  --ids a,b,c  --variants prod,new  --abstain-threshold -4

Retrieval metrics run without an LLM → fast iteration loop. Generation eval calls the real mini-PC qwen3:30b via Tailscale — budget ~10 s per question and per variant.