From b05eea24d5b50a65701d3a1a1d3dac9bd0e40073 Mon Sep 17 00:00:00 2001 From: grgmro Date: Thu, 17 Sep 2026 23:23:03 +0200 Subject: [PATCH] Docs: English pages for design/eval, linked from the English README GitHub and Forgejo only i18n the root README. Public rag pages now have *.en.md siblings so English links stay in English; French files keep the unprefixed names. Co-authored-by: Cursor --- README.fr.md | 3 +- README.md | 15 +- docs/rag/DESIGN.en.md | 238 +++++++++++++++++++++++ docs/rag/DESIGN.md | 4 +- docs/rag/EVAL.en.md | 395 ++++++++++++++++++++++++++++++++++++++ docs/rag/EVAL.md | 2 + docs/rag/PIPELINE.en.md | 207 ++++++++++++++++++++ docs/rag/PIPELINE.md | 2 + docs/rag/README.en.md | 75 ++++++++ docs/rag/README.md | 9 +- docs/rag/REFERENCES.en.md | 126 ++++++++++++ docs/rag/REFERENCES.md | 2 + docs/rag/SAFETY.en.md | 187 ++++++++++++++++++ docs/rag/SAFETY.md | 2 + 14 files changed, 1257 insertions(+), 10 deletions(-) create mode 100644 docs/rag/DESIGN.en.md create mode 100644 docs/rag/EVAL.en.md create mode 100644 docs/rag/PIPELINE.en.md create mode 100644 docs/rag/README.en.md create mode 100644 docs/rag/REFERENCES.en.md create mode 100644 docs/rag/SAFETY.en.md diff --git a/README.fr.md b/README.fr.md index e1457f4..4c1ede4 100644 --- a/README.fr.md +++ b/README.fr.md @@ -106,7 +106,8 @@ Détail : [`docs/rag/EVAL.md`](docs/rag/EVAL.md) · | Chemin | Contenu | |---|---| | racine (`plugin.rb`, `lib/`, `app/`, `config/`) | Le plugin Discourse, installable par `git clone` dans `plugins/` | -| `docs/rag/` | Design, éval, sécurité, références, recette dev local | +| `docs/rag/*.md` (sans suffixe) | Design / éval / sécurité / pipeline / références (FR) | +| `docs/rag/*.en.md` | Mêmes pages en anglais (liées depuis `README.md`) | | `prompts/` | System prompt de production (v3.1) | | `tools/` | Smoke test live (API Discourse) | diff --git a/README.md b/README.md index bd87a64..5a70c4b 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ rewrite (`think:false`) and generation. Deliberate choice: **no RAG framework** (LangChain & co.) — a linear pipeline in direct code, instrumented for evaluation. Full reasoning: -[`docs/rag/DESIGN.md` §9](docs/rag/DESIGN.md#user-content-9-pourquoi-pas-langchain). +[`docs/rag/DESIGN.en.md` §9](docs/rag/DESIGN.en.md#user-content-9-why-not-langchain). ## Measured results (real corpus, 812-question eval set) @@ -93,19 +93,20 @@ list → 0. 0.89 means the first good hit is usually 1st or 2nd, not merely Hit@5 = recall ("did we miss it?"). MRR = ranking ("did we put it first?"). Protocol and formulas : -[`docs/rag/EVAL.md` §2](docs/rag/EVAL.md#user-content-2-métriques). +[`docs/rag/EVAL.en.md` §2](docs/rag/EVAL.en.md#user-content-2-metrics). -Details: [`docs/rag/EVAL.md`](docs/rag/EVAL.md) · -[`docs/rag/SAFETY.md`](docs/rag/SAFETY.md) · -[`docs/rag/PIPELINE.md`](docs/rag/PIPELINE.md) · -[`docs/rag/REFERENCES.md`](docs/rag/REFERENCES.md) +Details: [`docs/rag/EVAL.en.md`](docs/rag/EVAL.en.md) · +[`docs/rag/SAFETY.en.md`](docs/rag/SAFETY.en.md) · +[`docs/rag/PIPELINE.en.md`](docs/rag/PIPELINE.en.md) · +[`docs/rag/REFERENCES.en.md`](docs/rag/REFERENCES.en.md) ## Repository layout | Path | Content | |---|---| | root (`plugin.rb`, `lib/`, `app/`, `config/`) | The Discourse plugin — installable via `git clone` into `plugins/` | -| `docs/rag/` | Design, evaluation, safety, references, local dev recipe | +| `docs/rag/*.en.md` | English design / eval / safety / pipeline / references | +| `docs/rag/*.md` (no suffix) | Same pages in French, plus operator runbooks | | `prompts/` | Production system prompt (v3.1) | | `tools/` | Live smoke test (Discourse API) | diff --git a/docs/rag/DESIGN.en.md b/docs/rag/DESIGN.en.md new file mode 100644 index 0000000..0cc0b02 --- /dev/null +++ b/docs/rag/DESIGN.en.md @@ -0,0 +1,238 @@ +# Design — post-grain RAG (Depiedencap) + +📘 Cette page est également disponible en [français 🇫🇷](./DESIGN.md) + +> **Last update**: 17/09/2026 · **Status**: winning config (E: native posts +> + rerank) measured; code in prod on the VPS, member flags still **OFF**, +> agent limited to a demo group. §0 below is a **historical snapshot** +> (topic grain) — current architecture: +> [`README.md`](../../README.md) · [`PIPELINE.en.md`](PIPELINE.en.md). +> Results: [`EVAL.en.md`](EVAL.en.md) · Safety: [`SAFETY.en.md`](SAFETY.en.md) · +> References: [`REFERENCES.en.md`](REFERENCES.en.md) + +## 0. Current state (snapshot) + +- **Index**: `ai_topics_embeddings` — 1 vector/topic (bge-m3, 1024 dims, + 4096-token input), 22 340 rows. pgvector storage in the forum DB. +- **Retrieval** (`retrieval.rb`): `Search.execute` (keyword) + + `SemanticSearch` (pgvector, HyDE off), custom fusion (base rank + title + bonus + hybrid bonus +12), mandatory title filter (distinctive word ≥ 6 + chars / stem), FR stopwords, hard-coded `INTENT_QUERIES`, 2 h cache, + max 6 sources. +- **Evidence**: titles + 320-char excerpts injected server-side; the LLM + emits bare `[[n]]`; sanitizer → URL whitelist (`Topic.exists?`), Sources + footer + CTA. + +### Known weaknesses + +| Weakness | Cause | +|---|---| +| Megathreads invisible in detail | 1 vector / 300 posts = dilution + 4096 truncation | +| Citation to the whole thread | `[[n]]` → `/t/slug/id`, not the precise message | +| Evidence without context | Isolated 320-char excerpt, unresolved anaphora | +| Ad-hoc fusion | Custom score, not a standard method (RRF) | + +## 1. Storage schema (decided) + +**Vectors in the forum Postgres** (pgvector), no external store. +Rationale: atomicity with `posts`, free backup inside the `.tar.gz`, the +local dev forum gets its copy automatically. + +| Table | Content | Status | +|---|---|---| +| `ai_topics_embeddings` | 1 vector/topic (native discourse-ai) | existing, **kept** (related topics + fallback) | +| `ai_posts_embeddings` | 1 vector/post, native `strategy_id=1` + enriched `strategy_id=2` (test) | **implemented** — see note below, replaces `dpec_post_embeddings` | + +> **Revision D2 (14/09)**: the custom table `dpec_post_embeddings` was +> **not** created — decision changed mid-course. Native +> `ai_posts_embeddings` with key `(model_id, strategy_id, post_id)` does +> the same job (several representations of the same post coexist) without +> duplicating infra (index, purge, backfill job). `strategy_id=1` = native +> discourse-ai (title+cat+tags+body, free, **this is the winner**), +> `strategy_id=2` = OP/parent enriched (measured, rejected — see +> EVAL.en.md §4.2). Custom would have remained necessary only for C4 +> (LLM blurb), never implemented (see §8). + +> Why a **custom** table rather than native `ai_posts_embeddings`: +> native backfill embeds bare `post.raw` — no way to inject context +> (title/OP/parent). Custom table = full control of the embedded text + +> metadata (post_number, topic_id, text hash, model). +> `ai_posts_embeddings` stays disabled (`ai_embeddings_per_post_enabled=false`). +> +> Findings from the local restore (12/09): the native table already exists +> with `embeddings halfvec` + **HNSW binary-quantized** index per +> `(model_id, strategy_id)` — unique key `(model_id, strategy_id, post_id)`. +> **And** native strategy `Truncation#post_truncation` already embeds +> "title + category + tags + post body" — partial enrichment is **free** +> by turning on `ai_embeddings_per_post_enabled`. What remains custom: +> parent/OP excerpt and our metadata. A custom `strategy_id` in the native +> table would give fully enriched vectors while reusing index + purge — +> a serious option before creating `dpec_post_embeddings` — see D2. + +Proposed schema: + +```sql +post_id bigint PK FK → posts.id +topic_id bigint (denormalised, fast filter/aggregation) +post_number int (citation anchor) +embedding vector(1024) +source_hash char(40) -- sha1 of the embedded enriched text (detects edits) +model text -- 'bge-m3' (a model change = incompatible vectors) +embedded_at timestamptz +``` + +Index: **to be measured** — brute force `ORDER BY embedding <=> $q` +(cosine) on ~300-400k rows ≈ 200-500 ms, may be enough on a path where +the LLM takes seconds. Otherwise HNSW (`vector_cosine_ops`, tune `m`/`ef`) +— recall ~95-99 %, RAM-heavy build on the 8 GB VPS. + +## 2. Embedded text (enrichment) + +A bare post fails on anaphora ("it was rubbish" → rubbish **what**?). +Context resolves most references. Candidate text to embed: + +``` +[topic title] › [OP excerpt if post_number > 1] › [parent-post excerpt +if a formal reply] › [cleaned post.raw] +``` + +Variants to test (matrix §4): + +| Code | Embedded text | Index cost | +|---|---|---| +| `B` | bare `post.raw` | minimal | +| `C1` | `title + post.raw` | minimal | +| `C2` | `title + OP excerpt + post.raw` | minimal | +| `C3` | `C2 + parent excerpt` (reply_to_post_number / quote) | minimal | +| `C4` | `C3 + LLM blurb` ("in this thread about X, this message says…") | 1 LLM call/post — expensive, bound to a subset | + +> C4 = Anthropic "Contextual Retrieval" literally (−35 % top-20 failures +> on its own). C1-C3 = the free version. The literature suggests the main +> gain is *situating* the chunk, not necessarily the LLM. + +Note: an orphan post (a reply that does not quote a neighbour) does +**not** need to be retrieved itself — it will be read via neighbourhood +expansion of the neighbouring post that *does* carry the context (§3). + +## 3. Two-stage retrieval (small-to-big) + +``` +question + → embed query (bge-m3, same instance) + → top-K posts (pgvector) + top-M topics (existing keyword Search) + → RRF fusion (k=60) ← replace custom scoring? + → aggregate by topic: 1 entry/thread, score = max(posts in thread) + → evidence block: per entry, + title + targeted post excerpt + ±N neighbouring posts + → LLM: citations [[n]] → sanitizer → /t/slug/id/post_number +``` + +- **Aggregate by topic**: stops a megathread monopolising the 6 evidence + slots. Topic score = best post (or mean of top-2, to test). +- **Neighbourhood** (`window` ±2 or ±3 posts, parameter): carries context, + resolves anaphora at **read** time — not at index time. Sentence-window + retrieval applied to posts. +- **Post-level citation**: `[[n]]` → `/t/slug/topic_id/post_number`. + Sanitizer whitelist: `Post.exists?` (not only `Topic.exists?`). + +## 4. Config matrix — results (14/09/2026) + +| Config | Description | Hit@5 | MRR | Verdict | +|---|---|---|---|---| +| `A` | topics (then-current prod) | 0.68 | 0.63 | live baseline | +| `B` | native posts (strategy_id=1) | **0.94** | 0.84 | **kept** — the main gain | +| `C` | OP/parent-enriched posts (strategy_id=2) | 0.85 | 0.80 | **measured, rejected** — dilutes the vector | +| `D` | ±N neighbourhood at generation time | — | — | not tested in isolation (rerank + 320c evidence covers the need in practice) | +| `E` | B + mmarco cross-encoder rerank | **0.97** | **0.89** | **kept — winning config** | +| `F` | RRF posts×hybrid fusion | 0.95 | 0.88 | measured, marginal on E (+0.01 MRR), optional | +| `G` | routing by topic size | 0.93-0.95 | 0.77-0.86 | **measured, rejected** — degrades MRR vs free fusion | + +Full detail, methodology and measured n: `EVAL.en.md` §1-4. +Each config compared by changing **one** factor at a time. +Safety of config E (refusal, gate, injection): `SAFETY.en.md`. + +Secondary variables to explore later: top-K (10/20/50), embed input +length (1024 vs 4096), title-filter threshold, window size. + +## 5. Corpus to index + +| Choice | Position | +|---|---| +| Categories | **All public ones** ("forum life" questions are legitimate) | +| PMs | **Never** (existing product decision) | +| Min length | To calibrate — posts < ~30-40 chars likely excluded ("+1", "thanks!") | +| Deleted / staff posts | excluded (`deleted_at`, non-public categories) | +| Estimated volume | ~300-350k indexable posts out of ~425k | + +## 6. Cost and load + +- **Backfill**: ~300k bge-m3 embeddings. Laptop CPU ~10-20/s → 4-8 h + (several nights OK). Prod: same model on the mini-PC iGPU, faster. +- **Query**: 1 embed (~50-200 ms) + 1 SQL similarity + expansion = + negligible next to LLM generation. +- **VPS storage**: ~300-400k × (1024 dims × 4 B + row overhead) + ≈ **2-3 GB** — tenable vs ~13 GB free, to watch. +- **Incremental**: 1 embed per new post (async Sidekiq job), + re-embed + if `source_hash` changes after an edit. + +## 7. Decisions + +| # | Date | Decision | Why | +|---|---|---|---| +| D1 | 12/09/2026 | Vectors in the forum Postgres, no external store | Atomicity, backup in the `.tar.gz`, free local replica | +| D2 | 12/09/2026, **revised 14/09** | Native `ai_posts_embeddings` (`strategy_id`), no custom table | Same need (multi-representation coexistence) without duplicating index/purge/backfill | +| D3 | 12/09/2026 | Dev: bge-m3 **local laptop CPU/GPU**, LLM on mini PC | Dev/prod isolation; NUM_PARALLEL=2 is risky for the 30B | +| D4 | 13/09/2026 | RRF measured vs custom fusion — kept as option (F), not mandatory | Marginal gain (+0.01 MRR) on winning config E | +| D5 | 13/09/2026 | Config B (bare posts) kept as base, C (enriched) rejected | Measured: 0.94 vs 0.85 Hit@5 — OP/parent context dilutes the vector | +| D6 | 14/09/2026 | Config E (B + cross-encoder rerank) = final retrieval | Measured: 0.97 Hit@5 / 0.89 MRR, ~100 ms cost negligible | +| D7 | 14/09/2026 | G (routing by size) rejected | Measured at scale (796): degrades MRR vs free fusion | +| D8 | 14/09/2026 | Abstention/scope: graduated zone on CE score (no hard threshold) + deterministic input gate + refusal clause in the prompt | No threshold cleanly separates in-domain/off-topic (overlapping distributions) — see SAFETY.en.md | +| D9 | 17/09/2026 | **No LangChain** — patterns reused without the dependency | See [§9](#user-content-9-why-not-langchain) | +| D10 | 17/09/2026, **revised same day** | Multi-turn query rewrite (`condense_query`) via **`qwen3:30b-a3b-q6k`** — the main LLM, not a dedicated model | Retrieval only saw the last message; real bug in usage. First cut `llama3.1:8b` (qwen3:4b swallowed its budget in `reasoning`); comparative test → the MoE (~3B active) condenses in ~1.0s vs 1.6s, equal quality, **one fewer resident model** in RAM. `think:false` forced in the request (thinking guardrail, ignored without error by non-thinking models) | + +## 8. Open questions — status + +- ~~`OP excerpt` in the enriched text~~ → **settled**: measured and rejected (D5/D2). +- Neighbourhood window (config D): **not measured in isolation** — rerank + + 320c excerpt is enough in practice on the end-to-end eval (`SAFETY.en.md` + run 812); revisit only if unresolved anaphora shows up in real use. +- Reranker: **settled** — yes, +0.03 Hit@5 / +0.05 MRR for ~100 ms. + Runs on CPU; mini PC or VPS EPYC Rome both sufficient + (measured: `PIPELINE.en.md`). +- Abstention: **settled** — not a single threshold, graduated zone + (`< -4` abstain, `[-4,0)` cautious, `≥0` normal) + input gate + for the safety axis (independent of relevance). See SAFETY.en.md. +- Hand-coded `INTENT_QUERIES` (then-current prod): **not carried** into the + new pipeline — the cross-encoder rerank covers that need without ad-hoc + rules to maintain; confirm if a regression case appears. +- Config D (generation neighbourhood), config F (RRF): optional, + measured marginal gain — not needed for the MVP. + +## 9. Why not LangChain + +LangChain is a **library** (Python/JS — not an LLM, not a service to +deploy) that provides LLM orchestration bricks: retrieval, memory, query +rewriting, tools. It is the most common framework for *prototyping* a RAG +— hence its presence in job ads. + +**Why we do not use it here** — an architecture choice, not a rejection of +the idea: + +- Our pipeline lives in a **Discourse Ruby plugin**, inside the forum + process. LangChain would mean a separate Python service to deploy, + maintain and secure — a new failure point between the forum and the + mini PC, to re-implement logic that is already linear. +- We already do the same steps in direct code (~600 lines): + embed → pgvector → cross-encoder rerank → LLM → sanitizer. No need + for an abstraction on a linear, instrumented pipeline (the + `eval_full.rb` eval calls the functions directly). +- If we one day wanted **agentic** capabilities (the bot picks its tools, + chains searches), a dedicated service would become defensible — + LangChain/LangGraph would then be candidates. Not the current roadmap. + +**What we still took from that world**: **conversational query rewriting** +(*condense question* / *history-aware retriever* pattern from LangChain, +and QReCC/CANARD literature) — implemented in `Retrieval.condense_query`: +latest message + the previous 3 turns → a standalone query before +embedding. The rewriting model is **the same Qwen3-30B as generation** +(D10) — see PIPELINE.en.md "why a MoE". diff --git a/docs/rag/DESIGN.md b/docs/rag/DESIGN.md index 85d4184..2054168 100644 --- a/docs/rag/DESIGN.md +++ b/docs/rag/DESIGN.md @@ -1,10 +1,12 @@ # Design — RAG grain post (Depiedencap) +📘 This page is also available in [English 🇬🇧](./DESIGN.en.md) + > **Dernière màj** : 17/09/2026 · **Statut** : config gagnante (E : posts > natifs + rerank) mesurée ; code en prod sur le VPS, flags membre encore > **OFF**, agent limité à un groupe démo. Le §0 ci-dessous est un > **instantané historique** (grain topic) — l'architecture actuelle : -> [`README.md`](../../README.md) · [`PIPELINE.md`](PIPELINE.md). +> [`README.fr.md`](../../README.fr.md) · [`PIPELINE.md`](PIPELINE.md). > Résultats : [`EVAL.md`](EVAL.md) · Sécurité : [`SAFETY.md`](SAFETY.md) · > Références : [`REFERENCES.md`](REFERENCES.md) diff --git a/docs/rag/EVAL.en.md b/docs/rag/EVAL.en.md new file mode 100644 index 0000000..08f9eed --- /dev/null +++ b/docs/rag/EVAL.en.md @@ -0,0 +1,395 @@ +# RAG evaluation — protocol and question set + +📘 Cette page est également disponible en [français 🇫🇷](./EVAL.md) + +> **Status**: active (13/09/2026). Set expanded to **101 questions** sourced +> from the local DB (goldens verified as existing in the DB; relevance of +> canonical threads still to be validated by a forum human). +> Principle: **measure by layer** (retrieval ≠ generation), see +> [`REFERENCES.en.md`](REFERENCES.en.md) §5. +> +> **Public-facing intent**: this `rag/` folder (protocol, timestamped +> results, comparison table) is a deliverable meant to be **shown publicly +> later** (Forgejo). Consequences for how we work: +> - each run produces a timestamped JSON committed in `results/`; +> - the §4 table is filled for every tested config, with qualitative notes; +> - commits tell the reasoning (why a config, what the measurement showed) +> — not just the mechanics; +> - one factor at a time (§3), so the table reads as a demonstration. + +## 1. Eval-set structure + +101 questions (initial 15–20 target exceeded — a larger corpus gives less +noisy signals), each with **golden documents** (threads/messages that +*should* come back). Format: + +```yaml +- id: calceophilie + question: "C'est quoi la calcéophilie ?" + expected_topic_ids: [34091] # /t/du-terme-calceophile/34091 + expected_post_ids: [] # empty = topic grain is enough + difficulty: facile + tags: [définition] + +- id: + question: "" + expected_topics: [] + expected_posts: [] # the precise message — the real post-grain test + difficulty: megathread + tags: [contexte, anaphore] +``` + +Actual coverage (101 questions, `questions.yml` sections A–J): + +| Question type | n | What it tests | +|---|---|---| +| Definition / manufacturing technique | 7 | semantic retrieval on concepts (constructions, gemming…) | +| Brand opinions | 24 | brand→canonical thread, megathreads up to 2.7k posts | +| Care / repair | 21 | fine grain (wax, glazing, fading…) | +| Buying / second-hand / budget | 8 | transactional intents | +| Sizing / width | 7 | sizing — answers scattered across posts | +| Shoe types | 12 | product taxonomy + colour megathreads | +| Forum life / rules | 10 | non-"shoes" categories, sensitive topics | +| Answer inside a megathread (post grain) | 2 | **the core of the workstream** — `expected_post_ids` | +| Off-topic | 6 | correct abstention | +| Anaphora / conversational context | — | out of scope of the single-turn harness — manual bot test | +| LLM unavailable | — | manual test (humorous message) | + +## 2. Metrics + +The two README numbers (**Hit@5 = 0.97**, **MRR = 0.89**) measure +**retrieval only** (does the right thread/post come back?), not the quality +of the generated sentence. They are computed on the golden set +(`questions.yml`): each question has one or more `expected_topic_ids` +(and sometimes `expected_post_ids`). + +### Hit@5 (recall in the top-5) + +For one question, retrieval **succeeds** if **at least one** golden +document appears in the first 5 results. Hit@5 = that success averaged +over the set (0 or 1 per question). + +- **0.97** (config E, n=796): 97 questions out of 100 see the right + thread in the top-5. +- A Hit@5 of 1.0 with the right thread always in 5th place would still + be "perfect" as recall, and bad as ranking → hence MRR beside it. +- Variants in the tables: **Hit@k topic** (the golden thread is in the + top-k); **Hit@k post** (the golden *message* is in the top-k, post grain + only). The headline figure is scale Hit@5 **topic** (batch K). Early + passes also used k ∈ {3, 6} on curated batch A–J. + +### MRR — Mean Reciprocal Rank (ranking of the first hit) + +For each question: `1 / rank` of the **first** relevant result +(1st → 1, 2nd → 0.5, 3rd → ≈ 0.33, 5th → 0.2). If the golden is missing +from the returned list: **0**. MRR = mean of those scores. + +- **0.89** (config E): the first relevant hit is in practice 1st or 2nd, + rarely 4th/5th. +- Unlike Hit@5, a golden moving from #1 to #5 **lowers** MRR (1 → 0.2) + even if it stays "in the top-5". + +In one line: **Hit@5 = did we miss it?** · **MRR = did we put it first?** +The topic → post jump (0.68 → 0.94 Hit@5) is the main gain; rerank +(0.94 → 0.97 Hit@5, 0.84 → 0.89 MRR) mainly improves order. + +### Retrieval (automatic) — recap + +| Metric | Definition | +|---|---| +| **Hit@5 topic** | at least one `expected_topic` in the top-5 (README figure) | +| **Hit@k topic** | same for a given k (batches A–J: also k ∈ {3, 6}) | +| **Hit@k post** | the `expected_post` in the top-k posts (post grain only) | +| **MRR** | mean of `1/rank` of the first relevant hit (0 if missing) | +| Pollution | evidence slots occupied by the same topic (measures aggregation) | + +### Generation (semi-auto) + +| Metric | How | +|---|---| +| Real cited links | % `[[n]]` → `Post.exists?` — should be 100 % (sanitizer) | +| Relevant citation | does the cited message carry the answer? (human, 0/1) | +| Faithfulness | claims supported by the evidence (LLM-judge or review) | +| Correct abstention | off-topic → abstain, no invention | +| CTA present | "To go further" footer with an invitation to post | + +## 3. Protocol + +1. **Baseline `A`** (live config, topic grain): run the set on the local + forum → fill the table. That is the bar to beat. +2. Each config (`B`, `C`, `D`, `E`…): **one factor changed** vs the + previous config, same set, same metrics. +3. Results → table below + qualitative notes (striking examples). +4. A question that regresses = we understand it before concluding. + +## 4. Results table (to fill) + +| Config | Hit@5 topic | Hit@k post | Real links | Abstention OK | Notes | +|---|---|---|---|---|---| +| A (live, topics) | **39/95** (kw 17/95, sem 32/95) | n/a | | | 13/09 v2, goldens corrected — hybrid=41% | +| B (native posts) | **88/95** | exact post `#1` ×2 (carmina-sizing, glaçage-sèche-cheveux) | | | 13/09 v2 — 425k posts embedded | +| C (enriched posts) | **0.85** (n=796) | — | | | 14/09 — **measured, rejected**: enriching the embedded text (OP + parent excerpt) degrades vs B (0.94) — dilution of post content, §4quater | +| D (+ neighbourhood) | | | | | generation side (LLM context), not Hit@k | +| E (+ rerank) | **97%** (n=796) | `p✓` kept | | | 13/09 — CPU cross-encoder, Hit@5=0.97/MRR=0.89 (rrf rerank×hybrid) — detail §4ter | +| F (RRF fusions) | **87-88/95** | `p✓` ×2 kept at #1 | | | 13/09 v2 — posts×hybrid MRR **0.80** | +| G (size routing) | 84-88/95 | `p✓` ×2 | | | 13/09 — **rejected**: degrades vs F, RRF already routes implicitly | + +Size-routing findings (13/09, thresholds 25/50/100/250 posts): filtering +topics > N posts out of the topic channel before fusion **degrades +monotonically** (MRR 0.68@25 → 0.76@250 < 0.80 with no filter). Long +threads almost never get a "topic" vote anyway (diluted embedding = +invisible), so there is no noise to remove; cutting medium threads removes +useful votes. **RRF already implements the spirit of routing**: a +megathread only comes back via its posts, a short thread benefits from +both channels. Variant documented then dropped — negative result, kept +for the demonstration. + +Baseline A→B findings (13/09, n=95, goldens v2): the native posts channel +alone does **Hit@5 0.93 / MRR 0.77** vs topic hybrid 0.41. Golden +correction by pooling (13/09): the initial "failures" were bringing back +**valid** threads missing from the expected list — widened for +`corthay-avis`, `noeuds-lacets`, `presentation-nouveau-membre`, +`bienvenue-nouveau`; `richelieu-bout-rapporte` rephrased (a member names +the brand). **Method lesson: the initial 0.87→0.93 gap comes from golden +completeness, not retrieval** — keeping goldens current is the most +fragile part of the benchmark. + +Fusion findings (13/09): RRF posts×{semantic, hybrid} **does not gain +recall** (posts is enough) but **improves MRR** (0.77→0.80) and rescues +posts failures (`loding-pointure` #23→**#1**, `grande-pointure` #11→#1 — +sizing diluted in posts benefits from the semantic topic signal). RRF +posts×keyword **degrades** (BM25 noise: MRR 0.74). Posts-channel +pollution: top-5 = 2.8 distinct topics on average, dominant topic 63 % of +slots. + +Baseline A findings (12/09): the Carmina megathread (2690 posts) absent +from semantic top-20 **and** keyword top-25 → topic-embedding dilution is +real and measured. The title gate (`relevance_score` requires a +distinctive word in the title) also rejects relevant threads whose title +does not contain the keyword — keep that in mind when reading post-grain +metrics. + +## 4bis. Scale confirmation (812 questions, 13/09) + +Batch A-J (101 curated questions, goldens verified by pooling) was +completed by a **scale-generated batch** (section K of `questions.yml`, ++701 questions + 10 off-topic, sourced by mining all categories/tags in +the DB — different methodology, documented at the top of the section, +golden = single source topic without multi-thread union). Goal: check +that curated-batch results are not a small-sample artefact. + +| Channel | Hit@5 (n=796) | MRR | +|---|---|---| +| semantic (topic) | 0.55 | 0.43 | +| keyword (topic) | 0.41 | 0.38 | +| native posts | **0.94** | **0.84** | +| hybrid (then-current prod) | 0.68 | 0.63 | +| rrf posts×hybrid | **0.95** | **0.88** | +| routed_250 | 0.95 | 0.86 | +| enriched posts (C) | 0.85 | 0.80 | +| rrf enriched×hybrid | 0.92 | 0.85 | +| rerank enriched | 0.91 | 0.86 | + +**Confirmed**: the native-posts / RRF-fusion advantage holds at scale, +slightly improving (less statistical noise). **Notable gap vs the curated +batch**: prod `hybrid` scores 0.68 here vs 0.41 on A-J — the generated +batch contains many highly distinctive titles (named brands/products) +that BM25 matches easily, while A-J deliberately tests rephrasings and +harder megathreads. The two measurements are complementary: A-J documents +fine cases, K measures general robustness. Size routing (rejected on A-J) +is confirmed rejected here too (routed_250 < rrf_post_hyb with no filter). + +## 4ter. Config E — cross-encoder rerank (13/09, n=796) + +Model: `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` (multilingual, light, +CPU — loaded once via `sentence-transformers`, `--rerank` in +`run_eval.py`). Re-sorts the posts-channel top-25 (`{title}. {first 300 +characters of the post}` vs question) before topic dedup. + +| Channel | Hit@5 | MRR | +|---|---|---| +| native posts | 0.94 | 0.84 | +| rrf posts×hybrid | 0.95 | 0.88 | +| **rerank** | **0.97** | 0.88 | +| **rrf rerank×hybrid** | **0.97** | **0.89** | + +Rerank mainly gains **recall** (0.94→0.97 Hit@5): it promotes relevant +posts poorly ranked by cosine distance alone, already in the top-25 but +not in the top-5. Net but incremental gain versus the initial topic→post +jump (still the dominant factor of the workstream). `eval_retrieval.rb` +now exposes `text` (title + 300c excerpt) per post so the host can rerank +without an extra step inside the container. + +## 4quater. Config C (enriched posts) — measured, rejected (14/09, n=796) + +Implemented for the vitrine (comparable metrics on the whole dataset): +full `strategy_id=2` backfill on 425 505 posts (`backfill_enriched.rb`, +~3h GPU). Embedded text = native context (title+cat+tags) **+ 500c +excerpt of the thread's first message + 400c excerpt of the parent post** +(`reply_to_post_number`), truncated at max_seq−2 like the native strategy. +Hypothesis: disambiguate short replies ("same in 8.5") that the bare post +does not contextualise. + +| Channel | Hit@5 | MRR | +|---|---|---| +| native posts (s1) | **0.94** | **0.84** | +| **enriched posts (s2)** | 0.85 | 0.80 | +| rrf posts×hybrid | **0.95** | **0.88** | +| rrf enriched×hybrid | 0.92 | 0.85 | +| rerank (s1) | **0.97** | 0.88 | +| rerank enriched (s2) | 0.91 | 0.86 | + +**Negative result, uniform across all variants**: enrichment costs ~9 +Hit@5 points and ~4 MRR points on the channel alone, and the degradation +propagates to fusions and rerank. Interpretation: the OP/parent excerpt +consumes token budget and **dilutes the post's own content** in the +vector — contextual noise hurts more than disambiguation helps. Typical +case: `scaled-faq-les-diff-rents-montages` — the golden (36778) is #1 in +s1 but **drops out of the top-25 in s2**: the shared OP context +homogenises vectors of posts in the same thread and flips toward a +neighbouring thread (36856) that occupies the first 5 slots at ~equal +distance. + +**Decision: rejected on measurement.** The bare post (title+cat+tags+body) +is the right granularity. Context belongs **at generation time** +(config D — neighbourhood injected into the LLM prompt, without touching +embeddings), not in the indexed vector. + +Dataset note: residual failures (~3 % after rerank) remain dominated by +generic titles in batch K ("Présentation": 1671 close threads in the DB, +"Question": 211, "ceinture": 123) whose single-topic golden is +structurally ambiguous — a measurement limit, not a retrieval one. + +## 4quinquies. FULL eval — real generation (14/09, n=17→20) + +First end-to-end test with the **real prod LLM** (`qwen3:30b-a3b-q6k`, +vLLM mini PC `:32134`). `eval_full.rb` emits per question the prod +evidence (`instructions_block`/`for_question` as-is) + the native-posts +top-25; `run_full_eval.py` rebuilds the "new" block (posts+rerank, same +template), calls the LLM with `system = persona + block`, and collects +the answer + `[[n]]` markers for validation. Readable digest in +`results/*-full-*.yml`. + +### Lessons (17 questions: 14 in-domain, 3 off-topic) + +| | prod (topic grain) | new (posts+rerank) | +|---|---|---| +| in-domain | answers 12/14; **wrong abstention ×2** (`montage-norvegien-def`, `carmina-sizing-megathread` — empty evidence) | answers 14/14 | +| off-topic | abstains ×3 cleanly | **hallucinates ×3** with citations (invented recipe, "42 °C in Paris", "DPEC investment committee") | +| invalid citations | 2 (`scaled-aston-martin` cites [[2]][[3]] with only 1 source) | 0 | +| LLM latency | ~3-10 s/answer (mini PC) | same | + +**Central finding**: the prod title gate was serving as abstention *by +accident* — off-topic → no distinctive word in titles → empty evidence → +abstention. Removing it gains in-domain recall (the 2 wrong abstentions +are fixed) but **loses out-of-domain protection**: the posts channel +always returns 25 candidates and the LLM uses them, even off-brief. + +**CE score alone does not split cleanly** (calibration 14/09): +top-1 CE on 796 in-domain questions: min −4.74, p25 +0.42, 4/796 < −4; +on 16 off-topic: −5.9 → **+2.77**. The overlap comes from Le Comptoir — +the forum talks food, wine, cinema: "bœuf bourguignon" has legitimate +tangential threads (off-topic cinema = +2.77). A −4 threshold cuts +extreme cases with no measured in-domain false negative, but lets the +grey zone through; that is as much a **product** decision (should the +bot answer Comptoir questions?) as a technical knob. + +**To decide for the final abstention design** (options, not yet settled): +calibrated CE threshold, minimum number of sources above a score, second +lexical signal, or categories to exclude from retrieval. The full eval +did its job: reveal the gap Hit@k does not measure — 0.97 recall ≠ safe +answers. + +### 4.4 Measured solution: graduated zone (weak evidence) — 14/09 run `weak-zone` + +The binary gate (full evidence / empty) is replaced by **three zones** +on top-1 CE score: `< −4` → empty evidence (abstention); +`[−4, +1[` → `WEAK_EVIDENCE` block: sources are provided but with the +instruction to own the weak relevance ("honestly say no excerpt really +answers, point to the closest thread with caution, invite to post"); +`≥ +1` → normal evidence. + +Result on the 17 representative cases (`--variants new`, +`--weak-threshold 1.0`): + +- **the 3 off-topic hallucinations disappear**: cooking → "no excerpt + provides a recipe"; weather → "no precise information"; finance → "no + recommendation available". No more asserted "42 °C" or "DPEC + investment committee". +- **no in-domain regression**: `calceophilie` (CE −2.32, weak zone!), + `bannissement` (−1.87), `carmina-megathread` (−0.91), + `presentation` (−0.21) all answer correctly — graduated framing does + not turn legitimate low-CE questions into abstentions. +- confident cases (CE ≥ +1, including Comptoir: aston-martin +3.8, + finsbury +3.4) answer normally. + +Key lesson: overlapping CE distributions (in-domain min −4.7 vs +off-topic max +2.8) make any hard threshold imperfect — **but the +graduated zone does not need a clean split**: ambiguous cases are +handled honestly rather than cut. That is the measured answer to the +"post grain without OOD protection" problem in §4.3. + +**Watch the quality/safety trade-off** — top-1 CE distribution recomputed +on the 796 in-domain questions (offline, no LLM): + +| Weak threshold | % legitimate questions in the "cautious" zone | +|---|---| +| +2.0 | 33 % | +| +1.0 | **25 %** (first-test value — too high) | +| **0.0** | **17 %** | +| −1.0 | 10 % | +| −4.0 (abstention) | 0 % | + +The 3 measured off-topic cases (−0.74, −0.46, −3.60) all fall in +`[−4, 0)` → threshold **0.0** captures the same cases while almost +halving exposure of legitimate questions (25 % → 17 %). The cost of the +WEAK block is stylistic (more cautious answer), not an abstention — +legitimate low-CE questions still answer correctly (calceophilie −2.3). +Trade-off to confirm with a scale A/B run. + +File: `results/20260914-115544-weak-zone.*`. See also +[`SAFETY.en.md`](SAFETY.en.md) (refusal/abuse axis — complementary: the +weak block handles weak relevance, the refusal clause handles out-of-role +requests). + +## 5. Harness + +`rag/eval/`: + +- `questions.yml` — question set + `expected_topic_ids` (golden). The + initial seed is **to validate/enrich** (some expected items are + guesses — check that the listed thread is the canonical one). +- `safety.yml` — refusal/injection probes (safety axis, see + [`SAFETY.en.md`](SAFETY.en.md)). +- `input_gate.yml` — input-gate regex rules (intent, not topic). +- `prompt_clauses/` — system-prompt variants testable without touching + the DB (`--system-append`). +- `eval_retrieval.rb` — run **inside the container** (`rails runner`): + replays `Retrieval` without cache, splits semantic / keyword / hybrid. +- `run_eval.py` — host side: copies the ruby, runs, computes Hit@k/MRR + per channel, saves `results/-