# RAG pipeline β€” from Postgres to the bot reply πŸ“˜ Cette page est Γ©galement disponible en [franΓ§ais πŸ‡«πŸ‡·](./PIPELINE.md) Architecture document: the full path of a user question, from raw forum data to the cited answer. See [`DESIGN.en.md`](DESIGN.en.md) for the structural choices and [`EVAL.en.md`](EVAL.en.md) for measured metrics. ## Overview ``` POSTGRES (posts, topics) ← Discourse replica/prod β”‚ β”‚ [offline, once per piece of content] β–Ό bge-m3 β†’ 1024-d vector ← embedding model (Ollama) β”‚ ai_posts_embeddings (pgvector) β”‚ USER QUESTION ← at query time β”‚ β”œβ”€β–Ί STAGE 0 β€” input gate (optional, deterministic) β”‚ regex on INTENT (not topics) β†’ immediate decline β”‚ if danger/illegal/misuse detected β€” see SAFETY.en.md β”‚ β”œβ”€β–Ί STAGE 1 β€” recall β”‚ question β†’ bge-m3 β†’ cosine over 425k post vectors β†’ top-25 β”‚ (+ BM25 keyword channel / semantic topic in RRF fusion) β”‚ β”œβ”€β–Ί STAGE 2 β€” precision (rerank) β”‚ mmarco cross-encoder: score (question, candidate) Γ—25 β†’ re-sort β”‚ β”œβ”€β–Ί aggregation: dedup by topic β†’ top-5/6 sources β”‚ + graduated zone on top-1 CE score: β”‚ < βˆ’4 β†’ empty-evidence block (abstention) β”‚ [βˆ’4, 0) β†’ WEAK block (sources + "assume the weakness") β”‚ β‰₯ 0 β†’ normal "STABLE FORUM EVIDENCE" block β”‚ (config D: Β± neighbouring posts for context) β”‚ β–Ό System prompt = persona + refusal clause (SAFETY.en.md) β”‚ + evidence block: titles + excerpts + citation rules β”‚ strict [[1]]..[[6]] + "evidence does not widen your scope" β–Ό LLM (mini PC) β†’ drafted answer + [[n]] markers β”‚ β–Ό Server sanitizer: [[n]] β†’ /t/slug/id URLs ← links built server-side, never by the LLM ``` ## Models in play | Model | Type | Role | Where it runs | |---|---|---|---| | **bge-m3** | bi-encoder (embedding) | text β†’ 1024-dim vector. Encodes each post **once** (backfill) and each question **at query time** | Ollama β€” RTX 2060 laptop in dev; Ollama mini PC `:32134` in prod | | **cross-encoder mmarco-mMiniLMv2-L12-H384** | cross-encoder | relevance score of a **pair** (question, candidate) β€” reads both together | dev: `sentence-transformers` laptop CPU; **prod: TEI `/rerank` on the mini PC** (k3s pod `reranker`, NodePort 32136, Tailscale only) | | **chat LLM `qwen3:30b-a3b-q6k`** | generative (MoE ~3B active) | writes the answer from the 6 provided sources **+** multi-turn query rewrite (`condense_query`, `think:false`) | mini PC (Ollama `:32134`) β€” **one resident model** for both roles | | Postgres/pgvector | β€” | storage + cosine/hamming distance | Discourse container (VPS in prod) | ### Why a 30B MoE on a modest iGPU β€” the structural choice The mini PC 780M iGPU has **no dedicated VRAM**: weights live in unified RAM (GTT, tens of GiB). Two consequences: - **Capacity**: a quantized 30B (~20+ GiB) fits where a same-form-factor dGPU would cap at 8-16 GB VRAM β€” the limiter is system RAM, of which we have 48 GB. - **Speed**: iGPU inference is **memory-bandwidth bound** (each token rereads the active weights). A MoE activates only ~3B parameters per token β†’ throughput stays high despite total model size. Measured: query rewrite in **~1.0 s** hot vs 1.6 s for a dense llama3.1:8B β€” the "big" model is faster than the small one **because it is MoE**. That is why query rewriting moved from `llama3.1:8b` to the main LLM (D10, 17/09): faster, equal or better quality, and one fewer resident model to keep warm in RAM. ### Bi-encoder vs cross-encoder β€” why both They are **not competitors** but two complementary stages: - **bge-m3 (bi-encoder)**: question and post are encoded **separately**. The post vector is precomputed β†’ comparing 425k vectors = a few ms. But the model never "sees" the pair together: relevance is indirect (geometric proximity). - **cross-encoder**: takes the concatenated pair `(question, candidate)` as input β†’ direct, token-level relevance judgment. More precise, but impossible to precompute: using it alone would cost 425k scorings per question. So it only **re-sorts the top-25** from pgvector. Hence: cosine for **recall** (miss nothing), cross-encoder for **fine ranking** (order well). Cosine is never dropped. ## Measured latency (i5-10300H laptop) | Step | Cost | |---|---| | question embedding (bge-m3) | ~ms (one vector) | | pgvector cosine top-25 on 425k rows | ~ms (HNSW/scan index) | | cross-encoder rerank, 25 pairs | **~75-100 ms** (~3-4 ms/pair) | | model load (once, at boot) | ~5 s | | LLM generation | **several seconds** (the real bottleneck) | β†’ Rerank adds ~100 ms per query: **negligible** next to LLM generation. In prod it runs on the **mini PC** (k3s pod `rag-depiedencap/reranker`, TEI `cpu-1.6`, NodePort 32136 β€” not on the VPS, whose RAM is tight ~300-400 MiB free). Checked: TEI/ONNX logits are **identical** to `sentence-transformers` PyTorch β†’ thresholds measured in eval (-4 / 0) stay valid in prod; the plugin converts the TEI sigmoid score back to a logit. ## Detailed steps ### 1. Indexing (offline) Each eligible post β†’ `ai_posts_embeddings` (one row per post): - embedded text = `thread title + category + tags + post body` (native `post_truncation` strategy, `strategy_id=1`, truncated at `max_sequence_length - 2`) - **post grain** is the structural decision: a 2 700-post thread has a diluted, invisible topic vector, while its individual posts stay sharp (measured: Hit@5 0.68 β†’ 0.94, see `EVAL.en.md`) - rejected after measurement: OP/parent-enriched text (`strategy_id=2`, -9 Hit@5 points β€” dilutes the post, EVAL.en.md Β§4quater) ### 2. Retrieval (per question) - **posts**: direct cosine on `ai_posts_embeddings` β†’ top-25 posts - **keyword**: `Search.execute` (Discourse BM25) on distinctive words - **semantic**: native `SemanticSearch` on topic embeddings - **hybrid (prod fallback)**: custom fusion `Retrieval.retrieve` + title gate + boosts β€” prod state before this work, kept as fallback (`depiedencap_ai_citations_posts_pipeline=false` or empty posts table or error) - **RRF fusions**: `Ξ£ 1/(60+rank)` over the lists β€” postsΓ—hybrid is the best measured combination (+0.01 MRR vs posts alone) - rejected: routing by thread size (RRF already routes implicitly) ### 3. Rerank (config E) Top-25 posts β†’ `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` score `(question, title + 300c excerpt)` β†’ re-sort β†’ dedup by topic. Measured: Hit@5 0.94 β†’ **0.97**, MRR 0.84 β†’ **0.89** (with hybrid fusion). ### 4. Generation (evaluated β€” `run_full_eval.py`, real mini-PC LLM) - the 6 best sources β†’ `STABLE FORUM EVIDENCE` prompt with strict rules (cite `[[n]]`, never write a URL, abstain if empty) - **three evidence zones** from top-1 CE score (EVAL.en.md Β§4.4): abstention `< βˆ’4`, cautious framing `[βˆ’4, 0)`, normal `β‰₯ 0` - **refusal clause** in the system prompt + role-limit reminder in the evidence template (SAFETY.en.md β€” 20/20 probes) - the LLM writes; the sanitizer resolves `[[n]]` β†’ `/t/slug/id` **server-side** β€” the LLM cannot invent a link - measured: valid citations, OOD abstention, refusal/abuse, evidence injection, quality regression (A/B n=133 β€” no degradation outside the weak zone) ## File map | What | Where | |---|---| | Prod retrieval (fusion, gate, prompt) | `discourse_plugin/depiedencap-ai-citations/lib/retrieval.rb` | | Enriched backfill (config C, rejected) | `rag/eval/backfill_enriched.rb` | | Retrieval eval (inside the container) | `rag/eval/eval_retrieval.rb` | | Metrics + fusions + rerank harness | `rag/eval/run_eval.py` | | End-to-end generation eval | `rag/eval/eval_full.rb` + `run_full_eval.py` | | Question set + goldens | `rag/eval/questions.yml` | | Safety probes (refusal/injection) | `rag/eval/safety.yml` | | Input gate (regex rules) | `rag/eval/input_gate.yml` | | System-prompt variants | `rag/eval/prompt_clauses/` | | Timestamped results | `rag/eval/results/*.json` | | Measurements and decisions | `rag/EVAL.md` + `rag/SAFETY.md` (FR) / `EVAL.en.md` + `SAFETY.en.md` | | Local forum environment | `rag/LOCAL_DEV.md` (French working doc) | | Pre-deploy prod snapshots | `rag/prod_snapshots/` | ## Production port (15/09/2026) The v2 channel lives in `Retrieval` behind two site settings (`depiedencap_ai_citations_posts_pipeline`, `depiedencap_ai_citations_input_gate`, both **false by default** β†’ deploying the code changes nothing until the flags are flipped): - `retrieve_posts_pack`: `vector_from(question)` (bge-m3 mini PC) β†’ cosine `ai_posts_embeddings` top-25 β†’ `POST /rerank` (mini PC :32136) β†’ topic dedup β†’ 6 sources with post-grain URL `/t/slug/id/post_number` - **exclusions added vs eval** (prod only): `post_type=regular`, non-deleted posts/topics, `archetype='regular'` (no PMs), `read_restricted` categories excluded β€” the legacy channel had `Guardian`/`Search`; direct SQL must filter itself - **input gate**: `Retrieval.gate_match` in `PlaygroundHook` (reply_to + schedule_bot_reply) β†’ `publish_canned_reply!` posts the fixed reply with no LLM; tagged `depiedencap_ai_citations=t` β†’ the sanitizer skips it - **measured fallbacks**: flag off β†’ legacy; empty `ai_posts_embeddings` β†’ legacy (`posts_table_ready?` guard, 60 s cache); embeddings endpoint down β†’ legacy; reranker down β†’ cosine order only, zone `:ok` (config B measured 0.94) - deploy: `docker cp` plugin β†’ container + `/shared/…` + `sv restart unicorn` (Ruby only, no precompile) Native posts backfill (`Jobs::EmbeddingsBackfill`, 5 min, `low` queue) was enabled with `ai_embeddings_per_post_enabled=true` + `ai_embeddings_backfill_batch_size` raised temporarily β€” it produces `(model_id=1, strategy_id=1, strategy_version=1)`, exactly the winning evaluated strategy. Flags flip only once backfill is done.