discadmin/disclosure-bureau
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
disclosure-bureau/docs/adrs/ADR-001-embedding-rerank-stack.md
Luiz Gustavo eaf282c535
Some checks failed
CI / Web — typecheck + lint + build (push) Failing after 40s
Details
CI / Scripts — Python smoke (push) Failing after 3s
Details
CI / Web — npm audit (push) Failing after 29s
Details
CI / Retrieval — golden set (Recall@5 + MRR) (push) Failing after 3s
Details
W2: rerank opt-in, analyze_image_region tool, RAG eval, graph cleanup, ADRs 
- TD#8 hybrid.ts: rerank_strategy {always|when_top_k_gt|never} + threshold
  (default skips rerank for top_k ≤ 15; chat tool uses threshold 10)
- O11 vision.ts + tools.ts: analyze_image_region tool — sharp-crops the
  bbox, claude CLI reads the temp PNG via Read tool, Sonnet vision answers
- TD#12 /graph: SigmaGraph replaces ForceGraphCanvas; react-force-graph-2d
  uninstalled (-37 transitive deps); force-graph-canvas.tsx deleted
- TD#27 messages/route.ts gatherContext slice sizes via CTX_* env vars
- TD#22 tests/rag/: golden.yaml (15 queries) + run.py (Recall@k + MRR +
  negative-pass rate) + baseline.json + CI job in .forgejo/workflows/ci.yml
- docs/adrs/: ADR-001..005 published from systems-atelier deliverables

Verified live on disclosure.top: top_k=5 path skips rerank (6.7s embed-only,
was 12-15s with rerank); rerank=always still available on demand.
First RAG baseline: Recall@5 = 0.2083, MRR = 0.25, Negative pass = 1.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 19:20:09 -03:00

2.6 KiB
Raw Permalink Blame History

adr title status date deciders project
ADR-001 Embedding e reranker stack — manter BGE-M3 self-hosted CPU; tornar reranker opt-in; reavaliar GPU em 6 meses accepted 2026-05-23 sa-principal, sa-architecture-lead, sa-platform-lead disclosure-bureau

Context

A retrieval pipeline atual (BGE-M3 dense + BM25 + RRF + BGE-Reranker-v2-M3 cross-encoder) entrega Recall@5 aceitavel para 20.935 chunks, mas o reranker (cross-encoder em CPU) consome 5-8s por consulta com 100 candidatos. Esse e o gargalo dominante de UX no chat sincrono.

Alternativas conhecidas:

  1. Manter status quo: CPU rerank sempre on.
  2. Skip rerank: confiar so em RRF do RPC hybrid_search_chunks. Mais rapido, perde precisao em queries ambiguas.
  3. Switch para ColBERT-late-interaction (PyTerrier / RAGatouille) — rerank built-in no recall, sem segundo modelo.
  4. GPU para reranker: VPS com GPU pequena (Hetzner GPU-1: ~$20/mes). BGE-Reranker em GPU caem de 5-8s para <1s.
  5. External managed (Voyage, Cohere): viola politica "self-hosted by default".

Decision

  1. Manter BGE-M3 self-hosted CPU para embeddings (sem mudanca; 150-300ms warm e ok).
  2. Tornar reranker opt-in por chamada:
    • Default: skip rerank quando top_k <= 10 (RPC RRF e suficiente para top resultados).
    • Aplicar rerank quando top_k > 10 OU explicit rerank=1 no API.
  3. Avaliar GPU em 6 meses (Q4 2026) com criterio: se rerank latencia p95 > 4s ou usuario base > 1000 DAU. Se ambos, provisionar GPU 1.
  4. ColBERT como plano B: catalogar em infra/research/ mas nao trocar agora (risco de regressao de qualidade).
  5. Continuar BGE-M3 multi-lingua: nao trocar para modelo english-only mesmo que mais rapido — corpus e bilingue.

Consequences

Positivas:

  • Latencia mediana p95 do chat cai de ~10s para ~6s (estimativa baseada em remocao do rerank para top_k <=10).
  • Custo continua $0/mes alem do VPS (sem GPU upgrade).
  • Reranker continua disponivel para queries complexas (top_k=20+).

Negativas:

  • Quando user pede explicitamente "top 20 results", latencia volta a ser 8s.
  • Recall@5 pode cair marginalmente em queries muito ambiguas. Ver eval harness W2.

Trade-off aceito: UX media melhora; UX pior caso mantem. Eval harness do W2 vai pegar regressao real.

Verification

  • tests/rag/golden.yaml mede Recall@5 antes/depois.
  • Sentry timing histogram chat_query_latency_ms p95 antes/depois.
  • Manual smoke test: 5 queries cobrindo cada top_k bucket.

References

  • infra/RETRIEVAL.md (performance budget).
  • web/lib/retrieval/hybrid.ts (codigo).
  • BGE-M3 paper: arxiv:2402.03216.
  • ColBERT-late-interaction: arxiv:2004.12832.