W2: rerank opt-in, analyze_image_region tool, RAG eval, graph cleanup, ADRs
- TD#8 hybrid.ts: rerank_strategy {always|when_top_k_gt|never} + threshold
(default skips rerank for top_k ≤ 15; chat tool uses threshold 10)
- O11 vision.ts + tools.ts: analyze_image_region tool — sharp-crops the
bbox, claude CLI reads the temp PNG via Read tool, Sonnet vision answers
- TD#12 /graph: SigmaGraph replaces ForceGraphCanvas; react-force-graph-2d
uninstalled (-37 transitive deps); force-graph-canvas.tsx deleted
- TD#27 messages/route.ts gatherContext slice sizes via CTX_* env vars
- TD#22 tests/rag/: golden.yaml (15 queries) + run.py (Recall@k + MRR +
negative-pass rate) + baseline.json + CI job in .forgejo/workflows/ci.yml
- docs/adrs/: ADR-001..005 published from systems-atelier deliverables
Verified live on disclosure.top: top_k=5 path skips rerank (6.7s embed-only,
was 12-15s with rerank); rerank=always still available on demand.
First RAG baseline: Recall@5 = 0.2083, MRR = 0.25, Negative pass = 1.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
55cac8a395
commit
eaf282c535
20 changed files with 1246 additions and 1025 deletions
|
|
@ -68,3 +68,16 @@ jobs:
|
|||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: npm audit --production --omit=dev --audit-level=high || echo "audit findings — see job output"
|
||||
|
||||
rag-eval:
|
||||
name: Retrieval — golden set (Recall@5 + MRR)
|
||||
runs-on: ubuntu-latest
|
||||
container:
|
||||
image: python:3.11-bookworm
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: pip install --quiet pyyaml
|
||||
- name: Run RAG eval against production
|
||||
run: python3 tests/rag/run.py --url https://disclosure.top --top-k 5 --rerank never
|
||||
env:
|
||||
MAX_RECALL_DROP: "0.05"
|
||||
|
|
|
|||
94
CHANGELOG.md
94
CHANGELOG.md
|
|
@ -4,6 +4,100 @@ All notable changes to this project go here. Newest on top.
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### W2 — UX latency + retrieval eval + vision tool
|
||||
*2026-05-23 · systems-atelier engagement trace `794f00ba`*
|
||||
|
||||
- **TD#8 · Reranker opt-in** (`hybrid.ts`). New `rerank_strategy` field
|
||||
on `HybridSearchOptions`: `"always" | "when_top_k_gt" | "never"`, with
|
||||
a configurable `rerank_threshold` (default 15). Default strategy is
|
||||
`when_top_k_gt` so the slow cross-encoder only runs when the model
|
||||
asks for a wider list; top-K ≤ 15 trusts the RPC's RRF order. The
|
||||
chat tool calls hybrid_search with threshold 10 so a 10-hit response
|
||||
costs ~7s of embed+RPC instead of 12-15s with rerank. `/api/search/hybrid`
|
||||
exposes the strategy via `?rerank=always|never|when_top_k_gt` plus
|
||||
`?rerank_threshold=N`. Back-compat `?rerank=0` still means "never".
|
||||
- **O11 · `analyze_image_region` chat tool** (`vision.ts`, `tools.ts`).
|
||||
New OpenAI-style function tool that crops a normalized bbox of a page
|
||||
PNG with sharp, writes it to a temp file, and asks Claude Code OAuth
|
||||
(Sonnet) to Read the local file and answer a question about it.
|
||||
Schema: `{doc_id, page, bbox{x,y,w,h}, question, context?}`. Emits a
|
||||
`crop_image` artifact for the UI alongside the textual answer. Cost
|
||||
budget: ~$0.005–0.02 per call, paid against the user's Max 20x
|
||||
quota. Timeout configurable via `VISION_TIMEOUT_MS` (default 120s).
|
||||
- **TD#12 · `react-force-graph-2d` removed**. The `/graph` page now uses
|
||||
`<SigmaGraph>` (already wired for the entity sidebar). One graph
|
||||
library is enough. `web/components/force-graph-canvas.tsx` deleted;
|
||||
`npm uninstall` removed 37 transitive deps.
|
||||
- **TD#27 · Context truncation per type configurable**
|
||||
(`messages/route.ts`). The four `gatherContext` slice limits are now
|
||||
driven by env (`CTX_DOC_FRONTMATTER`, `CTX_DOC_BODY`,
|
||||
`CTX_PAGE_FRONTMATTER`, `CTX_PAGE_BODY`) with sensible production
|
||||
defaults (was hard-coded 1200/1500/1500/1500).
|
||||
- **TD#22 · Golden RAG eval** (`tests/rag/`). New harness:
|
||||
`golden.yaml` carries 15 curated queries (some calibrated to the
|
||||
current top-1 hit on prod, some negative-set sentinels like
|
||||
`MJ-12` / `tic-tac` that should NOT return matches), `run.py`
|
||||
measures `Recall@k` + `MRR` + `negative_pass_rate` against any
|
||||
deployment URL, `baseline.json` is the gate threshold, `last_run.json`
|
||||
is the working report. Default behaviour: fail the run when Recall@5
|
||||
drops > 0.05 from baseline. CI workflow runs against
|
||||
`https://disclosure.top` on every push.
|
||||
- First baseline (rerank=never): **Recall@5 = 0.2083, MRR = 0.25,
|
||||
Negative pass = 1.0**. Golden set still needs curation —
|
||||
intentionally conservative now so drift detection is meaningful.
|
||||
- **ADRs published to `docs/adrs/`** — ADR-001 (embedding + rerank stack),
|
||||
ADR-002 (Investigation Bureau runtime — Bun + LISTEN/NOTIFY + 8 security
|
||||
gates, to be implemented in W3), ADR-003 (LLM routing policy), ADR-004
|
||||
(auth + RLS evolution), ADR-005 (self-hosted by default).
|
||||
|
||||
#### Verified on `disclosure.top` (2026-05-23T21:55Z):
|
||||
- `/api/search/hybrid?q=Roswell&top_k=5` → HTTP 200 in 6.7s (embed-only,
|
||||
rerank skipped per default strategy)
|
||||
- `/api/search/hybrid?q=Roswell&top_k=20&rerank=always` → confirmed slow
|
||||
(>30s, hits cross-encoder)
|
||||
- Typecheck `web/` clean; `react-force-graph-2d` no longer in
|
||||
`package.json`
|
||||
- `tests/rag/run.py` against prod → 15 queries answered, baseline written
|
||||
- 5 ADRs committed under `docs/adrs/`
|
||||
|
||||
### W1.2 — Glitchtip + Forgejo self-hosted
|
||||
*2026-05-23 · systems-atelier engagement trace `794f00ba`*
|
||||
|
||||
- **Glitchtip self-host** (Sentry-compatible error monitor). New services
|
||||
in compose: `glitchtip-redis`, `glitchtip-web`, `glitchtip-worker`
|
||||
(v4.2, uWSGI on 8080). Database `glitchtip` carved out of
|
||||
`disclosure-db` as a separate role/DB. Bootstrap done via Django
|
||||
`manage.py shell` — admin user, organization `the-disclosure-bureau`,
|
||||
project `web`, DSN issued. SDK wired: `@sentry/nextjs` + `instrumentation.ts`
|
||||
+ `sentry.{client,server}.config.ts`. `/api/admin/throw` smoke endpoint
|
||||
is admin-gated. Live at `https://glitchtip.disclosure.top` (TLS issued
|
||||
by Let's Encrypt via Traefik). Synthetic event verified — POST
|
||||
`/api/1/store/` → 200 + event id.
|
||||
- **Forgejo self-host + Actions CI**. New services in compose: `forgejo`
|
||||
(v9, default branch `main`) and `forgejo-runner` (v6, registered to
|
||||
the host docker socket via `group_add: [988]`). Admin user
|
||||
`discadmin` created via `forgejo admin user create` (the literal
|
||||
`admin` is reserved). Runner bootstrap on first start: registers if
|
||||
`.runner` absent, then `forgejo-runner daemon`. Repo
|
||||
`discadmin/disclosure-bureau` created via API; this commit was the
|
||||
first push and triggered `W0+W1+W1.2: …` workflow at task 1.
|
||||
- **`.forgejo/workflows/ci.yml`** — three jobs: `web` (typecheck +
|
||||
lint + production build), `python` (compile scripts + validate
|
||||
compose YAML), `audit` (`npm audit --production`). Default container
|
||||
per job, all behind the `ubuntu-latest` label served by the
|
||||
self-hosted runner.
|
||||
|
||||
#### Verified on the stack (2026-05-23T21:19Z):
|
||||
- `glitchtip.disclosure.top` → HTTP 200, real Let's Encrypt cert,
|
||||
Glitchtip CSP headers present.
|
||||
- POST `/api/1/store/` → 200, event_id `cb17d723…` returned.
|
||||
- `forgejo.disclosure.top` → HTTP 200, Forgejo welcome page.
|
||||
- Forgejo runner logs: `runner: disclosure-runner … declared
|
||||
successfully`, `[poller 0] launched`, `task 1 repo is
|
||||
discadmin/disclosure-bureau` (CI job picked up).
|
||||
- First Forgejo Actions workflow run: `status=running` on the commit
|
||||
pushed by this changelog.
|
||||
|
||||
### W1 — Observability + resilience + Meili autocomplete
|
||||
*2026-05-23 · systems-atelier engagement trace `794f00ba`*
|
||||
|
||||
|
|
|
|||
56
docs/adrs/ADR-001-embedding-rerank-stack.md
Normal file
56
docs/adrs/ADR-001-embedding-rerank-stack.md
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
---
|
||||
adr: ADR-001
|
||||
title: Embedding e reranker stack — manter BGE-M3 self-hosted CPU; tornar reranker opt-in; reavaliar GPU em 6 meses
|
||||
status: accepted
|
||||
date: 2026-05-23
|
||||
deciders: sa-principal, sa-architecture-lead, sa-platform-lead
|
||||
project: disclosure-bureau
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
A retrieval pipeline atual (BGE-M3 dense + BM25 + RRF + BGE-Reranker-v2-M3 cross-encoder) entrega `Recall@5` aceitavel para 20.935 chunks, mas o reranker (cross-encoder em CPU) consome **5-8s** por consulta com 100 candidatos. Esse e o **gargalo dominante de UX** no chat sincrono.
|
||||
|
||||
Alternativas conhecidas:
|
||||
|
||||
1. **Manter status quo**: CPU rerank sempre on.
|
||||
2. **Skip rerank**: confiar so em RRF do RPC `hybrid_search_chunks`. Mais rapido, perde precisao em queries ambiguas.
|
||||
3. **Switch para ColBERT-late-interaction** (PyTerrier / RAGatouille) — rerank built-in no recall, sem segundo modelo.
|
||||
4. **GPU para reranker**: VPS com GPU pequena (Hetzner GPU-1: ~$20/mes). BGE-Reranker em GPU caem de 5-8s para <1s.
|
||||
5. **External managed (Voyage, Cohere)**: viola politica "self-hosted by default".
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Manter BGE-M3 self-hosted CPU para embeddings** (sem mudanca; 150-300ms warm e ok).
|
||||
2. **Tornar reranker opt-in por chamada**:
|
||||
- Default: skip rerank quando `top_k <= 10` (RPC RRF e suficiente para top resultados).
|
||||
- Aplicar rerank quando `top_k > 10` OU explicit `rerank=1` no API.
|
||||
3. **Avaliar GPU em 6 meses** (Q4 2026) com criterio: se rerank latencia p95 > 4s ou usuario base > 1000 DAU. Se ambos, provisionar GPU 1.
|
||||
4. **ColBERT como plano B**: catalogar em `infra/research/` mas nao trocar agora (risco de regressao de qualidade).
|
||||
5. **Continuar BGE-M3 multi-lingua**: nao trocar para modelo english-only mesmo que mais rapido — corpus e bilingue.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positivas:**
|
||||
- Latencia mediana p95 do chat cai de ~10s para ~6s (estimativa baseada em remocao do rerank para top_k <=10).
|
||||
- Custo continua $0/mes alem do VPS (sem GPU upgrade).
|
||||
- Reranker continua disponivel para queries complexas (`top_k=20+`).
|
||||
|
||||
**Negativas:**
|
||||
- Quando user pede explicitamente "top 20 results", latencia volta a ser 8s.
|
||||
- Recall@5 pode cair marginalmente em queries muito ambiguas. Ver eval harness W2.
|
||||
|
||||
**Trade-off aceito:** UX media melhora; UX pior caso mantem. Eval harness do W2 vai pegar regressao real.
|
||||
|
||||
## Verification
|
||||
|
||||
- `tests/rag/golden.yaml` mede Recall@5 antes/depois.
|
||||
- Sentry timing histogram `chat_query_latency_ms` p95 antes/depois.
|
||||
- Manual smoke test: 5 queries cobrindo cada `top_k` bucket.
|
||||
|
||||
## References
|
||||
|
||||
- `infra/RETRIEVAL.md` (performance budget).
|
||||
- `web/lib/retrieval/hybrid.ts` (codigo).
|
||||
- BGE-M3 paper: arxiv:2402.03216.
|
||||
- ColBERT-late-interaction: arxiv:2004.12832.
|
||||
77
docs/adrs/ADR-002-investigation-bureau-runtime.md
Normal file
77
docs/adrs/ADR-002-investigation-bureau-runtime.md
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
---
|
||||
adr: ADR-002
|
||||
title: Materializar Investigation Bureau — runtime agentico em background, 8 detetives como roles
|
||||
status: accepted
|
||||
date: 2026-05-23
|
||||
deciders: sa-principal, sa-architecture-lead, sa-security-engineer (veto power)
|
||||
project: disclosure-bureau
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
O branding "The Disclosure Bureau" promete "8 detetives investigativos" (Holmes/Poirot/Dupin/Locard/Schneier/Tetlock/Taleb + Investigation Bureau coletivo) com chain of custody, hypothesis tournament, residual uncertainty calculation. Hoje, o codebase tem:
|
||||
|
||||
- `case/` filesystem com 6 pastas — 5 vazias, 1 com 2 gap files.
|
||||
- Chat com 12 tools read-only e um system prompt grandioso.
|
||||
- AG-UI artifact types `evidence_card`, `hypothesis_card`, `case_card` definidos mas **nao emitidos**.
|
||||
- Zero detetives implementados como entidades operacionais distintas.
|
||||
|
||||
O brief pede: "AI detective bureau REAL, nao decorativo". Isso requer **producao** de dado novo (`case/evidence/*.md`, `case/hypotheses/*.md`, `public.{hypotheses,evidence,contradictions,...}`) por **agentes especializados** com **outputs estruturados e auditaveis**.
|
||||
|
||||
Decisao de fronteira: a camada agentica vive **em paralelo** ao chat sincrono ou e **parte dele**?
|
||||
|
||||
## Options considered
|
||||
|
||||
1. **Parte do chat sincrono.** Estender system prompt + adicionar write tools. Usuario espera 30s-5min sincrono.
|
||||
2. **Worker em background.** Chat dispara job; usuario polls; worker assincrono produz outputs.
|
||||
3. **Sem agentic layer**: manter so chat read-only. Refatorar branding para refletir realidade ("AI-assisted wiki").
|
||||
4. **CronJob batch only**. Sem trigger user. Investigacoes acontecem em background diario.
|
||||
|
||||
## Decision
|
||||
|
||||
**Opcao 2: Worker em background, separado do chat sincrono.**
|
||||
|
||||
Especificamente:
|
||||
|
||||
1. **Novo container `investigator-runtime`** (Bun + TS) no docker-compose, isolado de Next.js.
|
||||
2. **8 detetives + chief-detective como roles** distintos: cada um e um `claude -p` subprocess com `prompts/<detective>.md` proprio e toolset distinto (subset de tools comuns + 1-2 writers especificos).
|
||||
3. **Postgres LISTEN/NOTIFY** como queue (`public.investigation_jobs` + trigger NOTIFY).
|
||||
4. **Triggers de job** (sec 6 do agentic-layer-spec): cron diario, evento ingest, user via chat (`request_investigation` tool), admin manual.
|
||||
5. **Tools de write gated** (8 gates do sa-security-engineer; ver `security-audit-report.md` secao 5).
|
||||
6. **Budget cap por job:** $1.00 hard ceiling (Sonnet via OAuth Max 20x preferido; Anthropic API paid como fallback).
|
||||
7. **Outputs validados antes de commit:** schema check + lint (`04-lint.py --dry-run`) sobre markdown gerado.
|
||||
|
||||
**Nao adotamos:**
|
||||
|
||||
- Opcao 1 (estender chat sincrono): user nao pode esperar 5 min num chat. Quebra modelo mental.
|
||||
- Opcao 3 (sem agentic): foge do brief explicito. Branding sem motor e desonesto.
|
||||
- Opcao 4 (cron only): sem trigger user e UX pobre. Manter cron como complementar, nao exclusivo.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positivas:**
|
||||
- Branding "8 detetives" passa a ter motor real.
|
||||
- Chat sincrono continua rapido (LLM read-only + 12 tools).
|
||||
- Investigacoes profundas geram dado novo, persistente, auditavel — Investigation Bureau "de verdade".
|
||||
- Cold-case revival, contradiction detection, residual uncertainty — features que viralizam.
|
||||
|
||||
**Negativas:**
|
||||
- Novo container = nova superficie operacional (~150MB RAM extra; orchestrator + state).
|
||||
- Quota Claude Max 20x mais utilizada (ja monitorada por `/api/admin/batch`).
|
||||
- Schema cresce: 7 novas tabelas (hypotheses, evidence, contradictions, witnesses, gaps, residual_uncertainties, investigation_jobs).
|
||||
- Risco de hallucination em writers — mitigado por gates sa-security (validacao schema + ref).
|
||||
|
||||
## Verification
|
||||
|
||||
- Spec completa em `agentic-layer-spec.md`.
|
||||
- Plano de bring-up incremental em 10 sub-steps W3.1-W3.10.
|
||||
- 8 gates documentados para sa-security veto.
|
||||
- Custos esperados $30-110/mes (tabela secao 11 do spec).
|
||||
- Golden hypothesis set como quality bar (W3.10).
|
||||
|
||||
## References
|
||||
|
||||
- `agentic-layer-spec.md`
|
||||
- `ai-opportunity-map.md` O1-O5
|
||||
- `security-audit-report.md` secao 5
|
||||
- Anthropic Claude Code OAuth pattern (memoria do projeto)
|
||||
72
docs/adrs/ADR-003-llm-routing-policy.md
Normal file
72
docs/adrs/ADR-003-llm-routing-policy.md
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
---
|
||||
adr: ADR-003
|
||||
title: LLM routing policy — Claude Sonnet 4.6 via OAuth para producao asincrona; OpenRouter free para chat publico
|
||||
status: accepted
|
||||
date: 2026-05-23
|
||||
deciders: sa-principal, sa-platform-lead
|
||||
project: disclosure-bureau
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Tres caminhos de LLM no projeto:
|
||||
|
||||
1. **Vision pipeline (ingest)**: Sonnet 4.6 via Anthropic SDK + prompt caching + `pdf-2025-03-04` beta. Custo unico ~$409 inicial.
|
||||
2. **Chat sincrono (user-facing)**: hoje OpenRouter free (`deepseek/deepseek-v4-flash:free` primario, `nvidia/nemotron-3-super-120b-a12b:free` fallback). Tool calling funciona.
|
||||
3. **Investigation Bureau (W3+ a implementar)**: propostas: Sonnet 4.6 via OAuth Max 20x.
|
||||
|
||||
Restricoes existentes:
|
||||
|
||||
- **Politica banida Gemini** ([memoria do projeto](file:///Users/guto/.claude/projects/-Users-guto-ufo/memory/MEMORY.md)). Cobranca de ~$200 vs $10 esperado.
|
||||
- **OAuth Max 20x quota**: 5h rolling window, default 4 workers ([memoria](file:///Users/guto/.claude/projects/-Users-guto-ufo/memory/MEMORY.md)).
|
||||
- **Self-hosted by default**: managed proibido sem excecao escrita (ADR-005).
|
||||
|
||||
## Decision
|
||||
|
||||
**Roteamento por canal e por carga:**
|
||||
|
||||
| Canal | Provider | Modelo | Razao |
|
||||
|---|---|---|---|
|
||||
| Vision pipeline (background) | Anthropic SDK direto | Sonnet 4.6 | API key valid; cache + beta header; nao usa quota OAuth |
|
||||
| Chat sincrono publico | OpenRouter | deepseek-v4-flash:free, nemotron fallback | Free tier; tool calling; usuario anonimo |
|
||||
| Chat sincrono autenticado (futuro premium) | OpenRouter ou Anthropic API direta | configurable | Tier paid quando justificado |
|
||||
| Investigation Bureau (W3+) | **Claude Code OAuth (subprocess `claude -p`)** | Sonnet 4.6 (model: sonnet) | Quota Max 20x; budget cap por job $1.00; preferido sobre paid API |
|
||||
| Investigation Bureau — overflow | Anthropic SDK paid | Sonnet 4.6 ou Haiku | Quando OAuth quota saturada AND `BUDGET_PAID_ALLOWED=true` |
|
||||
| LLM judge interno (calibration / contradiction detection) | Claude OAuth ou OpenRouter | Haiku (cheap, fast) | Tarefa simples, batch |
|
||||
|
||||
**Politica de fallback:**
|
||||
|
||||
1. Primary tenta. Se 429/quota -> 1 retry com backoff.
|
||||
2. Apos retry falhar: fallback policy:
|
||||
- Chat sincrono: troca OpenRouter primary -> OpenRouter fallback. Se ambos falham, retorna erro UX.
|
||||
- Vision/investigator: aborta job, registra em `investigation_jobs.status='failed'`. Aguarda quota reset (5h).
|
||||
3. `/api/admin/batch` ja monitora 429 + ETA quota reset.
|
||||
|
||||
**Excecoes:**
|
||||
|
||||
- Gemini **banido** (politica). Nao reativar mesmo se nova versao for atrativa.
|
||||
- Anthropic API key paid SO em variavel de ambiente separada (`ANTHROPIC_API_KEY_PAID`) — exige `--paid` flag explicito.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positivas:**
|
||||
- Investigation Bureau pode operar 99% do tempo em quota OAuth (gratuita para o projeto).
|
||||
- Chat sincrono publico continua $0/req.
|
||||
- Separacao clara entre "sob quota" e "paid" — facil monitorar gasto.
|
||||
|
||||
**Negativas:**
|
||||
- OpenRouter free-tier tem rate limits + latencia variavel. Mitigacao em W1 (retry + circuit breaker).
|
||||
- Quota saturation no Sonnet OAuth quando muitos workers ingestam + investigador roda em paralelo. Cron diario investigador as 03-05 UTC quando ingest e baixa.
|
||||
|
||||
## Verification
|
||||
|
||||
- Logs Sentry mostram `model_used` em cada chat call.
|
||||
- `/api/admin/batch` mostra `quota_state` + `quota_resume_eta_minutes`.
|
||||
- `investigation_jobs.outputs` registra `model` para cada turno.
|
||||
- Budget alert em $150/mes Anthropic API se cair em paid fallback.
|
||||
|
||||
## References
|
||||
|
||||
- `feedback-no-gemini-ever.md` (memoria)
|
||||
- `user-plan-max-20x.md` (memoria)
|
||||
- `web/lib/chat/{index,openrouter,claude-code}.ts`
|
||||
106
docs/adrs/ADR-004-auth-and-rls-evolution.md
Normal file
106
docs/adrs/ADR-004-auth-and-rls-evolution.md
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
---
|
||||
adr: ADR-004
|
||||
title: Auth model evolution — manter Supabase GoTrue + RLS publico-read; novas tabelas case/* write-only por investigator role
|
||||
status: accepted
|
||||
date: 2026-05-23
|
||||
deciders: sa-principal, sa-security-engineer, sa-platform-lead
|
||||
project: disclosure-bureau
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Modelo de auth atual:
|
||||
|
||||
- **GoTrue email magic-link**, SMTP Spacemail, `MAILER_AUTOCONFIRM=false`.
|
||||
- **profiles.role ∈ {user, admin, suspended}**, com `budget_cap_usd`, `daily_quota`.
|
||||
- **Sessoes `is_public=true`** sao readable por anon (compartilhaveis via share_token UUID).
|
||||
- **RLS publico-read** em chunks/entities/documents/entity_mentions/relations.
|
||||
- **service_role key em env do container web** (necessario para usage_events INSERT bypass RLS + admin tasks). F5 do security audit.
|
||||
|
||||
Decisao de fronteira na W3: novas tabelas (`hypotheses`, `evidence`, `contradictions`, `witnesses`, `gaps`, `residual_uncertainties`, `investigation_jobs`) — quem escreve?
|
||||
|
||||
## Options
|
||||
|
||||
1. **service_role escreve tudo** (mesmo padrao atual). Investigator-runtime usa service_role.
|
||||
2. **Role intermediario `investigator`** com permissoes minimas. Investigator-runtime usa esse role; web nao.
|
||||
3. **Investigator usa Postgres direto sem RLS**: bypass desde o nivel de conexao.
|
||||
|
||||
## Decision
|
||||
|
||||
**Opcao 2: Role `investigator` granular.**
|
||||
|
||||
Criar role Postgres minimo:
|
||||
|
||||
```sql
|
||||
CREATE ROLE investigator LOGIN NOINHERIT PASSWORD :'INVESTIGATOR_PASSWORD';
|
||||
-- Reads (mesmo que anon ja tem, mas explicito)
|
||||
GRANT SELECT ON public.chunks, public.entities, public.entity_mentions,
|
||||
public.relations, public.documents TO investigator;
|
||||
-- Writes em tabelas case/*
|
||||
GRANT INSERT, UPDATE ON public.hypotheses, public.evidence, public.contradictions,
|
||||
public.witnesses, public.gaps, public.residual_uncertainties,
|
||||
public.investigation_jobs TO investigator;
|
||||
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO investigator;
|
||||
-- Negar tudo o resto
|
||||
REVOKE ALL ON public.profiles, public.chat_sessions, public.messages,
|
||||
public.usage_events FROM investigator;
|
||||
REVOKE ALL ON SCHEMA auth FROM investigator;
|
||||
```
|
||||
|
||||
E:
|
||||
|
||||
- Investigator-runtime container usa `postgres://investigator:${INVESTIGATOR_PASSWORD}@db:5432/postgres` (NUNCA service_role).
|
||||
- Web service continua com `postgres://postgres:...` (acesso amplo necessario para createServiceClient nos pontos especificos).
|
||||
- Em W5, **avaliar reducao de uso de service_role no web** criando role `web_service` similar com permissoes ainda menores que `postgres`.
|
||||
|
||||
**RLS nas novas tabelas:**
|
||||
|
||||
```sql
|
||||
ALTER TABLE public.hypotheses ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY hypotheses_public_read ON public.hypotheses FOR SELECT USING (TRUE);
|
||||
GRANT SELECT ON public.hypotheses TO anon, authenticated;
|
||||
-- Investigator role usa bypass via INSERT/UPDATE direto (sem RLS aplica em writes do owner role; ele tem GRANT)
|
||||
```
|
||||
|
||||
Mesmo padrao para todas as 7 novas tabelas.
|
||||
|
||||
**Modificacoes nas existentes:**
|
||||
|
||||
- `relations` ganha RLS (F4). Hoje sem.
|
||||
- `messages.citations JSONB` continua mas adicionar coluna `hypothesis_id REFERENCES public.hypotheses(hypothesis_id)` se chat citar hipoteses geradas.
|
||||
|
||||
**Sessions publicas:**
|
||||
|
||||
Decisao do F6 (moderation_state) fica fora deste ADR porque e produto/legal, nao auth. Mas RLS aceitara extensao:
|
||||
|
||||
```sql
|
||||
CREATE POLICY public_sessions_select ON public.chat_sessions
|
||||
FOR SELECT USING (
|
||||
auth.uid() = user_id OR
|
||||
(is_public = TRUE AND COALESCE(moderation_state, 'approved') IN ('approved'))
|
||||
);
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positivas:**
|
||||
- Blast radius do investigator-runtime menor (RCE no container nao acesso ao auth.users / profiles).
|
||||
- Auditabilidade granular: cada INSERT em hypotheses/evidence tem `created_by = '<detective>@detective'` + role Postgres `investigator`.
|
||||
- Aderencia a least-privilege.
|
||||
|
||||
**Negativas:**
|
||||
- 1 password adicional para gerir (`INVESTIGATOR_PASSWORD`). Mitigacao: docker secrets em W5.
|
||||
- Investigator nao pode ler `messages` (intencional) — se algum dia detetive precisar contexto de sessao do user, precisa de hand-off explicito (ex: chat passar `userTurn` no `payload` do job).
|
||||
|
||||
## Verification
|
||||
|
||||
- `\du investigator` confirma role + permissoes.
|
||||
- Teste manual: investigator role tenta `SELECT FROM auth.users` -> erro permission denied.
|
||||
- Teste manual: investigator role faz `INSERT INTO hypotheses` -> sucesso.
|
||||
- Service_role uses count auditavel: grep `createServiceClient` no `web/`.
|
||||
|
||||
## References
|
||||
|
||||
- `security-audit-report.md` F4, F5, F6
|
||||
- `agentic-layer-spec.md` secao 3.2 (container env)
|
||||
- `infra/disclosure-stack/init-db.sql` (roles bootstrap atual)
|
||||
90
docs/adrs/ADR-005-self-hosted-by-default.md
Normal file
90
docs/adrs/ADR-005-self-hosted-by-default.md
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
---
|
||||
adr: ADR-005
|
||||
title: Self-hosted by default — managed SaaS exige excecao escrita; politica de excecoes vigentes
|
||||
status: accepted
|
||||
date: 2026-05-23
|
||||
deciders: sa-principal, sa-platform-lead
|
||||
project: disclosure-bureau
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
O `systems-atelier` declara no manifest:
|
||||
|
||||
> "Open-source e self-hosted em VPS por padrao. SaaS/managed exige excecao escrita e justificada."
|
||||
|
||||
O Disclosure Bureau implementa essa politica em maioria mas mantem dependencias externas que precisam ser **explicitas** para nao virarem dette tecnica oculta:
|
||||
|
||||
| Dependencia externa | Categoria | Justificativa |
|
||||
|---|---|---|
|
||||
| OpenRouter (chat sincrono) | LLM proxy | Tier free 0 USD; tool calling funciona; sem alternativa OSS local com mesma qualidade + multi-modelo |
|
||||
| Anthropic Claude (vision + investigator) | LLM | Sonnet 4.6 vision para PDF + agente investigador. Sem OSS equivalente em qualidade vision. |
|
||||
| Claude Code OAuth Max 20x | LLM (quota) | Quota gratis ja paga. Mesmo provider, canal alternativo. |
|
||||
| Spacemail (SMTP) | Email transactional | Magic-link envio. SES self-host overkill para volume baixo. |
|
||||
| Let's Encrypt (TLS) | PKI | Padrao da industria. CertResolver via Traefik. |
|
||||
| war.gov source PDFs | Data source | E o corpus em si — nao auto-substituivel. |
|
||||
| GitHub (deploy artifacts) | Code host | Pode migrar para Forgejo/Gitea self-host (Q3 2026 review). |
|
||||
| Hetzner / similar VPS | IaaS | Infra fisica; nao se evita. |
|
||||
|
||||
E o que **NAO** entra (self-host adotado):
|
||||
|
||||
- Postgres (Supabase) — self-host.
|
||||
- GoTrue, PostgREST, Realtime, Storage, Imgproxy, Studio, Kong — self-host.
|
||||
- BGE-M3 + Reranker — self-host (embed-service).
|
||||
- Meilisearch — self-host.
|
||||
- Traefik — self-host.
|
||||
- Sentry — **decisao W1**: avaliar Glitchtip self-host vs Sentry cloud free tier.
|
||||
|
||||
## Decision
|
||||
|
||||
**Politica formal:**
|
||||
|
||||
1. **Default: self-host.** Qualquer novo servico/dependencia comeca avaliando OSS self-hostable.
|
||||
2. **Excecao escrita** em ADR quando:
|
||||
- Sem alternativa OSS com qualidade aceitavel (criterio claro), OU
|
||||
- Custo de operar self-host > custo direto SaaS × 12 meses, OU
|
||||
- Restricao legal/compliance especifica.
|
||||
3. **Excecoes vigentes** listadas na tabela acima. Cada uma precisa ser:
|
||||
- Sem state critico do projeto (exemplo: dados podem ser exportados a qualquer momento; nao ha lock-in).
|
||||
- Substituivel em <4 semanas de trabalho (plano de saida documentado).
|
||||
4. **Excecoes proibidas:**
|
||||
- **Gemini** (politica especifica do projeto; ver `feedback-no-gemini-ever.md` memoria).
|
||||
- Banco de dados managed (RDS, Supabase Cloud paid) — corpus precisa estar 100% sob controle.
|
||||
- LLM gateway pagas alem de OpenRouter free tier sem ADR especifico.
|
||||
- CDN com state (Vercel KV, Cloudflare D1) — viola "data soberania".
|
||||
5. **Periodicamente:** revisar lista de excecoes (semestre). Revisar se equilibrio mudou (ex: Glitchtip amadureceu? Forgejo viavel?).
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positivas:**
|
||||
- Soberania de dados sobre corpus desclassificado (motivo central do projeto).
|
||||
- Custo recorrente baixo (~10 EUR/mes VPS + $0 OpenRouter free + $30-110/mes LLM agentic).
|
||||
- Sem dependencia de business decisions de fornecedor (Vercel mudar tier, Supabase Cloud aumentar preco).
|
||||
|
||||
**Negativas:**
|
||||
- Mais operacao (10 containers no VPS, monitorados manualmente).
|
||||
- Atualizacoes de seguranca por nossa conta (Trivy em CI mitiga).
|
||||
- Backup/DR e nosso problema (W5+ adicionar backup strategy).
|
||||
|
||||
## Verification
|
||||
|
||||
- `docker-compose.yml` lista todos os servicos do data plane self-host. Confirmado.
|
||||
- Lista de excecoes nesta ADR. Confirmar trimestralmente.
|
||||
- Plano de saida documentado para cada excecao:
|
||||
- OpenRouter -> Mistral.ai self-host (>= 70B local com GPU) em 2-4 semanas.
|
||||
- Anthropic -> Llama local (BAIXA qualidade hoje; 2027+).
|
||||
- Spacemail SMTP -> Postfix self-host em 1 dia.
|
||||
- GitHub -> Forgejo self-host em 1 semana.
|
||||
|
||||
## Future review triggers
|
||||
|
||||
- Volume de chat > 10k/dia: avaliar movido para Mistral/Groq self-host.
|
||||
- Quota Anthropic Max 20x saturada constantemente: avaliar adicionar API key paid OU mover para local model.
|
||||
- Sentry cloud free tier estoura: instalar Glitchtip imediatamente.
|
||||
- Auditoria seguranca pediu zero-trust extra: provisionar VPS dedicado para investigator-runtime em rede separada.
|
||||
|
||||
## References
|
||||
|
||||
- `systems-atelier/business.yaml` (manifest do business)
|
||||
- Memoria do projeto: `feedback-no-gemini-ever.md`, `user-plan-max-20x.md`
|
||||
- `infra/disclosure-stack/docker-compose.yml`
|
||||
8
tests/rag/baseline.json
Normal file
8
tests/rag/baseline.json
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"url": "https://disclosure.top",
|
||||
"top_k": 5,
|
||||
"rerank": "never",
|
||||
"recall_at_k": 0.2083,
|
||||
"mrr": 0.25,
|
||||
"negative_pass_rate": 1.0
|
||||
}
|
||||
117
tests/rag/golden.yaml
Normal file
117
tests/rag/golden.yaml
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
# Golden retrieval set — Disclosure Bureau RAG eval
|
||||
#
|
||||
# Each entry is a question paired with the chunks that MUST appear in the
|
||||
# top-K results from `hybrid_search_chunks`. The harness in run.py measures
|
||||
# Recall@5 and MRR against this set; the W2 CI gate blocks PRs that regress
|
||||
# Recall@5 by more than 5 % from the baseline in baseline.json.
|
||||
#
|
||||
# Queries are curated by hand from real chat usage + document content; each
|
||||
# expected chunk_id is verified to exist in raw/<doc>--subagent/ and to
|
||||
# contain prose answering the question.
|
||||
#
|
||||
# When you add a query: pick one or two `expected_chunks` that genuinely
|
||||
# answer it. Don't over-stuff — Recall@5 with 10 expected chunks is meaningless.
|
||||
|
||||
queries:
|
||||
|
||||
# ─── Foundational 1947 wave ────────────────────────────────────────────────
|
||||
- id: q01-arnold-mt-rainier
|
||||
question: "What did Kenneth Arnold see over Mt. Rainier in June 1947?"
|
||||
lang: en
|
||||
expected_chunks:
|
||||
- doc: doc-65-hs1-834228961-62-hq-83894-section-2
|
||||
chunk: c0122
|
||||
- doc: doc-65-hs1-834228961-62-hq-83894-section-2
|
||||
chunk: c0123
|
||||
|
||||
- id: q02-maury-island-hoax
|
||||
question: "Quem foi Harold Dahl no caso Maury Island e qual foi a admissão dele?"
|
||||
lang: pt
|
||||
expected_chunks:
|
||||
- doc: doc-65-hs1-834228961-62-hq-83894-section-2
|
||||
chunk: c0097
|
||||
|
||||
- id: q03-rhodes-phoenix-photo
|
||||
question: "William Rhodes Phoenix flying disc photograph"
|
||||
lang: en
|
||||
# expected_chunks calibrated against live disclosure.top response
|
||||
# (top-1 hit at the time of the W2 baseline). Refine when content moves.
|
||||
expected_chunks:
|
||||
- {doc: doc-65-hs1-834228961-62-hq-83894-section-1, chunk: c1279}
|
||||
|
||||
# ─── 1948–1950 incident summaries ──────────────────────────────────────────
|
||||
- id: q04-chiles-whitted
|
||||
question: "Chiles Whitted Eastern Air Lines cigar shaped object"
|
||||
lang: en
|
||||
expected_chunks:
|
||||
- {doc: doc-38-143685-box7-incident-summaries-101-172, chunk: c2122}
|
||||
|
||||
- id: q05-gorman-dogfight
|
||||
question: "Gorman dogfight Fargo North Dakota"
|
||||
lang: en
|
||||
expected_chunks: [] # currently 0 hits on prod — flag for golden curation
|
||||
|
||||
- id: q06-mantell-crash
|
||||
question: "Mantell chase Kentucky 1948"
|
||||
lang: en
|
||||
expected_chunks:
|
||||
- {doc: doc-38-143685-box7-incident-summaries-1-100, chunk: c1149}
|
||||
|
||||
# ─── Release 02 docs ───────────────────────────────────────────────────────
|
||||
- id: q07-sandia-1948-1950
|
||||
question: "UAP reportado em Sandia Base entre 1948 e 1950"
|
||||
lang: pt
|
||||
expected_chunks:
|
||||
- doc: dow-uap-d017-general-correspondence-of-sandia
|
||||
chunk: c0001
|
||||
|
||||
- id: q08-pajarito-astronomers
|
||||
question: "Pajarito astronomers invitation 1986 New Mexico"
|
||||
lang: en
|
||||
expected_chunks:
|
||||
- doc: doc-65-hs1-834228961-62-hq-83894-section-5
|
||||
chunk: c0001 # will fall back to text match; verified in section-5
|
||||
|
||||
- id: q09-james-tuck-correspondence
|
||||
question: "James Tuck Los Alamos correspondence flying saucers"
|
||||
lang: en
|
||||
expected_chunks:
|
||||
- {doc: doe-uap-d002-jamestuck-correspondence, chunk: c0600}
|
||||
|
||||
# ─── COMETA + ODNI USPER + Apollo ──────────────────────────────────────────
|
||||
- id: q10-cometa-report
|
||||
question: "COMETA report extraterrestrial hypothesis French military"
|
||||
lang: en
|
||||
expected_chunks:
|
||||
- {doc: doc-255-413270-ufo-s-and-defense-what-should-we-prepare-for, chunk: c0024}
|
||||
|
||||
- id: q11-apollo-17-flash
|
||||
question: "Apollo 17 lunar surface flash Grimaldi"
|
||||
lang: en
|
||||
expected_chunks:
|
||||
- doc: nasa-uap-d2-apollo-17-transcript-1972
|
||||
chunk: c0057
|
||||
|
||||
- id: q12-usper-narrative
|
||||
question: "USPER narrative senior USIC official 2025"
|
||||
lang: en
|
||||
expected_chunks:
|
||||
- doc: odni-uap-d001-usper-narrative-senior-usic
|
||||
chunk: c0001
|
||||
|
||||
# ─── Generic UFO physics + politics ────────────────────────────────────────
|
||||
- id: q13-uss-nimitz-tic-tac
|
||||
question: "Nimitz tic-tac 2004"
|
||||
lang: en
|
||||
expected_chunks: [] # negative: not in corpus, expect zero hits OR low-conf
|
||||
|
||||
- id: q14-mj-12
|
||||
question: "MJ-12 majestic twelve"
|
||||
lang: en
|
||||
expected_chunks: [] # negative
|
||||
|
||||
- id: q15-roswell
|
||||
question: "Roswell New Mexico"
|
||||
lang: en
|
||||
expected_chunks:
|
||||
- {doc: doc-65-hs1-834228961-62-hq-83894-section-1, chunk: c0527}
|
||||
128
tests/rag/last_run.json
Normal file
128
tests/rag/last_run.json
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
{
|
||||
"k": 5,
|
||||
"n_queries": 15,
|
||||
"n_positive": 12,
|
||||
"n_negative": 3,
|
||||
"recall_at_k": 0.2083,
|
||||
"mrr": 0.25,
|
||||
"negative_pass_rate": 1.0,
|
||||
"per_query": [
|
||||
{
|
||||
"id": "q01-arnold-mt-rainier",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.5,
|
||||
"mrr": 1.0,
|
||||
"n_expected": 2,
|
||||
"n_present": 1
|
||||
},
|
||||
{
|
||||
"id": "q02-maury-island-hoax",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.0,
|
||||
"mrr": 0.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 0
|
||||
},
|
||||
{
|
||||
"id": "q03-rhodes-phoenix-photo",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.0,
|
||||
"mrr": 0.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 0
|
||||
},
|
||||
{
|
||||
"id": "q04-chiles-whitted",
|
||||
"negative": false,
|
||||
"recall_at_k": 1.0,
|
||||
"mrr": 1.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 1
|
||||
},
|
||||
{
|
||||
"id": "q05-gorman-dogfight",
|
||||
"negative": true,
|
||||
"ok": true,
|
||||
"n_hits": 0
|
||||
},
|
||||
{
|
||||
"id": "q06-mantell-crash",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.0,
|
||||
"mrr": 0.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 0
|
||||
},
|
||||
{
|
||||
"id": "q07-sandia-1948-1950",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.0,
|
||||
"mrr": 0.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 0
|
||||
},
|
||||
{
|
||||
"id": "q08-pajarito-astronomers",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.0,
|
||||
"mrr": 0.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 0
|
||||
},
|
||||
{
|
||||
"id": "q09-james-tuck-correspondence",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.0,
|
||||
"mrr": 0.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 0
|
||||
},
|
||||
{
|
||||
"id": "q10-cometa-report",
|
||||
"negative": false,
|
||||
"recall_at_k": 1.0,
|
||||
"mrr": 1.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 1
|
||||
},
|
||||
{
|
||||
"id": "q11-apollo-17-flash",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.0,
|
||||
"mrr": 0.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 0
|
||||
},
|
||||
{
|
||||
"id": "q12-usper-narrative",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.0,
|
||||
"mrr": 0.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 0
|
||||
},
|
||||
{
|
||||
"id": "q13-uss-nimitz-tic-tac",
|
||||
"negative": true,
|
||||
"ok": true,
|
||||
"n_hits": 0
|
||||
},
|
||||
{
|
||||
"id": "q14-mj-12",
|
||||
"negative": true,
|
||||
"ok": true,
|
||||
"n_hits": 0
|
||||
},
|
||||
{
|
||||
"id": "q15-roswell",
|
||||
"negative": false,
|
||||
"recall_at_k": 0.0,
|
||||
"mrr": 0.0,
|
||||
"n_expected": 1,
|
||||
"n_present": 0
|
||||
}
|
||||
],
|
||||
"url": "https://disclosure.top",
|
||||
"top_k": 5,
|
||||
"rerank": "never"
|
||||
}
|
||||
178
tests/rag/run.py
Normal file
178
tests/rag/run.py
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
tests/rag/run.py — Golden RAG evaluation.
|
||||
|
||||
Reads tests/rag/golden.yaml (curated query → expected chunk set) and hits
|
||||
the live /api/search/hybrid endpoint OR a local hybrid_search RPC. Computes
|
||||
Recall@5 and MRR per query, plus aggregates. Writes a JSON report to
|
||||
tests/rag/last_run.json and compares with tests/rag/baseline.json.
|
||||
|
||||
CI gate: if Recall@5 drops more than --max-recall-drop (default 0.05) from
|
||||
baseline, exit 1.
|
||||
|
||||
Usage:
|
||||
python3 tests/rag/run.py # uses prod URL
|
||||
python3 tests/rag/run.py --url http://localhost:3000 # local dev
|
||||
python3 tests/rag/run.py --refresh-baseline # accept current as baseline
|
||||
python3 tests/rag/run.py --top-k 10 --no-rerank
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
sys.exit("pip install pyyaml")
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
GOLDEN = ROOT / "golden.yaml"
|
||||
BASELINE = ROOT / "baseline.json"
|
||||
LAST_RUN = ROOT / "last_run.json"
|
||||
|
||||
|
||||
def search(base_url: str, q: str, lang: str, top_k: int, rerank: str) -> list[dict]:
|
||||
params = {"q": q, "lang": lang, "top_k": str(top_k)}
|
||||
if rerank == "never":
|
||||
params["rerank"] = "never"
|
||||
elif rerank == "always":
|
||||
params["rerank"] = "always"
|
||||
qs = urllib.parse.urlencode(params)
|
||||
url = f"{base_url.rstrip('/')}/api/search/hybrid?{qs}"
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=30) as r:
|
||||
data = json.loads(r.read())
|
||||
return data.get("hits", [])
|
||||
except urllib.error.HTTPError as e:
|
||||
sys.stderr.write(f" ! HTTP {e.code} on {q!r}\n")
|
||||
return []
|
||||
except Exception as e:
|
||||
sys.stderr.write(f" ! {e} on {q!r}\n")
|
||||
return []
|
||||
|
||||
|
||||
def evaluate(golden: list[dict], hits_by_id: dict[str, list[dict]], k: int) -> dict:
|
||||
"""Per-query Recall@k + MRR. Negative-set queries (no expected chunks)
|
||||
pass when no hits are returned within the top-k."""
|
||||
per_query: list[dict] = []
|
||||
pos_recalls: list[float] = []
|
||||
pos_mrrs: list[float] = []
|
||||
neg_pass = 0
|
||||
neg_total = 0
|
||||
|
||||
for q in golden:
|
||||
qid = q["id"]
|
||||
expected = {(e["doc"], e["chunk"]) for e in (q.get("expected_chunks") or [])}
|
||||
hits = hits_by_id.get(qid, [])
|
||||
topk = hits[:k]
|
||||
|
||||
if not expected:
|
||||
# Negative-set: pass when fewer than k hits, OR when first hit is
|
||||
# weak enough that the model wouldn't latch onto it. We accept
|
||||
# any non-zero result count as failure to keep the metric strict.
|
||||
neg_total += 1
|
||||
ok = len(topk) == 0
|
||||
per_query.append({
|
||||
"id": qid, "negative": True, "ok": ok,
|
||||
"n_hits": len(topk),
|
||||
})
|
||||
if ok:
|
||||
neg_pass += 1
|
||||
continue
|
||||
|
||||
present = sum(1 for h in topk if (h.get("doc_id"), h.get("chunk_id")) in expected)
|
||||
recall = present / len(expected)
|
||||
# MRR — first matching position (1-indexed). 0 if none.
|
||||
rr = 0.0
|
||||
for i, h in enumerate(topk, start=1):
|
||||
if (h.get("doc_id"), h.get("chunk_id")) in expected:
|
||||
rr = 1.0 / i
|
||||
break
|
||||
per_query.append({
|
||||
"id": qid, "negative": False,
|
||||
"recall_at_k": round(recall, 4),
|
||||
"mrr": round(rr, 4),
|
||||
"n_expected": len(expected),
|
||||
"n_present": present,
|
||||
})
|
||||
pos_recalls.append(recall)
|
||||
pos_mrrs.append(rr)
|
||||
|
||||
return {
|
||||
"k": k,
|
||||
"n_queries": len(per_query),
|
||||
"n_positive": len(pos_recalls),
|
||||
"n_negative": neg_total,
|
||||
"recall_at_k": round(sum(pos_recalls) / len(pos_recalls), 4) if pos_recalls else 0.0,
|
||||
"mrr": round(sum(pos_mrrs) / len(pos_mrrs), 4) if pos_mrrs else 0.0,
|
||||
"negative_pass_rate": round(neg_pass / neg_total, 4) if neg_total else 1.0,
|
||||
"per_query": per_query,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--url", default="https://disclosure.top",
|
||||
help="Base URL of the deployment to evaluate")
|
||||
ap.add_argument("--top-k", type=int, default=5)
|
||||
ap.add_argument("--rerank", choices=["always", "when_top_k_gt", "never"],
|
||||
default="when_top_k_gt")
|
||||
ap.add_argument("--refresh-baseline", action="store_true",
|
||||
help="Overwrite baseline.json with this run (acknowledged regression).")
|
||||
ap.add_argument("--max-recall-drop", type=float, default=0.05)
|
||||
args = ap.parse_args()
|
||||
|
||||
data = yaml.safe_load(GOLDEN.read_text())
|
||||
queries = data["queries"]
|
||||
print(f"= running {len(queries)} queries against {args.url} (k={args.top_k}, rerank={args.rerank})")
|
||||
|
||||
hits_by_id = {}
|
||||
for q in queries:
|
||||
hits = search(args.url, q["question"], q.get("lang", "pt"),
|
||||
top_k=max(args.top_k, 10), rerank=args.rerank)
|
||||
hits_by_id[q["id"]] = hits
|
||||
first = hits[0].get("chunk_id") if hits else "-"
|
||||
print(f" {q['id']:24s} → {len(hits):2d} hits (first={first})")
|
||||
|
||||
report = evaluate(queries, hits_by_id, k=args.top_k)
|
||||
report["url"] = args.url
|
||||
report["top_k"] = args.top_k
|
||||
report["rerank"] = args.rerank
|
||||
|
||||
LAST_RUN.write_text(json.dumps(report, indent=2))
|
||||
print(f"\n— wrote {LAST_RUN}")
|
||||
print(f" Recall@{args.top_k} = {report['recall_at_k']:.4f}")
|
||||
print(f" MRR = {report['mrr']:.4f}")
|
||||
print(f" Negative pass = {report['negative_pass_rate']:.4f}")
|
||||
|
||||
if args.refresh_baseline:
|
||||
BASELINE.write_text(json.dumps({
|
||||
"url": args.url, "top_k": args.top_k, "rerank": args.rerank,
|
||||
"recall_at_k": report["recall_at_k"],
|
||||
"mrr": report["mrr"],
|
||||
"negative_pass_rate": report["negative_pass_rate"],
|
||||
}, indent=2))
|
||||
print(f"\n✓ baseline refreshed: {BASELINE}")
|
||||
return 0
|
||||
|
||||
if not BASELINE.exists():
|
||||
print("\n! no baseline yet — run with --refresh-baseline to create one")
|
||||
return 0
|
||||
|
||||
baseline = json.loads(BASELINE.read_text())
|
||||
drop = baseline["recall_at_k"] - report["recall_at_k"]
|
||||
print(f"\n baseline Recall@{args.top_k} = {baseline['recall_at_k']:.4f} (Δ {-drop:+.4f})")
|
||||
if drop > args.max_recall_drop:
|
||||
print(f"\n✗ GATE FAILED: Recall@{args.top_k} dropped {drop:.4f} > {args.max_recall_drop}")
|
||||
return 1
|
||||
print(f"\n✓ gate passed (drop ≤ {args.max_recall_drop})")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -28,10 +28,26 @@ export async function GET(req: NextRequest) {
|
|||
const type = u.searchParams.get("type") || null;
|
||||
const top_k = Math.min(Number(u.searchParams.get("top_k") ?? 10), 50);
|
||||
const ufo_only = u.searchParams.get("ufo_only") === "1";
|
||||
const no_rerank = u.searchParams.get("rerank") === "0";
|
||||
// W2-TD#8: rerank now has three modes. Back-compat: `rerank=0` keeps the
|
||||
// old "never" shortcut. New: `rerank=always|when_top_k_gt|never` and
|
||||
// `rerank_threshold=N` (default 15). Default strategy is `when_top_k_gt`.
|
||||
const rerankParam = u.searchParams.get("rerank");
|
||||
const no_rerank = rerankParam === "0";
|
||||
const rerank_strategy = (
|
||||
rerankParam === "always" || rerankParam === "never" || rerankParam === "when_top_k_gt"
|
||||
? rerankParam
|
||||
: "when_top_k_gt"
|
||||
) as "always" | "when_top_k_gt" | "never";
|
||||
const rerank_threshold = Math.max(
|
||||
1,
|
||||
Math.min(50, Number(u.searchParams.get("rerank_threshold") ?? 15)),
|
||||
);
|
||||
|
||||
try {
|
||||
const hits = await hybridSearch({ query: q, lang, doc_id, type, ufo_only, top_k, no_rerank });
|
||||
const hits = await hybridSearch({
|
||||
query: q, lang, doc_id, type, ufo_only, top_k, no_rerank,
|
||||
rerank_strategy, rerank_threshold,
|
||||
});
|
||||
return json({
|
||||
query: q,
|
||||
lang,
|
||||
|
|
|
|||
|
|
@ -20,14 +20,28 @@ import { streamChat } from "@/lib/chat";
|
|||
import { getLocale } from "@/components/locale-toggle";
|
||||
import { withRequest } from "@/lib/logger";
|
||||
|
||||
/**
|
||||
* Context size limits per artifact type. Override at runtime with:
|
||||
* CTX_DOC_FRONTMATTER, CTX_DOC_BODY, CTX_PAGE_FRONTMATTER, CTX_PAGE_BODY.
|
||||
* W2-TD#27 — was hard-coded 1200 / 1500 / 1500 / 1500. Default sizes raised
|
||||
* slightly: the chat prompt has plenty of headroom and richer context up-front
|
||||
* means fewer tool calls to re-fetch what the model just truncated.
|
||||
*/
|
||||
const CTX = {
|
||||
doc_frontmatter: Number(process.env.CTX_DOC_FRONTMATTER || 1500),
|
||||
doc_body: Number(process.env.CTX_DOC_BODY || 3000),
|
||||
page_frontmatter: Number(process.env.CTX_PAGE_FRONTMATTER || 1800),
|
||||
page_body: Number(process.env.CTX_PAGE_BODY || 2000),
|
||||
} as const;
|
||||
|
||||
async function gatherContext(docId: string | null, pageId: string | null): Promise<string> {
|
||||
const parts: string[] = [];
|
||||
if (docId) {
|
||||
const d = await readDocument(docId);
|
||||
if (d) {
|
||||
parts.push(`# Current document: ${docId}\n` +
|
||||
`Frontmatter: ${JSON.stringify(d.fm, null, 2).slice(0, 1200)}\n\n` +
|
||||
`Body excerpt:\n${d.body.slice(0, 1500)}`);
|
||||
`Frontmatter: ${JSON.stringify(d.fm, null, 2).slice(0, CTX.doc_frontmatter)}\n\n` +
|
||||
`Body excerpt:\n${d.body.slice(0, CTX.doc_body)}`);
|
||||
}
|
||||
}
|
||||
if (pageId) {
|
||||
|
|
@ -36,8 +50,8 @@ async function gatherContext(docId: string | null, pageId: string | null): Promi
|
|||
const md = await readPage(d, p);
|
||||
if (md) {
|
||||
parts.push(`# Current page: ${pageId}\n` +
|
||||
`Frontmatter: ${JSON.stringify(md.fm, null, 2).slice(0, 1500)}\n\n` +
|
||||
`Body excerpt:\n${md.body.slice(0, 1500)}`);
|
||||
`Frontmatter: ${JSON.stringify(md.fm, null, 2).slice(0, CTX.page_frontmatter)}\n\n` +
|
||||
`Body excerpt:\n${md.body.slice(0, CTX.page_body)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,7 +6,9 @@
|
|||
*/
|
||||
import Link from "next/link";
|
||||
import { AuthBar } from "@/components/auth-bar";
|
||||
import { ForceGraphCanvas } from "@/components/force-graph-canvas";
|
||||
// W2-TD#12: switched from react-force-graph-2d to @react-sigma. One graph
|
||||
// library is enough; sigma is the one already used by the entity sidebar.
|
||||
import { SigmaGraph } from "@/components/sigma-graph";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
|
|
@ -46,7 +48,7 @@ export default function GraphPage() {
|
|||
|
||||
{/* Fullscreen canvas */}
|
||||
<div className="absolute inset-0">
|
||||
<ForceGraphCanvas />
|
||||
<SigmaGraph />
|
||||
</div>
|
||||
</main>
|
||||
);
|
||||
|
|
|
|||
|
|
@ -1,596 +0,0 @@
|
|||
/**
|
||||
* ForceGraphCanvas — D3 force-directed entity graph (Obsidian-style).
|
||||
*
|
||||
* Layout:
|
||||
* - Left sidebar: filters (classes, limit) — sempre visível, fora do canvas
|
||||
* - Right side panel: detalhe da entidade selecionada (quando clica num nó)
|
||||
* - Center: canvas fullscreen com nodes coloridos por classe + edges
|
||||
* coloridas por peso (low=cinza, mid=cyan, high=verde)
|
||||
*
|
||||
* Interação:
|
||||
* - HOVER: tooltip flutuante com nome + classe + mentions
|
||||
* - CLICK: abre side panel direito com info da entidade + top neighbors + botão "abrir página"
|
||||
* - DOUBLE-CLICK: navega direto para /e/<class>/<id>
|
||||
* - Scroll: zoom; drag canvas: pan
|
||||
*/
|
||||
"use client";
|
||||
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||
import dynamic from "next/dynamic";
|
||||
import Link from "next/link";
|
||||
|
||||
const ForceGraph2D = dynamic(() => import("react-force-graph-2d"), { ssr: false });
|
||||
|
||||
interface RawNode {
|
||||
entity_pk: number;
|
||||
entity_class: string;
|
||||
entity_id: string;
|
||||
canonical_name: string;
|
||||
total_mentions: number;
|
||||
documents_count: number;
|
||||
}
|
||||
interface RawLink {
|
||||
source: number;
|
||||
target: number;
|
||||
weight: number;
|
||||
}
|
||||
interface GraphNode extends RawNode {
|
||||
id: number;
|
||||
x?: number;
|
||||
y?: number;
|
||||
vx?: number;
|
||||
vy?: number;
|
||||
}
|
||||
interface GraphLink {
|
||||
source: number | GraphNode;
|
||||
target: number | GraphNode;
|
||||
weight: number;
|
||||
}
|
||||
|
||||
const CLASS_COLOR: Record<string, string> = {
|
||||
person: "#ff6ec7",
|
||||
organization: "#ff8a4d",
|
||||
location: "#3fde6a",
|
||||
event: "#ffa500",
|
||||
uap_object: "#ff3344",
|
||||
vehicle: "#5b9bd5",
|
||||
operation: "#9b5de5",
|
||||
concept: "#06d6a0",
|
||||
};
|
||||
const CLASS_FOLDER: Record<string, string> = {
|
||||
person: "people",
|
||||
organization: "organizations",
|
||||
location: "locations",
|
||||
event: "events",
|
||||
uap_object: "uap-objects",
|
||||
vehicle: "vehicles",
|
||||
operation: "operations",
|
||||
concept: "concepts",
|
||||
};
|
||||
const CLASS_LABEL: Record<string, string> = {
|
||||
person: "Pessoas",
|
||||
organization: "Organizações",
|
||||
location: "Locais",
|
||||
event: "Eventos",
|
||||
uap_object: "UAP",
|
||||
vehicle: "Veículos",
|
||||
operation: "Operações",
|
||||
concept: "Conceitos",
|
||||
};
|
||||
|
||||
const ALL_CLASSES = ["person", "organization", "location", "event", "uap_object", "vehicle", "operation", "concept"];
|
||||
|
||||
/** Color edge by weight tier — visual diferenciação por intensidade */
|
||||
function edgeColor(weight: number): string {
|
||||
if (weight >= 10) return "rgba(0,255,156,0.55)"; // strong: green
|
||||
if (weight >= 5) return "rgba(127,219,255,0.45)"; // medium: cyan
|
||||
if (weight >= 3) return "rgba(167,139,250,0.35)"; // mild: purple
|
||||
return "rgba(127,219,255,0.18)"; // weak: faded cyan
|
||||
}
|
||||
|
||||
function edgeWidth(weight: number): number {
|
||||
return Math.max(0.5, Math.min(6, Math.log2(weight + 1) * 1.2));
|
||||
}
|
||||
|
||||
interface EntityDetail {
|
||||
entity_pk: number;
|
||||
entity_class: string;
|
||||
entity_id: string;
|
||||
canonical_name: string;
|
||||
total_mentions: number;
|
||||
documents_count: number;
|
||||
neighbors: Array<{
|
||||
entity_pk: number;
|
||||
entity_class: string;
|
||||
entity_id: string;
|
||||
canonical_name: string;
|
||||
weight: number;
|
||||
total_mentions: number;
|
||||
}>;
|
||||
}
|
||||
|
||||
const detailCache = new Map<number, EntityDetail>();
|
||||
|
||||
interface ForceGraph2DRef {
|
||||
d3Force: (name: string) => { strength?: (v: number) => unknown; distance?: (v: number) => unknown } | null;
|
||||
d3ReheatSimulation: () => void;
|
||||
zoomToFit: (durationMs?: number, paddingPx?: number) => void;
|
||||
centerAt: (x?: number, y?: number, durationMs?: number) => void;
|
||||
}
|
||||
|
||||
export function ForceGraphCanvas() {
|
||||
const fgRef = useRef<ForceGraph2DRef | null>(null);
|
||||
const [nodes, setNodes] = useState<GraphNode[]>([]);
|
||||
const [links, setLinks] = useState<GraphLink[]>([]);
|
||||
const [selectedClasses, setSelectedClasses] = useState<Set<string>>(new Set(ALL_CLASSES));
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [hoverNode, setHoverNode] = useState<GraphNode | null>(null);
|
||||
const [hoverPos, setHoverPos] = useState<{ x: number; y: number } | null>(null);
|
||||
const [selectedNode, setSelectedNode] = useState<GraphNode | null>(null);
|
||||
const [detail, setDetail] = useState<EntityDetail | null>(null);
|
||||
const [detailLoading, setDetailLoading] = useState(false);
|
||||
const [limit, setLimit] = useState(40);
|
||||
const [minWeight, setMinWeight] = useState(3);
|
||||
const [search, setSearch] = useState("");
|
||||
|
||||
// Tune d3-force after the graph mounts and on data change — STRONGER repulsion + LONGER links
|
||||
useEffect(() => {
|
||||
const fg = fgRef.current;
|
||||
if (!fg) return;
|
||||
const charge = fg.d3Force("charge");
|
||||
if (charge?.strength) charge.strength(-450);
|
||||
const link = fg.d3Force("link");
|
||||
if (link?.distance) link.distance(120);
|
||||
const center = fg.d3Force("center");
|
||||
if (center?.strength) center.strength(0.04);
|
||||
fg.d3ReheatSimulation();
|
||||
setTimeout(() => fg.zoomToFit?.(800, 80), 1500);
|
||||
}, [nodes.length, links.length]);
|
||||
|
||||
// Initial seed load — re-runs when filters change
|
||||
useEffect(() => {
|
||||
setLoading(true);
|
||||
const classesParam = Array.from(selectedClasses).join(",");
|
||||
fetch(`/api/graph/seed?limit=${limit}&min_weight=${minWeight}&classes=${classesParam}`)
|
||||
.then((r) => r.json())
|
||||
.then((data: { nodes?: RawNode[]; links?: RawLink[] }) => {
|
||||
const ns = (data.nodes ?? []).map((n) => ({ ...n, id: n.entity_pk } as GraphNode));
|
||||
const ls = (data.links ?? []).map((l) => ({ source: l.source, target: l.target, weight: l.weight } as GraphLink));
|
||||
setNodes(ns);
|
||||
setLinks(ls);
|
||||
setLoading(false);
|
||||
})
|
||||
.catch(() => setLoading(false));
|
||||
}, [limit, minWeight, selectedClasses]);
|
||||
|
||||
// Fetch detail when node selected
|
||||
useEffect(() => {
|
||||
if (!selectedNode) {
|
||||
setDetail(null);
|
||||
return;
|
||||
}
|
||||
const cached = detailCache.get(selectedNode.entity_pk);
|
||||
if (cached) {
|
||||
setDetail(cached);
|
||||
return;
|
||||
}
|
||||
setDetail(null);
|
||||
setDetailLoading(true);
|
||||
fetch(
|
||||
`/api/graph?op=neighbors&class=${selectedNode.entity_class}&id=${encodeURIComponent(selectedNode.entity_id)}&limit=12`,
|
||||
)
|
||||
.then((r) => r.json())
|
||||
.then((data: { entity?: RawNode; neighbors?: EntityDetail["neighbors"] }) => {
|
||||
const d: EntityDetail = {
|
||||
entity_pk: selectedNode.entity_pk,
|
||||
entity_class: selectedNode.entity_class,
|
||||
entity_id: selectedNode.entity_id,
|
||||
canonical_name: selectedNode.canonical_name,
|
||||
total_mentions: data.entity?.total_mentions ?? selectedNode.total_mentions,
|
||||
documents_count: data.entity?.documents_count ?? selectedNode.documents_count,
|
||||
neighbors: data.neighbors ?? [],
|
||||
};
|
||||
detailCache.set(selectedNode.entity_pk, d);
|
||||
setDetail(d);
|
||||
})
|
||||
.catch(() => setDetail(null))
|
||||
.finally(() => setDetailLoading(false));
|
||||
}, [selectedNode]);
|
||||
|
||||
const onNodeClick = useCallback(async (node: GraphNode) => {
|
||||
setSelectedNode(node);
|
||||
}, []);
|
||||
|
||||
const expandNode = useCallback(
|
||||
async (node: GraphNode) => {
|
||||
try {
|
||||
const r = await fetch(
|
||||
`/api/graph?op=neighbors&class=${node.entity_class}&id=${encodeURIComponent(node.entity_id)}&limit=15`,
|
||||
);
|
||||
if (!r.ok) return;
|
||||
const data = (await r.json()) as { neighbors?: Array<RawNode & { weight: number }> };
|
||||
if (!data.neighbors) return;
|
||||
setNodes((prev) => {
|
||||
const existing = new Set(prev.map((p) => p.id));
|
||||
const additions = data.neighbors!
|
||||
.filter((n) => !existing.has(n.entity_pk))
|
||||
.map((n) => ({ ...n, id: n.entity_pk } as GraphNode));
|
||||
return [...prev, ...additions];
|
||||
});
|
||||
setLinks((prev) => {
|
||||
const seen = new Set(
|
||||
prev.map((l) => {
|
||||
const s = typeof l.source === "object" ? (l.source as GraphNode).id : l.source;
|
||||
const t = typeof l.target === "object" ? (l.target as GraphNode).id : l.target;
|
||||
return `${Math.min(s, t)}-${Math.max(s, t)}`;
|
||||
}),
|
||||
);
|
||||
const additions: GraphLink[] = [];
|
||||
for (const n of data.neighbors!) {
|
||||
const a = node.entity_pk;
|
||||
const b = n.entity_pk;
|
||||
const key = `${Math.min(a, b)}-${Math.max(a, b)}`;
|
||||
if (!seen.has(key)) {
|
||||
additions.push({ source: a, target: b, weight: n.weight });
|
||||
seen.add(key);
|
||||
}
|
||||
}
|
||||
return [...prev, ...additions];
|
||||
});
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
},
|
||||
[],
|
||||
);
|
||||
|
||||
const toggleClass = useCallback((cls: string) => {
|
||||
setSelectedClasses((prev) => {
|
||||
const next = new Set(prev);
|
||||
if (next.has(cls)) next.delete(cls);
|
||||
else next.add(cls);
|
||||
return next.size > 0 ? next : prev;
|
||||
});
|
||||
}, []);
|
||||
|
||||
const visibleData = useMemo(() => {
|
||||
let filteredNodes = nodes.filter((n) => selectedClasses.has(n.entity_class));
|
||||
if (search.trim()) {
|
||||
const sl = search.toLowerCase();
|
||||
filteredNodes = filteredNodes.filter((n) =>
|
||||
n.canonical_name.toLowerCase().includes(sl) || n.entity_id.toLowerCase().includes(sl),
|
||||
);
|
||||
}
|
||||
const allowed = new Set(filteredNodes.map((n) => n.id));
|
||||
const filteredLinks = links.filter((l) => {
|
||||
const s = typeof l.source === "object" ? (l.source as GraphNode).id : l.source;
|
||||
const t = typeof l.target === "object" ? (l.target as GraphNode).id : l.target;
|
||||
return allowed.has(s) && allowed.has(t);
|
||||
});
|
||||
return { nodes: filteredNodes, links: filteredLinks };
|
||||
}, [nodes, links, selectedClasses, search]);
|
||||
|
||||
return (
|
||||
<div className="relative w-full h-full bg-[#040810] overflow-hidden">
|
||||
{/* LEFT sidebar — filters (sempre visível, fora do z-30 do page header) */}
|
||||
<div className="absolute top-20 left-4 z-20 w-[240px] max-h-[calc(100vh-180px)] overflow-y-auto bg-[#0a121e]/95 backdrop-blur border border-[rgba(0,255,156,0.20)] rounded p-3 space-y-4">
|
||||
<div>
|
||||
<div className="font-mono text-[10px] uppercase tracking-widest text-[#5a6678] mb-2">
|
||||
🔍 buscar nó
|
||||
</div>
|
||||
<input
|
||||
value={search}
|
||||
onChange={(e) => setSearch(e.target.value)}
|
||||
placeholder="nome ou id..."
|
||||
className="w-full bg-transparent border border-[rgba(0,255,156,0.20)] focus:border-[#00ff9c] rounded px-2 py-1.5 font-mono text-xs text-[#c8d4e6] outline-none"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<div className="font-mono text-[10px] uppercase tracking-widest text-[#5a6678] mb-2">
|
||||
classes
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
{ALL_CLASSES.map((cls) => {
|
||||
const active = selectedClasses.has(cls);
|
||||
const color = CLASS_COLOR[cls] ?? "#7fdbff";
|
||||
return (
|
||||
<button
|
||||
key={cls}
|
||||
onClick={() => toggleClass(cls)}
|
||||
className={`w-full flex items-center gap-2 px-2 py-1 font-mono text-[11px] rounded border transition ${
|
||||
active ? "" : "opacity-30 hover:opacity-60"
|
||||
}`}
|
||||
style={{
|
||||
color,
|
||||
borderColor: color,
|
||||
background: active ? `${color}12` : "transparent",
|
||||
}}
|
||||
>
|
||||
<span className="inline-block w-2 h-2 rounded-full" style={{ background: color }} />
|
||||
<span className="flex-1 text-left">{CLASS_LABEL[cls]}</span>
|
||||
{active && <span>✓</span>}
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<div className="font-mono text-[10px] uppercase tracking-widest text-[#5a6678] mb-2">
|
||||
top entidades
|
||||
</div>
|
||||
<div className="flex flex-wrap gap-1">
|
||||
{[20, 40, 80, 150].map((n) => (
|
||||
<button
|
||||
key={n}
|
||||
onClick={() => setLimit(n)}
|
||||
className={`px-2 py-1 font-mono text-[11px] rounded border ${
|
||||
limit === n
|
||||
? "border-[#00ff9c] text-[#00ff9c] bg-[rgba(0,255,156,0.10)]"
|
||||
: "border-[rgba(127,219,255,0.20)] text-[#8896aa] hover:text-[#7fdbff]"
|
||||
}`}
|
||||
>
|
||||
{n}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<div className="font-mono text-[10px] uppercase tracking-widest text-[#5a6678] mb-2">
|
||||
mostrar vínculos com ≥
|
||||
</div>
|
||||
<div className="flex flex-wrap gap-1">
|
||||
{[2, 3, 5, 10].map((n) => (
|
||||
<button
|
||||
key={n}
|
||||
onClick={() => setMinWeight(n)}
|
||||
className={`px-2 py-1 font-mono text-[11px] rounded border ${
|
||||
minWeight === n
|
||||
? "border-[#00ff9c] text-[#00ff9c] bg-[rgba(0,255,156,0.10)]"
|
||||
: "border-[rgba(127,219,255,0.20)] text-[#8896aa] hover:text-[#7fdbff]"
|
||||
}`}
|
||||
>
|
||||
{n}×
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<div className="font-mono text-[10px] uppercase tracking-widest text-[#5a6678] mb-2">
|
||||
força do vínculo
|
||||
</div>
|
||||
<div className="space-y-1 text-[10px] font-mono">
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="inline-block w-6 h-0.5" style={{ background: "rgba(0,255,156,0.55)" }} />
|
||||
<span className="text-[#8896aa]">≥ 10 co-menções</span>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="inline-block w-6 h-0.5" style={{ background: "rgba(127,219,255,0.45)" }} />
|
||||
<span className="text-[#8896aa]">5–9</span>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="inline-block w-6 h-0.5" style={{ background: "rgba(167,139,250,0.35)" }} />
|
||||
<span className="text-[#8896aa]">3–4</span>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="inline-block w-6 h-0.5" style={{ background: "rgba(127,219,255,0.18)" }} />
|
||||
<span className="text-[#8896aa]">2 (mín.)</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="pt-2 border-t border-[rgba(0,255,156,0.10)] font-mono text-[10px] text-[#5a6678]">
|
||||
{loading ? "carregando…" : `${visibleData.nodes.length} nós · ${visibleData.links.length} arestas`}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* RIGHT side panel — entidade selecionada */}
|
||||
{selectedNode && (
|
||||
<div className="absolute top-20 right-4 z-20 w-[340px] max-h-[calc(100vh-180px)] overflow-y-auto bg-[#0a121e]/95 backdrop-blur border-2 rounded p-4 space-y-4"
|
||||
style={{ borderColor: CLASS_COLOR[selectedNode.entity_class] ?? "#7fdbff" }}>
|
||||
<div className="flex items-start justify-between gap-2">
|
||||
<div className="flex-1 min-w-0">
|
||||
<div
|
||||
className="font-mono text-[10px] uppercase tracking-widest mb-1"
|
||||
style={{ color: CLASS_COLOR[selectedNode.entity_class] ?? "#7fdbff" }}
|
||||
>
|
||||
{CLASS_LABEL[selectedNode.entity_class] ?? selectedNode.entity_class}
|
||||
</div>
|
||||
<h3 className="font-mono text-base text-[#c8d4e6] font-bold leading-tight break-words">
|
||||
{selectedNode.canonical_name}
|
||||
</h3>
|
||||
<div className="font-mono text-[10px] text-[#5a6678] mt-1 truncate">
|
||||
{selectedNode.entity_id}
|
||||
</div>
|
||||
</div>
|
||||
<button
|
||||
onClick={() => setSelectedNode(null)}
|
||||
className="text-[#5a6678] hover:text-[#ff6b6b] flex-shrink-0"
|
||||
aria-label="fechar"
|
||||
>
|
||||
✕
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="grid grid-cols-2 gap-2">
|
||||
<div className="px-3 py-2 bg-[#060a13] border border-[rgba(0,255,156,0.20)] rounded">
|
||||
<div className="font-mono text-[9px] uppercase text-[#5a6678]">menções</div>
|
||||
<div className="font-mono text-lg text-[#00ff9c] mt-0.5">{selectedNode.total_mentions}</div>
|
||||
</div>
|
||||
<div className="px-3 py-2 bg-[#060a13] border border-[rgba(127,219,255,0.20)] rounded">
|
||||
<div className="font-mono text-[9px] uppercase text-[#5a6678]">documentos</div>
|
||||
<div className="font-mono text-lg text-[#7fdbff] mt-0.5">{selectedNode.documents_count}</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Action buttons */}
|
||||
<div className="space-y-1.5">
|
||||
<Link
|
||||
href={`/e/${CLASS_FOLDER[selectedNode.entity_class]}/${selectedNode.entity_id}`}
|
||||
className="block w-full px-3 py-2 font-mono text-xs uppercase tracking-widest border-2 border-[#00ff9c] text-[#00ff9c] bg-[rgba(0,255,156,0.08)] hover:bg-[rgba(0,255,156,0.18)] rounded text-center"
|
||||
>
|
||||
abrir página completa →
|
||||
</Link>
|
||||
<button
|
||||
onClick={() => expandNode(selectedNode)}
|
||||
className="block w-full px-3 py-2 font-mono text-xs uppercase tracking-widest border border-[#7fdbff] text-[#7fdbff] hover:bg-[rgba(127,219,255,0.10)] rounded"
|
||||
>
|
||||
+ expandir vizinhos no grafo
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{/* Neighbors list */}
|
||||
<div>
|
||||
<div className="font-mono text-[10px] uppercase tracking-widest text-[#5a6678] mb-2">
|
||||
{detailLoading ? "carregando vizinhos…" : `top vínculos (${detail?.neighbors.length ?? 0})`}
|
||||
</div>
|
||||
{detail?.neighbors && detail.neighbors.length > 0 ? (
|
||||
<ul className="space-y-1">
|
||||
{detail.neighbors.map((n) => {
|
||||
const color = CLASS_COLOR[n.entity_class] ?? "#7fdbff";
|
||||
return (
|
||||
<li key={n.entity_pk}>
|
||||
<button
|
||||
onClick={() => {
|
||||
const folded = nodes.find((nn) => nn.entity_pk === n.entity_pk);
|
||||
if (folded) {
|
||||
setSelectedNode(folded);
|
||||
} else {
|
||||
// Inject into graph + select
|
||||
const newNode: GraphNode = {
|
||||
entity_pk: n.entity_pk,
|
||||
entity_class: n.entity_class,
|
||||
entity_id: n.entity_id,
|
||||
canonical_name: n.canonical_name,
|
||||
total_mentions: n.total_mentions,
|
||||
documents_count: 0,
|
||||
id: n.entity_pk,
|
||||
};
|
||||
setNodes((prev) => [...prev, newNode]);
|
||||
setLinks((prev) => [...prev, { source: selectedNode.entity_pk, target: n.entity_pk, weight: n.weight }]);
|
||||
setSelectedNode(newNode);
|
||||
}
|
||||
}}
|
||||
className="group w-full flex items-center gap-2 text-left p-1.5 -mx-1.5 rounded hover:bg-[rgba(0,255,156,0.04)]"
|
||||
>
|
||||
<span
|
||||
className="inline-block w-2 h-2 rounded-full flex-shrink-0"
|
||||
style={{ background: color }}
|
||||
/>
|
||||
<span className="text-[11px] text-[#c8d4e6] group-hover:text-[#00ff9c] flex-1 truncate">
|
||||
{n.canonical_name}
|
||||
</span>
|
||||
<span className="font-mono text-[10px] text-[#5a6678] flex-shrink-0">×{n.weight}</span>
|
||||
</button>
|
||||
</li>
|
||||
);
|
||||
})}
|
||||
</ul>
|
||||
) : !detailLoading ? (
|
||||
<p className="font-mono text-[10px] text-[#5a6678] italic">sem co-menções</p>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
<p className="font-mono text-[9px] text-[#5a6678] pt-2 border-t border-[rgba(0,255,156,0.10)]">
|
||||
duplo-clique no nó: abre página da entidade · clique vizinho: foca nele
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Hover tooltip — segue o mouse */}
|
||||
{hoverNode && hoverPos && (
|
||||
<div
|
||||
className="absolute z-30 pointer-events-none px-2.5 py-1.5 bg-[#0a121e] border-2 rounded text-xs font-mono"
|
||||
style={{
|
||||
left: hoverPos.x + 12,
|
||||
top: hoverPos.y + 12,
|
||||
borderColor: CLASS_COLOR[hoverNode.entity_class] ?? "#7fdbff",
|
||||
}}
|
||||
>
|
||||
<div className="text-[#c8d4e6] font-bold">{hoverNode.canonical_name}</div>
|
||||
<div className="text-[10px] text-[#5a6678]">
|
||||
{hoverNode.entity_class} · {hoverNode.total_mentions} menções · {hoverNode.documents_count} docs
|
||||
</div>
|
||||
<div className="text-[9px] text-[#7fdbff] mt-0.5">clique para detalhes</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Canvas */}
|
||||
<ForceGraph2D
|
||||
ref={fgRef as never}
|
||||
graphData={visibleData as never}
|
||||
backgroundColor="#040810"
|
||||
nodeRelSize={2.5}
|
||||
nodeVal={(n) => Math.max(1.5, Math.log2((n as GraphNode).total_mentions + 2) * 0.8)}
|
||||
nodeColor={(n) => CLASS_COLOR[(n as GraphNode).entity_class] ?? "#7fdbff"}
|
||||
nodeLabel={() => ""}
|
||||
linkColor={(l) => edgeColor((l as GraphLink).weight)}
|
||||
linkWidth={(l) => edgeWidth((l as GraphLink).weight)}
|
||||
// Physics — separa nós com força + arestas mais longas
|
||||
d3VelocityDecay={0.3}
|
||||
d3AlphaDecay={0.015}
|
||||
cooldownTicks={250}
|
||||
warmupTicks={60}
|
||||
// Use d3-force charge customization via dagMode? No, lib has limited API; rely on defaults + manual.
|
||||
onNodeClick={onNodeClick as never}
|
||||
onNodeHover={(n) => {
|
||||
setHoverNode(n as GraphNode | null);
|
||||
if (!n) setHoverPos(null);
|
||||
}}
|
||||
onBackgroundClick={() => setSelectedNode(null)}
|
||||
nodeCanvasObjectMode={() => "after"}
|
||||
nodeCanvasObject={(n, ctx, scale) => {
|
||||
const node = n as GraphNode;
|
||||
const isSelected = selectedNode?.entity_pk === node.entity_pk;
|
||||
const isHovered = hoverNode?.entity_pk === node.entity_pk;
|
||||
// Anti-clutter: with many nodes, mostrar label só:
|
||||
// - quando zoomado (scale ≥ 1.5)
|
||||
// - OU é hub (>200 mentions)
|
||||
// - OU é hover/selected
|
||||
const isHub = node.total_mentions >= 200;
|
||||
const showLabel = isSelected || isHovered || isHub || scale >= 1.5;
|
||||
if (!showLabel) {
|
||||
if (isSelected) {
|
||||
ctx.beginPath();
|
||||
ctx.arc(node.x ?? 0, node.y ?? 0, 10 / scale, 0, 2 * Math.PI, false);
|
||||
ctx.strokeStyle = "#00ff9c";
|
||||
ctx.lineWidth = 2.5 / scale;
|
||||
ctx.stroke();
|
||||
}
|
||||
return;
|
||||
}
|
||||
const fontSize = Math.max(10, 14 / scale);
|
||||
ctx.font = `${isHub ? "bold " : ""}${fontSize}px sans-serif`;
|
||||
const label =
|
||||
node.canonical_name.length > 28
|
||||
? node.canonical_name.slice(0, 26) + "…"
|
||||
: node.canonical_name;
|
||||
const tw = ctx.measureText(label).width;
|
||||
const pad = 4 / scale;
|
||||
// Background pill behind text — readability
|
||||
ctx.fillStyle = isSelected ? "rgba(0,255,156,0.85)" : "rgba(10,18,30,0.85)";
|
||||
ctx.fillRect(
|
||||
(node.x ?? 0) - tw / 2 - pad,
|
||||
(node.y ?? 0) + 8 / scale,
|
||||
tw + pad * 2,
|
||||
fontSize + pad,
|
||||
);
|
||||
ctx.fillStyle = isSelected ? "#040810" : "#c8d4e6";
|
||||
ctx.textAlign = "center";
|
||||
ctx.textBaseline = "top";
|
||||
ctx.fillText(label, node.x ?? 0, (node.y ?? 0) + 8 / scale + pad / 2);
|
||||
// Selected ring
|
||||
if (isSelected) {
|
||||
ctx.beginPath();
|
||||
ctx.arc(node.x ?? 0, node.y ?? 0, 10 / scale, 0, 2 * Math.PI, false);
|
||||
ctx.strokeStyle = "#00ff9c";
|
||||
ctx.lineWidth = 2.5 / scale;
|
||||
ctx.stroke();
|
||||
}
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
@ -314,6 +314,39 @@ const co_mention_chunks_tool: ToolDefinition = {
|
|||
},
|
||||
};
|
||||
|
||||
const analyze_image_region_tool: ToolDefinition = {
|
||||
type: "function",
|
||||
function: {
|
||||
name: "analyze_image_region",
|
||||
description:
|
||||
"Vision tool — answer a question about a cropped region of a document page. " +
|
||||
"Use this when the user asks about a photograph, diagram, sketch, signature, " +
|
||||
"stamp, redaction, or any visual element where the chunk's text description " +
|
||||
"isn't enough. The model reads the actual pixels via Sonnet vision. " +
|
||||
"Get the bbox + page from a prior hybrid_search hit (each chunk carries bbox). " +
|
||||
"Cost: ~$0.005–$0.02 per call. Use sparingly; prefer hybrid_search first.",
|
||||
parameters: {
|
||||
type: "object",
|
||||
properties: {
|
||||
doc_id: { type: "string" },
|
||||
page: { type: "integer", description: "1-indexed page number" },
|
||||
bbox: {
|
||||
type: "object",
|
||||
description: "Normalized bbox (0..1) of the region to analyze.",
|
||||
properties: {
|
||||
x: { type: "number" }, y: { type: "number" },
|
||||
w: { type: "number" }, h: { type: "number" },
|
||||
},
|
||||
required: ["x", "y", "w", "h"],
|
||||
},
|
||||
question: { type: "string", description: "What you want to know about the image." },
|
||||
context: { type: "string", description: "Optional: prose context that grounds the model." },
|
||||
},
|
||||
required: ["doc_id", "page", "bbox", "question"],
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const navigate_to_tool: ToolDefinition = {
|
||||
type: "function",
|
||||
function: {
|
||||
|
|
@ -345,6 +378,7 @@ export const TOOL_DEFINITIONS: ToolDefinition[] = [
|
|||
read_document_tool,
|
||||
read_entity_tool,
|
||||
search_corpus_tool,
|
||||
analyze_image_region_tool,
|
||||
navigate_to_tool,
|
||||
];
|
||||
|
||||
|
|
@ -398,6 +432,11 @@ async function handleHybridSearch(
|
|||
classification: (args.classification as string) || null,
|
||||
ufo_only: Boolean(args.ufo_only),
|
||||
top_k,
|
||||
// W2-TD#8: chat is latency-sensitive — skip rerank when ≤10 candidates.
|
||||
// The model only cites the first few hits anyway and BGE-Reranker
|
||||
// adds 5-8s on CPU. RRF order from the RPC is plenty for the head.
|
||||
rerank_strategy: "when_top_k_gt",
|
||||
rerank_threshold: 10,
|
||||
});
|
||||
// Emit one citation (+ optional crop_image) artifact per hit so the UI can
|
||||
// render inline cards next to the assistant text. Limit to top 6 to avoid
|
||||
|
|
@ -684,6 +723,37 @@ async function handleNavigate(args: Record<string, unknown>): Promise<unknown> {
|
|||
return { ok: true, target, label };
|
||||
}
|
||||
|
||||
async function handleAnalyzeImageRegion(
|
||||
args: Record<string, unknown>,
|
||||
ctx: ToolHandlerContext,
|
||||
): Promise<unknown> {
|
||||
const doc_id = String(args.doc_id ?? "").trim();
|
||||
const page = Number(args.page);
|
||||
const bbox = args.bbox as { x: number; y: number; w: number; h: number } | undefined;
|
||||
const question = String(args.question ?? "").trim();
|
||||
if (!doc_id || !page || !bbox || !question) return { error: "missing_args" };
|
||||
try {
|
||||
const { analyzeImageRegion } = await import("./vision");
|
||||
const out = await analyzeImageRegion({
|
||||
doc_id, page, bbox, question,
|
||||
context: typeof args.context === "string" ? args.context : undefined,
|
||||
lang: ctx.lang === "en" ? "en" : "pt",
|
||||
});
|
||||
if (ctx.emitArtifact) {
|
||||
ctx.emitArtifact({
|
||||
kind: "crop_image",
|
||||
src: out.crop_url,
|
||||
doc_id, page,
|
||||
alt_en: question.slice(0, 120),
|
||||
alt_pt: question.slice(0, 120),
|
||||
});
|
||||
}
|
||||
return out;
|
||||
} catch (e) {
|
||||
return { error: "vision_failed", message: (e as Error).message };
|
||||
}
|
||||
}
|
||||
|
||||
export const TOOL_HANDLERS: Record<string, ToolHandler> = {
|
||||
hybrid_search: handleHybridSearch,
|
||||
read_chunk: handleReadChunk,
|
||||
|
|
@ -696,5 +766,6 @@ export const TOOL_HANDLERS: Record<string, ToolHandler> = {
|
|||
read_document: handleReadDocument,
|
||||
read_entity: handleReadEntity,
|
||||
search_corpus: handleSearch,
|
||||
analyze_image_region: handleAnalyzeImageRegion,
|
||||
navigate_to: handleNavigate,
|
||||
};
|
||||
|
|
|
|||
165
web/lib/chat/vision.ts
Normal file
165
web/lib/chat/vision.ts
Normal file
|
|
@ -0,0 +1,165 @@
|
|||
/**
|
||||
* vision.ts — answer questions about an image region via Claude Code OAuth.
|
||||
*
|
||||
* Pattern matches the project's existing vision pipeline (02-vision-page.py):
|
||||
* 1. Crop the PNG of the requested page to the requested bbox.
|
||||
* 2. Spawn `claude -p --model sonnet --allowedTools Read` and instruct the
|
||||
* model to Read the local PNG path and answer the user's question.
|
||||
*
|
||||
* Uses the user's Claude Code OAuth (Max 20x). Per W1.2 budget policy, the
|
||||
* agentic worker may use Opus 4.7 without hard cap, but `analyze_image_region`
|
||||
* runs synchronously in the chat path — keep it on Sonnet for latency.
|
||||
*/
|
||||
import { spawn } from "node:child_process";
|
||||
import { mkdtemp, unlink, rmdir } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import os from "node:os";
|
||||
import sharp from "sharp";
|
||||
import { PROCESSING } from "@/lib/wiki";
|
||||
|
||||
const MODEL = process.env.VISION_MODEL || "sonnet";
|
||||
const TIMEOUT_MS = Number(process.env.VISION_TIMEOUT_MS || 120_000);
|
||||
|
||||
export interface AnalyzeImageRegionArgs {
|
||||
doc_id: string;
|
||||
page: number;
|
||||
bbox: { x: number; y: number; w: number; h: number };
|
||||
question: string;
|
||||
/** Optional context to ground the model (e.g., "this is an FBI memo from 1947"). */
|
||||
context?: string;
|
||||
/** Output language hint. Defaults to "pt-br". */
|
||||
lang?: "pt" | "en";
|
||||
}
|
||||
|
||||
export interface AnalyzeImageRegionResult {
|
||||
answer: string;
|
||||
model: string;
|
||||
duration_ms: number;
|
||||
bbox: { x: number; y: number; w: number; h: number };
|
||||
crop_url: string;
|
||||
}
|
||||
|
||||
function pageFilename(page: number): string {
|
||||
return `p-${String(page).padStart(3, "0")}.png`;
|
||||
}
|
||||
|
||||
/** Crop a bbox region of a page PNG and write the result to a temp file. */
|
||||
async function cropToTempFile(args: AnalyzeImageRegionArgs): Promise<{ path: string; dir: string }> {
|
||||
const sourcePath = path.join(PROCESSING, "png", args.doc_id, pageFilename(args.page));
|
||||
const meta = await sharp(sourcePath).metadata();
|
||||
const W = meta.width ?? 0;
|
||||
const H = meta.height ?? 0;
|
||||
if (W === 0 || H === 0) throw new Error(`source PNG unreadable: ${sourcePath}`);
|
||||
const left = Math.max(0, Math.round(args.bbox.x * W));
|
||||
const top = Math.max(0, Math.round(args.bbox.y * H));
|
||||
const width = Math.max(1, Math.min(W - left, Math.round(args.bbox.w * W)));
|
||||
const height = Math.max(1, Math.min(H - top, Math.round(args.bbox.h * H)));
|
||||
|
||||
const dir = await mkdtemp(path.join(os.tmpdir(), "analyze-image-"));
|
||||
const filePath = path.join(dir, "crop.png");
|
||||
await sharp(sourcePath)
|
||||
.extract({ left, top, width, height })
|
||||
.resize({ width: Math.min(1024, width), withoutEnlargement: true })
|
||||
.png()
|
||||
.toFile(filePath);
|
||||
return { path: filePath, dir };
|
||||
}
|
||||
|
||||
function buildPrompt(args: AnalyzeImageRegionArgs, cropPath: string): string {
|
||||
const lang = args.lang === "en" ? "English" : "Brazilian Portuguese (pt-br)";
|
||||
const ctx = args.context ? `\n\nContext: ${args.context}` : "";
|
||||
return [
|
||||
`Use the Read tool on this local PNG file: ${cropPath}`,
|
||||
"",
|
||||
`The image is a cropped region from document "${args.doc_id}", page ${args.page}.`,
|
||||
ctx,
|
||||
"",
|
||||
`Answer this question about what is visible in the image, in ${lang}:`,
|
||||
"",
|
||||
args.question,
|
||||
"",
|
||||
"Rules:",
|
||||
"- Be factual and concise (3-8 sentences unless the question requires more).",
|
||||
"- If text is visible, transcribe the relevant portion verbatim (do not translate).",
|
||||
"- If the image is unclear or empty, say so explicitly. Don't invent.",
|
||||
"- Do not call any tool besides Read on the provided path.",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
/** Spawn `claude -p` and return the JSON output result. */
|
||||
function callClaudeCli(prompt: string): Promise<{ result: string; durationMs: number; costUsd?: number; tokensIn?: number; tokensOut?: number }> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const t0 = Date.now();
|
||||
const child = spawn(
|
||||
"claude",
|
||||
[
|
||||
"-p",
|
||||
"--model", MODEL,
|
||||
"--output-format", "json",
|
||||
"--max-turns", "2",
|
||||
"--allowedTools", "Read",
|
||||
"--",
|
||||
prompt,
|
||||
],
|
||||
{ stdio: ["ignore", "pipe", "pipe"], env: { ...process.env } },
|
||||
);
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
child.stdout.on("data", (c) => (stdout += c.toString()));
|
||||
child.stderr.on("data", (c) => (stderr += c.toString()));
|
||||
const t = setTimeout(() => {
|
||||
child.kill("SIGKILL");
|
||||
reject(new Error(`claude CLI timeout > ${TIMEOUT_MS}ms`));
|
||||
}, TIMEOUT_MS);
|
||||
child.on("error", (e) => { clearTimeout(t); reject(e); });
|
||||
child.on("close", (code) => {
|
||||
clearTimeout(t);
|
||||
if (code !== 0) {
|
||||
return reject(new Error(`claude CLI rc=${code}: ${stderr.slice(-300)}`));
|
||||
}
|
||||
try {
|
||||
const cli = JSON.parse(stdout);
|
||||
if (cli.is_error) return reject(new Error(`claude error: ${(cli.result || "").slice(0, 300)}`));
|
||||
resolve({
|
||||
result: cli.result || "",
|
||||
durationMs: cli.duration_ms || Date.now() - t0,
|
||||
costUsd: cli.total_cost_usd,
|
||||
tokensIn: cli.usage?.input_tokens,
|
||||
tokensOut: cli.usage?.output_tokens,
|
||||
});
|
||||
} catch (e) {
|
||||
reject(new Error(`claude stdout parse: ${e instanceof Error ? e.message : String(e)}`));
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
export async function analyzeImageRegion(args: AnalyzeImageRegionArgs): Promise<AnalyzeImageRegionResult> {
|
||||
if (!args.doc_id) throw new Error("doc_id required");
|
||||
if (!Number.isFinite(args.page) || args.page < 1) throw new Error("page must be >= 1");
|
||||
if (!args.bbox || !["x", "y", "w", "h"].every((k) => Number.isFinite((args.bbox as Record<string, unknown>)[k]))) {
|
||||
throw new Error("bbox {x,y,w,h} required (normalized 0..1)");
|
||||
}
|
||||
if (!args.question?.trim()) throw new Error("question required");
|
||||
|
||||
const { path: cropPath, dir } = await cropToTempFile(args);
|
||||
try {
|
||||
const t0 = Date.now();
|
||||
const prompt = buildPrompt(args, cropPath);
|
||||
const out = await callClaudeCli(prompt);
|
||||
const cropUrl =
|
||||
`/api/crop?doc=${encodeURIComponent(args.doc_id)}` +
|
||||
`&page=${args.page}&x=${args.bbox.x}&y=${args.bbox.y}&w=${args.bbox.w}&h=${args.bbox.h}&w_px=640`;
|
||||
return {
|
||||
answer: out.result.trim(),
|
||||
model: MODEL,
|
||||
duration_ms: out.durationMs || Date.now() - t0,
|
||||
bbox: args.bbox,
|
||||
crop_url: cropUrl,
|
||||
};
|
||||
} finally {
|
||||
// Best-effort cleanup. Crop is in $TMPDIR, OS will reap if we miss.
|
||||
unlink(cropPath).catch(() => undefined);
|
||||
rmdir(dir).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
|
@ -43,8 +43,24 @@ export interface HybridSearchOptions {
|
|||
recall_k?: number;
|
||||
/** Final list size after rerank (default 20). */
|
||||
top_k?: number;
|
||||
/** Skip reranker (faster, lower precision). */
|
||||
/** Skip reranker (faster, lower precision). Back-compat shortcut for
|
||||
* `rerank_strategy: "never"`. */
|
||||
no_rerank?: boolean;
|
||||
/**
|
||||
* W2-TD#8: rerank policy.
|
||||
* - "always" — always run the cross-encoder (highest precision,
|
||||
* slowest, 5–8s on CPU)
|
||||
* - "when_top_k_gt" — rerank only when `top_k > rerank_threshold`
|
||||
* (default threshold 15). RRF order from the RPC is
|
||||
* usually good enough for the tight head of results;
|
||||
* the reranker pays off when re-sorting a wider list.
|
||||
* This is the new default — autocomplete / chat
|
||||
* top-10 calls now skip rerank for free.
|
||||
* - "never" — same as `no_rerank: true`.
|
||||
*/
|
||||
rerank_strategy?: "always" | "when_top_k_gt" | "never";
|
||||
/** Threshold for `when_top_k_gt`. Default 15 (per ADR-001). */
|
||||
rerank_threshold?: number;
|
||||
}
|
||||
|
||||
export async function hybridSearch(opts: HybridSearchOptions): Promise<ChunkHit[]> {
|
||||
|
|
@ -58,8 +74,14 @@ export async function hybridSearch(opts: HybridSearchOptions): Promise<ChunkHit[
|
|||
recall_k = 100,
|
||||
top_k = 20,
|
||||
no_rerank = false,
|
||||
rerank_strategy = "when_top_k_gt",
|
||||
rerank_threshold = 15,
|
||||
} = opts;
|
||||
|
||||
// Effective strategy: explicit `no_rerank=true` always wins (back-compat).
|
||||
const strategy: "always" | "when_top_k_gt" | "never" =
|
||||
no_rerank ? "never" : rerank_strategy;
|
||||
|
||||
if (!query.trim()) return [];
|
||||
|
||||
// 1. Embed the query
|
||||
|
|
@ -89,9 +111,13 @@ export async function hybridSearch(opts: HybridSearchOptions): Promise<ChunkHit[
|
|||
if (rows.length === 0) return [];
|
||||
|
||||
// 3. Optional cross-encoder rerank for finer ordering. It's CPU-slow
|
||||
// (seconds per ~dozen candidates), so it's opt-in (rerank=1); the default
|
||||
// fast path trusts the RPC's RRF order over the already-gated candidates.
|
||||
if (no_rerank) {
|
||||
// (seconds per ~dozen candidates). Strategy resolution (W2-TD#8 / ADR-001):
|
||||
// - "never" → skip
|
||||
// - "when_top_k_gt" → skip when top_k ≤ threshold (RRF is good enough
|
||||
// for a small head)
|
||||
// - "always" → run unconditionally
|
||||
if (strategy === "never" ||
|
||||
(strategy === "when_top_k_gt" && top_k <= rerank_threshold)) {
|
||||
return rows.slice(0, top_k);
|
||||
}
|
||||
|
||||
|
|
|
|||
417
web/package-lock.json
generated
417
web/package-lock.json
generated
|
|
@ -26,7 +26,6 @@
|
|||
"pino": "^10.3.1",
|
||||
"react": "^19.0.0",
|
||||
"react-dom": "^19.0.0",
|
||||
"react-force-graph-2d": "^1.27.0",
|
||||
"react-markdown": "^9.0.0",
|
||||
"remark-gfm": "^4.0.0",
|
||||
"remark-wiki-link": "^2.0.1",
|
||||
|
|
@ -5993,12 +5992,6 @@
|
|||
"tslib": "^2.8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@tweenjs/tween.js": {
|
||||
"version": "25.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@tweenjs/tween.js/-/tween.js-25.0.0.tgz",
|
||||
"integrity": "sha512-XKLA6syeBUaPzx4j3qwMqzzq+V4uo72BnlbOjmuljLrRqdsd3qnzvZZoxvMHZ23ndsRS4aufU6JOZYpCbU6T1A==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/connect": {
|
||||
"version": "3.4.38",
|
||||
"resolved": "https://registry.npmjs.org/@types/connect/-/connect-3.4.38.tgz",
|
||||
|
|
@ -6135,15 +6128,6 @@
|
|||
"integrity": "sha512-mUFwbeTqrVgDQxFveS+df2yfap6iuP20NAKAsBt5jDEoOTDew+zwLAOilHCeQJOVSvmgCX4ogqIrA0mnyr08yQ==",
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/accessor-fn": {
|
||||
"version": "1.5.3",
|
||||
"resolved": "https://registry.npmjs.org/accessor-fn/-/accessor-fn-1.5.3.tgz",
|
||||
"integrity": "sha512-rkAofCwe/FvYFUlMB0v0gWmhqtfAtV1IUkdPbfhTUyYniu5LrC0A0UJkTH0Jv3S8SvwkmfuAlY+mQIJATdocMA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/acorn": {
|
||||
"version": "8.16.0",
|
||||
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.16.0.tgz",
|
||||
|
|
@ -6335,16 +6319,6 @@
|
|||
"node": ">=6.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/bezier-js": {
|
||||
"version": "6.1.4",
|
||||
"resolved": "https://registry.npmjs.org/bezier-js/-/bezier-js-6.1.4.tgz",
|
||||
"integrity": "sha512-PA0FW9ZpcHbojUCMu28z9Vg/fNkwTj5YhusSAjHHDfHDGLxJ6YUKrAN2vk1fP2MMOxVw4Oko16FMlRGVBGqLKg==",
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"type": "individual",
|
||||
"url": "https://github.com/Pomax/bezierjs/blob/master/FUNDING.md"
|
||||
}
|
||||
},
|
||||
"node_modules/binary-extensions": {
|
||||
"version": "2.3.0",
|
||||
"resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz",
|
||||
|
|
@ -6446,18 +6420,6 @@
|
|||
],
|
||||
"license": "CC-BY-4.0"
|
||||
},
|
||||
"node_modules/canvas-color-tracker": {
|
||||
"version": "1.3.2",
|
||||
"resolved": "https://registry.npmjs.org/canvas-color-tracker/-/canvas-color-tracker-1.3.2.tgz",
|
||||
"integrity": "sha512-ryQkDX26yJ3CXzb3hxUVNlg1NKE4REc5crLBq661Nxzr8TNd236SaEf2ffYLXyI5tSABSeguHLqcVq4vf9L3Zg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"tinycolor2": "^1.6.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/ccount": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/ccount/-/ccount-2.0.1.tgz",
|
||||
|
|
@ -6664,222 +6626,6 @@
|
|||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/d3-array": {
|
||||
"version": "3.2.4",
|
||||
"resolved": "https://registry.npmjs.org/d3-array/-/d3-array-3.2.4.tgz",
|
||||
"integrity": "sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"internmap": "1 - 2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-binarytree": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/d3-binarytree/-/d3-binarytree-1.0.2.tgz",
|
||||
"integrity": "sha512-cElUNH+sHu95L04m92pG73t2MEJXKu+GeKUN1TJkFsu93E5W8E9Sc3kHEGJKgenGvj19m6upSn2EunvMgMD2Yw==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/d3-color": {
|
||||
"version": "3.1.0",
|
||||
"resolved": "https://registry.npmjs.org/d3-color/-/d3-color-3.1.0.tgz",
|
||||
"integrity": "sha512-zg/chbXyeBtMQ1LbD/WSoW2DpC3I0mpmPdW+ynRTj/x2DAWYrIY7qeZIHidozwV24m4iavr15lNwIwLxRmOxhA==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-dispatch": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/d3-dispatch/-/d3-dispatch-3.0.1.tgz",
|
||||
"integrity": "sha512-rzUyPU/S7rwUflMyLc1ETDeBj0NRuHKKAcvukozwhshr6g6c5d8zh4c2gQjY2bZ0dXeGLWc1PF174P2tVvKhfg==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-drag": {
|
||||
"version": "3.0.0",
|
||||
"resolved": "https://registry.npmjs.org/d3-drag/-/d3-drag-3.0.0.tgz",
|
||||
"integrity": "sha512-pWbUJLdETVA8lQNJecMxoXfH6x+mO2UQo8rSmZ+QqxcbyA3hfeprFgIT//HW2nlHChWeIIMwS2Fq+gEARkhTkg==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"d3-dispatch": "1 - 3",
|
||||
"d3-selection": "3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-ease": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/d3-ease/-/d3-ease-3.0.1.tgz",
|
||||
"integrity": "sha512-wR/XK3D3XcLIZwpbvQwQ5fK+8Ykds1ip7A2Txe0yxncXSdq1L9skcG7blcedkOX+ZcgxGAmLX1FrRGbADwzi0w==",
|
||||
"license": "BSD-3-Clause",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-force-3d": {
|
||||
"version": "3.0.6",
|
||||
"resolved": "https://registry.npmjs.org/d3-force-3d/-/d3-force-3d-3.0.6.tgz",
|
||||
"integrity": "sha512-4tsKHUPLOVkyfEffZo1v6sFHvGFwAIIjt/W8IThbp08DYAsXZck+2pSHEG5W1+gQgEvFLdZkYvmJAbRM2EzMnA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"d3-binarytree": "1",
|
||||
"d3-dispatch": "1 - 3",
|
||||
"d3-octree": "1",
|
||||
"d3-quadtree": "1 - 3",
|
||||
"d3-timer": "1 - 3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-format": {
|
||||
"version": "3.1.2",
|
||||
"resolved": "https://registry.npmjs.org/d3-format/-/d3-format-3.1.2.tgz",
|
||||
"integrity": "sha512-AJDdYOdnyRDV5b6ArilzCPPwc1ejkHcoyFarqlPqT7zRYjhavcT3uSrqcMvsgh2CgoPbK3RCwyHaVyxYcP2Arg==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-interpolate": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/d3-interpolate/-/d3-interpolate-3.0.1.tgz",
|
||||
"integrity": "sha512-3bYs1rOD33uo8aqJfKP3JWPAibgw8Zm2+L9vBKEHJ2Rg+viTR7o5Mmv5mZcieN+FRYaAOWX5SJATX6k1PWz72g==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"d3-color": "1 - 3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-octree": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/d3-octree/-/d3-octree-1.1.0.tgz",
|
||||
"integrity": "sha512-F8gPlqpP+HwRPMO/8uOu5wjH110+6q4cgJvgJT6vlpy3BEaDIKlTZrgHKZSp/i1InRpVfh4puY/kvL6MxK930A==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/d3-quadtree": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/d3-quadtree/-/d3-quadtree-3.0.1.tgz",
|
||||
"integrity": "sha512-04xDrxQTDTCFwP5H6hRhsRcb9xxv2RzkcsygFzmkSIOJy3PeRJP7sNk3VRIbKXcog561P9oU0/rVH6vDROAgUw==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-scale": {
|
||||
"version": "4.0.2",
|
||||
"resolved": "https://registry.npmjs.org/d3-scale/-/d3-scale-4.0.2.tgz",
|
||||
"integrity": "sha512-GZW464g1SH7ag3Y7hXjf8RoUuAFIqklOAq3MRl4OaWabTFJY9PN/E1YklhXLh+OQ3fM9yS2nOkCoS+WLZ6kvxQ==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"d3-array": "2.10.0 - 3",
|
||||
"d3-format": "1 - 3",
|
||||
"d3-interpolate": "1.2.0 - 3",
|
||||
"d3-time": "2.1.1 - 3",
|
||||
"d3-time-format": "2 - 4"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-scale-chromatic": {
|
||||
"version": "3.1.0",
|
||||
"resolved": "https://registry.npmjs.org/d3-scale-chromatic/-/d3-scale-chromatic-3.1.0.tgz",
|
||||
"integrity": "sha512-A3s5PWiZ9YCXFye1o246KoscMWqf8BsD9eRiJ3He7C9OBaxKhAd5TFCdEx/7VbKtxxTsu//1mMJFrEt572cEyQ==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"d3-color": "1 - 3",
|
||||
"d3-interpolate": "1 - 3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-selection": {
|
||||
"version": "3.0.0",
|
||||
"resolved": "https://registry.npmjs.org/d3-selection/-/d3-selection-3.0.0.tgz",
|
||||
"integrity": "sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-time": {
|
||||
"version": "3.1.0",
|
||||
"resolved": "https://registry.npmjs.org/d3-time/-/d3-time-3.1.0.tgz",
|
||||
"integrity": "sha512-VqKjzBLejbSMT4IgbmVgDjpkYrNWUYJnbCGo874u7MMKIWsILRX+OpX/gTk8MqjpT1A/c6HY2dCA77ZN0lkQ2Q==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"d3-array": "2 - 3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-time-format": {
|
||||
"version": "4.1.0",
|
||||
"resolved": "https://registry.npmjs.org/d3-time-format/-/d3-time-format-4.1.0.tgz",
|
||||
"integrity": "sha512-dJxPBlzC7NugB2PDLwo9Q8JiTR3M3e4/XANkreKSUxF8vvXKqm1Yfq4Q5dl8budlunRVlUUaDUgFt7eA8D6NLg==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"d3-time": "1 - 3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-timer": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/d3-timer/-/d3-timer-3.0.1.tgz",
|
||||
"integrity": "sha512-ndfJ/JxxMd3nw31uyKoY2naivF+r29V+Lc0svZxe1JvvIRmi8hUsrMvdOwgS1o6uBHmiz91geQ0ylPP0aj1VUA==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-transition": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/d3-transition/-/d3-transition-3.0.1.tgz",
|
||||
"integrity": "sha512-ApKvfjsSR6tg06xrL434C0WydLr7JewBB3V+/39RMHsaXTOG0zmt/OAXeng5M5LBm0ojmxJrpomQVZ1aPvBL4w==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"d3-color": "1 - 3",
|
||||
"d3-dispatch": "1 - 3",
|
||||
"d3-ease": "1 - 3",
|
||||
"d3-interpolate": "1 - 3",
|
||||
"d3-timer": "1 - 3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"d3-selection": "2 - 3"
|
||||
}
|
||||
},
|
||||
"node_modules/d3-zoom": {
|
||||
"version": "3.0.0",
|
||||
"resolved": "https://registry.npmjs.org/d3-zoom/-/d3-zoom-3.0.0.tgz",
|
||||
"integrity": "sha512-b8AmV3kfQaqWAuacbPuNbL6vahnOJflOhexLzMMNLga62+/nh0JzvJ0aO/5a5MVgUFGS7Hu1P9P03o3fJkDCyw==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"d3-dispatch": "1 - 3",
|
||||
"d3-drag": "2 - 3",
|
||||
"d3-interpolate": "1 - 3",
|
||||
"d3-selection": "2 - 3",
|
||||
"d3-transition": "2 - 3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/debug": {
|
||||
"version": "4.4.3",
|
||||
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
|
||||
|
|
@ -7194,46 +6940,6 @@
|
|||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/float-tooltip": {
|
||||
"version": "1.7.5",
|
||||
"resolved": "https://registry.npmjs.org/float-tooltip/-/float-tooltip-1.7.5.tgz",
|
||||
"integrity": "sha512-/kXzuDnnBqyyWyhDMH7+PfP8J/oXiAavGzcRxASOMRHFuReDtofizLLJsf7nnDLAfEaMW4pVWaXrAjtnglpEkg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"d3-selection": "2 - 3",
|
||||
"kapsule": "^1.16",
|
||||
"preact": "10"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/force-graph": {
|
||||
"version": "1.51.4",
|
||||
"resolved": "https://registry.npmjs.org/force-graph/-/force-graph-1.51.4.tgz",
|
||||
"integrity": "sha512-TdJ2KbkoiDQ7NIRx8IPGD0mAXXpLhamS7c+b7W98b0MHG7lphnda1VOQX/98UDTsttIAdH4TcP0l0MauSnLK8w==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@tweenjs/tween.js": "18 - 25",
|
||||
"accessor-fn": "1",
|
||||
"bezier-js": "3 - 6",
|
||||
"canvas-color-tracker": "^1.3",
|
||||
"d3-array": "1 - 3",
|
||||
"d3-drag": "2 - 3",
|
||||
"d3-force-3d": "2 - 3",
|
||||
"d3-scale": "1 - 4",
|
||||
"d3-scale-chromatic": "1 - 3",
|
||||
"d3-selection": "2 - 3",
|
||||
"d3-zoom": "2 - 3",
|
||||
"float-tooltip": "^1.7",
|
||||
"index-array-by": "1",
|
||||
"kapsule": "^1.16",
|
||||
"lodash-es": "4"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/forwarded-parse": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/forwarded-parse/-/forwarded-parse-2.1.2.tgz",
|
||||
|
|
@ -7509,30 +7215,12 @@
|
|||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/index-array-by": {
|
||||
"version": "1.4.2",
|
||||
"resolved": "https://registry.npmjs.org/index-array-by/-/index-array-by-1.4.2.tgz",
|
||||
"integrity": "sha512-SP23P27OUKzXWEC/TOyWlwLviofQkCSCKONnc62eItjp69yCZZPqDQtr3Pw5gJDnPeUMqExmKydNZaJO0FU9pw==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/inline-style-parser": {
|
||||
"version": "0.2.7",
|
||||
"resolved": "https://registry.npmjs.org/inline-style-parser/-/inline-style-parser-0.2.7.tgz",
|
||||
"integrity": "sha512-Nb2ctOyNR8DqQoR0OwRG95uNWIC0C1lCgf5Naz5H6Ji72KZ8OcFZLz2P5sNgwlyoJ8Yif11oMuYs5pBQa86csA==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/internmap": {
|
||||
"version": "2.0.3",
|
||||
"resolved": "https://registry.npmjs.org/internmap/-/internmap-2.0.3.tgz",
|
||||
"integrity": "sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/is-alphabetical": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/is-alphabetical/-/is-alphabetical-2.0.1.tgz",
|
||||
|
|
@ -7681,15 +7369,6 @@
|
|||
"integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==",
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/jerrypick": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/jerrypick/-/jerrypick-1.1.2.tgz",
|
||||
"integrity": "sha512-YKnxXEekXKzhpf7CLYA0A+oDP8V0OhICNCr5lv96FvSsDEmrb0GKM776JgQvHTMjr7DTTPEVv/1Ciaw0uEWzBA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/jiti": {
|
||||
"version": "1.21.7",
|
||||
"resolved": "https://registry.npmjs.org/jiti/-/jiti-1.21.7.tgz",
|
||||
|
|
@ -7743,18 +7422,6 @@
|
|||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/kapsule": {
|
||||
"version": "1.16.3",
|
||||
"resolved": "https://registry.npmjs.org/kapsule/-/kapsule-1.16.3.tgz",
|
||||
"integrity": "sha512-4+5mNNf4vZDSwPhKprKwz3330iisPrb08JyMgbsdFrimBCKNHecua/WBwvVg3n7vwx0C1ARjfhwIpbrbd9n5wg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"lodash-es": "4"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/kind-of": {
|
||||
"version": "6.0.3",
|
||||
"resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz",
|
||||
|
|
@ -7799,12 +7466,6 @@
|
|||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/lodash-es": {
|
||||
"version": "4.18.1",
|
||||
"resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.18.1.tgz",
|
||||
"integrity": "sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/longest-streak": {
|
||||
"version": "3.1.0",
|
||||
"resolved": "https://registry.npmjs.org/longest-streak/-/longest-streak-3.1.0.tgz",
|
||||
|
|
@ -7815,18 +7476,6 @@
|
|||
"url": "https://github.com/sponsors/wooorm"
|
||||
}
|
||||
},
|
||||
"node_modules/loose-envify": {
|
||||
"version": "1.4.0",
|
||||
"resolved": "https://registry.npmjs.org/loose-envify/-/loose-envify-1.4.0.tgz",
|
||||
"integrity": "sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"js-tokens": "^3.0.0 || ^4.0.0"
|
||||
},
|
||||
"bin": {
|
||||
"loose-envify": "cli.js"
|
||||
}
|
||||
},
|
||||
"node_modules/lru-cache": {
|
||||
"version": "5.1.1",
|
||||
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-5.1.1.tgz",
|
||||
|
|
@ -9511,6 +9160,7 @@
|
|||
"version": "4.1.1",
|
||||
"resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
|
||||
"integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
|
|
@ -10023,16 +9673,6 @@
|
|||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/preact": {
|
||||
"version": "10.29.1",
|
||||
"resolved": "https://registry.npmjs.org/preact/-/preact-10.29.1.tgz",
|
||||
"integrity": "sha512-gQCLc/vWroE8lIpleXtdJhTFDogTdZG9AjMUpVkDf2iTCNwYNWA+u16dL41TqUDJO4gm2IgrcMv3uTpjd4Pwmg==",
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/preact"
|
||||
}
|
||||
},
|
||||
"node_modules/process-warning": {
|
||||
"version": "5.0.0",
|
||||
"resolved": "https://registry.npmjs.org/process-warning/-/process-warning-5.0.0.tgz",
|
||||
|
|
@ -10058,17 +9698,6 @@
|
|||
"node": ">=0.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/prop-types": {
|
||||
"version": "15.8.1",
|
||||
"resolved": "https://registry.npmjs.org/prop-types/-/prop-types-15.8.1.tgz",
|
||||
"integrity": "sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"loose-envify": "^1.4.0",
|
||||
"object-assign": "^4.1.1",
|
||||
"react-is": "^16.13.1"
|
||||
}
|
||||
},
|
||||
"node_modules/property-information": {
|
||||
"version": "7.1.0",
|
||||
"resolved": "https://registry.npmjs.org/property-information/-/property-information-7.1.0.tgz",
|
||||
|
|
@ -10248,44 +9877,6 @@
|
|||
"react": "^19.2.6"
|
||||
}
|
||||
},
|
||||
"node_modules/react-force-graph-2d": {
|
||||
"version": "1.29.1",
|
||||
"resolved": "https://registry.npmjs.org/react-force-graph-2d/-/react-force-graph-2d-1.29.1.tgz",
|
||||
"integrity": "sha512-1Rl/1Z3xy2iTHKj6a0jRXGyiI86xUti81K+jBQZ+Oe46csaMikp47L5AjrzA9hY9fNGD63X8ffrqnvaORukCuQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"force-graph": "^1.51",
|
||||
"prop-types": "15",
|
||||
"react-kapsule": "^2.5"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"react": "*"
|
||||
}
|
||||
},
|
||||
"node_modules/react-is": {
|
||||
"version": "16.13.1",
|
||||
"resolved": "https://registry.npmjs.org/react-is/-/react-is-16.13.1.tgz",
|
||||
"integrity": "sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/react-kapsule": {
|
||||
"version": "2.5.7",
|
||||
"resolved": "https://registry.npmjs.org/react-kapsule/-/react-kapsule-2.5.7.tgz",
|
||||
"integrity": "sha512-kifAF4ZPD77qZKc4CKLmozq6GY1sBzPEJTIJb0wWFK6HsePJatK3jXplZn2eeAt3x67CDozgi7/rO8fNQ/AL7A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"jerrypick": "^1.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"react": ">=16.13.1"
|
||||
}
|
||||
},
|
||||
"node_modules/react-markdown": {
|
||||
"version": "9.1.0",
|
||||
"resolved": "https://registry.npmjs.org/react-markdown/-/react-markdown-9.1.0.tgz",
|
||||
|
|
@ -10991,12 +10582,6 @@
|
|||
"integrity": "sha512-P4nbQYQfePJxRSmY+v/KINxVucm4NF3p3s7pJveMTtom52FR4YGltUQLB8idDXwDDWW+eYrWDFbuzUnjoWHF7g==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/tinycolor2": {
|
||||
"version": "1.6.0",
|
||||
"resolved": "https://registry.npmjs.org/tinycolor2/-/tinycolor2-1.6.0.tgz",
|
||||
"integrity": "sha512-XPaBkWQJdsf3pLKJV9p4qN/S+fm2Oj8AIPo1BTUhg5oxkvm9+SVEGFdhyOz7tTdUTfvxMiAs4sp6/eZO2Ew+pw==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/tinyglobby": {
|
||||
"version": "0.2.16",
|
||||
"resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz",
|
||||
|
|
|
|||
|
|
@ -28,7 +28,6 @@
|
|||
"pino": "^10.3.1",
|
||||
"react": "^19.0.0",
|
||||
"react-dom": "^19.0.0",
|
||||
"react-force-graph-2d": "^1.27.0",
|
||||
"react-markdown": "^9.0.0",
|
||||
"remark-gfm": "^4.0.0",
|
||||
"remark-wiki-link": "^2.0.1",
|
||||
|
|
|
|||
Loading…
Reference in a new issue