diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml index 35ed31b..f20d48f 100644 --- a/.forgejo/workflows/ci.yml +++ b/.forgejo/workflows/ci.yml @@ -68,3 +68,16 @@ jobs: steps: - uses: actions/checkout@v4 - run: npm audit --production --omit=dev --audit-level=high || echo "audit findings — see job output" + + rag-eval: + name: Retrieval — golden set (Recall@5 + MRR) + runs-on: ubuntu-latest + container: + image: python:3.11-bookworm + steps: + - uses: actions/checkout@v4 + - run: pip install --quiet pyyaml + - name: Run RAG eval against production + run: python3 tests/rag/run.py --url https://disclosure.top --top-k 5 --rerank never + env: + MAX_RECALL_DROP: "0.05" diff --git a/CHANGELOG.md b/CHANGELOG.md index 085ecbc..be22467 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,100 @@ All notable changes to this project go here. Newest on top. ## [Unreleased] +### W2 — UX latency + retrieval eval + vision tool +*2026-05-23 · systems-atelier engagement trace `794f00ba`* + +- **TD#8 · Reranker opt-in** (`hybrid.ts`). New `rerank_strategy` field + on `HybridSearchOptions`: `"always" | "when_top_k_gt" | "never"`, with + a configurable `rerank_threshold` (default 15). Default strategy is + `when_top_k_gt` so the slow cross-encoder only runs when the model + asks for a wider list; top-K ≤ 15 trusts the RPC's RRF order. The + chat tool calls hybrid_search with threshold 10 so a 10-hit response + costs ~7s of embed+RPC instead of 12-15s with rerank. `/api/search/hybrid` + exposes the strategy via `?rerank=always|never|when_top_k_gt` plus + `?rerank_threshold=N`. Back-compat `?rerank=0` still means "never". +- **O11 · `analyze_image_region` chat tool** (`vision.ts`, `tools.ts`). + New OpenAI-style function tool that crops a normalized bbox of a page + PNG with sharp, writes it to a temp file, and asks Claude Code OAuth + (Sonnet) to Read the local file and answer a question about it. + Schema: `{doc_id, page, bbox{x,y,w,h}, question, context?}`. Emits a + `crop_image` artifact for the UI alongside the textual answer. Cost + budget: ~$0.005–0.02 per call, paid against the user's Max 20x + quota. Timeout configurable via `VISION_TIMEOUT_MS` (default 120s). +- **TD#12 · `react-force-graph-2d` removed**. The `/graph` page now uses + `` (already wired for the entity sidebar). One graph + library is enough. `web/components/force-graph-canvas.tsx` deleted; + `npm uninstall` removed 37 transitive deps. +- **TD#27 · Context truncation per type configurable** + (`messages/route.ts`). The four `gatherContext` slice limits are now + driven by env (`CTX_DOC_FRONTMATTER`, `CTX_DOC_BODY`, + `CTX_PAGE_FRONTMATTER`, `CTX_PAGE_BODY`) with sensible production + defaults (was hard-coded 1200/1500/1500/1500). +- **TD#22 · Golden RAG eval** (`tests/rag/`). New harness: + `golden.yaml` carries 15 curated queries (some calibrated to the + current top-1 hit on prod, some negative-set sentinels like + `MJ-12` / `tic-tac` that should NOT return matches), `run.py` + measures `Recall@k` + `MRR` + `negative_pass_rate` against any + deployment URL, `baseline.json` is the gate threshold, `last_run.json` + is the working report. Default behaviour: fail the run when Recall@5 + drops > 0.05 from baseline. CI workflow runs against + `https://disclosure.top` on every push. + - First baseline (rerank=never): **Recall@5 = 0.2083, MRR = 0.25, + Negative pass = 1.0**. Golden set still needs curation — + intentionally conservative now so drift detection is meaningful. +- **ADRs published to `docs/adrs/`** — ADR-001 (embedding + rerank stack), + ADR-002 (Investigation Bureau runtime — Bun + LISTEN/NOTIFY + 8 security + gates, to be implemented in W3), ADR-003 (LLM routing policy), ADR-004 + (auth + RLS evolution), ADR-005 (self-hosted by default). + +#### Verified on `disclosure.top` (2026-05-23T21:55Z): +- `/api/search/hybrid?q=Roswell&top_k=5` → HTTP 200 in 6.7s (embed-only, + rerank skipped per default strategy) +- `/api/search/hybrid?q=Roswell&top_k=20&rerank=always` → confirmed slow + (>30s, hits cross-encoder) +- Typecheck `web/` clean; `react-force-graph-2d` no longer in + `package.json` +- `tests/rag/run.py` against prod → 15 queries answered, baseline written +- 5 ADRs committed under `docs/adrs/` + +### W1.2 — Glitchtip + Forgejo self-hosted +*2026-05-23 · systems-atelier engagement trace `794f00ba`* + +- **Glitchtip self-host** (Sentry-compatible error monitor). New services + in compose: `glitchtip-redis`, `glitchtip-web`, `glitchtip-worker` + (v4.2, uWSGI on 8080). Database `glitchtip` carved out of + `disclosure-db` as a separate role/DB. Bootstrap done via Django + `manage.py shell` — admin user, organization `the-disclosure-bureau`, + project `web`, DSN issued. SDK wired: `@sentry/nextjs` + `instrumentation.ts` + + `sentry.{client,server}.config.ts`. `/api/admin/throw` smoke endpoint + is admin-gated. Live at `https://glitchtip.disclosure.top` (TLS issued + by Let's Encrypt via Traefik). Synthetic event verified — POST + `/api/1/store/` → 200 + event id. +- **Forgejo self-host + Actions CI**. New services in compose: `forgejo` + (v9, default branch `main`) and `forgejo-runner` (v6, registered to + the host docker socket via `group_add: [988]`). Admin user + `discadmin` created via `forgejo admin user create` (the literal + `admin` is reserved). Runner bootstrap on first start: registers if + `.runner` absent, then `forgejo-runner daemon`. Repo + `discadmin/disclosure-bureau` created via API; this commit was the + first push and triggered `W0+W1+W1.2: …` workflow at task 1. +- **`.forgejo/workflows/ci.yml`** — three jobs: `web` (typecheck + + lint + production build), `python` (compile scripts + validate + compose YAML), `audit` (`npm audit --production`). Default container + per job, all behind the `ubuntu-latest` label served by the + self-hosted runner. + +#### Verified on the stack (2026-05-23T21:19Z): +- `glitchtip.disclosure.top` → HTTP 200, real Let's Encrypt cert, + Glitchtip CSP headers present. +- POST `/api/1/store/` → 200, event_id `cb17d723…` returned. +- `forgejo.disclosure.top` → HTTP 200, Forgejo welcome page. +- Forgejo runner logs: `runner: disclosure-runner … declared + successfully`, `[poller 0] launched`, `task 1 repo is + discadmin/disclosure-bureau` (CI job picked up). +- First Forgejo Actions workflow run: `status=running` on the commit + pushed by this changelog. + ### W1 — Observability + resilience + Meili autocomplete *2026-05-23 · systems-atelier engagement trace `794f00ba`* diff --git a/docs/adrs/ADR-001-embedding-rerank-stack.md b/docs/adrs/ADR-001-embedding-rerank-stack.md new file mode 100644 index 0000000..ef696eb --- /dev/null +++ b/docs/adrs/ADR-001-embedding-rerank-stack.md @@ -0,0 +1,56 @@ +--- +adr: ADR-001 +title: Embedding e reranker stack — manter BGE-M3 self-hosted CPU; tornar reranker opt-in; reavaliar GPU em 6 meses +status: accepted +date: 2026-05-23 +deciders: sa-principal, sa-architecture-lead, sa-platform-lead +project: disclosure-bureau +--- + +## Context + +A retrieval pipeline atual (BGE-M3 dense + BM25 + RRF + BGE-Reranker-v2-M3 cross-encoder) entrega `Recall@5` aceitavel para 20.935 chunks, mas o reranker (cross-encoder em CPU) consome **5-8s** por consulta com 100 candidatos. Esse e o **gargalo dominante de UX** no chat sincrono. + +Alternativas conhecidas: + +1. **Manter status quo**: CPU rerank sempre on. +2. **Skip rerank**: confiar so em RRF do RPC `hybrid_search_chunks`. Mais rapido, perde precisao em queries ambiguas. +3. **Switch para ColBERT-late-interaction** (PyTerrier / RAGatouille) — rerank built-in no recall, sem segundo modelo. +4. **GPU para reranker**: VPS com GPU pequena (Hetzner GPU-1: ~$20/mes). BGE-Reranker em GPU caem de 5-8s para <1s. +5. **External managed (Voyage, Cohere)**: viola politica "self-hosted by default". + +## Decision + +1. **Manter BGE-M3 self-hosted CPU para embeddings** (sem mudanca; 150-300ms warm e ok). +2. **Tornar reranker opt-in por chamada**: + - Default: skip rerank quando `top_k <= 10` (RPC RRF e suficiente para top resultados). + - Aplicar rerank quando `top_k > 10` OU explicit `rerank=1` no API. +3. **Avaliar GPU em 6 meses** (Q4 2026) com criterio: se rerank latencia p95 > 4s ou usuario base > 1000 DAU. Se ambos, provisionar GPU 1. +4. **ColBERT como plano B**: catalogar em `infra/research/` mas nao trocar agora (risco de regressao de qualidade). +5. **Continuar BGE-M3 multi-lingua**: nao trocar para modelo english-only mesmo que mais rapido — corpus e bilingue. + +## Consequences + +**Positivas:** +- Latencia mediana p95 do chat cai de ~10s para ~6s (estimativa baseada em remocao do rerank para top_k <=10). +- Custo continua $0/mes alem do VPS (sem GPU upgrade). +- Reranker continua disponivel para queries complexas (`top_k=20+`). + +**Negativas:** +- Quando user pede explicitamente "top 20 results", latencia volta a ser 8s. +- Recall@5 pode cair marginalmente em queries muito ambiguas. Ver eval harness W2. + +**Trade-off aceito:** UX media melhora; UX pior caso mantem. Eval harness do W2 vai pegar regressao real. + +## Verification + +- `tests/rag/golden.yaml` mede Recall@5 antes/depois. +- Sentry timing histogram `chat_query_latency_ms` p95 antes/depois. +- Manual smoke test: 5 queries cobrindo cada `top_k` bucket. + +## References + +- `infra/RETRIEVAL.md` (performance budget). +- `web/lib/retrieval/hybrid.ts` (codigo). +- BGE-M3 paper: arxiv:2402.03216. +- ColBERT-late-interaction: arxiv:2004.12832. diff --git a/docs/adrs/ADR-002-investigation-bureau-runtime.md b/docs/adrs/ADR-002-investigation-bureau-runtime.md new file mode 100644 index 0000000..ed5f74c --- /dev/null +++ b/docs/adrs/ADR-002-investigation-bureau-runtime.md @@ -0,0 +1,77 @@ +--- +adr: ADR-002 +title: Materializar Investigation Bureau — runtime agentico em background, 8 detetives como roles +status: accepted +date: 2026-05-23 +deciders: sa-principal, sa-architecture-lead, sa-security-engineer (veto power) +project: disclosure-bureau +--- + +## Context + +O branding "The Disclosure Bureau" promete "8 detetives investigativos" (Holmes/Poirot/Dupin/Locard/Schneier/Tetlock/Taleb + Investigation Bureau coletivo) com chain of custody, hypothesis tournament, residual uncertainty calculation. Hoje, o codebase tem: + +- `case/` filesystem com 6 pastas — 5 vazias, 1 com 2 gap files. +- Chat com 12 tools read-only e um system prompt grandioso. +- AG-UI artifact types `evidence_card`, `hypothesis_card`, `case_card` definidos mas **nao emitidos**. +- Zero detetives implementados como entidades operacionais distintas. + +O brief pede: "AI detective bureau REAL, nao decorativo". Isso requer **producao** de dado novo (`case/evidence/*.md`, `case/hypotheses/*.md`, `public.{hypotheses,evidence,contradictions,...}`) por **agentes especializados** com **outputs estruturados e auditaveis**. + +Decisao de fronteira: a camada agentica vive **em paralelo** ao chat sincrono ou e **parte dele**? + +## Options considered + +1. **Parte do chat sincrono.** Estender system prompt + adicionar write tools. Usuario espera 30s-5min sincrono. +2. **Worker em background.** Chat dispara job; usuario polls; worker assincrono produz outputs. +3. **Sem agentic layer**: manter so chat read-only. Refatorar branding para refletir realidade ("AI-assisted wiki"). +4. **CronJob batch only**. Sem trigger user. Investigacoes acontecem em background diario. + +## Decision + +**Opcao 2: Worker em background, separado do chat sincrono.** + +Especificamente: + +1. **Novo container `investigator-runtime`** (Bun + TS) no docker-compose, isolado de Next.js. +2. **8 detetives + chief-detective como roles** distintos: cada um e um `claude -p` subprocess com `prompts/.md` proprio e toolset distinto (subset de tools comuns + 1-2 writers especificos). +3. **Postgres LISTEN/NOTIFY** como queue (`public.investigation_jobs` + trigger NOTIFY). +4. **Triggers de job** (sec 6 do agentic-layer-spec): cron diario, evento ingest, user via chat (`request_investigation` tool), admin manual. +5. **Tools de write gated** (8 gates do sa-security-engineer; ver `security-audit-report.md` secao 5). +6. **Budget cap por job:** $1.00 hard ceiling (Sonnet via OAuth Max 20x preferido; Anthropic API paid como fallback). +7. **Outputs validados antes de commit:** schema check + lint (`04-lint.py --dry-run`) sobre markdown gerado. + +**Nao adotamos:** + +- Opcao 1 (estender chat sincrono): user nao pode esperar 5 min num chat. Quebra modelo mental. +- Opcao 3 (sem agentic): foge do brief explicito. Branding sem motor e desonesto. +- Opcao 4 (cron only): sem trigger user e UX pobre. Manter cron como complementar, nao exclusivo. + +## Consequences + +**Positivas:** +- Branding "8 detetives" passa a ter motor real. +- Chat sincrono continua rapido (LLM read-only + 12 tools). +- Investigacoes profundas geram dado novo, persistente, auditavel — Investigation Bureau "de verdade". +- Cold-case revival, contradiction detection, residual uncertainty — features que viralizam. + +**Negativas:** +- Novo container = nova superficie operacional (~150MB RAM extra; orchestrator + state). +- Quota Claude Max 20x mais utilizada (ja monitorada por `/api/admin/batch`). +- Schema cresce: 7 novas tabelas (hypotheses, evidence, contradictions, witnesses, gaps, residual_uncertainties, investigation_jobs). +- Risco de hallucination em writers — mitigado por gates sa-security (validacao schema + ref). + +## Verification + +- Spec completa em `agentic-layer-spec.md`. +- Plano de bring-up incremental em 10 sub-steps W3.1-W3.10. +- 8 gates documentados para sa-security veto. +- Custos esperados $30-110/mes (tabela secao 11 do spec). +- Golden hypothesis set como quality bar (W3.10). + +## References + +- `agentic-layer-spec.md` +- `ai-opportunity-map.md` O1-O5 +- `security-audit-report.md` secao 5 +- Anthropic Claude Code OAuth pattern (memoria do projeto) diff --git a/docs/adrs/ADR-003-llm-routing-policy.md b/docs/adrs/ADR-003-llm-routing-policy.md new file mode 100644 index 0000000..0c2440b --- /dev/null +++ b/docs/adrs/ADR-003-llm-routing-policy.md @@ -0,0 +1,72 @@ +--- +adr: ADR-003 +title: LLM routing policy — Claude Sonnet 4.6 via OAuth para producao asincrona; OpenRouter free para chat publico +status: accepted +date: 2026-05-23 +deciders: sa-principal, sa-platform-lead +project: disclosure-bureau +--- + +## Context + +Tres caminhos de LLM no projeto: + +1. **Vision pipeline (ingest)**: Sonnet 4.6 via Anthropic SDK + prompt caching + `pdf-2025-03-04` beta. Custo unico ~$409 inicial. +2. **Chat sincrono (user-facing)**: hoje OpenRouter free (`deepseek/deepseek-v4-flash:free` primario, `nvidia/nemotron-3-super-120b-a12b:free` fallback). Tool calling funciona. +3. **Investigation Bureau (W3+ a implementar)**: propostas: Sonnet 4.6 via OAuth Max 20x. + +Restricoes existentes: + +- **Politica banida Gemini** ([memoria do projeto](file:///Users/guto/.claude/projects/-Users-guto-ufo/memory/MEMORY.md)). Cobranca de ~$200 vs $10 esperado. +- **OAuth Max 20x quota**: 5h rolling window, default 4 workers ([memoria](file:///Users/guto/.claude/projects/-Users-guto-ufo/memory/MEMORY.md)). +- **Self-hosted by default**: managed proibido sem excecao escrita (ADR-005). + +## Decision + +**Roteamento por canal e por carga:** + +| Canal | Provider | Modelo | Razao | +|---|---|---|---| +| Vision pipeline (background) | Anthropic SDK direto | Sonnet 4.6 | API key valid; cache + beta header; nao usa quota OAuth | +| Chat sincrono publico | OpenRouter | deepseek-v4-flash:free, nemotron fallback | Free tier; tool calling; usuario anonimo | +| Chat sincrono autenticado (futuro premium) | OpenRouter ou Anthropic API direta | configurable | Tier paid quando justificado | +| Investigation Bureau (W3+) | **Claude Code OAuth (subprocess `claude -p`)** | Sonnet 4.6 (model: sonnet) | Quota Max 20x; budget cap por job $1.00; preferido sobre paid API | +| Investigation Bureau — overflow | Anthropic SDK paid | Sonnet 4.6 ou Haiku | Quando OAuth quota saturada AND `BUDGET_PAID_ALLOWED=true` | +| LLM judge interno (calibration / contradiction detection) | Claude OAuth ou OpenRouter | Haiku (cheap, fast) | Tarefa simples, batch | + +**Politica de fallback:** + +1. Primary tenta. Se 429/quota -> 1 retry com backoff. +2. Apos retry falhar: fallback policy: + - Chat sincrono: troca OpenRouter primary -> OpenRouter fallback. Se ambos falham, retorna erro UX. + - Vision/investigator: aborta job, registra em `investigation_jobs.status='failed'`. Aguarda quota reset (5h). +3. `/api/admin/batch` ja monitora 429 + ETA quota reset. + +**Excecoes:** + +- Gemini **banido** (politica). Nao reativar mesmo se nova versao for atrativa. +- Anthropic API key paid SO em variavel de ambiente separada (`ANTHROPIC_API_KEY_PAID`) — exige `--paid` flag explicito. + +## Consequences + +**Positivas:** +- Investigation Bureau pode operar 99% do tempo em quota OAuth (gratuita para o projeto). +- Chat sincrono publico continua $0/req. +- Separacao clara entre "sob quota" e "paid" — facil monitorar gasto. + +**Negativas:** +- OpenRouter free-tier tem rate limits + latencia variavel. Mitigacao em W1 (retry + circuit breaker). +- Quota saturation no Sonnet OAuth quando muitos workers ingestam + investigador roda em paralelo. Cron diario investigador as 03-05 UTC quando ingest e baixa. + +## Verification + +- Logs Sentry mostram `model_used` em cada chat call. +- `/api/admin/batch` mostra `quota_state` + `quota_resume_eta_minutes`. +- `investigation_jobs.outputs` registra `model` para cada turno. +- Budget alert em $150/mes Anthropic API se cair em paid fallback. + +## References + +- `feedback-no-gemini-ever.md` (memoria) +- `user-plan-max-20x.md` (memoria) +- `web/lib/chat/{index,openrouter,claude-code}.ts` diff --git a/docs/adrs/ADR-004-auth-and-rls-evolution.md b/docs/adrs/ADR-004-auth-and-rls-evolution.md new file mode 100644 index 0000000..7cc0adf --- /dev/null +++ b/docs/adrs/ADR-004-auth-and-rls-evolution.md @@ -0,0 +1,106 @@ +--- +adr: ADR-004 +title: Auth model evolution — manter Supabase GoTrue + RLS publico-read; novas tabelas case/* write-only por investigator role +status: accepted +date: 2026-05-23 +deciders: sa-principal, sa-security-engineer, sa-platform-lead +project: disclosure-bureau +--- + +## Context + +Modelo de auth atual: + +- **GoTrue email magic-link**, SMTP Spacemail, `MAILER_AUTOCONFIRM=false`. +- **profiles.role ∈ {user, admin, suspended}**, com `budget_cap_usd`, `daily_quota`. +- **Sessoes `is_public=true`** sao readable por anon (compartilhaveis via share_token UUID). +- **RLS publico-read** em chunks/entities/documents/entity_mentions/relations. +- **service_role key em env do container web** (necessario para usage_events INSERT bypass RLS + admin tasks). F5 do security audit. + +Decisao de fronteira na W3: novas tabelas (`hypotheses`, `evidence`, `contradictions`, `witnesses`, `gaps`, `residual_uncertainties`, `investigation_jobs`) — quem escreve? + +## Options + +1. **service_role escreve tudo** (mesmo padrao atual). Investigator-runtime usa service_role. +2. **Role intermediario `investigator`** com permissoes minimas. Investigator-runtime usa esse role; web nao. +3. **Investigator usa Postgres direto sem RLS**: bypass desde o nivel de conexao. + +## Decision + +**Opcao 2: Role `investigator` granular.** + +Criar role Postgres minimo: + +```sql +CREATE ROLE investigator LOGIN NOINHERIT PASSWORD :'INVESTIGATOR_PASSWORD'; +-- Reads (mesmo que anon ja tem, mas explicito) +GRANT SELECT ON public.chunks, public.entities, public.entity_mentions, + public.relations, public.documents TO investigator; +-- Writes em tabelas case/* +GRANT INSERT, UPDATE ON public.hypotheses, public.evidence, public.contradictions, + public.witnesses, public.gaps, public.residual_uncertainties, + public.investigation_jobs TO investigator; +GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO investigator; +-- Negar tudo o resto +REVOKE ALL ON public.profiles, public.chat_sessions, public.messages, + public.usage_events FROM investigator; +REVOKE ALL ON SCHEMA auth FROM investigator; +``` + +E: + +- Investigator-runtime container usa `postgres://investigator:${INVESTIGATOR_PASSWORD}@db:5432/postgres` (NUNCA service_role). +- Web service continua com `postgres://postgres:...` (acesso amplo necessario para createServiceClient nos pontos especificos). +- Em W5, **avaliar reducao de uso de service_role no web** criando role `web_service` similar com permissoes ainda menores que `postgres`. + +**RLS nas novas tabelas:** + +```sql +ALTER TABLE public.hypotheses ENABLE ROW LEVEL SECURITY; +CREATE POLICY hypotheses_public_read ON public.hypotheses FOR SELECT USING (TRUE); +GRANT SELECT ON public.hypotheses TO anon, authenticated; +-- Investigator role usa bypass via INSERT/UPDATE direto (sem RLS aplica em writes do owner role; ele tem GRANT) +``` + +Mesmo padrao para todas as 7 novas tabelas. + +**Modificacoes nas existentes:** + +- `relations` ganha RLS (F4). Hoje sem. +- `messages.citations JSONB` continua mas adicionar coluna `hypothesis_id REFERENCES public.hypotheses(hypothesis_id)` se chat citar hipoteses geradas. + +**Sessions publicas:** + +Decisao do F6 (moderation_state) fica fora deste ADR porque e produto/legal, nao auth. Mas RLS aceitara extensao: + +```sql +CREATE POLICY public_sessions_select ON public.chat_sessions + FOR SELECT USING ( + auth.uid() = user_id OR + (is_public = TRUE AND COALESCE(moderation_state, 'approved') IN ('approved')) + ); +``` + +## Consequences + +**Positivas:** +- Blast radius do investigator-runtime menor (RCE no container nao acesso ao auth.users / profiles). +- Auditabilidade granular: cada INSERT em hypotheses/evidence tem `created_by = '@detective'` + role Postgres `investigator`. +- Aderencia a least-privilege. + +**Negativas:** +- 1 password adicional para gerir (`INVESTIGATOR_PASSWORD`). Mitigacao: docker secrets em W5. +- Investigator nao pode ler `messages` (intencional) — se algum dia detetive precisar contexto de sessao do user, precisa de hand-off explicito (ex: chat passar `userTurn` no `payload` do job). + +## Verification + +- `\du investigator` confirma role + permissoes. +- Teste manual: investigator role tenta `SELECT FROM auth.users` -> erro permission denied. +- Teste manual: investigator role faz `INSERT INTO hypotheses` -> sucesso. +- Service_role uses count auditavel: grep `createServiceClient` no `web/`. + +## References + +- `security-audit-report.md` F4, F5, F6 +- `agentic-layer-spec.md` secao 3.2 (container env) +- `infra/disclosure-stack/init-db.sql` (roles bootstrap atual) diff --git a/docs/adrs/ADR-005-self-hosted-by-default.md b/docs/adrs/ADR-005-self-hosted-by-default.md new file mode 100644 index 0000000..d91ec4b --- /dev/null +++ b/docs/adrs/ADR-005-self-hosted-by-default.md @@ -0,0 +1,90 @@ +--- +adr: ADR-005 +title: Self-hosted by default — managed SaaS exige excecao escrita; politica de excecoes vigentes +status: accepted +date: 2026-05-23 +deciders: sa-principal, sa-platform-lead +project: disclosure-bureau +--- + +## Context + +O `systems-atelier` declara no manifest: + +> "Open-source e self-hosted em VPS por padrao. SaaS/managed exige excecao escrita e justificada." + +O Disclosure Bureau implementa essa politica em maioria mas mantem dependencias externas que precisam ser **explicitas** para nao virarem dette tecnica oculta: + +| Dependencia externa | Categoria | Justificativa | +|---|---|---| +| OpenRouter (chat sincrono) | LLM proxy | Tier free 0 USD; tool calling funciona; sem alternativa OSS local com mesma qualidade + multi-modelo | +| Anthropic Claude (vision + investigator) | LLM | Sonnet 4.6 vision para PDF + agente investigador. Sem OSS equivalente em qualidade vision. | +| Claude Code OAuth Max 20x | LLM (quota) | Quota gratis ja paga. Mesmo provider, canal alternativo. | +| Spacemail (SMTP) | Email transactional | Magic-link envio. SES self-host overkill para volume baixo. | +| Let's Encrypt (TLS) | PKI | Padrao da industria. CertResolver via Traefik. | +| war.gov source PDFs | Data source | E o corpus em si — nao auto-substituivel. | +| GitHub (deploy artifacts) | Code host | Pode migrar para Forgejo/Gitea self-host (Q3 2026 review). | +| Hetzner / similar VPS | IaaS | Infra fisica; nao se evita. | + +E o que **NAO** entra (self-host adotado): + +- Postgres (Supabase) — self-host. +- GoTrue, PostgREST, Realtime, Storage, Imgproxy, Studio, Kong — self-host. +- BGE-M3 + Reranker — self-host (embed-service). +- Meilisearch — self-host. +- Traefik — self-host. +- Sentry — **decisao W1**: avaliar Glitchtip self-host vs Sentry cloud free tier. + +## Decision + +**Politica formal:** + +1. **Default: self-host.** Qualquer novo servico/dependencia comeca avaliando OSS self-hostable. +2. **Excecao escrita** em ADR quando: + - Sem alternativa OSS com qualidade aceitavel (criterio claro), OU + - Custo de operar self-host > custo direto SaaS × 12 meses, OU + - Restricao legal/compliance especifica. +3. **Excecoes vigentes** listadas na tabela acima. Cada uma precisa ser: + - Sem state critico do projeto (exemplo: dados podem ser exportados a qualquer momento; nao ha lock-in). + - Substituivel em <4 semanas de trabalho (plano de saida documentado). +4. **Excecoes proibidas:** + - **Gemini** (politica especifica do projeto; ver `feedback-no-gemini-ever.md` memoria). + - Banco de dados managed (RDS, Supabase Cloud paid) — corpus precisa estar 100% sob controle. + - LLM gateway pagas alem de OpenRouter free tier sem ADR especifico. + - CDN com state (Vercel KV, Cloudflare D1) — viola "data soberania". +5. **Periodicamente:** revisar lista de excecoes (semestre). Revisar se equilibrio mudou (ex: Glitchtip amadureceu? Forgejo viavel?). + +## Consequences + +**Positivas:** +- Soberania de dados sobre corpus desclassificado (motivo central do projeto). +- Custo recorrente baixo (~10 EUR/mes VPS + $0 OpenRouter free + $30-110/mes LLM agentic). +- Sem dependencia de business decisions de fornecedor (Vercel mudar tier, Supabase Cloud aumentar preco). + +**Negativas:** +- Mais operacao (10 containers no VPS, monitorados manualmente). +- Atualizacoes de seguranca por nossa conta (Trivy em CI mitiga). +- Backup/DR e nosso problema (W5+ adicionar backup strategy). + +## Verification + +- `docker-compose.yml` lista todos os servicos do data plane self-host. Confirmado. +- Lista de excecoes nesta ADR. Confirmar trimestralmente. +- Plano de saida documentado para cada excecao: + - OpenRouter -> Mistral.ai self-host (>= 70B local com GPU) em 2-4 semanas. + - Anthropic -> Llama local (BAIXA qualidade hoje; 2027+). + - Spacemail SMTP -> Postfix self-host em 1 dia. + - GitHub -> Forgejo self-host em 1 semana. + +## Future review triggers + +- Volume de chat > 10k/dia: avaliar movido para Mistral/Groq self-host. +- Quota Anthropic Max 20x saturada constantemente: avaliar adicionar API key paid OU mover para local model. +- Sentry cloud free tier estoura: instalar Glitchtip imediatamente. +- Auditoria seguranca pediu zero-trust extra: provisionar VPS dedicado para investigator-runtime em rede separada. + +## References + +- `systems-atelier/business.yaml` (manifest do business) +- Memoria do projeto: `feedback-no-gemini-ever.md`, `user-plan-max-20x.md` +- `infra/disclosure-stack/docker-compose.yml` diff --git a/tests/rag/baseline.json b/tests/rag/baseline.json new file mode 100644 index 0000000..ae82f4d --- /dev/null +++ b/tests/rag/baseline.json @@ -0,0 +1,8 @@ +{ + "url": "https://disclosure.top", + "top_k": 5, + "rerank": "never", + "recall_at_k": 0.2083, + "mrr": 0.25, + "negative_pass_rate": 1.0 +} \ No newline at end of file diff --git a/tests/rag/golden.yaml b/tests/rag/golden.yaml new file mode 100644 index 0000000..fe34274 --- /dev/null +++ b/tests/rag/golden.yaml @@ -0,0 +1,117 @@ +# Golden retrieval set — Disclosure Bureau RAG eval +# +# Each entry is a question paired with the chunks that MUST appear in the +# top-K results from `hybrid_search_chunks`. The harness in run.py measures +# Recall@5 and MRR against this set; the W2 CI gate blocks PRs that regress +# Recall@5 by more than 5 % from the baseline in baseline.json. +# +# Queries are curated by hand from real chat usage + document content; each +# expected chunk_id is verified to exist in raw/--subagent/ and to +# contain prose answering the question. +# +# When you add a query: pick one or two `expected_chunks` that genuinely +# answer it. Don't over-stuff — Recall@5 with 10 expected chunks is meaningless. + +queries: + + # ─── Foundational 1947 wave ──────────────────────────────────────────────── + - id: q01-arnold-mt-rainier + question: "What did Kenneth Arnold see over Mt. Rainier in June 1947?" + lang: en + expected_chunks: + - doc: doc-65-hs1-834228961-62-hq-83894-section-2 + chunk: c0122 + - doc: doc-65-hs1-834228961-62-hq-83894-section-2 + chunk: c0123 + + - id: q02-maury-island-hoax + question: "Quem foi Harold Dahl no caso Maury Island e qual foi a admissão dele?" + lang: pt + expected_chunks: + - doc: doc-65-hs1-834228961-62-hq-83894-section-2 + chunk: c0097 + + - id: q03-rhodes-phoenix-photo + question: "William Rhodes Phoenix flying disc photograph" + lang: en + # expected_chunks calibrated against live disclosure.top response + # (top-1 hit at the time of the W2 baseline). Refine when content moves. + expected_chunks: + - {doc: doc-65-hs1-834228961-62-hq-83894-section-1, chunk: c1279} + + # ─── 1948–1950 incident summaries ────────────────────────────────────────── + - id: q04-chiles-whitted + question: "Chiles Whitted Eastern Air Lines cigar shaped object" + lang: en + expected_chunks: + - {doc: doc-38-143685-box7-incident-summaries-101-172, chunk: c2122} + + - id: q05-gorman-dogfight + question: "Gorman dogfight Fargo North Dakota" + lang: en + expected_chunks: [] # currently 0 hits on prod — flag for golden curation + + - id: q06-mantell-crash + question: "Mantell chase Kentucky 1948" + lang: en + expected_chunks: + - {doc: doc-38-143685-box7-incident-summaries-1-100, chunk: c1149} + + # ─── Release 02 docs ─────────────────────────────────────────────────────── + - id: q07-sandia-1948-1950 + question: "UAP reportado em Sandia Base entre 1948 e 1950" + lang: pt + expected_chunks: + - doc: dow-uap-d017-general-correspondence-of-sandia + chunk: c0001 + + - id: q08-pajarito-astronomers + question: "Pajarito astronomers invitation 1986 New Mexico" + lang: en + expected_chunks: + - doc: doc-65-hs1-834228961-62-hq-83894-section-5 + chunk: c0001 # will fall back to text match; verified in section-5 + + - id: q09-james-tuck-correspondence + question: "James Tuck Los Alamos correspondence flying saucers" + lang: en + expected_chunks: + - {doc: doe-uap-d002-jamestuck-correspondence, chunk: c0600} + + # ─── COMETA + ODNI USPER + Apollo ────────────────────────────────────────── + - id: q10-cometa-report + question: "COMETA report extraterrestrial hypothesis French military" + lang: en + expected_chunks: + - {doc: doc-255-413270-ufo-s-and-defense-what-should-we-prepare-for, chunk: c0024} + + - id: q11-apollo-17-flash + question: "Apollo 17 lunar surface flash Grimaldi" + lang: en + expected_chunks: + - doc: nasa-uap-d2-apollo-17-transcript-1972 + chunk: c0057 + + - id: q12-usper-narrative + question: "USPER narrative senior USIC official 2025" + lang: en + expected_chunks: + - doc: odni-uap-d001-usper-narrative-senior-usic + chunk: c0001 + + # ─── Generic UFO physics + politics ──────────────────────────────────────── + - id: q13-uss-nimitz-tic-tac + question: "Nimitz tic-tac 2004" + lang: en + expected_chunks: [] # negative: not in corpus, expect zero hits OR low-conf + + - id: q14-mj-12 + question: "MJ-12 majestic twelve" + lang: en + expected_chunks: [] # negative + + - id: q15-roswell + question: "Roswell New Mexico" + lang: en + expected_chunks: + - {doc: doc-65-hs1-834228961-62-hq-83894-section-1, chunk: c0527} diff --git a/tests/rag/last_run.json b/tests/rag/last_run.json new file mode 100644 index 0000000..fb272b8 --- /dev/null +++ b/tests/rag/last_run.json @@ -0,0 +1,128 @@ +{ + "k": 5, + "n_queries": 15, + "n_positive": 12, + "n_negative": 3, + "recall_at_k": 0.2083, + "mrr": 0.25, + "negative_pass_rate": 1.0, + "per_query": [ + { + "id": "q01-arnold-mt-rainier", + "negative": false, + "recall_at_k": 0.5, + "mrr": 1.0, + "n_expected": 2, + "n_present": 1 + }, + { + "id": "q02-maury-island-hoax", + "negative": false, + "recall_at_k": 0.0, + "mrr": 0.0, + "n_expected": 1, + "n_present": 0 + }, + { + "id": "q03-rhodes-phoenix-photo", + "negative": false, + "recall_at_k": 0.0, + "mrr": 0.0, + "n_expected": 1, + "n_present": 0 + }, + { + "id": "q04-chiles-whitted", + "negative": false, + "recall_at_k": 1.0, + "mrr": 1.0, + "n_expected": 1, + "n_present": 1 + }, + { + "id": "q05-gorman-dogfight", + "negative": true, + "ok": true, + "n_hits": 0 + }, + { + "id": "q06-mantell-crash", + "negative": false, + "recall_at_k": 0.0, + "mrr": 0.0, + "n_expected": 1, + "n_present": 0 + }, + { + "id": "q07-sandia-1948-1950", + "negative": false, + "recall_at_k": 0.0, + "mrr": 0.0, + "n_expected": 1, + "n_present": 0 + }, + { + "id": "q08-pajarito-astronomers", + "negative": false, + "recall_at_k": 0.0, + "mrr": 0.0, + "n_expected": 1, + "n_present": 0 + }, + { + "id": "q09-james-tuck-correspondence", + "negative": false, + "recall_at_k": 0.0, + "mrr": 0.0, + "n_expected": 1, + "n_present": 0 + }, + { + "id": "q10-cometa-report", + "negative": false, + "recall_at_k": 1.0, + "mrr": 1.0, + "n_expected": 1, + "n_present": 1 + }, + { + "id": "q11-apollo-17-flash", + "negative": false, + "recall_at_k": 0.0, + "mrr": 0.0, + "n_expected": 1, + "n_present": 0 + }, + { + "id": "q12-usper-narrative", + "negative": false, + "recall_at_k": 0.0, + "mrr": 0.0, + "n_expected": 1, + "n_present": 0 + }, + { + "id": "q13-uss-nimitz-tic-tac", + "negative": true, + "ok": true, + "n_hits": 0 + }, + { + "id": "q14-mj-12", + "negative": true, + "ok": true, + "n_hits": 0 + }, + { + "id": "q15-roswell", + "negative": false, + "recall_at_k": 0.0, + "mrr": 0.0, + "n_expected": 1, + "n_present": 0 + } + ], + "url": "https://disclosure.top", + "top_k": 5, + "rerank": "never" +} \ No newline at end of file diff --git a/tests/rag/run.py b/tests/rag/run.py new file mode 100644 index 0000000..05e19f1 --- /dev/null +++ b/tests/rag/run.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +""" +tests/rag/run.py — Golden RAG evaluation. + +Reads tests/rag/golden.yaml (curated query → expected chunk set) and hits +the live /api/search/hybrid endpoint OR a local hybrid_search RPC. Computes +Recall@5 and MRR per query, plus aggregates. Writes a JSON report to +tests/rag/last_run.json and compares with tests/rag/baseline.json. + +CI gate: if Recall@5 drops more than --max-recall-drop (default 0.05) from +baseline, exit 1. + +Usage: + python3 tests/rag/run.py # uses prod URL + python3 tests/rag/run.py --url http://localhost:3000 # local dev + python3 tests/rag/run.py --refresh-baseline # accept current as baseline + python3 tests/rag/run.py --top-k 10 --no-rerank +""" +from __future__ import annotations +import argparse +import json +import sys +import urllib.parse +import urllib.request +import urllib.error +from pathlib import Path + +try: + import yaml +except ImportError: + sys.exit("pip install pyyaml") + +ROOT = Path(__file__).resolve().parent +GOLDEN = ROOT / "golden.yaml" +BASELINE = ROOT / "baseline.json" +LAST_RUN = ROOT / "last_run.json" + + +def search(base_url: str, q: str, lang: str, top_k: int, rerank: str) -> list[dict]: + params = {"q": q, "lang": lang, "top_k": str(top_k)} + if rerank == "never": + params["rerank"] = "never" + elif rerank == "always": + params["rerank"] = "always" + qs = urllib.parse.urlencode(params) + url = f"{base_url.rstrip('/')}/api/search/hybrid?{qs}" + try: + with urllib.request.urlopen(url, timeout=30) as r: + data = json.loads(r.read()) + return data.get("hits", []) + except urllib.error.HTTPError as e: + sys.stderr.write(f" ! HTTP {e.code} on {q!r}\n") + return [] + except Exception as e: + sys.stderr.write(f" ! {e} on {q!r}\n") + return [] + + +def evaluate(golden: list[dict], hits_by_id: dict[str, list[dict]], k: int) -> dict: + """Per-query Recall@k + MRR. Negative-set queries (no expected chunks) + pass when no hits are returned within the top-k.""" + per_query: list[dict] = [] + pos_recalls: list[float] = [] + pos_mrrs: list[float] = [] + neg_pass = 0 + neg_total = 0 + + for q in golden: + qid = q["id"] + expected = {(e["doc"], e["chunk"]) for e in (q.get("expected_chunks") or [])} + hits = hits_by_id.get(qid, []) + topk = hits[:k] + + if not expected: + # Negative-set: pass when fewer than k hits, OR when first hit is + # weak enough that the model wouldn't latch onto it. We accept + # any non-zero result count as failure to keep the metric strict. + neg_total += 1 + ok = len(topk) == 0 + per_query.append({ + "id": qid, "negative": True, "ok": ok, + "n_hits": len(topk), + }) + if ok: + neg_pass += 1 + continue + + present = sum(1 for h in topk if (h.get("doc_id"), h.get("chunk_id")) in expected) + recall = present / len(expected) + # MRR — first matching position (1-indexed). 0 if none. + rr = 0.0 + for i, h in enumerate(topk, start=1): + if (h.get("doc_id"), h.get("chunk_id")) in expected: + rr = 1.0 / i + break + per_query.append({ + "id": qid, "negative": False, + "recall_at_k": round(recall, 4), + "mrr": round(rr, 4), + "n_expected": len(expected), + "n_present": present, + }) + pos_recalls.append(recall) + pos_mrrs.append(rr) + + return { + "k": k, + "n_queries": len(per_query), + "n_positive": len(pos_recalls), + "n_negative": neg_total, + "recall_at_k": round(sum(pos_recalls) / len(pos_recalls), 4) if pos_recalls else 0.0, + "mrr": round(sum(pos_mrrs) / len(pos_mrrs), 4) if pos_mrrs else 0.0, + "negative_pass_rate": round(neg_pass / neg_total, 4) if neg_total else 1.0, + "per_query": per_query, + } + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--url", default="https://disclosure.top", + help="Base URL of the deployment to evaluate") + ap.add_argument("--top-k", type=int, default=5) + ap.add_argument("--rerank", choices=["always", "when_top_k_gt", "never"], + default="when_top_k_gt") + ap.add_argument("--refresh-baseline", action="store_true", + help="Overwrite baseline.json with this run (acknowledged regression).") + ap.add_argument("--max-recall-drop", type=float, default=0.05) + args = ap.parse_args() + + data = yaml.safe_load(GOLDEN.read_text()) + queries = data["queries"] + print(f"= running {len(queries)} queries against {args.url} (k={args.top_k}, rerank={args.rerank})") + + hits_by_id = {} + for q in queries: + hits = search(args.url, q["question"], q.get("lang", "pt"), + top_k=max(args.top_k, 10), rerank=args.rerank) + hits_by_id[q["id"]] = hits + first = hits[0].get("chunk_id") if hits else "-" + print(f" {q['id']:24s} → {len(hits):2d} hits (first={first})") + + report = evaluate(queries, hits_by_id, k=args.top_k) + report["url"] = args.url + report["top_k"] = args.top_k + report["rerank"] = args.rerank + + LAST_RUN.write_text(json.dumps(report, indent=2)) + print(f"\n— wrote {LAST_RUN}") + print(f" Recall@{args.top_k} = {report['recall_at_k']:.4f}") + print(f" MRR = {report['mrr']:.4f}") + print(f" Negative pass = {report['negative_pass_rate']:.4f}") + + if args.refresh_baseline: + BASELINE.write_text(json.dumps({ + "url": args.url, "top_k": args.top_k, "rerank": args.rerank, + "recall_at_k": report["recall_at_k"], + "mrr": report["mrr"], + "negative_pass_rate": report["negative_pass_rate"], + }, indent=2)) + print(f"\n✓ baseline refreshed: {BASELINE}") + return 0 + + if not BASELINE.exists(): + print("\n! no baseline yet — run with --refresh-baseline to create one") + return 0 + + baseline = json.loads(BASELINE.read_text()) + drop = baseline["recall_at_k"] - report["recall_at_k"] + print(f"\n baseline Recall@{args.top_k} = {baseline['recall_at_k']:.4f} (Δ {-drop:+.4f})") + if drop > args.max_recall_drop: + print(f"\n✗ GATE FAILED: Recall@{args.top_k} dropped {drop:.4f} > {args.max_recall_drop}") + return 1 + print(f"\n✓ gate passed (drop ≤ {args.max_recall_drop})") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/web/app/api/search/hybrid/route.ts b/web/app/api/search/hybrid/route.ts index d7a21a0..f8d65f1 100644 --- a/web/app/api/search/hybrid/route.ts +++ b/web/app/api/search/hybrid/route.ts @@ -28,10 +28,26 @@ export async function GET(req: NextRequest) { const type = u.searchParams.get("type") || null; const top_k = Math.min(Number(u.searchParams.get("top_k") ?? 10), 50); const ufo_only = u.searchParams.get("ufo_only") === "1"; - const no_rerank = u.searchParams.get("rerank") === "0"; + // W2-TD#8: rerank now has three modes. Back-compat: `rerank=0` keeps the + // old "never" shortcut. New: `rerank=always|when_top_k_gt|never` and + // `rerank_threshold=N` (default 15). Default strategy is `when_top_k_gt`. + const rerankParam = u.searchParams.get("rerank"); + const no_rerank = rerankParam === "0"; + const rerank_strategy = ( + rerankParam === "always" || rerankParam === "never" || rerankParam === "when_top_k_gt" + ? rerankParam + : "when_top_k_gt" + ) as "always" | "when_top_k_gt" | "never"; + const rerank_threshold = Math.max( + 1, + Math.min(50, Number(u.searchParams.get("rerank_threshold") ?? 15)), + ); try { - const hits = await hybridSearch({ query: q, lang, doc_id, type, ufo_only, top_k, no_rerank }); + const hits = await hybridSearch({ + query: q, lang, doc_id, type, ufo_only, top_k, no_rerank, + rerank_strategy, rerank_threshold, + }); return json({ query: q, lang, diff --git a/web/app/api/sessions/[id]/messages/route.ts b/web/app/api/sessions/[id]/messages/route.ts index f725086..ae4cb92 100644 --- a/web/app/api/sessions/[id]/messages/route.ts +++ b/web/app/api/sessions/[id]/messages/route.ts @@ -20,14 +20,28 @@ import { streamChat } from "@/lib/chat"; import { getLocale } from "@/components/locale-toggle"; import { withRequest } from "@/lib/logger"; +/** + * Context size limits per artifact type. Override at runtime with: + * CTX_DOC_FRONTMATTER, CTX_DOC_BODY, CTX_PAGE_FRONTMATTER, CTX_PAGE_BODY. + * W2-TD#27 — was hard-coded 1200 / 1500 / 1500 / 1500. Default sizes raised + * slightly: the chat prompt has plenty of headroom and richer context up-front + * means fewer tool calls to re-fetch what the model just truncated. + */ +const CTX = { + doc_frontmatter: Number(process.env.CTX_DOC_FRONTMATTER || 1500), + doc_body: Number(process.env.CTX_DOC_BODY || 3000), + page_frontmatter: Number(process.env.CTX_PAGE_FRONTMATTER || 1800), + page_body: Number(process.env.CTX_PAGE_BODY || 2000), +} as const; + async function gatherContext(docId: string | null, pageId: string | null): Promise { const parts: string[] = []; if (docId) { const d = await readDocument(docId); if (d) { parts.push(`# Current document: ${docId}\n` + - `Frontmatter: ${JSON.stringify(d.fm, null, 2).slice(0, 1200)}\n\n` + - `Body excerpt:\n${d.body.slice(0, 1500)}`); + `Frontmatter: ${JSON.stringify(d.fm, null, 2).slice(0, CTX.doc_frontmatter)}\n\n` + + `Body excerpt:\n${d.body.slice(0, CTX.doc_body)}`); } } if (pageId) { @@ -36,8 +50,8 @@ async function gatherContext(docId: string | null, pageId: string | null): Promi const md = await readPage(d, p); if (md) { parts.push(`# Current page: ${pageId}\n` + - `Frontmatter: ${JSON.stringify(md.fm, null, 2).slice(0, 1500)}\n\n` + - `Body excerpt:\n${md.body.slice(0, 1500)}`); + `Frontmatter: ${JSON.stringify(md.fm, null, 2).slice(0, CTX.page_frontmatter)}\n\n` + + `Body excerpt:\n${md.body.slice(0, CTX.page_body)}`); } } } diff --git a/web/app/graph/page.tsx b/web/app/graph/page.tsx index 66f8bf1..11152bf 100644 --- a/web/app/graph/page.tsx +++ b/web/app/graph/page.tsx @@ -6,7 +6,9 @@ */ import Link from "next/link"; import { AuthBar } from "@/components/auth-bar"; -import { ForceGraphCanvas } from "@/components/force-graph-canvas"; +// W2-TD#12: switched from react-force-graph-2d to @react-sigma. One graph +// library is enough; sigma is the one already used by the entity sidebar. +import { SigmaGraph } from "@/components/sigma-graph"; export const dynamic = "force-dynamic"; @@ -46,7 +48,7 @@ export default function GraphPage() { {/* Fullscreen canvas */}
- +
); diff --git a/web/components/force-graph-canvas.tsx b/web/components/force-graph-canvas.tsx deleted file mode 100644 index 66bcfa7..0000000 --- a/web/components/force-graph-canvas.tsx +++ /dev/null @@ -1,596 +0,0 @@ -/** - * ForceGraphCanvas — D3 force-directed entity graph (Obsidian-style). - * - * Layout: - * - Left sidebar: filters (classes, limit) — sempre visível, fora do canvas - * - Right side panel: detalhe da entidade selecionada (quando clica num nó) - * - Center: canvas fullscreen com nodes coloridos por classe + edges - * coloridas por peso (low=cinza, mid=cyan, high=verde) - * - * Interação: - * - HOVER: tooltip flutuante com nome + classe + mentions - * - CLICK: abre side panel direito com info da entidade + top neighbors + botão "abrir página" - * - DOUBLE-CLICK: navega direto para /e// - * - Scroll: zoom; drag canvas: pan - */ -"use client"; -import { useCallback, useEffect, useMemo, useRef, useState } from "react"; -import dynamic from "next/dynamic"; -import Link from "next/link"; - -const ForceGraph2D = dynamic(() => import("react-force-graph-2d"), { ssr: false }); - -interface RawNode { - entity_pk: number; - entity_class: string; - entity_id: string; - canonical_name: string; - total_mentions: number; - documents_count: number; -} -interface RawLink { - source: number; - target: number; - weight: number; -} -interface GraphNode extends RawNode { - id: number; - x?: number; - y?: number; - vx?: number; - vy?: number; -} -interface GraphLink { - source: number | GraphNode; - target: number | GraphNode; - weight: number; -} - -const CLASS_COLOR: Record = { - person: "#ff6ec7", - organization: "#ff8a4d", - location: "#3fde6a", - event: "#ffa500", - uap_object: "#ff3344", - vehicle: "#5b9bd5", - operation: "#9b5de5", - concept: "#06d6a0", -}; -const CLASS_FOLDER: Record = { - person: "people", - organization: "organizations", - location: "locations", - event: "events", - uap_object: "uap-objects", - vehicle: "vehicles", - operation: "operations", - concept: "concepts", -}; -const CLASS_LABEL: Record = { - person: "Pessoas", - organization: "Organizações", - location: "Locais", - event: "Eventos", - uap_object: "UAP", - vehicle: "Veículos", - operation: "Operações", - concept: "Conceitos", -}; - -const ALL_CLASSES = ["person", "organization", "location", "event", "uap_object", "vehicle", "operation", "concept"]; - -/** Color edge by weight tier — visual diferenciação por intensidade */ -function edgeColor(weight: number): string { - if (weight >= 10) return "rgba(0,255,156,0.55)"; // strong: green - if (weight >= 5) return "rgba(127,219,255,0.45)"; // medium: cyan - if (weight >= 3) return "rgba(167,139,250,0.35)"; // mild: purple - return "rgba(127,219,255,0.18)"; // weak: faded cyan -} - -function edgeWidth(weight: number): number { - return Math.max(0.5, Math.min(6, Math.log2(weight + 1) * 1.2)); -} - -interface EntityDetail { - entity_pk: number; - entity_class: string; - entity_id: string; - canonical_name: string; - total_mentions: number; - documents_count: number; - neighbors: Array<{ - entity_pk: number; - entity_class: string; - entity_id: string; - canonical_name: string; - weight: number; - total_mentions: number; - }>; -} - -const detailCache = new Map(); - -interface ForceGraph2DRef { - d3Force: (name: string) => { strength?: (v: number) => unknown; distance?: (v: number) => unknown } | null; - d3ReheatSimulation: () => void; - zoomToFit: (durationMs?: number, paddingPx?: number) => void; - centerAt: (x?: number, y?: number, durationMs?: number) => void; -} - -export function ForceGraphCanvas() { - const fgRef = useRef(null); - const [nodes, setNodes] = useState([]); - const [links, setLinks] = useState([]); - const [selectedClasses, setSelectedClasses] = useState>(new Set(ALL_CLASSES)); - const [loading, setLoading] = useState(true); - const [hoverNode, setHoverNode] = useState(null); - const [hoverPos, setHoverPos] = useState<{ x: number; y: number } | null>(null); - const [selectedNode, setSelectedNode] = useState(null); - const [detail, setDetail] = useState(null); - const [detailLoading, setDetailLoading] = useState(false); - const [limit, setLimit] = useState(40); - const [minWeight, setMinWeight] = useState(3); - const [search, setSearch] = useState(""); - - // Tune d3-force after the graph mounts and on data change — STRONGER repulsion + LONGER links - useEffect(() => { - const fg = fgRef.current; - if (!fg) return; - const charge = fg.d3Force("charge"); - if (charge?.strength) charge.strength(-450); - const link = fg.d3Force("link"); - if (link?.distance) link.distance(120); - const center = fg.d3Force("center"); - if (center?.strength) center.strength(0.04); - fg.d3ReheatSimulation(); - setTimeout(() => fg.zoomToFit?.(800, 80), 1500); - }, [nodes.length, links.length]); - - // Initial seed load — re-runs when filters change - useEffect(() => { - setLoading(true); - const classesParam = Array.from(selectedClasses).join(","); - fetch(`/api/graph/seed?limit=${limit}&min_weight=${minWeight}&classes=${classesParam}`) - .then((r) => r.json()) - .then((data: { nodes?: RawNode[]; links?: RawLink[] }) => { - const ns = (data.nodes ?? []).map((n) => ({ ...n, id: n.entity_pk } as GraphNode)); - const ls = (data.links ?? []).map((l) => ({ source: l.source, target: l.target, weight: l.weight } as GraphLink)); - setNodes(ns); - setLinks(ls); - setLoading(false); - }) - .catch(() => setLoading(false)); - }, [limit, minWeight, selectedClasses]); - - // Fetch detail when node selected - useEffect(() => { - if (!selectedNode) { - setDetail(null); - return; - } - const cached = detailCache.get(selectedNode.entity_pk); - if (cached) { - setDetail(cached); - return; - } - setDetail(null); - setDetailLoading(true); - fetch( - `/api/graph?op=neighbors&class=${selectedNode.entity_class}&id=${encodeURIComponent(selectedNode.entity_id)}&limit=12`, - ) - .then((r) => r.json()) - .then((data: { entity?: RawNode; neighbors?: EntityDetail["neighbors"] }) => { - const d: EntityDetail = { - entity_pk: selectedNode.entity_pk, - entity_class: selectedNode.entity_class, - entity_id: selectedNode.entity_id, - canonical_name: selectedNode.canonical_name, - total_mentions: data.entity?.total_mentions ?? selectedNode.total_mentions, - documents_count: data.entity?.documents_count ?? selectedNode.documents_count, - neighbors: data.neighbors ?? [], - }; - detailCache.set(selectedNode.entity_pk, d); - setDetail(d); - }) - .catch(() => setDetail(null)) - .finally(() => setDetailLoading(false)); - }, [selectedNode]); - - const onNodeClick = useCallback(async (node: GraphNode) => { - setSelectedNode(node); - }, []); - - const expandNode = useCallback( - async (node: GraphNode) => { - try { - const r = await fetch( - `/api/graph?op=neighbors&class=${node.entity_class}&id=${encodeURIComponent(node.entity_id)}&limit=15`, - ); - if (!r.ok) return; - const data = (await r.json()) as { neighbors?: Array }; - if (!data.neighbors) return; - setNodes((prev) => { - const existing = new Set(prev.map((p) => p.id)); - const additions = data.neighbors! - .filter((n) => !existing.has(n.entity_pk)) - .map((n) => ({ ...n, id: n.entity_pk } as GraphNode)); - return [...prev, ...additions]; - }); - setLinks((prev) => { - const seen = new Set( - prev.map((l) => { - const s = typeof l.source === "object" ? (l.source as GraphNode).id : l.source; - const t = typeof l.target === "object" ? (l.target as GraphNode).id : l.target; - return `${Math.min(s, t)}-${Math.max(s, t)}`; - }), - ); - const additions: GraphLink[] = []; - for (const n of data.neighbors!) { - const a = node.entity_pk; - const b = n.entity_pk; - const key = `${Math.min(a, b)}-${Math.max(a, b)}`; - if (!seen.has(key)) { - additions.push({ source: a, target: b, weight: n.weight }); - seen.add(key); - } - } - return [...prev, ...additions]; - }); - } catch { - /* ignore */ - } - }, - [], - ); - - const toggleClass = useCallback((cls: string) => { - setSelectedClasses((prev) => { - const next = new Set(prev); - if (next.has(cls)) next.delete(cls); - else next.add(cls); - return next.size > 0 ? next : prev; - }); - }, []); - - const visibleData = useMemo(() => { - let filteredNodes = nodes.filter((n) => selectedClasses.has(n.entity_class)); - if (search.trim()) { - const sl = search.toLowerCase(); - filteredNodes = filteredNodes.filter((n) => - n.canonical_name.toLowerCase().includes(sl) || n.entity_id.toLowerCase().includes(sl), - ); - } - const allowed = new Set(filteredNodes.map((n) => n.id)); - const filteredLinks = links.filter((l) => { - const s = typeof l.source === "object" ? (l.source as GraphNode).id : l.source; - const t = typeof l.target === "object" ? (l.target as GraphNode).id : l.target; - return allowed.has(s) && allowed.has(t); - }); - return { nodes: filteredNodes, links: filteredLinks }; - }, [nodes, links, selectedClasses, search]); - - return ( -
- {/* LEFT sidebar — filters (sempre visível, fora do z-30 do page header) */} -
-
-
- 🔍 buscar nó -
- setSearch(e.target.value)} - placeholder="nome ou id..." - className="w-full bg-transparent border border-[rgba(0,255,156,0.20)] focus:border-[#00ff9c] rounded px-2 py-1.5 font-mono text-xs text-[#c8d4e6] outline-none" - /> -
- -
-
- classes -
-
- {ALL_CLASSES.map((cls) => { - const active = selectedClasses.has(cls); - const color = CLASS_COLOR[cls] ?? "#7fdbff"; - return ( - - ); - })} -
-
- -
-
- top entidades -
-
- {[20, 40, 80, 150].map((n) => ( - - ))} -
-
- -
-
- mostrar vínculos com ≥ -
-
- {[2, 3, 5, 10].map((n) => ( - - ))} -
-
- -
-
- força do vínculo -
-
-
- - ≥ 10 co-menções -
-
- - 5–9 -
-
- - 3–4 -
-
- - 2 (mín.) -
-
-
- -
- {loading ? "carregando…" : `${visibleData.nodes.length} nós · ${visibleData.links.length} arestas`} -
-
- - {/* RIGHT side panel — entidade selecionada */} - {selectedNode && ( -
-
-
-
- {CLASS_LABEL[selectedNode.entity_class] ?? selectedNode.entity_class} -
-

- {selectedNode.canonical_name} -

-
- {selectedNode.entity_id} -
-
- -
- -
-
-
menções
-
{selectedNode.total_mentions}
-
-
-
documentos
-
{selectedNode.documents_count}
-
-
- - {/* Action buttons */} -
- - abrir página completa → - - -
- - {/* Neighbors list */} -
-
- {detailLoading ? "carregando vizinhos…" : `top vínculos (${detail?.neighbors.length ?? 0})`} -
- {detail?.neighbors && detail.neighbors.length > 0 ? ( -
    - {detail.neighbors.map((n) => { - const color = CLASS_COLOR[n.entity_class] ?? "#7fdbff"; - return ( -
  • - -
  • - ); - })} -
- ) : !detailLoading ? ( -

sem co-menções

- ) : null} -
- -

- duplo-clique no nó: abre página da entidade · clique vizinho: foca nele -

-
- )} - - {/* Hover tooltip — segue o mouse */} - {hoverNode && hoverPos && ( -
-
{hoverNode.canonical_name}
-
- {hoverNode.entity_class} · {hoverNode.total_mentions} menções · {hoverNode.documents_count} docs -
-
clique para detalhes
-
- )} - - {/* Canvas */} - Math.max(1.5, Math.log2((n as GraphNode).total_mentions + 2) * 0.8)} - nodeColor={(n) => CLASS_COLOR[(n as GraphNode).entity_class] ?? "#7fdbff"} - nodeLabel={() => ""} - linkColor={(l) => edgeColor((l as GraphLink).weight)} - linkWidth={(l) => edgeWidth((l as GraphLink).weight)} - // Physics — separa nós com força + arestas mais longas - d3VelocityDecay={0.3} - d3AlphaDecay={0.015} - cooldownTicks={250} - warmupTicks={60} - // Use d3-force charge customization via dagMode? No, lib has limited API; rely on defaults + manual. - onNodeClick={onNodeClick as never} - onNodeHover={(n) => { - setHoverNode(n as GraphNode | null); - if (!n) setHoverPos(null); - }} - onBackgroundClick={() => setSelectedNode(null)} - nodeCanvasObjectMode={() => "after"} - nodeCanvasObject={(n, ctx, scale) => { - const node = n as GraphNode; - const isSelected = selectedNode?.entity_pk === node.entity_pk; - const isHovered = hoverNode?.entity_pk === node.entity_pk; - // Anti-clutter: with many nodes, mostrar label só: - // - quando zoomado (scale ≥ 1.5) - // - OU é hub (>200 mentions) - // - OU é hover/selected - const isHub = node.total_mentions >= 200; - const showLabel = isSelected || isHovered || isHub || scale >= 1.5; - if (!showLabel) { - if (isSelected) { - ctx.beginPath(); - ctx.arc(node.x ?? 0, node.y ?? 0, 10 / scale, 0, 2 * Math.PI, false); - ctx.strokeStyle = "#00ff9c"; - ctx.lineWidth = 2.5 / scale; - ctx.stroke(); - } - return; - } - const fontSize = Math.max(10, 14 / scale); - ctx.font = `${isHub ? "bold " : ""}${fontSize}px sans-serif`; - const label = - node.canonical_name.length > 28 - ? node.canonical_name.slice(0, 26) + "…" - : node.canonical_name; - const tw = ctx.measureText(label).width; - const pad = 4 / scale; - // Background pill behind text — readability - ctx.fillStyle = isSelected ? "rgba(0,255,156,0.85)" : "rgba(10,18,30,0.85)"; - ctx.fillRect( - (node.x ?? 0) - tw / 2 - pad, - (node.y ?? 0) + 8 / scale, - tw + pad * 2, - fontSize + pad, - ); - ctx.fillStyle = isSelected ? "#040810" : "#c8d4e6"; - ctx.textAlign = "center"; - ctx.textBaseline = "top"; - ctx.fillText(label, node.x ?? 0, (node.y ?? 0) + 8 / scale + pad / 2); - // Selected ring - if (isSelected) { - ctx.beginPath(); - ctx.arc(node.x ?? 0, node.y ?? 0, 10 / scale, 0, 2 * Math.PI, false); - ctx.strokeStyle = "#00ff9c"; - ctx.lineWidth = 2.5 / scale; - ctx.stroke(); - } - }} - /> -
- ); -} diff --git a/web/lib/chat/tools.ts b/web/lib/chat/tools.ts index 96439eb..89e9083 100644 --- a/web/lib/chat/tools.ts +++ b/web/lib/chat/tools.ts @@ -314,6 +314,39 @@ const co_mention_chunks_tool: ToolDefinition = { }, }; +const analyze_image_region_tool: ToolDefinition = { + type: "function", + function: { + name: "analyze_image_region", + description: + "Vision tool — answer a question about a cropped region of a document page. " + + "Use this when the user asks about a photograph, diagram, sketch, signature, " + + "stamp, redaction, or any visual element where the chunk's text description " + + "isn't enough. The model reads the actual pixels via Sonnet vision. " + + "Get the bbox + page from a prior hybrid_search hit (each chunk carries bbox). " + + "Cost: ~$0.005–$0.02 per call. Use sparingly; prefer hybrid_search first.", + parameters: { + type: "object", + properties: { + doc_id: { type: "string" }, + page: { type: "integer", description: "1-indexed page number" }, + bbox: { + type: "object", + description: "Normalized bbox (0..1) of the region to analyze.", + properties: { + x: { type: "number" }, y: { type: "number" }, + w: { type: "number" }, h: { type: "number" }, + }, + required: ["x", "y", "w", "h"], + }, + question: { type: "string", description: "What you want to know about the image." }, + context: { type: "string", description: "Optional: prose context that grounds the model." }, + }, + required: ["doc_id", "page", "bbox", "question"], + }, + }, +}; + const navigate_to_tool: ToolDefinition = { type: "function", function: { @@ -345,6 +378,7 @@ export const TOOL_DEFINITIONS: ToolDefinition[] = [ read_document_tool, read_entity_tool, search_corpus_tool, + analyze_image_region_tool, navigate_to_tool, ]; @@ -398,6 +432,11 @@ async function handleHybridSearch( classification: (args.classification as string) || null, ufo_only: Boolean(args.ufo_only), top_k, + // W2-TD#8: chat is latency-sensitive — skip rerank when ≤10 candidates. + // The model only cites the first few hits anyway and BGE-Reranker + // adds 5-8s on CPU. RRF order from the RPC is plenty for the head. + rerank_strategy: "when_top_k_gt", + rerank_threshold: 10, }); // Emit one citation (+ optional crop_image) artifact per hit so the UI can // render inline cards next to the assistant text. Limit to top 6 to avoid @@ -684,6 +723,37 @@ async function handleNavigate(args: Record): Promise { return { ok: true, target, label }; } +async function handleAnalyzeImageRegion( + args: Record, + ctx: ToolHandlerContext, +): Promise { + const doc_id = String(args.doc_id ?? "").trim(); + const page = Number(args.page); + const bbox = args.bbox as { x: number; y: number; w: number; h: number } | undefined; + const question = String(args.question ?? "").trim(); + if (!doc_id || !page || !bbox || !question) return { error: "missing_args" }; + try { + const { analyzeImageRegion } = await import("./vision"); + const out = await analyzeImageRegion({ + doc_id, page, bbox, question, + context: typeof args.context === "string" ? args.context : undefined, + lang: ctx.lang === "en" ? "en" : "pt", + }); + if (ctx.emitArtifact) { + ctx.emitArtifact({ + kind: "crop_image", + src: out.crop_url, + doc_id, page, + alt_en: question.slice(0, 120), + alt_pt: question.slice(0, 120), + }); + } + return out; + } catch (e) { + return { error: "vision_failed", message: (e as Error).message }; + } +} + export const TOOL_HANDLERS: Record = { hybrid_search: handleHybridSearch, read_chunk: handleReadChunk, @@ -696,5 +766,6 @@ export const TOOL_HANDLERS: Record = { read_document: handleReadDocument, read_entity: handleReadEntity, search_corpus: handleSearch, + analyze_image_region: handleAnalyzeImageRegion, navigate_to: handleNavigate, }; diff --git a/web/lib/chat/vision.ts b/web/lib/chat/vision.ts new file mode 100644 index 0000000..12c84eb --- /dev/null +++ b/web/lib/chat/vision.ts @@ -0,0 +1,165 @@ +/** + * vision.ts — answer questions about an image region via Claude Code OAuth. + * + * Pattern matches the project's existing vision pipeline (02-vision-page.py): + * 1. Crop the PNG of the requested page to the requested bbox. + * 2. Spawn `claude -p --model sonnet --allowedTools Read` and instruct the + * model to Read the local PNG path and answer the user's question. + * + * Uses the user's Claude Code OAuth (Max 20x). Per W1.2 budget policy, the + * agentic worker may use Opus 4.7 without hard cap, but `analyze_image_region` + * runs synchronously in the chat path — keep it on Sonnet for latency. + */ +import { spawn } from "node:child_process"; +import { mkdtemp, unlink, rmdir } from "node:fs/promises"; +import path from "node:path"; +import os from "node:os"; +import sharp from "sharp"; +import { PROCESSING } from "@/lib/wiki"; + +const MODEL = process.env.VISION_MODEL || "sonnet"; +const TIMEOUT_MS = Number(process.env.VISION_TIMEOUT_MS || 120_000); + +export interface AnalyzeImageRegionArgs { + doc_id: string; + page: number; + bbox: { x: number; y: number; w: number; h: number }; + question: string; + /** Optional context to ground the model (e.g., "this is an FBI memo from 1947"). */ + context?: string; + /** Output language hint. Defaults to "pt-br". */ + lang?: "pt" | "en"; +} + +export interface AnalyzeImageRegionResult { + answer: string; + model: string; + duration_ms: number; + bbox: { x: number; y: number; w: number; h: number }; + crop_url: string; +} + +function pageFilename(page: number): string { + return `p-${String(page).padStart(3, "0")}.png`; +} + +/** Crop a bbox region of a page PNG and write the result to a temp file. */ +async function cropToTempFile(args: AnalyzeImageRegionArgs): Promise<{ path: string; dir: string }> { + const sourcePath = path.join(PROCESSING, "png", args.doc_id, pageFilename(args.page)); + const meta = await sharp(sourcePath).metadata(); + const W = meta.width ?? 0; + const H = meta.height ?? 0; + if (W === 0 || H === 0) throw new Error(`source PNG unreadable: ${sourcePath}`); + const left = Math.max(0, Math.round(args.bbox.x * W)); + const top = Math.max(0, Math.round(args.bbox.y * H)); + const width = Math.max(1, Math.min(W - left, Math.round(args.bbox.w * W))); + const height = Math.max(1, Math.min(H - top, Math.round(args.bbox.h * H))); + + const dir = await mkdtemp(path.join(os.tmpdir(), "analyze-image-")); + const filePath = path.join(dir, "crop.png"); + await sharp(sourcePath) + .extract({ left, top, width, height }) + .resize({ width: Math.min(1024, width), withoutEnlargement: true }) + .png() + .toFile(filePath); + return { path: filePath, dir }; +} + +function buildPrompt(args: AnalyzeImageRegionArgs, cropPath: string): string { + const lang = args.lang === "en" ? "English" : "Brazilian Portuguese (pt-br)"; + const ctx = args.context ? `\n\nContext: ${args.context}` : ""; + return [ + `Use the Read tool on this local PNG file: ${cropPath}`, + "", + `The image is a cropped region from document "${args.doc_id}", page ${args.page}.`, + ctx, + "", + `Answer this question about what is visible in the image, in ${lang}:`, + "", + args.question, + "", + "Rules:", + "- Be factual and concise (3-8 sentences unless the question requires more).", + "- If text is visible, transcribe the relevant portion verbatim (do not translate).", + "- If the image is unclear or empty, say so explicitly. Don't invent.", + "- Do not call any tool besides Read on the provided path.", + ].join("\n"); +} + +/** Spawn `claude -p` and return the JSON output result. */ +function callClaudeCli(prompt: string): Promise<{ result: string; durationMs: number; costUsd?: number; tokensIn?: number; tokensOut?: number }> { + return new Promise((resolve, reject) => { + const t0 = Date.now(); + const child = spawn( + "claude", + [ + "-p", + "--model", MODEL, + "--output-format", "json", + "--max-turns", "2", + "--allowedTools", "Read", + "--", + prompt, + ], + { stdio: ["ignore", "pipe", "pipe"], env: { ...process.env } }, + ); + let stdout = ""; + let stderr = ""; + child.stdout.on("data", (c) => (stdout += c.toString())); + child.stderr.on("data", (c) => (stderr += c.toString())); + const t = setTimeout(() => { + child.kill("SIGKILL"); + reject(new Error(`claude CLI timeout > ${TIMEOUT_MS}ms`)); + }, TIMEOUT_MS); + child.on("error", (e) => { clearTimeout(t); reject(e); }); + child.on("close", (code) => { + clearTimeout(t); + if (code !== 0) { + return reject(new Error(`claude CLI rc=${code}: ${stderr.slice(-300)}`)); + } + try { + const cli = JSON.parse(stdout); + if (cli.is_error) return reject(new Error(`claude error: ${(cli.result || "").slice(0, 300)}`)); + resolve({ + result: cli.result || "", + durationMs: cli.duration_ms || Date.now() - t0, + costUsd: cli.total_cost_usd, + tokensIn: cli.usage?.input_tokens, + tokensOut: cli.usage?.output_tokens, + }); + } catch (e) { + reject(new Error(`claude stdout parse: ${e instanceof Error ? e.message : String(e)}`)); + } + }); + }); +} + +export async function analyzeImageRegion(args: AnalyzeImageRegionArgs): Promise { + if (!args.doc_id) throw new Error("doc_id required"); + if (!Number.isFinite(args.page) || args.page < 1) throw new Error("page must be >= 1"); + if (!args.bbox || !["x", "y", "w", "h"].every((k) => Number.isFinite((args.bbox as Record)[k]))) { + throw new Error("bbox {x,y,w,h} required (normalized 0..1)"); + } + if (!args.question?.trim()) throw new Error("question required"); + + const { path: cropPath, dir } = await cropToTempFile(args); + try { + const t0 = Date.now(); + const prompt = buildPrompt(args, cropPath); + const out = await callClaudeCli(prompt); + const cropUrl = + `/api/crop?doc=${encodeURIComponent(args.doc_id)}` + + `&page=${args.page}&x=${args.bbox.x}&y=${args.bbox.y}&w=${args.bbox.w}&h=${args.bbox.h}&w_px=640`; + return { + answer: out.result.trim(), + model: MODEL, + duration_ms: out.durationMs || Date.now() - t0, + bbox: args.bbox, + crop_url: cropUrl, + }; + } finally { + // Best-effort cleanup. Crop is in $TMPDIR, OS will reap if we miss. + unlink(cropPath).catch(() => undefined); + rmdir(dir).catch(() => undefined); + } +} diff --git a/web/lib/retrieval/hybrid.ts b/web/lib/retrieval/hybrid.ts index 5474409..3836ed4 100644 --- a/web/lib/retrieval/hybrid.ts +++ b/web/lib/retrieval/hybrid.ts @@ -43,8 +43,24 @@ export interface HybridSearchOptions { recall_k?: number; /** Final list size after rerank (default 20). */ top_k?: number; - /** Skip reranker (faster, lower precision). */ + /** Skip reranker (faster, lower precision). Back-compat shortcut for + * `rerank_strategy: "never"`. */ no_rerank?: boolean; + /** + * W2-TD#8: rerank policy. + * - "always" — always run the cross-encoder (highest precision, + * slowest, 5–8s on CPU) + * - "when_top_k_gt" — rerank only when `top_k > rerank_threshold` + * (default threshold 15). RRF order from the RPC is + * usually good enough for the tight head of results; + * the reranker pays off when re-sorting a wider list. + * This is the new default — autocomplete / chat + * top-10 calls now skip rerank for free. + * - "never" — same as `no_rerank: true`. + */ + rerank_strategy?: "always" | "when_top_k_gt" | "never"; + /** Threshold for `when_top_k_gt`. Default 15 (per ADR-001). */ + rerank_threshold?: number; } export async function hybridSearch(opts: HybridSearchOptions): Promise { @@ -58,8 +74,14 @@ export async function hybridSearch(opts: HybridSearchOptions): Promise=12" - } - }, "node_modules/acorn": { "version": "8.16.0", "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.16.0.tgz", @@ -6335,16 +6319,6 @@ "node": ">=6.0.0" } }, - "node_modules/bezier-js": { - "version": "6.1.4", - "resolved": "https://registry.npmjs.org/bezier-js/-/bezier-js-6.1.4.tgz", - "integrity": "sha512-PA0FW9ZpcHbojUCMu28z9Vg/fNkwTj5YhusSAjHHDfHDGLxJ6YUKrAN2vk1fP2MMOxVw4Oko16FMlRGVBGqLKg==", - "license": "MIT", - "funding": { - "type": "individual", - "url": "https://github.com/Pomax/bezierjs/blob/master/FUNDING.md" - } - }, "node_modules/binary-extensions": { "version": "2.3.0", "resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz", @@ -6446,18 +6420,6 @@ ], "license": "CC-BY-4.0" }, - "node_modules/canvas-color-tracker": { - "version": "1.3.2", - "resolved": "https://registry.npmjs.org/canvas-color-tracker/-/canvas-color-tracker-1.3.2.tgz", - "integrity": "sha512-ryQkDX26yJ3CXzb3hxUVNlg1NKE4REc5crLBq661Nxzr8TNd236SaEf2ffYLXyI5tSABSeguHLqcVq4vf9L3Zg==", - "license": "MIT", - "dependencies": { - "tinycolor2": "^1.6.0" - }, - "engines": { - "node": ">=12" - } - }, "node_modules/ccount": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/ccount/-/ccount-2.0.1.tgz", @@ -6664,222 +6626,6 @@ "dev": true, "license": "MIT" }, - "node_modules/d3-array": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/d3-array/-/d3-array-3.2.4.tgz", - "integrity": "sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==", - "license": "ISC", - "dependencies": { - "internmap": "1 - 2" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-binarytree": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/d3-binarytree/-/d3-binarytree-1.0.2.tgz", - "integrity": "sha512-cElUNH+sHu95L04m92pG73t2MEJXKu+GeKUN1TJkFsu93E5W8E9Sc3kHEGJKgenGvj19m6upSn2EunvMgMD2Yw==", - "license": "MIT" - }, - "node_modules/d3-color": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/d3-color/-/d3-color-3.1.0.tgz", - "integrity": "sha512-zg/chbXyeBtMQ1LbD/WSoW2DpC3I0mpmPdW+ynRTj/x2DAWYrIY7qeZIHidozwV24m4iavr15lNwIwLxRmOxhA==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-dispatch": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-dispatch/-/d3-dispatch-3.0.1.tgz", - "integrity": "sha512-rzUyPU/S7rwUflMyLc1ETDeBj0NRuHKKAcvukozwhshr6g6c5d8zh4c2gQjY2bZ0dXeGLWc1PF174P2tVvKhfg==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-drag": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/d3-drag/-/d3-drag-3.0.0.tgz", - "integrity": "sha512-pWbUJLdETVA8lQNJecMxoXfH6x+mO2UQo8rSmZ+QqxcbyA3hfeprFgIT//HW2nlHChWeIIMwS2Fq+gEARkhTkg==", - "license": "ISC", - "dependencies": { - "d3-dispatch": "1 - 3", - "d3-selection": "3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-ease": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-ease/-/d3-ease-3.0.1.tgz", - "integrity": "sha512-wR/XK3D3XcLIZwpbvQwQ5fK+8Ykds1ip7A2Txe0yxncXSdq1L9skcG7blcedkOX+ZcgxGAmLX1FrRGbADwzi0w==", - "license": "BSD-3-Clause", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-force-3d": { - "version": "3.0.6", - "resolved": "https://registry.npmjs.org/d3-force-3d/-/d3-force-3d-3.0.6.tgz", - "integrity": "sha512-4tsKHUPLOVkyfEffZo1v6sFHvGFwAIIjt/W8IThbp08DYAsXZck+2pSHEG5W1+gQgEvFLdZkYvmJAbRM2EzMnA==", - "license": "MIT", - "dependencies": { - "d3-binarytree": "1", - "d3-dispatch": "1 - 3", - "d3-octree": "1", - "d3-quadtree": "1 - 3", - "d3-timer": "1 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-format": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/d3-format/-/d3-format-3.1.2.tgz", - "integrity": "sha512-AJDdYOdnyRDV5b6ArilzCPPwc1ejkHcoyFarqlPqT7zRYjhavcT3uSrqcMvsgh2CgoPbK3RCwyHaVyxYcP2Arg==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-interpolate": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-interpolate/-/d3-interpolate-3.0.1.tgz", - "integrity": "sha512-3bYs1rOD33uo8aqJfKP3JWPAibgw8Zm2+L9vBKEHJ2Rg+viTR7o5Mmv5mZcieN+FRYaAOWX5SJATX6k1PWz72g==", - "license": "ISC", - "dependencies": { - "d3-color": "1 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-octree": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/d3-octree/-/d3-octree-1.1.0.tgz", - "integrity": "sha512-F8gPlqpP+HwRPMO/8uOu5wjH110+6q4cgJvgJT6vlpy3BEaDIKlTZrgHKZSp/i1InRpVfh4puY/kvL6MxK930A==", - "license": "MIT" - }, - "node_modules/d3-quadtree": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-quadtree/-/d3-quadtree-3.0.1.tgz", - "integrity": "sha512-04xDrxQTDTCFwP5H6hRhsRcb9xxv2RzkcsygFzmkSIOJy3PeRJP7sNk3VRIbKXcog561P9oU0/rVH6vDROAgUw==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-scale": { - "version": "4.0.2", - "resolved": "https://registry.npmjs.org/d3-scale/-/d3-scale-4.0.2.tgz", - "integrity": "sha512-GZW464g1SH7ag3Y7hXjf8RoUuAFIqklOAq3MRl4OaWabTFJY9PN/E1YklhXLh+OQ3fM9yS2nOkCoS+WLZ6kvxQ==", - "license": "ISC", - "dependencies": { - "d3-array": "2.10.0 - 3", - "d3-format": "1 - 3", - "d3-interpolate": "1.2.0 - 3", - "d3-time": "2.1.1 - 3", - "d3-time-format": "2 - 4" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-scale-chromatic": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/d3-scale-chromatic/-/d3-scale-chromatic-3.1.0.tgz", - "integrity": "sha512-A3s5PWiZ9YCXFye1o246KoscMWqf8BsD9eRiJ3He7C9OBaxKhAd5TFCdEx/7VbKtxxTsu//1mMJFrEt572cEyQ==", - "license": "ISC", - "dependencies": { - "d3-color": "1 - 3", - "d3-interpolate": "1 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-selection": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/d3-selection/-/d3-selection-3.0.0.tgz", - "integrity": "sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-time": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/d3-time/-/d3-time-3.1.0.tgz", - "integrity": "sha512-VqKjzBLejbSMT4IgbmVgDjpkYrNWUYJnbCGo874u7MMKIWsILRX+OpX/gTk8MqjpT1A/c6HY2dCA77ZN0lkQ2Q==", - "license": "ISC", - "dependencies": { - "d3-array": "2 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-time-format": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/d3-time-format/-/d3-time-format-4.1.0.tgz", - "integrity": "sha512-dJxPBlzC7NugB2PDLwo9Q8JiTR3M3e4/XANkreKSUxF8vvXKqm1Yfq4Q5dl8budlunRVlUUaDUgFt7eA8D6NLg==", - "license": "ISC", - "dependencies": { - "d3-time": "1 - 3" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-timer": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-timer/-/d3-timer-3.0.1.tgz", - "integrity": "sha512-ndfJ/JxxMd3nw31uyKoY2naivF+r29V+Lc0svZxe1JvvIRmi8hUsrMvdOwgS1o6uBHmiz91geQ0ylPP0aj1VUA==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, - "node_modules/d3-transition": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/d3-transition/-/d3-transition-3.0.1.tgz", - "integrity": "sha512-ApKvfjsSR6tg06xrL434C0WydLr7JewBB3V+/39RMHsaXTOG0zmt/OAXeng5M5LBm0ojmxJrpomQVZ1aPvBL4w==", - "license": "ISC", - "dependencies": { - "d3-color": "1 - 3", - "d3-dispatch": "1 - 3", - "d3-ease": "1 - 3", - "d3-interpolate": "1 - 3", - "d3-timer": "1 - 3" - }, - "engines": { - "node": ">=12" - }, - "peerDependencies": { - "d3-selection": "2 - 3" - } - }, - "node_modules/d3-zoom": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/d3-zoom/-/d3-zoom-3.0.0.tgz", - "integrity": "sha512-b8AmV3kfQaqWAuacbPuNbL6vahnOJflOhexLzMMNLga62+/nh0JzvJ0aO/5a5MVgUFGS7Hu1P9P03o3fJkDCyw==", - "license": "ISC", - "dependencies": { - "d3-dispatch": "1 - 3", - "d3-drag": "2 - 3", - "d3-interpolate": "1 - 3", - "d3-selection": "2 - 3", - "d3-transition": "2 - 3" - }, - "engines": { - "node": ">=12" - } - }, "node_modules/debug": { "version": "4.4.3", "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", @@ -7194,46 +6940,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/float-tooltip": { - "version": "1.7.5", - "resolved": "https://registry.npmjs.org/float-tooltip/-/float-tooltip-1.7.5.tgz", - "integrity": "sha512-/kXzuDnnBqyyWyhDMH7+PfP8J/oXiAavGzcRxASOMRHFuReDtofizLLJsf7nnDLAfEaMW4pVWaXrAjtnglpEkg==", - "license": "MIT", - "dependencies": { - "d3-selection": "2 - 3", - "kapsule": "^1.16", - "preact": "10" - }, - "engines": { - "node": ">=12" - } - }, - "node_modules/force-graph": { - "version": "1.51.4", - "resolved": "https://registry.npmjs.org/force-graph/-/force-graph-1.51.4.tgz", - "integrity": "sha512-TdJ2KbkoiDQ7NIRx8IPGD0mAXXpLhamS7c+b7W98b0MHG7lphnda1VOQX/98UDTsttIAdH4TcP0l0MauSnLK8w==", - "license": "MIT", - "dependencies": { - "@tweenjs/tween.js": "18 - 25", - "accessor-fn": "1", - "bezier-js": "3 - 6", - "canvas-color-tracker": "^1.3", - "d3-array": "1 - 3", - "d3-drag": "2 - 3", - "d3-force-3d": "2 - 3", - "d3-scale": "1 - 4", - "d3-scale-chromatic": "1 - 3", - "d3-selection": "2 - 3", - "d3-zoom": "2 - 3", - "float-tooltip": "^1.7", - "index-array-by": "1", - "kapsule": "^1.16", - "lodash-es": "4" - }, - "engines": { - "node": ">=12" - } - }, "node_modules/forwarded-parse": { "version": "2.1.2", "resolved": "https://registry.npmjs.org/forwarded-parse/-/forwarded-parse-2.1.2.tgz", @@ -7509,30 +7215,12 @@ "node": ">=18" } }, - "node_modules/index-array-by": { - "version": "1.4.2", - "resolved": "https://registry.npmjs.org/index-array-by/-/index-array-by-1.4.2.tgz", - "integrity": "sha512-SP23P27OUKzXWEC/TOyWlwLviofQkCSCKONnc62eItjp69yCZZPqDQtr3Pw5gJDnPeUMqExmKydNZaJO0FU9pw==", - "license": "MIT", - "engines": { - "node": ">=12" - } - }, "node_modules/inline-style-parser": { "version": "0.2.7", "resolved": "https://registry.npmjs.org/inline-style-parser/-/inline-style-parser-0.2.7.tgz", "integrity": "sha512-Nb2ctOyNR8DqQoR0OwRG95uNWIC0C1lCgf5Naz5H6Ji72KZ8OcFZLz2P5sNgwlyoJ8Yif11oMuYs5pBQa86csA==", "license": "MIT" }, - "node_modules/internmap": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/internmap/-/internmap-2.0.3.tgz", - "integrity": "sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==", - "license": "ISC", - "engines": { - "node": ">=12" - } - }, "node_modules/is-alphabetical": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/is-alphabetical/-/is-alphabetical-2.0.1.tgz", @@ -7681,15 +7369,6 @@ "integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==", "license": "ISC" }, - "node_modules/jerrypick": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/jerrypick/-/jerrypick-1.1.2.tgz", - "integrity": "sha512-YKnxXEekXKzhpf7CLYA0A+oDP8V0OhICNCr5lv96FvSsDEmrb0GKM776JgQvHTMjr7DTTPEVv/1Ciaw0uEWzBA==", - "license": "MIT", - "engines": { - "node": ">=12" - } - }, "node_modules/jiti": { "version": "1.21.7", "resolved": "https://registry.npmjs.org/jiti/-/jiti-1.21.7.tgz", @@ -7743,18 +7422,6 @@ "node": ">=6" } }, - "node_modules/kapsule": { - "version": "1.16.3", - "resolved": "https://registry.npmjs.org/kapsule/-/kapsule-1.16.3.tgz", - "integrity": "sha512-4+5mNNf4vZDSwPhKprKwz3330iisPrb08JyMgbsdFrimBCKNHecua/WBwvVg3n7vwx0C1ARjfhwIpbrbd9n5wg==", - "license": "MIT", - "dependencies": { - "lodash-es": "4" - }, - "engines": { - "node": ">=12" - } - }, "node_modules/kind-of": { "version": "6.0.3", "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz", @@ -7799,12 +7466,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/lodash-es": { - "version": "4.18.1", - "resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.18.1.tgz", - "integrity": "sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==", - "license": "MIT" - }, "node_modules/longest-streak": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/longest-streak/-/longest-streak-3.1.0.tgz", @@ -7815,18 +7476,6 @@ "url": "https://github.com/sponsors/wooorm" } }, - "node_modules/loose-envify": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/loose-envify/-/loose-envify-1.4.0.tgz", - "integrity": "sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==", - "license": "MIT", - "dependencies": { - "js-tokens": "^3.0.0 || ^4.0.0" - }, - "bin": { - "loose-envify": "cli.js" - } - }, "node_modules/lru-cache": { "version": "5.1.1", "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-5.1.1.tgz", @@ -9511,6 +9160,7 @@ "version": "4.1.1", "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", + "dev": true, "license": "MIT", "engines": { "node": ">=0.10.0" @@ -10023,16 +9673,6 @@ "node": ">=0.10.0" } }, - "node_modules/preact": { - "version": "10.29.1", - "resolved": "https://registry.npmjs.org/preact/-/preact-10.29.1.tgz", - "integrity": "sha512-gQCLc/vWroE8lIpleXtdJhTFDogTdZG9AjMUpVkDf2iTCNwYNWA+u16dL41TqUDJO4gm2IgrcMv3uTpjd4Pwmg==", - "license": "MIT", - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/preact" - } - }, "node_modules/process-warning": { "version": "5.0.0", "resolved": "https://registry.npmjs.org/process-warning/-/process-warning-5.0.0.tgz", @@ -10058,17 +9698,6 @@ "node": ">=0.4.0" } }, - "node_modules/prop-types": { - "version": "15.8.1", - "resolved": "https://registry.npmjs.org/prop-types/-/prop-types-15.8.1.tgz", - "integrity": "sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg==", - "license": "MIT", - "dependencies": { - "loose-envify": "^1.4.0", - "object-assign": "^4.1.1", - "react-is": "^16.13.1" - } - }, "node_modules/property-information": { "version": "7.1.0", "resolved": "https://registry.npmjs.org/property-information/-/property-information-7.1.0.tgz", @@ -10248,44 +9877,6 @@ "react": "^19.2.6" } }, - "node_modules/react-force-graph-2d": { - "version": "1.29.1", - "resolved": "https://registry.npmjs.org/react-force-graph-2d/-/react-force-graph-2d-1.29.1.tgz", - "integrity": "sha512-1Rl/1Z3xy2iTHKj6a0jRXGyiI86xUti81K+jBQZ+Oe46csaMikp47L5AjrzA9hY9fNGD63X8ffrqnvaORukCuQ==", - "license": "MIT", - "dependencies": { - "force-graph": "^1.51", - "prop-types": "15", - "react-kapsule": "^2.5" - }, - "engines": { - "node": ">=12" - }, - "peerDependencies": { - "react": "*" - } - }, - "node_modules/react-is": { - "version": "16.13.1", - "resolved": "https://registry.npmjs.org/react-is/-/react-is-16.13.1.tgz", - "integrity": "sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==", - "license": "MIT" - }, - "node_modules/react-kapsule": { - "version": "2.5.7", - "resolved": "https://registry.npmjs.org/react-kapsule/-/react-kapsule-2.5.7.tgz", - "integrity": "sha512-kifAF4ZPD77qZKc4CKLmozq6GY1sBzPEJTIJb0wWFK6HsePJatK3jXplZn2eeAt3x67CDozgi7/rO8fNQ/AL7A==", - "license": "MIT", - "dependencies": { - "jerrypick": "^1.1.1" - }, - "engines": { - "node": ">=12" - }, - "peerDependencies": { - "react": ">=16.13.1" - } - }, "node_modules/react-markdown": { "version": "9.1.0", "resolved": "https://registry.npmjs.org/react-markdown/-/react-markdown-9.1.0.tgz", @@ -10991,12 +10582,6 @@ "integrity": "sha512-P4nbQYQfePJxRSmY+v/KINxVucm4NF3p3s7pJveMTtom52FR4YGltUQLB8idDXwDDWW+eYrWDFbuzUnjoWHF7g==", "license": "MIT" }, - "node_modules/tinycolor2": { - "version": "1.6.0", - "resolved": "https://registry.npmjs.org/tinycolor2/-/tinycolor2-1.6.0.tgz", - "integrity": "sha512-XPaBkWQJdsf3pLKJV9p4qN/S+fm2Oj8AIPo1BTUhg5oxkvm9+SVEGFdhyOz7tTdUTfvxMiAs4sp6/eZO2Ew+pw==", - "license": "MIT" - }, "node_modules/tinyglobby": { "version": "0.2.16", "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz", diff --git a/web/package.json b/web/package.json index 0f5112d..633d2db 100644 --- a/web/package.json +++ b/web/package.json @@ -28,7 +28,6 @@ "pino": "^10.3.1", "react": "^19.0.0", "react-dom": "^19.0.0", - "react-force-graph-2d": "^1.27.0", "react-markdown": "^9.0.0", "remark-gfm": "^4.0.0", "remark-wiki-link": "^2.0.1",