disclosure-bureau/infra/DEPLOY-CHECKLIST.md

Deploy Checklist — Retrieval Layer Activation

Runbook end-to-end para ativar pgvector + BGE-M3 + reranker no Disclosure Bureau VPS.

Assume: VPS já tem disclosure-stack (Supabase + Next.js + Meilisearch) rodando, e o batch scripts/28-batch-rebuild-all.py já produziu chunks em raw/<doc>--subagent/ (parcial ou completo).

0. Pré-condições

# checa que chunks existem
ls -d /Users/guto/ufo/raw/*--subagent | wc -l   # esperado ≥ 1

# checa que .env tem POSTGRES_PASSWORD + DATABASE_URL + EMBED_SERVICE_URL
grep -E "POSTGRES_PASSWORD|DATABASE_URL|EMBED_SERVICE_URL" infra/disclosure-stack/.env

Se o .env ainda não tem as novas linhas, copia do .env.example:

DATABASE_URL=postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres
EMBED_SERVICE_URL=http://embed:8000

1. Aplicar migração 02 (pgvector + chunks schema)

Sobe o stack — bootstrap.sh já aplica as migrações automaticamente (vê linhas 86-92):

cd infra/disclosure-stack
./scripts/deploy.sh        # rsync + docker compose up; aplica 00-init, 01-chat, 02-chunks

Se quiser aplicar só a 02 manualmente:

./scripts/ssh.sh
cd /data/disclosure
docker exec -i disclosure-db psql -U postgres < migrations/02-chunks-retrieval.sql

Verificar:

docker exec -i disclosure-db psql -U postgres -c "\dx" | grep vector
# extensão `vector` instalada ✓

docker exec -i disclosure-db psql -U postgres -c "\dt public.*"
# documents, chunks, entities, entity_mentions ✓

docker exec -i disclosure-db psql -U postgres -c "\df public.hybrid_search_chunks"
# função RPC ✓

2. Subir o embed-service (BGE-M3 + reranker)

Já está no docker-compose.yml. Primeiro build leva 5-10 min (baixa torch CPU + FlagEmbedding). Primeira request leva mais 5-8 s pra carregar modelos.

./scripts/ssh.sh
cd /data/disclosure
docker compose build embed
docker compose up -d embed
docker compose logs -f embed

Verificar (de dentro do VPS via internal network):

docker exec disclosure-embed curl -s http://localhost:8000/health
# {"status":"ok","embed_loaded":false,"rerank_loaded":false}

docker exec disclosure-embed curl -s -X POST http://localhost:8000/embed \
  -H 'content-type: application/json' \
  -d '{"texts":["UAP sobre Kansas em 1950"]}'
# primeira call: ~5s (model load). retorna {model,dim:1024,embeddings:[[...]]}

3. Indexar chunks → Postgres

./scripts/ssh.sh
cd /data/disclosure

# instalar deps Python (se ainda não)
pip3 install psycopg[binary] pyyaml requests

# pegar a senha do postgres pra montar DATABASE_URL local
source /data/disclosure/.env

# rodar indexer dentro de um container que tem rede internal
docker run --rm \
  --network disclosure-internal \
  -v /data/ufo:/data/ufo:ro \
  -e DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres" \
  -e EMBED_SERVICE_URL=http://embed:8000 \
  python:3.11-slim \
  bash -c "pip install -q psycopg[binary] pyyaml requests && \
           python3 /data/ufo/scripts/30-index-chunks-to-db.py --skip-existing"

Verificar:

docker exec -i disclosure-db psql -U postgres -c \
  "SELECT COUNT(*) FROM public.chunks WHERE embedding IS NOT NULL;"
# esperado: total = soma de chunks em raw/*--subagent/chunks/

4. Materializar entity_mentions

docker run --rm \
  --network disclosure-internal \
  -v /data/ufo:/data/ufo:ro \
  -e DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres" \
  python:3.11-slim \
  bash -c "pip install -q psycopg[binary] pyyaml && \
           python3 /data/ufo/scripts/31-populate-entity-mentions.py"

# tempo: ~30min para 34k entidades

Verificar:

docker exec -i disclosure-db psql -U postgres -c \
  "SELECT COUNT(*) FROM public.entity_mentions;"
# esperado: dezenas de milhares

5. Sync mentioned_in[] → markdown (opcional, fecha loop)

Esta é a única etapa que ESCREVE em wiki/. Use --dry-run primeiro.

docker run --rm \
  --network disclosure-internal \
  -v /data/ufo:/data/ufo \
  -e DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres" \
  python:3.11-slim \
  bash -c "pip install -q psycopg[binary] pyyaml && \
           python3 /data/ufo/scripts/32-sync-mentioned-in-yaml.py"

6. Configurar Next.js para usar a DB

No disclosure-stack/.env:

DATABASE_URL=postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres
EMBED_SERVICE_URL=http://embed:8000

Reiniciar o container web:

docker compose restart web

7. Smoke test end-to-end

Acessar pelas URLs do Traefik:

URL	Esperado
https://app.disclosure.top/admin/stats	dashboard com counts da DB
https://app.disclosure.top/admin/indexer	"✓ retrieval operacional"
https://app.disclosure.top/search?q=Olathe	hits semânticos
https://app.disclosure.top/graph	force graph com co-mentions
https://app.disclosure.top/timeline	eventos por década
https://app.disclosure.top/d/doc-342-.../v2	chunks renderizados inline
https://app.disclosure.top/d/doc-342-.../v2/p001	single page view
Cmd+K em qualquer página	command palette com hybrid_search
chat: "qual foi a forma dos objetos em Olathe?"	resposta cita [[doc-id/p001#c0008]] que vira card com crop

8. Manutenção periódica

# compactar progress.jsonl (após várias retries do batch)
python3 scripts/33-compact-progress-log.py

# reindexar docs novos que apareceram no disco mas ainda não estão na DB
python3 scripts/30-index-chunks-to-db.py --skip-existing

# regenerar entity_mentions quando 31 novos chunks são adicionados
python3 scripts/31-populate-entity-mentions.py --reset

9. Troubleshooting

Sintoma	Causa	Fix
/api/search/hybrid 503	embed-service down	docker compose logs embed
/api/admin/indexer mostra "db_error"	DATABASE_URL errada ou DB parado	docker compose logs db
chat retorna "retrieval_unavailable"	DB ou embed-service inacessíveis	restart no compose
/graph vazio	entity_mentions não populado	rodar 31-populate-entity-mentions.py
/timeline vazio	events sem date_start no frontmatter	revisar wiki/entities/events/
bbox crop 500	PNG faltando em processing/	rodar 01-convert-pdfs.sh
Anthropic 429 no batch	quota Max 20x (5h window)	esperar reset; orchestrator agora aborta cedo

10. Custos recorrentes

• Embedding (BGE-M3 self-host): $0
• Reranker (BGE-Reranker-v2-M3 self-host): $0
• Postgres + pgvector: já incluso no plano VPS
• LLM (chat agent): OpenRouter — deepseek-v4-flash:free ($0) ou paid model conforme tier
• Re-rebuild de docs: só quando schema mudar (Anthropic API ou Claude Code Max quota)