disclosure-bureau/infra/DEPLOY-CHECKLIST.md

# Deploy Checklist — Retrieval Layer Activation

Runbook end-to-end para ativar pgvector + BGE-M3 + reranker no Disclosure Bureau VPS.

> Assume: VPS já tem disclosure-stack (Supabase + Next.js + Meilisearch) rodando, e o batch `scripts/28-batch-rebuild-all.py` já produziu chunks em `raw/<doc>--subagent/` (parcial ou completo).

## 0. Pré-condições

```bash
# checa que chunks existem
ls -d /Users/guto/ufo/raw/*--subagent | wc -l   # esperado ≥ 1

# checa que .env tem POSTGRES_PASSWORD + DATABASE_URL + EMBED_SERVICE_URL
grep -E "POSTGRES_PASSWORD|DATABASE_URL|EMBED_SERVICE_URL" infra/disclosure-stack/.env
```

Se o `.env` ainda não tem as novas linhas, copia do `.env.example`:

```
DATABASE_URL=postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres
EMBED_SERVICE_URL=http://embed:8000
```

## 1. Aplicar migração 02 (pgvector + chunks schema)

Sobe o stack — bootstrap.sh já aplica as migrações automaticamente (vê linhas 86-92):

```bash
cd infra/disclosure-stack
./scripts/deploy.sh        # rsync + docker compose up; aplica 00-init, 01-chat, 02-chunks
```

Se quiser aplicar só a 02 manualmente:

```bash
./scripts/ssh.sh
cd /data/disclosure
docker exec -i disclosure-db psql -U postgres < migrations/02-chunks-retrieval.sql
```

**Verificar:**

```bash
docker exec -i disclosure-db psql -U postgres -c "\dx" | grep vector
# extensão `vector` instalada ✓

docker exec -i disclosure-db psql -U postgres -c "\dt public.*"
# documents, chunks, entities, entity_mentions ✓

docker exec -i disclosure-db psql -U postgres -c "\df public.hybrid_search_chunks"
# função RPC ✓
```

## 2. Subir o embed-service (BGE-M3 + reranker)

Já está no `docker-compose.yml`. Primeiro build leva 5-10 min (baixa torch CPU + FlagEmbedding). Primeira request leva mais 5-8 s pra carregar modelos.

```bash
./scripts/ssh.sh
cd /data/disclosure
docker compose build embed
docker compose up -d embed
docker compose logs -f embed
```

**Verificar (de dentro do VPS via internal network):**

```bash
docker exec disclosure-embed curl -s http://localhost:8000/health
# {"status":"ok","embed_loaded":false,"rerank_loaded":false}

docker exec disclosure-embed curl -s -X POST http://localhost:8000/embed \
  -H 'content-type: application/json' \
  -d '{"texts":["UAP sobre Kansas em 1950"]}'
# primeira call: ~5s (model load). retorna {model,dim:1024,embeddings:[[...]]}
```

## 3. Indexar chunks → Postgres

```bash
./scripts/ssh.sh
cd /data/disclosure

# instalar deps Python (se ainda não)
pip3 install psycopg[binary] pyyaml requests

# pegar a senha do postgres pra montar DATABASE_URL local
source /data/disclosure/.env

# rodar indexer dentro de um container que tem rede internal
docker run --rm \
  --network disclosure-internal \
  -v /data/ufo:/data/ufo:ro \
  -e DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres" \
  -e EMBED_SERVICE_URL=http://embed:8000 \
  python:3.11-slim \
  bash -c "pip install -q psycopg[binary] pyyaml requests && \
           python3 /data/ufo/scripts/30-index-chunks-to-db.py --skip-existing"
```

**Verificar:**

```bash
docker exec -i disclosure-db psql -U postgres -c \
  "SELECT COUNT(*) FROM public.chunks WHERE embedding IS NOT NULL;"
# esperado: total = soma de chunks em raw/*--subagent/chunks/
```

## 4. Materializar `entity_mentions`

```bash
docker run --rm \
  --network disclosure-internal \
  -v /data/ufo:/data/ufo:ro \
  -e DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres" \
  python:3.11-slim \
  bash -c "pip install -q psycopg[binary] pyyaml && \
           python3 /data/ufo/scripts/31-populate-entity-mentions.py"

# tempo: ~30min para 34k entidades
```

**Verificar:**

```bash
docker exec -i disclosure-db psql -U postgres -c \
  "SELECT COUNT(*) FROM public.entity_mentions;"
# esperado: dezenas de milhares
```

## 5. Sync `mentioned_in[]` → markdown (opcional, fecha loop)

Esta é a única etapa que ESCREVE em `wiki/`. Use `--dry-run` primeiro.

```bash
docker run --rm \
  --network disclosure-internal \
  -v /data/ufo:/data/ufo \
  -e DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres" \
  python:3.11-slim \
  bash -c "pip install -q psycopg[binary] pyyaml && \
           python3 /data/ufo/scripts/32-sync-mentioned-in-yaml.py"
```

## 6. Configurar Next.js para usar a DB

No `disclosure-stack/.env`:

```
DATABASE_URL=postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres
EMBED_SERVICE_URL=http://embed:8000
```

Reiniciar o container web:

```bash
docker compose restart web
```

## 7. Smoke test end-to-end

Acessar pelas URLs do Traefik:

| URL | Esperado |
|---|---|
| `https://app.disclosure.top/admin/stats` | dashboard com counts da DB |
| `https://app.disclosure.top/admin/indexer` | "✓ retrieval operacional" |
| `https://app.disclosure.top/search?q=Olathe` | hits semânticos |
| `https://app.disclosure.top/graph` | force graph com co-mentions |
| `https://app.disclosure.top/timeline` | eventos por década |
| `https://app.disclosure.top/d/doc-342-.../v2` | chunks renderizados inline |
| `https://app.disclosure.top/d/doc-342-.../v2/p001` | single page view |
| `Cmd+K` em qualquer página | command palette com hybrid_search |
| chat: "qual foi a forma dos objetos em Olathe?" | resposta cita `[[doc-id/p001#c0008]]` que vira card com crop |

## 8. Manutenção periódica

```bash
# compactar progress.jsonl (após várias retries do batch)
python3 scripts/33-compact-progress-log.py

# reindexar docs novos que apareceram no disco mas ainda não estão na DB
python3 scripts/30-index-chunks-to-db.py --skip-existing

# regenerar entity_mentions quando 31 novos chunks são adicionados
python3 scripts/31-populate-entity-mentions.py --reset
```

## 9. Troubleshooting

| Sintoma | Causa | Fix |
|---|---|---|
| `/api/search/hybrid` 503 | embed-service down | `docker compose logs embed` |
| `/api/admin/indexer` mostra "db_error" | DATABASE_URL errada ou DB parado | `docker compose logs db` |
| chat retorna `"retrieval_unavailable"` | DB ou embed-service inacessíveis | restart no compose |
| `/graph` vazio | entity_mentions não populado | rodar `31-populate-entity-mentions.py` |
| `/timeline` vazio | events sem `date_start` no frontmatter | revisar wiki/entities/events/ |
| bbox crop 500 | PNG faltando em processing/ | rodar `01-convert-pdfs.sh` |
| Anthropic 429 no batch | quota Max 20x (5h window) | esperar reset; orchestrator agora aborta cedo |

## 10. Custos recorrentes

- **Embedding (BGE-M3 self-host)**: $0
- **Reranker (BGE-Reranker-v2-M3 self-host)**: $0
- **Postgres + pgvector**: já incluso no plano VPS
- **LLM (chat agent)**: OpenRouter — deepseek-v4-flash:free ($0) ou paid model conforme tier
- **Re-rebuild de docs**: só quando schema mudar (Anthropic API ou Claude Code Max quota)