discadmin/disclosure-bureau
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

baseline: Disclosure Bureau pipeline + Next.js UI + Supabase stack

Browse source
This commit is contained in:
guto 2026-05-17 22:44:36 -03:00
commit 19d0678e55
191 changed files with 51488 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

31
.gitignore vendored Normal file
View file

@ -0,0 +1,31 @@
# Bulk data — managed separately, not source
raw/
processing/
wiki/
# Build artifacts
web/.next/
web/node_modules/
web/.env.local
web/tsconfig.tsbuildinfo
# Logs / temp / OS
*.log
.DS_Store
*.swp
*.bak
# Secrets
infra/disclosure-stack/.env
**/*.env.local
# Python
__pycache__/
*.pyc
.venv/
.python-version
# Local-only output
case/case-report.md
case/residual-uncertainty.md
infra/disclosure-stack/.env.backup.*

1205
CLAUDE-schema-full.md Normal file
View file

File diff suppressed because it is too large Load diff

229
CLAUDE.md Normal file
View file

@ -0,0 +1,229 @@
# CLAUDE.md — Contrato Vinculante da Wiki UFO/UAP
> Versão `0.1.0` · Última atualização `2026-05-13` · Schema canônico em [`CLAUDE-schema-full.md`](CLAUDE-schema-full.md)
Toda agente que tocar este projeto **lê este arquivo no boot**. Operar lendo apenas este contrato é suficiente para tarefas correntes — detalhes de schema vivem em `CLAUDE-schema-full.md`.
## 1. Filosofia em uma frase
Wiki investigativa estilo **Karpathy LLM Wiki** + **Investigation Bureau** (8 detetives Holmes/Poirot/Dupin/Locard + Schneier/Tetlock/Taleb). Markdown puro, sem RAG, com procedência absoluta de cada claim.
## 2. Layout
```
/Users/guto/ufo/
├── CLAUDE.md ← este arquivo (contrato)
├── CLAUDE-schema-full.md ← schema completo dos 24 tipos
├── raw/ ← IMUTÁVEL (115 PDFs + 14 JPG/PNG)
├── processing/ ← intermediário (PNGs, OCR, vision raw)
├── wiki/ ← GERADO (documents, pages, entities, tables, images)
├── case/ ← Investigation Bureau (evidence, witnesses, hypotheses, ...)
└── scripts/ ← pipelines de ingest, dedup, lint
```
**Regra de ouro:** nada escreve em `raw/`. Referências usam path relativo `../raw/<file>.pdf`.
## 3. Idioma — bilíngue EN + PT-BR (português brasileiro)
A wiki é **bilíngue** desde o ingest. A mesma chamada Haiku vision gera EN e PT-BR juntos (single pass, preserva contexto visual da imagem).
| Categoria de campo | Idioma |
|---|---|
| YAML keys | **English** (international standard) |
| OCR raw text | **Source language only** (verbatim, no translation) |
| `verbatim_excerpt` (evidence), `verbatim_quotes` (person), `caption_ocr` (image) | **Source language only** |
| Enums (`page_type`, `content_classification`, `evidence_grade`, `confidence_band`, redaction codes, classification markings) | **English** (universal) |
| `canonical_name`, technical IDs | **Source language**; aliases array can hold PT-BR forms |
| Narrative descriptions (`vision_description`, `narrative_summary`, `executive_summary`, `description` in gaps, `definition_short` in concepts, `verdict_rationale` in witnesses) | **Both EN and PT-BR** via sibling fields `vision_description` + `vision_description_pt_br` etc. |
| Markdown body sections (headings + commentary) | **Both EN and PT-BR** in adjacent sections: `## Vision Description (EN)` then `## Descrição Vision (PT-BR)` |
**PT-BR rules:**
- Must be **Brazilian Portuguese** (`pt-br`), NOT European Portuguese. Use Brazilian vocabulary and spelling.
- Preserve UTF-8 accents correctly: `ç`, `ã`, `á`, `é`, `í`, `ó`, `ú`, `â`, `ê`, `ô`, `à`. Never strip accents.
- When a verbatim quote from the document appears inside a narrative paragraph, keep the **quote** in source language and translate only the surrounding narration.
- IDs always ASCII-fold (kebab-case without accents). Display fields (`canonical_name`) preserve accents when applicable.
Encoding: **always UTF-8**.
## 4. Os 24 tipos de markdown
| Tipo | Caminho | Owner |
|---|---|---|
| `document` | `wiki/documents/<doc-id>.md` | archivist |
| `page` | `wiki/pages/<doc-id>/p<NNN>.md` | archivist + evidence-officer |
| `person` | `wiki/entities/people/<id>.md` | profiler |
| `organization` | `wiki/entities/organizations/<id>.md` | profiler |
| `location` | `wiki/entities/locations/<id>.md` | archivist |
| `event` | `wiki/entities/events/<id>.md` | timeline-analyst |
| `uap_object` | `wiki/entities/uap-objects/<id>.md` | evidence-officer |
| `vehicle` | `wiki/entities/vehicles/<id>.md` | archivist |
| `operation` | `wiki/entities/operations/<id>.md` | archivist |
| `concept` | `wiki/entities/concepts/<id>.md` | archivist |
| `table` | `wiki/tables/<table-id>.md` | archivist |
| `image` | `wiki/images/<image-id>.md` | evidence-officer |
| `evidence` | `case/evidence/<E-NNNN>.md` | evidence-officer |
| `witness_analysis` | `case/witnesses/<W-NNNN>.md` | witness-officer |
| `timeline` | `case/timelines/<scope>.md` | timeline-analyst |
| `hypothesis` | `case/hypotheses/<H-NNNN>.md` | hypothesis-lead |
| `actor_profile` | `case/profiles/<AP-NNNN>.md` | profiler |
| `gap` | `case/gaps/<G-NNNN>.md` | archivist + chief-detective |
| `relation` | `case/connect-the-dots/<R-NNNN>.md` | chief-detective |
| `case_report` | `case/case-report.md` | case-writer |
| `residual_uncertainty` | `case/residual-uncertainty.md` | chief-detective |
| `index` | `wiki/index.md` | archivist |
| `log` | `wiki/log.md` | archivist (append-only) |
| (este) | `CLAUDE.md` | chief-detective |
Schemas de frontmatter detalhados em [`CLAUDE-schema-full.md`](CLAUDE-schema-full.md).
## 5. Frontmatter obrigatório universal
Todo arquivo `.md` em `wiki/` e `case/` tem:
```yaml
---
schema_version: "0.1.0"
type: <enum> # document | page | person | ... (24 tipos)
canonical_title: "..." # OU canonical_name (entidades)
wiki_version: "0.1.0"
last_ingest: "2026-05-13T14:22:11Z" # OU last_revised
---
```
## 6. Naming canônico (regex)
| Tipo | Regex | Exemplo |
|---|---|---|
| `doc_id` | `^[a-z0-9][a-z0-9-]*$` | `dow-uap-d54-mission-report-mediterranean-sea-na` |
| `page_id` | `^[a-z0-9-]+/p\d{3}$` | `dow-uap-d54-.../p007` |
| `person_id` | `^[a-z][a-z0-9-]*$` (ASCII-fold) | `j-edgar-hoover` |
| `event_id` | `^EV-\d{4}-(\d{2}\|XX)-(\d{2}\|XX)-[a-z0-9-]+$` | `EV-2004-11-14-tic-tac-nimitz` |
| `uap_object_id` | `^OBJ-[A-Z0-9-]+-\d{2}$` | `OBJ-EV2004-NIMITZ-01` |
| `evidence_id` | `^E-\d{4}$` | `E-0042` |
| `witness_id` | `^W-\d{4}$` | `W-0007` |
| `hypothesis_id` | `^H-\d{4}$` | `H-0003` |
| `table_id` | `^TBL-[A-Z0-9]+-\d{4}$` | `TBL-DOWD54-0003` |
| `image_id` | `^IMG-[A-Z0-9]+-p\d{3}-\d{2}$` | `IMG-DOWD54-p007-01` |
| `gap_id` | `^G-\d{4}$` | `G-0012` |
| `relation_id` | `^R-\d{4}$` | `R-0028` |
| `actor_profile_id` | `^AP-\d{4}$` | `AP-0001` |
### Algoritmo `filename → doc_id`
```
1. Strip extension (.pdf, .jpg, .png)
2. NFD + remove combining marks (ASCII fold)
3. Lowercase
4. Replace whitespace/underscore/non-[a-z0-9-] com "-"
5. Collapse "-" repetidos
6. Trim "-" inicial/final
7. Se começa com dígito, prefixa "doc-"
```
## 7. Wiki-links — 18 namespaces
```
[[doc-id]] → wiki/documents/<doc-id>.md
[[doc-id/pNNN]] → wiki/pages/<doc-id>/p<NNN>.md
[[people/<id>]] → wiki/entities/people/<id>.md
[[org/<id>]] → wiki/entities/organizations/<id>.md
[[loc/<id>]] → wiki/entities/locations/<id>.md
[[event/<id>]] → wiki/entities/events/<id>.md
[[uap/<id>]] → wiki/entities/uap-objects/<id>.md
[[vehicle/<id>]] → wiki/entities/vehicles/<id>.md
[[op/<id>]] → wiki/entities/operations/<id>.md
[[concept/<id>]] → wiki/entities/concepts/<id>.md
[[table/<id>]] [[image/<id>]] → wiki/tables|images/<id>.md
[[evidence/<id>]] [[witness/<id>]]
[[hypothesis/<id>]] [[profile/<id>]]
[[gap/<id>]] [[relation/<id>]] → case/...
[[people/...|Grusch]] → custom display text
```
**Backlinks** (`mentioned_in[]` em entidades) são **materializados pelo Lint, NÃO escritos à mão**.
## 8. Confidence calibration (Tetlock)
| Banda | Faixa | Linguagem permitida |
|---|---|---|
| `high` | ≥0.90 | "demonstra", "estabelece" |
| `medium` | 0.60–0.89 | "sugere fortemente", "indica" |
| `low` | 0.30–0.59 | "possivelmente", "pode" |
| `speculation` | <0.30 | "hipótese", "especulação" — sempre rotulado |
Toda claim em sumário executivo carrega `confidence_band`.
## 9. Classificação de conteúdo (`content_classification`)
Array enum em `document` e `page`:
- `text-only` · `contains-photos` · `contains-sketches` · `contains-diagrams` · `contains-maps` · `contains-tables` · `contains-signatures` · `contains-stamps` · `redaction-heavy` (>30% redacted) · `mixed` · `blank`
Doc-level = união dos valores das páginas.
## 10. Procedência (Locard)
- Toda `evidence` aponta `source_page` + `bbox` (opcional).
- Toda claim em entidade tem `mentioned_in[]` com `page_ref`.
- `chain_of_custody[]` obrigatório em evidence; `custody_gaps[]` explícitos.
- Grade A → ≥3 custody steps · Grade B → ≥2 · Grade C → ≥1
## 11. Operações canônicas
1. **INGEST** — PDF → PNG por página → vision Haiku → `page.md` + entity upsert
2. **LINT** — scan reverso, materializa `mentioned_in[]`, valida wiki-links, reporta orphans
3. **QUERY** — leitura por wiki-link traversal; nunca via embeddings
Log toda operação em `wiki/log.md` (append-only, formato fixo).
## 12. Quality gates (chief-detective enforça)
Threshold global **0.85** em 6 rubrics no `case-report.md`:
1. `chain_of_custody_completeness`
2. `confidence_calibration_match`
3. `hypothesis_tournament_discipline` (≥3 hipóteses)
4. `residual_uncertainty_presence`
5. `audit_trail_per_claim`
6. `red_team_pass`
Lint adicional **bloqueante**:
- Wiki-links resolvem 100%
- `entity.mentioned_in` ↔ `page.entities_extracted` consistente
- Nenhum `canonical_name` duplicado sem `disambiguation_note`
- `pages[]` contínuo `1..page_count` por documento
## 13. Triggers de enrichment externo
- **≥3 menções OU central claim** → `enrichment_status: deep` (WebSearch + ≥2 `external_sources`)
- **1-2 menções** → `enrichment_status: shallow` (1 query + knowledge interno)
- **0 menções** (inferida) → `enrichment_status: none`
## 14. Idempotência
Re-ingest do mesmo PDF (mesmo `sha256`) atualiza `last_ingest`, preserva `created_at`. Re-lint sobrescreve `mentioned_in[]` mas não duplica.
## 15. Escalation
Agente encontra:
- **Contradição entre evidências grade A/B** → escalar `chief-detective`
- **Hypothesis sobrevivente com posterior >0.70** → revisão multi-detective
- **Gap critical** → criar `[[gap/G-NNNN]]` + linkar em `case-report`
## 16. Modelo
Default para ingest, vision, dedup, lint, enrichment, e geração de markdown: **`claude-haiku-4-5`**.
`case-writer` (narrativa Holmes-Watson final) e `chief-detective` (red team review) podem opcionalmente usar Sonnet para qualidade final.
## 17. Stack de execução
- **PDF → PNG**: `pdftoppm -r 200` (Poppler)
- **PDF → texto**: `pdftotext -layout`
- **Vision**: Anthropic SDK Python + Haiku, com prompt caching e `pdf-2025-03-04` beta header se aplicável
- **Linting**: Python (PyYAML + regex)
Scripts em `/Users/guto/ufo/scripts/`.

95
CORPUS-SNAPSHOT.md Normal file
View file

@ -0,0 +1,95 @@
# Corpus Snapshot — Disclosure Bureau v0.2.0
> Generated: 2026-05-17 · Após batch rebuild completo
## Totais
| Métrica | Valor |
|---|---|
| **Documentos arquivados** | 116/115 (100% — extra: doc-342 test) |
| **Chunks totais** | 20.935 |
| **Páginas processadas** | 3.359 |
| **Imagens cropadas** | 752 (bilíngue desc + UAP check) |
| **UFO anomaly flagged** | 3.020 chunks (14.4%) |
| **Cryptid anomaly flagged** | 21 chunks (0.1%) |
| **Disk usage** | 634 MB (`raw/*--subagent`) |
| **Custo cumulativo** | ~$409 USD |
| **Validação harness** | 93 ok · 23 com warnings YAML (body OK) |
## Cobertura por coleção
- **DOW-UAP D1-D75** (75 docs Mission Reports DoD 2020-2025): completos
- **65 HS1 FBI Vault** (12 sections + 16 serials + sub-a): completos
- **NASA Apollo** (transcripts 11/12/17/Skylab — 5 docs): completos
- **DOS-UAP** (cables diplomáticos 3 docs): completos
- **FBI photos b2-b24** (10 docs): completos
- **059UAP** (FOIA — 3 docs): completos
- **doc-18, doc-255, doc-331, doc-341, doc-342**: completos
- **serials redacted** (3-5): completos
- **western-us-event-slides** + **usper-statement**: completos
## Estrutura por documento
Cada `raw/<doc-id>--subagent/` contém:
```
├── document.md assembled bilingual master (EN+PT-BR inline)
├── _index.json ordered chunk index (harness reassembly)
├── chunks/
│ ├── c0001.md 1 file per chunk, frontmatter rico (bbox+type+anomaly)
│ ├── c0002.md
│ └── ...
├── images/ cropped bbox regions PNG (with bilingual descriptions)
└── tables/ stitched multi-page tables CSV
```
## Próximos passos pré-deploy
Toda infra do retrieval layer está construída (`scripts/30,31,32,33` + `embed-service/` + migration `0002_chunks_retrieval.sql`). Para ativar:
```bash
# Sobe stack na VPS (embed-service + pgvector migration aplicada)
cd infra/disclosure-stack && ./scripts/deploy.sh
# Index chunks → Postgres + BGE-M3 embeddings
python3 scripts/30-index-chunks-to-db.py
# Materialize entity_mentions (chunk ↔ entity, ~30min)
python3 scripts/31-populate-entity-mentions.py
# Sync mentioned_in[] → markdown (fecha loop wiki ↔ DB)
python3 scripts/32-sync-mentioned-in-yaml.py
# (Manutenção) Compact progress.jsonl
python3 scripts/33-compact-progress-log.py
```
Detalhes em [`infra/DEPLOY-CHECKLIST.md`](infra/DEPLOY-CHECKLIST.md).
## Frontend já operacional
13 rotas + Cmd+K + chat agente:
- `/` lista de docs com summaries + filtros
- `/d/<doc>/v2` chunks rebuilt rico
- `/d/<doc>/v2/<page>` single page side-by-side
- `/search?q=…` URL-shareable hybrid search
- `/timeline` cronologia decade-grouped
- `/graph` force-directed entity network
- `/e/<class>` entity list por classe
- `/e/<class>/<id>` entity detail + co-mentions live
- `/admin/stats` corpus analytics
- `/admin/batch` rebuild monitor
- `/admin/indexer` retrieval health
- Chat: 12 tools (hybrid_search / read_chunk / entity_neighbors / etc)
## Warnings conhecidos
Os 23 docs com YAML warnings (todos `body OK` graças ao parser resiliente):
- doc-38-143685, doc-59-64634, doc-65-hs1-101634279
- doc-65-hs1-834228961 sections 1, 2, 4, 5, 6, 7, 8, 10
- doc-65-hs1-834228961 serials 130, 403, 438, 449
- doc-65-hs1-834228961 sub-a
- fbi-photo-b4, b5, b8, b11, b17, b23
- nasa-uap-d7-skylab
Causa: Sonnet 4.6 escreveu aspas duplas mal-escapadas em campos `ocr_source_lines` ou outros. Fix preventivo já aplicado no `page-rebuilder.md` para próximos rebuilds. Os atuais funcionam 100% para retrieval (texto OK), só perdem alguns metadados estruturados em ~140 chunks.

138
README.md Normal file
View file

@ -0,0 +1,138 @@
# The Disclosure Bureau
> Investigative wiki + agentic chat sobre o corpus declassificado do US Department of War em `war.gov/ufo` (116 PDFs, 3.435 páginas, 34k+ entidades, 28 vídeos UAP).
**Live:** [disclosure.top](https://disclosure.top)
## O que é
Pipeline de IA que transforma documentos UAP/UFO declassificados em uma **wiki investigativa navegável** + **chat agêntico com retrieval semântico bilíngue (EN + PT-BR)** e **citações com bbox crop no PDF original**.
A premissa metodológica é o padrão **Karpathy LLM Wiki**: ler tudo, compilar conhecimento em markdown cross-referenciado, navegar via wiki-links — não por busca vetorial. Em cima dessa wiki rodamos uma camada de **hybrid retrieval** (BM25 + BGE-M3 dense + cross-encoder rerank) para perguntas livres no chat.
A camada investigativa segue protocolo **Investigation Bureau** (Holmes/Poirot/Dupin/Locard + Schneier/Tetlock/Taleb): chain-of-custody, hypothesis tournament, residual uncertainty.
## Arquitetura
```
PDFs (raw/)
↓ pdftoppm 72 DPI + pdftotext
processing/ (png + ocr)
↓ Sonnet 4.6 subagents (page-rebuilder, image-analyst, table-stitcher)
raw/<doc>--subagent/ (chunks bilíngues + bbox + anomaly flags)
↓ scripts/30 (BGE-M3 embed) + 31 (entity_mentions)
Postgres + pgvector + tsvector
↓ hybrid_search RPC + reranker
chat agente (OpenRouter) cita [[doc/p007#c0042]] → frontend renderiza crop bbox
```
## Stack
- **Embedding**: BGE-M3 self-hosted (1024-dim, multilíngue, $0)
- **Reranker**: BGE-Reranker-v2-M3 self-hosted ($0)
- **Vetor + texto**: Postgres 15 + pgvector + tsvector bilíngue (`pt_unaccent`, `en_unaccent`)
- **LLM (chat)**: OpenRouter — DeepSeek v4 free como default
- **Frontend**: Next.js 15 + React 19 + Tailwind + assistant-ui (Pattern C streaming)
- **Auth + persistência**: Supabase self-hosted (GoTrue, PostgREST, Storage, Imgproxy)
- **Reverse proxy**: Traefik + Let's Encrypt
- **Imagens**: sharp via `/api/crop` (bbox on-demand, cached 1ano)
## Layout
```
/Users/guto/ufo/
├── CLAUDE.md # contrato vinculante (24 tipos de markdown)
├── CLAUDE-schema-full.md # schema detalhado
├── README.md # este arquivo
├── raw/ # 116 PDFs imutáveis + chunks v0.2.0 derivados
│ ├── <pdfs>
│ ├── <doc-id>--subagent/ # chunks rebuilt (chunks/c*.md + _index.json + document.md)
│ └── _batch-rebuild/ # logs do orchestrator
├── processing/ # intermediários (PNG, OCR, vision JSON)
├── wiki/ # markdown gerado (documents/, pages/, entities/, tables/, images/)
├── case/ # artefatos Investigation Bureau (case-report, hypotheses, gaps)
├── scripts/ # 33 scripts numerados (Phase 0 → manutenção)
├── infra/ # docker-compose, embed-service, migrations, deploy
└── web/ # Next.js frontend
```
## Quick start
```bash
# 1. Converter PDFs em PNG + OCR (uma vez)
./scripts/01-convert-pdfs.sh
# 2. Rebuild chunks bilíngues (Sonnet 4.6 via Claude Code subagents)
python3 scripts/28-batch-rebuild-all.py --workers 2
# 3. (após batch) Indexar em Postgres + embeddings
python3 scripts/30-index-chunks-to-db.py --skip-existing
# 4. (opcional) Materializar entity_mentions p/ grafo
python3 scripts/31-populate-entity-mentions.py
# 5. Deploy
cd infra/disclosure-stack && ./scripts/deploy.sh
```
Detalhes completos em [`infra/DEPLOY-CHECKLIST.md`](infra/DEPLOY-CHECKLIST.md).
## Features do frontend
| URL | Função |
|---|---|
| `/` | Lista de documentos com resumo de 3 linhas, filtros (collection, classification, sort), busca |
| `/d/<doc>` | Visão legado (page grid + frontmatter) |
| `/d/<doc>/v2` | Render rico de chunks com lang toggle (PT/EN/both), paged vs flow |
| `/d/<doc>/v2/<page>` | Single page V2 com PNG side-by-side |
| `/d/<doc>/full` | Texto consolidado bilíngue |
| `/e/<class>` | Lista paginada de entidades por classe (people, locations, ...) |
| `/e/<class>/<id>` | Detalhe da entidade + co-mentions + chunks live |
| `/search?q=...` | Hybrid search URL-shareable |
| `/timeline` | Cronologia de eventos por década |
| `/graph` | Grafo força-direcionado de co-menções (Obsidian-style) |
| `/admin/stats` | Analytics do corpus (FS + DB) |
| `/admin/batch` | Monitor de progresso do rebuild |
| `/admin/indexer` | Estado da camada de retrieval |
**Atalhos globais:**
- `⌘K` / `Ctrl+K` em qualquer página → command palette com hybrid_search
- Toggle 🌐 EN ↔ PT-BR fixo bottom-left (cookie 1ano)
- Chat 💬 botão flutuante bottom-right com 12 ferramentas
## Os 12 tools do agente
🔍 Retrieval: `hybrid_search`, `read_chunk`, `get_page_chunks`, `list_anomalies`
🔗 Grafo: `entity_neighbors`, `entity_path`, `co_mention_chunks`
📄 Wiki: `read_document`, `read_page`, `read_entity`, `search_corpus`
🧭 UI: `navigate_to`
Citações tipo `[[doc-id/p007#c0042]]` viram cards interativos com crop bbox + texto bilíngue + link.
## Custos
| Item | Custo |
|---|---|
| Rebuild chunks (Sonnet 4.6 via Claude Code Max 20x) | ~$200 one-shot p/ 116 docs |
| Embedding BGE-M3 self-host | $0/mês |
| Reranker BGE-Reranker-v2-M3 self-host | $0/mês |
| Postgres + pgvector | já incluso no VPS |
| Chat LLM (DeepSeek free via OpenRouter) | $0/req |
| VPS (16GB / 4 CPU) | ~€10/mês |
## Documentação
- [`CLAUDE.md`](CLAUDE.md) — contrato vinculante para agentes (schema v0.2.0)
- [`CLAUDE-schema-full.md`](CLAUDE-schema-full.md) — schema dos 24 tipos
- [`infra/RETRIEVAL.md`](infra/RETRIEVAL.md) — arquitetura da camada de retrieval
- [`infra/DEPLOY-CHECKLIST.md`](infra/DEPLOY-CHECKLIST.md) — runbook end-to-end
- [`infra/embed-service/README.md`](infra/embed-service/README.md) — microsserviço BGE-M3
## Licença + procedência
- PDFs declassificados: domínio público (US Department of War / FBI / DOS / NASA)
- Código deste projeto: MIT
- Modelos: BGE-M3 (MIT), DeepSeek v4 (proprietary via OpenRouter free tier)
- Branding: The Disclosure Bureau / disclosure.top — pessoal
> Wiki investigativa, não advocacy. Toda claim tem chain-of-custody até a página + bbox do PDF original.

207
infra/DEPLOY-CHECKLIST.md Normal file
View file

@ -0,0 +1,207 @@
# Deploy Checklist — Retrieval Layer Activation
Runbook end-to-end para ativar pgvector + BGE-M3 + reranker no Disclosure Bureau VPS.
> Assume: VPS já tem disclosure-stack (Supabase + Next.js + Meilisearch) rodando, e o batch `scripts/28-batch-rebuild-all.py` já produziu chunks em `raw/<doc>--subagent/` (parcial ou completo).
## 0. Pré-condições
```bash
# checa que chunks existem
ls -d /Users/guto/ufo/raw/*--subagent | wc -l # esperado ≥ 1
# checa que .env tem POSTGRES_PASSWORD + DATABASE_URL + EMBED_SERVICE_URL
grep -E "POSTGRES_PASSWORD|DATABASE_URL|EMBED_SERVICE_URL" infra/disclosure-stack/.env
```
Se o `.env` ainda não tem as novas linhas, copia do `.env.example`:
```
DATABASE_URL=postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres
EMBED_SERVICE_URL=http://embed:8000
```
## 1. Aplicar migração 02 (pgvector + chunks schema)
Sobe o stack — bootstrap.sh já aplica as migrações automaticamente (vê linhas 86-92):
```bash
cd infra/disclosure-stack
./scripts/deploy.sh # rsync + docker compose up; aplica 00-init, 01-chat, 02-chunks
```
Se quiser aplicar só a 02 manualmente:
```bash
./scripts/ssh.sh
cd /data/disclosure
docker exec -i disclosure-db psql -U postgres < migrations/02-chunks-retrieval.sql
```
**Verificar:**
```bash
docker exec -i disclosure-db psql -U postgres -c "\dx" | grep vector
# extensão `vector` instalada ✓
docker exec -i disclosure-db psql -U postgres -c "\dt public.*"
# documents, chunks, entities, entity_mentions ✓
docker exec -i disclosure-db psql -U postgres -c "\df public.hybrid_search_chunks"
# função RPC ✓
```
## 2. Subir o embed-service (BGE-M3 + reranker)
Já está no `docker-compose.yml`. Primeiro build leva 5-10 min (baixa torch CPU + FlagEmbedding). Primeira request leva mais 5-8 s pra carregar modelos.
```bash
./scripts/ssh.sh
cd /data/disclosure
docker compose build embed
docker compose up -d embed
docker compose logs -f embed
```
**Verificar (de dentro do VPS via internal network):**
```bash
docker exec disclosure-embed curl -s http://localhost:8000/health
# {"status":"ok","embed_loaded":false,"rerank_loaded":false}
docker exec disclosure-embed curl -s -X POST http://localhost:8000/embed \
-H 'content-type: application/json' \
-d '{"texts":["UAP sobre Kansas em 1950"]}'
# primeira call: ~5s (model load). retorna {model,dim:1024,embeddings:[[...]]}
```
## 3. Indexar chunks → Postgres
```bash
./scripts/ssh.sh
cd /data/disclosure
# instalar deps Python (se ainda não)
pip3 install psycopg[binary] pyyaml requests
# pegar a senha do postgres pra montar DATABASE_URL local
source /data/disclosure/.env
# rodar indexer dentro de um container que tem rede internal
docker run --rm \
--network disclosure-internal \
-v /data/ufo:/data/ufo:ro \
-e DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres" \
-e EMBED_SERVICE_URL=http://embed:8000 \
python:3.11-slim \
bash -c "pip install -q psycopg[binary] pyyaml requests && \
python3 /data/ufo/scripts/30-index-chunks-to-db.py --skip-existing"
```
**Verificar:**
```bash
docker exec -i disclosure-db psql -U postgres -c \
"SELECT COUNT(*) FROM public.chunks WHERE embedding IS NOT NULL;"
# esperado: total = soma de chunks em raw/*--subagent/chunks/
```
## 4. Materializar `entity_mentions`
```bash
docker run --rm \
--network disclosure-internal \
-v /data/ufo:/data/ufo:ro \
-e DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres" \
python:3.11-slim \
bash -c "pip install -q psycopg[binary] pyyaml && \
python3 /data/ufo/scripts/31-populate-entity-mentions.py"
# tempo: ~30min para 34k entidades
```
**Verificar:**
```bash
docker exec -i disclosure-db psql -U postgres -c \
"SELECT COUNT(*) FROM public.entity_mentions;"
# esperado: dezenas de milhares
```
## 5. Sync `mentioned_in[]` → markdown (opcional, fecha loop)
Esta é a única etapa que ESCREVE em `wiki/`. Use `--dry-run` primeiro.
```bash
docker run --rm \
--network disclosure-internal \
-v /data/ufo:/data/ufo \
-e DATABASE_URL="postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres" \
python:3.11-slim \
bash -c "pip install -q psycopg[binary] pyyaml && \
python3 /data/ufo/scripts/32-sync-mentioned-in-yaml.py"
```
## 6. Configurar Next.js para usar a DB
No `disclosure-stack/.env`:
```
DATABASE_URL=postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres
EMBED_SERVICE_URL=http://embed:8000
```
Reiniciar o container web:
```bash
docker compose restart web
```
## 7. Smoke test end-to-end
Acessar pelas URLs do Traefik:
| URL | Esperado |
|---|---|
| `https://app.disclosure.top/admin/stats` | dashboard com counts da DB |
| `https://app.disclosure.top/admin/indexer` | "✓ retrieval operacional" |
| `https://app.disclosure.top/search?q=Olathe` | hits semânticos |
| `https://app.disclosure.top/graph` | force graph com co-mentions |
| `https://app.disclosure.top/timeline` | eventos por década |
| `https://app.disclosure.top/d/doc-342-.../v2` | chunks renderizados inline |
| `https://app.disclosure.top/d/doc-342-.../v2/p001` | single page view |
| `Cmd+K` em qualquer página | command palette com hybrid_search |
| chat: "qual foi a forma dos objetos em Olathe?" | resposta cita `[[doc-id/p001#c0008]]` que vira card com crop |
## 8. Manutenção periódica
```bash
# compactar progress.jsonl (após várias retries do batch)
python3 scripts/33-compact-progress-log.py
# reindexar docs novos que apareceram no disco mas ainda não estão na DB
python3 scripts/30-index-chunks-to-db.py --skip-existing
# regenerar entity_mentions quando 31 novos chunks são adicionados
python3 scripts/31-populate-entity-mentions.py --reset
```
## 9. Troubleshooting
| Sintoma | Causa | Fix |
|---|---|---|
| `/api/search/hybrid` 503 | embed-service down | `docker compose logs embed` |
| `/api/admin/indexer` mostra "db_error" | DATABASE_URL errada ou DB parado | `docker compose logs db` |
| chat retorna `"retrieval_unavailable"` | DB ou embed-service inacessíveis | restart no compose |
| `/graph` vazio | entity_mentions não populado | rodar `31-populate-entity-mentions.py` |
| `/timeline` vazio | events sem `date_start` no frontmatter | revisar wiki/entities/events/ |
| bbox crop 500 | PNG faltando em processing/ | rodar `01-convert-pdfs.sh` |
| Anthropic 429 no batch | quota Max 20x (5h window) | esperar reset; orchestrator agora aborta cedo |
## 10. Custos recorrentes
- **Embedding (BGE-M3 self-host)**: $0
- **Reranker (BGE-Reranker-v2-M3 self-host)**: $0
- **Postgres + pgvector**: já incluso no plano VPS
- **LLM (chat agent)**: OpenRouter — deepseek-v4-flash:free ($0) ou paid model conforme tier
- **Re-rebuild de docs**: só quando schema mudar (Anthropic API ou Claude Code Max quota)

51
infra/README.md Normal file
View file

@ -0,0 +1,51 @@
# Infrastructure — Disclosure Bureau
Self-hosted stack on a single VPS (16 GB / 4 CPU / 200 GB NVMe) managed via **Coolify**.
```
Internet (443/80)
│
┌─────────▼─────────┐
│ Caddy (Coolify) │ ← auto-TLS Let's Encrypt
└────┬──────────────┘
│
┌─────────────┼──────────────────────┬──────────────────┐
▼ ▼ ▼ ▼
┌─────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Next.js │ │ Supabase │ │ Supabase │ │ shared │
│ web │ │ disclosure│ │ project-B │ │ services │
│ :3000 │ │ stack │ │ stack │ │ Meili··· │
└─────────┘ │ ┌─────┐ │ │ ┌─────┐ │ │ Imgproxy │
│ │PG/GT│ │ │ │PG/GT│ │ │ Dragonfly│
│ └─────┘ │ │ └─────┘ │ └──────────┘
└──────────┘ └──────────┘
disclosure.top projeto-b.com
```
## Components
| Layer | Service | Notes |
|---|---|---|
| **Orchestration** | [Coolify](https://coolify.io) v4 | Self-hosted PaaS — manages all containers, TLS, backups |
| **Database + Auth + Storage** | Supabase self-hosted (one per project) | Each project gets own Postgres + GoTrue + Storage |
| **Frontend** | Next.js 15 (this repo's `/web`) | Deployed via Coolify Git integration |
| **Search** | Meilisearch (shared) | Full-text search across pages + entities |
| **Cache + Queue** | Dragonfly (shared) | Redis-compatible, multi-threaded |
| **Images** | Imgproxy (shared) | On-the-fly resize / WebP conversion |
| **Backups** | restic + Backblaze B2 | Nightly Postgres + Storage dumps |
## Quick path
1. **[`coolify/INSTALL.md`](coolify/INSTALL.md)** — install Coolify on the fresh VPS (~10 min)
2. **[`coolify/SUPABASE.md`](coolify/SUPABASE.md)** — create the `disclosure` Supabase project (~5 min)
3. Run [`supabase/migrations/0001_chat_schema.sql`](supabase/migrations/0001_chat_schema.sql) via Supabase Studio SQL editor
4. **[`coolify/NEXTJS.md`](coolify/NEXTJS.md)** — deploy the `/web` app pointing at the Supabase URL
5. **[`coolify/SHARED.md`](coolify/SHARED.md)** — bring up Meilisearch, Dragonfly, Imgproxy
## Adding more projects later
For each new project, repeat step 2 (new Supabase project in Coolify UI) and step 4 (new Next.js app). They get their own subdomain, own auth, own data. Total isolation.
## Local development
For dev on macOS/Linux without the VPS, see [`../web/README.md`](../web/README.md) — uses the Supabase CLI to spin up a local stack on `localhost:54321`.

80
infra/RETRIEVAL.md Normal file
View file

@ -0,0 +1,80 @@
# Retrieval Pipeline — disclosure.top chunks layer
Hybrid retrieval over the agentic chunks (raw/<doc-id>--subagent/) using:
- **BGE-M3** dense embeddings (1024-dim, multilingual, self-hosted, gratis)
- **pgvector HNSW** index (Postgres 15.8.1 in disclosure-stack, supabase image ships pgvector)
- **Postgres tsvector** BM25 (`pt_unaccent` + `en_unaccent` configs)
- **BGE-Reranker-v2-M3** cross-encoder rerank (self-hosted)
- **RRF fusion** of BM25 + dense → reranker → final top-k
Cost: ~$0/month after initial $0 setup. LLM stays on OpenRouter (deepseek-v4-flash:free or paid model of choice).
## Components
| Path | Purpose |
|---|---|
| `infra/embed-service/` | Python FastAPI on CPU — BGE-M3 + reranker |
| `infra/supabase/migrations/0002_chunks_retrieval.sql` | pgvector + tsvector + chunks/documents/entities tables + `public.hybrid_search_chunks` RPC |
| `scripts/30-index-chunks-to-db.py` | Reads `raw/<doc>--subagent/_index.json + chunks/c*.md`, embeds via embed-service, UPSERTs to Postgres |
| `web/lib/retrieval/` | TS client (db.ts, embed.ts, hybrid.ts) |
| `web/lib/chat/tools.ts` | `hybrid_search`, `read_chunk`, `get_page_chunks`, `list_anomalies` tools |
| `web/app/api/crop/` | On-demand bbox crop service (sharp) — used by chunk views + chat citations |
| `web/app/d/[docId]/v2/` | Rich render using chunks (inline images + tables + cite anchors) |
## End-to-end flow
```
PDFs (raw/*.pdf)
│ pdftoppm 72 DPI + pdftotext
▼
processing/png/<doc>/p-NNN.png + processing/ocr/<doc>/p-NNN.txt
│ Sonnet 4.6 via Claude Code subagents (scripts/28-batch-rebuild-all.py)
▼
raw/<doc>--subagent/
├── _index.json
├── chunks/c0001.md ... c<NNNN>.md (bilingual EN+PT, bbox, anomaly flags)
├── images/IMG-c<NNNN>.png (crops with image-analyst description)
└── tables/TBL-NNN.csv (stitched multi-page tables)
│ scripts/30-index-chunks-to-db.py + embed-service
▼
Postgres
├── public.documents (1 row per doc)
├── public.chunks (1 row per chunk; embedding vector(1024))
├── public.entities (1 row per canonical entity)
└── public.entity_mentions (chunk ↔ entity link, materialized by lint)
│ web/lib/retrieval/hybrid.ts → public.hybrid_search_chunks RPC + /rerank
▼
Chat agent (OpenRouter) calls `hybrid_search` tool → cites [[doc/p007#c0042]]
│ /api/crop returns the bbox region
▼
Frontend renders inline crop + bilingual text + link to original page
```
## Deploy
```bash
# 1. Build & ship embed-service image to VPS
cd infra/disclosure-stack
./scripts/bootstrap.sh # picks up new embed service + 0002 migration
# 2. Index after batch rebuild completes (or incrementally)
ssh vps "cd /data/disclosure && docker exec -i disclosure-db psql -U postgres \
< migrations/02-chunks-retrieval.sql"
# 3. Run indexer (on VPS, after embed-service is healthy)
cd /Users/guto/ufo
DATABASE_URL='postgres://...' EMBED_SERVICE_URL='http://localhost:8000' \
python3 scripts/30-index-chunks-to-db.py --skip-existing
```
## Performance budget (CPU-only VPS, 16GB RAM)
- BGE-M3 cold load: ~5-8 s; warm embed (single text): ~150-300 ms
- Embed batch of 16 chunks: ~800-1500 ms
- Indexing 100 chunks/doc × 115 docs = ~11,500 chunks → ~15-25 min total
- pgvector HNSW recall@100 from 150k chunks: <30 ms
- BGE-Reranker on 100 candidates: 5-8 s
- End-to-end chat query (recall + rerank + LLM): ~6-12 s
Tune later: switch reranker to batch of 50 if latency feels slow; use BGE-M3 fp16 if GPU available.

48
infra/coolify/INSTALL.md Normal file
View file

@ -0,0 +1,48 @@
# Coolify — VPS install
Coolify is the self-hosted PaaS that runs everything else. One curl command installs it.
## Pre-reqs on the VPS
- Ubuntu 22.04+ or Debian 12+
- root or sudo
- Open ports: **22, 80, 443, 8000** (Coolify dashboard during install; can be locked behind subdomain later)
- A DNS A record like `coolify.disclosure.top` pointing at the VPS IP (any subdomain works)
## Install
SSH into the VPS, then:
```bash
curl -fsSL https://cdn.coollabs.io/coolify/install.sh | sudo bash
```
This sets up:
- Docker + Docker Compose
- The Coolify control plane in `/data/coolify/`
- A built-in Traefik that handles TLS
Boot takes ~5 min. When done you'll see a URL like `http://<your-ip>:8000` — open it, create the admin user.
## Post-install — hardening
1. **Point a subdomain at the panel** in Coolify Settings → "Instance Domain": `coolify.disclosure.top`. Save. It auto-issues TLS via Let's Encrypt.
2. **Disable port 8000 publicly** — once subdomain works, edit firewall:
```bash
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw deny 8000/tcp
ufw enable
```
3. **Add S3 backup destination** in Coolify Settings → Backups (Backblaze B2 recommended — $0.005/GB/mo).
4. **Optional: Cloudflare proxy** — pointing the wildcard `*.disclosure.top` through Cloudflare as proxy adds DDoS mitigation + edge caching. Set DNS-only (grey cloud) for first TLS issuance, then re-enable proxy (orange) after the cert is issued.
## What's next
Once you can log in to the Coolify dashboard at `https://coolify.disclosure.top`:
→ [`SUPABASE.md`](SUPABASE.md) — create the first Supabase project

84
infra/coolify/NEXTJS.md Normal file
View file

@ -0,0 +1,84 @@
# Coolify — Deploy the Next.js web app
## Source
Coolify pulls from Git. Set up the repo on GitHub/GitLab/Gitea (private OK; Coolify supports deploy keys).
This repo's `/web` directory is the app root.
## Create the application
1. Coolify → **+ New Resource** → **Application** → **Public Git** (or Private with deploy key).
2. Repository: `git@github.com:youruser/ufo.git` (whatever you push to)
3. Branch: `main`
4. **Base directory**: `/web` ← important, the Next app isn't at the repo root
5. **Build pack**: `Nixpacks` (or `Dockerfile` if you commit one — Nixpacks is simpler for Next 15)
6. **Port**: `3000`
7. **Domain**: `disclosure.top` and `www.disclosure.top`
## Environment variables
Set in the application's **Environment Variables** tab:
```env
# Supabase (from Coolify's Supabase project page)
NEXT_PUBLIC_SUPABASE_URL=https://db.disclosure.top
NEXT_PUBLIC_SUPABASE_ANON_KEY=<paste-anon-key>
SUPABASE_SERVICE_ROLE_KEY=<paste-service-role-key>
# Anthropic (the Claude API for the chat agent)
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_CHAT_MODEL=claude-haiku-4-5
# Paths inside the container (we mount the wiki/ + processing/ volumes)
UFO_ROOT=/data/ufo
# Public URL (for magic-link redirects)
NEXT_PUBLIC_SITE_URL=https://disclosure.top
```
## Volume mounts — the wiki data
The Next.js app reads markdown directly from `/Users/guto/ufo/{wiki,processing,raw}`. On the VPS, mount the data dir into the container at `/data/ufo`.
In Coolify app → **Storages**:
| Source (host) | Target (container) | Mode |
|---|---|---|
| `/data/ufo/wiki` | `/data/ufo/wiki` | read-only |
| `/data/ufo/processing` | `/data/ufo/processing` | read-only |
| `/data/ufo/raw` | `/data/ufo/raw` | read-only |
**How to populate these on the VPS first time:**
```bash
# On your laptop, after pipeline finishes:
rsync -avz --progress \
/Users/guto/ufo/wiki/ \
/Users/guto/ufo/processing/ \
/Users/guto/ufo/raw/ \
root@<vps-ip>:/data/ufo/
```
Subsequent updates: just re-rsync. The container reads live (with Next.js page revalidation set appropriately).
## Deploy
Coolify pulls the repo, runs `npm install && npm run build`, then `npm run start` on port 3000.
First deploy ~5 min. Subsequent (cached layer) ~1 min.
## Verify
```bash
curl https://disclosure.top/api/documents | head -c 300
# → {"documents":[{"doc_id":"doc-059uap00011",...
```
## Continuous deploy
Coolify can listen to a Git webhook so every push to `main` triggers a rebuild. Set in Application → Webhooks.
## Next
→ [`SHARED.md`](SHARED.md) — Meilisearch + Dragonfly + Imgproxy

113
infra/coolify/SHARED.md Normal file
View file

@ -0,0 +1,113 @@
# Coolify — Shared services (Meilisearch + Dragonfly + Imgproxy)
These are stateless or namespaceable — one instance can serve all your projects. Saves RAM.
## Meilisearch
Coolify → + New Service → Database → **Meilisearch**.
| Field | Value |
|---|---|
| Project name | `meilisearch-shared` |
| Domain | `search.disclosure.top` |
| Master key | (generate; copy for `MEILI_MASTER_KEY` env in web app) |
| Resource limits | 1 CPU, 1 GB RAM |
Each project uses different index names: `disclosure_pages`, `disclosure_entities`, `projeto_b_xxx` — no cross-talk.
After deploy, in the web app env vars:
```env
MEILISEARCH_URL=https://search.disclosure.top
MEILISEARCH_API_KEY=<master-key>
```
The web app will create + populate indexes on first deploy (see `web/scripts/seed-meili.ts`).
## Dragonfly
Redis-compatible, drop-in, 25× faster than Redis. Coolify has no template, use a custom Docker Compose service:
```yaml
# In Coolify → + New Resource → Service → Custom Docker Compose
services:
dragonfly:
image: docker.dragonflydb.io/dragonflydb/dragonfly:latest
restart: unless-stopped
ulimits:
memlock: -1
ports:
- "6379:6379"
volumes:
- dragonfly-data:/data
command: ["--logtostderr", "--cache_mode=true", "--maxmemory=512mb"]
mem_limit: 600m
volumes:
dragonfly-data:
```
Web app env:
```env
REDIS_URL=redis://dragonfly:6379
```
(Coolify networks the services internally; you reach `dragonfly:6379` from inside the network.)
## Imgproxy
Stateless image resizer:
```yaml
services:
imgproxy:
image: ghcr.io/imgproxy/imgproxy:latest
restart: unless-stopped
environment:
IMGPROXY_KEY: ${IMGPROXY_KEY}
IMGPROXY_SALT: ${IMGPROXY_SALT}
IMGPROXY_USE_ETAG: "true"
IMGPROXY_TTL: "31536000"
IMGPROXY_MAX_SRC_RESOLUTION: "50"
IMGPROXY_ENABLE_WEBP_DETECTION: "true"
IMGPROXY_LOCAL_FILESYSTEM_ROOT: "/data"
volumes:
- /data/ufo/processing:/data:ro
ports:
- "8080:8080"
mem_limit: 256m
```
Generate `IMGPROXY_KEY` and `IMGPROXY_SALT`:
```bash
openssl rand -hex 64 # → key
openssl rand -hex 64 # → salt
```
Add subdomain `img.disclosure.top` → :8080.
Web app uses signed URLs like:
```
https://img.disclosure.top/<signature>/rs:fit:800:0/plain/local:///png/doc-x/p-001.png
```
The `web/lib/imgproxy.ts` helper generates these signatures.
## Backups (restic + Backblaze B2)
Optional but strongly recommended. Coolify has a built-in backup feature per service:
1. Each Supabase project's stack → **Backups** → set schedule (e.g., `0 3 * * *` daily 3am).
2. Destination: configure B2 bucket in Coolify Settings → Backups (one-time setup).
Restic runs encrypted, deduplicated. ~$0.005/GB/mo on B2.
## Done
Stack complete:
- `disclosure.top` (Next.js)
- `db.disclosure.top` (Supabase Kong)
- `studio.disclosure.top` (Supabase Studio)
- `search.disclosure.top` (Meilisearch)
- `img.disclosure.top` (Imgproxy)
- `coolify.disclosure.top` (Coolify panel)
Total RAM in production: ~5-6 GB. Plenty of room for 2-3 more projects.

59
infra/coolify/SUPABASE.md Normal file
View file

@ -0,0 +1,59 @@
# Coolify — Add a Supabase project
Each Supabase "project" in Coolify is a Docker Compose stack containing Postgres + GoTrue + PostgREST + Storage + Realtime + Studio + Kong. They are fully isolated.
## Create the `disclosure` project
1. Coolify dashboard → **+ New Resource** → **Service** → search **"Supabase"** → Deploy.
2. Configure:
- **Project name**: `disclosure`
- **Server**: your VPS (it's the default if you only have one)
- **Domain**: `db.disclosure.top` (the Kong API gateway will live here)
- **Studio domain**: `studio.disclosure.top` (the admin UI)
3. Coolify generates the secrets automatically. Copy these for later:
- `POSTGRES_PASSWORD`
- `JWT_SECRET`
- `ANON_KEY`
- `SERVICE_ROLE_KEY`
- `DASHBOARD_USERNAME` / `DASHBOARD_PASSWORD` (for Studio basic auth)
4. Click **Deploy**. Coolify pulls all images and starts the stack (~3 min).
5. When green: open `https://studio.disclosure.top`, log in with the dashboard creds.
## Apply the chat schema
In Studio → SQL Editor, paste the contents of [`../supabase/migrations/0001_chat_schema.sql`](../supabase/migrations/0001_chat_schema.sql) and run. This creates `profiles`, `chat_sessions`, `messages` with RLS enabled.
Then run [`../supabase/seed.sql`](../supabase/seed.sql) to seed your admin user (edit the email in the file first).
## Verify
```bash
# From your laptop, with the ANON_KEY copied from Coolify:
curl -H "apikey: <ANON_KEY>" https://db.disclosure.top/rest/v1/profiles
# → [] (empty array, but valid response = it's working)
```
## Adding another project later
Same flow — Coolify → + New Service → Supabase. Pick a new project name (`projeto-b`), new domains (`db.projeto-b.com`, `studio.projeto-b.com`). Coolify isolates everything: own Postgres, own GoTrue, own secrets. Different anon/service keys.
## Resource tuning
In each Supabase project's stack settings, increase Postgres memory if needed:
```yaml
# In the Postgres service env vars:
POSTGRES_SHARED_BUFFERS: 256MB
POSTGRES_WORK_MEM: 16MB
POSTGRES_MAINTENANCE_WORK_MEM: 64MB
```
Defaults are fine for <10k users. For chat-heavy workloads bump shared_buffers.
## Next
→ [`NEXTJS.md`](NEXTJS.md) — deploy the web app

129
infra/disclosure-stack/.env.example Normal file
View file

@ -0,0 +1,129 @@
# =============================================================================
# DISCLOSURE BUREAU — DEPLOYMENT CONFIG
# =============================================================================
# Copy this file to `.env`, fill in the values, never commit `.env`.
# When migrating to a different VPS, change ONLY the "VPS CONNECTION" block.
# Everything else stays portable.
#
# To regenerate the secrets block: `./scripts/gen-secrets.sh`
# =============================================================================
# ─── VPS CONNECTION ─────────────────────────────────────────────────────────
# The only block you change when moving to another VPS.
# scripts/ssh.sh, scripts/deploy.sh, scripts/logs.sh all read these.
VPS_HOST= # e.g., 187.77.40.19 OR disclosure.example.com
VPS_USER=root
VPS_PORT=22
VPS_AUTH=password # 'password' or 'key'
VPS_PASSWORD= # only if VPS_AUTH=password (kept ONLY in this gitignored .env)
VPS_SSH_KEY=~/.ssh/id_ed25519 # only if VPS_AUTH=key
VPS_DEPLOY_ROOT=/data/disclosure # where this stack lives on the VPS
# ─── PROJECT IDENTITY ───────────────────────────────────────────────────────
PROJECT_NAME=disclosure # docker-compose project name; prefixes all containers
STACK_PREFIX=disclosure- # matches the existing pattern on the VPS (unimed-, irmed-, ...)
# ─── DOMAINS ────────────────────────────────────────────────────────────────
# Set DNS A records pointing all of these to VPS_HOST before deploying.
DOMAIN_MAIN=disclosure.top # Next.js public app
DOMAIN_API=api.disclosure.top # Kong (Supabase API gateway)
DOMAIN_STUDIO=studio.disclosure.top # Supabase Studio (admin UI — protect with basic-auth)
DOMAIN_SEARCH=search.disclosure.top # Meilisearch
DOMAIN_IMG=img.disclosure.top # Imgproxy
ACME_EMAIL= # for Let's Encrypt cert notifications
# ─── HOST PORTS (must NOT collide with other projects on the VPS) ───────────
# Internal ports in containers stay default; these are the host-side mappings.
# Each stack on the VPS uses its own range. Pick free ones via `ss -tlnp`.
PORT_KONG_HTTP=18001 # Supabase API gateway HTTP
PORT_KONG_HTTPS=18444 # Supabase API gateway HTTPS
PORT_STUDIO=18002 # Studio UI
PORT_NEXT=18003 # Next.js app
PORT_MEILI=18004 # Meilisearch
PORT_IMGPROXY=18005 # Imgproxy
# Postgres NEVER exposed to host; reach it via internal network only.
# ─── SECRETS (auto-generated, regenerate per VPS) ───────────────────────────
# Generate fresh values with: ./scripts/gen-secrets.sh
POSTGRES_PASSWORD=
JWT_SECRET= # 64+ chars, used to sign ANON_KEY and SERVICE_ROLE_KEY
ANON_KEY= # JWT signed with role=anon (generated by gen-secrets.sh)
SERVICE_ROLE_KEY= # JWT signed with role=service_role
DASHBOARD_USERNAME=admin # Studio basic-auth
DASHBOARD_PASSWORD=
SECRET_KEY_BASE= # used by realtime + auth
VAULT_ENC_KEY= # 32-char hex for Supabase Vault
MEILI_MASTER_KEY=
IMGPROXY_KEY=
IMGPROXY_SALT=
# ─── CHAT AGENT — providers ─────────────────────────────────────────────────
# Primary: Claude Code SDK via OAuth (NEVER use ANTHROPIC_API_KEY in this project).
# The container ships the `claude` CLI and authenticates with CLAUDE_CODE_OAUTH_TOKEN.
# Get a long-lived OAuth token by running `claude setup-token` locally.
CLAUDE_CODE_OAUTH_TOKEN=
CLAUDE_CODE_MODEL=haiku # claude-haiku-4-5 alias; or 'sonnet' for harder cases
# Fallback: OpenRouter — free or paid models when Claude Code is rate-limited.
OPENROUTER_API_KEY=
OPENROUTER_MODEL=deepseek/deepseek-v4-flash:free # primary (free, supports tool calls)
OPENROUTER_FALLBACK_MODEL=nvidia/nemotron-3-super-120b-a12b:free
# Provider selection:
# 'openrouter' → tool calling + AG-UI streaming (Pattern C)
# 'claude-code' → simple Q&A via OAuth subprocess, no tools
# 'auto' → claude-code first, fallback OpenRouter on rate-limit
CHAT_PROVIDER=openrouter
# ─── EMAIL (for Supabase magic-link delivery) ───────────────────────────────
# Pick ONE:
# A) Resend (recommended — free tier 3k/mo)
SMTP_HOST=smtp.resend.com
SMTP_PORT=465
SMTP_USER=resend
SMTP_PASS=
SMTP_FROM=noreply@disclosure.top
SMTP_FROM_NAME="The Disclosure Bureau"
# ─── DATA PATHS (on the VPS) ────────────────────────────────────────────────
# These mount the wiki/, processing/, raw/ trees into the Next.js container.
# Sync via `./scripts/sync-data.sh` (rsync from your laptop).
DATA_WIKI=/data/disclosure/wiki
DATA_PROCESSING=/data/disclosure/processing
DATA_RAW=/data/disclosure/raw
# ─── RETRIEVAL (embed-service + pgvector) ───────────────────────────────────
# DATABASE_URL: Next.js connects to Postgres inside compose (service name `db`).
# For local indexing from your laptop, set this to a tunneled VPS port or local PG.
DATABASE_URL=postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres
EMBED_SERVICE_URL=http://embed:8000
# ─── BACKUPS (optional but recommended) ─────────────────────────────────────
BACKUP_ENABLED=false
BACKUP_DESTINATION= # e.g., b2:my-bucket/disclosure or s3:bucket/path
BACKUP_PASSWORD= # restic repo encryption key
B2_ACCOUNT_ID=
B2_ACCOUNT_KEY=
BACKUP_CRON="0 3 * * *" # daily 3am UTC
# ─── RESOURCE LIMITS (tuned for 16GB/4cpu VPS) ──────────────────────────────
# Conservative — adjust upward when you see fit.
POSTGRES_SHARED_BUFFERS=256MB
POSTGRES_WORK_MEM=8MB
POSTGRES_MAINTENANCE_WORK_MEM=64MB
POSTGRES_MAX_CONNECTIONS=80
MEILI_MAX_INDEXING_MEMORY=512MB
NEXT_NODE_OPTIONS="--max-old-space-size=768"

10
infra/disclosure-stack/.gitignore vendored Normal file
View file

@ -0,0 +1,10 @@
# Real config — contains secrets + VPS connection
.env
# Local runtime state
.local/
*.log
backup-*.tar.gz
# Allow the example through
!.env.example

97
infra/disclosure-stack/README.md Normal file
View file

@ -0,0 +1,97 @@
# disclosure-stack — portable deployment
Single-folder deployment unit. Edit `.env`, run scripts, app deploys to the VPS.
When migrating to another VPS: **change ONLY the VPS_* block in `.env`**, run `./scripts/gen-secrets.sh` (regenerates per-VPS secrets), then `./scripts/deploy.sh`. Done.
## Layout
```
infra/disclosure-stack/
├── .env ← active config (gitignored, secrets in here)
├── .env.example ← template, safe to commit
├── docker-compose.yml ← TODO — supabase + next + meili + imgproxy
└── scripts/
├── _lib.sh ← shared SSH/rsync helpers
├── ssh.sh ← interactive SSH or one-shot remote command
├── status.sh ← VPS + stack health report
├── gen-secrets.sh ← rotate per-VPS secrets (JWT, Postgres, etc.)
├── sync-data.sh ← rsync wiki/processing/raw to VPS
├── deploy.sh ← upload + docker compose up
└── logs.sh ← tail logs of a service
```
## Pre-reqs on your laptop
```bash
brew install hudochenkov/sshpass/sshpass # for password SSH (testing VPS)
```
For real production, generate an SSH key, copy it to the VPS, and switch `VPS_AUTH=key` in `.env`.
## Daily ops
```bash
# Open shell on VPS
./scripts/ssh.sh
# One-shot command
./scripts/ssh.sh "docker ps"
# Full health report
./scripts/status.sh
# Tail Postgres logs
./scripts/logs.sh postgres
# Push fresh wiki data (after running the local pipeline)
./scripts/sync-data.sh
# Deploy stack changes
./scripts/deploy.sh
```
## Migrating to a different VPS
1. **Edit `.env`** — change `VPS_HOST`, `VPS_PASSWORD` (or switch to `VPS_AUTH=key`), `VPS_DEPLOY_ROOT` if needed.
2. **Rotate secrets**:
```bash
./scripts/gen-secrets.sh
```
This regenerates `POSTGRES_PASSWORD`, `JWT_SECRET`, `ANON_KEY`, `SERVICE_ROLE_KEY`, `DASHBOARD_PASSWORD`, etc., and writes them back to `.env`. The old `.env` is backed up.
3. **Sync data**:
```bash
./scripts/sync-data.sh
```
4. **Deploy**:
```bash
./scripts/deploy.sh
```
That's it. The new VPS now hosts the full stack with fresh secrets, isolated from the old one.
## What still needs to be built
The `docker-compose.yml` itself. Will include:
- Supabase Postgres + GoTrue + PostgREST + Storage + Kong + Studio + Realtime
- Next.js (built from this repo's `/web` dir)
- Meilisearch
- Imgproxy
- Caddy (TLS + reverse proxy on subdomains from `.env`)
- restic-cron for backups (if `BACKUP_ENABLED=true`)
I'll generate that next.
## Coexistence with existing VPS projects
On the testing VPS, 8 other Supabase-based stacks are already running (unimed-*, irmed-*, v2irmed-*, top10-*, cf-*, nirvana-*, plegal-*). This stack:
- Uses unique container names (`disclosure-*` prefix)
- Uses unique host ports (`PORT_*` block in `.env`, all 18xxx)
- Mounts its own data volumes under `/data/disclosure/`
- Caddy on this stack only binds to `PORT_KONG_HTTP/HTTPS` and friends — does NOT take 80/443
When you move to the dedicated 4cpu/16GB VPS, you can:
- Keep ports as-is (works)
- OR remap PORT_KONG_HTTP=80, PORT_KONG_HTTPS=443 since nothing else uses them
The stack is **portable in both directions**.

367
infra/disclosure-stack/docker-compose.yml Normal file
View file

@ -0,0 +1,367 @@
# Disclosure Bureau — full deployment stack.
# Routed via the host's existing plegal-traefik (network: traefik-public).
# Internal services share the disclosure-internal network and are NOT exposed
# to the host. Public services (web, kong, studio, search) get Traefik labels.
name: disclosure
networks:
internal:
name: disclosure-internal
driver: bridge
traefik:
name: traefik-public
external: true
volumes:
db-data:
storage-data:
meili-data:
hf-cache:
services:
# ─── Database ─────────────────────────────────────────────────────────────
db:
container_name: disclosure-db
image: supabase/postgres:15.8.1.060
restart: unless-stopped
networks: [internal]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -h localhost"]
interval: 10s
timeout: 5s
retries: 12
environment:
POSTGRES_HOST: /var/run/postgresql
POSTGRES_PORT: 5432
POSTGRES_DB: postgres
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
PGPASSWORD: ${POSTGRES_PASSWORD}
JWT_SECRET: ${JWT_SECRET}
JWT_EXP: 3600
POSTGRES_INITDB_ARGS: "--data-checksums"
command:
- postgres
- -c
- shared_buffers=${POSTGRES_SHARED_BUFFERS:-384MB}
- -c
- work_mem=${POSTGRES_WORK_MEM:-12MB}
- -c
- maintenance_work_mem=${POSTGRES_MAINTENANCE_WORK_MEM:-96MB}
- -c
- max_connections=${POSTGRES_MAX_CONNECTIONS:-80}
volumes:
- db-data:/var/lib/postgresql/data
# ─── Auth (GoTrue) ────────────────────────────────────────────────────────
auth:
container_name: disclosure-auth
image: supabase/gotrue:v2.170.0
restart: unless-stopped
networks: [internal]
depends_on:
db: { condition: service_healthy }
environment:
GOTRUE_API_HOST: 0.0.0.0
GOTRUE_API_PORT: 9999
API_EXTERNAL_URL: https://${DOMAIN_API}
GOTRUE_DB_DRIVER: postgres
GOTRUE_DB_DATABASE_URL: postgres://supabase_auth_admin:${POSTGRES_PASSWORD}@db:5432/postgres?search_path=auth
GOTRUE_SITE_URL: https://${DOMAIN_MAIN}
GOTRUE_URI_ALLOW_LIST: https://${DOMAIN_MAIN},https://www.${DOMAIN_MAIN}
GOTRUE_DISABLE_SIGNUP: "false"
GOTRUE_JWT_ADMIN_ROLES: service_role
GOTRUE_JWT_AUD: authenticated
GOTRUE_JWT_DEFAULT_GROUP_NAME: authenticated
GOTRUE_JWT_EXP: 3600
GOTRUE_JWT_SECRET: ${JWT_SECRET}
GOTRUE_EXTERNAL_EMAIL_ENABLED: "true"
# SMTP is not configured yet, so we auto-confirm signups (skips email
# verification). Switch to "false" once SMTP_PASS is set.
GOTRUE_MAILER_AUTOCONFIRM: "true"
GOTRUE_MAILER_OTP_EXP: 3600
GOTRUE_SMTP_HOST: ${SMTP_HOST}
GOTRUE_SMTP_PORT: ${SMTP_PORT}
GOTRUE_SMTP_USER: ${SMTP_USER}
GOTRUE_SMTP_PASS: ${SMTP_PASS}
GOTRUE_SMTP_ADMIN_EMAIL: ${SMTP_FROM}
GOTRUE_SMTP_SENDER_NAME: ${SMTP_FROM_NAME}
GOTRUE_MAILER_URLPATHS_INVITE: /auth/callback
GOTRUE_MAILER_URLPATHS_CONFIRMATION: /auth/callback
GOTRUE_MAILER_URLPATHS_RECOVERY: /auth/callback
GOTRUE_MAILER_URLPATHS_EMAIL_CHANGE: /auth/callback
# ─── PostgREST ────────────────────────────────────────────────────────────
rest:
container_name: disclosure-rest
image: postgrest/postgrest:v12.2.8
restart: unless-stopped
networks: [internal]
depends_on:
db: { condition: service_healthy }
environment:
PGRST_DB_URI: postgres://authenticator:${POSTGRES_PASSWORD}@db:5432/postgres
PGRST_DB_SCHEMAS: public,storage
PGRST_DB_ANON_ROLE: anon
PGRST_JWT_SECRET: ${JWT_SECRET}
PGRST_DB_USE_LEGACY_GUCS: "false"
PGRST_APP_SETTINGS_JWT_SECRET: ${JWT_SECRET}
PGRST_APP_SETTINGS_JWT_EXP: 3600
# ─── Realtime ─────────────────────────────────────────────────────────────
realtime:
container_name: disclosure-realtime
image: supabase/realtime:v2.30.34
restart: unless-stopped
networks: [internal]
depends_on:
db: { condition: service_healthy }
environment:
PORT: 4000
DB_HOST: db
DB_PORT: 5432
DB_USER: supabase_admin
DB_PASSWORD: ${POSTGRES_PASSWORD}
DB_NAME: postgres
DB_ENC_KEY: ${VAULT_ENC_KEY}
API_JWT_SECRET: ${JWT_SECRET}
SECRET_KEY_BASE: ${SECRET_KEY_BASE}
ERL_AFLAGS: -proto_dist inet_tcp
DNS_NODES: "''"
RLIMIT_NOFILE: "10000"
APP_NAME: realtime
SEED_SELF_HOST: "true"
RUN_JANITOR: "true"
# ─── Storage ──────────────────────────────────────────────────────────────
storage:
container_name: disclosure-storage
image: supabase/storage-api:v1.14.3
restart: unless-stopped
networks: [internal]
depends_on:
db: { condition: service_healthy }
rest: { condition: service_started }
imgproxy: { condition: service_started }
environment:
ANON_KEY: ${ANON_KEY}
SERVICE_KEY: ${SERVICE_ROLE_KEY}
POSTGREST_URL: http://rest:3000
PGRST_JWT_SECRET: ${JWT_SECRET}
DATABASE_URL: postgres://supabase_storage_admin:${POSTGRES_PASSWORD}@db:5432/postgres
FILE_SIZE_LIMIT: 52428800
STORAGE_BACKEND: file
FILE_STORAGE_BACKEND_PATH: /var/lib/storage
TENANT_ID: stub
REGION: stub
GLOBAL_S3_BUCKET: stub
ENABLE_IMAGE_TRANSFORMATION: "true"
IMGPROXY_URL: http://imgproxy:5001
volumes:
- storage-data:/var/lib/storage
imgproxy:
container_name: disclosure-imgproxy
image: darthsim/imgproxy:v3.8.0
restart: unless-stopped
networks: [internal]
environment:
IMGPROXY_BIND: ":5001"
IMGPROXY_LOCAL_FILESYSTEM_ROOT: /
IMGPROXY_USE_ETAG: "true"
IMGPROXY_ENABLE_WEBP_DETECTION: "true"
volumes:
- storage-data:/var/lib/storage
# ─── pg-meta + Studio ─────────────────────────────────────────────────────
meta:
container_name: disclosure-meta
image: supabase/postgres-meta:v0.83.2
restart: unless-stopped
networks: [internal]
depends_on:
db: { condition: service_healthy }
environment:
PG_META_PORT: 8080
PG_META_DB_HOST: db
PG_META_DB_PORT: 5432
PG_META_DB_NAME: postgres
PG_META_DB_USER: supabase_admin
PG_META_DB_PASSWORD: ${POSTGRES_PASSWORD}
studio:
container_name: disclosure-studio
image: supabase/studio:20241202-71e5240
restart: unless-stopped
networks: [internal, traefik]
depends_on:
meta: { condition: service_started }
environment:
STUDIO_PG_META_URL: http://meta:8080
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
DEFAULT_ORGANIZATION_NAME: "Disclosure Bureau"
DEFAULT_PROJECT_NAME: "disclosure"
SUPABASE_URL: http://kong:8000
SUPABASE_PUBLIC_URL: https://${DOMAIN_API}
SUPABASE_ANON_KEY: ${ANON_KEY}
SUPABASE_SERVICE_KEY: ${SERVICE_ROLE_KEY}
AUTH_JWT_SECRET: ${JWT_SECRET}
DASHBOARD_USERNAME: ${DASHBOARD_USERNAME}
DASHBOARD_PASSWORD: ${DASHBOARD_PASSWORD}
labels:
- traefik.enable=true
- traefik.docker.network=traefik-public
- traefik.http.routers.disclosure-studio.rule=Host(`${DOMAIN_STUDIO}`)
- traefik.http.routers.disclosure-studio.entrypoints=websecure
- traefik.http.routers.disclosure-studio.tls=true
- traefik.http.routers.disclosure-studio.tls.certresolver=letsencrypt
- traefik.http.services.disclosure-studio.loadbalancer.server.port=3000
- traefik.http.middlewares.disclosure-studio-auth.basicauth.usersfile=/dev/null
# Studio is sensitive — protect with basic auth. We use the dashboard creds via labels:
# Generate htpasswd format with: htpasswd -nbB admin <pass>
# ─── Kong API gateway ─────────────────────────────────────────────────────
kong:
container_name: disclosure-kong
image: kong:2.8.1
restart: unless-stopped
networks: [internal, traefik]
depends_on:
auth: { condition: service_started }
rest: { condition: service_started }
realtime: { condition: service_started }
storage: { condition: service_started }
environment:
KONG_DATABASE: "off"
# Read rendered config (envsubst happens at startup, see entrypoint below)
KONG_DECLARATIVE_CONFIG: /tmp/kong.yml
KONG_DNS_ORDER: LAST,A,CNAME
KONG_PLUGINS: request-transformer,cors,key-auth,acl,basic-auth
KONG_NGINX_PROXY_PROXY_BUFFER_SIZE: 160k
KONG_NGINX_PROXY_PROXY_BUFFERS: 64 160k
SUPABASE_ANON_KEY: ${ANON_KEY}
SUPABASE_SERVICE_KEY: ${SERVICE_ROLE_KEY}
DASHBOARD_USERNAME: ${DASHBOARD_USERNAME}
DASHBOARD_PASSWORD: ${DASHBOARD_PASSWORD}
# Kong declarative config does NOT do env substitution by itself. We
# render the template into /tmp/kong.yml at container start so the JWT
# keys land literally in the config Kong actually loads.
user: root
entrypoint:
- /bin/sh
- -c
- |
apk add --no-cache gettext >/dev/null 2>&1 || true
envsubst < /usr/local/kong/kong.yml.tmpl > /tmp/kong.yml
chown kong:kong /tmp/kong.yml
exec /docker-entrypoint.sh kong docker-start
volumes:
- ./kong.yml:/usr/local/kong/kong.yml.tmpl:ro
labels:
- traefik.enable=true
- traefik.docker.network=traefik-public
- traefik.http.routers.disclosure-api.rule=Host(`${DOMAIN_API}`)
- traefik.http.routers.disclosure-api.entrypoints=websecure
- traefik.http.routers.disclosure-api.tls=true
- traefik.http.routers.disclosure-api.tls.certresolver=letsencrypt
- traefik.http.services.disclosure-api.loadbalancer.server.port=8000
# ─── Meilisearch ──────────────────────────────────────────────────────────
meilisearch:
container_name: disclosure-meili
image: getmeili/meilisearch:v1.10
restart: unless-stopped
networks: [internal, traefik]
environment:
MEILI_MASTER_KEY: ${MEILI_MASTER_KEY}
MEILI_NO_ANALYTICS: "true"
MEILI_ENV: production
MEILI_MAX_INDEXING_MEMORY: ${MEILI_MAX_INDEXING_MEMORY:-512MB}
volumes:
- meili-data:/meili_data
labels:
- traefik.enable=true
- traefik.docker.network=traefik-public
- traefik.http.routers.disclosure-search.rule=Host(`${DOMAIN_SEARCH}`)
- traefik.http.routers.disclosure-search.entrypoints=websecure
- traefik.http.routers.disclosure-search.tls=true
- traefik.http.routers.disclosure-search.tls.certresolver=letsencrypt
- traefik.http.services.disclosure-search.loadbalancer.server.port=7700
# ─── Next.js web (Disclosure Bureau frontend) ─────────────────────────────
web:
container_name: disclosure-web
build:
context: /data/disclosure/web # rsynced from laptop, see scripts/sync-data.sh
dockerfile: Dockerfile
args:
NEXT_PUBLIC_SUPABASE_URL: https://${DOMAIN_API}
NEXT_PUBLIC_SUPABASE_ANON_KEY: ${ANON_KEY}
NEXT_PUBLIC_SITE_URL: https://${DOMAIN_MAIN}
restart: unless-stopped
networks: [internal, traefik]
depends_on:
kong: { condition: service_started }
environment:
NODE_ENV: production
NODE_OPTIONS: ${NEXT_NODE_OPTIONS:---max-old-space-size=768}
NEXT_PUBLIC_SUPABASE_URL: https://${DOMAIN_API}
NEXT_PUBLIC_SUPABASE_ANON_KEY: ${ANON_KEY}
SUPABASE_SERVICE_ROLE_KEY: ${SERVICE_ROLE_KEY}
NEXT_PUBLIC_SITE_URL: https://${DOMAIN_MAIN}
UFO_ROOT: /data/ufo
# Chat agent
CLAUDE_CODE_OAUTH_TOKEN: ${CLAUDE_CODE_OAUTH_TOKEN}
CLAUDE_CODE_MODEL: ${CLAUDE_CODE_MODEL}
OPENROUTER_API_KEY: ${OPENROUTER_API_KEY}
OPENROUTER_MODEL: ${OPENROUTER_MODEL}
OPENROUTER_FALLBACK_MODEL: ${OPENROUTER_FALLBACK_MODEL}
CHAT_PROVIDER: ${CHAT_PROVIDER}
# Meilisearch (used by /api/search)
MEILISEARCH_URL: http://meilisearch:7700
MEILISEARCH_API_KEY: ${MEILI_MASTER_KEY}
# Embed service (used by /lib/retrieval)
EMBED_SERVICE_URL: http://embed:8000
# pgvector + chunks (hybrid_search)
DATABASE_URL: postgres://postgres:${POSTGRES_PASSWORD}@db:5432/postgres
volumes:
- ${DATA_WIKI}:/data/ufo/wiki:ro
- ${DATA_PROCESSING}:/data/ufo/processing:ro
- ${DATA_RAW}:/data/ufo/raw:ro
labels:
- traefik.enable=true
- traefik.docker.network=traefik-public
- traefik.http.routers.disclosure-web.rule=Host(`app.${DOMAIN_MAIN}`) || Host(`${DOMAIN_MAIN}`) || Host(`www.${DOMAIN_MAIN}`)
- traefik.http.routers.disclosure-web.entrypoints=websecure
- traefik.http.routers.disclosure-web.tls=true
- traefik.http.routers.disclosure-web.tls.certresolver=letsencrypt
- traefik.http.services.disclosure-web.loadbalancer.server.port=3000
# www → apex redirect
- traefik.http.middlewares.disclosure-www-redir.redirectregex.regex=^https?://www\.${DOMAIN_MAIN}/(.*)
- traefik.http.middlewares.disclosure-www-redir.redirectregex.replacement=https://${DOMAIN_MAIN}/$${1}
- traefik.http.middlewares.disclosure-www-redir.redirectregex.permanent=true
# ─── BGE-M3 embedding + reranker service (CPU only) ───────────────────────
embed:
container_name: disclosure-embed
build:
context: ../embed-service
restart: unless-stopped
networks: [internal]
environment:
DEVICE: cpu
EMBED_MODEL: BAAI/bge-m3
RERANK_MODEL: BAAI/bge-reranker-v2-m3
HF_HUB_DOWNLOAD_TIMEOUT: 600
volumes:
- hf-cache:/cache
healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8000/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 180s
deploy:
resources:
limits:
memory: 3g

57
infra/disclosure-stack/init-db.sql Normal file
View file

@ -0,0 +1,57 @@
-- Bootstrap roles + schemas that Supabase services expect.
-- Run AFTER the db container is up but BEFORE auth/rest/storage/realtime start.
-- Pattern matches supabase/postgres official image; if you use that image as base,
-- it auto-runs migrations from /docker-entrypoint-initdb.d/.
-- Roles
DO $$
BEGIN
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'anon') THEN
CREATE ROLE anon NOLOGIN NOINHERIT;
END IF;
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'authenticated') THEN
CREATE ROLE authenticated NOLOGIN NOINHERIT;
END IF;
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'service_role') THEN
CREATE ROLE service_role NOLOGIN NOINHERIT BYPASSRLS;
END IF;
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'authenticator') THEN
EXECUTE format('CREATE ROLE authenticator LOGIN NOINHERIT PASSWORD %L', current_setting('app.pg_password', true));
END IF;
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'supabase_admin') THEN
EXECUTE format('CREATE ROLE supabase_admin LOGIN CREATEROLE CREATEDB REPLICATION BYPASSRLS PASSWORD %L', current_setting('app.pg_password', true));
END IF;
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'supabase_auth_admin') THEN
EXECUTE format('CREATE ROLE supabase_auth_admin LOGIN NOINHERIT CREATEROLE PASSWORD %L', current_setting('app.pg_password', true));
END IF;
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'supabase_storage_admin') THEN
EXECUTE format('CREATE ROLE supabase_storage_admin LOGIN NOINHERIT CREATEROLE PASSWORD %L', current_setting('app.pg_password', true));
END IF;
END
$$;
GRANT anon TO authenticator;
GRANT authenticated TO authenticator;
GRANT service_role TO authenticator;
GRANT supabase_admin TO authenticator;
-- Schemas
CREATE SCHEMA IF NOT EXISTS auth AUTHORIZATION supabase_auth_admin;
CREATE SCHEMA IF NOT EXISTS storage AUTHORIZATION supabase_storage_admin;
CREATE SCHEMA IF NOT EXISTS extensions;
CREATE SCHEMA IF NOT EXISTS realtime AUTHORIZATION supabase_admin;
-- Extensions used by Supabase
CREATE EXTENSION IF NOT EXISTS pgcrypto;
CREATE EXTENSION IF NOT EXISTS pgjwt SCHEMA extensions;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp" SCHEMA extensions;
-- Grant permissions
GRANT USAGE ON SCHEMA public TO postgres, anon, authenticated, service_role;
GRANT ALL ON ALL TABLES IN SCHEMA public TO postgres, anon, authenticated, service_role;
GRANT ALL ON ALL ROUTINES IN SCHEMA public TO postgres, anon, authenticated, service_role;
GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO postgres, anon, authenticated, service_role;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA public GRANT ALL ON TABLES TO postgres, anon, authenticated, service_role;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA public GRANT ALL ON ROUTINES TO postgres, anon, authenticated, service_role;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA public GRANT ALL ON SEQUENCES TO postgres, anon, authenticated, service_role;

116
infra/disclosure-stack/kong.yml Normal file
View file

@ -0,0 +1,116 @@
# Kong declarative config — Supabase API gateway.
# Routes:
# /auth/v1/* → GoTrue
# /rest/v1/* → PostgREST
# /realtime/v1/* → Realtime
# /storage/v1/* → Storage API
# /pg/* → postgres-meta (Studio backend)
_format_version: "2.1"
_transform: true
consumers:
- username: anon
keyauth_credentials:
- key: ${SUPABASE_ANON_KEY}
- username: service_role
keyauth_credentials:
- key: ${SUPABASE_SERVICE_KEY}
acls:
- consumer: anon
group: anon
- consumer: service_role
group: admin
services:
- name: auth-v1-open
url: http://auth:9999/verify
routes:
- name: auth-v1-open
strip_path: true
paths: [/auth/v1/verify]
plugins:
- name: cors
- name: auth-v1-open-callback
url: http://auth:9999/callback
routes:
- name: auth-v1-open-callback
strip_path: true
paths: [/auth/v1/callback]
plugins:
- name: cors
- name: auth-v1-open-authorize
url: http://auth:9999/authorize
routes:
- name: auth-v1-open-authorize
strip_path: true
paths: [/auth/v1/authorize]
plugins:
- name: cors
- name: auth-v1
_comment: "GoTrue: /auth/v1/* -> http://auth:9999/*"
url: http://auth:9999/
routes:
- name: auth-v1-all
strip_path: true
paths: [/auth/v1/]
plugins:
- name: cors
- name: key-auth
config: { hide_credentials: false }
- name: acl
config: { hide_groups_header: true, allow: [admin, anon] }
- name: rest-v1
_comment: "PostgREST: /rest/v1/* -> http://rest:3000/*"
url: http://rest:3000/
routes:
- name: rest-v1-all
strip_path: true
paths: [/rest/v1/]
plugins:
- name: cors
- name: key-auth
config: { hide_credentials: true }
- name: acl
config: { hide_groups_header: true, allow: [admin, anon] }
- name: realtime-v1
_comment: "Realtime: /realtime/v1/* -> ws://realtime:4000/socket/*"
url: http://realtime:4000/socket/
routes:
- name: realtime-v1-all
strip_path: true
paths: [/realtime/v1/]
plugins:
- name: cors
- name: key-auth
config: { hide_credentials: false }
- name: acl
config: { hide_groups_header: true, allow: [admin, anon] }
- name: storage-v1
_comment: "Storage: /storage/v1/* -> http://storage:5000/*"
url: http://storage:5000/
routes:
- name: storage-v1-all
strip_path: true
paths: [/storage/v1/]
plugins:
- name: cors
- name: meta
_comment: "pg-meta: /pg/* -> http://meta:8080/*"
url: http://meta:8080/
routes:
- name: meta-all
strip_path: true
paths: [/pg/]
plugins:
- name: key-auth
- name: acl
config: { hide_groups_header: true, allow: [admin] }

70
infra/disclosure-stack/scripts/_lib.sh Executable file
View file

@ -0,0 +1,70 @@
#!/usr/bin/env bash
# Shared library: loads .env and exposes SSH helpers.
# Sourced by every script in this directory.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
STACK_DIR="$(dirname "$SCRIPT_DIR")"
ENV_FILE="${STACK_DIR}/.env"
if [ ! -f "$ENV_FILE" ]; then
echo "❌ $ENV_FILE not found. Copy .env.example to .env and fill it in." >&2
exit 1
fi
# Load .env without leaking variables to the shell history
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
: "${VPS_HOST:?VPS_HOST not set in .env}"
: "${VPS_USER:?VPS_USER not set in .env}"
: "${VPS_PORT:=22}"
: "${VPS_AUTH:=password}"
# ssh wrapper — uses password (sshpass) OR key, transparently
vps_ssh() {
local cmd="${1:-}"
if [ "$VPS_AUTH" = "password" ]; then
if ! command -v sshpass >/dev/null; then
echo "❌ sshpass not installed. Install with: brew install hudochenkov/sshpass/sshpass" >&2
exit 1
fi
if [ -n "$cmd" ]; then
SSHPASS="$VPS_PASSWORD" sshpass -e \
ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 \
-p "$VPS_PORT" "${VPS_USER}@${VPS_HOST}" "$cmd"
else
SSHPASS="$VPS_PASSWORD" sshpass -e \
ssh -o StrictHostKeyChecking=accept-new \
-p "$VPS_PORT" "${VPS_USER}@${VPS_HOST}"
fi
else
local key="${VPS_SSH_KEY/#\~/$HOME}"
if [ -n "$cmd" ]; then
ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 \
-p "$VPS_PORT" -i "$key" "${VPS_USER}@${VPS_HOST}" "$cmd"
else
ssh -o StrictHostKeyChecking=accept-new \
-p "$VPS_PORT" -i "$key" "${VPS_USER}@${VPS_HOST}"
fi
fi
}
# rsync wrapper
vps_rsync() {
local src="$1" dst="$2"
if [ "$VPS_AUTH" = "password" ]; then
SSHPASS="$VPS_PASSWORD" sshpass -e \
rsync -avz --progress \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT" \
"$src" "${VPS_USER}@${VPS_HOST}:${dst}"
else
local key="${VPS_SSH_KEY/#\~/$HOME}"
rsync -avz --progress \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT -i $key" \
"$src" "${VPS_USER}@${VPS_HOST}:${dst}"
fi
}

114
infra/disclosure-stack/scripts/bootstrap.sh Executable file
View file

@ -0,0 +1,114 @@
#!/usr/bin/env bash
# Full first-time bootstrap on the VPS.
# Idempotent — safe to re-run; rsync only ships diffs, docker compose only restarts changed services.
#
# Steps:
# 1. mkdir /data/disclosure on VPS
# 2. rsync compose files + web/ source + wiki/ + processing/png + processing/crops
# 3. SSH: docker compose up -d db
# 4. Wait for db healthy
# 5. Apply Supabase init + chat schema
# 6. docker compose up -d (everything else)
# 7. Report URLs
set -euo pipefail
source "$(dirname "$0")/_lib.sh"
LAPTOP_UFO_ROOT="${LAPTOP_UFO_ROOT:-/Users/guto/ufo}"
STACK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
echo "================================================================"
echo " STAGE A — create deploy dir + sync code (fast)"
echo "================================================================"
vps_ssh "mkdir -p ${VPS_DEPLOY_ROOT}/{web,wiki,processing,raw,migrations}"
echo ""
echo "→ Syncing infra/disclosure-stack/ (compose + kong + .env + scripts)"
vps_rsync "${STACK_DIR}/docker-compose.yml" "${VPS_DEPLOY_ROOT}/docker-compose.yml"
vps_rsync "${STACK_DIR}/kong.yml" "${VPS_DEPLOY_ROOT}/kong.yml"
vps_rsync "${STACK_DIR}/init-db.sql" "${VPS_DEPLOY_ROOT}/migrations/00-init.sql"
vps_rsync "${STACK_DIR}/../supabase/migrations/0001_chat_schema.sql" "${VPS_DEPLOY_ROOT}/migrations/01-chat-schema.sql"
vps_rsync "${STACK_DIR}/../supabase/migrations/0002_chunks_retrieval.sql" "${VPS_DEPLOY_ROOT}/migrations/02-chunks-retrieval.sql"
vps_rsync "${STACK_DIR}/.env" "${VPS_DEPLOY_ROOT}/.env"
echo ""
echo "→ Syncing web/ (Next.js source, excl. node_modules/.next)"
if [ "$VPS_AUTH" = "password" ]; then
SSHPASS="$VPS_PASSWORD" sshpass -e rsync -avz --delete \
--exclude node_modules --exclude .next --exclude .env.local \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT" \
"${LAPTOP_UFO_ROOT}/web/" "${VPS_USER}@${VPS_HOST}:${VPS_DEPLOY_ROOT}/web/"
else
rsync -avz --delete \
--exclude node_modules --exclude .next --exclude .env.local \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT -i ${VPS_SSH_KEY/#\~/$HOME}" \
"${LAPTOP_UFO_ROOT}/web/" "${VPS_USER}@${VPS_HOST}:${VPS_DEPLOY_ROOT}/web/"
fi
echo ""
echo "================================================================"
echo " STAGE B — sync wiki + processing (data — can be slow first time)"
echo "================================================================"
echo "→ Syncing wiki/ (markdown)"
vps_rsync "${LAPTOP_UFO_ROOT}/wiki/" "${VPS_DEPLOY_ROOT}/wiki/"
echo ""
echo "→ Syncing processing/png/ (page images — large)"
vps_rsync "${LAPTOP_UFO_ROOT}/processing/png/" "${VPS_DEPLOY_ROOT}/processing/png/"
echo ""
echo "→ Syncing processing/ocr/ (text)"
vps_rsync "${LAPTOP_UFO_ROOT}/processing/ocr/" "${VPS_DEPLOY_ROOT}/processing/ocr/"
echo ""
echo "→ Syncing processing/crops/ + processing/tables/ + processing/uap-frames/"
vps_rsync "${LAPTOP_UFO_ROOT}/processing/crops/" "${VPS_DEPLOY_ROOT}/processing/crops/" 2>&1 || true
vps_rsync "${LAPTOP_UFO_ROOT}/processing/tables/" "${VPS_DEPLOY_ROOT}/processing/tables/" 2>&1 || true
vps_rsync "${LAPTOP_UFO_ROOT}/processing/case-images/" "${VPS_DEPLOY_ROOT}/processing/case-images/" 2>&1 || true
vps_rsync "${LAPTOP_UFO_ROOT}/processing/uap-frames/" "${VPS_DEPLOY_ROOT}/processing/uap-frames/" 2>&1 || true
vps_rsync "${LAPTOP_UFO_ROOT}/processing/video-analysis/" "${VPS_DEPLOY_ROOT}/processing/video-analysis/" 2>&1 || true
echo ""
echo "================================================================"
echo " STAGE C — start stack on VPS"
echo "================================================================"
vps_ssh "set -e
cd ${VPS_DEPLOY_ROOT}
echo '→ docker compose up -d db'
docker compose up -d db
echo '→ Waiting for db healthy…'
for i in {1..30}; do
STATUS=\$(docker inspect --format='{{.State.Health.Status}}' disclosure-db 2>/dev/null || echo starting)
if [ \"\$STATUS\" = healthy ]; then echo ' ✓ db healthy'; break; fi
sleep 2
done
echo ''
echo '→ Applying init-db.sql (roles + schemas)'
docker exec -i disclosure-db psql -U postgres < migrations/00-init.sql 2>&1 | tail -5 || true
echo ''
echo '→ Applying chat schema'
docker exec -i disclosure-db psql -U postgres < migrations/01-chat-schema.sql 2>&1 | tail -5 || true
echo ''
echo '→ Applying chunks retrieval schema (pgvector + hybrid_search)'
docker exec -i disclosure-db psql -U postgres < migrations/02-chunks-retrieval.sql 2>&1 | tail -5 || true
echo ''
echo '→ docker compose up -d (entire stack incl. embed-service)'
docker compose up -d
echo ''
echo '→ Status:'
docker compose ps
"
echo ""
echo "================================================================"
echo " ✓ Bootstrap complete"
echo "================================================================"
echo ""
echo "URLs (give DNS + TLS issuance ~5 min):"
echo " Main app: https://${DOMAIN_MAIN}"
echo " Studio: https://${DOMAIN_STUDIO}"
echo " Supa API: https://${DOMAIN_API}"
echo " Search: https://${DOMAIN_SEARCH}"
echo ""
echo "Tail container logs: ./scripts/logs.sh <service>"
echo "Service list: ./scripts/ssh.sh 'cd ${VPS_DEPLOY_ROOT} && docker compose ps'"

126
infra/disclosure-stack/scripts/deploy-incremental.sh Executable file
View file

@ -0,0 +1,126 @@
#!/usr/bin/env bash
# Incremental deploy — syncs only new/changed code + raw/--subagent data.
# Skips re-syncing wiki/, processing/png/, processing/ocr/ (already on VPS).
#
# Steps:
# 1. Sync infra updates (compose, embed-service Dockerfile, migration 02)
# 2. Sync web/ Next.js source (V2, search, graph, timeline, stats, command palette)
# 3. Sync scripts/ (30-33 new scripts)
# 4. Sync raw/*--subagent/ (chunks, 634MB — needed for V2 view)
# 5. Apply migration 02 (pgvector + chunks_retrieval schema)
# 6. Build & start embed-service (BGE-M3 + reranker) — runs in background
# 7. Rebuild web container
# 8. Report URLs
source "$(dirname "$0")/_lib.sh"
set -euo pipefail
STACK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
UFO_ROOT="${LAPTOP_UFO_ROOT:-/Users/guto/ufo}"
echo "════════════════════════════════════════════════════════════════"
echo " STAGE 1 — sync infra (compose, embed-service, migration 02)"
echo "════════════════════════════════════════════════════════════════"
vps_ssh "mkdir -p ${VPS_DEPLOY_ROOT}/{migrations,embed-service}"
vps_rsync "${STACK_DIR}/docker-compose.yml" "${VPS_DEPLOY_ROOT}/docker-compose.yml"
vps_rsync "${STACK_DIR}/kong.yml" "${VPS_DEPLOY_ROOT}/kong.yml"
vps_rsync "${STACK_DIR}/.env" "${VPS_DEPLOY_ROOT}/.env"
vps_rsync "${STACK_DIR}/../supabase/migrations/0002_chunks_retrieval.sql" \
"${VPS_DEPLOY_ROOT}/migrations/02-chunks-retrieval.sql"
vps_rsync "${STACK_DIR}/../embed-service/Dockerfile" "${VPS_DEPLOY_ROOT}/embed-service/Dockerfile"
vps_rsync "${STACK_DIR}/../embed-service/app.py" "${VPS_DEPLOY_ROOT}/embed-service/app.py"
vps_rsync "${STACK_DIR}/../embed-service/requirements.txt" "${VPS_DEPLOY_ROOT}/embed-service/requirements.txt"
echo ""
echo "════════════════════════════════════════════════════════════════"
echo " STAGE 2 — sync web/ source (Next.js — V2 + retrieval features)"
echo "════════════════════════════════════════════════════════════════"
if [ "$VPS_AUTH" = "password" ]; then
SSHPASS="$VPS_PASSWORD" sshpass -e rsync -avz --delete \
--exclude node_modules --exclude .next --exclude .env.local \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT" \
"${UFO_ROOT}/web/" "${VPS_USER}@${VPS_HOST}:${VPS_DEPLOY_ROOT}/web/"
else
rsync -avz --delete \
--exclude node_modules --exclude .next --exclude .env.local \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT -i ${VPS_SSH_KEY/#\~/$HOME}" \
"${UFO_ROOT}/web/" "${VPS_USER}@${VPS_HOST}:${VPS_DEPLOY_ROOT}/web/"
fi
echo ""
echo "════════════════════════════════════════════════════════════════"
echo " STAGE 3 — sync scripts/ (30-33 retrieval pipeline scripts)"
echo "════════════════════════════════════════════════════════════════"
vps_ssh "mkdir -p ${VPS_DEPLOY_ROOT}/scripts"
if [ "$VPS_AUTH" = "password" ]; then
SSHPASS="$VPS_PASSWORD" sshpass -e rsync -avz \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT" \
"${UFO_ROOT}/scripts/" "${VPS_USER}@${VPS_HOST}:${VPS_DEPLOY_ROOT}/scripts/"
else
rsync -avz \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT -i ${VPS_SSH_KEY/#\~/$HOME}" \
"${UFO_ROOT}/scripts/" "${VPS_USER}@${VPS_HOST}:${VPS_DEPLOY_ROOT}/scripts/"
fi
echo ""
echo "════════════════════════════════════════════════════════════════"
echo " STAGE 4 — sync raw/*--subagent/ (chunks v0.2.0, ~634MB, 116 docs)"
echo "════════════════════════════════════════════════════════════════"
echo "Note: only --subagent/ archives are synced (chunks + crops + index)"
echo " Raw PDFs already on VPS — not re-synced"
if [ "$VPS_AUTH" = "password" ]; then
SSHPASS="$VPS_PASSWORD" sshpass -e rsync -avz \
--include='*--subagent/' --include='*--subagent/**' --include='*.pdf' --exclude='*' \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT" \
"${UFO_ROOT}/raw/" "${VPS_USER}@${VPS_HOST}:${DATA_RAW:-${VPS_DEPLOY_ROOT}/raw}/"
else
rsync -avz \
--include='*--subagent/' --include='*--subagent/**' --include='*.pdf' --exclude='*' \
-e "ssh -o StrictHostKeyChecking=accept-new -p $VPS_PORT -i ${VPS_SSH_KEY/#\~/$HOME}" \
"${UFO_ROOT}/raw/" "${VPS_USER}@${VPS_HOST}:${DATA_RAW:-${VPS_DEPLOY_ROOT}/raw}/"
fi
echo ""
echo "════════════════════════════════════════════════════════════════"
echo " STAGE 5 — apply migration 02 (pgvector + chunks tables)"
echo "════════════════════════════════════════════════════════════════"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker exec -i disclosure-db psql -U postgres < migrations/02-chunks-retrieval.sql 2>&1 | tail -10"
echo ""
echo "════════════════════════════════════════════════════════════════"
echo " STAGE 6 — build + start embed-service (BGE-M3, ~10min first build)"
echo "════════════════════════════════════════════════════════════════"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose build embed 2>&1 | tail -5"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose up -d embed"
echo ""
echo "════════════════════════════════════════════════════════════════"
echo " STAGE 7 — rebuild + restart web (Next.js with V2/search/graph)"
echo "════════════════════════════════════════════════════════════════"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose build web 2>&1 | tail -5"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose up -d --force-recreate web"
echo ""
echo "════════════════════════════════════════════════════════════════"
echo " STAGE 8 — status"
echo "════════════════════════════════════════════════════════════════"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose ps"
echo ""
echo "✓ Incremental deploy complete."
echo ""
echo "Test URLs:"
echo " https://${DOMAIN_MAIN:-disclosure.top}/ (home with filters + summaries + ✨v2 badges)"
echo " https://${DOMAIN_MAIN:-disclosure.top}/d/doc-342-.../v2 (rich chunks view)"
echo " https://${DOMAIN_MAIN:-disclosure.top}/search (hybrid search palette)"
echo " https://${DOMAIN_MAIN:-disclosure.top}/timeline (decade timeline)"
echo " https://${DOMAIN_MAIN:-disclosure.top}/graph (force-directed)"
echo " https://${DOMAIN_MAIN:-disclosure.top}/admin/stats (corpus analytics)"
echo " https://${DOMAIN_MAIN:-disclosure.top}/admin/batch (batch monitor)"
echo " https://${DOMAIN_MAIN:-disclosure.top}/admin/indexer (retrieval health)"
echo ""
echo "Embed-service may take ~5 minutes to download BGE-M3 weights on first /embed call."
echo "Indexer (next step, not run by this script):"
echo " ssh vps && docker exec disclosure-embed curl -s http://localhost:8000/health"
echo " # then run: scripts/30-index-chunks-to-db.py from a container with internal network"

46
infra/disclosure-stack/scripts/deploy.sh Executable file
View file

@ -0,0 +1,46 @@
#!/usr/bin/env bash
# Deploy / update the disclosure stack on the VPS.
#
# Steps:
# 1. Ensure VPS_DEPLOY_ROOT exists.
# 2. Upload docker-compose.yml + .env to the VPS.
# 3. Pull images.
# 4. docker compose up -d.
# 5. Print status.
#
# Idempotent: re-run anytime, only changed services restart.
source "$(dirname "$0")/_lib.sh"
set -euo pipefail
STACK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
COMPOSE_FILE="${STACK_DIR}/docker-compose.yml"
ENV_FILE="${STACK_DIR}/.env"
if [ ! -f "$COMPOSE_FILE" ]; then
echo "❌ $COMPOSE_FILE not found. The docker-compose.yml hasn't been generated yet — see README."
exit 1
fi
echo "→ Creating deploy root on VPS: $VPS_DEPLOY_ROOT"
vps_ssh "mkdir -p ${VPS_DEPLOY_ROOT}"
echo ""
echo "→ Uploading docker-compose.yml and .env"
vps_rsync "$COMPOSE_FILE" "${VPS_DEPLOY_ROOT}/docker-compose.yml"
vps_rsync "$ENV_FILE" "${VPS_DEPLOY_ROOT}/.env"
echo ""
echo "→ Pulling latest images on VPS"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose pull"
echo ""
echo "→ Bringing stack up (will recreate changed services only)"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose up -d --remove-orphans"
echo ""
echo "→ Current stack status:"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose ps"
echo ""
echo "✓ Deploy complete. Logs: ./scripts/logs.sh <service>"

82
infra/disclosure-stack/scripts/gen-secrets.sh Executable file
View file

@ -0,0 +1,82 @@
#!/usr/bin/env bash
# Regenerate all per-VPS secrets in .env. Run ONCE per new VPS deployment.
# Backs up the existing .env first.
#
# Generates:
# POSTGRES_PASSWORD, JWT_SECRET, DASHBOARD_PASSWORD,
# SECRET_KEY_BASE, VAULT_ENC_KEY, MEILI_MASTER_KEY,
# IMGPROXY_KEY, IMGPROXY_SALT,
# ANON_KEY and SERVICE_ROLE_KEY (JWTs signed with JWT_SECRET)
#
# Usage: ./gen-secrets.sh
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
STACK_DIR="$(dirname "$SCRIPT_DIR")"
ENV_FILE="${STACK_DIR}/.env"
if [ ! -f "$ENV_FILE" ]; then
echo "❌ $ENV_FILE not found. Copy .env.example to .env first."
exit 1
fi
cp "$ENV_FILE" "${ENV_FILE}.backup.$(date +%s)"
echo "✓ backed up to ${ENV_FILE}.backup.<ts>"
POSTGRES_PASSWORD=$(openssl rand -hex 32 | head -c 48)
JWT_SECRET=$(openssl rand -hex 64)
DASHBOARD_PASSWORD=$(openssl rand -base64 24 | tr -d '/=+')
SECRET_KEY_BASE=$(openssl rand -hex 64)
VAULT_ENC_KEY=$(openssl rand -hex 32 | head -c 32)
MEILI_MASTER_KEY=$(openssl rand -hex 32)
IMGPROXY_KEY=$(openssl rand -hex 64)
IMGPROXY_SALT=$(openssl rand -hex 64)
# Generate Supabase ANON_KEY and SERVICE_ROLE_KEY (HS256 JWTs)
# Standard payload Supabase expects:
# iss: supabase ref: <project-ref> role: anon|service_role
# iat: now exp: now + 10 years
generate_jwt() {
local role="$1"
local now=$(date +%s)
local exp=$((now + 315360000)) # +10 years
local header_b64=$(printf '%s' '{"alg":"HS256","typ":"JWT"}' | openssl base64 -A | tr -d '=' | tr '/+' '_-')
local payload_b64=$(printf '{"iss":"supabase","ref":"disclosure","role":"%s","iat":%s,"exp":%s}' "$role" "$now" "$exp" | openssl base64 -A | tr -d '=' | tr '/+' '_-')
local signing_input="${header_b64}.${payload_b64}"
local sig=$(printf '%s' "$signing_input" | openssl dgst -sha256 -hmac "$JWT_SECRET" -binary | openssl base64 -A | tr -d '=' | tr '/+' '_-')
echo "${signing_input}.${sig}"
}
ANON_KEY=$(generate_jwt "anon")
SERVICE_ROLE_KEY=$(generate_jwt "service_role")
# Replace values in .env (only the lines that match these keys)
replace() {
local key="$1" value="$2"
# macOS sed needs '' after -i; this is portable enough for both BSD and GNU
if sed --version >/dev/null 2>&1; then
sed -i "s|^${key}=.*|${key}=${value}|" "$ENV_FILE"
else
sed -i '' "s|^${key}=.*|${key}=${value}|" "$ENV_FILE"
fi
}
replace POSTGRES_PASSWORD "$POSTGRES_PASSWORD"
replace JWT_SECRET "$JWT_SECRET"
replace DASHBOARD_PASSWORD "$DASHBOARD_PASSWORD"
replace SECRET_KEY_BASE "$SECRET_KEY_BASE"
replace VAULT_ENC_KEY "$VAULT_ENC_KEY"
replace MEILI_MASTER_KEY "$MEILI_MASTER_KEY"
replace IMGPROXY_KEY "$IMGPROXY_KEY"
replace IMGPROXY_SALT "$IMGPROXY_SALT"
replace ANON_KEY "$ANON_KEY"
replace SERVICE_ROLE_KEY "$SERVICE_ROLE_KEY"
echo "✓ secrets rotated in $ENV_FILE"
echo ""
echo "ANON_KEY (length=${#ANON_KEY}): ${ANON_KEY:0:40}…"
echo "SERVICE_ROLE_KEY (length=${#SERVICE_ROLE_KEY}): ${SERVICE_ROLE_KEY:0:40}…"
echo ""
echo "Both are JWTs signed with JWT_SECRET. Supabase uses them to authorize requests."
echo "Distribute ANON_KEY to clients (NEXT_PUBLIC_SUPABASE_ANON_KEY)."
echo "Keep SERVICE_ROLE_KEY private (server-only — bypasses RLS)."

18
infra/disclosure-stack/scripts/logs.sh Executable file
View file

@ -0,0 +1,18 @@
#!/usr/bin/env bash
# Tail the logs of a specific service on the VPS.
# Usage:
# ./logs.sh # interactive picker
# ./logs.sh postgres # tail postgres logs
# ./logs.sh next # tail Next.js
# ./logs.sh kong # tail Supabase API gateway
source "$(dirname "$0")/_lib.sh"
svc="${1:-}"
if [ -z "$svc" ]; then
echo "Available services:"
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose config --services 2>/dev/null || echo '(stack not yet deployed)'"
echo ""
read -rp "Service name: " svc
fi
vps_ssh "cd ${VPS_DEPLOY_ROOT} && docker compose logs --tail=200 -f ${svc}"

7
infra/disclosure-stack/scripts/ssh.sh Executable file
View file

@ -0,0 +1,7 @@
#!/usr/bin/env bash
# Open an interactive SSH session to the VPS using credentials from .env.
# Usage:
# ./ssh.sh # interactive shell
# ./ssh.sh "docker ps" # one-shot remote command
source "$(dirname "$0")/_lib.sh"
vps_ssh "${1:-}"

37
infra/disclosure-stack/scripts/status.sh Executable file
View file

@ -0,0 +1,37 @@
#!/usr/bin/env bash
# Print a comprehensive status report of the VPS + disclosure stack.
source "$(dirname "$0")/_lib.sh"
vps_ssh "bash -s" <<EOF
echo "=== HOST ==="
hostname; uname -r
[ -f /etc/os-release ] && . /etc/os-release && echo "\$PRETTY_NAME"
uptime
echo ""
echo "=== RESOURCES ==="
free -h
echo ""
df -h | grep -vE 'tmpfs|loop|overlay'
echo ""
echo "=== DISCLOSURE STACK ==="
if [ -d "$VPS_DEPLOY_ROOT" ]; then
cd "$VPS_DEPLOY_ROOT" || exit
if [ -f docker-compose.yml ]; then
docker compose ps 2>/dev/null || echo "(docker compose not running for this stack)"
else
echo "(no docker-compose.yml in $VPS_DEPLOY_ROOT yet — run scripts/deploy.sh first)"
fi
else
echo "(deploy root not yet created: $VPS_DEPLOY_ROOT)"
fi
echo ""
echo "=== ALL CONTAINERS ON HOST ==="
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' | head -40
echo ""
echo "=== EXPOSED PORTS ==="
ss -tlnp 2>/dev/null | awk '/LISTEN/' | head -30
EOF

29
infra/disclosure-stack/scripts/sync-data.sh Executable file
View file

@ -0,0 +1,29 @@
#!/usr/bin/env bash
# Push the wiki/, processing/, raw/ trees from your laptop to the VPS.
# First run: full sync (~minutes). Subsequent runs: only diffs (~seconds).
#
# Usage: ./sync-data.sh
source "$(dirname "$0")/_lib.sh"
set -euo pipefail
LAPTOP_UFO_ROOT="${LAPTOP_UFO_ROOT:-/Users/guto/ufo}"
echo "→ Ensuring $VPS_DEPLOY_ROOT exists on VPS…"
vps_ssh "mkdir -p ${VPS_DEPLOY_ROOT}/{wiki,processing,raw}"
echo ""
echo "→ Syncing wiki/ → ${DATA_WIKI:-$VPS_DEPLOY_ROOT/wiki}"
vps_rsync "${LAPTOP_UFO_ROOT}/wiki/" "${DATA_WIKI:-$VPS_DEPLOY_ROOT/wiki}/"
echo ""
echo "→ Syncing processing/ → ${DATA_PROCESSING:-$VPS_DEPLOY_ROOT/processing}"
vps_rsync "${LAPTOP_UFO_ROOT}/processing/" "${DATA_PROCESSING:-$VPS_DEPLOY_ROOT/processing}/"
echo ""
echo "→ Syncing raw/ → ${DATA_RAW:-$VPS_DEPLOY_ROOT/raw}"
vps_rsync "${LAPTOP_UFO_ROOT}/raw/" "${DATA_RAW:-$VPS_DEPLOY_ROOT/raw}/"
echo ""
echo "✓ Data synced. Sizes on VPS:"
vps_ssh "du -sh ${VPS_DEPLOY_ROOT}/{wiki,processing,raw}"

26
infra/embed-service/Dockerfile Normal file
View file

@ -0,0 +1,26 @@
FROM python:3.11-slim
ENV PYTHONUNBUFFERED=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_NO_CACHE_DIR=1 \
HF_HOME=/cache/huggingface \
TORCH_HOME=/cache/torch \
TRANSFORMERS_OFFLINE=0
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential git curl ca-certificates && \
rm -rf /var/lib/apt/lists/*
COPY requirements.txt /app/requirements.txt
RUN pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements.txt
COPY app.py /app/app.py
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=10s --start-period=120s --retries=3 \
CMD curl -fsS http://127.0.0.1:8000/health || exit 1
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]

50
infra/embed-service/README.md Normal file
View file

@ -0,0 +1,50 @@
# embed-service — BGE-M3 + BGE-Reranker-v2-M3 microservice
Self-hosted on the VPS, CPU-only. ~2.5 GB RAM in steady state. Powers hybrid retrieval over the chunks corpus.
## Endpoints
- `POST /embed` — batch dense embedding (1024 dim, normalized cosine)
- `POST /rerank` — cross-encoder rerank for candidate lists
- `GET /health`, `GET /info`
## Resource expectations
| Op | Cold | Warm | Notes |
|---|---|---|---|
| Embed 1 chunk (~400 tokens) | ~5s (load) | 100-200 ms | first request loads model |
| Embed batch of 16 | — | 800-1500 ms | use during indexing |
| Rerank 100 candidates | — | 5-8 s | called per query post-recall |
## Add to disclosure-stack
```yaml
embed:
build: ../embed-service
restart: unless-stopped
networks: [internal]
environment:
DEVICE: cpu
EMBED_MODEL: BAAI/bge-m3
RERANK_MODEL: BAAI/bge-reranker-v2-m3
volumes:
- hf-cache:/cache
deploy:
resources:
limits:
memory: 3g
```
## Test locally
```bash
docker build -t embed-service .
docker run --rm -p 8000:8000 -v hf-cache:/cache embed-service
curl -s http://localhost:8000/health
curl -s -X POST http://localhost:8000/embed \
-H 'Content-Type: application/json' \
-d '{"texts":["UAP avistado sobre Olathe, Kansas em 6 de janeiro de 1950"]}'
```
Note: First request downloads model weights (~2.3 GB total). Subsequent requests
hit the cache. Mount `hf-cache` as a named volume to persist across restarts.

148
infra/embed-service/app.py Normal file
View file

@ -0,0 +1,148 @@
"""BGE-M3 embedding + reranker microservice.
Self-hosted on the VPS, CPU-only. Loaded lazily on first request, kept warm
in memory thereafter. Two HuggingFace models share ~2.5 GB RAM:
- BGE-M3 (BAAI/bge-m3) — multilingual dense embedding, 1024-dim, 8k context
- BGE-Reranker-v2-M3 (BAAI/bge-reranker-v2-m3) — cross-encoder for reranking
Endpoints:
- POST /embed { texts: string[], normalize?: bool }
- POST /rerank { query: string, docs: string[] }
- GET /health
- GET /info
"""
from __future__ import annotations
import os
import time
from threading import Lock
from typing import List, Optional
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
EMBED_MODEL_NAME = os.getenv("EMBED_MODEL", "BAAI/bge-m3")
RERANK_MODEL_NAME = os.getenv("RERANK_MODEL", "BAAI/bge-reranker-v2-m3")
DEVICE = os.getenv("DEVICE", "cpu")
_embed_model = None
_rerank_model = None
_embed_lock = Lock()
_rerank_lock = Lock()
def get_embed_model():
global _embed_model
with _embed_lock:
if _embed_model is None:
from FlagEmbedding import BGEM3FlagModel
_embed_model = BGEM3FlagModel(EMBED_MODEL_NAME, use_fp16=False, device=DEVICE)
return _embed_model
def get_rerank_model():
global _rerank_model
with _rerank_lock:
if _rerank_model is None:
from FlagEmbedding import FlagReranker
_rerank_model = FlagReranker(RERANK_MODEL_NAME, use_fp16=False, device=DEVICE)
return _rerank_model
app = FastAPI(title="Disclosure Bureau Embed Service", version="0.1.0")
class EmbedRequest(BaseModel):
texts: List[str] = Field(..., min_items=1, max_items=512)
normalize: bool = True
class EmbedResponse(BaseModel):
model: str
dim: int
count: int
elapsed_ms: int
embeddings: List[List[float]]
class RerankRequest(BaseModel):
query: str
docs: List[str] = Field(..., min_items=1, max_items=200)
normalize: bool = True
class RerankResponse(BaseModel):
model: str
elapsed_ms: int
scores: List[float]
@app.get("/health")
def health():
return {
"status": "ok",
"embed_loaded": _embed_model is not None,
"rerank_loaded": _rerank_model is not None,
}
@app.get("/info")
def info():
return {
"embed_model": EMBED_MODEL_NAME,
"rerank_model": RERANK_MODEL_NAME,
"device": DEVICE,
"embed_dim": 1024,
}
@app.post("/embed", response_model=EmbedResponse)
def embed(req: EmbedRequest):
t0 = time.time()
try:
model = get_embed_model()
out = model.encode(
req.texts,
batch_size=min(len(req.texts), 16),
max_length=8192,
return_dense=True,
return_sparse=False,
return_colbert_vecs=False,
)
vectors = out["dense_vecs"]
if req.normalize:
import numpy as np
arr = np.asarray(vectors)
norms = np.linalg.norm(arr, axis=1, keepdims=True) + 1e-12
arr = arr / norms
vectors = arr
return EmbedResponse(
model=EMBED_MODEL_NAME,
dim=len(vectors[0]),
count=len(vectors),
elapsed_ms=int((time.time() - t0) * 1000),
embeddings=[list(map(float, v)) for v in vectors],
)
except Exception as e:
raise HTTPException(status_code=500, detail=f"embed failed: {e}")
@app.post("/rerank", response_model=RerankResponse)
def rerank(req: RerankRequest):
t0 = time.time()
try:
model = get_rerank_model()
pairs = [[req.query, d] for d in req.docs]
scores = model.compute_score(pairs, normalize=req.normalize)
if isinstance(scores, float):
scores = [scores]
return RerankResponse(
model=RERANK_MODEL_NAME,
elapsed_ms=int((time.time() - t0) * 1000),
scores=[float(s) for s in scores],
)
except Exception as e:
raise HTTPException(status_code=500, detail=f"rerank failed: {e}")

15
infra/embed-service/requirements.txt Normal file
View file

@ -0,0 +1,15 @@
fastapi==0.115.5
uvicorn[standard]==0.32.1
pydantic==2.10.3
FlagEmbedding==1.3.4
# Pinned EXACT versions known to work with FlagEmbedding 1.3.4
# (FlagEmbedding imports `is_torch_fx_available` and `GEMMA2_START_DOCSTRING`
# which were both removed in transformers 5.x and 4.47+)
transformers==4.46.3
tokenizers==0.20.3
huggingface_hub==0.26.5
peft==0.13.2
sentence-transformers==3.3.1
torch==2.5.1
numpy==1.26.4
accelerate==1.1.1

221
infra/supabase/migrations/0001_chat_schema.sql Normal file
View file

@ -0,0 +1,221 @@
-- The Disclosure Bureau — chat schema
-- Apply via Supabase Studio SQL editor OR psql on the production DB.
-- Safe to re-run (uses IF NOT EXISTS guards).
-- 1. profiles — 1:1 with auth.users, holds budget + role
CREATE TABLE IF NOT EXISTS public.profiles (
id UUID PRIMARY KEY REFERENCES auth.users(id) ON DELETE CASCADE,
display_name TEXT,
avatar_url TEXT,
role TEXT NOT NULL DEFAULT 'user' CHECK (role IN ('user','admin','suspended')),
budget_cap_usd NUMERIC(10,4) NOT NULL DEFAULT 5.0,
total_cost_usd NUMERIC(10,4) NOT NULL DEFAULT 0,
daily_quota INT NOT NULL DEFAULT 100,
daily_used INT NOT NULL DEFAULT 0,
quota_reset_at TIMESTAMPTZ NOT NULL DEFAULT (DATE_TRUNC('day', NOW()) + INTERVAL '1 day'),
preferred_locale TEXT DEFAULT 'pt-BR',
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Auto-create profile on user signup
CREATE OR REPLACE FUNCTION public.handle_new_user()
RETURNS TRIGGER LANGUAGE plpgsql SECURITY DEFINER SET search_path = public AS $$
BEGIN
INSERT INTO public.profiles (id, display_name)
VALUES (NEW.id, COALESCE(NEW.raw_user_meta_data->>'full_name', NEW.email));
RETURN NEW;
END;
$$;
DROP TRIGGER IF EXISTS on_auth_user_created ON auth.users;
CREATE TRIGGER on_auth_user_created
AFTER INSERT ON auth.users
FOR EACH ROW EXECUTE FUNCTION public.handle_new_user();
-- 2. chat_sessions — one conversation per row
CREATE TABLE IF NOT EXISTS public.chat_sessions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
title TEXT,
summary TEXT,
context_doc_id TEXT,
context_page_id TEXT,
is_public BOOLEAN NOT NULL DEFAULT FALSE,
share_token TEXT UNIQUE,
archived BOOLEAN NOT NULL DEFAULT FALSE,
message_count INT NOT NULL DEFAULT 0,
total_tokens INT NOT NULL DEFAULT 0,
total_cost_usd NUMERIC(10,4) NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_sessions_user_updated
ON public.chat_sessions(user_id, updated_at DESC) WHERE NOT archived;
CREATE INDEX IF NOT EXISTS idx_sessions_share
ON public.chat_sessions(share_token) WHERE share_token IS NOT NULL;
-- 3. messages — one row per turn
CREATE TABLE IF NOT EXISTS public.messages (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
session_id UUID NOT NULL REFERENCES public.chat_sessions(id) ON DELETE CASCADE,
role TEXT NOT NULL CHECK (role IN ('user','assistant','tool','system')),
content TEXT NOT NULL,
tool_calls JSONB,
tool_results JSONB,
citations JSONB,
model TEXT,
tokens_in INT,
tokens_out INT,
cost_usd NUMERIC(10,6),
duration_ms INT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_messages_session
ON public.messages(session_id, created_at);
-- 4. usage_events — audit log of every billable action (optional but cheap)
CREATE TABLE IF NOT EXISTS public.usage_events (
id BIGSERIAL PRIMARY KEY,
user_id UUID NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
session_id UUID REFERENCES public.chat_sessions(id) ON DELETE SET NULL,
event_type TEXT NOT NULL, -- 'message','tool_call','enrichment',...
cost_usd NUMERIC(10,6) NOT NULL DEFAULT 0,
metadata JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_usage_user_created
ON public.usage_events(user_id, created_at DESC);
-- 5. updated_at trigger
CREATE OR REPLACE FUNCTION public.touch_updated_at()
RETURNS TRIGGER LANGUAGE plpgsql AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;
$$;
DROP TRIGGER IF EXISTS profiles_touch ON public.profiles;
CREATE TRIGGER profiles_touch BEFORE UPDATE ON public.profiles
FOR EACH ROW EXECUTE FUNCTION public.touch_updated_at();
DROP TRIGGER IF EXISTS sessions_touch ON public.chat_sessions;
CREATE TRIGGER sessions_touch BEFORE UPDATE ON public.chat_sessions
FOR EACH ROW EXECUTE FUNCTION public.touch_updated_at();
-- 6. message-count + cost rollup trigger
CREATE OR REPLACE FUNCTION public.rollup_session_stats()
RETURNS TRIGGER LANGUAGE plpgsql AS $$
BEGIN
UPDATE public.chat_sessions
SET
message_count = message_count + 1,
total_tokens = total_tokens + COALESCE(NEW.tokens_in, 0) + COALESCE(NEW.tokens_out, 0),
total_cost_usd = total_cost_usd + COALESCE(NEW.cost_usd, 0),
updated_at = NOW()
WHERE id = NEW.session_id;
-- Also bump the user's total
UPDATE public.profiles p
SET total_cost_usd = total_cost_usd + COALESCE(NEW.cost_usd, 0),
daily_used = daily_used + 1
FROM public.chat_sessions s
WHERE s.id = NEW.session_id AND p.id = s.user_id;
RETURN NEW;
END;
$$;
DROP TRIGGER IF EXISTS messages_rollup ON public.messages;
CREATE TRIGGER messages_rollup AFTER INSERT ON public.messages
FOR EACH ROW EXECUTE FUNCTION public.rollup_session_stats();
-- 7. ROW LEVEL SECURITY — defense in depth
ALTER TABLE public.profiles ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.chat_sessions ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.messages ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.usage_events ENABLE ROW LEVEL SECURITY;
-- profiles: user sees/updates only their own
DROP POLICY IF EXISTS "own_profile_select" ON public.profiles;
CREATE POLICY "own_profile_select" ON public.profiles
FOR SELECT USING (auth.uid() = id);
DROP POLICY IF EXISTS "own_profile_update" ON public.profiles;
CREATE POLICY "own_profile_update" ON public.profiles
FOR UPDATE USING (auth.uid() = id);
-- chat_sessions: user's own + anyone can read public-shared ones
DROP POLICY IF EXISTS "own_sessions_select" ON public.chat_sessions;
CREATE POLICY "own_sessions_select" ON public.chat_sessions
FOR SELECT USING (auth.uid() = user_id OR is_public = TRUE);
DROP POLICY IF EXISTS "own_sessions_modify" ON public.chat_sessions;
CREATE POLICY "own_sessions_modify" ON public.chat_sessions
FOR ALL USING (auth.uid() = user_id) WITH CHECK (auth.uid() = user_id);
-- messages: only owners of the parent session (or public if session is_public)
DROP POLICY IF EXISTS "session_messages_select" ON public.messages;
CREATE POLICY "session_messages_select" ON public.messages
FOR SELECT USING (
EXISTS (SELECT 1 FROM public.chat_sessions s
WHERE s.id = messages.session_id
AND (s.user_id = auth.uid() OR s.is_public = TRUE))
);
DROP POLICY IF EXISTS "session_messages_insert" ON public.messages;
CREATE POLICY "session_messages_insert" ON public.messages
FOR INSERT WITH CHECK (
EXISTS (SELECT 1 FROM public.chat_sessions s
WHERE s.id = messages.session_id AND s.user_id = auth.uid())
);
DROP POLICY IF EXISTS "session_messages_delete" ON public.messages;
CREATE POLICY "session_messages_delete" ON public.messages
FOR DELETE USING (
EXISTS (SELECT 1 FROM public.chat_sessions s
WHERE s.id = messages.session_id AND s.user_id = auth.uid())
);
-- usage_events: insert by service role only; user can read their own
DROP POLICY IF EXISTS "own_usage_select" ON public.usage_events;
CREATE POLICY "own_usage_select" ON public.usage_events
FOR SELECT USING (auth.uid() = user_id);
-- 8. Helper RPC: get_or_create_session
CREATE OR REPLACE FUNCTION public.get_or_create_session(
p_context_doc_id TEXT DEFAULT NULL,
p_context_page_id TEXT DEFAULT NULL,
p_title TEXT DEFAULT NULL
) RETURNS UUID LANGUAGE plpgsql SECURITY INVOKER SET search_path = public AS $$
DECLARE
new_id UUID;
BEGIN
IF auth.uid() IS NULL THEN
RAISE EXCEPTION 'unauthenticated';
END IF;
new_id := gen_random_uuid();
INSERT INTO chat_sessions (id, user_id, title, context_doc_id, context_page_id)
VALUES (new_id, auth.uid(), p_title, p_context_doc_id, p_context_page_id);
RETURN new_id;
END;
$$;
GRANT EXECUTE ON FUNCTION public.get_or_create_session TO authenticated;
-- 9. Helper RPC: enforce budget cap
CREATE OR REPLACE FUNCTION public.check_budget(p_user_id UUID)
RETURNS BOOLEAN LANGUAGE plpgsql SECURITY DEFINER SET search_path = public AS $$
DECLARE
prof RECORD;
BEGIN
SELECT * INTO prof FROM profiles WHERE id = p_user_id;
IF prof.role = 'suspended' THEN RETURN FALSE; END IF;
IF prof.total_cost_usd >= prof.budget_cap_usd THEN RETURN FALSE; END IF;
-- Reset daily counter if past midnight
IF prof.quota_reset_at <= NOW() THEN
UPDATE profiles
SET daily_used = 0,
quota_reset_at = DATE_TRUNC('day', NOW()) + INTERVAL '1 day'
WHERE id = p_user_id;
RETURN TRUE;
END IF;
IF prof.daily_used >= prof.daily_quota THEN RETURN FALSE; END IF;
RETURN TRUE;
END;
$$;
GRANT EXECUTE ON FUNCTION public.check_budget TO authenticated;

253
infra/supabase/migrations/0002_chunks_retrieval.sql Normal file
View file

@ -0,0 +1,253 @@
-- The Disclosure Bureau — chunks retrieval schema (v0.2.0)
-- Enables hybrid retrieval (BM25 + dense embeddings + reranker) over the
-- agentic chunks produced by scripts/28-batch-rebuild-all.py.
--
-- Safe to re-run. Apply via Supabase Studio SQL editor OR psql.
-- 1. pgvector + trigram extensions (Supabase image ships both)
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS unaccent;
-- 2. Multilingual unaccent text search config (EN + PT-BR)
DO $$ BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_ts_config WHERE cfgname = 'pt_unaccent') THEN
CREATE TEXT SEARCH CONFIGURATION public.pt_unaccent ( COPY = pg_catalog.portuguese );
ALTER TEXT SEARCH CONFIGURATION public.pt_unaccent
ALTER MAPPING FOR hword, hword_part, word
WITH unaccent, portuguese_stem;
END IF;
IF NOT EXISTS (SELECT 1 FROM pg_ts_config WHERE cfgname = 'en_unaccent') THEN
CREATE TEXT SEARCH CONFIGURATION public.en_unaccent ( COPY = pg_catalog.english );
ALTER TEXT SEARCH CONFIGURATION public.en_unaccent
ALTER MAPPING FOR hword, hword_part, word
WITH unaccent, english_stem;
END IF;
END $$;
-- 3. documents — 1 row per doc (mirrors wiki/documents/<doc-id>.md frontmatter highlights)
CREATE TABLE IF NOT EXISTS public.documents (
doc_id TEXT PRIMARY KEY,
canonical_title TEXT,
collection TEXT,
document_class TEXT,
page_count INT,
classification TEXT,
content_class TEXT[],
schema_version TEXT NOT NULL DEFAULT '0.2.0',
build_approach TEXT,
build_model TEXT,
built_at TIMESTAMPTZ,
ingested_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
raw_path TEXT
);
CREATE INDEX IF NOT EXISTS documents_collection_idx ON public.documents (collection);
CREATE INDEX IF NOT EXISTS documents_built_at_idx ON public.documents (built_at DESC);
-- 4. chunks — the retrieval unit. 1 row per chunk file (raw/<doc>/chunks/c*.md).
-- 1024 dims = BGE-M3 dense.
CREATE TABLE IF NOT EXISTS public.chunks (
chunk_pk BIGSERIAL PRIMARY KEY,
doc_id TEXT NOT NULL REFERENCES public.documents(doc_id) ON DELETE CASCADE,
chunk_id TEXT NOT NULL,
page INT NOT NULL,
order_in_page INT NOT NULL,
order_global INT NOT NULL,
type TEXT NOT NULL,
bbox JSONB,
content_en TEXT,
content_pt TEXT,
ocr_confidence REAL,
classification TEXT,
formatting TEXT[],
cross_page_hint TEXT,
prev_chunk TEXT,
next_chunk TEXT,
related_image TEXT,
related_table TEXT,
redaction_code TEXT,
redaction_inferred TEXT,
image_type TEXT,
ufo_anomaly BOOLEAN NOT NULL DEFAULT FALSE,
ufo_anomaly_type TEXT,
ufo_rationale TEXT,
cryptid_anomaly BOOLEAN NOT NULL DEFAULT FALSE,
cryptid_anomaly_type TEXT,
cryptid_rationale TEXT,
image_desc_en TEXT,
image_desc_pt TEXT,
source_png TEXT,
embedding vector(1024),
ts_en tsvector GENERATED ALWAYS AS (
to_tsvector('public.en_unaccent', COALESCE(content_en, ''))
) STORED,
ts_pt tsvector GENERATED ALWAYS AS (
to_tsvector('public.pt_unaccent', COALESCE(content_pt, ''))
) STORED,
CONSTRAINT chunks_doc_chunk_uk UNIQUE (doc_id, chunk_id)
);
-- 5. indexes
CREATE INDEX IF NOT EXISTS chunks_doc_id_page_idx
ON public.chunks (doc_id, page, order_in_page);
CREATE INDEX IF NOT EXISTS chunks_type_idx
ON public.chunks (type);
CREATE INDEX IF NOT EXISTS chunks_classification_idx
ON public.chunks (classification) WHERE classification IS NOT NULL;
CREATE INDEX IF NOT EXISTS chunks_ufo_idx
ON public.chunks (ufo_anomaly) WHERE ufo_anomaly = TRUE;
CREATE INDEX IF NOT EXISTS chunks_cryptid_idx
ON public.chunks (cryptid_anomaly) WHERE cryptid_anomaly = TRUE;
CREATE INDEX IF NOT EXISTS chunks_ts_en_idx ON public.chunks USING GIN (ts_en);
CREATE INDEX IF NOT EXISTS chunks_ts_pt_idx ON public.chunks USING GIN (ts_pt);
-- HNSW vector index — m=16, ef_construction=64 (defaults; tune later)
CREATE INDEX IF NOT EXISTS chunks_embedding_hnsw_idx
ON public.chunks USING hnsw (embedding vector_cosine_ops);
-- Trigram index on content for fuzzy ILIKE
CREATE INDEX IF NOT EXISTS chunks_content_en_trgm_idx
ON public.chunks USING GIN (content_en gin_trgm_ops);
CREATE INDEX IF NOT EXISTS chunks_content_pt_trgm_idx
ON public.chunks USING GIN (content_pt gin_trgm_ops);
-- 6. entities — flattened from wiki/entities/<class>/<id>.md (post-lint)
CREATE TABLE IF NOT EXISTS public.entities (
entity_pk BIGSERIAL PRIMARY KEY,
entity_class TEXT NOT NULL,
entity_id TEXT NOT NULL,
canonical_name TEXT NOT NULL,
aliases TEXT[],
embedding vector(1024),
total_mentions INT NOT NULL DEFAULT 0,
documents_count INT NOT NULL DEFAULT 0,
enrichment_status TEXT,
last_ingest TIMESTAMPTZ,
CONSTRAINT entities_uk UNIQUE (entity_class, entity_id)
);
CREATE INDEX IF NOT EXISTS entities_canonical_name_idx ON public.entities (canonical_name);
CREATE INDEX IF NOT EXISTS entities_aliases_idx ON public.entities USING GIN (aliases);
CREATE INDEX IF NOT EXISTS entities_embedding_hnsw_idx
ON public.entities USING hnsw (embedding vector_cosine_ops);
-- 7. entity_mentions — link table chunk ↔ entity (materialized from lint)
CREATE TABLE IF NOT EXISTS public.entity_mentions (
mention_pk BIGSERIAL PRIMARY KEY,
chunk_pk BIGINT NOT NULL REFERENCES public.chunks(chunk_pk) ON DELETE CASCADE,
entity_pk BIGINT NOT NULL REFERENCES public.entities(entity_pk) ON DELETE CASCADE,
surface_form TEXT,
CONSTRAINT entity_mentions_uk UNIQUE (chunk_pk, entity_pk)
);
CREATE INDEX IF NOT EXISTS entity_mentions_chunk_idx ON public.entity_mentions (chunk_pk);
CREATE INDEX IF NOT EXISTS entity_mentions_entity_idx ON public.entity_mentions (entity_pk);
-- 8. Hybrid search RPC: BM25 + dense + RRF fusion server-side
CREATE OR REPLACE FUNCTION public.hybrid_search_chunks(
q_text TEXT,
q_embedding vector(1024),
q_lang TEXT DEFAULT 'pt', -- 'pt' | 'en'
q_doc_id TEXT DEFAULT NULL,
q_type TEXT DEFAULT NULL,
q_classification TEXT DEFAULT NULL,
q_ufo_only BOOLEAN DEFAULT FALSE,
k INT DEFAULT 100,
rrf_k INT DEFAULT 60
)
RETURNS TABLE (
chunk_pk BIGINT,
doc_id TEXT,
chunk_id TEXT,
page INT,
type TEXT,
bbox JSONB,
content_en TEXT,
content_pt TEXT,
classification TEXT,
score DOUBLE PRECISION,
bm25_rank INT,
dense_rank INT
)
LANGUAGE plpgsql STABLE AS $$
BEGIN
RETURN QUERY
WITH
ts_q AS (
SELECT CASE WHEN q_lang = 'en'
THEN websearch_to_tsquery('public.en_unaccent'::regconfig, q_text)
ELSE websearch_to_tsquery('public.pt_unaccent'::regconfig, q_text)
END AS q
),
bm25 AS (
SELECT c.chunk_pk,
row_number() OVER (ORDER BY
ts_rank_cd(
CASE WHEN q_lang = 'en' THEN c.ts_en ELSE c.ts_pt END,
(SELECT q FROM ts_q)
) DESC NULLS LAST
)::INT AS r
FROM public.chunks c
WHERE (CASE WHEN q_lang = 'en' THEN c.ts_en ELSE c.ts_pt END) @@ (SELECT q FROM ts_q)
AND (q_doc_id IS NULL OR c.doc_id = q_doc_id)
AND (q_type IS NULL OR c.type = q_type)
AND (q_classification IS NULL OR c.classification = q_classification)
AND (NOT q_ufo_only OR c.ufo_anomaly = TRUE)
LIMIT k
),
dense AS (
SELECT c.chunk_pk,
row_number() OVER (ORDER BY c.embedding <=> q_embedding)::INT AS r
FROM public.chunks c
WHERE c.embedding IS NOT NULL
AND (q_doc_id IS NULL OR c.doc_id = q_doc_id)
AND (q_type IS NULL OR c.type = q_type)
AND (q_classification IS NULL OR c.classification = q_classification)
AND (NOT q_ufo_only OR c.ufo_anomaly = TRUE)
ORDER BY c.embedding <=> q_embedding
LIMIT k
),
fused AS (
SELECT COALESCE(b.chunk_pk, d.chunk_pk) AS chunk_pk,
((1.0::DOUBLE PRECISION / (rrf_k + COALESCE(b.r, k + 1))::DOUBLE PRECISION) +
(1.0::DOUBLE PRECISION / (rrf_k + COALESCE(d.r, k + 1))::DOUBLE PRECISION)) AS score,
b.r AS bm25_rank,
d.r AS dense_rank
FROM bm25 b
FULL OUTER JOIN dense d USING (chunk_pk)
)
SELECT c.chunk_pk, c.doc_id, c.chunk_id, c.page, c.type, c.bbox,
c.content_en, c.content_pt, c.classification,
f.score, f.bm25_rank, f.dense_rank
FROM fused f
JOIN public.chunks c USING (chunk_pk)
ORDER BY f.score DESC
LIMIT k;
END
$$;
-- 9. RLS — chunks/entities are public read; writes via service_role
ALTER TABLE public.documents ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.chunks ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.entities ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.entity_mentions ENABLE ROW LEVEL SECURITY;
DROP POLICY IF EXISTS documents_read ON public.documents;
DROP POLICY IF EXISTS chunks_read ON public.chunks;
DROP POLICY IF EXISTS entities_read ON public.entities;
DROP POLICY IF EXISTS entity_mentions_read ON public.entity_mentions;
CREATE POLICY documents_read ON public.documents FOR SELECT USING (TRUE);
CREATE POLICY chunks_read ON public.chunks FOR SELECT USING (TRUE);
CREATE POLICY entities_read ON public.entities FOR SELECT USING (TRUE);
CREATE POLICY entity_mentions_read ON public.entity_mentions FOR SELECT USING (TRUE);
GRANT SELECT ON public.documents, public.chunks, public.entities, public.entity_mentions
TO anon, authenticated;
GRANT EXECUTE ON FUNCTION public.hybrid_search_chunks TO anon, authenticated;

14
infra/supabase/seed.sql Normal file
View file

@ -0,0 +1,14 @@
-- One-time seed — promote your account to admin.
-- Run AFTER first signing in via magic-link at https://disclosure.top.
-- Edit the email below before running.
UPDATE public.profiles
SET role = 'admin',
budget_cap_usd = 9999.0,
daily_quota = 10000
WHERE id = (SELECT id FROM auth.users WHERE email = 'YOUR-EMAIL@example.com');
-- Verify
SELECT id, display_name, role, budget_cap_usd, daily_quota
FROM public.profiles
WHERE role = 'admin';

181
scripts/00-extract-war-gov.js Normal file
View file

@ -0,0 +1,181 @@
/**
* 00-extract-war-gov.js — Console-based extractor for war.gov/UFO/Release-NN/
*
* Works on any release page (Release-01, Release-02, etc.) because it derives
* everything from the DOM, not from a hardcoded release number.
*
* USAGE (Chrome on https://www.war.gov/UFO/Release-NN/):
* 1. Wait for the page to load — scroll to the bottom to trigger lazy-load
* if there are images you haven't scrolled past
* 2. Open DevTools (Cmd+Option+I / F12) → Console
* 3. Paste this ENTIRE file. Press Enter.
* 4. Wait ~3-5 minutes (158 docs × ~2s each click+wait).
* 5. The JSON is opened in a new tab AND copied to clipboard.
* Save it under /Users/guto/ufo/processing/war-gov-metadata/
* as all-documents-release-NN.json (or paste it back to me).
*
* What it captures per document:
* - record_id (record-001..record-NNN — internal id)
* - title (as printed in the modal heading; case correct)
* - agency
* - release_date, incident_date, incident_location, document_type
* - description (the unique paragraph shown in the detail overlay)
* - thumbnail_url (Akamai-hosted JPG preview)
* - pdf_url_inferred (replaces "/thumbnail/" with "/" and ".jpg" with the
* proper extension based on document_type)
*
* The script is READ-ONLY — it never submits, never modifies the page beyond
* opening and closing the detail modal.
*/
(async function extractWarGovFull() {
const sleep = ms => new Promise(r => setTimeout(r, ms));
function pollUntil(predicate, opts = {}) {
const timeout = opts.timeout ?? 3000;
const interval = opts.interval ?? 50;
return new Promise((resolve, reject) => {
const start = Date.now();
const tick = () => {
const v = predicate();
if (v) return resolve(v);
if (Date.now() - start > timeout) return reject(new Error("timeout"));
setTimeout(tick, interval);
};
tick();
});
}
function strip(s) { return (s || "").replace(/^\s*\[(.*)\]\s*$/, "$1").trim(); }
function parseModal() {
const modal = document.querySelector('.record-modal-shell, [data-record-modal-shell]');
if (!modal) return null;
const out = {};
const titleEl = modal.querySelector('[data-record-modal-title], #record-modal-title');
out.title = titleEl ? titleEl.innerText.trim() : null;
const agencyEl = modal.querySelector('[data-record-modal-agency]');
out.agency = strip(agencyEl?.innerText);
const descEl = modal.querySelector('[data-record-modal-copy], .record-modal-copy');
out.description = descEl ? descEl.innerText.trim() : null;
// dl facts
modal.querySelectorAll('.record-modal-fact').forEach(fact => {
const dt = fact.querySelector('dt');
const dd = fact.querySelector('dd');
if (dt && dd) {
const key = dt.innerText.trim().toLowerCase().replace(/[^a-z0-9]+/g, "_").replace(/^_|_$/g, "");
out[key] = strip(dd.innerText);
}
});
// Thumbnail
const img = modal.querySelector('#record-main-image, img');
if (img && img.src) {
out.thumbnail_url = img.src;
// Infer PDF/asset url: drop "/thumbnail" segment, restore extension based on document_type
const ext = (out.document_type || ".pdf").toLowerCase().replace(/^\[?\.?/, ".").replace(/\]$/, "");
out.pdf_url_inferred = img.src.replace("/thumbnail/", "/").replace(/\.jpg$/i, ext);
}
// Record kind from modal data attr (pdf|vid|img)
out.record_kind = modal.getAttribute("data-record-kind") || null;
return out;
}
async function clickRowAndCapture(row) {
const recordId = row.dataset.recordId || row.getAttribute("data-record-id");
// open modal
row.click();
let modalData = null;
try {
await pollUntil(() => {
const m = document.querySelector('.record-modal-shell');
if (!m) return null;
// Wait until the title matches the row's title (modal can be stale from previous open)
const t = m.querySelector('[data-record-modal-title]')?.innerText?.trim();
const expected = row.querySelector('.record-title')?.innerText?.trim();
if (t && expected && t.toLowerCase() === expected.toLowerCase()) return m;
// Or just any visible modal after some delay
return null;
}, { timeout: 2500 });
modalData = parseModal();
} catch (e) {
console.warn(` ${recordId}: modal did not load for "${row.querySelector('.record-title')?.innerText}"`);
}
// close modal
const closeBtn = document.querySelector('.record-modal-close, [data-record-modal-close]');
if (closeBtn) closeBtn.click();
else {
// Press Escape
document.dispatchEvent(new KeyboardEvent("keydown", { key: "Escape" }));
}
await pollUntil(() => !document.querySelector('.record-modal-shell'), { timeout: 1500 }).catch(() => {});
return { record_id: recordId, ...(modalData || {}) };
}
async function captureCurrentPage() {
const rows = Array.from(document.querySelectorAll('button.record-row'));
const out = [];
for (const row of rows) {
const record = await clickRowAndCapture(row);
out.push(record);
await sleep(120); // small breather between cards
}
return out;
}
const findNext = () => document.querySelector('button.pagination-next');
const release = (location.pathname.match(/Release-(\d+)/i) || [, "01"])[1].padStart(2, "0");
console.log(`[extract] starting on Release-${release}`);
const all = [];
const seen = new Set();
const MAX_PAGES = 25;
let pageIdx = 0;
while (pageIdx < MAX_PAGES) {
pageIdx++;
const firstBefore = document.querySelector('button.record-row')?.dataset.recordId;
const t0 = performance.now();
const captured = await captureCurrentPage();
let added = 0;
for (const r of captured) {
const key = r.record_id || `${r.title}|${r.incident_date}`;
if (seen.has(key)) continue;
seen.add(key);
all.push(r);
added++;
}
console.log(`[extract] page ${pageIdx}: captured ${captured.length} (+${added} new, total ${all.length}, ${(performance.now()-t0|0)}ms)`);
const next = findNext();
if (!next || next.disabled || next.getAttribute("aria-disabled") === "true") {
console.log("[extract] NEXT disabled — last page reached");
break;
}
next.click();
// Wait until row content changes
await pollUntil(() => {
const f = document.querySelector('button.record-row')?.dataset.recordId;
return f && f !== firstBefore ? f : null;
}, { timeout: 3000 }).catch(() => {});
await sleep(200);
}
const result = {
extracted_at: new Date().toISOString(),
source_url: location.href,
release: `Release-${release}`,
total_documents: all.length,
pages_visited: pageIdx,
documents: all,
};
const jsonStr = JSON.stringify(result, null, 2);
console.log(`[extract] DONE — ${all.length} documents extracted across ${pageIdx} pages`);
console.log(`[extract] full-metadata count: ${all.filter(d => d.description && d.asset_file_name).length}`);
try { await navigator.clipboard.writeText(jsonStr); console.log("[extract] ✓ JSON copied to clipboard"); }
catch (e) { console.warn("[extract] clipboard failed (focus the tab and re-run if needed):", e.message); }
const blob = new Blob([jsonStr], { type: "application/json" });
window.open(URL.createObjectURL(blob), "_blank");
console.log("[extract] ✓ JSON opened in new tab — save with Cmd+S");
console.log("[extract] sample doc:", all[0]);
return result;
})();

250
scripts/00b-coverage-report.py Executable file
View file

@ -0,0 +1,250 @@
#!/usr/bin/env python3
"""
00b-coverage-report.py — Compare /Users/guto/ufo/raw/ against the war.gov
metadata JSON extracted by 00-extract-war-gov.js. Reports:
- documents in war.gov that ARE present in raw/
- documents in war.gov that are MISSING from raw/ (need to be downloaded)
- files in raw/ that DO NOT appear in war.gov (manual additions / older releases / renamed)
Matching is lenient: both sides normalize to ASCII-folded lowercase kebab-case,
with extra noise stripped. We try filename match first, then title match.
Usage:
./00b-coverage-report.py # uses release-01 by default
./00b-coverage-report.py --json <path> # custom JSON path
./00b-coverage-report.py --json <path> --out <path> # custom output report path
"""
from __future__ import annotations
import argparse
import json
import re
import sys
import unicodedata
from pathlib import Path
UFO_ROOT = Path("/Users/guto/ufo")
RAW = UFO_ROOT / "raw"
DEFAULT_JSON = UFO_ROOT / "processing" / "war-gov-metadata" / "all-documents-release-01-basic.json"
DEFAULT_OUT = UFO_ROOT / "processing" / "war-gov-metadata" / "coverage-report.md"
def normalize(s: str) -> str:
"""ASCII fold + lowercase + collapse non-alnum to hyphens + strip zero-padding."""
if not s:
return ""
nfkd = unicodedata.normalize("NFKD", s)
ascii_s = "".join(c for c in nfkd if not unicodedata.combining(c))
lower = ascii_s.lower()
lower = lower.replace("'", "").replace(",", "-").replace("[", "").replace("]", "")
replaced = re.sub(r"[^a-z0-9]+", "-", lower)
norm = re.sub(r"-+", "-", replaced).strip("-")
# Strip zero-padding inside "letter+digits" tokens: d074 → d74, b001 → b1, section-001 → section-1
# Apply repeatedly because regex doesn't recurse.
prev = None
while prev != norm:
prev = norm
norm = re.sub(r"(?<=[a-z-])0+(\d)", r"\1", norm)
return norm
def main():
ap = argparse.ArgumentParser(description="Compare raw/ vs war.gov metadata JSON.")
ap.add_argument("--json", default=str(DEFAULT_JSON), help="path to war-gov metadata JSON")
ap.add_argument("--out", default=str(DEFAULT_OUT), help="output report path (markdown)")
args = ap.parse_args()
json_path = Path(args.json)
if not json_path.exists():
sys.stderr.write(f"JSON not found: {json_path}\n")
sys.exit(1)
data = json.loads(json_path.read_text(encoding="utf-8"))
war_docs = data.get("documents", [])
print(f"war.gov JSON: {json_path.name} — {len(war_docs)} docs")
# Build raw inventory by normalized basename (no extension)
raw_files = sorted(p for p in RAW.iterdir() if p.is_file() and not p.name.startswith("."))
raw_norm_to_path: dict[str, Path] = {}
for p in raw_files:
stem = p.stem
raw_norm_to_path[normalize(stem)] = p
print(f"raw/: {len(raw_files)} files")
print()
# Common noise tokens that hurt Jaccard accuracy
COMMON = {
"mission", "report", "uap", "the", "of", "and", "a", "in", "on", "for",
"with", "to", "from", "department", "war", "fbi", "nasa", "state",
"unresolved", "debrief", "summary", "transcript", "crew", "general",
"vol", "incident", "summaries", "photo", "video", "cable", "email",
"correspondence", "correspondance", "launch", "range", "fouler",
"force", "air", "navy", "between", "or", "year", "month",
"january", "february", "march", "april", "may", "june", "july",
"august", "september", "october", "november", "december", "redacted",
"sub", "sighting", "about", "kuwait", "kazakhstan", "papua", "guinea",
"syria", "iraq", "iran", "yemen", "djibouti", "japan", "greece",
"mexico", "germany", "turkey", "turkmenistan", "georgia", "tbilisi",
"indopacom", "middle", "east", "africa", "europe", "western", "united",
"states", "north", "south", "america",
}
def signature_tokens(s: str) -> set[str]:
return {t for t in normalize(s).split("-") if t and t not in COMMON}
def jaccard(a: set, b: set) -> float:
if not a or not b:
return 0.0
return len(a & b) / len(a | b)
def primary_id(s: str) -> str | None:
"""Extract a stable prefix identifier from titles/filenames.
Examples:
'DOW-UAP-D074, MISSION REPORT, ...' → 'dow-uap-d74'
'DOW-UAP-D57-Mission-Report-Gulf-of-Aden-September-2020' → 'dow-uap-d57'
'NASA-UAP-D003, GEMINI 7 TRANSCRIPT, 1965' → 'nasa-uap-d3'
'FBI PHOTO B001' → 'fbi-photo-b1'
Returns None if no ID prefix found.
"""
n = normalize(s)
patterns = [
r"^(dow-uap-[a-z]{1,4}\d+)",
r"^(dos-uap-d\d+)",
r"^(nasa-uap-[a-z]{1,3}\d+[a-z]?)",
r"^(fbi-photo-[a-z]\d+)",
]
for p in patterns:
m = re.match(p, n)
if m:
return m.group(1)
return None
raw_tokens_index = [(p, signature_tokens(p.stem), normalize(p.stem), primary_id(p.stem)) for p in raw_files]
present: list[tuple[dict, Path, str, float]] = []
missing: list[dict] = []
matched_raw_paths: set[Path] = set()
for doc in war_docs:
title = doc.get("title", "")
norm_title = normalize(title)
sig_war = signature_tokens(title)
asset = doc.get("asset_file_name") or ""
match = None
reason = ""
score = 1.0
war_pid = primary_id(title)
# Tier 1: direct normalized match
for p, _sig, raw_norm, _pid in raw_tokens_index:
if raw_norm == norm_title or (asset and raw_norm == normalize(asset)):
match = p; reason = "exact-norm"; break
# Tier 2: primary-id match (DOW-UAP-D74, etc.) — strongest semantic anchor
if not match and war_pid:
for p, _sig, _raw_norm, raw_pid in raw_tokens_index:
if p in matched_raw_paths:
continue
if raw_pid and raw_pid == war_pid:
match = p; reason = f"primary-id={war_pid}"; break
# Tier 3: containment (one inside the other) — high specificity
if not match:
for p, _sig, raw_norm, _pid in raw_tokens_index:
if p in matched_raw_paths:
continue
if len(norm_title) >= 12 and len(raw_norm) >= 12 and (
norm_title in raw_norm or raw_norm in norm_title
):
match = p; reason = "containment"; break
# Tier 4: signature-token Jaccard with threshold
if not match and sig_war:
best = None
best_score = 0.0
for p, sig_raw, _raw_norm, _pid in raw_tokens_index:
if p in matched_raw_paths:
continue
j = jaccard(sig_war, sig_raw)
if j > best_score:
best_score = j
best = p
if best is not None and best_score >= 0.50:
match = best; reason = f"jaccard={best_score:.2f}"; score = best_score
if match:
present.append((doc, match, reason, score))
matched_raw_paths.add(match)
else:
missing.append(doc)
# raw files NOT mentioned in war.gov
orphan_raw = [p for p in raw_files if p not in matched_raw_paths]
# Summary
print(f"{'='*60}")
print(f"Present in raw/: {len(present)} / {len(war_docs)}")
print(f"Missing from raw/: {len(missing)}")
print(f"Orphan files in raw/ (not in war.gov metadata): {len(orphan_raw)}")
print(f"{'='*60}")
# Build report
lines: list[str] = []
lines.append("# Coverage Report — war.gov/UFO vs /Users/guto/ufo/raw/")
lines.append("")
lines.append(f"- Source JSON: `{json_path}`")
lines.append(f"- raw/ inventory: {len(raw_files)} files")
lines.append(f"- war.gov inventory: {len(war_docs)} documents")
lines.append(f"- **Present**: {len(present)}")
lines.append(f"- **Missing**: {len(missing)} (need to be downloaded)")
lines.append(f"- **Orphan in raw/**: {len(orphan_raw)} (not in war.gov metadata)")
lines.append("")
lines.append("## Missing from raw/ (must be downloaded)")
lines.append("")
if missing:
lines.append("| record_id | title | agency | document_type | pdf_url_inferred |")
lines.append("|---|---|---|---|---|")
for d in missing:
url = d.get("pdf_url_inferred") or d.get("pdf_url") or ""
lines.append(
f"| {d.get('record_id','')} "
f"| {d.get('title','')} "
f"| {d.get('agency','')} "
f"| {d.get('document_type','')} "
f"| {url} |"
)
else:
lines.append("_(none)_")
lines.append("")
lines.append("## Present in raw/ (no action needed)")
lines.append("")
if present:
lines.append("| record_id | title | matched raw/ file | match reason |")
lines.append("|---|---|---|---|")
for d, p, reason, _score in present:
lines.append(f"| {d.get('record_id','')} | {d.get('title','')} | `{p.name}` | {reason} |")
else:
lines.append("_(none)_")
lines.append("")
lines.append("## Orphan files in raw/ (likely older releases or manual additions)")
lines.append("")
if orphan_raw:
for p in orphan_raw:
lines.append(f"- `{p.name}`")
else:
lines.append("_(none)_")
lines.append("")
out_path = Path(args.out)
out_path.parent.mkdir(parents=True, exist_ok=True)
out_path.write_text("\n".join(lines), encoding="utf-8")
print(f"\nReport written: {out_path}")
if __name__ == "__main__":
main()

164
scripts/00c-download-missing.js Normal file
View file

@ -0,0 +1,164 @@
/**
* 00c-download-missing.js — Programmatic download (fetch+blob) via the war.gov UI
*
* Improved over the previous version:
* - Uses fetch() to grab the asset from the same origin (browser cookies +
* same-origin policy → Akamai accepts).
* - Creates a Blob and triggers `<a download="filename">` to control the
* filename exactly (no "(1)" duplicates).
* - Reads the PDF URL from the modal's download button data and/or thumbnail
* src pattern (`/thumbnail/foo.jpg` → `/foo.pdf|.mp4|.jpg`).
* - Skips records whose file already exists (best-effort by checking the
* expected filename — you can also clear the lists below).
*
* USAGE (Chrome on https://www.war.gov/UFO/Release-NN/):
* 1. Set Chrome download folder to /Users/guto/ufo/raw/ (Settings → Downloads).
* For videos, the script will rename to land in /Users/guto/ufo/raw/videos/
* after — just MOVE them manually after this finishes.
* 2. Open DevTools → Console.
* 3. Paste this whole file. Press Enter.
* 4. Chrome prompts "Allow multiple downloads" → click **Allow**.
* 5. Wait ~30s (1s between downloads). Files land in Downloads folder.
*/
(async function downloadMissing() {
// ============================================================
// PDFs still missing (Release-01, verified 2026-05-13).
// The 28 .VID videos are already in /Users/guto/ufo/raw/videos/ from
// a prior bulk download — they share file names like DOD_111688723.mp4.
// ============================================================
const TARGETS = [
"record-140", // NASA-UAP-D003 GEMINI 7 TRANSCRIPT 1965
"record-154", // STATE CABLE 003 TBILISI GEORGIA
"record-155", // STATE CABLE 004 ASHGABAT TURKMENISTAN
"record-156", // STATE CABLE 005 MEXICO
];
console.log(`[dl] ${TARGETS.length} records to download`);
// ----------------------------------------------------------------------
const sleep = ms => new Promise(r => setTimeout(r, ms));
function pollUntil(predicate, opts = {}) {
const timeout = opts.timeout ?? 4000;
const interval = opts.interval ?? 50;
return new Promise((resolve, reject) => {
const start = Date.now();
const tick = () => {
const v = predicate();
if (v) return resolve(v);
if (Date.now() - start > timeout) return reject(new Error("timeout"));
setTimeout(tick, interval);
};
tick();
});
}
async function findRowOnAllPages(recordId) {
for (let i = 0; i < 25; i++) {
const row = document.querySelector(`button.record-row[data-record-id="${recordId}"]`);
if (row) return row;
const next = document.querySelector("button.pagination-next");
if (!next || next.disabled || next.getAttribute("aria-disabled") === "true") return null;
const before = document.querySelector("button.record-row")?.dataset.recordId;
next.click();
try {
await pollUntil(() => {
const f = document.querySelector("button.record-row")?.dataset.recordId;
return f && f !== before ? f : null;
});
} catch { /* ignore */ }
await sleep(150);
}
return null;
}
async function goToFirstPage() {
const firstBtn = Array.from(document.querySelectorAll(".pagination-button")).find(b => b.innerText.trim() === "1");
if (firstBtn) { firstBtn.click(); await sleep(400); return; }
}
function buildFilenameFromThumb(thumbUrl, extHint) {
// .../thumbnail/foo.jpg → foo + ext
const m = thumbUrl.match(/\/thumbnail\/([^?#]+)\.[a-z]+$/i);
if (!m) return null;
const base = decodeURIComponent(m[1]);
return `${base}${extHint}`;
}
function buildAssetUrlFromThumb(thumbUrl, extHint) {
// strip "/thumbnail/" segment, swap extension
return thumbUrl.replace("/thumbnail/", "/").replace(/\.[a-z]+$/i, extHint);
}
async function downloadOne(recordId) {
const row = await findRowOnAllPages(recordId);
if (!row) { console.warn(` ✗ ${recordId}: row not found`); return false; }
row.click();
try {
await pollUntil(() => document.querySelector(".record-modal-shell"));
} catch {
console.warn(` ✗ ${recordId}: modal didn't open`);
return false;
}
const modal = document.querySelector(".record-modal-shell");
const kind = (modal.getAttribute("data-record-kind") || "pdf").toLowerCase();
const title = modal.querySelector("[data-record-modal-title]")?.innerText?.trim() || recordId;
const docTypeEl = Array.from(modal.querySelectorAll(".record-modal-fact dd"))
.find(d => d.previousElementSibling?.innerText?.trim() === "Document Type");
let ext = ".pdf";
if (docTypeEl) {
const raw = docTypeEl.innerText.trim().replace(/[\[\]]/g, "").toLowerCase();
ext = raw.startsWith(".") ? raw : "." + raw;
// Normalize uncommon: .vid → .mp4 (guess; site serves mp4 for videos), .img → .jpg
if (ext === ".vid") ext = ".mp4";
if (ext === ".img") ext = ".jpg";
}
const img = modal.querySelector("#record-main-image, img");
const thumb = img?.src;
if (!thumb) {
console.warn(` ✗ ${recordId}: no thumbnail src — cannot infer URL`);
const close = document.querySelector(".record-modal-close, [data-record-modal-close]");
if (close) close.click(); await sleep(300);
return false;
}
const assetUrl = buildAssetUrlFromThumb(thumb, ext);
const filename = buildFilenameFromThumb(thumb, ext) || `${recordId}${ext}`;
console.log(` ↓ ${recordId}: fetching "${filename}" from ${assetUrl}`);
try {
const res = await fetch(assetUrl, { credentials: "include", referrer: location.href });
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const blob = await res.blob();
const url = URL.createObjectURL(blob);
const a = document.createElement("a");
a.href = url;
a.download = filename;
document.body.appendChild(a);
a.click();
a.remove();
setTimeout(() => URL.revokeObjectURL(url), 5000);
console.log(` ✓ ${recordId}: ${filename} (${(blob.size/1024/1024).toFixed(2)} MB)`);
} catch (e) {
console.warn(` ✗ ${recordId}: fetch failed — ${e.message}`);
const close = document.querySelector(".record-modal-close, [data-record-modal-close]");
if (close) close.click();
return false;
}
const close = document.querySelector(".record-modal-close, [data-record-modal-close]");
if (close) close.click();
await sleep(800);
return true;
}
await goToFirstPage();
let ok = 0, fail = [];
for (const id of TARGETS) {
const success = await downloadOne(id);
if (success) ok++;
else fail.push(id);
await sleep(500);
}
console.log(`\n[dl] DONE — ok=${ok}, failed=${fail.length}`);
if (fail.length) console.log("failed:", fail);
console.log("Move videos from Downloads/ → /Users/guto/ufo/raw/videos/ when done.");
})();

206
scripts/01-convert-pdfs.sh Executable file
View file

@ -0,0 +1,206 @@
#!/usr/bin/env bash
# 01-convert-pdfs.sh — Fase 2: PDF → PNG (200 DPI) + OCR (pdftotext -layout)
#
# Uso:
# ./01-convert-pdfs.sh --doc-id <doc-id> # single doc
# ./01-convert-pdfs.sh --filename <filename> # single PDF by filename
# ./01-convert-pdfs.sh --all # all 115 PDFs in raw/
#
# Idempotente: pula PNGs/OCR já gerados. Re-roda --force para sobrescrever.
set -euo pipefail
UFO_ROOT="/Users/guto/ufo"
RAW_DIR="$UFO_ROOT/raw"
PNG_BASE="$UFO_ROOT/processing/png"
OCR_BASE="$UFO_ROOT/processing/ocr"
DPI=72 # LLM vision downscales internally; 72 DPI matches PDF point grid and is sufficient for OCR + vision
FORCE=0
TARGET_FILENAME=""
TARGET_DOC_ID=""
PROCESS_ALL=0
usage() {
cat <<EOF
Usage: $0 [--doc-id <doc-id>] [--filename <filename.pdf>] [--all] [--force]
Options:
--doc-id ID Process single PDF by doc_id (kebab-case)
--filename F Process single PDF by raw filename
--all Process every PDF in $RAW_DIR
--force Re-convert even if outputs exist
-h, --help Show this help
EOF
exit "${1:-0}"
}
# Canonicalize filename → doc_id (matches CLAUDE.md algorithm)
filename_to_doc_id() {
local fname="$1"
local base="${fname%.*}" # strip extension
# ASCII fold (best-effort via iconv) + lowercase + replace non-alnum with -
local id
id=$(printf '%s' "$base" \
| iconv -f UTF-8 -t ASCII//TRANSLIT 2>/dev/null \
| tr '[:upper:]' '[:lower:]' \
| sed -e 's/[^a-z0-9-]/-/g' -e 's/--*/-/g' -e 's/^-//' -e 's/-$//')
# Prefix with doc- if starts with digit
if [[ "$id" =~ ^[0-9] ]]; then
id="doc-$id"
fi
printf '%s' "$id"
}
# Reverse: doc_id → filename (scan raw/ for match)
doc_id_to_filename() {
local target_id="$1"
for f in "$RAW_DIR"/*.pdf; do
[[ -f "$f" ]] || continue
local fname
fname=$(basename "$f")
local id
id=$(filename_to_doc_id "$fname")
if [[ "$id" == "$target_id" ]]; then
printf '%s' "$fname"
return 0
fi
done
return 1
}
convert_one_pdf() {
local pdf_path="$1"
local fname
fname=$(basename "$pdf_path")
local doc_id
doc_id=$(filename_to_doc_id "$fname")
local png_dir="$PNG_BASE/$doc_id"
local ocr_dir="$OCR_BASE/$doc_id"
mkdir -p "$png_dir" "$ocr_dir"
# Get page count
local page_count
page_count=$(pdfinfo "$pdf_path" 2>/dev/null | awk -F': +' '/^Pages/ {print $2}')
if [[ -z "$page_count" ]]; then
printf ' [skip] %s — could not read pdfinfo\n' "$fname" >&2
return 1
fi
printf '\n=== %s (%d pages) → %s ===\n' "$fname" "$page_count" "$doc_id"
# PNG generation (pdftoppm produces p-001.png, p-002.png, ...)
local need_png=1
if [[ $FORCE -eq 0 ]]; then
# Check if last expected PNG exists
local last_page_num
last_page_num=$(printf '%03d' "$page_count")
if [[ -f "$png_dir/p-$last_page_num.png" ]]; then
need_png=0
printf ' PNG: skip (already generated)\n'
fi
fi
if [[ $need_png -eq 1 ]]; then
printf ' PNG: pdftoppm @ %d DPI...\n' "$DPI"
pdftoppm -r "$DPI" -png "$pdf_path" "$png_dir/p"
# pdftoppm names files like p-1.png for pages 1-9, p-10.png for 10+
# Normalize to zero-padded p-001.png
for f in "$png_dir"/p-*.png; do
[[ -f "$f" ]] || continue
local bn
bn=$(basename "$f")
# Extract number, zero-pad to 3 digits
local num
num=$(printf '%s' "$bn" | sed -E 's/^p-([0-9]+)\.png$/\1/')
if [[ "$num" =~ ^[0-9]+$ ]]; then
local padded
padded=$(printf '%03d' "$num")
local new_name="p-$padded.png"
if [[ "$bn" != "$new_name" ]]; then
mv "$f" "$png_dir/$new_name"
fi
fi
done
printf ' PNG: done\n'
fi
# OCR per page (pdftotext -f N -l N -layout)
local need_ocr=1
if [[ $FORCE -eq 0 ]]; then
local last_page_num
last_page_num=$(printf '%03d' "$page_count")
if [[ -f "$ocr_dir/p-$last_page_num.txt" ]]; then
need_ocr=0
printf ' OCR: skip (already generated)\n'
fi
fi
if [[ $need_ocr -eq 1 ]]; then
printf ' OCR: pdftotext -layout per page...\n'
for (( p=1; p<=page_count; p++ )); do
local padded
padded=$(printf '%03d' "$p")
pdftotext -f "$p" -l "$p" -layout "$pdf_path" "$ocr_dir/p-$padded.txt" 2>/dev/null || true
done
printf ' OCR: done\n'
fi
printf ' ✓ %s\n' "$doc_id"
}
# Parse args
while [[ $# -gt 0 ]]; do
case "$1" in
--doc-id)
TARGET_DOC_ID="$2"
shift 2
;;
--filename)
TARGET_FILENAME="$2"
shift 2
;;
--all)
PROCESS_ALL=1
shift
;;
--force)
FORCE=1
shift
;;
-h|--help)
usage 0
;;
*)
printf 'Unknown arg: %s\n' "$1" >&2
usage 1
;;
esac
done
if [[ $PROCESS_ALL -eq 0 && -z "$TARGET_DOC_ID" && -z "$TARGET_FILENAME" ]]; then
usage 1
fi
if [[ -n "$TARGET_DOC_ID" ]]; then
fname=$(doc_id_to_filename "$TARGET_DOC_ID") || {
printf 'No PDF in %s matches doc_id %s\n' "$RAW_DIR" "$TARGET_DOC_ID" >&2
exit 1
}
convert_one_pdf "$RAW_DIR/$fname"
elif [[ -n "$TARGET_FILENAME" ]]; then
if [[ ! -f "$RAW_DIR/$TARGET_FILENAME" ]]; then
printf 'File not found: %s\n' "$RAW_DIR/$TARGET_FILENAME" >&2
exit 1
fi
convert_one_pdf "$RAW_DIR/$TARGET_FILENAME"
else
# --all
count=0
for pdf in "$RAW_DIR"/*.pdf; do
[[ -f "$pdf" ]] || continue
convert_one_pdf "$pdf" || true
count=$((count + 1))
done
printf '\n=== Total processed: %d PDFs ===\n' "$count"
fi

627
scripts/02-vision-page.py Executable file
View file

@ -0,0 +1,627 @@
#!/usr/bin/env python3
"""
02-vision-page.py — Fase 3 — Vision Haiku via Claude Code CLI (OAuth)
Usa o `claude` CLI (plano Max 20x do usuário) — NÃO usa ANTHROPIC_API_KEY direta.
Invoca `claude -p --model haiku` por subprocess para cada PNG.
Para cada PNG em processing/png/<doc-id>/p-NNN.png:
1. Lê OCR raw (processing/ocr/<doc-id>/p-NNN.txt)
2. Chama claude CLI com prompt estruturado pedindo que use Read no PNG
3. Recebe JSON com page_type, content_classification, entities_extracted, etc.
4. Salva JSON em processing/vision/<doc-id>/p-NNN.json
5. Escreve wiki/pages/<doc-id>/p<NNN>.md (frontmatter + corpo) — idioma ORIGINAL
Idempotente: pula se vision JSON + page.md já existem (use --force para refazer).
Uso:
./02-vision-page.py --doc-id dow-uap-d54-mission-report-mediterranean-sea-na [--force] [--max-pages N]
./02-vision-page.py --all
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import random
import re
import subprocess
import sys
import threading
import time
import unicodedata
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
try:
from PIL import Image
except ImportError:
sys.stderr.write("Missing pillow. Run: pip3 install pillow\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
RAW_DIR = UFO_ROOT / "raw"
PNG_BASE = UFO_ROOT / "processing" / "png"
OCR_BASE = UFO_ROOT / "processing" / "ocr"
VISION_BASE = UFO_ROOT / "processing" / "vision"
PAGES_BASE = UFO_ROOT / "wiki" / "pages"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
MODEL = "haiku" # claude-haiku-4-5 alias
VISION_MODEL_FULL = "claude-haiku-4-5"
WIKI_VERSION = "0.1.0"
SCHEMA_VERSION = "0.1.0"
MAX_TURNS = 3
DEFAULT_WORKERS = 4
DEFAULT_RETRIES = 3
DEFAULT_TIMEOUT = 180
_print_lock = threading.Lock()
def safe_print(*args, **kwargs):
"""Thread-safe print."""
with _print_lock:
print(*args, **kwargs, flush=True)
VISION_JSON_SCHEMA = {
"type": "object",
"properties": {
"page_type": {"type": "string"},
"content_classification": {"type": "array", "items": {"type": "string"}},
"language_detected": {"type": "string"},
"classification_markings": {"type": "array"},
"redactions": {"type": "array"},
"signatures_observed": {"type": "array"},
"tables_detected": {"type": "array"},
"images_detected": {"type": "array"},
"entities_extracted": {"type": "object"},
"uap_observation_fields": {},
"vision_description": {"type": "string"},
"ocr_quality_score": {"type": "number"},
"vision_quality_score": {"type": "number"},
"flags": {"type": "array"},
},
"required": [
"page_type",
"content_classification",
"language_detected",
"vision_description",
"entities_extracted",
"redactions",
"classification_markings",
],
}
def build_prompt(png_path: Path, ocr_text: str) -> str:
"""Build the prompt sent to claude CLI."""
return f"""You are an evidence officer in the Investigation Bureau, analyzing one page of a US Department of War UAP/UFO document released at war.gov/ufo.
STEP 1: Use the Read tool to view this PNG of the page:
{png_path}
STEP 2: Combine what you SEE in the image with the raw pdftotext OCR below.
OCR raw (pdftotext -layout):
```
{ocr_text}
```
STEP 3: Output ONE JSON object (no markdown fence, no commentary, no preamble) matching this exact schema:
{{
"page_type": "cover|toc|body|signature|photo|sketch|map|stamp|blank|appendix|redaction-heavy|table-page|mixed",
"content_classification": ["text-only"|"contains-photos"|"contains-sketches"|"contains-diagrams"|"contains-maps"|"contains-tables"|"contains-signatures"|"contains-stamps"|"redaction-heavy"|"mixed"|"blank"],
"language_detected": "en|pt|es|fr|de|ru|unknown",
"classification_markings": [
{{"level":"UNCLASSIFIED|CUI|CONFIDENTIAL|SECRET|TOP SECRET","caveats":["NOFORN"],"location":"header|footer|banner|stamp","bbox":{{"x":0.0,"y":0.0,"w":0.0,"h":0.0}}}}
],
"redactions": [
{{"code":"(b)(1) 1.4(a)|(b)(3)|(b)(6)|other","description":"...","bbox":{{"x":0.0,"y":0.0,"w":0.0,"h":0.0}},"text_inferred":null}}
],
"signatures_observed": [
{{"signer_inferred":null,"confidence_band":"low|medium|high","bbox":{{"x":0.0,"y":0.0,"w":0.0,"h":0.0}},"notes":"..."}}
],
"tables_detected": [
{{"local_table_index":1,"bbox":{{"x":0.0,"y":0.0,"w":0.0,"h":0.0}},"spans_multi_page":false,"continues_from_prev_page":false,"likely_continues_next_page":false,"row_count_estimate":0,"col_count_estimate":0,"headers_summary":"..."}}
],
"images_detected": [
{{"local_image_index":1,"image_type":"photo|sketch|map|chart|stamp|signature|redaction|logo|seal|diagram|other","bbox":{{"x":0.0,"y":0.0,"w":0.0,"h":0.0}},"caption_ocr":"..."}}
],
"vision_description": "Rich English description (2-5 sentences) of the page layout, visible elements, redaction extent, stamps, sketches, etc. PRESERVE ORIGINAL LANGUAGE of any quoted text from the document.",
"vision_description_pt_br": "Same content as vision_description, translated to Brazilian Portuguese (pt-br). Use Brazilian spelling and idioms (NOT European Portuguese). Preserve UTF-8 accents (ã, é, ç, etc.). KEEP verbatim English quotes from the document in English (do not translate quoted text from the page itself); only the narrative description is translated.",
"entities_extracted": {{
"people": [{{"name":"As written","role_in_page":"subject|witness|author|signer|mentioned"}}],
"organizations": [{{"name":"As written","aliases":[]}}],
"locations": [{{"name":"As written","type":"city|region|country|sea|strait|airbase|naval-base|mountain|desert|other"}}],
"events": [{{"label":"Short label","date":"YYYY-MM-DD|YYYY|NA"}}],
"uap_objects": [{{"shape":"sphere|disc|triangle|cylinder|cube|elongated-ellipsoid|cigar|irregular|unknown","color":"...","size_estimate":"..."}}],
"vehicles": [{{"name":"...","class":"aircraft|ship|submarine|spacecraft|satellite|ground|other"}}],
"operations": [{{"name":"...","type":"military-operation|reporting-protocol|research-program|task-force|foia-disclosure|other"}}],
"concepts": [{{"name":"...","class":"legal-instrument|phenomenon-type|doctrine|scientific-term|jargon|program-name|other"}}]
}},
"uap_observation_fields": {{
"date_time_utc":"...","duration_seconds":null,"shape":"...","color":"...","size_estimate":"...","altitude_ft":null,"speed_kts":null,"bearing_deg":null,"distance_nm":null,"coordinates":{{"lat":null,"lon":null,"raw_text":"..."}}
}},
"ocr_quality_score": 0.0,
"vision_quality_score": 0.0,
"flags": ["low-ocr"|"heavy-redaction"|"rotated"|"scanned-twice"|"missing-page-number"]
}}
Rules:
- Empty arrays for not-applicable fields. Do not omit keys.
- bbox is normalized 0..1 (x,y,w,h) relative to the page image.
- Entity NAMES, OCR-extracted strings, verbatim quotes, classification markings, redaction codes: ALWAYS in ORIGINAL source language (do NOT translate). Preserve original spelling, including any typos (e.g., "TRIANGLUAR" must stay as written).
- ONLY `vision_description_pt_br` is the translation. Everything else stays in source language.
- `vision_description_pt_br` must be Brazilian Portuguese (pt-br), NOT European Portuguese (pt-pt). Use Brazilian vocabulary and spelling. Preserve UTF-8 accentuation correctly (ç, ã, á, é, í, ó, ú, â, ê, ô, à).
- uap_observation_fields = null when page has no UAP encounter block.
- Output ONLY the JSON. No preamble, no fence, no commentary.
"""
def utc_now_iso():
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def filename_to_doc_id(filename: str) -> str:
base = filename.rsplit(".", 1)[0]
nfkd = unicodedata.normalize("NFKD", base)
ascii_str = "".join(c for c in nfkd if not unicodedata.combining(c))
lower = ascii_str.lower()
replaced = re.sub(r"[^a-z0-9-]", "-", lower)
collapsed = re.sub(r"-+", "-", replaced).strip("-")
if collapsed and collapsed[0].isdigit():
collapsed = "doc-" + collapsed
return collapsed
def sha256_file(p: Path) -> str:
h = hashlib.sha256()
with open(p, "rb") as fh:
for chunk in iter(lambda: fh.read(65536), b""):
h.update(chunk)
return h.hexdigest()
def extract_json(text: str) -> dict:
"""Extract JSON object from claude CLI output (may have markdown fences)."""
text = text.strip()
if text.startswith("```"):
text = re.sub(r"^```(?:json)?\s*", "", text)
text = re.sub(r"\s*```$", "", text)
# Find first { and matching last }
start = text.find("{")
if start == -1:
raise ValueError("No JSON object in response")
# Track depth to find matching close
depth = 0
for i, c in enumerate(text[start:], start):
if c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
return json.loads(text[start : i + 1])
raise ValueError("Unclosed JSON object")
def call_claude_vision(png_path: Path, ocr_text: str, timeout: int = DEFAULT_TIMEOUT) -> tuple[dict, dict]:
"""Invoke `claude -p --model haiku` and return (vision_data, metadata). Single attempt."""
prompt = build_prompt(png_path, ocr_text)
cmd = [
"claude",
"-p",
"--model", MODEL,
"--output-format", "json",
"--max-turns", str(MAX_TURNS),
"--allowedTools", "Read",
"--add-dir", str(png_path.parent),
"--",
prompt,
]
res = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
check=False,
)
if res.returncode != 0:
raise RuntimeError(f"claude CLI failed (rc={res.returncode}): {res.stderr[-2000:]}")
cli_output = json.loads(res.stdout)
if cli_output.get("is_error"):
raise RuntimeError(f"claude reported error: {cli_output.get('result', '')[:500]}")
result_text = cli_output.get("result", "")
vision_data = extract_json(result_text)
metadata = {
"duration_ms": cli_output.get("duration_ms"),
"duration_api_ms": cli_output.get("duration_api_ms"),
"total_cost_usd": cli_output.get("total_cost_usd"),
"num_turns": cli_output.get("num_turns"),
"session_id": cli_output.get("session_id"),
"usage": cli_output.get("usage"),
}
return vision_data, metadata
def call_with_retry(
png_path: Path,
ocr_text: str,
retries: int = DEFAULT_RETRIES,
base_backoff: float = 5.0,
timeout: int = DEFAULT_TIMEOUT,
) -> tuple[dict, dict]:
"""Call vision with exponential backoff + jitter. Raises on final failure."""
last_err: Exception | None = None
for attempt in range(1, retries + 1):
try:
return call_claude_vision(png_path, ocr_text, timeout=timeout)
except subprocess.TimeoutExpired as e:
last_err = e
backoff = base_backoff * (2 ** (attempt - 1)) + random.uniform(0, 2)
safe_print(f" timeout (attempt {attempt}/{retries}); sleeping {backoff:.1f}s")
time.sleep(backoff)
except RuntimeError as e:
last_err = e
msg = str(e).lower()
transient = any(s in msg for s in ("overloaded", "rate", "429", "500", "502", "503", "504", "timeout", "connection"))
if not transient or attempt == retries:
raise
backoff = base_backoff * (2 ** (attempt - 1)) + random.uniform(0, 2)
safe_print(f" transient error (attempt {attempt}/{retries}); sleeping {backoff:.1f}s")
time.sleep(backoff)
except json.JSONDecodeError as e:
last_err = e
if attempt == retries:
raise
backoff = base_backoff * (2 ** (attempt - 1)) + random.uniform(0, 2)
safe_print(f" JSON parse error (attempt {attempt}/{retries}); sleeping {backoff:.1f}s")
time.sleep(backoff)
if last_err:
raise last_err
raise RuntimeError("unreachable")
def render_page_md(
*,
doc_id: str,
page_num: int,
total_pages: int,
png_path: Path,
ocr_path: Path,
vision_path: Path,
vision_data: dict,
png_dimensions: tuple[int, int],
now_iso: str,
) -> str:
padded = f"{page_num:03d}"
page_id = f"{doc_id}/p{padded}"
frontmatter = {
"schema_version": SCHEMA_VERSION,
"type": "page",
"page_id": page_id,
"doc_id": doc_id,
"page_number": page_num,
"total_pages": total_pages,
"png_path": f"../../../processing/png/{doc_id}/{png_path.name}",
"png_sha256": sha256_file(png_path),
"png_dpi": 200,
"png_width": png_dimensions[0],
"png_height": png_dimensions[1],
"ocr_raw_path": f"../../../processing/ocr/{doc_id}/{ocr_path.name}",
"vision_raw_path": f"../../../processing/vision/{doc_id}/{vision_path.name}",
"vision_model": VISION_MODEL_FULL,
"vision_run_at": now_iso,
"page_type": vision_data.get("page_type", "body"),
"content_classification": vision_data.get("content_classification", []),
"language_detected": vision_data.get("language_detected", "unknown"),
"classification_markings": vision_data.get("classification_markings", []),
"redactions": vision_data.get("redactions", []),
"signatures_observed": vision_data.get("signatures_observed", []),
"tables_detected": vision_data.get("tables_detected", []),
"images_detected": vision_data.get("images_detected", []),
"entities_extracted": vision_data.get("entities_extracted", {}),
"uap_observation_fields": vision_data.get("uap_observation_fields"),
"vision_description": vision_data.get("vision_description", ""),
"vision_description_pt_br": vision_data.get("vision_description_pt_br", ""),
"ocr_quality_score": vision_data.get("ocr_quality_score", 0.0),
"vision_quality_score": vision_data.get("vision_quality_score", 0.0),
"flags": vision_data.get("flags", []),
"last_ingest": now_iso,
"last_lint": None,
"wiki_version": WIKI_VERSION,
}
yaml_str = yaml.dump(frontmatter, allow_unicode=True, sort_keys=False, default_flow_style=False)
ocr_text = ocr_path.read_text(encoding="utf-8", errors="replace").strip()
pt_desc = vision_data.get("vision_description_pt_br", "").strip()
en_desc = vision_data.get("vision_description", "").strip()
body = f"""# [[{doc_id}]] — Page {page_num} of {total_pages}
![Page {page_num}](../../../processing/png/{doc_id}/{png_path.name})
## OCR Text (raw, original language)
```
{ocr_text}
```
## Vision Description (EN)
{en_desc}
## Descrição Vision (PT-BR)
{pt_desc}
## Investigation Notes
- `page_type`: `{vision_data.get("page_type", "unknown")}`
- `content_classification`: {', '.join(f"`{c}`" for c in vision_data.get("content_classification", [])) or "_n/a_"}
- `language_detected`: `{vision_data.get("language_detected", "unknown")}`
- `flags`: {', '.join(f"`{f}`" for f in vision_data.get("flags", [])) or "_none_"}
"""
return f"---\n{yaml_str}---\n\n{body}"
def _process_page(
*,
doc_id: str,
png_path: Path,
ocr_path: Path,
vision_json_path: Path,
page_md_path: Path,
page_num: int,
total_pages: int,
retries: int,
timeout: int,
) -> tuple[str, float, float, str | None]:
"""Process a single page. Returns (label, elapsed_seconds, cost_usd, error_or_none)."""
padded = f"{page_num:03d}"
t0 = time.time()
try:
with Image.open(png_path) as im:
png_dimensions = im.size
except Exception:
png_dimensions = (0, 0)
ocr_text = ocr_path.read_text(encoding="utf-8", errors="replace").strip()
try:
vision_data, meta = call_with_retry(png_path, ocr_text, retries=retries, timeout=timeout)
except Exception as e:
return (f"p{padded}", time.time() - t0, 0.0, str(e)[:300])
vision_json_path.write_text(
json.dumps({"vision_data": vision_data, "meta": meta}, indent=2, ensure_ascii=False),
encoding="utf-8",
)
md = render_page_md(
doc_id=doc_id,
page_num=page_num,
total_pages=total_pages,
png_path=png_path,
ocr_path=ocr_path,
vision_path=vision_json_path,
vision_data=vision_data,
png_dimensions=png_dimensions,
now_iso=utc_now_iso(),
)
page_md_path.write_text(md, encoding="utf-8")
elapsed = time.time() - t0
cost = meta.get("total_cost_usd", 0.0) or 0.0
return (f"p{padded}", elapsed, cost, None)
def find_pdf_filename_for_doc_id(doc_id: str) -> str | None:
"""Reverse-canonicalize: scan raw/ for a PDF whose canonical doc_id matches."""
def _canon(fname: str) -> str:
base = fname.rsplit(".", 1)[0]
nfkd = unicodedata.normalize("NFKD", base)
ascii_str = "".join(c for c in nfkd if not unicodedata.combining(c))
lower = ascii_str.lower()
replaced = re.sub(r"[^a-z0-9-]", "-", lower)
collapsed = re.sub(r"-+", "-", replaced).strip("-")
if collapsed and collapsed[0].isdigit():
collapsed = "doc-" + collapsed
return collapsed
raw_dir = UFO_ROOT / "raw"
for p in raw_dir.glob("*.pdf"):
if _canon(p.name) == doc_id:
return p.name
return None
def try_reconvert_from_raw(doc_id: str) -> bool:
"""Attempt to regenerate PNGs/OCR via scripts/01-convert-pdfs.sh.
Returns True if reconvert succeeded (PNGs now exist), False otherwise."""
fname = find_pdf_filename_for_doc_id(doc_id)
if not fname:
safe_print(f" ⚠ PDF for {doc_id} not in raw/ — manual download required from https://www.war.gov/ufo/<filename>.pdf")
return False
script = UFO_ROOT / "scripts" / "01-convert-pdfs.sh"
safe_print(f" ↻ re-converting from raw/{fname} ...")
res = subprocess.run(
[str(script), "--filename", fname],
capture_output=True,
text=True,
timeout=300,
check=False,
)
if res.returncode != 0:
safe_print(f" ✗ re-conversion failed: {res.stderr[-500:]}")
return False
return True
def process_doc(
doc_id: str,
force: bool = False,
max_pages: int | None = None,
workers: int = DEFAULT_WORKERS,
retries: int = DEFAULT_RETRIES,
timeout: int = DEFAULT_TIMEOUT,
):
png_dir = PNG_BASE / doc_id
ocr_dir = OCR_BASE / doc_id
vision_dir = VISION_BASE / doc_id
pages_dir = PAGES_BASE / doc_id
vision_dir.mkdir(parents=True, exist_ok=True)
pages_dir.mkdir(parents=True, exist_ok=True)
pngs = sorted(png_dir.glob("p-*.png"))
if not pngs:
# Fallback: try to re-convert from raw/<file>.pdf
safe_print(f"No PNGs for doc_id={doc_id} in {png_dir} — attempting re-conversion from raw/")
if try_reconvert_from_raw(doc_id):
pngs = sorted(png_dir.glob("p-*.png"))
if not pngs:
sys.stderr.write(
f"FATAL: no PNGs for doc_id={doc_id} after re-conversion attempt.\n"
f" Expected at: {png_dir}\n"
f" Manual recovery: download the PDF from https://www.war.gov/ufo/<filename>.pdf\n"
f" and place it in /Users/guto/ufo/raw/, then re-run this script.\n"
)
return
total_pages = len(pngs)
if max_pages:
pngs = pngs[:max_pages]
# Build worklist (after skip filter)
worklist = []
for png_path in pngs:
m = re.match(r"p-(\d+)\.png$", png_path.name)
if not m:
continue
page_num = int(m.group(1))
padded = f"{page_num:03d}"
vision_json_path = vision_dir / f"p-{padded}.json"
page_md_path = pages_dir / f"p{padded}.md"
ocr_path = ocr_dir / f"p-{padded}.txt"
if not ocr_path.exists():
safe_print(f" p{padded}: skip (missing OCR)")
continue
if not force and vision_json_path.exists() and page_md_path.exists():
continue # silently skip already-processed
worklist.append((png_path, ocr_path, vision_json_path, page_md_path, page_num))
skipped = len(pngs) - len(worklist)
safe_print(f"\n=== {doc_id} ({total_pages} total, {len(worklist)} to process, {skipped} skipped, {workers} workers) ===")
if not worklist:
return
log_entries: list[str] = []
total_cost = 0.0
done = 0
started_at = time.time()
def _job(args):
png_path, ocr_path, vision_json_path, page_md_path, page_num = args
return _process_page(
doc_id=doc_id,
png_path=png_path,
ocr_path=ocr_path,
vision_json_path=vision_json_path,
page_md_path=page_md_path,
page_num=page_num,
total_pages=total_pages,
retries=retries,
timeout=timeout,
)
with ThreadPoolExecutor(max_workers=workers) as pool:
futures = {pool.submit(_job, item): item for item in worklist}
for fut in as_completed(futures):
label, elapsed, cost, err = fut.result()
done += 1
total_cost += cost
wall = time.time() - started_at
if err:
safe_print(f" [{done}/{len(worklist)}] {label}: FAILED ({elapsed:.1f}s) — {err}")
log_entries.append(f" - {label}: vision error: {err}")
else:
rate = done / wall if wall > 0 else 0
eta = (len(worklist) - done) / rate if rate > 0 else 0
safe_print(f" [{done}/{len(worklist)}] {label}: ok ({elapsed:.1f}s, ${cost:.4f}) — wall {wall:.0f}s eta {eta:.0f}s")
log_entries.append(f" - {label}: ok ({elapsed:.1f}s, ${cost:.4f})")
wall = time.time() - started_at
safe_print(f" Total: {done} pages in {wall:.0f}s ({wall / max(done,1):.1f}s/page avg), ${total_cost:.4f}")
# Append to log
if log_entries:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(f"\n## {utc_now_iso()} — VISION INGEST\n")
fh.write(
f"- operator: archivist (via claude CLI OAuth)\n"
f"- doc_id: {doc_id}\n"
f"- model: {VISION_MODEL_FULL}\n"
f"- workers: {workers}\n"
f"- pages_processed: {len(log_entries)}\n"
f"- wall_seconds: {wall:.0f}\n"
f"- total_cost_usd: {total_cost:.4f}\n"
f"- results:\n"
)
for entry in sorted(log_entries):
fh.write(entry + "\n")
def main():
ap = argparse.ArgumentParser(description="Vision-process each PNG of a UFO doc via claude CLI (OAuth).")
g = ap.add_mutually_exclusive_group(required=True)
g.add_argument("--doc-id", help="single doc_id (kebab-case)")
g.add_argument("--all", action="store_true", help="process all docs in processing/png/")
ap.add_argument("--force", action="store_true", help="reprocess existing pages")
ap.add_argument("--max-pages", type=int, default=None, help="cap pages per doc (for smoke test)")
ap.add_argument("--workers", type=int, default=DEFAULT_WORKERS, help=f"parallel workers per doc (default {DEFAULT_WORKERS})")
ap.add_argument("--retries", type=int, default=DEFAULT_RETRIES, help=f"retries on transient errors (default {DEFAULT_RETRIES})")
ap.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT, help=f"per-call timeout seconds (default {DEFAULT_TIMEOUT})")
args = ap.parse_args()
# Verify claude CLI is available
try:
subprocess.run(["claude", "--version"], capture_output=True, check=True, timeout=10)
except (FileNotFoundError, subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
sys.stderr.write(f"claude CLI not found or not working: {e}\n")
sys.exit(2)
common = dict(
force=args.force,
max_pages=args.max_pages,
workers=args.workers,
retries=args.retries,
timeout=args.timeout,
)
if args.doc_id:
process_doc(args.doc_id, **common)
else:
for doc_dir in sorted(PNG_BASE.iterdir()):
if doc_dir.is_dir():
process_doc(doc_dir.name, **common)
if __name__ == "__main__":
main()

362
scripts/02b-enrich-with-web-metadata.py Executable file
View file

@ -0,0 +1,362 @@
#!/usr/bin/env python3
"""
02b-enrich-with-web-metadata.py — Phase 0.5
Injects the war.gov-extracted metadata (record_id, incident_date,
incident_location, agency, etc.) into each wiki/documents/<doc-id>.md
frontmatter. Also marks the 4 placeholder records as `availability:
pending-upstream`.
For each document.md we already created from a local PDF, we find the
matching war.gov record using the same 3-tier matcher as 00b-coverage:
1. exact-norm
2. primary-id (DOW-UAP-D74, DOS-UAP-D1, etc.)
3. Jaccard ≥0.5 on signature tokens
The matched record's fields are added under a `war_gov` block in the
frontmatter (non-destructive — never overwrites existing manual data).
If `--rename-events` is passed, events file `EV-XXXX-XX-XX-…` are renamed
to `EV-YYYY-MM-DD-…` based on the matched document's incident_date.
The script updates all wiki-link references to the renamed event ids.
Usage:
./02b-enrich-with-web-metadata.py [--dry-run] [--rename-events]
"""
from __future__ import annotations
import argparse
import json
import re
import sys
import unicodedata
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
DOCS_DIR = UFO_ROOT / "wiki" / "documents"
EVENTS_DIR = UFO_ROOT / "wiki" / "entities" / "events"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
METADATA_JSON = UFO_ROOT / "processing" / "war-gov-metadata" / "all-documents-release-01-basic.json"
# Records whose Download serves a placeholder file (verified 2026-05-13)
PLACEHOLDER_RECORDS = {"record-140", "record-154", "record-155", "record-156"}
COMMON = {
"mission", "report", "uap", "the", "of", "and", "a", "in", "on", "for",
"with", "to", "from", "department", "war", "fbi", "nasa", "state",
"unresolved", "debrief", "summary", "transcript", "crew", "general",
"vol", "incident", "summaries", "photo", "video", "cable", "email",
"correspondence", "correspondance", "launch", "range", "fouler",
"force", "air", "navy", "between", "or", "year", "month",
"january", "february", "march", "april", "may", "june", "july",
"august", "september", "october", "november", "december", "redacted",
"sub", "sighting", "about", "kuwait", "kazakhstan", "papua", "guinea",
"syria", "iraq", "iran", "yemen", "djibouti", "japan", "greece",
"mexico", "germany", "turkey", "turkmenistan", "georgia", "tbilisi",
"indopacom", "middle", "east", "africa", "europe", "western", "united",
"states", "north", "south", "america",
}
def normalize(s: str) -> str:
if not s:
return ""
nfkd = unicodedata.normalize("NFKD", s)
ascii_s = "".join(c for c in nfkd if not unicodedata.combining(c))
lower = ascii_s.lower().replace("'", "").replace(",", "-").replace("[", "").replace("]", "")
replaced = re.sub(r"[^a-z0-9]+", "-", lower)
norm = re.sub(r"-+", "-", replaced).strip("-")
prev = None
while prev != norm:
prev = norm
norm = re.sub(r"(?<=[a-z-])0+(\d)", r"\1", norm)
return norm
def signature_tokens(s: str) -> set[str]:
return {t for t in normalize(s).split("-") if t and t not in COMMON}
def jaccard(a: set, b: set) -> float:
return len(a & b) / len(a | b) if a and b else 0.0
def primary_id(s: str) -> str | None:
n = normalize(s)
for p in (
r"^(dow-uap-[a-z]{1,4}\d+)",
r"^(dos-uap-d\d+)",
r"^(nasa-uap-[a-z]{1,3}\d+[a-z]?)",
r"^(fbi-photo-[a-z]\d+)",
):
m = re.match(p, n)
if m:
return m.group(1)
return None
def parse_us_date(s: str) -> tuple[str, str]:
"""Parse a US-format date like '12/30/47' or '11/9/23' into
(iso_date, confidence). Year handling: 2-digit years <=30 → 20xx, else 19xx.
Returns (iso, confidence_band) e.g. ('1947-12-30','high').
Special cases: 'N/A' → ('NA','none'), 'LATE 2025' → ('2025-12-XX','low').
Year-only '1969' → ('1969-XX-XX','medium').
Range '4/10/2025-4/11/2025' → first date with confidence medium.
"""
if not s or s.strip() == "" or s.strip().upper() in ("N/A", "NA", "NULL"):
return ("NA", "none")
s = s.strip()
# Take first half of range
if "-" in s and any(c.isdigit() for c in s.split("-")[0]):
first = s.split("-")[0].strip()
# Try parsing the first half
iso, conf = parse_us_date(first)
if iso != "NA":
return (iso, "medium")
# Fuzzy patterns
if re.match(r"^late\s+\d{4}$", s, re.I):
y = re.search(r"\d{4}", s).group(0)
return (f"{y}-12-XX", "low")
if re.match(r"^\d{4}$", s):
return (f"{s}-XX-XX", "medium")
# M/D/YY or M/D/YYYY
m = re.match(r"^(\d{1,2})/(\d{1,2})/(\d{2,4})$", s)
if m:
mo, d, y = m.groups()
y_int = int(y)
if len(y) == 2:
y_int = 2000 + y_int if y_int <= 30 else 1900 + y_int
iso = f"{y_int:04d}-{int(mo):02d}-{int(d):02d}"
return (iso, "high")
return ("NA", "speculation")
def event_id_from_date_and_slug(iso_date: str, slug_seed: str) -> str:
"""Build EV-YYYY-MM-DD-<slug> id."""
if iso_date == "NA":
y, mo, d = "XXXX", "XX", "XX"
else:
parts = iso_date.split("-")
y = parts[0] if len(parts) > 0 else "XXXX"
mo = parts[1] if len(parts) > 1 else "XX"
d = parts[2] if len(parts) > 2 else "XX"
slug = normalize(slug_seed)[:50].strip("-") or "unlabeled"
return f"EV-{y}-{mo}-{d}-{slug}"
def read_md(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end == -1:
return {}, c
try:
return (yaml.safe_load(c[3:end].strip()) or {}), c[end + 3 :].lstrip("\n")
except yaml.YAMLError:
return {}, c[end + 3 :].lstrip("\n")
def write_md(path: Path, fm: dict, body: str, dry_run: bool = False) -> bool:
yaml_str = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
new = f"---\n{yaml_str}---\n\n{body}" if not body.startswith("\n") else f"---\n{yaml_str}---\n{body}"
if path.exists() and path.read_text(encoding="utf-8") == new:
return False
if dry_run:
return True
path.write_text(new, encoding="utf-8")
return True
# ---------------------------------------------------------------------- main
def build_war_index(records: list[dict]) -> list[tuple[dict, str, set[str], str | None]]:
"""Return list of (record, norm_title, sig_tokens, primary_id)."""
out = []
for r in records:
t = r.get("title", "")
out.append((r, normalize(t), signature_tokens(t), primary_id(t)))
return out
def match_doc_to_war(doc_norm: str, doc_sig: set[str], doc_pid: str | None, war_index: list) -> tuple[dict | None, str]:
# Tier 1
for r, wnorm, _wsig, _wpid in war_index:
if wnorm == doc_norm:
return r, "exact-norm"
# Tier 2
if doc_pid:
for r, _wnorm, _wsig, wpid in war_index:
if wpid and wpid == doc_pid:
return r, f"primary-id={doc_pid}"
# Tier 3 containment
for r, wnorm, _wsig, _wpid in war_index:
if len(doc_norm) >= 12 and len(wnorm) >= 12 and (doc_norm in wnorm or wnorm in doc_norm):
return r, "containment"
# Tier 4 Jaccard
best, best_j = None, 0.0
for r, _wnorm, wsig, _wpid in war_index:
j = jaccard(doc_sig, wsig)
if j > best_j:
best_j = j; best = r
if best and best_j >= 0.50:
return best, f"jaccard={best_j:.2f}"
return None, "no-match"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--dry-run", action="store_true")
ap.add_argument("--rename-events", action="store_true", help="Rename EV-XXXX events to EV-YYYY-MM-DD")
args = ap.parse_args()
if not METADATA_JSON.exists():
sys.stderr.write(f"Metadata JSON not found: {METADATA_JSON}\n")
sys.exit(1)
data = json.loads(METADATA_JSON.read_text(encoding="utf-8"))
records = data.get("documents", [])
print(f"war.gov records: {len(records)}")
war_index = build_war_index(records)
docs = sorted(DOCS_DIR.glob("*.md"))
print(f"local document.md files: {len(docs)}")
enriched = 0
unchanged = 0
unmatched = []
event_renames: list[tuple[str, str]] = [] # (old_event_id, new_event_id)
for doc_path in docs:
fm, body = read_md(doc_path)
if fm.get("type") != "document":
continue
title_candidates = [
fm.get("canonical_title", ""),
fm.get("original_filename", ""),
doc_path.stem,
]
doc_norm = normalize(title_candidates[0]) or normalize(title_candidates[1]) or normalize(title_candidates[2])
doc_sig = signature_tokens(title_candidates[0]) | signature_tokens(title_candidates[1])
doc_pid = primary_id(title_candidates[0]) or primary_id(title_candidates[1]) or primary_id(doc_path.stem)
match, reason = match_doc_to_war(doc_norm, doc_sig, doc_pid, war_index)
if not match:
unmatched.append(doc_path.name)
continue
# Build war_gov block
incident_iso, date_conf = parse_us_date(match.get("incident_date") or "")
release_iso, _ = parse_us_date(match.get("release_date") or "")
war_block = {
"record_id": match["record_id"],
"title_official": match.get("title"),
"agency_official": match.get("agency"),
"release_date_official": release_iso,
"release_date_raw": match.get("release_date"),
"incident_date_official": incident_iso,
"incident_date_raw": match.get("incident_date"),
"incident_date_confidence": date_conf,
"incident_location_official": match.get("incident_location"),
"document_type_official": match.get("document_type"),
"match_reason": reason,
"availability": "pending-upstream" if match["record_id"] in PLACEHOLDER_RECORDS else "downloaded",
"extracted_from_war_gov_at": data.get("extracted_at"),
}
new_fm = dict(fm)
new_fm["war_gov"] = war_block
# Promote some fields to top-level if they were "NA" or empty
if (new_fm.get("document_date") in (None, "", "NA")) and incident_iso != "NA":
new_fm["document_date"] = incident_iso
if write_md(doc_path, new_fm, body, dry_run=args.dry_run):
enriched += 1
print(f" ✓ {doc_path.name} ← {match['record_id']} ({reason})")
# Compute potential event rename if applicable
if args.rename_events and incident_iso != "NA":
# Look for events referenced in this document that start with EV-XXXX-
key_events = (new_fm.get("key_entities") or {}).get("events") or []
for ref in key_events:
if isinstance(ref, str):
m = re.search(r"\[\[event/(EV-XXXX-XX-XX-[a-z0-9-]+)\]\]", ref)
if m:
old = m.group(1)
slug = old.replace("EV-XXXX-XX-XX-", "", 1)
new_id = event_id_from_date_and_slug(incident_iso, slug)
if new_id != old:
event_renames.append((old, new_id))
else:
unchanged += 1
# Apply event renames
rename_count = 0
for old, new in set(event_renames):
old_path = EVENTS_DIR / f"{old}.md"
new_path = EVENTS_DIR / f"{new}.md"
if not old_path.exists():
continue
if new_path.exists() and new_path != old_path:
print(f" ⚠ skip rename {old} → {new} (target exists)")
continue
if args.dry_run:
print(f" [dry] rename {old} → {new}")
rename_count += 1
continue
# Read, update event_id field, write to new path, delete old
fm, body = read_md(old_path)
fm["event_id"] = new
# Update date_start/date_end if currently NA
parts = new.split("-")
if len(parts) >= 4:
y, mo, d = parts[1], parts[2], parts[3]
if y != "XXXX" and (fm.get("date_start") in (None, "NA")):
if mo != "XX" and d != "XX":
fm["date_start"] = f"{y}-{mo}-{d}"
fm["date_end"] = fm.get("date_end") or f"{y}-{mo}-{d}"
fm["date_confidence"] = "high"
elif mo != "XX":
fm["date_start"] = f"{y}-{mo}"
write_md(new_path, fm, body)
old_path.unlink()
rename_count += 1
# Update all wiki-links pointing to the old event_id everywhere
for f in list(UFO_ROOT.rglob("*.md")):
if "/processing/" in str(f) or f == new_path:
continue
c = f.read_text(encoding="utf-8")
if old not in c:
continue
c2 = c.replace(f"[[event/{old}]]", f"[[event/{new}]]")
if c2 != c:
f.write_text(c2, encoding="utf-8")
print(f" ↺ renamed {old} → {new}")
# Log
print(f"\nEnriched: {enriched}, unchanged: {unchanged}, unmatched: {len(unmatched)}, event renames: {rename_count}")
if unmatched:
print("Unmatched docs (no war.gov record found):")
for n in unmatched[:20]:
print(f" - {n}")
if len(unmatched) > 20:
print(f" … and {len(unmatched) - 20} more")
if not args.dry_run and enriched > 0:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')} — ENRICH WAR.GOV (Phase 0.5)\n"
f"- operator: archivist\n- script: scripts/02b-enrich-with-web-metadata.py\n"
f"- json_source: {METADATA_JSON.name}\n"
f"- enriched: {enriched}\n- unchanged: {unchanged}\n- unmatched: {len(unmatched)}\n"
f"- event_renames: {rename_count}\n"
)
if __name__ == "__main__":
main()

666
scripts/03-dedup-entities.py Executable file
View file

@ -0,0 +1,666 @@
#!/usr/bin/env python3
"""
03-dedup-entities.py — Phase 5 — Entity dedup + upsert
For every page.md under wiki/pages/**/*.md:
1. Read frontmatter.entities_extracted
2. Canonicalize each entity name → kebab-case ASCII-fold id
3. Aggregate occurrences across pages (same kebab-case = same entity)
4. Upsert wiki/entities/<class>/<id>.md:
- If file missing: create with stub frontmatter + bilingual body
- If file exists: merge aliases, preserve manual edits to body, refresh
derived stats (mention_count per page, total_mentions, documents_count)
Does NOT populate mentioned_in[] — that's lint's job (script 04). This script
just creates/updates entity stubs so wiki-links resolve.
Idempotent: re-running with no new pages produces no changes (atomic write
suppresses writes when output is identical).
Uso:
./03-dedup-entities.py # process every page in wiki/pages/
./03-dedup-entities.py --doc-id <id> # only one document
./03-dedup-entities.py --dry-run # report what would change, don't write
"""
from __future__ import annotations
import argparse
import re
import sys
import unicodedata
from collections import defaultdict
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
PAGES_BASE = UFO_ROOT / "wiki" / "pages"
ENTITIES_BASE = UFO_ROOT / "wiki" / "entities"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
WIKI_VERSION = "0.1.0"
SCHEMA_VERSION = "0.1.0"
# (class_name_in_page_extraction, dir_name_under_wiki/entities/, frontmatter type, entity_class field, id_field)
ENTITY_CLASSES = [
("people", "people", "entity", "person", "person_id"),
("organizations", "organizations", "entity", "organization", "organization_id"),
("locations", "locations", "entity", "location", "location_id"),
("vehicles", "vehicles", "entity", "vehicle", "vehicle_id"),
("operations", "operations", "entity", "operation", "operation_id"),
("concepts", "concepts", "entity", "concept", "concept_id"),
# events and uap_objects have non-trivial ID schemes — handled separately
]
def utc_now_iso():
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def canonicalize_name(name: str) -> str:
"""Generic name → kebab-case ASCII-fold id."""
if not name:
return ""
nfkd = unicodedata.normalize("NFKD", name)
ascii_str = "".join(c for c in nfkd if not unicodedata.combining(c))
lower = ascii_str.lower()
replaced = re.sub(r"[^a-z0-9-]", "-", lower)
collapsed = re.sub(r"-+", "-", replaced).strip("-")
if collapsed and collapsed[0].isdigit():
# IDs cannot start with digit (per CLAUDE.md rule)
collapsed = "x-" + collapsed
return collapsed
def event_id_from_entry(entry: dict) -> str:
"""Build event_id from {label, date}. Date is YYYY-MM-DD, YYYY, or NA."""
label = entry.get("label", "")
date = entry.get("date", "NA") or "NA"
slug = canonicalize_name(label)[:40].strip("-") or "unlabeled"
# Parse date
m = re.match(r"^(\d{4})-(\d{2})-(\d{2})$", date)
if m:
return f"EV-{m.group(1)}-{m.group(2)}-{m.group(3)}-{slug}"
m = re.match(r"^(\d{4})-(\d{2})$", date)
if m:
return f"EV-{m.group(1)}-{m.group(2)}-XX-{slug}"
m = re.match(r"^(\d{4})$", date)
if m:
return f"EV-{m.group(1)}-XX-XX-{slug}"
return f"EV-XXXX-XX-XX-{slug}"
def uap_object_id_from_entry(entry: dict, event_id: str, index: int) -> str:
"""OBJ-<EVENT_SLUG_UPPERCASE>-<NN>."""
# Strip "EV-" prefix and dashes from date part to make compact slug
if event_id.startswith("EV-"):
rest = event_id[3:] # "2004-11-14-tic-tac-nimitz"
# Take first 2 parts (year + slug) as compact event ref
parts = rest.split("-", 4)
if len(parts) >= 4:
# parts: [year, month, day, ...slug...]
year = parts[0]
slug_part = "-".join(parts[3:]) if len(parts) > 3 else "unk"
slug_compact = slug_part.replace("-", "").upper()[:20] or "UNK"
event_short = f"EV{year}-{slug_compact}"
else:
event_short = "UNK"
else:
event_short = "UNK"
return f"OBJ-{event_short}-{index:02d}"
def read_frontmatter_and_body(path: Path) -> tuple[dict, str]:
"""Parse a markdown file. Returns (frontmatter_dict, body_str)."""
content = path.read_text(encoding="utf-8")
if not content.startswith("---"):
return {}, content
end = content.find("---", 4)
if end == -1:
return {}, content
fm_str = content[3:end].strip()
body = content[end + 3 :].lstrip("\n")
try:
fm = yaml.safe_load(fm_str) or {}
except yaml.YAMLError as e:
sys.stderr.write(f"YAML error in {path}: {e}\n")
fm = {}
return fm, body
def write_frontmatter_and_body(path: Path, frontmatter: dict, body: str, dry_run: bool = False) -> bool:
"""Atomic write. Returns True if file was changed.
For idempotency: if the file exists and the only differences are
`last_ingest` / `last_lint` timestamps, do NOT rewrite.
"""
new_yaml = yaml.dump(frontmatter, allow_unicode=True, sort_keys=False, default_flow_style=False)
new_content = f"---\n{new_yaml}---\n\n{body}" if not body.startswith("\n") else f"---\n{new_yaml}---\n{body}"
if path.exists():
existing = path.read_text(encoding="utf-8")
if existing == new_content:
return False
# Compare frontmatter excluding volatile timestamps
existing_fm, existing_body = read_frontmatter_and_body(path)
VOLATILE = {"last_ingest", "last_lint"}
snap_old = {k: v for k, v in existing_fm.items() if k not in VOLATILE}
snap_new = {k: v for k, v in frontmatter.items() if k not in VOLATILE}
if snap_old == snap_new and existing_body == body:
return False # only timestamps differ; treat as unchanged
if dry_run:
return True
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(new_content, encoding="utf-8")
return True
def collect_entities_from_pages(doc_filter: str | None = None) -> dict:
"""
Walk wiki/pages/**/*.md and collect all entity references.
Returns: {
'people': { canonical_id: { 'aliases': set, 'mentions': [(page_id, role, doc_id), ...], 'roles': set } },
'organizations': { ... },
...
'events': { event_id: { 'labels': set, 'date': '...', 'mentions': [...] } },
'uap_objects': { obj_id: { 'shape': ..., 'color': ..., 'mentions': [...], 'event_id': ... } },
}
"""
collected = {
"people": defaultdict(lambda: {"aliases": set(), "mentions": [], "roles": set()}),
"organizations": defaultdict(lambda: {"aliases": set(), "mentions": []}),
"locations": defaultdict(lambda: {"aliases": set(), "mentions": [], "type": None}),
"vehicles": defaultdict(lambda: {"aliases": set(), "mentions": [], "class": None}),
"operations": defaultdict(lambda: {"aliases": set(), "mentions": [], "type": None}),
"concepts": defaultdict(lambda: {"aliases": set(), "mentions": [], "class": None}),
"events": defaultdict(lambda: {"labels": set(), "date": "NA", "mentions": []}),
"uap_objects": defaultdict(lambda: {"shape": None, "color": None, "size_estimate": None, "mentions": [], "event_id": None}),
}
pattern = "**/*.md"
pages = sorted(PAGES_BASE.glob(pattern))
for page_path in pages:
if doc_filter and doc_filter not in str(page_path):
continue
fm, _body = read_frontmatter_and_body(page_path)
if not fm or fm.get("type") != "page":
continue
page_id = fm.get("page_id", "")
doc_id = fm.get("doc_id", "")
if not page_id or not doc_id:
continue
entities = fm.get("entities_extracted") or {}
# Standard entity classes
for class_name, _, _, _, _ in ENTITY_CLASSES:
entries = entities.get(class_name) or []
for entry in entries:
name = entry.get("name") if isinstance(entry, dict) else None
if not name:
continue
canonical = canonicalize_name(name)
if not canonical:
continue
bucket = collected[class_name][canonical]
bucket["aliases"].add(name)
role = (entry.get("role_in_page") if class_name == "people" else None) or "mentioned"
bucket["mentions"].append((page_id, role, doc_id))
if class_name == "people":
bucket["roles"].add(role)
elif class_name == "locations":
if not bucket.get("type"):
bucket["type"] = entry.get("type")
elif class_name == "vehicles":
if not bucket.get("class"):
bucket["class"] = entry.get("class")
elif class_name == "operations":
if not bucket.get("type"):
bucket["type"] = entry.get("type")
elif class_name == "concepts":
if not bucket.get("class"):
bucket["class"] = entry.get("class")
# Events
events = entities.get("events") or []
page_event_ids: list[str] = []
for entry in events:
label = entry.get("label")
if not label:
continue
ev_id = event_id_from_entry(entry)
page_event_ids.append(ev_id)
bucket = collected["events"][ev_id]
bucket["labels"].add(label)
bucket["mentions"].append((page_id, "documented_in", doc_id))
date = entry.get("date") or "NA"
if date != "NA" and bucket["date"] == "NA":
bucket["date"] = date
# UAP objects — link to first event on the page if available
uaps = entities.get("uap_objects") or []
for idx, entry in enumerate(uaps, start=1):
event_for_obj = page_event_ids[0] if page_event_ids else f"EV-XXXX-XX-XX-{canonicalize_name(doc_id)[:30]}"
obj_id = uap_object_id_from_entry(entry, event_for_obj, idx)
bucket = collected["uap_objects"][obj_id]
bucket["shape"] = bucket["shape"] or entry.get("shape")
bucket["color"] = bucket["color"] or entry.get("color")
bucket["size_estimate"] = bucket["size_estimate"] or entry.get("size_estimate")
bucket["event_id"] = bucket["event_id"] or event_for_obj
bucket["mentions"].append((page_id, "observation", doc_id))
return collected
def _stub_body(entity_class: str, canonical_name: str) -> str:
"""Standard bilingual stub body for new entities."""
return (
f"# {canonical_name}\n\n"
"## Description (EN)\n\n"
"_Stub generated by entity dedup. Will be enriched in Phase 6._\n\n"
"## Descrição (PT-BR)\n\n"
"_Stub gerado pela deduplicação de entidades. Será enriquecido na Fase 6._\n"
)
# Pre-built alias index: {dir_name: {alias_lower: path}} cached on first access.
_ALIAS_INDEX: dict[str, dict[str, Path]] = {}
def _ensure_alias_index(dir_name: str) -> dict[str, Path]:
"""Build alias→path map for a class folder once, cached. O(N) initial scan."""
if dir_name in _ALIAS_INDEX:
return _ALIAS_INDEX[dir_name]
target_dir = ENTITIES_BASE / dir_name
index: dict[str, Path] = {}
if target_dir.exists():
for entity_path in target_dir.glob("*.md"):
try:
fm, _ = read_frontmatter_and_body(entity_path)
except Exception:
continue
# Index by stem (canonical_id) AND by all aliases
index[entity_path.stem.lower()] = entity_path
cname = fm.get("canonical_name")
if isinstance(cname, str) and cname.strip():
index[cname.lower().strip()] = entity_path
for alias in (fm.get("aliases") or []):
if isinstance(alias, str) and alias.strip():
index[alias.lower().strip()] = entity_path
_ALIAS_INDEX[dir_name] = index
return index
def _find_existing_entity_by_alias(
dir_name: str,
names: set[str],
canonical_id_candidate: str,
) -> Path | None:
"""O(1) lookup via pre-built alias index."""
idx = _ensure_alias_index(dir_name)
canon_needle = canonical_id_candidate.lower()
if canon_needle in idx:
return idx[canon_needle]
for n in names:
if not n:
continue
key = n.lower().strip()
if key in idx:
return idx[key]
return None
def _register_in_index(dir_name: str, path: Path, names: set[str], canonical_name: str | None = None) -> None:
"""Add a newly-created or updated entity to the in-memory alias index."""
idx = _ensure_alias_index(dir_name)
idx[path.stem.lower()] = path
if canonical_name:
idx[canonical_name.lower().strip()] = path
for n in names:
if isinstance(n, str) and n.strip():
idx[n.lower().strip()] = path
def _upsert_simple_entity(
class_name: str,
dir_name: str,
type_value: str,
entity_class: str,
id_field: str,
canonical_id: str,
data: dict,
dry_run: bool,
) -> tuple[str, bool, Path]:
"""Upsert a person/org/location/vehicle/operation/concept entity file.
Returns (action, changed_bool, real_path).
Action is 'created'|'updated'|'unchanged'|'merged-into-existing'.
"""
# Check if an existing entity matches by alias — avoid creating duplicates
existing = _find_existing_entity_by_alias(dir_name, data.get("aliases", set()), canonical_id)
merged = False
if existing and existing.stem != canonical_id:
path = existing
merged = True
else:
path = ENTITIES_BASE / dir_name / f"{canonical_id}.md"
aliases_sorted = sorted(data.get("aliases", set()))
# canonical_name = most common alias (first by sort) — could be improved
canonical_name = aliases_sorted[0] if aliases_sorted else canonical_id
unique_docs = {doc_id for _, _, doc_id in data["mentions"]}
total_mentions = len(data["mentions"])
documents_count = len(unique_docs)
if path.exists():
fm, body = read_frontmatter_and_body(path)
# Merge aliases (preserve existing + add new)
existing_aliases = set(fm.get("aliases", []) or [])
merged_aliases = sorted(existing_aliases | set(aliases_sorted))
fm["aliases"] = merged_aliases
fm["total_mentions"] = total_mentions
fm["documents_count"] = documents_count
fm["last_ingest"] = utc_now_iso()
# Refresh entity-specific fields if missing
if class_name == "locations" and not fm.get("location_type") and data.get("type"):
fm["location_type"] = data["type"]
if class_name == "vehicles" and not fm.get("vehicle_class") and data.get("class"):
fm["vehicle_class"] = data["class"]
if class_name == "operations" and not fm.get("operation_type") and data.get("type"):
fm["operation_type"] = data["type"]
if class_name == "concepts" and not fm.get("concept_class") and data.get("class"):
fm["concept_class"] = data["class"]
changed = write_frontmatter_and_body(path, fm, body, dry_run=dry_run)
action = "merged-into-existing" if merged else ("updated" if changed else "unchanged")
return (action, changed, path)
# Create new
fm = {
"schema_version": SCHEMA_VERSION,
"type": type_value,
"entity_class": entity_class,
id_field: canonical_id,
"canonical_name": canonical_name,
"aliases": aliases_sorted,
}
if class_name == "people":
fm["roles"] = []
fm["dates"] = {"born": None, "died": None}
elif class_name == "organizations":
fm["organization_type"] = None
fm["country"] = None
elif class_name == "locations":
fm["location_type"] = data.get("type")
fm["country"] = []
fm["coordinates"] = None
elif class_name == "vehicles":
fm["vehicle_class"] = data.get("class")
elif class_name == "operations":
fm["operation_type"] = data.get("type")
fm["status"] = None
elif class_name == "concepts":
fm["concept_class"] = data.get("class")
fm["domain"] = None
fm["definition_short"] = None
fm["definition_short_pt_br"] = None
fm["mentioned_in"] = [] # populated by lint
fm["total_mentions"] = total_mentions
fm["documents_count"] = documents_count
fm["related_concepts" if class_name == "concepts" else "related"] = []
fm["enrichment_status"] = "none"
fm["external_sources"] = []
fm["last_ingest"] = utc_now_iso()
fm["last_lint"] = None
fm["wiki_version"] = WIKI_VERSION
body = _stub_body(entity_class, canonical_name)
write_frontmatter_and_body(path, fm, body, dry_run=dry_run)
_register_in_index(dir_name, path, set(aliases_sorted), canonical_name)
return ("created", True, path)
def _upsert_event(event_id: str, data: dict, dry_run: bool) -> tuple[str, bool, Path]:
labels = sorted(data["labels"])
canonical_name = labels[0] if labels else event_id
unique_docs = {doc_id for _, _, doc_id in data["mentions"]}
total_mentions = len(data["mentions"])
# Alias-match against existing events
existing = _find_existing_entity_by_alias("events", set(labels), event_id)
merged = False
if existing and existing.stem != event_id:
path = existing
merged = True
else:
path = ENTITIES_BASE / "events" / f"{event_id}.md"
# Date parse from event_id
m = re.match(r"^EV-(\d{4}|XXXX)-(\d{2}|XX)-(\d{2}|XX)-", event_id)
date_start = "NA"
if m:
y, mo, d = m.groups()
if y != "XXXX":
if mo != "XX" and d != "XX":
date_start = f"{y}-{mo}-{d}"
elif mo != "XX":
date_start = f"{y}-{mo}"
else:
date_start = y
if path.exists():
fm, body = read_frontmatter_and_body(path)
existing_aliases = set(fm.get("aliases", []) or [])
fm["aliases"] = sorted(existing_aliases | set(labels))
fm["total_mentions"] = total_mentions
fm["documents_count"] = len(unique_docs)
fm["last_ingest"] = utc_now_iso()
changed = write_frontmatter_and_body(path, fm, body, dry_run=dry_run)
action = "merged-into-existing" if merged else ("updated" if changed else "unchanged")
return (action, changed, path)
fm = {
"schema_version": SCHEMA_VERSION,
"type": "entity",
"entity_class": "event",
"event_id": event_id,
"canonical_name": canonical_name,
"aliases": labels,
"event_class": "uap-encounter",
"date_start": date_start,
"date_end": date_start,
"date_confidence": "low",
"primary_location": None,
"observers": [],
"uap_objects": [],
"documented_in": [],
"total_mentions": total_mentions,
"documents_count": len(unique_docs),
"narrative_summary_confidence": "low",
"narrative_summary": "_Stub. Will be enriched in Phase 7._",
"narrative_summary_pt_br": "_Stub. Será enriquecido na Fase 7._",
"enrichment_status": "none",
"external_sources": [],
"last_ingest": utc_now_iso(),
"last_lint": None,
"wiki_version": WIKI_VERSION,
}
body = _stub_body("events", canonical_name)
write_frontmatter_and_body(path, fm, body, dry_run=dry_run)
_register_in_index("events", path, set(labels), canonical_name)
return ("created", True, path)
def _find_existing_uap_object_by_event(event_id: str | None, shape: str, color: str, current_id: str) -> Path | None:
"""If an existing uap_object is observed in the same event with matching shape (or unknown),
treat as the same object."""
if not event_id:
return None
target_dir = ENTITIES_BASE / "uap-objects"
if not target_dir.exists():
return None
event_ref = f"[[event/{event_id}]]"
for p in target_dir.glob("*.md"):
if p.stem == current_id:
return p
try:
fm, _ = read_frontmatter_and_body(p)
except Exception:
continue
if fm.get("observed_in_event") != event_ref:
continue
existing_shape = (fm.get("shape") or "unknown").lower()
existing_color = (fm.get("color") or "unknown").lower()
if existing_shape in ("unknown", "", shape.lower()) and (
existing_color in ("unknown", "", color.lower())
):
return p
return None
def _upsert_uap_object(obj_id: str, data: dict, dry_run: bool) -> tuple[str, bool, Path]:
shape = data.get("shape") or "unknown"
color = data.get("color") or "unknown"
canonical_name = f"{shape} {color} UAP ({obj_id})"
event_id = data.get("event_id")
unique_docs = {doc_id for _, _, doc_id in data["mentions"]}
total_mentions = len(data["mentions"])
# If an existing uap_object is anchored to the same event with compatible shape/color, merge
existing = _find_existing_uap_object_by_event(event_id, shape, color, obj_id)
merged = False
if existing and existing.stem != obj_id:
path = existing
merged = True
else:
path = ENTITIES_BASE / "uap-objects" / f"{obj_id}.md"
if path.exists():
fm, body = read_frontmatter_and_body(path)
fm["total_mentions"] = total_mentions
fm["documents_count"] = len(unique_docs)
fm["last_ingest"] = utc_now_iso()
changed = write_frontmatter_and_body(path, fm, body, dry_run=dry_run)
action = "merged-into-existing" if merged else ("updated" if changed else "unchanged")
return (action, changed, path)
fm = {
"schema_version": SCHEMA_VERSION,
"type": "entity",
"entity_class": "uap_object",
"uap_object_id": obj_id,
"canonical_name": canonical_name,
"observed_in_event": f"[[event/{event_id}]]" if event_id else None,
"secondary_events": [],
"shape": shape,
"color": color,
"size_estimate_m": {"min": None, "max": None, "confidence_band": "speculation"},
"features": [],
"altitude_ft": {"min": None, "max": None, "confidence_band": "speculation"},
"speed_kts": {"min": None, "max": None, "confidence_band": "speculation"},
"maneuver_descriptors": [],
"sensor_observations": [],
"visual_records": [],
"total_mentions": total_mentions,
"documents_count": len(unique_docs),
"evidence_anchored": [],
"hypotheses_addressing": [],
"confidence_band_overall": "low",
"last_ingest": utc_now_iso(),
"last_lint": None,
"wiki_version": WIKI_VERSION,
}
body = _stub_body("uap_objects", canonical_name)
write_frontmatter_and_body(path, fm, body, dry_run=dry_run)
_register_in_index("uap-objects", path, set(), canonical_name)
return ("created", True, path)
def main():
ap = argparse.ArgumentParser(description="Dedup and upsert entities from page extractions.")
ap.add_argument("--doc-id", help="Only process pages of this doc_id")
ap.add_argument("--dry-run", action="store_true", help="Report would-be changes without writing")
args = ap.parse_args()
print(f"Scanning {PAGES_BASE} for entity references...", flush=True)
collected = collect_entities_from_pages(doc_filter=args.doc_id)
totals = {k: len(v) for k, v in collected.items()}
print(f"Found unique entities: {totals}", flush=True)
stats = {"created": 0, "updated": 0, "unchanged": 0, "merged-into-existing": 0}
# Simple classes
for class_name, dir_name, type_value, entity_class, id_field in ENTITY_CLASSES:
for canonical_id, data in collected[class_name].items():
action, changed, real_path = _upsert_simple_entity(
class_name, dir_name, type_value, entity_class, id_field,
canonical_id, data, dry_run=args.dry_run,
)
# Bucket merged-but-unchanged into "unchanged"
if action == "merged-into-existing" and not changed:
stats["unchanged"] += 1
else:
stats[action] += 1
if changed:
rel = real_path.relative_to(UFO_ROOT)
tag = f"merged ({canonical_id} → {real_path.stem})" if action == "merged-into-existing" else action
print(f" [{tag}] {rel}", flush=True)
# Events
for event_id, data in collected["events"].items():
action, changed, real_path = _upsert_event(event_id, data, dry_run=args.dry_run)
if action == "merged-into-existing" and not changed:
stats["unchanged"] += 1
else:
stats[action] += 1
if changed:
tag = f"merged ({event_id} → {real_path.stem})" if action == "merged-into-existing" else action
print(f" [{tag}] {real_path.relative_to(UFO_ROOT)}", flush=True)
# UAP objects — need to resolve event_id reference first via event upsert
# The event_id stored in data may have been merged into a different existing event.
# Pass through the event merge map to remap.
event_merge_map = {}
for event_id, edata in collected["events"].items():
# Re-derive what _upsert_event would have decided
labels = sorted(edata["labels"])
existing = _find_existing_entity_by_alias("events", set(labels), event_id)
if existing and existing.stem != event_id:
event_merge_map[event_id] = existing.stem
for obj_id, data in collected["uap_objects"].items():
# Remap event_id if it was merged
if data.get("event_id") in event_merge_map:
data["event_id"] = event_merge_map[data["event_id"]]
action, changed, real_path = _upsert_uap_object(obj_id, data, dry_run=args.dry_run)
if action == "merged-into-existing" and not changed:
stats["unchanged"] += 1
else:
stats[action] += 1
if changed:
tag = f"merged ({obj_id} → {real_path.stem})" if action == "merged-into-existing" else action
print(f" [{tag}] {real_path.relative_to(UFO_ROOT)}", flush=True)
print(f"\nSummary: created={stats['created']}, updated={stats['updated']}, "
f"merged={stats['merged-into-existing']}, unchanged={stats['unchanged']}", flush=True)
if not args.dry_run and (stats["created"] or stats["updated"]):
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(f"\n## {utc_now_iso()} — ENTITY DEDUP (Phase 5)\n")
fh.write(f"- operator: archivist\n")
fh.write(f"- script: scripts/03-dedup-entities.py\n")
fh.write(f"- doc_filter: {args.doc_id or '(all)'}\n")
fh.write(f"- created: {stats['created']}\n- updated: {stats['updated']}\n- unchanged: {stats['unchanged']}\n")
fh.write(f"- totals_after: {totals}\n")
if __name__ == "__main__":
main()

545
scripts/04-lint.py Executable file
View file

@ -0,0 +1,545 @@
#!/usr/bin/env python3
"""
04-lint.py — Phase 8 — Lint + backlink rebuild
Scans all .md files in wiki/ and case/ and:
1. Parses frontmatter
2. Collects all entity files + all wiki-links
3. Validates schema:
- Required universal fields (schema_version, type, canonical_title|canonical_name, wiki_version)
- Type-specific required fields
- Page sequence continuity per document
- Evidence grade ↔ chain_of_custody steps
4. Validates wiki-links: every [[link]] must resolve
5. Rebuilds mentioned_in[] in entity files (reverse scan from pages)
6. Reports: orphans, broken links, duplicate canonical names, missing fields
7. Appends LINT entry to wiki/log.md
Default mode = report-only (read-only safe). Use --fix to write back rebuilt
mentioned_in[] and last_lint timestamps.
Uso:
./04-lint.py # report only
./04-lint.py --fix # rebuild backlinks + write
./04-lint.py --scope wiki # restrict to wiki/ (skip case/)
./04-lint.py --strict # exit non-zero on any error
"""
from __future__ import annotations
import argparse
import re
import sys
from collections import defaultdict
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
WIKI_BASE = UFO_ROOT / "wiki"
CASE_BASE = UFO_ROOT / "case"
LOG_PATH = WIKI_BASE / "log.md"
# ----------------------------------------------------------------------
# Required-field tables
# ----------------------------------------------------------------------
UNIVERSAL_REQUIRED = ["schema_version", "type", "wiki_version"]
# For most types, at least one of these name fields is required.
# Exceptions are listed in TYPES_WITHOUT_CANONICAL_NAME — they identify
# themselves via a type-specific id (e.g. page_id, log files have no id).
NAME_FIELDS = ["canonical_title", "canonical_name"]
TYPES_WITHOUT_CANONICAL_NAME = {"page"} # page uses page_id as unique identifier
TYPE_REQUIRED = {
"document": ["doc_id", "original_filename", "raw_path", "sha256", "page_count", "collection", "document_class", "content_classification", "pages"],
"page": ["page_id", "doc_id", "page_number", "png_path", "vision_model", "page_type", "content_classification", "entities_extracted"],
"entity": ["entity_class"], # plus class-specific id
"table": ["table_id", "source_doc", "spans_pages"],
"image": ["image_id", "image_type", "source_page", "bbox_on_page", "vision_description"],
"evidence": ["evidence_id", "evidence_grade", "evidence_class", "source_page", "chain_of_custody", "supports_claims"],
"witness_analysis": ["witness_id", "witness_person", "event_witnessed", "statements", "verdict"],
"timeline": ["timeline_scope", "period", "entries"],
"hypothesis": ["hypothesis_id", "hypothesis_class", "status", "falsification_tests", "evidence_for", "evidence_against"],
"actor_profile": ["actor_profile_id", "actor", "motive", "means", "opportunity", "modus_operandi"],
"gap": ["gap_id", "gap_class", "description", "detected_in", "severity"],
"relation": ["relation_id", "relation_class", "nodes", "connection_description", "confidence_band"],
"case_report": ["case_id", "chapters", "quality_rubrics", "overall_quality_score"],
"residual_uncertainty": ["unknowns_known", "calibration_table", "what_would_change_conclusion"],
"index": ["stats", "hubs"],
"log": [],
}
ENTITY_CLASS_ID = {
"person": "person_id",
"organization": "organization_id",
"location": "location_id",
"event": "event_id",
"uap_object": "uap_object_id",
"vehicle": "vehicle_id",
"operation": "operation_id",
"concept": "concept_id",
}
# wiki-link namespace → directory under UFO_ROOT
NAMESPACE_DIR = {
"people": "wiki/entities/people",
"org": "wiki/entities/organizations",
"loc": "wiki/entities/locations",
"event": "wiki/entities/events",
"uap": "wiki/entities/uap-objects",
"vehicle": "wiki/entities/vehicles",
"op": "wiki/entities/operations",
"concept": "wiki/entities/concepts",
"table": "wiki/tables",
"image": "wiki/images",
"evidence": "case/evidence",
"witness": "case/witnesses",
"hypothesis": "case/hypotheses",
"profile": "case/profiles",
"gap": "case/gaps",
"relation": "case/connect-the-dots",
"case": "case", # for [[case/case-report]], [[case/residual-uncertainty]]
}
WIKI_LINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_md(path: Path) -> tuple[dict, str]:
content = path.read_text(encoding="utf-8")
if not content.startswith("---"):
return {}, content
end = content.find("---", 4)
if end == -1:
return {}, content
try:
fm = yaml.safe_load(content[3:end].strip()) or {}
except yaml.YAMLError as e:
return {"_yaml_error": str(e)}, content[end + 3 :]
return fm, content[end + 3 :].lstrip("\n")
def write_md(path: Path, fm: dict, body: str) -> bool:
yaml_str = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
new_content = f"---\n{yaml_str}---\n\n{body}" if not body.startswith("\n") else f"---\n{yaml_str}---\n{body}"
if path.exists() and path.read_text(encoding="utf-8") == new_content:
return False
path.write_text(new_content, encoding="utf-8")
return True
def iter_md_files(scope: str) -> list[Path]:
"""List all .md files under wiki/ and/or case/."""
out: list[Path] = []
if scope in ("wiki", "all"):
out.extend(WIKI_BASE.rglob("*.md"))
if scope in ("case", "all"):
out.extend(CASE_BASE.rglob("*.md"))
return sorted(out)
def resolve_link(target: str) -> tuple[str, Path | None]:
"""Resolve a wiki-link target to a filesystem path. Returns (kind, path_or_None)."""
target = target.strip()
# Page link: <doc-id>/p<NNN>
m = re.match(r"^([a-z0-9][a-z0-9-]*)/p(\d{3})$", target)
if m:
doc_id, padded = m.group(1), m.group(2)
return ("page", UFO_ROOT / "wiki" / "pages" / doc_id / f"p{padded}.md")
# Namespaced link: <ns>/<id>
if "/" in target:
ns, rest = target.split("/", 1)
if ns in NAMESPACE_DIR:
return (ns, UFO_ROOT / NAMESPACE_DIR[ns] / f"{rest}.md")
# Bare doc_id
candidate = UFO_ROOT / "wiki" / "documents" / f"{target}.md"
return ("document", candidate)
def collect_inventory(scope: str) -> dict:
"""Walk all .md files; return inventory of frontmatters and wiki-links."""
files = iter_md_files(scope)
inv = {
"files": [],
"by_path": {},
"links_out": defaultdict(list), # source_path → [(target, resolved_path)]
"links_in": defaultdict(list), # target_path_str → [source_path]
"entity_files": {}, # canonical_id → path (for dedup detection)
"canonical_name_index": defaultdict(list), # name → [paths]
"page_files_by_doc": defaultdict(list), # doc_id → [(page_num, path)]
}
for path in files:
fm, body = read_md(path)
rel = path.relative_to(UFO_ROOT)
inv["files"].append(path)
inv["by_path"][str(path)] = {"fm": fm, "body": body, "rel": rel}
if fm.get("type") == "page":
doc_id = fm.get("doc_id", "")
page_num = fm.get("page_number")
if doc_id and isinstance(page_num, int):
inv["page_files_by_doc"][doc_id].append((page_num, path))
# Track canonical name uniqueness
cname = fm.get("canonical_name") or fm.get("canonical_title")
if cname:
inv["canonical_name_index"][cname].append(path)
# Find all wiki-links in body
for match in WIKI_LINK_RE.findall(body):
kind, resolved = resolve_link(match)
inv["links_out"][str(path)].append({"target": match, "kind": kind, "resolved": resolved})
if resolved is not None:
inv["links_in"][str(resolved)].append(path)
return inv
def validate_required_fields(fm: dict, path: Path) -> list[str]:
"""Return list of missing-field errors."""
errors: list[str] = []
# YAML parse error
if "_yaml_error" in fm:
errors.append(f"yaml-parse-error: {fm['_yaml_error']}")
return errors
# Universal
for f in UNIVERSAL_REQUIRED:
if f not in fm:
errors.append(f"missing-universal-field: {f}")
if fm.get("type") not in TYPES_WITHOUT_CANONICAL_NAME:
if not any(k in fm for k in NAME_FIELDS):
errors.append(f"missing-name-field: need one of {NAME_FIELDS}")
# Type-specific
t = fm.get("type")
if t in TYPE_REQUIRED:
for f in TYPE_REQUIRED[t]:
if f not in fm or fm[f] is None or fm[f] == []:
errors.append(f"missing-{t}-field: {f}")
# Entity-specific id field
if t == "entity":
cls = fm.get("entity_class")
if cls in ENTITY_CLASS_ID:
id_field = ENTITY_CLASS_ID[cls]
if id_field not in fm:
errors.append(f"missing-entity-id: {id_field} for entity_class={cls}")
else:
errors.append(f"unknown-entity-class: {cls!r}")
# Evidence: grade A → ≥3 custody, B → ≥2, C → ≥1
if t == "evidence":
grade = fm.get("evidence_grade")
custody = fm.get("chain_of_custody") or []
min_steps = {"A": 3, "B": 2, "C": 1}.get(grade, 0)
if len(custody) < min_steps:
errors.append(f"evidence-grade-{grade}-needs-{min_steps}-custody-steps (has {len(custody)})")
# Hypothesis posterior > 0.50 → ≥2 evidence_for
if t == "hypothesis":
post = fm.get("posterior_probability") or 0
if isinstance(post, (int, float)) and post > 0.50:
ev_for = fm.get("evidence_for") or []
if len(ev_for) < 2:
errors.append(f"hypothesis-posterior-{post}-needs-2-evidence_for (has {len(ev_for)})")
return errors
def validate_page_sequences(inv: dict) -> list[str]:
"""For each document, pages must be 1..page_count contiguous."""
errors = []
for path_str, info in inv["by_path"].items():
fm = info["fm"]
if fm.get("type") != "document":
continue
doc_id = fm.get("doc_id")
page_count = fm.get("page_count")
if not doc_id or not isinstance(page_count, int):
continue
actual = inv["page_files_by_doc"].get(doc_id, [])
actual_nums = sorted({n for n, _ in actual})
expected = list(range(1, page_count + 1))
missing = set(expected) - set(actual_nums)
extra = set(actual_nums) - set(expected)
if missing or extra:
errors.append(f"doc {doc_id}: page sequence broken (missing={sorted(missing)}, extra={sorted(extra)})")
return errors
def validate_canonical_uniqueness(inv: dict) -> list[str]:
"""Two distinct files cannot share canonical_name without disambiguation_note."""
errors = []
for name, paths in inv["canonical_name_index"].items():
if len(paths) <= 1:
continue
# Allow duplicates if ALL files declare disambiguation_note
all_have_note = all(inv["by_path"][str(p)]["fm"].get("disambiguation_note") for p in paths)
if not all_have_note:
rels = [str(p.relative_to(UFO_ROOT)) for p in paths]
errors.append(f"duplicate-canonical-name {name!r}: in {rels}")
return errors
def validate_links(inv: dict) -> tuple[list[str], list[str]]:
"""Check that every wiki-link resolves."""
broken = []
warned = []
for source_path_str, links in inv["links_out"].items():
for ln in links:
target_path = ln["resolved"]
if target_path is None:
broken.append(f"unparseable-link in {Path(source_path_str).relative_to(UFO_ROOT)}: [[{ln['target']}]]")
continue
if not target_path.exists():
broken.append(f"broken-link in {Path(source_path_str).relative_to(UFO_ROOT)}: [[{ln['target']}]] → {target_path.relative_to(UFO_ROOT)}")
return broken, warned
def detect_orphans(inv: dict) -> list[str]:
"""Entity files with zero inbound links (not referenced anywhere)."""
orphans = []
for path_str, info in inv["by_path"].items():
fm = info["fm"]
if fm.get("type") != "entity":
continue
path = Path(path_str)
if not inv["links_in"].get(str(path)):
rel = path.relative_to(UFO_ROOT)
orphans.append(f"orphan: {rel}")
return orphans
# ----------------------------------------------------------------------
# Backlink rebuild
# ----------------------------------------------------------------------
def _canonicalize_name(name: str) -> str:
"""Same algorithm used by script 03 (kebab-case ASCII-fold)."""
import unicodedata as ud
nfkd = ud.normalize("NFKD", name or "")
ascii_str = "".join(c for c in nfkd if not ud.combining(c))
lower = ascii_str.lower()
replaced = re.sub(r"[^a-z0-9-]", "-", lower)
collapsed = re.sub(r"-+", "-", replaced).strip("-")
if collapsed and collapsed[0].isdigit():
collapsed = "x-" + collapsed
return collapsed
PAGE_CLASS_TO_ENTITY_CLASS = {
"people": "person",
"organizations": "organization",
"locations": "location",
"vehicles": "vehicle",
"operations": "operation",
"concepts": "concept",
}
def _build_alias_index(inv: dict) -> dict[tuple[str, str], Path]:
"""Build {(entity_class, alias_key): entity_path} where alias_key is the
canonicalized form of every name/alias/canonical_name/concept_id under that
entity. Used to resolve free-text entity names extracted by Haiku back to
the curated entity file (which may have a friendlier canonical_id).
"""
index: dict[tuple[str, str], Path] = {}
for path_str, info in inv["by_path"].items():
fm = info["fm"]
if fm.get("type") != "entity":
continue
ec = fm.get("entity_class")
if not ec:
continue
keys: set[str] = set()
# canonical name + aliases + canonical_id itself
cname = fm.get("canonical_name")
if cname:
keys.add(_canonicalize_name(cname))
for alias in (fm.get("aliases") or []):
if isinstance(alias, str):
keys.add(_canonicalize_name(alias))
id_field = ENTITY_CLASS_ID.get(ec)
if id_field and id_field in fm:
keys.add(_canonicalize_name(fm[id_field]))
# Also include filename stem
keys.add(_canonicalize_name(Path(path_str).stem))
for key in keys:
if key:
index[(ec, key)] = Path(path_str)
return index
def rebuild_backlinks(inv: dict, dry_run: bool) -> tuple[int, int]:
"""For each entity file, materialize mentioned_in[] from page entities_extracted.
Resolution of "free-text entity name from Haiku" → "curated entity file" uses
the alias index (canonical_name + aliases + canonical_id all match).
Returns (entities_updated, entities_unchanged).
"""
updated = unchanged = 0
alias_index = _build_alias_index(inv)
# entity_file_path → list[(page_id, doc_id, role)]
mentions_by_entity: dict[str, list[tuple[str, str, str]]] = defaultdict(list)
for path_str, info in inv["by_path"].items():
fm = info["fm"]
if fm.get("type") != "page":
continue
page_id = fm.get("page_id", "")
doc_id = fm.get("doc_id", "")
ents = fm.get("entities_extracted") or {}
for cls, entries in ents.items():
if cls not in PAGE_CLASS_TO_ENTITY_CLASS:
continue
ec = PAGE_CLASS_TO_ENTITY_CLASS[cls]
for entry in (entries or []):
if not isinstance(entry, dict):
continue
name = entry.get("name")
if not name:
continue
key = _canonicalize_name(name)
target = alias_index.get((ec, key))
if not target:
continue
role = entry.get("role_in_page", "mentioned") if cls == "people" else "mentioned"
mentions_by_entity[str(target)].append((page_id, doc_id, role))
# Walk all entities and write mentioned_in[]
for path_str, info in inv["by_path"].items():
fm = info["fm"]
if fm.get("type") != "entity":
continue
ec = fm.get("entity_class")
if ec not in PAGE_CLASS_TO_ENTITY_CLASS.values():
# event, uap_object: their links come via documented_in/observed_in_event, not page entities_extracted
continue
mentions_raw = mentions_by_entity.get(path_str, [])
per_page: dict[str, dict] = {}
for page_id, doc_id, role in mentions_raw:
if page_id not in per_page:
per_page[page_id] = {"page": f"[[{page_id}]]", "mention_count": 0, "role_in_page": role}
per_page[page_id]["mention_count"] += 1
mentioned_in = sorted(per_page.values(), key=lambda x: -x["mention_count"])
total = sum(x["mention_count"] for x in mentioned_in)
unique_docs = {pg.split("/", 1)[0] for pg in per_page.keys()}
new_fm = dict(fm)
new_fm["mentioned_in"] = mentioned_in
new_fm["total_mentions"] = total
new_fm["documents_count"] = len(unique_docs)
# Idempotency: only bump last_lint if the substantive data changed
prev_lint = fm.get("last_lint")
snapshot_prev = {k: v for k, v in fm.items() if k != "last_lint"}
snapshot_new = {k: v for k, v in new_fm.items() if k != "last_lint"}
if snapshot_prev == snapshot_new:
unchanged += 1
continue
new_fm["last_lint"] = utc_now_iso()
if dry_run:
updated += 1
else:
changed = write_md(Path(path_str), new_fm, info["body"])
if changed:
updated += 1
else:
unchanged += 1
return updated, unchanged
# ----------------------------------------------------------------------
# Main
# ----------------------------------------------------------------------
def main():
ap = argparse.ArgumentParser(description="Lint wiki/case + rebuild backlinks.")
ap.add_argument("--scope", choices=["wiki", "case", "all"], default="all", help="scope to scan")
ap.add_argument("--fix", action="store_true", help="actually rewrite backlinks (default = report only)")
ap.add_argument("--strict", action="store_true", help="exit non-zero on any error")
args = ap.parse_args()
print(f"Scanning scope={args.scope}...", flush=True)
inv = collect_inventory(args.scope)
print(f" files: {len(inv['files'])}", flush=True)
all_errors: list[str] = []
all_warnings: list[str] = []
# 1. Required fields
field_errors = []
for path_str, info in inv["by_path"].items():
for err in validate_required_fields(info["fm"], Path(path_str)):
field_errors.append(f"{Path(path_str).relative_to(UFO_ROOT)}: {err}")
all_errors.extend(field_errors)
# 2. Page sequence
page_errors = validate_page_sequences(inv)
all_errors.extend(page_errors)
# 3. Canonical uniqueness
name_errors = validate_canonical_uniqueness(inv)
all_errors.extend(name_errors)
# 4. Links
broken, link_warnings = validate_links(inv)
all_errors.extend(broken)
all_warnings.extend(link_warnings)
# 5. Orphans (warning, not error)
orphans = detect_orphans(inv)
all_warnings.extend(orphans)
# 6. Rebuild backlinks
updated, unchanged = rebuild_backlinks(inv, dry_run=not args.fix)
# Report
print("\n=== LINT REPORT ===")
print(f" files scanned: {len(inv['files'])}")
print(f" errors: {len(all_errors)}")
for e in all_errors[:50]:
print(f" ✗ {e}")
if len(all_errors) > 50:
print(f" … and {len(all_errors) - 50} more")
print(f" warnings: {len(all_warnings)}")
for w in all_warnings[:20]:
print(f" ⚠ {w}")
if len(all_warnings) > 20:
print(f" … and {len(all_warnings) - 20} more")
action = "would-update" if not args.fix else "updated"
print(f" backlinks: {action}={updated}, unchanged={unchanged}")
# Log entry
if args.fix:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(f"\n## {utc_now_iso()} — LINT (Phase 8)\n")
fh.write(f"- operator: archivist\n- scope: {args.scope}\n- files_scanned: {len(inv['files'])}\n")
fh.write(f"- errors: {len(all_errors)}\n- warnings: {len(all_warnings)}\n")
fh.write(f"- backlinks_updated: {updated}\n- backlinks_unchanged: {unchanged}\n")
if all_errors:
fh.write("- top_errors:\n")
for e in all_errors[:10]:
fh.write(f" - {e}\n")
if args.strict and all_errors:
sys.exit(1)
if __name__ == "__main__":
main()

252
scripts/05-crop-bboxes.py Executable file
View file

@ -0,0 +1,252 @@
#!/usr/bin/env python3
"""
05-crop-bboxes.py — Eager crop generation from bounding boxes
For each page.md, read `images_detected[]`, `tables_detected[]`, and
`signatures_observed[]` (the elements whose visual content is worth showing
inline in chat replies). For each element with a bbox, crop the corresponding
region from the page PNG using Pillow and save to:
processing/crops/<doc-id>/<crop-id>.png
Where <crop-id> follows the convention:
IMG-<DOC>-p<NNN>-<NN> for images_detected
TBL-<DOC>-p<NNN>-<NN> for tables_detected
SIG-<DOC>-p<NNN>-<NN> for signatures_observed
Padding: 1% of page dimensions around each bbox to avoid tight clipping.
Idempotent: skips crops whose output PNG already exists with non-zero size
(unless --force).
Usage:
./05-crop-bboxes.py # all docs
./05-crop-bboxes.py --doc-id <id> # single doc
./05-crop-bboxes.py --force # overwrite existing crops
"""
from __future__ import annotations
import argparse
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
try:
from PIL import Image
except ImportError:
sys.stderr.write("Missing pillow. Run: pip3 install pillow\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
PAGES_BASE = UFO_ROOT / "wiki" / "pages"
PNG_BASE = UFO_ROOT / "processing" / "png"
CROPS_BASE = UFO_ROOT / "processing" / "crops"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
PADDING_FRACTION = 0.01 # 1% padding around bbox
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_frontmatter(path: Path) -> dict:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}
end = c.find("---", 4)
if end == -1:
return {}
try:
return yaml.safe_load(c[3:end].strip()) or {}
except yaml.YAMLError:
return {}
def doc_id_short(doc_id: str) -> str:
"""Compact uppercase tag for use inside crop ids.
'dow-uap-d54-mission-report-mediterranean-sea-na' → 'DOWD54'
'doc-65-hs1-...' → 'D65HS1' (drop common prefixes, keep first signal)
"""
s = doc_id.upper()
# Remove common prefixes / fillers
for prefix in ("DOW-UAP-", "DOS-UAP-", "NASA-UAP-", "FBI-PHOTO-", "DOC-"):
if s.startswith(prefix):
s = s[len(prefix):]
break
# Take first ~6 alphanumeric chars
s = re.sub(r"[^A-Z0-9]", "", s)[:8]
return s or "X"
def make_crop_id(prefix: str, doc_id: str, page_num: int, idx: int) -> str:
return f"{prefix}-{doc_id_short(doc_id)}-p{page_num:03d}-{idx:02d}"
def crop_bbox(
*,
src_png: Path,
dest_png: Path,
bbox: dict,
padding: float,
force: bool,
) -> tuple[bool, str | None]:
"""Crop src_png by bbox and save to dest_png.
Returns (created, reason_skipped_or_error).
"""
if not force and dest_png.exists() and dest_png.stat().st_size > 0:
return (False, "exists")
try:
with Image.open(src_png) as im:
W, H = im.size
x = float(bbox.get("x", 0))
y = float(bbox.get("y", 0))
w = float(bbox.get("w", 0))
h = float(bbox.get("h", 0))
if w <= 0 or h <= 0:
return (False, "zero-size-bbox")
# Apply padding
x_pad = max(0.0, x - padding)
y_pad = max(0.0, y - padding)
w_pad = min(1.0 - x_pad, w + 2 * padding)
h_pad = min(1.0 - y_pad, h + 2 * padding)
# Pixel coords
px = int(round(x_pad * W))
py = int(round(y_pad * H))
pw = max(1, int(round(w_pad * W)))
ph = max(1, int(round(h_pad * H)))
crop = im.crop((px, py, px + pw, py + ph))
dest_png.parent.mkdir(parents=True, exist_ok=True)
crop.save(dest_png, "PNG", optimize=True)
return (True, None)
except Exception as e:
return (False, f"error: {e}")
def process_page(page_md: Path, force: bool) -> dict:
"""Returns counts {created, skipped, error} for this page."""
fm = read_frontmatter(page_md)
if not fm or fm.get("type") != "page":
return {"created": 0, "skipped": 0, "error": 0}
doc_id = fm.get("doc_id", "")
page_id = fm.get("page_id", "")
png_rel = fm.get("png_path", "")
if not doc_id or not page_id or not png_rel:
return {"created": 0, "skipped": 0, "error": 0}
src_png = (page_md.parent / png_rel).resolve()
if not src_png.exists():
sys.stderr.write(f" ✗ source PNG missing: {src_png}\n")
return {"created": 0, "skipped": 0, "error": 1}
page_num = int(fm.get("page_number", 0))
counts = {"created": 0, "skipped": 0, "error": 0}
# 1. Images detected
for idx, item in enumerate(fm.get("images_detected") or [], start=1):
bbox = item.get("bbox")
if not bbox:
continue
crop_id = make_crop_id("IMG", doc_id, page_num, idx)
dest = CROPS_BASE / doc_id / f"{crop_id}.png"
created, reason = crop_bbox(
src_png=src_png, dest_png=dest, bbox=bbox,
padding=PADDING_FRACTION, force=force,
)
if created:
counts["created"] += 1
elif reason == "exists":
counts["skipped"] += 1
else:
counts["error"] += 1
# 2. Tables detected
for idx, item in enumerate(fm.get("tables_detected") or [], start=1):
bbox = item.get("bbox")
if not bbox:
continue
crop_id = make_crop_id("TBL", doc_id, page_num, idx)
dest = CROPS_BASE / doc_id / f"{crop_id}.png"
created, reason = crop_bbox(
src_png=src_png, dest_png=dest, bbox=bbox,
padding=PADDING_FRACTION * 2, force=force,
)
if created:
counts["created"] += 1
elif reason == "exists":
counts["skipped"] += 1
else:
counts["error"] += 1
# 3. Signatures observed
for idx, item in enumerate(fm.get("signatures_observed") or [], start=1):
bbox = item.get("bbox")
if not bbox:
continue
crop_id = make_crop_id("SIG", doc_id, page_num, idx)
dest = CROPS_BASE / doc_id / f"{crop_id}.png"
created, reason = crop_bbox(
src_png=src_png, dest_png=dest, bbox=bbox,
padding=PADDING_FRACTION, force=force,
)
if created:
counts["created"] += 1
elif reason == "exists":
counts["skipped"] += 1
else:
counts["error"] += 1
return counts
def main():
ap = argparse.ArgumentParser(description="Eager-crop bounding boxes from page PNGs.")
g = ap.add_mutually_exclusive_group()
g.add_argument("--doc-id", help="restrict to a single doc_id")
g.add_argument("--all", action="store_true", help="process all pages (default)")
ap.add_argument("--force", action="store_true", help="overwrite existing crops")
args = ap.parse_args()
if args.doc_id:
glob = PAGES_BASE / args.doc_id / "*.md"
pages = sorted(Path(str(glob).rsplit("/", 1)[0]).glob("*.md"))
else:
pages = sorted(PAGES_BASE.rglob("*.md"))
totals = {"created": 0, "skipped": 0, "error": 0}
doc_summary: dict[str, dict] = {}
print(f"Cropping bboxes from {len(pages)} page(s)...", flush=True)
for page_md in pages:
c = process_page(page_md, args.force)
doc_id = page_md.parent.name
doc_summary.setdefault(doc_id, {"created": 0, "skipped": 0, "error": 0})
for k in c:
totals[k] += c[k]
doc_summary[doc_id][k] += c[k]
for doc_id, c in sorted(doc_summary.items()):
if c["created"] or c["error"]:
print(f" {doc_id}: created={c['created']} skipped={c['skipped']} error={c['error']}", flush=True)
print(f"\nTotal: created={totals['created']}, skipped={totals['skipped']}, error={totals['error']}", flush=True)
if totals["created"] > 0 or totals["error"] > 0:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(f"\n## {utc_now_iso()} — CROP BBOXES (Phase 5.5)\n")
fh.write(f"- operator: archivist\n- script: scripts/05-crop-bboxes.py\n")
fh.write(f"- target: {args.doc_id or '(all)'}\n")
fh.write(f"- created: {totals['created']}\n- skipped: {totals['skipped']}\n- error: {totals['error']}\n")
if __name__ == "__main__":
main()

422
scripts/06-graph-export.py Executable file
View file

@ -0,0 +1,422 @@
#!/usr/bin/env python3
"""
06-graph-export.py — Export a graph JSON of the wiki for client-side viz
Walks wiki/ and case/, builds:
nodes:
- one per document
- one per entity (person, organization, location, event, uap_object,
vehicle, operation, concept)
- one per gap, evidence, hypothesis, witness, profile (case artifacts)
edges:
- document → page (contains)
- page → entity (mentions, via entities_extracted)
- entity → entity (related_*, observed_in_event, primary_location, etc.)
- relation node nodes[] → its members (connect-the-dots)
- gap.detected_in[] → page/document
Output:
wiki/graph.json
The JSON shape is friendly to Cytoscape / Sigma.js / react-flow. Each node
carries `type`, `label`, `entity_class` (when applicable), and `data` with the
frontmatter fields useful for filters (collection, country, date, confidence,
etc.).
Usage:
./06-graph-export.py
./06-graph-export.py --out /path/to/graph.json
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from collections import defaultdict
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
WIKI_BASE = UFO_ROOT / "wiki"
CASE_BASE = UFO_ROOT / "case"
DEFAULT_OUT = WIKI_BASE / "graph.json"
LOG_PATH = WIKI_BASE / "log.md"
WIKI_LINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_md(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end == -1:
return {}, c
try:
return (yaml.safe_load(c[3:end].strip()) or {}), c[end + 3 :]
except yaml.YAMLError:
return {}, c[end + 3 :]
def parse_wiki_link(target: str) -> tuple[str, str] | None:
"""Extract (namespace, id) from a wiki-link target string.
Returns None for unrecognized targets.
"""
t = target.strip()
# Page link: doc-id/pNNN
m = re.match(r"^([a-z0-9][a-z0-9-]*)/p\d{3}$", t)
if m:
return ("page", t)
if "/" in t:
ns, rest = t.split("/", 1)
return (ns, rest)
# Bare doc_id
return ("document", t)
def node_id_from_link(target: str) -> str | None:
"""Compute the canonical node id used in the graph from a wiki-link target."""
parsed = parse_wiki_link(target)
if not parsed:
return None
ns, rest = parsed
return f"{ns}:{rest}"
def make_node(node_id: str, ntype: str, label: str, **extra) -> dict:
n = {"id": node_id, "type": ntype, "label": label}
if extra:
n["data"] = extra
return n
def make_edge(source: str, target: str, kind: str, weight: float = 1.0) -> dict:
return {"source": source, "target": target, "kind": kind, "weight": weight}
# ----------------------------------------------------------------------
def collect_documents(graph: dict):
docs_dir = WIKI_BASE / "documents"
for p in sorted(docs_dir.glob("*.md")):
fm, _ = read_md(p)
if fm.get("type") != "document":
continue
doc_id = fm.get("doc_id", p.stem)
node_id = f"document:{doc_id}"
graph["nodes"][node_id] = make_node(
node_id, "document",
fm.get("canonical_title", doc_id),
collection=fm.get("collection"),
document_class=fm.get("document_class"),
page_count=fm.get("page_count"),
content_classification=fm.get("content_classification"),
document_date=fm.get("document_date"),
highest_classification=fm.get("highest_classification"),
)
# document → page edges
for page_ref in (fm.get("pages") or []):
page_id_link = page_ref.get("page_id") if isinstance(page_ref, dict) else None
if not page_id_link:
continue
# extract the [[doc/pNNN]] target
m = WIKI_LINK_RE.search(page_id_link)
if not m:
continue
target = node_id_from_link(m.group(1))
if target:
graph["edges"].append(make_edge(node_id, target, "contains"))
# document → key entities
key_entities = fm.get("key_entities") or {}
for cls, refs in key_entities.items():
for ref in (refs or []):
if not isinstance(ref, str):
continue
m = WIKI_LINK_RE.search(ref)
if m:
tgt = node_id_from_link(m.group(1))
if tgt:
graph["edges"].append(make_edge(node_id, tgt, "key-entity"))
# gaps_flagged
for ref in (fm.get("gaps_flagged") or []):
if isinstance(ref, str):
m = WIKI_LINK_RE.search(ref)
if m:
tgt = node_id_from_link(m.group(1))
if tgt:
graph["edges"].append(make_edge(node_id, tgt, "flags-gap"))
def collect_pages(graph: dict):
for p in sorted((WIKI_BASE / "pages").rglob("*.md")):
fm, _ = read_md(p)
if fm.get("type") != "page":
continue
page_id = fm.get("page_id")
if not page_id:
continue
node_id = f"page:{page_id}"
graph["nodes"][node_id] = make_node(
node_id, "page",
f"{fm.get('doc_id', '?')} p{fm.get('page_number', '?'):>03}",
page_number=fm.get("page_number"),
page_type=fm.get("page_type"),
content_classification=fm.get("content_classification"),
language_detected=fm.get("language_detected"),
)
# page → entities (mentions, via entities_extracted)
page_entity_map = {
"people": "people",
"organizations": "org",
"locations": "loc",
"vehicles": "vehicle",
"operations": "op",
"concepts": "concept",
}
ents = fm.get("entities_extracted") or {}
# We can't easily resolve the canonicalized id here without doing the
# alias-match lookup. The lint script's `mentioned_in[]` is the
# source-of-truth for who-mentions-who, so we'll add edges from
# entity → page later, not page → entity here.
def collect_entities(graph: dict):
entities_root = WIKI_BASE / "entities"
for p in sorted(entities_root.rglob("*.md")):
fm, _ = read_md(p)
if fm.get("type") != "entity":
continue
ec = fm.get("entity_class")
id_field_map = {
"person": "person_id",
"organization": "organization_id",
"location": "location_id",
"event": "event_id",
"uap_object": "uap_object_id",
"vehicle": "vehicle_id",
"operation": "operation_id",
"concept": "concept_id",
}
ns_map = {
"person": "people",
"organization": "org",
"location": "loc",
"event": "event",
"uap_object": "uap",
"vehicle": "vehicle",
"operation": "op",
"concept": "concept",
}
id_field = id_field_map.get(ec)
ns = ns_map.get(ec)
if not id_field or not ns:
continue
eid = fm.get(id_field) or p.stem
node_id = f"{ns}:{eid}"
graph["nodes"][node_id] = make_node(
node_id, ns,
fm.get("canonical_name", eid),
entity_class=ec,
aliases=fm.get("aliases"),
country=fm.get("country"),
location_type=fm.get("location_type"),
organization_type=fm.get("organization_type"),
shape=fm.get("shape"),
color=fm.get("color"),
event_class=fm.get("event_class"),
date_start=fm.get("date_start"),
concept_class=fm.get("concept_class"),
total_mentions=fm.get("total_mentions"),
enrichment_status=fm.get("enrichment_status"),
)
# entity → page edges (via mentioned_in[])
for m in (fm.get("mentioned_in") or []):
if not isinstance(m, dict):
continue
link = m.get("page")
if not link:
continue
mm = WIKI_LINK_RE.search(link)
if mm:
tgt = node_id_from_link(mm.group(1))
if tgt:
graph["edges"].append(make_edge(
node_id, tgt, "mentioned-in",
weight=m.get("mention_count", 1),
))
# event-specific links
if ec == "event":
pl = fm.get("primary_location")
if isinstance(pl, str):
mm = WIKI_LINK_RE.search(pl)
if mm:
tgt = node_id_from_link(mm.group(1))
if tgt:
graph["edges"].append(make_edge(node_id, tgt, "occurred-at"))
for obj in (fm.get("uap_objects") or []):
if isinstance(obj, str):
mm = WIKI_LINK_RE.search(obj)
if mm:
tgt = node_id_from_link(mm.group(1))
if tgt:
graph["edges"].append(make_edge(node_id, tgt, "observed-uap"))
# uap_object → event
if ec == "uap_object":
ev = fm.get("observed_in_event")
if isinstance(ev, str):
mm = WIKI_LINK_RE.search(ev)
if mm:
tgt = node_id_from_link(mm.group(1))
if tgt:
graph["edges"].append(make_edge(node_id, tgt, "observed-in-event"))
def collect_case_artifacts(graph: dict):
"""Add gaps, evidence, witnesses, hypotheses, profiles, relations."""
type_to_ns = {
"gap": "gap",
"evidence": "evidence",
"witness_analysis": "witness",
"hypothesis": "hypothesis",
"actor_profile": "profile",
"relation": "relation",
"case_report": "case",
"residual_uncertainty": "case",
"timeline": "timeline",
}
for p in sorted(CASE_BASE.rglob("*.md")):
fm, _ = read_md(p)
t = fm.get("type")
if t not in type_to_ns:
continue
ns = type_to_ns[t]
# ID detection
id_field = {
"gap": "gap_id",
"evidence": "evidence_id",
"witness_analysis": "witness_id",
"hypothesis": "hypothesis_id",
"actor_profile": "actor_profile_id",
"relation": "relation_id",
"case_report": "case_id",
"timeline": "scope_id",
}.get(t)
eid = (fm.get(id_field) if id_field else None) or p.stem
node_id = f"{ns}:{eid}"
graph["nodes"][node_id] = make_node(
node_id, ns,
fm.get("canonical_title", eid),
t=t,
severity=fm.get("severity"),
evidence_grade=fm.get("evidence_grade"),
status=fm.get("status"),
verdict=fm.get("verdict"),
connection_strength=fm.get("connection_strength"),
)
# gap → detected_in (pages)
if t == "gap":
for ref in (fm.get("detected_in") or []):
if isinstance(ref, str):
mm = WIKI_LINK_RE.search(ref)
if mm:
tgt = node_id_from_link(mm.group(1))
if tgt:
graph["edges"].append(make_edge(node_id, tgt, "detected-in"))
# relation → nodes[]
if t == "relation":
for ref in (fm.get("nodes") or []):
if isinstance(ref, str):
mm = WIKI_LINK_RE.search(ref)
if mm:
tgt = node_id_from_link(mm.group(1))
if tgt:
graph["edges"].append(make_edge(node_id, tgt, "relates"))
# ----------------------------------------------------------------------
def main():
ap = argparse.ArgumentParser(description="Export wiki graph (nodes + edges) as JSON.")
ap.add_argument("--out", default=str(DEFAULT_OUT), help=f"output path (default: {DEFAULT_OUT})")
args = ap.parse_args()
graph: dict = {
"generated_at": utc_now_iso(),
"wiki_version": "0.1.0",
"nodes": {}, # dict keyed by node_id for dedup, flattened to list at end
"edges": [],
}
collect_documents(graph)
collect_pages(graph)
collect_entities(graph)
collect_case_artifacts(graph)
# Dedup edges
edge_seen = set()
deduped_edges = []
for e in graph["edges"]:
key = (e["source"], e["target"], e["kind"])
if key in edge_seen:
continue
edge_seen.add(key)
deduped_edges.append(e)
graph["edges"] = deduped_edges
# Filter out edges pointing to nodes we don't have (broken refs)
valid_ids = set(graph["nodes"].keys())
pre = len(graph["edges"])
graph["edges"] = [e for e in graph["edges"] if e["source"] in valid_ids and e["target"] in valid_ids]
dropped = pre - len(graph["edges"])
nodes_list = sorted(graph["nodes"].values(), key=lambda n: (n["type"], n["id"]))
graph["nodes"] = nodes_list
# Summary by type
by_type: dict[str, int] = defaultdict(int)
for n in nodes_list:
by_type[n["type"]] += 1
edges_by_kind: dict[str, int] = defaultdict(int)
for e in graph["edges"]:
edges_by_kind[e["kind"]] += 1
out_path = Path(args.out)
out_path.parent.mkdir(parents=True, exist_ok=True)
out_path.write_text(json.dumps(graph, indent=2, ensure_ascii=False), encoding="utf-8")
print(f"Graph written to {out_path}", flush=True)
print(f" nodes total: {len(nodes_list)}", flush=True)
for t, n in sorted(by_type.items()):
print(f" {t}: {n}", flush=True)
print(f" edges total: {len(graph['edges'])} (dropped {dropped} dangling)", flush=True)
for k, n in sorted(edges_by_kind.items()):
print(f" {k}: {n}", flush=True)
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(f"\n## {utc_now_iso()} — GRAPH EXPORT\n")
fh.write(f"- operator: archivist\n- script: scripts/06-graph-export.py\n")
fh.write(f"- output: {out_path.relative_to(UFO_ROOT)}\n")
fh.write(f"- nodes: {len(nodes_list)}\n- edges: {len(graph['edges'])}\n")
if __name__ == "__main__":
main()

278
scripts/07-test-agent.py Executable file
View file

@ -0,0 +1,278 @@
#!/usr/bin/env python3
"""
07-test-agent.py — Minimal chat-agent CLI that validates the schema end-to-end
Simulates one chat-bubble round trip:
1. User asks a free-text query.
2. Agent walks wiki/ + case/ and collects relevant context markdowns.
3. Calls Claude Haiku (via claude CLI OAuth — same path as 02-vision-page.py)
with a system prompt that asks for STRUCTURED output:
{
"answer_en": "...",
"answer_pt_br": "...",
"citations": [
{
"kind": "page|crop|entity",
"page_id": "doc-id/pNNN", # for kind=page/crop
"entity_link": "[[loc/.../...]]", # for kind=entity
"png_url": "/static/png/doc-id/p-NNN.png",
"crop_url": "/static/crops/doc-id/CROP-ID.png", # if available
"bbox": {"x": .., "y": .., "w": .., "h": ..}, # if applicable
"snippet_en": "...",
"snippet_pt_br": "..."
}
]
}
4. Renders the JSON pretty-printed so the schema-to-UI contract is visible.
This is NOT the production agent — it's a smoke test that proves the wiki
schema carries everything the future chat UI will need (citations at page +
bbox, bilingual snippets, crop URLs).
Usage:
./07-test-agent.py "What UAP was observed in the Mediterranean?"
./07-test-agent.py --max-context 20 "How many redacted pages does D54 have?"
"""
from __future__ import annotations
import argparse
import json
import re
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
WIKI_BASE = UFO_ROOT / "wiki"
CASE_BASE = UFO_ROOT / "case"
PNG_BASE = UFO_ROOT / "processing" / "png"
CROPS_BASE = UFO_ROOT / "processing" / "crops"
MODEL = "haiku"
MAX_TURNS = 3
DEFAULT_MAX_CONTEXT_FILES = 12
# Future server prefixes (placeholder; real server resolves these to actual paths)
PNG_URL_PREFIX = "/static/png"
CROP_URL_PREFIX = "/static/crops"
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_md(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end == -1:
return {}, c
try:
return (yaml.safe_load(c[3:end].strip()) or {}), c[end + 3 :]
except yaml.YAMLError:
return {}, c[end + 3 :]
def tokenize(text: str) -> set[str]:
return {t.lower() for t in re.findall(r"[a-zA-Z0-9À-ſ]{3,}", text or "")}
def score_file(query_tokens: set[str], file_text: str, file_fm: dict) -> float:
"""Trivial keyword-overlap score; good enough for smoke test."""
file_tokens = tokenize(file_text)
# Boost: include canonical_name and aliases in search tokens
if file_fm:
for k in ("canonical_name", "canonical_title", "aliases"):
v = file_fm.get(k)
if isinstance(v, str):
file_tokens |= tokenize(v)
elif isinstance(v, list):
for it in v:
if isinstance(it, str):
file_tokens |= tokenize(it)
if not file_tokens:
return 0.0
overlap = len(query_tokens & file_tokens)
return overlap / max(1, len(query_tokens))
def gather_context(query: str, max_files: int) -> list[Path]:
"""Return list of markdown paths most relevant to the query, by keyword overlap."""
q_tokens = tokenize(query)
scored: list[tuple[float, Path]] = []
for base in (WIKI_BASE, CASE_BASE):
for p in base.rglob("*.md"):
if p.name == "graph.json":
continue
try:
fm, body = read_md(p)
except Exception:
continue
score = score_file(q_tokens, body, fm)
if score > 0:
scored.append((score, p))
scored.sort(key=lambda x: -x[0])
return [p for _, p in scored[:max_files]]
def crop_url_for(image_id: str) -> str:
"""Return URL for a crop image."""
# image_id format: IMG-DOCSHORT-pNNN-NN, TBL-..., SIG-...
# Convert to file path: processing/crops/<doc-id>/<image_id>.png
# but doc-id is encoded compactly in the crop_id. We need to scan instead.
matches = list(CROPS_BASE.rglob(f"{image_id}.png"))
if matches:
rel = matches[0].relative_to(UFO_ROOT / "processing" / "crops")
return f"{CROP_URL_PREFIX}/{rel}"
return ""
def page_url_for(page_id: str) -> str:
"""page_id format: <doc-id>/pNNN. PNG file: processing/png/<doc-id>/p-NNN.png"""
m = re.match(r"^(.+)/p(\d{3})$", page_id)
if not m:
return ""
doc_id, num = m.group(1), m.group(2)
return f"{PNG_URL_PREFIX}/{doc_id}/p-{num}.png"
def build_system_prompt() -> str:
return """You are a research assistant for the war.gov/ufo UAP/UFO document corpus.
The user asks a question. You receive a set of markdown files from a curated wiki (Karpathy-style LLM wiki) plus case-investigation artifacts. Each file's frontmatter carries strict provenance: doc_id, page_id, bbox coordinates, classifications, etc. Body text is bilingual (EN + PT-BR).
Your output MUST be a single JSON object with this exact shape (no markdown fence, no commentary, no preamble):
{
"answer_en": "2-5 sentence English answer grounded in the provided files. Every factual claim must be traceable to a citation below.",
"answer_pt_br": "Same answer translated to Brazilian Portuguese (pt-br). Use Brazilian vocabulary and spelling.",
"citations": [
{
"kind": "page",
"page_id": "doc-id/pNNN",
"snippet_en": "short verbatim or near-verbatim excerpt supporting the claim (English)",
"snippet_pt_br": "same in Brazilian Portuguese",
"bbox": null
},
{
"kind": "entity",
"entity_link": "[[loc/aegean-sea-off-santorini-greece]] or similar wiki-link",
"snippet_en": "...",
"snippet_pt_br": "..."
}
]
}
Rules:
- ONLY cite files that you were given. Do not invent page_ids or entity links.
- snippet_en and snippet_pt_br must be SHORT (1-2 sentences each).
- Brazilian Portuguese only for *_pt_br fields. Preserve UTF-8 accents.
- Verbatim quotes FROM the source documents stay in their original language (English) inside snippets — only the surrounding narrative is translated.
- If no file supports an answer, return: {"answer_en":"Insufficient evidence in corpus.","answer_pt_br":"Evidências insuficientes no corpus.","citations":[]}
- Output ONLY the JSON. No fence."""
def call_claude(user_prompt: str, system_prompt: str) -> dict:
cmd = [
"claude",
"-p",
"--model", MODEL,
"--output-format", "json",
"--max-turns", str(MAX_TURNS),
"--allowedTools", "Read",
"--add-dir", str(UFO_ROOT),
"--append-system-prompt", system_prompt,
"--",
user_prompt,
]
res = subprocess.run(cmd, capture_output=True, text=True, timeout=600, check=False)
if res.returncode != 0:
raise RuntimeError(f"claude CLI failed (rc={res.returncode}): {res.stderr[-1000:]}")
if not res.stdout.strip():
raise RuntimeError(f"claude CLI returned empty stdout. stderr: {res.stderr[-1000:]}")
try:
cli = json.loads(res.stdout)
except json.JSONDecodeError as e:
raise RuntimeError(f"claude CLI returned invalid JSON: {e}. stdout[:500]={res.stdout[:500]!r}")
if cli.get("is_error"):
raise RuntimeError(f"claude reported error: {cli.get('result','')[:500]}")
result = cli.get("result", "").strip()
if result.startswith("```"):
result = re.sub(r"^```(?:json)?\s*", "", result)
result = re.sub(r"\s*```$", "", result)
return {
"parsed": json.loads(result),
"meta": {
"duration_ms": cli.get("duration_ms"),
"total_cost_usd": cli.get("total_cost_usd"),
"session_id": cli.get("session_id"),
},
}
def enrich_citations(parsed: dict) -> dict:
"""Add png_url and crop_url to each page citation when possible."""
for cit in parsed.get("citations", []):
if cit.get("kind") == "page":
pid = cit.get("page_id", "")
cit["png_url"] = page_url_for(pid)
elif cit.get("kind") == "crop":
crop_id = cit.get("crop_id", "")
if crop_id:
cit["crop_url"] = crop_url_for(crop_id)
return parsed
def main():
ap = argparse.ArgumentParser(description="Minimal chat-agent smoke test for the UFO wiki.")
ap.add_argument("query", help="user question (in English or PT-BR)")
ap.add_argument("--max-context", type=int, default=DEFAULT_MAX_CONTEXT_FILES,
help=f"max number of markdown files to surface as context (default {DEFAULT_MAX_CONTEXT_FILES})")
args = ap.parse_args()
print(f"Query: {args.query}\n", flush=True)
print(f"Gathering context (max {args.max_context} files)...", flush=True)
context_files = gather_context(args.query, args.max_context)
for f in context_files:
print(f" - {f.relative_to(UFO_ROOT)}", flush=True)
if not context_files:
print(" (no relevant files found)", flush=True)
result = {"answer_en": "No relevant files found in the wiki.", "answer_pt_br": "Nenhum arquivo relevante encontrado.", "citations": []}
print("\n" + json.dumps(result, indent=2, ensure_ascii=False))
return
# Build user prompt: list of file paths for the agent to Read
file_list = "\n".join(f"- {p.relative_to(UFO_ROOT)}" for p in context_files)
user_prompt = (
f"User question:\n{args.query}\n\n"
f"Read the following files from /Users/guto/ufo/ "
f"(use the Read tool on each one as needed):\n{file_list}\n\n"
f"Then output the structured JSON answer per the system prompt."
)
print("\nCalling Haiku...", flush=True)
try:
out = call_claude(user_prompt, build_system_prompt())
except Exception as e:
sys.stderr.write(f"FATAL: {e}\n")
sys.exit(1)
parsed = enrich_citations(out["parsed"])
print(f"\n=== Agent reply (cost ${out['meta'].get('total_cost_usd', 0):.4f}, "
f"latency {out['meta'].get('duration_ms', 0)/1000:.1f}s) ===\n", flush=True)
print(json.dumps(parsed, indent=2, ensure_ascii=False))
if __name__ == "__main__":
main()

455
scripts/08-video-analysis.py Executable file
View file

@ -0,0 +1,455 @@
#!/usr/bin/env python3
"""
08-video-analysis.py — Sherlock Holmes-style deep video analysis with Gemini 3.1 Pro
For each .mp4 in /Users/guto/ufo/raw/videos/:
1. Upload to Gemini Files API
2. Wait for ACTIVE state
3. Call gemini-3.1-pro-preview with structured Sherlock prompt
4. Receive JSON containing:
- audio_transcript_verbatim (original language, with timestamps)
- vision_description (rich English description, frame-by-frame)
- vision_description_pt_br
- entities_extracted (people/voices, organizations, locations, equipment, UAP objects)
- uap_observations (shape, color, motion descriptors, sensor info, kinematics)
- timeline (events with timestamps in mm:ss)
- anomalies (sensor artifacts vs candidate phenomena, with Locard-style reasoning)
- sherlock_observations (what Holmes/Poirot/Dupin would notice — non-obvious details)
- classification_markings, redactions (visible on screen)
- confidence_band per major claim
5. Save raw JSON to processing/video-analysis/<video-id>.json
6. Write markdown to wiki/videos/<video-id>.md with bilingual frontmatter + body
Idempotent: skips videos whose .md + .json already exist (use --force to redo).
Usage:
./08-video-analysis.py # process all videos in raw/videos/
./08-video-analysis.py --video DOD_111688970.mp4 # single file
./08-video-analysis.py --max-files 3 # cap for testing
./08-video-analysis.py --model gemini-3.1-flash-lite # cheaper fallback
./08-video-analysis.py --force # re-process even if output exists
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import sys
import time
import unicodedata
from datetime import datetime, timezone
from pathlib import Path
try:
from google import genai
from google.genai import types as genai_types
except ImportError:
sys.stderr.write("Missing google-genai. Run: pip3 install google-genai\n")
sys.exit(1)
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
VIDEOS_DIR = UFO_ROOT / "raw" / "videos"
VIDEO_ANALYSIS_DIR = UFO_ROOT / "processing" / "video-analysis"
WIKI_VIDEOS_DIR = UFO_ROOT / "wiki" / "videos"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
DEFAULT_MODEL = "gemini-3.1-pro-preview"
FALLBACK_MODELS = ["gemini-3-pro-preview", "gemini-3.1-flash-lite"]
SCHEMA_VERSION = "0.1.0"
WIKI_VERSION = "0.1.0"
SHERLOCK_VIDEO_PROMPT = """You are an evidence officer in the Investigation Bureau, applying the combined methodologies of Sherlock Holmes (observation + elimination), Hercule Poirot (psychology of behavior), Auguste Dupin (ratiocination, hidden-in-plain-sight), and Edmond Locard (trace evidence, chain of custody) to a UAP/UFO video released by the U.S. Department of War at war.gov/ufo.
Your task: extract EVERYTHING from this video — visual content, audio, transcription, contextual signals, equipment, persons, and any anomalies. Be exhaustive. A great detective notices what others miss.
Output a SINGLE JSON object (no markdown fence, no preamble, no commentary) matching this exact schema:
{
"video_overview": {
"duration_seconds": <float>,
"primary_subject": "what the video is fundamentally about, one sentence",
"camera_perspective": "cockpit | ground | aerial | satellite | unknown",
"sensor_type": "visual_eo | infrared_FLIR | radar_screen | mixed | unknown",
"platform_inferred": "F/A-18 | helicopter | naval ship | satellite | unknown — based on cockpit layout, HUD elements, screen overlays etc.",
"primary_language_spoken": "en | pt | es | other | none"
},
"audio_transcript_verbatim": [
{
"t_start": "mm:ss",
"t_end": "mm:ss",
"speaker": "pilot_1 | pilot_2 | unknown_male | unknown_female | radio | narrator | n/a",
"text": "EXACT VERBATIM TRANSCRIPTION in the original language, preserve all 'uh', stutters, military jargon, callsigns. Do NOT translate.",
"confidence": "high | medium | low"
}
],
"vision_description": "Comprehensive English narrative of what is visible on screen, with timestamps. Frame-by-frame for key moments. Describe HUD overlays, sensor readouts, on-screen text, classification markings, redactions, target lock indicators, altitude/heading/airspeed values visible, any UAP morphology and motion (track its path with timestamps). 8-15 sentences for a typical 1-3 minute clip.",
"vision_description_pt_br": "Same content as vision_description, translated to Brazilian Portuguese (pt-br). Brazilian vocabulary and spelling. Preserve UTF-8 accents. Keep verbatim quotes from audio/screen in original language.",
"classification_markings_visible": [
{"t_start": "mm:ss", "t_end": "mm:ss", "level": "UNCLASSIFIED | CUI | CONFIDENTIAL | SECRET | TOP SECRET", "caveats": ["NOFORN", "ORCON", ...], "location_on_screen": "header_banner | corner | watermark | stamp"}
],
"redactions_visible": [
{"t_start": "mm:ss", "t_end": "mm:ss", "code": "(b)(1) 1.4(a) | (b)(3) | (b)(6) | other_or_unknown", "description": "what is being blacked/blurred (face, callsign, location, sensor frequency, etc.)"}
],
"entities_extracted": {
"people": [{"label": "Pilot 1 | Air Crew | Officer X", "role": "primary observer | radio operator | passenger | narrator | unknown", "voice_only": true, "first_appearance": "mm:ss"}],
"organizations": [{"name": "USS Nimitz | VFA-41 | AARO | ...", "evidence_for": "patch visible | radio callsign | hull number"}],
"locations": [{"name": "Pacific Ocean off San Diego | Strait of Hormuz | ...", "evidence_for": "coordinates on HUD | named in audio | identifiable landmark", "coordinates": {"lat": null, "lon": null, "raw_text": "..."}}],
"events": [{"label": "UAP intercept during routine patrol", "date": "YYYY-MM-DD | YYYY | NA"}],
"uap_objects": [{"shape": "sphere | disc | triangle | cylinder | cube | elongated-ellipsoid | cigar | irregular | unknown", "color": "...", "size_estimate": "...", "motion_descriptors": ["hover", "instantaneous-direction-change", "descent-from-X-to-Y", "no-visible-exhaust"]}],
"vehicles": [{"name": "F/A-18 Super Hornet | AH-64 | ...", "class": "aircraft | ship | submarine | spacecraft | satellite | ground | other"}],
"equipment_visible": [{"name": "ATFLIR pod | AESA radar | binoculars | EO/IR turret", "purpose": "sensor used to observe UAP"}],
"operations": [{"name": "Range Fouler | Operation X", "type": "military-operation | reporting-protocol | research-program | task-force | other"}],
"concepts": [{"name": "FLIR | Mark I eyeball | SECRET//NOFORN | ...", "class": "jargon | legal-instrument | sensor-term | scientific-term | other"}]
},
"uap_observation_fields": {
"first_visible_at": "mm:ss",
"last_visible_at": "mm:ss",
"duration_visible_seconds": <int>,
"shape": "sphere | disc | triangle | cylinder | cube | elongated-ellipsoid | cigar | irregular | unknown",
"color": "metallic | white | dark | luminous | ...",
"size_estimate": "1-3 m | 10 m | car-sized | etc.",
"altitude_ft": <int or null>,
"speed_kts": <int or null or "supersonic">,
"bearing_deg": <int or null>,
"distance_nm": <float or null>,
"coordinates": {"lat": null, "lon": null, "raw_text": "..."},
"maneuver_descriptors": ["hover", "instantaneous-direction-change", ...],
"sensor_observations": [{"sensor": "ATFLIR", "type": "infrared", "duration_min": "..."}],
"kinematic_anomalies": "describe any motion that defies known physics: no inertia, no sonic boom at high speed, instantaneous turns, etc."
},
"timeline": [
{"t": "mm:ss", "event": "Operator says 'There's a whole fleet of them, look on the SA'", "kind": "audio_quote | visual_event | sensor_event | redaction | classification_change"}
],
"anomalies_detected": [
{
"kind": "kinematic | sensor_artifact | atmospheric | optical_illusion | hoax_indicator | unredacted_slip | inconsistency",
"description": "what is anomalous",
"evidence": "at timestamp mm:ss the object does X while expected Y",
"candidate_explanations": ["prosaic-advanced-tech", "sensor-glare", "atmospheric-refraction", "extraterrestrial", "hoax", ...],
"confidence_band": "high | medium | low | speculation"
}
],
"sherlock_observations": [
{
"detective_lens": "holmes | poirot | dupin | locard",
"observation": "Non-obvious detail a regular viewer would miss. E.g., 'the operator's voice quaver at 1:42 increases at the moment the SA reading changes — emotional reaction precedes the sensor change by 0.5s, suggesting the operator saw the target visually before the radar updated.'",
"implication": "why it matters investigatively",
"confidence_band": "high | medium | low | speculation"
}
],
"executive_summary_en": "3-5 sentence English summary of WHAT happened in this video, suitable for a chat citation in the future Sherlock UI.",
"executive_summary_pt_br": "Same summary in Brazilian Portuguese (pt-br).",
"quality_signals": {
"video_quality_overall": "high | medium | low",
"audio_quality_overall": "high | medium | low | none",
"redaction_density": "none | light | heavy | full-blackout",
"completeness": "complete | truncated | partial",
"extraction_confidence": "high | medium | low"
},
"flags": ["sensitive-content", "audio-only", "redaction-heavy", "low-resolution", "interlaced-artifacts", "thermal-only", "no-audio"]
}
Rules:
- Output ONLY the JSON. No fence, no preamble.
- Empty arrays for not-applicable fields; null for unknown scalars. Never omit keys.
- ALL extracted text (transcript, on-screen text, callsigns) stays in ORIGINAL source language. Do NOT translate.
- ONLY `vision_description_pt_br` and `executive_summary_pt_br` are translations — Brazilian Portuguese (pt-br), NOT European Portuguese. Preserve UTF-8 accents.
- Verbatim quotes from audio INSIDE narrative fields stay in original language; only the surrounding narration is translated.
- Be EXHAUSTIVE in sherlock_observations — aim for 5-15 observations, including subtle audio cues, sensor metadata, behavioral signals.
- For anomalies, list ≥3 candidate explanations including a mundane one (sensor artifact, parallax, atmospheric).
- If duration_seconds = 0 or no content, still return the JSON with empty arrays and flags=["empty-or-corrupt"].
"""
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def filename_to_video_id(filename: str) -> str:
base = filename.rsplit(".", 1)[0]
nfkd = unicodedata.normalize("NFKD", base)
ascii_str = "".join(c for c in nfkd if not unicodedata.combining(c))
lower = ascii_str.lower()
replaced = re.sub(r"[^a-z0-9-]", "-", lower)
collapsed = re.sub(r"-+", "-", replaced).strip("-")
if collapsed and collapsed[0].isdigit():
collapsed = "vid-" + collapsed
return collapsed
def sha256_file(p: Path) -> str:
h = hashlib.sha256()
with open(p, "rb") as fh:
for chunk in iter(lambda: fh.read(65536), b""):
h.update(chunk)
return h.hexdigest()
def upload_and_wait(client, video_path: Path, poll_interval: float = 2.0, timeout: int = 600):
"""Upload video to Files API and wait until ACTIVE."""
print(f" uploading {video_path.name} ({video_path.stat().st_size/1024/1024:.1f} MB)…", flush=True)
t0 = time.time()
f = client.files.upload(file=str(video_path))
while f.state.name == "PROCESSING":
if time.time() - t0 > timeout:
raise TimeoutError(f"upload still PROCESSING after {timeout}s")
time.sleep(poll_interval)
f = client.files.get(name=f.name)
if f.state.name != "ACTIVE":
raise RuntimeError(f"file state is {f.state.name} (not ACTIVE) — cannot use")
print(f" ✓ file ready ({time.time() - t0:.1f}s upload+process)", flush=True)
return f
def call_gemini_for_video(client, video_file, model: str, attempt: int = 1, timeout: int = 240):
"""Generate Sherlock analysis for a video file. Retries with fallback models on failure.
Uses a thread-based timeout to avoid SDK hangs (genai sometimes hangs forever on rate limit)."""
import concurrent.futures
def _call():
return client.models.generate_content(
model=model,
contents=[video_file, SHERLOCK_VIDEO_PROMPT],
config=genai_types.GenerateContentConfig(
response_mime_type="application/json",
temperature=0.2,
max_output_tokens=32768,
),
)
try:
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:
future = ex.submit(_call)
try:
resp = future.result(timeout=timeout)
except concurrent.futures.TimeoutError:
raise RuntimeError(f"Gemini hung >{timeout}s — treating as failure")
return resp.text, model
except Exception as e:
if attempt < len(FALLBACK_MODELS):
next_model = FALLBACK_MODELS[attempt - 1]
print(f" ⚠ {model} failed ({e}); falling back to {next_model}", flush=True)
return call_gemini_for_video(client, video_file, next_model, attempt + 1, timeout)
raise
def render_video_md(
*,
video_id: str,
video_path: Path,
analysis: dict,
meta: dict,
now_iso: str,
) -> str:
"""Render wiki/videos/<video-id>.md (bilingual)."""
frontmatter = {
"schema_version": SCHEMA_VERSION,
"type": "video",
"video_id": video_id,
"original_filename": video_path.name,
"raw_path": f"../../raw/videos/{video_path.name}",
"sha256": sha256_file(video_path),
"size_bytes": video_path.stat().st_size,
"collection": "DOW-UAP-Video",
"vision_model": meta.get("model"),
"analyzed_at": now_iso,
# Promote video_overview
**{f"overview_{k}": v for k, v in (analysis.get("video_overview") or {}).items()},
"uap_observation_fields": analysis.get("uap_observation_fields"),
"classification_markings_visible": analysis.get("classification_markings_visible") or [],
"redactions_visible": analysis.get("redactions_visible") or [],
"entities_extracted": analysis.get("entities_extracted") or {},
"timeline": analysis.get("timeline") or [],
"anomalies_detected": analysis.get("anomalies_detected") or [],
"sherlock_observations": analysis.get("sherlock_observations") or [],
"audio_transcript_verbatim": analysis.get("audio_transcript_verbatim") or [],
"vision_description": analysis.get("vision_description", ""),
"vision_description_pt_br": analysis.get("vision_description_pt_br", ""),
"executive_summary_en": analysis.get("executive_summary_en", ""),
"executive_summary_pt_br": analysis.get("executive_summary_pt_br", ""),
"quality_signals": analysis.get("quality_signals") or {},
"flags": analysis.get("flags") or [],
"last_ingest": now_iso,
"wiki_version": WIKI_VERSION,
}
yaml_str = yaml.dump(frontmatter, allow_unicode=True, sort_keys=False, default_flow_style=False)
body = f"""# Video Analysis — {video_id}
> Source: `{video_path.name}` · Gemini model: `{meta.get("model")}` · Analyzed: {now_iso}
## Executive Summary (EN)
{analysis.get("executive_summary_en", "").strip()}
## Sumário Executivo (PT-BR)
{analysis.get("executive_summary_pt_br", "").strip()}
## Vision Description (EN)
{analysis.get("vision_description", "").strip()}
## Descrição Vision (PT-BR)
{analysis.get("vision_description_pt_br", "").strip()}
## Audio Transcript (verbatim, original language)
"""
for seg in (analysis.get("audio_transcript_verbatim") or []):
body += f"- **[{seg.get('t_start','?')}–{seg.get('t_end','?')}] {seg.get('speaker','?')}**: {seg.get('text','')} _(confidence: {seg.get('confidence','?')})_\n"
body += "\n## Sherlock Observations\n\n"
for obs in (analysis.get("sherlock_observations") or []):
body += f"- **[{obs.get('detective_lens','?')}]** {obs.get('observation','')}\n - _Implication:_ {obs.get('implication','')}\n - _Confidence:_ `{obs.get('confidence_band','?')}`\n\n"
body += "## Anomalies Detected\n\n"
for a in (analysis.get("anomalies_detected") or []):
body += f"- **{a.get('kind','?')}**: {a.get('description','')}\n - _Evidence:_ {a.get('evidence','')}\n - _Candidates:_ {', '.join(a.get('candidate_explanations', []))}\n - _Confidence:_ `{a.get('confidence_band','?')}`\n\n"
return f"---\n{yaml_str}---\n\n{body}"
def process_video(client, video_path: Path, model: str, force: bool = False) -> bool:
video_id = filename_to_video_id(video_path.name)
json_out = VIDEO_ANALYSIS_DIR / f"{video_id}.json"
md_out = WIKI_VIDEOS_DIR / f"{video_id}.md"
if not force and json_out.exists() and md_out.exists():
print(f" skip {video_id} (already processed)", flush=True)
return True
json_out.parent.mkdir(parents=True, exist_ok=True)
md_out.parent.mkdir(parents=True, exist_ok=True)
print(f"\n=== {video_path.name} → {video_id} ===", flush=True)
t0 = time.time()
try:
video_file = upload_and_wait(client, video_path)
except Exception as e:
print(f" ✗ upload failed: {e}", flush=True)
return False
print(f" calling {model} for Sherlock analysis…", flush=True)
try:
text, model_used = call_gemini_for_video(client, video_file, model)
except Exception as e:
print(f" ✗ generation failed: {e}", flush=True)
return False
# Strip optional fence
text = text.strip()
if text.startswith("```"):
text = re.sub(r"^```(?:json)?\s*", "", text)
text = re.sub(r"\s*```$", "", text)
try:
analysis = json.loads(text)
except json.JSONDecodeError as e:
print(f" ✗ JSON parse failed: {e}", flush=True)
# Save raw output anyway for inspection
json_out.with_suffix(".raw.txt").write_text(text, encoding="utf-8")
return False
meta = {"model": model_used, "duration_seconds": round(time.time() - t0, 1)}
json_out.write_text(
json.dumps({"analysis": analysis, "meta": meta}, indent=2, ensure_ascii=False),
encoding="utf-8",
)
md = render_video_md(
video_id=video_id,
video_path=video_path,
analysis=analysis,
meta=meta,
now_iso=utc_now_iso(),
)
md_out.write_text(md, encoding="utf-8")
# Clean up uploaded file to free quota
try:
client.files.delete(name=video_file.name)
except Exception:
pass
elapsed = time.time() - t0
print(f" ✓ {video_id} done ({elapsed:.1f}s)", flush=True)
return True
def main():
ap = argparse.ArgumentParser(description="Sherlock-style video analysis via Gemini 3.1 Pro.")
g = ap.add_mutually_exclusive_group()
g.add_argument("--video", help="single video filename in raw/videos/")
g.add_argument("--all", action="store_true", help="process all videos (default)")
ap.add_argument("--max-files", type=int, default=None, help="cap number of videos (for testing)")
ap.add_argument("--model", default=DEFAULT_MODEL, help=f"Gemini model (default {DEFAULT_MODEL})")
ap.add_argument("--force", action="store_true", help="reprocess existing outputs")
ap.add_argument("--sort-by-size", action="store_true", help="process smallest videos first (for cheap testing)")
args = ap.parse_args()
api_key = os.environ.get("GEMINI_API_KEY")
if not api_key:
sys.stderr.write("GEMINI_API_KEY not set\n")
sys.exit(2)
client = genai.Client(api_key=api_key)
if args.video:
v = VIDEOS_DIR / args.video
if not v.exists():
sys.stderr.write(f"Video not found: {v}\n"); sys.exit(1)
videos = [v]
else:
videos = sorted(VIDEOS_DIR.glob("*.mp4"))
if args.sort_by_size:
videos.sort(key=lambda p: p.stat().st_size)
if args.max_files:
videos = videos[: args.max_files]
print(f"Processing {len(videos)} video(s) with model {args.model}")
ok = 0
fail = []
for v in videos:
if process_video(client, v, args.model, force=args.force):
ok += 1
else:
fail.append(v.name)
print(f"\nDone. ok={ok}, failed={len(fail)}")
if fail:
print("Failed:", fail)
if ok > 0:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {utc_now_iso()} — VIDEO ANALYSIS (Gemini Phase 4)\n"
f"- operator: archivist + evidence-officer\n- script: scripts/08-video-analysis.py\n"
f"- model: {args.model}\n- videos_ok: {ok}\n- videos_failed: {len(fail)}\n"
)
if __name__ == "__main__":
main()

234
scripts/09-extract-uap-frames.py Executable file
View file

@ -0,0 +1,234 @@
#!/usr/bin/env python3
"""
09-extract-uap-frames.py — Extract key UAP frames from videos via ffmpeg
For each video analyzed by 08-video-analysis.py, read the Gemini JSON output
and extract still frames at the moments where the UAP is visible:
- first_visible_at (UAP enters frame)
- midpoint (visual peak)
- last_visible_at (UAP exits frame)
- additional samples every 1s within the visible window
Frames are written to /Users/guto/ufo/processing/uap-frames/<video-id>/
as JPEG at high quality (q=2). Filenames encode the timestamp:
frame-00-00-first.jpg
frame-00-02-mid.jpg
frame-00-04-last.jpg
frame-00-01-sample.jpg
...
The frame paths are appended back to the video's frontmatter under
`uap_frames` for traceability.
Usage:
./09-extract-uap-frames.py # all analyzed videos
./09-extract-uap-frames.py --video-id dod-111689005 # single video
./09-extract-uap-frames.py --force # re-extract
"""
from __future__ import annotations
import argparse
import json
import re
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
VIDEOS_DIR = UFO_ROOT / "raw" / "videos"
ANALYSIS_DIR = UFO_ROOT / "processing" / "video-analysis"
FRAMES_DIR = UFO_ROOT / "processing" / "uap-frames"
WIKI_VIDEOS_DIR = UFO_ROOT / "wiki" / "videos"
def parse_timestamp(ts: str) -> float | None:
"""Parse 'mm:ss' or 'h:mm:ss' or 'ss' into seconds (float)."""
if not ts:
return None
ts = ts.strip()
parts = ts.split(":")
try:
if len(parts) == 1:
return float(parts[0])
if len(parts) == 2:
return int(parts[0]) * 60 + float(parts[1])
if len(parts) == 3:
return int(parts[0]) * 3600 + int(parts[1]) * 60 + float(parts[2])
except ValueError:
return None
return None
def find_video_path(video_id: str) -> Path | None:
"""Map video_id back to the original .mp4 in raw/videos/."""
# Try a few derivations
for v in VIDEOS_DIR.glob("*.mp4"):
stem = v.stem
# video_id is lowercase kebab; raw is uppercase with underscores
normalized = re.sub(r"[^a-z0-9]+", "-", stem.lower()).strip("-")
if normalized == video_id or f"vid-{normalized}" == video_id:
return v
return None
def extract_frame(video_path: Path, timestamp_s: float, out_path: Path) -> bool:
"""Extract a single JPEG frame at the given timestamp using ffmpeg."""
out_path.parent.mkdir(parents=True, exist_ok=True)
cmd = [
"ffmpeg",
"-y", # overwrite
"-loglevel", "error",
"-ss", f"{timestamp_s:.3f}",
"-i", str(video_path),
"-frames:v", "1",
"-q:v", "2", # high quality JPEG
str(out_path),
]
res = subprocess.run(cmd, capture_output=True, text=True, check=False)
if res.returncode != 0 or not out_path.exists() or out_path.stat().st_size == 0:
sys.stderr.write(f" ✗ ffmpeg failed for {timestamp_s:.2f}s: {res.stderr[:200]}\n")
return False
return True
def collect_extraction_points(analysis: dict) -> list[tuple[float, str]]:
"""Return list of (timestamp_seconds, label) to extract."""
uap = analysis.get("uap_observation_fields") or {}
first = parse_timestamp(uap.get("first_visible_at", ""))
last = parse_timestamp(uap.get("last_visible_at", ""))
overview = analysis.get("video_overview") or {}
duration = overview.get("duration_seconds") or 0
points: list[tuple[float, str]] = []
if first is None and last is None:
# No UAP timestamps — sample evenly
if duration > 0:
for i in range(min(5, int(duration) + 1)):
t = duration * (i + 0.5) / 5
points.append((t, f"sample-{i:02d}"))
return points
first = first if first is not None else 0.0
last = last if last is not None else first + 1.0
if last <= first:
last = first + 0.5
# Always include first, mid, last
points.append((first, "first"))
mid = (first + last) / 2
if abs(mid - first) > 0.4:
points.append((mid, "mid"))
if last - first > 0.6:
points.append((last, "last"))
# Sample every ~1s within window
cur = first + 1.0
sample_idx = 0
while cur < last - 0.2:
# avoid duplicating mid
if abs(cur - mid) > 0.5:
points.append((cur, f"sample-{sample_idx:02d}"))
sample_idx += 1
cur += 1.0
return points
def format_filename(t: float, label: str) -> str:
"""frame-MM-SS[-fff]-label.jpg"""
mm = int(t // 60)
ss = t - mm * 60
return f"frame-{mm:02d}-{ss:05.2f}-{label}.jpg".replace(".", "_", 1).replace(".jpg", "", 1)[:-1] + ".jpg"
def process_video(video_id: str, force: bool = False) -> dict:
json_path = ANALYSIS_DIR / f"{video_id}.json"
if not json_path.exists():
return {"video_id": video_id, "status": "no-analysis", "frames": []}
data = json.loads(json_path.read_text(encoding="utf-8"))
analysis = data.get("analysis", {})
video_path = find_video_path(video_id)
if not video_path:
return {"video_id": video_id, "status": "no-source-video", "frames": []}
frames_subdir = FRAMES_DIR / video_id
if frames_subdir.exists() and not force and any(frames_subdir.glob("*.jpg")):
existing = [str(p.relative_to(UFO_ROOT)) for p in sorted(frames_subdir.glob("*.jpg"))]
return {"video_id": video_id, "status": "skipped-existing", "frames": existing}
points = collect_extraction_points(analysis)
if not points:
return {"video_id": video_id, "status": "no-extraction-points", "frames": []}
frames_subdir.mkdir(parents=True, exist_ok=True)
extracted = []
for t, label in points:
fname = format_filename(t, label)
out = frames_subdir / fname
if extract_frame(video_path, t, out):
extracted.append(str(out.relative_to(UFO_ROOT)))
print(f" ✓ {video_id} @ {t:6.2f}s [{label:8}] → {fname}", flush=True)
return {"video_id": video_id, "status": "ok", "frames": extracted}
def append_frames_to_md(video_id: str, frames: list[str]):
"""Add `uap_frames` list to the wiki/videos/<video-id>.md frontmatter."""
md_path = WIKI_VIDEOS_DIR / f"{video_id}.md"
if not md_path.exists():
return
content = md_path.read_text(encoding="utf-8")
if not content.startswith("---"):
return
end = content.find("---", 4)
if end == -1:
return
try:
fm = yaml.safe_load(content[3:end].strip()) or {}
except yaml.YAMLError:
return
body = content[end + 3 :].lstrip("\n")
fm["uap_frames"] = frames
fm["uap_frames_extracted_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
yaml_str = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
new = f"---\n{yaml_str}---\n\n{body}"
md_path.write_text(new, encoding="utf-8")
def main():
ap = argparse.ArgumentParser(description="Extract UAP frames from analyzed videos via ffmpeg.")
g = ap.add_mutually_exclusive_group()
g.add_argument("--video-id", help="single video id (e.g. dod-111689005)")
g.add_argument("--all", action="store_true", help="all analyzed videos (default)")
ap.add_argument("--force", action="store_true", help="re-extract even if frames exist")
args = ap.parse_args()
if args.video_id:
targets = [args.video_id]
else:
targets = sorted(p.stem for p in ANALYSIS_DIR.glob("*.json"))
print(f"Processing {len(targets)} video(s)…")
for vid in targets:
res = process_video(vid, force=args.force)
if res["status"] == "ok":
append_frames_to_md(vid, res["frames"])
print(f" → {vid}: {len(res['frames'])} frames extracted, md updated")
else:
print(f" → {vid}: {res['status']}")
if __name__ == "__main__":
main()

462
scripts/11-generate-case-images.py Executable file
View file

@ -0,0 +1,462 @@
#!/usr/bin/env python3
"""
11-generate-case-images.py — Generate "case images" (Nano Banana + Codex) per entity
For each completed video (wiki/videos/<id>.md) OR document (wiki/documents/<id>.md),
generate TWO conceptual images representing the case, using the executive_summary
and UAP observation fields as the prompt seed:
processing/case-images/<entity-id>/case-nanobanana.png
processing/case-images/<entity-id>/case-codex.png
These are "what the case might look like" reproductions — NOT evidence, NOT
real-data reconstructions. They are speculative visualizations for the chat UI
to display alongside citations (the future Sherlock chat app).
All output is tagged `synthetic: true` in the entity markdown and gets a
`synthesis_warnings` block.
Usage:
./11-generate-case-images.py --kind videos # process all wiki/videos/*.md
./11-generate-case-images.py --kind documents # process all wiki/documents/*.md
./11-generate-case-images.py --kind both # both
./11-generate-case-images.py --entity-id dod-111689005 # single entity (video or doc)
./11-generate-case-images.py --skip-codex # only Nano Banana (cheaper)
./11-generate-case-images.py --force # re-generate
"""
from __future__ import annotations
import argparse
import os
import re
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. Run: pip3 install pyyaml\n")
sys.exit(1)
try:
from google import genai
from PIL import Image as PILImage
from io import BytesIO
except ImportError:
sys.stderr.write("Missing google-genai or pillow. Run: pip3 install google-genai pillow\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
NANO_BANANA_MODEL = "gemini-3-pro-image-preview"
WIKI_VIDEOS_DIR = UFO_ROOT / "wiki" / "videos"
WIKI_DOCS_DIR = UFO_ROOT / "wiki" / "documents"
CASE_IMAGES_DIR = UFO_ROOT / "processing" / "case-images"
FRAMES_DIR = UFO_ROOT / "processing" / "uap-frames"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
NANO_BANANA_SCRIPT = Path.home() / ".claude" / "skills" / "nano-banana-pro" / "scripts" / "generate_image.py"
def find_best_frame(video_id: str) -> Path | None:
"""Return the most representative single frame jpg for a video."""
d = FRAMES_DIR / video_id
if not d.exists():
return None
candidates = sorted(d.glob("*.jpg"))
if not candidates:
return None
for keyword in ("-mi.jpg", "-firs.jpg", "-las.jpg", "-sample"):
for p in candidates:
if keyword in p.name:
return p
return candidates[0]
def find_all_frames(video_id: str, max_n: int = 5) -> list[Path]:
"""Return up to max_n frames covering the UAP timeline (first, mid, last, samples).
Order: first → samples → mid → last (chronological)."""
d = FRAMES_DIR / video_id
if not d.exists():
return []
all_jpgs = sorted(d.glob("*.jpg"))
if not all_jpgs:
return []
# Order by timestamp encoded in filename: frame-MM-SS_NN-label.jpg
def t_of(p: Path):
import re as _re
m = _re.match(r"frame-(\d+)-(\d+)_(\d+)", p.name)
if not m:
return 0
return int(m.group(1)) * 60 + int(m.group(2)) + int(m.group(3)) / 100
sorted_by_time = sorted(all_jpgs, key=t_of)
if len(sorted_by_time) <= max_n:
return sorted_by_time
# Evenly subsample
step = len(sorted_by_time) / max_n
indices = [int(i * step) for i in range(max_n)]
return [sorted_by_time[i] for i in indices]
def read_md(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end == -1:
return {}, c
try:
return (yaml.safe_load(c[3:end].strip()) or {}), c[end + 3 :].lstrip("\n")
except yaml.YAMLError:
return {}, c[end + 3 :].lstrip("\n")
def build_case_prompt(fm: dict, kind: str, has_reference_frame: bool) -> str:
"""Build the descriptive prompt for image generation. If has_reference_frame=True,
the prompt instructs to enhance the actual extracted frame; otherwise it's a
text-only conceptual scene."""
if kind == "video":
title = fm.get("video_id", "unknown")
summary = (fm.get("executive_summary_en", "") or "").strip()
uap = fm.get("uap_observation_fields") or {}
overview = {k.replace("overview_", ""): v for k, v in fm.items() if k.startswith("overview_")}
location_hint = uap.get("coordinates", {}).get("raw_text") or ""
sherlock = fm.get("sherlock_observations") or []
sherlock_summary = "; ".join(o.get("observation", "")[:120] for o in sherlock[:3])
else: # document
title = fm.get("canonical_title") or fm.get("doc_id", "unknown")
summary = (fm.get("executive_summary", "") or "").strip()
uap = {}
overview = {
"primary_subject": fm.get("document_class") or "",
"incident_date": (fm.get("war_gov") or {}).get("incident_date_official") or "",
"incident_location": (fm.get("war_gov") or {}).get("incident_location_official") or "",
}
location_hint = overview["incident_location"]
sherlock_summary = ""
shape = uap.get("shape") or "unknown"
color = uap.get("color") or "unknown"
altitude = uap.get("altitude_ft") or "unknown"
speed = uap.get("speed_kts") or "unknown"
maneuvers = ", ".join(uap.get("maneuver_descriptors") or []) or "no specific maneuvers reported"
sensor = (uap.get("sensor_observations") or [{}])
sensor_str = sensor[0].get("sensor", "unknown sensor") if sensor else "unknown sensor"
if has_reference_frame:
intro = f"""USE THE ATTACHED REFERENCE FRAME as your visual starting point. This is an actual frame extracted from the original UAP video at a moment when the UAP is visible. Enhance and re-interpret this exact scene cinematically while keeping ALL the real visual elements: same camera angle, same terrain/sensor view, same UAP position, same scale, same lighting conditions of the IR/FLIR/visible sensor.
The output should look like a CINEMATIC VERSION of the same moment captured in the frame — same scene, same UAP, but rendered with higher production value and atmospheric depth. DO NOT change the location of the UAP. DO NOT invent buildings, terrain, or atmosphere that aren't in the reference frame."""
else:
intro = """Create a photorealistic conceptual reproduction of a UAP/UFO incident scene from a U.S. Department of War declassified case."""
return f"""{intro}
CASE METADATA:
- title: {title}
- narrative: {summary[:600]}
- location: {location_hint or 'unknown'}
- primary subject: {overview.get('primary_subject', '')}
- camera vantage: {overview.get('camera_perspective', 'aerial')}
- sensor depicted: {sensor_str}
UAP CHARACTERISTICS:
- shape: {shape}
- color: {color}
- altitude: {altitude}
- speed: {speed}
- maneuvers: {maneuvers}
KEY OBSERVATIONS: {sherlock_summary[:400]}
ABSOLUTE RULES:
- Do NOT add any HUD telemetry text, altitude readouts, headings, coordinates, callsigns, or date/time stamps. These would be fabricated.
- Do NOT add classification banners with specific levels (SECRET, NOFORN, etc).
- Do NOT add ANY text at all.
- Cinematic photorealism, IMAX documentary aesthetic, somber investigative mood.
- 16:9 aspect ratio.
This is a CONCEPTUAL VISUALIZATION — artistic interpretation, not evidence."""
def build_diagram_prompt(fm: dict) -> str:
"""Sherlock investigation board annotation prompt — requires reference frame."""
sherlock = fm.get("sherlock_observations") or []
sherlock_text = " | ".join(
f"[{o.get('detective_lens','?')}] {o.get('observation','')[:100]}"
for o in sherlock[:4]
)
anomalies = fm.get("anomalies_detected") or []
anomaly_text = " | ".join(
f"{a.get('kind','?')}: {a.get('description','')[:80]}"
for a in anomalies[:2]
)
return f"""USE THE ATTACHED REFERENCE FRAME from the UAP video. Transform it into a Sherlock Holmes investigative diagram board. Keep the underlying scene (slightly brightened for legibility), and overlay handwritten-style red-pen detective annotations.
CONTEXT FROM ANALYSIS:
- Sherlock observations: {sherlock_text[:500]}
- Anomalies detected: {anomaly_text[:200]}
OVERLAY ANNOTATIONS (hand-drawn in red pen on transparent overlay):
- ◯ "UAP TARGET" circled around the most likely UAP position with arrow pointing to it
- ◯ "TRACKING LOCK" or "CROSSHAIR" if a tracking marker is visible in the frame
- ◯ "OBSERVED FROM" labeling the camera vantage (cockpit, ground, etc.)
- A dashed yellow-highlighter arrow showing the inferred motion direction with label "APPROXIMATE FLIGHT PATH"
- Bottom-left annotation in small red text summarizing ONE key observation (e.g. "IR signature: linear motion, no visible exhaust — anomalous")
- Top-right small annotation: "SOURCE: DOD VIDEO, DECLASSIFIED"
STYLE:
- Annotations look hand-drawn, slightly imperfect, like a real detective pinned the photo on a corkboard and circled clues
- The base scene from the frame stays intact (don't replace it, just annotate)
- Corkboard pins in the corners
- Slight grungy texture overlay
- Forensic investigation board / vintage detective work
ABSOLUTE RULES:
- Do NOT invent specific telemetry numbers (altitude, heading, coords, timestamps)
- All annotations are INTERPRETATIONS of what's visible, not data extracted from HUD
- Do NOT remove or alter the actual scene content"""
# Gemini SDK client (lazy)
_gemini_client = None
def _get_gemini_client():
global _gemini_client
if _gemini_client is None:
api_key = os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY")
if not api_key:
raise RuntimeError("GEMINI_API_KEY / GOOGLE_API_KEY not set")
_gemini_client = genai.Client(api_key=api_key)
return _gemini_client
def call_nano_banana(prompt: str, out_path: Path, input_images: list[Path] | None = None, resolution: str = "2K") -> bool:
"""Direct API call to Nano Banana Pro (Gemini 3 Pro Image) with support for
MULTIPLE reference images, which the official skill script does not support."""
# If 0 or 1 image, falls back to the simpler skill script (lets it handle resolution etc.)
if not input_images or len(input_images) <= 1:
cmd = [
"uv", "run", str(NANO_BANANA_SCRIPT),
"--prompt", prompt,
"--filename", str(out_path),
"--resolution", resolution,
]
if input_images:
cmd.extend(["--input-image", str(input_images[0])])
res = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
if res.returncode != 0:
sys.stderr.write(f" ✗ Nano Banana (skill) failed: {res.stderr[-400:]}\n")
return False
return out_path.exists() and out_path.stat().st_size > 0
# Multi-image path: direct SDK call
try:
client = _get_gemini_client()
pil_images = [PILImage.open(p) for p in input_images if p.exists()]
contents = [*pil_images, prompt]
response = client.models.generate_content(
model=NANO_BANANA_MODEL,
contents=contents,
)
# Extract image bytes from response.candidates[0].content.parts
for part in response.candidates[0].content.parts:
if hasattr(part, "inline_data") and part.inline_data and part.inline_data.data:
out_path.parent.mkdir(parents=True, exist_ok=True)
img = PILImage.open(BytesIO(part.inline_data.data))
img.save(out_path, "PNG")
return True
sys.stderr.write(f" ✗ Nano Banana: no image in response\n")
return False
except Exception as e:
sys.stderr.write(f" ✗ Nano Banana (SDK multi-image) failed: {e}\n")
return False
def call_codex(prompt: str, out_path: Path, input_images: list[Path] | None = None) -> bool:
"""Use codex CLI (logged via ChatGPT subscription) to generate the image.
Supports multiple reference frames — copies them all into the working dir
and references them in the prompt for image-to-image with timeline context."""
ref_section = ""
if input_images:
import shutil
existing_frames = []
for p in input_images:
if p.exists():
local = out_path.parent / p.name
if not local.exists():
shutil.copy(p, local)
existing_frames.append(p.name)
if existing_frames:
file_list = ", ".join(f"'{n}'" for n in existing_frames)
ref_section = f"""
REFERENCE FRAMES (in order of timeline): {file_list}.
These are real frames extracted from the original UAP video at different timestamps.
USE THEM as visual input for gpt-image-1's image edit/composition endpoint.
The UAP appears in these frames — preserve its position, scale, and the scene composition.
Use the multiple frames to understand UAP motion / trajectory and convey a coherent moment in the cinematic output.
"""
codex_instruction = f"""Generate ONE high-quality image and save it to '{out_path.name}' in the current directory.{ref_section}
PROMPT:
{prompt}
Use gpt-image-1's image edit (image-to-image) capability with the reference frame(s) above. Combine them as multi-image input if your tool supports it; otherwise pick the most representative one. Output resolution at least 1024x1024. Save only ONE PNG with the exact filename '{out_path.name}'. Confirm the filename after saving."""
cmd = [
"codex", "exec",
"--skip-git-repo-check",
"--sandbox", "workspace-write",
"--cd", str(out_path.parent),
codex_instruction,
]
res = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
if res.returncode != 0:
sys.stderr.write(f" ✗ Codex failed: {res.stderr[-400:]}\n")
return False
return out_path.exists() and out_path.stat().st_size > 0
def append_case_image_refs(md_path: Path, nano_path: Path | None, codex_path: Path | None, diagram_path: Path | None, ref_frame: Path | None):
"""Add case_images block to entity frontmatter."""
fm, body = read_md(md_path)
case_images = {}
if nano_path and nano_path.exists():
case_images["nano_banana"] = {
"path": str(nano_path.relative_to(UFO_ROOT)),
"model": "gemini-3-pro-image",
"synthetic": True,
"factual_data_extraction": "NONE",
"reference_frame": str(ref_frame.relative_to(UFO_ROOT)) if ref_frame else None,
}
if codex_path and codex_path.exists():
case_images["codex"] = {
"path": str(codex_path.relative_to(UFO_ROOT)),
"model": "gpt-image-1",
"synthetic": True,
"factual_data_extraction": "NONE",
"reference_frame": str(ref_frame.relative_to(UFO_ROOT)) if ref_frame else None,
}
if diagram_path and diagram_path.exists():
case_images["investigation_diagram"] = {
"path": str(diagram_path.relative_to(UFO_ROOT)),
"model": "gemini-3-pro-image",
"synthetic": True,
"factual_data_extraction": "NONE",
"reference_frame": str(ref_frame.relative_to(UFO_ROOT)) if ref_frame else None,
"annotation_style": "sherlock-holmes-investigation-board",
}
if not case_images:
return
fm["case_images"] = case_images
fm["case_images_warnings"] = [
"Conceptual visualizations only — not evidence.",
"Do NOT extract numerical claims (altitude, coords, timestamps) from these images.",
"AI-enhanced from a real video frame; UAP position and scene composition come from the frame, but rendering and any annotations are interpretive.",
]
fm["case_images_generated_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
yaml_str = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
new = f"---\n{yaml_str}---\n\n{body}" if not body.startswith("\n") else f"---\n{yaml_str}---\n{body}"
md_path.write_text(new, encoding="utf-8")
def process_entity(md_path: Path, kind: str, force: bool, skip_codex: bool, skip_nano: bool):
fm, _ = read_md(md_path)
entity_id = fm.get("video_id") if kind == "video" else fm.get("doc_id")
if not entity_id:
sys.stderr.write(f" ✗ no entity id in {md_path.name}\n")
return
out_dir = CASE_IMAGES_DIR / entity_id
out_dir.mkdir(parents=True, exist_ok=True)
nano_out = out_dir / "case-nanobanana.png"
codex_out = out_dir / "case-codex.png"
diagram_out = out_dir / "investigation-diagram.png"
# Collect frames as references (videos only). Multi-image lets the model
# understand motion/trajectory across the UAP timeline.
if kind == "video":
ref_frames = find_all_frames(entity_id, max_n=5)
else:
ref_frames = []
primary_frame = ref_frames[len(ref_frames) // 2] if ref_frames else None # for metadata
print(f"\n=== {entity_id} ({kind}) ===", flush=True)
if ref_frames:
print(f" reference frames ({len(ref_frames)}): {[p.name for p in ref_frames]}", flush=True)
else:
print(f" (no reference frames — text-only generation)", flush=True)
case_prompt = build_case_prompt(fm, kind, has_reference_frame=bool(ref_frames))
if not skip_nano and (force or not nano_out.exists()):
print(f" → Nano Banana (case, {len(ref_frames)} frames)…", flush=True)
if call_nano_banana(case_prompt, nano_out, input_images=ref_frames):
print(f" ✓ {nano_out.relative_to(UFO_ROOT)}", flush=True)
if not skip_codex and (force or not codex_out.exists()):
print(f" → Codex (case, {len(ref_frames)} frames)…", flush=True)
if call_codex(case_prompt, codex_out, input_images=ref_frames):
print(f" ✓ {codex_out.relative_to(UFO_ROOT)}", flush=True)
# Investigation diagram — Nano Banana, multi-image (videos only)
if ref_frames and not skip_nano and (force or not diagram_out.exists()):
diagram_prompt = build_diagram_prompt(fm)
print(f" → Nano Banana (investigation diagram, {len(ref_frames)} frames)…", flush=True)
if call_nano_banana(diagram_prompt, diagram_out, input_images=ref_frames):
print(f" ✓ {diagram_out.relative_to(UFO_ROOT)}", flush=True)
append_case_image_refs(md_path, nano_out, codex_out, diagram_out, primary_frame)
def collect_entities(kind: str, entity_id: str | None) -> list[tuple[Path, str]]:
out = []
if kind in ("videos", "both"):
for p in sorted(WIKI_VIDEOS_DIR.glob("*.md")):
if entity_id and p.stem != entity_id:
continue
out.append((p, "video"))
if kind in ("documents", "both"):
for p in sorted(WIKI_DOCS_DIR.glob("*.md")):
if entity_id and p.stem != entity_id:
continue
out.append((p, "document"))
return out
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--kind", choices=["videos", "documents", "both"], default="videos")
ap.add_argument("--entity-id", help="single entity (video_id or doc_id)")
ap.add_argument("--skip-nano", action="store_true", help="skip Nano Banana")
ap.add_argument("--skip-codex", action="store_true", help="skip Codex")
ap.add_argument("--force", action="store_true", help="re-generate even if exists")
args = ap.parse_args()
if not os.environ.get("GEMINI_API_KEY") and not args.skip_nano:
sys.stderr.write("GEMINI_API_KEY not set (needed for Nano Banana)\n")
sys.exit(2)
entities = collect_entities(args.kind, args.entity_id)
print(f"Processing {len(entities)} entit(y/ies)…")
for md_path, kind in entities:
try:
process_entity(md_path, kind, args.force, args.skip_codex, args.skip_nano)
except Exception as e:
sys.stderr.write(f"FATAL on {md_path.name}: {e}\n")
continue
# Log
if entities:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')} — CASE IMAGES (Phase 4.6)\n"
f"- operator: archivist + case-writer\n- script: scripts/11-generate-case-images.py\n"
f"- kind: {args.kind}\n- entities: {len(entities)}\n"
f"- skip_nano: {args.skip_nano}\n- skip_codex: {args.skip_codex}\n"
)
if __name__ == "__main__":
main()

201
scripts/12-incremental-orchestrator.py Executable file
View file

@ -0,0 +1,201 @@
#!/usr/bin/env python3
"""
12-incremental-orchestrator.py — Wait-and-process loop
Polls for newly-completed Gemini video analyses, then runs the downstream
steps (frame extraction + case-images generation) per video. Idempotent by
construction — each step checks the output and skips if already done.
A video is "ready for downstream" when BOTH exist:
- processing/video-analysis/<id>.json (Gemini analysis output)
- wiki/videos/<id>.md (markdown rendered by script 08)
For each ready video, this loop will:
1. If no `processing/uap-frames/<id>/*.jpg` exists → run 09-extract-uap-frames.py
2. If no `processing/case-images/<id>/case-nanobanana.png` AND
no `processing/case-images/<id>/case-codex.png` → run 11-generate-case-images.py
(steps 1 and 2 always check their own outputs first; they never re-do work)
Termination:
- Stops when all videos in raw/videos/ are fully downstream-processed
- Or when --max-iterations is reached
- Or on SIGINT (Ctrl+C)
Usage:
./12-incremental-orchestrator.py # poll every 90s
./12-incremental-orchestrator.py --interval 60 # custom poll interval
./12-incremental-orchestrator.py --max-iterations 50
./12-incremental-orchestrator.py --skip-codex # only Nano Banana case images
./12-incremental-orchestrator.py --once # single pass, no loop
"""
from __future__ import annotations
import argparse
import re
import subprocess
import sys
import time
import unicodedata
from datetime import datetime, timezone
from pathlib import Path
UFO_ROOT = Path("/Users/guto/ufo")
VIDEOS_DIR = UFO_ROOT / "raw" / "videos"
ANALYSIS_DIR = UFO_ROOT / "processing" / "video-analysis"
WIKI_VIDEOS_DIR = UFO_ROOT / "wiki" / "videos"
FRAMES_DIR = UFO_ROOT / "processing" / "uap-frames"
CASE_IMAGES_DIR = UFO_ROOT / "processing" / "case-images"
SCRIPTS = UFO_ROOT / "scripts"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
def now() -> str:
return datetime.now(timezone.utc).strftime("%H:%M:%S")
def filename_to_video_id(name: str) -> str:
base = name.rsplit(".", 1)[0]
nfkd = unicodedata.normalize("NFKD", base)
ascii_s = "".join(c for c in nfkd if not unicodedata.combining(c))
lower = ascii_s.lower()
replaced = re.sub(r"[^a-z0-9-]", "-", lower)
collapsed = re.sub(r"-+", "-", replaced).strip("-")
if collapsed and collapsed[0].isdigit():
collapsed = "vid-" + collapsed
return collapsed
def list_all_video_ids() -> list[str]:
"""All videos under raw/videos/ mapped to their canonical ids."""
return sorted(filename_to_video_id(p.name) for p in VIDEOS_DIR.glob("*.mp4"))
def is_analyzed(video_id: str) -> bool:
"""Ready for downstream: has both JSON and MD from script 08."""
return (ANALYSIS_DIR / f"{video_id}.json").exists() and (WIKI_VIDEOS_DIR / f"{video_id}.md").exists()
def has_frames(video_id: str) -> bool:
d = FRAMES_DIR / video_id
return d.exists() and any(d.glob("*.jpg"))
def has_case_images(video_id: str, want_codex: bool) -> bool:
d = CASE_IMAGES_DIR / video_id
nano = d / "case-nanobanana.png"
codex = d / "case-codex.png"
if not nano.exists():
return False
if want_codex and not codex.exists():
return False
return True
def run(cmd: list[str], description: str) -> bool:
"""Run a subprocess, streaming through. Returns success."""
print(f" [{now()}] → {description}", flush=True)
try:
res = subprocess.run(cmd, timeout=900, check=False)
return res.returncode == 0
except subprocess.TimeoutExpired:
print(f" [{now()}] ✗ timeout on {description}", flush=True)
return False
def process_one_pass(skip_codex: bool, skip_nano: bool) -> tuple[int, int, int]:
"""Single pass over all video ids. Returns (newly_processed, total_ready, total_videos)."""
all_ids = list_all_video_ids()
ready_ids = [v for v in all_ids if is_analyzed(v)]
actions_done = 0
for vid in ready_ids:
did_anything = False
# Step 1: frames
if not has_frames(vid):
cmd = ["python3", str(SCRIPTS / "09-extract-uap-frames.py"), "--video-id", vid]
if run(cmd, f"frames for {vid}"):
did_anything = True
else:
continue # don't proceed to case images if frames failed
# Step 2: case images
if not has_case_images(vid, want_codex=not skip_codex):
cmd = ["python3", str(SCRIPTS / "11-generate-case-images.py"),
"--kind", "videos",
"--entity-id", vid]
if skip_codex:
cmd.append("--skip-codex")
if skip_nano:
cmd.append("--skip-nano")
if run(cmd, f"case images for {vid}"):
did_anything = True
if did_anything:
actions_done += 1
return actions_done, len(ready_ids), len(all_ids)
def all_fully_processed(skip_codex: bool) -> bool:
"""True when every video has been Gemini-analyzed AND has frames + case images."""
all_ids = list_all_video_ids()
if not all_ids:
return False
for v in all_ids:
if not is_analyzed(v):
return False
if not has_frames(v):
return False
if not has_case_images(v, want_codex=not skip_codex):
return False
return True
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--interval", type=int, default=90, help="poll interval seconds (default 90)")
ap.add_argument("--max-iterations", type=int, default=120, help="cap (120 × 90s = 3h)")
ap.add_argument("--skip-codex", action="store_true")
ap.add_argument("--skip-nano", action="store_true")
ap.add_argument("--once", action="store_true", help="single pass, no loop")
args = ap.parse_args()
print(f"[{now()}] orchestrator started")
print(f" interval={args.interval}s max_iterations={args.max_iterations}")
print(f" skip_codex={args.skip_codex} skip_nano={args.skip_nano}")
print(f" watching: {len(list_all_video_ids())} videos in raw/videos/")
iteration = 0
total_actions = 0
try:
while iteration < args.max_iterations:
iteration += 1
actions, ready, total = process_one_pass(args.skip_codex, args.skip_nano)
total_actions += actions
print(f"[{now()}] iter {iteration}: ready={ready}/{total}, "
f"actions_this_pass={actions}, total_actions={total_actions}", flush=True)
if args.once:
break
if all_fully_processed(args.skip_codex):
print(f"[{now()}] ✓ all {total} videos fully processed — exiting", flush=True)
break
time.sleep(args.interval)
except KeyboardInterrupt:
print(f"\n[{now()}] interrupted by user")
# Log
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')} — ORCHESTRATOR\n"
f"- operator: archivist\n- script: scripts/12-incremental-orchestrator.py\n"
f"- iterations: {iteration}\n- total_actions: {total_actions}\n"
)
if __name__ == "__main__":
main()

381
scripts/13-analyze-loose-images.py Executable file
View file

@ -0,0 +1,381 @@
#!/usr/bin/env python3
"""
13-analyze-loose-images.py — Sherlock analysis for standalone images in raw/
Targets the loose image files in /Users/guto/ufo/raw/ that are NOT bundled in
PDFs (so they don't go through Phase 2 conversion). Currently:
FBI-Photo-A1.png .. FBI-Photo-A8.png (8 PNGs)
NASA-UAP-VM1-Apollo-12-1969.jpg .. NASA-UAP-VM6-Apollo-17-1972.jpg (6 JPGs)
For each, calls Gemini 3.1 Pro with a Sherlock-style prompt to extract:
- forensic_description (rich English)
- forensic_description_pt_br (Brazilian Portuguese)
- what_is_visible, classification_markings, redactions
- UAP morphology if applicable
- sherlock_observations (Holmes/Poirot/Dupin/Locard lenses)
- entities_extracted (people, places, equipment, UAP objects)
- quality_signals + flags
Output:
processing/image-analysis/<image-id>.json (raw analysis)
wiki/images-direct/<image-id>.md (bilingual frontmatter + body)
Usage:
./13-analyze-loose-images.py # all
./13-analyze-loose-images.py --image <name> # single file
./13-analyze-loose-images.py --max-files N # cap for testing
./13-analyze-loose-images.py --force # re-run
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import sys
import time
import unicodedata
from datetime import datetime, timezone
from pathlib import Path
try:
from google import genai
from google.genai import types as genai_types
except ImportError:
sys.stderr.write("Missing google-genai. pip3 install google-genai\n")
sys.exit(1)
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
RAW_DIR = UFO_ROOT / "raw"
ANALYSIS_DIR = UFO_ROOT / "processing" / "image-analysis"
WIKI_IMAGES_DIR = UFO_ROOT / "wiki" / "images-direct"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
DEFAULT_MODEL = "gemini-3.1-pro-preview"
FALLBACK = ["gemini-3-pro-preview", "gemini-3.1-flash-lite"]
SCHEMA_VERSION = "0.1.0"
WIKI_VERSION = "0.1.0"
SHERLOCK_IMAGE_PROMPT = """You are an evidence officer in the Investigation Bureau analyzing a single standalone image released by the U.S. government as part of a UAP/UFO disclosure (war.gov/ufo). Apply the combined methodologies of Sherlock Holmes (observation + elimination), Hercule Poirot (psychology of behavior), Auguste Dupin (ratiocination), and Edmond Locard (trace evidence).
Output ONE JSON object (no markdown fence, no preamble):
{
"image_overview": {
"primary_subject": "what the image is fundamentally showing, one sentence",
"camera_perspective": "ground | aerial | satellite | cockpit | underwater | unknown",
"sensor_or_medium": "color_photograph | bw_photograph | infrared_FLIR | radar_screen | sketch_handdrawn | document_scan | screen_capture | unknown",
"platform_inferred": "F/A-18 | helicopter | observer-handheld | naval ship | satellite | unknown",
"estimated_era": "1940s-50s | 1960s-70s | 1980s-90s | 2000s | 2010s | 2020s | unknown"
},
"forensic_description": "Comprehensive English description, 8-15 sentences. Describe everything: composition, persons present, equipment, geography/landmarks, atmospheric conditions, any text/labels visible, any UAP and its morphology, photographic anomalies (lens flare, dust spot, motion blur), any visible processing marks (scanner artifacts, fold lines, redaction tape). Cite verbatim any text visible on the image.",
"forensic_description_pt_br": "Same content as forensic_description, translated to Brazilian Portuguese (pt-br). Brazilian vocabulary and spelling. Preserve UTF-8 accents. Keep verbatim quoted text in original language.",
"classification_markings_visible": [
{"level": "UNCLASSIFIED | CUI | CONFIDENTIAL | SECRET | TOP SECRET", "caveats": ["NOFORN"], "location_on_image": "header | footer | corner | watermark | stamp"}
],
"redactions_visible": [
{"code": "(b)(1) 1.4(a) | (b)(3) | (b)(6) | other_or_unknown", "description": "what is being obscured", "bbox_normalized": {"x": 0.0, "y": 0.0, "w": 0.0, "h": 0.0}}
],
"entities_extracted": {
"people": [{"label": "Subject 1 | Officer X", "role": "subject | photographer | bystander | unknown", "facing_camera": true}],
"organizations": [{"name": "FBI | USAF | NASA | ...", "evidence_for": "patch visible | uniform | logo | scanner stamp"}],
"locations": [{"name": "where", "evidence_for": "landmark | sign | coordinates"}],
"events": [{"label": "...", "date": "YYYY-MM-DD | YYYY | NA"}],
"uap_objects": [{"shape": "sphere | disc | triangle | cylinder | cube | elongated-ellipsoid | cigar | irregular | unknown", "color": "...", "size_estimate": "...", "motion_descriptors": []}],
"vehicles": [{"name": "...", "class": "aircraft | ship | submarine | spacecraft | satellite | ground | other"}],
"equipment_visible": [{"name": "binoculars | sensor pod | camera | ...", "purpose": "..."}],
"concepts": [{"name": "FOIA exemption | sketch | photograph | ...", "class": "legal-instrument | jargon | scientific-term | other"}]
},
"uap_observation_fields": {
"shape": "...",
"color": "...",
"size_estimate": "...",
"altitude_ft": null,
"speed_kts": null,
"bearing_deg": null,
"distance_nm": null,
"coordinates": {"lat": null, "lon": null, "raw_text": "..."},
"morphology_notes": "describe any details of the apparent object",
"kinematic_anomalies": "anything physics-defying inferable from the still"
},
"sherlock_observations": [
{
"detective_lens": "holmes | poirot | dupin | locard",
"observation": "Non-obvious detail. e.g. 'The shadow direction does not match the apparent sun angle suggested by the object highlight, indicating either composite imaging or a light source different from the sun.'",
"implication": "why this matters investigatively",
"confidence_band": "high | medium | low | speculation"
}
],
"anomalies_detected": [
{
"kind": "photographic_artifact | optical_illusion | film_processing | hoax_indicator | inconsistency | unredacted_slip | morphological_anomaly",
"description": "...",
"candidate_explanations": ["lens-flare", "double-exposure", "physical-object", "post-processing", "atmospheric"],
"confidence_band": "high | medium | low | speculation"
}
],
"executive_summary_en": "3-5 sentence English summary suitable for citation in a chat reply.",
"executive_summary_pt_br": "Same in Brazilian Portuguese (pt-br).",
"quality_signals": {
"image_quality_overall": "high | medium | low",
"resolution_apparent": "high | medium | low",
"redaction_density": "none | light | heavy | full-blackout",
"completeness": "complete | truncated | partial",
"extraction_confidence": "high | medium | low"
},
"flags": ["sketch-handdrawn", "redaction-heavy", "low-resolution", "monochrome", "darkened", "scanner-artifact", "fold-marks", "stamp-overlay"]
}
Rules:
- Output ONLY the JSON. No fence. No preamble.
- Empty arrays / null for not applicable. Never omit keys.
- ALL extracted text in ORIGINAL language. Do NOT translate.
- ONLY `forensic_description_pt_br` and `executive_summary_pt_br` are translated to Brazilian Portuguese.
- bbox_normalized is 0..1 (x,y,w,h) relative to the image.
- Aim for ≥4 sherlock_observations including subtle photographic details.
- For anomalies, list ≥3 candidate explanations including a mundane one.
"""
def filename_to_image_id(name: str) -> str:
base = name.rsplit(".", 1)[0]
nfkd = unicodedata.normalize("NFKD", base)
ascii_s = "".join(c for c in nfkd if not unicodedata.combining(c))
lower = ascii_s.lower()
replaced = re.sub(r"[^a-z0-9-]", "-", lower)
collapsed = re.sub(r"-+", "-", replaced).strip("-")
if collapsed and collapsed[0].isdigit():
collapsed = "img-" + collapsed
return collapsed
def sha256_file(p: Path) -> str:
h = hashlib.sha256()
with open(p, "rb") as fh:
for chunk in iter(lambda: fh.read(65536), b""):
h.update(chunk)
return h.hexdigest()
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def call_gemini_image(client, image_path: Path, model: str, attempt: int = 1, timeout: int = 180):
"""Thread-based timeout to prevent SDK hangs on rate limit / network issues."""
import concurrent.futures
try:
if image_path.stat().st_size > 19 * 1024 * 1024:
file = client.files.upload(file=str(image_path))
while file.state.name == "PROCESSING":
time.sleep(2)
file = client.files.get(name=file.name)
content = [file, SHERLOCK_IMAGE_PROMPT]
else:
mime = "image/png" if image_path.suffix.lower() == ".png" else "image/jpeg"
content = [
genai_types.Part.from_bytes(data=image_path.read_bytes(), mime_type=mime),
SHERLOCK_IMAGE_PROMPT,
]
def _call():
return client.models.generate_content(
model=model,
contents=content,
config=genai_types.GenerateContentConfig(
response_mime_type="application/json",
temperature=0.2,
max_output_tokens=16384,
),
)
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:
future = ex.submit(_call)
try:
resp = future.result(timeout=timeout)
except concurrent.futures.TimeoutError:
raise RuntimeError(f"Gemini hung >{timeout}s — treating as failure")
return resp.text, model
except Exception as e:
if attempt < len(FALLBACK):
next_m = FALLBACK[attempt - 1]
print(f" ⚠ {model} failed ({e}); fallback {next_m}", flush=True)
return call_gemini_image(client, image_path, next_m, attempt + 1, timeout)
raise
def render_image_md(image_id: str, image_path: Path, analysis: dict, meta: dict, now_iso: str) -> str:
fm = {
"schema_version": SCHEMA_VERSION,
"type": "image",
"image_id": image_id,
"image_class": "standalone", # vs "page-extract"
"original_filename": image_path.name,
"raw_path": f"../../raw/{image_path.name}",
"sha256": sha256_file(image_path),
"size_bytes": image_path.stat().st_size,
"vision_model": meta.get("model"),
"analyzed_at": now_iso,
**{f"overview_{k}": v for k, v in (analysis.get("image_overview") or {}).items()},
"uap_observation_fields": analysis.get("uap_observation_fields"),
"classification_markings_visible": analysis.get("classification_markings_visible") or [],
"redactions_visible": analysis.get("redactions_visible") or [],
"entities_extracted": analysis.get("entities_extracted") or {},
"anomalies_detected": analysis.get("anomalies_detected") or [],
"sherlock_observations": analysis.get("sherlock_observations") or [],
"forensic_description": analysis.get("forensic_description", ""),
"forensic_description_pt_br": analysis.get("forensic_description_pt_br", ""),
"executive_summary_en": analysis.get("executive_summary_en", ""),
"executive_summary_pt_br": analysis.get("executive_summary_pt_br", ""),
"quality_signals": analysis.get("quality_signals") or {},
"flags": analysis.get("flags") or [],
"last_ingest": now_iso,
"wiki_version": WIKI_VERSION,
}
yaml_str = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
body = f"""# Image Analysis — {image_id}
> Source: `raw/{image_path.name}` · Gemini model: `{meta.get("model")}` · Analyzed: {now_iso}
![image](../../raw/{image_path.name})
## Executive Summary (EN)
{(analysis.get("executive_summary_en") or "").strip()}
## Sumário Executivo (PT-BR)
{(analysis.get("executive_summary_pt_br") or "").strip()}
## Forensic Description (EN)
{(analysis.get("forensic_description") or "").strip()}
## Descrição Forense (PT-BR)
{(analysis.get("forensic_description_pt_br") or "").strip()}
## Sherlock Observations
"""
for o in (analysis.get("sherlock_observations") or []):
body += f"- **[{o.get('detective_lens','?')}]** {o.get('observation','')}\n - _Implication:_ {o.get('implication','')}\n - _Confidence:_ `{o.get('confidence_band','?')}`\n\n"
body += "## Anomalies Detected\n\n"
for a in (analysis.get("anomalies_detected") or []):
body += f"- **{a.get('kind','?')}**: {a.get('description','')}\n - _Candidates:_ {', '.join(a.get('candidate_explanations', []))}\n - _Confidence:_ `{a.get('confidence_band','?')}`\n\n"
return f"---\n{yaml_str}---\n\n{body}"
def process_image(client, image_path: Path, model: str, force: bool) -> bool:
image_id = filename_to_image_id(image_path.name)
json_out = ANALYSIS_DIR / f"{image_id}.json"
md_out = WIKI_IMAGES_DIR / f"{image_id}.md"
if not force and json_out.exists() and md_out.exists():
print(f" skip {image_id} (already processed)", flush=True)
return True
json_out.parent.mkdir(parents=True, exist_ok=True)
md_out.parent.mkdir(parents=True, exist_ok=True)
print(f"\n=== {image_path.name} → {image_id} ===", flush=True)
t0 = time.time()
try:
text, model_used = call_gemini_image(client, image_path, model)
except Exception as e:
print(f" ✗ generation failed: {e}", flush=True)
return False
text = text.strip()
if text.startswith("```"):
text = re.sub(r"^```(?:json)?\s*", "", text)
text = re.sub(r"\s*```$", "", text)
try:
analysis = json.loads(text)
except json.JSONDecodeError as e:
print(f" ✗ JSON parse failed: {e}", flush=True)
json_out.with_suffix(".raw.txt").write_text(text, encoding="utf-8")
return False
meta = {"model": model_used, "duration_seconds": round(time.time() - t0, 1)}
json_out.write_text(json.dumps({"analysis": analysis, "meta": meta}, indent=2, ensure_ascii=False), encoding="utf-8")
md = render_image_md(image_id, image_path, analysis, meta, utc_now_iso())
md_out.write_text(md, encoding="utf-8")
print(f" ✓ {image_id} ({time.time() - t0:.1f}s)", flush=True)
return True
def find_loose_images() -> list[Path]:
images = []
for ext in (".png", ".jpg", ".jpeg"):
for p in RAW_DIR.glob(f"*{ext}"):
# Skip PDFs (FBI-Photo-B* etc. are PDFs)
images.append(p)
return sorted(images)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--image", help="single image filename in raw/")
ap.add_argument("--max-files", type=int, default=None)
ap.add_argument("--model", default=DEFAULT_MODEL)
ap.add_argument("--force", action="store_true")
args = ap.parse_args()
api_key = os.environ.get("GEMINI_API_KEY")
if not api_key:
sys.stderr.write("GEMINI_API_KEY not set\n"); sys.exit(2)
client = genai.Client(api_key=api_key)
if args.image:
targets = [RAW_DIR / args.image]
else:
targets = find_loose_images()
if args.max_files:
targets = targets[: args.max_files]
print(f"Processing {len(targets)} image(s) with {args.model}")
ok = 0
fail = []
for p in targets:
if process_image(client, p, args.model, args.force):
ok += 1
else:
fail.append(p.name)
print(f"\nDone. ok={ok}, failed={len(fail)}")
if fail:
print("failed:", fail)
if ok > 0:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {utc_now_iso()} — IMAGE ANALYSIS (Phase 4.7)\n"
f"- operator: archivist + evidence-officer\n- script: scripts/13-analyze-loose-images.py\n"
f"- model: {args.model}\n- images_ok: {ok}\n- images_failed: {len(fail)}\n"
)
if __name__ == "__main__":
main()

394
scripts/14-build-document-md.py Executable file
View file

@ -0,0 +1,394 @@
#!/usr/bin/env python3
"""
14-build-document-md.py — Build wiki/documents/<doc-id>.md from its pages
Walks each document directory under wiki/pages/<doc-id>/ and aggregates its
page-level analyses into ONE consolidated document.md with:
- Frontmatter (doc_id, page_count, sha256, content_classification UNION,
highest_classification, languages_detected, redaction_codes_present,
war_gov block if available, executive_summary_confidence, key_entities aggregated)
- Page index (table linking to each [[<doc-id>/pNNN]])
- Aggregated entities (union of all entities_extracted across pages, deduped)
- Aggregated UAP observations (concat of all uap_observation_fields)
- Aggregated classification markings + redactions stats
- Optional Haiku-generated executive summary (bilingual EN + PT-BR)
A document is "ready" when its pages count == total_pages (from page frontmatter).
Idempotent: re-running updates last_lint timestamp only if substantive data changed.
Usage:
./14-build-document-md.py # all ready docs
./14-build-document-md.py --doc-id <id> # single
./14-build-document-md.py --force # rebuild even if exists
./14-build-document-md.py --with-llm-summary # also call Haiku for executive_summary
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import subprocess
import sys
import unicodedata
from collections import Counter, defaultdict
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
PAGES_BASE = UFO_ROOT / "wiki" / "pages"
DOCS_BASE = UFO_ROOT / "wiki" / "documents"
PNG_BASE = UFO_ROOT / "processing" / "png"
RAW_DIR = UFO_ROOT / "raw"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
SCHEMA_VERSION = "0.1.0"
WIKI_VERSION = "0.1.0"
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_md(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end == -1:
return {}, c
try:
return (yaml.safe_load(c[3:end].strip()) or {}), c[end + 3 :].lstrip("\n")
except yaml.YAMLError:
return {}, c[end + 3 :].lstrip("\n")
def write_md(path: Path, fm: dict, body: str) -> bool:
yaml_str = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
new = f"---\n{yaml_str}---\n\n{body}" if not body.startswith("\n") else f"---\n{yaml_str}---\n{body}"
if path.exists():
existing = path.read_text(encoding="utf-8")
# Idempotency: compare excluding generated_at timestamps
if existing == new:
return False
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(new, encoding="utf-8")
return True
def sha256_file(p: Path) -> str:
h = hashlib.sha256()
with open(p, "rb") as fh:
for chunk in iter(lambda: fh.read(65536), b""):
h.update(chunk)
return h.hexdigest()
def find_pdf_for_doc(doc_id: str) -> Path | None:
"""Reverse-lookup: find the original PDF in raw/ matching this doc_id."""
def _canon(name: str) -> str:
base = name.rsplit(".", 1)[0]
nfkd = unicodedata.normalize("NFKD", base)
ascii_s = "".join(c for c in nfkd if not unicodedata.combining(c))
lower = ascii_s.lower()
replaced = re.sub(r"[^a-z0-9-]", "-", lower)
collapsed = re.sub(r"-+", "-", replaced).strip("-")
if collapsed and collapsed[0].isdigit():
collapsed = "doc-" + collapsed
return collapsed
for p in RAW_DIR.glob("*.pdf"):
if _canon(p.name) == doc_id:
return p
return None
def list_doc_dirs() -> list[Path]:
"""All wiki/pages/<doc-id>/ subdirectories."""
if not PAGES_BASE.exists():
return []
return sorted([d for d in PAGES_BASE.iterdir() if d.is_dir()])
MAX_MISSING_PAGES_TOLERATED = 3 # p000 OCR-missing cases are structural; tolerate small gaps
def is_doc_complete(doc_dir: Path) -> tuple[bool, int, int]:
"""Returns (is_complete, pages_done, total_pages_expected).
A doc is considered complete if it has at most MAX_MISSING_PAGES_TOLERATED
pages missing (typically the p000 with missing OCR — structurally irrecoverable).
"""
pages = sorted(doc_dir.glob("p*.md"))
if not pages:
return False, 0, 0
fm0, _ = read_md(pages[0])
total = fm0.get("total_pages") or 0
if total <= 0:
return False, len(pages), total
return len(pages) >= max(1, total - MAX_MISSING_PAGES_TOLERATED), len(pages), total
def aggregate(doc_id: str, doc_dir: Path) -> dict:
"""Walk all pages and aggregate frontmatter fields."""
pages = sorted(doc_dir.glob("p*.md"))
agg = {
"doc_id": doc_id,
"page_count": len(pages),
"pages": [],
"content_classification": set(),
"languages_detected": set(),
"redaction_codes": Counter(),
"classification_levels": Counter(),
"page_types": Counter(),
"entities": defaultdict(lambda: Counter()), # entities['people']['name'] = count
"uap_observations": [],
"first_vision_run_at": None,
"last_vision_run_at": None,
"total_redactions": 0,
"total_signatures": 0,
"total_tables": 0,
"total_images_detected": 0,
"ocr_quality_avg": 0.0,
"vision_quality_avg": 0.0,
"flags": Counter(),
}
ocr_scores = []
vis_scores = []
for p in pages:
fm, _ = read_md(p)
if not fm:
continue
m = re.match(r"p(\d+)", p.stem)
page_num = int(m.group(1)) if m else 0
agg["pages"].append({
"page": page_num,
"page_id": f"[[{fm.get('page_id','')}]]",
"page_type": fm.get("page_type", "unknown"),
"content_classification": fm.get("content_classification", []),
"language_detected": fm.get("language_detected", "unknown"),
})
for c in (fm.get("content_classification") or []):
agg["content_classification"].add(c)
lang = fm.get("language_detected")
if lang and lang != "unknown":
agg["languages_detected"].add(lang)
agg["page_types"][fm.get("page_type", "unknown")] += 1
for r in (fm.get("redactions") or []):
agg["total_redactions"] += 1
code = r.get("code")
if code:
agg["redaction_codes"][code] += 1
for cm in (fm.get("classification_markings") or []):
lv = cm.get("level")
if lv:
agg["classification_levels"][lv] += 1
agg["total_signatures"] += len(fm.get("signatures_observed") or [])
agg["total_tables"] += len(fm.get("tables_detected") or [])
agg["total_images_detected"] += len(fm.get("images_detected") or [])
for ent_class, items in (fm.get("entities_extracted") or {}).items():
for it in (items or []):
name = (it.get("name") if isinstance(it, dict) else None) \
or (it.get("label") if isinstance(it, dict) else None) \
or (it.get("shape") if isinstance(it, dict) else None)
if name:
agg["entities"][ent_class][name] += 1
if fm.get("uap_observation_fields"):
uap = dict(fm["uap_observation_fields"])
uap["_page"] = page_num
agg["uap_observations"].append(uap)
ocr = fm.get("ocr_quality_score")
if isinstance(ocr, (int, float)):
ocr_scores.append(ocr)
vis = fm.get("vision_quality_score")
if isinstance(vis, (int, float)):
vis_scores.append(vis)
for fg in (fm.get("flags") or []):
agg["flags"][fg] += 1
run_at = fm.get("vision_run_at")
if run_at:
if not agg["first_vision_run_at"] or run_at < agg["first_vision_run_at"]:
agg["first_vision_run_at"] = run_at
if not agg["last_vision_run_at"] or run_at > agg["last_vision_run_at"]:
agg["last_vision_run_at"] = run_at
agg["ocr_quality_avg"] = round(sum(ocr_scores) / len(ocr_scores), 3) if ocr_scores else 0.0
agg["vision_quality_avg"] = round(sum(vis_scores) / len(vis_scores), 3) if vis_scores else 0.0
return agg
def highest_classification(level_counter: Counter) -> str:
order = ["TOP SECRET", "SECRET", "CONFIDENTIAL", "CUI", "UNCLASSIFIED"]
for lv in order:
if level_counter.get(lv, 0) > 0:
return lv
return "UNCLASSIFIED"
def render_document_md(doc_id: str, agg: dict, pdf_path: Path | None) -> tuple[dict, str]:
"""Compose document.md frontmatter + body from aggregated data."""
top_people = [n for n, _ in agg["entities"].get("people", Counter()).most_common(20)]
top_orgs = [n for n, _ in agg["entities"].get("organizations", Counter()).most_common(20)]
top_locs = [n for n, _ in agg["entities"].get("locations", Counter()).most_common(20)]
top_events = [n for n, _ in agg["entities"].get("events", Counter()).most_common(10)]
top_uap = [n for n, _ in agg["entities"].get("uap_objects", Counter()).most_common(10)]
top_vehicles = [n for n, _ in agg["entities"].get("vehicles", Counter()).most_common(10)]
top_concepts = [n for n, _ in agg["entities"].get("concepts", Counter()).most_common(20)]
fm = {
"schema_version": SCHEMA_VERSION,
"type": "document",
"doc_id": doc_id,
"canonical_title": doc_id.replace("-", " ").title(),
"original_filename": pdf_path.name if pdf_path else None,
"raw_path": f"../raw/{pdf_path.name}" if pdf_path else None,
"sha256": sha256_file(pdf_path) if pdf_path and pdf_path.exists() else None,
"size_bytes": pdf_path.stat().st_size if pdf_path and pdf_path.exists() else None,
"page_count": agg["page_count"],
"mime_type": "application/pdf",
"collection": "DOW-UAP", # TODO: infer from doc_id prefix
"document_class": "unknown",
"content_classification": sorted(agg["content_classification"]),
"highest_classification": highest_classification(agg["classification_levels"]),
"languages_detected": sorted(agg["languages_detected"]),
"has_redactions": agg["total_redactions"] > 0,
"redaction_codes_present": sorted(agg["redaction_codes"].keys()),
"redactions_total": agg["total_redactions"],
"signatures_total": agg["total_signatures"],
"tables_total": agg["total_tables"],
"images_detected_total": agg["total_images_detected"],
"page_types_histogram": dict(agg["page_types"]),
"ocr_quality_avg": agg["ocr_quality_avg"],
"vision_quality_avg": agg["vision_quality_avg"],
"flags": dict(agg["flags"]),
"first_vision_run_at": agg["first_vision_run_at"],
"last_vision_run_at": agg["last_vision_run_at"],
"ingest_date": agg["last_vision_run_at"][:10] if agg["last_vision_run_at"] else None,
"last_ingest": utc_now_iso(),
"wiki_version": WIKI_VERSION,
"key_entities": {
"people": top_people,
"organizations": top_orgs,
"locations": top_locs,
"events": top_events,
"uap_objects": top_uap,
"vehicles": top_vehicles,
"concepts": top_concepts,
},
"uap_observations_count": len(agg["uap_observations"]),
"pages": [{"page": p["page"], "page_id": p["page_id"], "page_type": p["page_type"]} for p in agg["pages"]],
}
# Body
body = f"# {fm['canonical_title']}\n\n"
body += f"> **{agg['page_count']}-page document** · {fm['highest_classification']} · {len(agg['content_classification'])} content categories · {agg['total_redactions']} redactions across pages\n\n"
body += "## Quick stats\n\n"
body += f"- **Pages**: {agg['page_count']}\n"
body += f"- **Languages**: {', '.join(sorted(agg['languages_detected'])) or 'n/a'}\n"
body += f"- **Page types**: {dict(agg['page_types'])}\n"
body += f"- **Redaction codes**: {dict(agg['redaction_codes'])}\n"
body += f"- **Classification levels seen**: {dict(agg['classification_levels'])}\n"
body += f"- **Signatures observed**: {agg['total_signatures']}\n"
body += f"- **Tables detected**: {agg['total_tables']}\n"
body += f"- **Images detected**: {agg['total_images_detected']}\n"
body += f"- **OCR quality (avg)**: {agg['ocr_quality_avg']}\n"
body += f"- **Vision quality (avg)**: {agg['vision_quality_avg']}\n\n"
body += "## Key entities (aggregated across all pages)\n\n"
for label, lst in [
("People", top_people), ("Organizations", top_orgs), ("Locations", top_locs),
("Events", top_events), ("UAP objects", top_uap), ("Vehicles", top_vehicles),
("Concepts", top_concepts),
]:
if lst:
body += f"### {label}\n\n"
for item in lst:
body += f"- {item}\n"
body += "\n"
if agg["uap_observations"]:
body += "## UAP observations across pages\n\n"
for u in agg["uap_observations"]:
p = u.get("_page", "?")
shape = u.get("shape") or "unknown"
color = u.get("color") or ""
alt = u.get("altitude_ft")
spd = u.get("speed_kts")
body += f"- **Page {p}**: shape=`{shape}` color=`{color}` altitude={alt} speed={spd}\n"
body += "\n"
body += "## Page index\n\n"
body += "| Page | Type | Classification |\n|---|---|---|\n"
for p in agg["pages"]:
cc = ", ".join(p["content_classification"]) or "—"
body += f"| {p['page_id']} | `{p['page_type']}` | {cc} |\n"
body += "\n"
body += "## Notes\n\n"
body += "Document.md is a **consolidated view** of all pages. For per-page detail (OCR text, vision description bilingual, entities, etc.), open the individual `wiki/pages/<doc-id>/p<NNN>.md` files linked in the page index above.\n"
return fm, body
def process_doc(doc_dir: Path, force: bool) -> bool:
doc_id = doc_dir.name
complete, done, total = is_doc_complete(doc_dir)
if not complete:
print(f" ⏳ {doc_id}: {done}/{total} pages — not ready, skipping")
return False
out_path = DOCS_BASE / f"{doc_id}.md"
if out_path.exists() and not force:
# Check if mtime newer than last page mtime
latest_page_mtime = max((p.stat().st_mtime for p in doc_dir.glob("p*.md")), default=0)
if out_path.stat().st_mtime >= latest_page_mtime:
return False # already up-to-date
print(f" 📄 {doc_id}: {done}/{total} pages — building document.md")
agg = aggregate(doc_id, doc_dir)
pdf = find_pdf_for_doc(doc_id)
fm, body = render_document_md(doc_id, agg, pdf)
changed = write_md(out_path, fm, body)
if changed:
print(f" ✓ {out_path.relative_to(UFO_ROOT)}")
return changed
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id", help="single doc")
ap.add_argument("--force", action="store_true")
args = ap.parse_args()
DOCS_BASE.mkdir(parents=True, exist_ok=True)
targets = [PAGES_BASE / args.doc_id] if args.doc_id else list_doc_dirs()
print(f"Processing {len(targets)} doc(s)…")
built = 0
for d in targets:
if not d.exists():
sys.stderr.write(f" ✗ no pages dir for {d.name}\n")
continue
if process_doc(d, args.force):
built += 1
print(f"\nBuilt/updated: {built} document.md")
if built > 0:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {utc_now_iso()} — BUILD DOCUMENT.MD (Phase 4)\n"
f"- operator: archivist + case-writer\n- script: scripts/14-build-document-md.py\n"
f"- documents_built: {built}\n"
)
if __name__ == "__main__":
main()

322
scripts/15-consolidate-tables.py Executable file
View file

@ -0,0 +1,322 @@
#!/usr/bin/env python3
"""
15-consolidate-tables.py — Multi-page table consolidation
For each document, walks pages in order and stitches together tables that
span multiple pages (detected by Haiku's `tables_detected[]` flags
`spans_multi_page`, `continues_from_prev_page`, `likely_continues_next_page`).
Output:
- wiki/tables/<TBL-<DOCSHORT>-<NNNN>>.md per consolidated multi-page table
- Updates each page.md's tables_detected[].table_id to reference the new TBL id
(so pages can cross-link to the master table.md)
Single-page tables stay inline in page.md (no separate table.md). Only spanning
tables get promoted.
Algorithm:
1. For each doc-id directory under wiki/pages/, sort pages by page_number.
2. Maintain a list of "open" tables (started, not yet ended).
3. For each page's tables in order:
a. If table has `continues_from_prev_page=true` and there's an open table
compatible with similar bbox/column-count, append this page as a span.
If `likely_continues_next_page=false`, finalize.
b. Otherwise start a new table. If `likely_continues_next_page=false` and
`spans_multi_page=false`, single-page → skip (don't promote).
Otherwise add to open tables.
4. Finalize each open table at end-of-doc.
5. Write wiki/tables/<id>.md and inject `table_id` back into each page's
tables_detected entry.
Idempotent: writes only when content changes.
Usage:
./15-consolidate-tables.py
./15-consolidate-tables.py --doc-id <id>
./15-consolidate-tables.py --force
"""
from __future__ import annotations
import argparse
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
PAGES_BASE = UFO_ROOT / "wiki" / "pages"
TABLES_BASE = UFO_ROOT / "wiki" / "tables"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
SCHEMA_VERSION = "0.1.0"
WIKI_VERSION = "0.1.0"
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_md(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end == -1:
return {}, c
try:
return (yaml.safe_load(c[3:end].strip()) or {}), c[end + 3 :].lstrip("\n")
except yaml.YAMLError:
return {}, c[end + 3 :].lstrip("\n")
def write_md(path: Path, fm: dict, body: str) -> bool:
yaml_str = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
new = f"---\n{yaml_str}---\n\n{body}" if not body.startswith("\n") else f"---\n{yaml_str}---\n{body}"
if path.exists() and path.read_text(encoding="utf-8") == new:
return False
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(new, encoding="utf-8")
return True
def doc_short_id(doc_id: str) -> str:
"""Compact uppercase identifier for use in TBL-<DOCSHORT>-NNNN."""
s = re.sub(r"[^A-Z0-9]", "", doc_id.upper())
# Drop common prefixes
for prefix in ("DOWUAP", "DOSUAP", "NASAUAP", "FBIPHOTO", "DOC"):
if s.startswith(prefix):
s = s[len(prefix):]
break
return s[:8] or "X"
def list_doc_dirs() -> list[Path]:
if not PAGES_BASE.exists():
return []
return sorted(d for d in PAGES_BASE.iterdir() if d.is_dir())
def consolidate_doc(doc_dir: Path, force: bool) -> int:
"""Walk pages in order, identify multi-page tables, write wiki/tables/<id>.md.
Returns number of tables created/updated."""
doc_id = doc_dir.name
pages = sorted(doc_dir.glob("p*.md"))
if not pages:
return 0
# Load all pages' tables_detected in order
pages_data = []
for p in pages:
fm, body = read_md(p)
if not fm:
continue
m = re.match(r"p(\d+)", p.stem)
if not m:
continue
page_num = int(m.group(1))
pages_data.append({
"path": p,
"page_num": page_num,
"page_id": fm.get("page_id", f"{doc_id}/p{page_num:03d}"),
"tables": fm.get("tables_detected") or [],
"fm": fm,
"body": body,
})
if not pages_data:
return 0
# Walk and stitch
open_tables = [] # list of dicts with `spans`, last bbox, last col count
finalized = [] # list of finalized tables ready to write
short = doc_short_id(doc_id)
def new_table():
return {
"spans": [], # [{page_num, page_id, bbox, role}]
"headers_summaries": [],
"row_count_estimates": [],
"col_count_estimates": [],
}
for page in pages_data:
if not page["tables"]:
# Close any open tables — they didn't continue to this page
for ot in open_tables:
if ot["spans"]:
ot["spans"][-1]["role"] = "end"
finalized.append(ot)
open_tables = []
continue
# Match each table on this page
matched_open = []
for t in page["tables"]:
bbox = t.get("bbox") or {}
continues_from = bool(t.get("continues_from_prev_page"))
likely_continues = bool(t.get("likely_continues_next_page"))
spans_multi = bool(t.get("spans_multi_page"))
tbl = None
if continues_from and open_tables:
# Continue the oldest open table (simple FIFO)
tbl = open_tables.pop(0)
elif spans_multi or likely_continues:
tbl = new_table()
else:
# Single-page table — skip, lives inline in page.md
continue
role = "start" if not tbl["spans"] else ("middle" if likely_continues else "end")
tbl["spans"].append({
"page_num": page["page_num"],
"page_id": page["page_id"],
"bbox": bbox,
"role": role,
})
if t.get("headers_summary"):
tbl["headers_summaries"].append(t["headers_summary"])
if t.get("row_count_estimate"):
tbl["row_count_estimates"].append(t["row_count_estimate"])
if t.get("col_count_estimate"):
tbl["col_count_estimates"].append(t["col_count_estimate"])
if likely_continues:
matched_open.append(tbl)
else:
finalized.append(tbl)
# Any open tables not matched on this page are stranded
for ot in open_tables:
if ot["spans"]:
ot["spans"][-1]["role"] = "end"
finalized.append(ot)
open_tables = matched_open
# Finalize any remaining open at end-of-doc
for ot in open_tables:
if ot["spans"]:
ot["spans"][-1]["role"] = "end"
finalized.append(ot)
# Filter to multi-page only (single-page slipped in via spans_multi_page=true on 1 page)
multi_page = [t for t in finalized if len(t["spans"]) >= 2]
if not multi_page:
return 0
# Write wiki/tables/<id>.md and update page.md back-refs
page_table_refs: dict[str, list[str]] = {} # page_id → [table_id...]
n_written = 0
for idx, tbl in enumerate(multi_page, start=1):
tbl_id = f"TBL-{short}-{idx:04d}"
# Pick best canonical headers (first non-empty)
headers = next((h for h in tbl["headers_summaries"] if h), "")
row_est = max(tbl["row_count_estimates"], default=0)
col_est = max(tbl["col_count_estimates"], default=0)
span_pages_yaml = []
for sp in tbl["spans"]:
span_pages_yaml.append({
"page": f"[[{sp['page_id']}]]",
"bbox": sp["bbox"],
"role": sp["role"],
})
page_table_refs.setdefault(sp["page_id"], []).append(tbl_id)
fm = {
"schema_version": SCHEMA_VERSION,
"type": "table",
"table_id": tbl_id,
"canonical_title": (headers or f"Multi-page table {idx} of {doc_id}")[:200],
"source_doc": f"[[{doc_id}]]",
"multi_page": True,
"page_count": len(tbl["spans"]),
"spans_pages": span_pages_yaml,
"headers_summary": headers,
"total_rows_estimate": row_est,
"total_columns_estimate": col_est,
"extraction_quality": None, # to be set when actually extracted to CSV
"last_ingest": utc_now_iso(),
"wiki_version": WIKI_VERSION,
}
body = (
f"# {fm['canonical_title']}\n\n"
f"> Multi-page table spanning {len(tbl['spans'])} pages of [[{doc_id}]]\n\n"
f"## Pages\n\n"
)
for sp in tbl["spans"]:
body += f"- {sp['role']}: [[{sp['page_id']}]] · bbox {sp['bbox']}\n"
body += "\n## Headers\n\n"
body += f"{headers or '_(not extracted)_'}\n\n"
body += "## Notes\n\n"
body += "Per-page table snippets live in each page.md's `tables_detected[]`. This consolidated record stitches them together. Full data extraction (row-by-row CSV) is deferred to a future enrichment pass.\n"
out = TABLES_BASE / f"{tbl_id}.md"
if write_md(out, fm, body):
n_written += 1
# Inject table_id back into each page.md (idempotent)
for page in pages_data:
refs = page_table_refs.get(page["page_id"])
if not refs:
continue
fm = page["fm"]
tables = fm.get("tables_detected") or []
if not tables:
continue
# Mark the first N matching tables with table_id (simple sequential mapping)
modified = False
ref_iter = iter(refs)
for t in tables:
if t.get("spans_multi_page") or t.get("continues_from_prev_page") or t.get("likely_continues_next_page"):
try:
next_id = next(ref_iter)
except StopIteration:
break
if t.get("table_id") != next_id:
t["table_id"] = next_id
modified = True
if modified:
fm["tables_detected"] = tables
write_md(page["path"], fm, page["body"])
return n_written
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id", help="single doc")
ap.add_argument("--force", action="store_true")
args = ap.parse_args()
TABLES_BASE.mkdir(parents=True, exist_ok=True)
targets = [PAGES_BASE / args.doc_id] if args.doc_id else list_doc_dirs()
total_tables = 0
docs_with_tables = 0
for d in targets:
if not d.exists():
continue
n = consolidate_doc(d, args.force)
if n > 0:
print(f" ✓ {d.name}: {n} multi-page table(s)")
total_tables += n
docs_with_tables += 1
print(f"\nTotal: {total_tables} multi-page tables across {docs_with_tables} doc(s)")
if total_tables > 0:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {utc_now_iso()} — CONSOLIDATE TABLES\n"
f"- operator: archivist\n- script: scripts/15-consolidate-tables.py\n"
f"- tables_written: {total_tables}\n- docs_with_tables: {docs_with_tables}\n"
)
if __name__ == "__main__":
main()

390
scripts/16-extract-table-csv.py Executable file
View file

@ -0,0 +1,390 @@
#!/usr/bin/env python3
"""
16-extract-table-csv.py — Row-by-row extraction of multi-page tables → CSV
For each `wiki/tables/<TBL-id>.md`:
1. Resolve each span's PNG path (processing/png/<doc-id>/p-NNN.png)
2. Crop the table region using bbox (Pillow)
3. Send all crops in order to Haiku with a prompt to extract the full table
preserving multi-page row continuity
4. Receive JSON: { headers: [...], rows: [[...], ...] }
5. Save:
- processing/tables/<TBL-id>.csv (extracted CSV)
- processing/tables/<TBL-id>.json (raw extraction + metadata)
- processing/table-crops/<TBL-id>/ (the crop JPGs for inspection)
- Update wiki/tables/<TBL-id>.md frontmatter:
csv_path, extraction_quality, headers, row_count_extracted,
extracted_at, extraction_model
Idempotent: skip if CSV exists and not --force.
Usage:
./16-extract-table-csv.py # all multi-page tables
./16-extract-table-csv.py --table-id <id> # single
./16-extract-table-csv.py --force # re-extract
./16-extract-table-csv.py --model haiku # default; or sonnet
"""
from __future__ import annotations
import argparse
import csv
import json
import re
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. pip3 install pyyaml\n")
sys.exit(1)
try:
from PIL import Image
except ImportError:
sys.stderr.write("Missing pillow. pip3 install pillow\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
TABLES_BASE = UFO_ROOT / "wiki" / "tables"
PNG_BASE = UFO_ROOT / "processing" / "png"
CSV_BASE = UFO_ROOT / "processing" / "tables"
CROPS_BASE = UFO_ROOT / "processing" / "table-crops"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
DEFAULT_MODEL = "haiku"
MAX_TURNS = 4
DEFAULT_TIMEOUT = 240
EXTRACT_PROMPT = """You are extracting a multi-page table from a US Department of War declassified UAP document.
You will see {n_crops} image crops in order. They represent ONE logical table split across {n_pages} consecutive pages. The first crop is the start, the last is the end, and any middle ones continue the rows.
STEPS:
1. Use the Read tool on EACH of these crop image paths, IN ORDER:
{crop_list}
2. Identify the column headers (typically only on the first page; subsequent pages may repeat headers — skip those repeats).
3. Concatenate all rows from all pages into a single ordered list. A row that visually appears to span a page break (e.g. a cell continues onto the next page) should be merged into ONE row when possible.
4. Output ONE JSON object (no fence, no preamble) with this exact schema:
{{
"headers": ["col1", "col2", ...],
"rows": [
["row1_col1_value", "row1_col2_value", ...],
["row2_col1_value", "row2_col2_value", ...]
],
"row_count": <int — total rows extracted, excluding header repeats>,
"column_count": <int — number of columns>,
"headers_repeat_on_each_page": true|false,
"merged_cross_page_rows": <int — how many rows you merged across page breaks>,
"extraction_quality": <float 0..1 — your confidence the extraction is complete and accurate>,
"notes": "Any caveats: illegible cells, redactions inside cells, merged headers, ambiguous values, etc. Use 'REDACTED' for cell values that are blacked out, and '???' for illegible content."
}}
RULES:
- Preserve ORIGINAL LANGUAGE of all cell text. Do NOT translate.
- For redacted cells: "REDACTED" or "REDACTED (1.4(a))" if the code is visible.
- For illegible cells: "???".
- For empty cells: empty string "".
- If a cell contains a list (multiple values), preserve as comma-separated.
- Numbers stay as strings (preserve formatting like "24,989" or "1319Z").
- Headers should be short, snake_case-friendly (e.g. "incident_date", "shape", "altitude_ft").
- Output ONLY the JSON. No fence, no commentary."""
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_md(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end == -1:
return {}, c
try:
return (yaml.safe_load(c[3:end].strip()) or {}), c[end + 3 :].lstrip("\n")
except yaml.YAMLError:
return {}, c[end + 3 :].lstrip("\n")
def write_md(path: Path, fm: dict, body: str) -> bool:
yaml_str = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
new = f"---\n{yaml_str}---\n\n{body}" if not body.startswith("\n") else f"---\n{yaml_str}---\n{body}"
if path.exists() and path.read_text(encoding="utf-8") == new:
return False
path.write_text(new, encoding="utf-8")
return True
def resolve_page_png(page_link: str) -> Path | None:
"""[[doc-id/p059]] → /Users/guto/ufo/processing/png/doc-id/p-059.png"""
m = re.match(r"\[\[([a-z0-9-]+)/p(\d+)\]\]", page_link)
if not m:
return None
doc_id = m.group(1)
page_num = int(m.group(2))
png = PNG_BASE / doc_id / f"p-{page_num:03d}.png"
return png if png.exists() else None
def crop_table_region(png_path: Path, bbox: dict, out_path: Path, padding: float = 0.005) -> bool:
"""Crop bbox from page PNG (Pillow) and save as JPEG."""
try:
with Image.open(png_path) as im:
W, H = im.size
x = max(0.0, float(bbox.get("x", 0)) - padding)
y = max(0.0, float(bbox.get("y", 0)) - padding)
w = min(1.0 - x, float(bbox.get("w", 0)) + 2 * padding)
h = min(1.0 - y, float(bbox.get("h", 0)) + 2 * padding)
if w <= 0 or h <= 0:
return False
px = int(round(x * W))
py = int(round(y * H))
pw = max(1, int(round(w * W)))
ph = max(1, int(round(h * H)))
crop = im.crop((px, py, px + pw, py + ph))
out_path.parent.mkdir(parents=True, exist_ok=True)
if crop.mode != "RGB":
crop = crop.convert("RGB")
crop.save(out_path, "JPEG", quality=92)
return True
except Exception as e:
sys.stderr.write(f" ✗ crop failed: {e}\n")
return False
def call_haiku_extract(crops: list[Path], n_pages: int) -> tuple[dict | None, str]:
"""Call Haiku via claude CLI with the crops and structured-output prompt."""
crop_list = "\n".join(f" {i+1}. {str(p)}" for i, p in enumerate(crops))
prompt = EXTRACT_PROMPT.format(n_crops=len(crops), n_pages=n_pages, crop_list=crop_list)
cmd = [
"claude", "-p",
"--model", DEFAULT_MODEL,
"--output-format", "json",
"--max-turns", str(MAX_TURNS),
"--allowedTools", "Read",
"--add-dir", str(crops[0].parent),
"--",
prompt,
]
try:
res = subprocess.run(cmd, capture_output=True, text=True, timeout=DEFAULT_TIMEOUT, check=False)
except subprocess.TimeoutExpired:
return None, "timeout"
if res.returncode != 0:
return None, f"rc={res.returncode}: {res.stderr[-300:]}"
try:
cli = json.loads(res.stdout)
except json.JSONDecodeError:
return None, "cli-stdout-not-json"
if cli.get("is_error"):
return None, "is_error"
text = (cli.get("result") or "").strip()
parsed, err = robust_json_parse(text)
if parsed is not None:
return parsed, ""
return None, f"result-not-json: {err}"
def robust_json_parse(text: str) -> tuple[dict | None, str]:
"""Parse JSON tolerant of fences, trailing commentary, unbalanced edges.
Strategy:
1. Strip ``` fences.
2. Try direct json.loads.
3. Find first balanced { ... } block and parse it.
4. As a last resort: rewrite typical Haiku gotchas (smart quotes, trailing
comma before }, unescaped newlines inside strings).
"""
t = text.strip()
t = re.sub(r"^```(?:json)?\s*", "", t)
t = re.sub(r"\s*```$", "", t)
try:
return json.loads(t), ""
except json.JSONDecodeError as e:
first_err = str(e)
# Find balanced { ... }
start = t.find("{")
if start >= 0:
depth = 0
for i in range(start, len(t)):
if t[i] == "{":
depth += 1
elif t[i] == "}":
depth -= 1
if depth == 0:
cand = t[start:i + 1]
try:
return json.loads(cand), ""
except json.JSONDecodeError:
break
# Final pass: remove trailing commas before } or ]
cleaned = re.sub(r",\s*([}\]])", r"\1", t)
try:
return json.loads(cleaned), ""
except json.JSONDecodeError:
return None, first_err
def save_csv(out_csv: Path, headers: list[str], rows: list[list]) -> None:
out_csv.parent.mkdir(parents=True, exist_ok=True)
with out_csv.open("w", newline="", encoding="utf-8") as fh:
w = csv.writer(fh)
w.writerow(headers)
for row in rows:
# Normalize row length to header length
padded = list(row) + [""] * (len(headers) - len(row))
w.writerow(padded[: len(headers)])
def render_table_md_body(table_id: str, fm: dict, parsed: dict | None) -> str:
spans = fm.get("spans_pages") or []
body = f"# {fm.get('canonical_title', table_id)}\n\n"
body += f"> Multi-page table spanning {len(spans)} pages of {fm.get('source_doc','')}\n\n"
body += "## Pages\n\n"
for sp in spans:
body += f"- {sp.get('role','?')}: {sp.get('page','')} · bbox {sp.get('bbox')}\n"
body += "\n"
if parsed:
headers = parsed.get("headers") or []
rows = parsed.get("rows") or []
body += f"## Extracted data ({parsed.get('row_count', len(rows))} rows × {len(headers)} cols)\n\n"
body += f"_Extraction quality: `{parsed.get('extraction_quality')}` · "
body += f"merged cross-page rows: {parsed.get('merged_cross_page_rows', 0)} · "
body += f"CSV: `{fm.get('csv_path')}`_\n\n"
if parsed.get("notes"):
body += f"> **Notes from extraction:** {parsed['notes']}\n\n"
if headers and rows:
body += "| " + " | ".join(headers) + " |\n"
body += "|" + "|".join(["---"] * len(headers)) + "|\n"
for row in rows[:50]:
cells = [str(c).replace("|", "\\|").replace("\n", " ") for c in row]
# pad
cells = cells + [""] * (len(headers) - len(cells))
body += "| " + " | ".join(cells[: len(headers)]) + " |\n"
if len(rows) > 50:
body += f"\n_(showing first 50 of {len(rows)} rows — full CSV in `{fm.get('csv_path')}`)_\n"
else:
body += "## Extracted data\n\n_Extraction not yet run or failed. Run `scripts/16-extract-table-csv.py`._\n"
body += "\n## Notes\n\nPer-page table snippets live in each page.md's `tables_detected[]`. Full row-by-row data is in the CSV at `csv_path`.\n"
return body
def process_table(md_path: Path, force: bool) -> bool:
fm, _ = read_md(md_path)
if fm.get("type") != "table":
return False
if not fm.get("multi_page"):
return False # single-page tables stay inline
table_id = fm.get("table_id") or md_path.stem
csv_path = CSV_BASE / f"{table_id}.csv"
json_path = CSV_BASE / f"{table_id}.json"
crops_dir = CROPS_BASE / table_id
if csv_path.exists() and json_path.exists() and not force:
return False
spans = fm.get("spans_pages") or []
if len(spans) < 2:
return False
print(f"\n=== {table_id} — {len(spans)} pages ===", flush=True)
crops: list[Path] = []
for i, sp in enumerate(spans):
page_link = sp.get("page", "")
bbox = sp.get("bbox") or {}
png = resolve_page_png(page_link)
if not png:
sys.stderr.write(f" ✗ no PNG for {page_link}\n")
return False
crop_out = crops_dir / f"span-{i+1:02d}.jpg"
if not crop_out.exists() or force:
if not crop_table_region(png, bbox, crop_out):
return False
crops.append(crop_out)
print(f" ✓ crop {i+1}: {crop_out.name}", flush=True)
t0 = time.time()
parsed = None
err = ""
for attempt in range(1, 4):
print(f" → calling Haiku (attempt {attempt}/3) to extract CSV from {len(crops)} crops…", flush=True)
parsed, err = call_haiku_extract(crops, n_pages=len(spans))
if parsed:
break
print(f" · attempt {attempt} failed: {err[:120]}", flush=True)
time.sleep(4 * attempt)
elapsed = time.time() - t0
if not parsed:
print(f" ✗ extraction failed after 3 attempts ({elapsed:.1f}s): {err}", flush=True)
return False
headers = parsed.get("headers") or []
rows = parsed.get("rows") or []
if not headers or not rows:
print(f" ⚠ extraction returned empty headers/rows", flush=True)
return False
save_csv(csv_path, headers, rows)
json_path.write_text(json.dumps(parsed, indent=2, ensure_ascii=False), encoding="utf-8")
print(f" ✓ {csv_path.relative_to(UFO_ROOT)} ({len(rows)} rows × {len(headers)} cols, {elapsed:.1f}s)", flush=True)
# Update table.md frontmatter
fm["csv_path"] = str(csv_path.relative_to(UFO_ROOT))
fm["json_path"] = str(json_path.relative_to(UFO_ROOT))
fm["headers"] = headers
fm["row_count_extracted"] = parsed.get("row_count", len(rows))
fm["column_count_extracted"] = parsed.get("column_count", len(headers))
fm["extraction_quality"] = parsed.get("extraction_quality")
fm["extraction_notes"] = parsed.get("notes", "")
fm["extraction_model"] = "claude-haiku-4-5"
fm["extracted_at"] = utc_now_iso()
fm["last_ingest"] = utc_now_iso()
body = render_table_md_body(table_id, fm, parsed)
write_md(md_path, fm, body)
return True
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--table-id", help="single table")
ap.add_argument("--force", action="store_true")
args = ap.parse_args()
CSV_BASE.mkdir(parents=True, exist_ok=True)
CROPS_BASE.mkdir(parents=True, exist_ok=True)
if args.table_id:
targets = [TABLES_BASE / f"{args.table_id}.md"]
else:
targets = sorted(TABLES_BASE.glob("*.md"))
print(f"Processing {len(targets)} table(s)…")
extracted = 0
for t in targets:
if not t.exists():
sys.stderr.write(f" ✗ no table.md: {t}\n")
continue
if process_table(t, args.force):
extracted += 1
print(f"\nExtracted: {extracted} table(s)")
if extracted > 0:
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {utc_now_iso()} — EXTRACT TABLE CSV\n"
f"- operator: archivist + evidence-officer\n- script: scripts/16-extract-table-csv.py\n"
f"- tables_extracted: {extracted}\n"
)
if __name__ == "__main__":
main()

498
scripts/17-enrich-entities.py Executable file
View file

@ -0,0 +1,498 @@
#!/usr/bin/env python3
"""
17-enrich-entities.py — Fase 6 — Enrichment externo de entidades
Para cada entidade em wiki/entities/<class>/<id>.md:
- total_mentions >= 3 → enrichment_status: deep (WebSearch + WebFetch + >=2 sources)
- total_mentions 1-2 → enrichment_status: shallow (1 query + conhecimento interno)
- total_mentions == 0 → enrichment_status: none (skip)
Usa Claude CLI (`claude -p --model haiku`) com tools WebSearch e WebFetch,
mesmo padrão de OAuth/plano Max que 02-vision-page.py.
Pede ao modelo JSON estruturado com:
- biographical_summary EN + PT-BR
- external_sources[] (URL + título + publisher + key_facts + reliability_band)
- additional_aliases, verified_facts
- class-specific (dates pessoa, org_type, coordinates loc, etc.)
Atualiza:
- frontmatter: enrichment_status, external_sources, last_enriched, +campos específicos
- corpo: insere/atualiza seção "## Enrichment (EN)" + "## Enriquecimento (PT-BR)"
PRESERVANDO descrição original (mantém marcador `<!-- enrichment:start -->` ...
`<!-- enrichment:end -->` para idempotência)
Idempotente:
- pula se `last_enriched` < ENRICHMENT_TTL_DAYS atrás (a menos que --force)
- re-rodar não duplica seção (substitui entre marcadores)
Wrap em ThreadPoolExecutor por entidade (timeout 240s) — evita hang do CLI.
Uso:
./17-enrich-entities.py --all [--workers 3] [--force] [--max N] [--tier deep|shallow|all]
./17-enrich-entities.py --class people # apenas pessoas
./17-enrich-entities.py --entity-id j-edgar-hoover
"""
from __future__ import annotations
import argparse
import concurrent.futures
import json
import re
import subprocess
import sys
import threading
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("Missing pyyaml. pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
ENTITIES_BASE = UFO_ROOT / "wiki" / "entities"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
MODEL = "haiku"
WIKI_VERSION = "0.1.0"
ENRICHMENT_TTL_DAYS = 30
DEFAULT_WORKERS = 3
DEFAULT_TIMEOUT_S = 240
DEEP_THRESHOLD = 3 # >= 3 mentions = deep tier
ENRICH_START = "<!-- enrichment:start -->"
ENRICH_END = "<!-- enrichment:end -->"
# Class folder names under wiki/entities/
ENTITY_DIRS = ["people", "organizations", "locations", "events",
"uap-objects", "vehicles", "operations", "concepts"]
_print_lock = threading.Lock()
def safe_print(*args, **kwargs):
with _print_lock:
print(*args, **kwargs, flush=True)
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_md(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end == -1:
return {}, c
try:
return (yaml.safe_load(c[3:end].strip()) or {}), c[end + 3:].lstrip("\n")
except yaml.YAMLError:
return {}, c[end + 3:].lstrip("\n")
def write_md(path: Path, fm: dict, body: str) -> bool:
yaml_str = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
new = f"---\n{yaml_str}---\n\n{body}" if not body.startswith("\n") else f"---\n{yaml_str}---\n{body}"
if path.exists() and path.read_text(encoding="utf-8") == new:
return False
path.write_text(new, encoding="utf-8")
return True
def extract_json(text: str) -> dict:
"""Strip ```json fences then parse. Robust to leading/trailing junk."""
t = text.strip()
t = re.sub(r"^```(?:json)?\s*", "", t)
t = re.sub(r"\s*```$", "", t)
# Try direct
try:
return json.loads(t)
except json.JSONDecodeError:
pass
# Try to find first { ... } balanced block
start = t.find("{")
if start == -1:
raise json.JSONDecodeError("no { in response", t, 0)
depth = 0
for i in range(start, len(t)):
if t[i] == "{":
depth += 1
elif t[i] == "}":
depth -= 1
if depth == 0:
return json.loads(t[start:i + 1])
raise json.JSONDecodeError("unbalanced braces", t, 0)
def is_stale(last_enriched: str | None) -> bool:
if not last_enriched:
return True
try:
ts = datetime.strptime(last_enriched, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
except ValueError:
return True
return (datetime.now(timezone.utc) - ts) > timedelta(days=ENRICHMENT_TTL_DAYS)
def build_prompt(entity_class: str, fm: dict, tier: str) -> str:
canonical_name = fm.get("canonical_name") or fm.get("entity_id") or "?"
aliases = fm.get("aliases") or []
total_mentions = fm.get("total_mentions", 0)
# Class-specific context hints
role_hints = {
"person": "Look up biographical info, role, organization, dates of activity. Distinguish from people with same name (disambiguation).",
"organization": "Look up organization type, founding date, country, mission, leadership. Note any UAP/UFO involvement.",
"location": "Look up coordinates (decimal lat/lon), country, region, type (city/airbase/sea/etc.), notable UAP-related history if any.",
"event": "Look up historical accounts of this event — date, location, official statements, primary sources.",
"uap_object": "External enrichment usually not applicable. Mark enrichment_status: none and explain why in summary.",
"vehicle": "Look up vehicle/aircraft model, operator, specs (if applicable).",
"operation": "Look up operation type (program/task-force/exercise), agency, date range, public knowledge.",
"concept": "Look up canonical definition, legal/scientific context, related programs.",
}
class_hint = role_hints.get(entity_class, "Look up authoritative info; cite sources.")
deep_block = (
"Use the WebSearch tool with 2-4 queries to find authoritative sources "
"(Wikipedia, official government sites, peer-reviewed sources, established news outlets). "
"Use WebFetch on the 2-3 best results to extract key facts. "
"Provide >=2 distinct sources in external_sources[]."
) if tier == "deep" else (
"Use the WebSearch tool with 1 query to confirm/disambiguate. "
"Rely primarily on your own pretraining knowledge for the summary, but cite the 1 web source "
"in external_sources[] (if found). External_sources may be empty if no reliable source surfaced."
)
aliases_str = "\n".join(f" - {a}" for a in aliases[:8]) or " (none)"
prompt = f"""You are an OSINT analyst for the Investigation Bureau — enriching one entity from a US Department of War UAP/UFO archive.
ENTITY CONTEXT:
- Class: {entity_class}
- Canonical name: {canonical_name}
- Aliases / variants in corpus:
{aliases_str}
- Total mentions across corpus: {total_mentions}
- Tier: {tier} (>= {DEEP_THRESHOLD} mentions = deep)
GUIDANCE:
{class_hint}
RESEARCH PROTOCOL:
{deep_block}
Output ONE JSON object only (no markdown fence, no commentary, no preamble). Schema:
{{
"enrichment_status": "{tier}",
"disambiguation_note": "Brief note distinguishing from similar names (e.g., 'NOT to be confused with X who is Y'). Empty string if not applicable.",
"biographical_summary_en": "3-6 sentences English. Focus on identity, role, period of activity, UAP relevance (if any). If genuinely cannot identify the entity (too generic, no public record), say so explicitly.",
"biographical_summary_pt_br": "Same content in Brazilian Portuguese (pt-br, NOT European Portuguese). Preserve UTF-8 accents (ç, ã, é, etc.). Keep proper nouns and English-language verbatim quotes in English.",
"additional_aliases": ["any alternative names, transliterations, common nicknames not already in the aliases list"],
"verified_facts": [
{{ "fact": "single verifiable claim", "source_url": "URL where it was found", "confidence_band": "high|medium|low" }}
],
"external_sources": [
{{ "url": "https://...", "title": "Page title", "publisher": "Wikipedia | NYT | DoD | etc.", "accessed_at": "{utc_now_iso()}", "key_facts": ["short fact 1", "short fact 2"], "reliability_band": "high|medium|low" }}
],
"class_specific": {{
"person": {{"dates": {{"born": "YYYY-MM-DD or null", "died": "YYYY-MM-DD or null"}}, "primary_role": "...", "primary_organization": "..."}},
"organization": {{"organization_type": "intelligence-agency|military-branch|civilian-agency|private-company|ngo|other", "country": "ISO-2 or descriptor", "founded": "YYYY or null"}},
"location": {{"coordinates": {{"lat": 0.0, "lon": 0.0}}, "location_type": "city|airbase|sea|...", "country": ["ISO-2 codes"]}},
"event": {{"date_start": "YYYY-MM-DD or YYYY or null", "primary_location": "...", "event_class": "uap-encounter|disclosure|legal-filing|other"}},
"uap_object": {{"note": "External enrichment usually not applicable for UAP objects."}},
"vehicle": {{"vehicle_class": "aircraft|ship|...", "operator": "...", "model": "..."}},
"operation": {{"operation_type": "military-operation|research-program|task-force|exercise|other", "status": "active|concluded|classified|unknown"}},
"concept": {{"concept_class": "legal-instrument|phenomenon-type|doctrine|scientific-term|jargon|program-name|other", "definition_short_en": "1 sentence", "definition_short_pt_br": "1 frase em pt-br"}}
}}
}}
Rules:
- Provide ONLY the class_specific entry for `{entity_class}`. Other class entries can be omitted.
- If the entity is impossible to identify externally (generic descriptor, common name, redacted), set `external_sources: []` and explain in `biographical_summary_en`.
- ALWAYS preserve UTF-8 accents in PT-BR. Brazilian Portuguese, NOT European.
- Output ONLY the JSON. No fence, no preamble.
"""
return prompt
def call_claude(prompt: str, timeout: int = DEFAULT_TIMEOUT_S) -> tuple[dict, dict]:
"""Invoke claude CLI with WebSearch + WebFetch. Wrapped in ThreadPoolExecutor for hard timeout."""
def _run():
cmd = [
"claude", "-p", "--model", MODEL,
"--output-format", "json",
"--max-turns", "8",
"--allowedTools", "WebSearch,WebFetch",
"--",
prompt,
]
return subprocess.run(cmd, capture_output=True, text=True, timeout=timeout + 30, check=False)
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:
future = ex.submit(_run)
try:
res = future.result(timeout=timeout)
except concurrent.futures.TimeoutError:
raise RuntimeError(f"claude CLI hung > {timeout}s — aborted")
if res.returncode != 0:
raise RuntimeError(f"claude CLI rc={res.returncode}: {res.stderr[-1000:]}")
cli_out = json.loads(res.stdout)
if cli_out.get("is_error"):
raise RuntimeError(f"claude reported error: {cli_out.get('result', '')[:300]}")
enriched = extract_json(cli_out.get("result", ""))
meta = {
"duration_ms": cli_out.get("duration_ms"),
"total_cost_usd": cli_out.get("total_cost_usd"),
"num_turns": cli_out.get("num_turns"),
"session_id": cli_out.get("session_id"),
}
return enriched, meta
def merge_into_frontmatter(fm: dict, enriched: dict, tier: str, now_iso: str) -> dict:
"""Update fm in-place with enrichment results. Returns fm."""
cs = (enriched.get("class_specific") or {}).copy()
# class_specific arrives as a single-key dict in many cases; flatten it
class_specific_payload = {}
if isinstance(cs, dict):
# If it's nested {person: {...}} unwrap; otherwise treat as direct
for v in cs.values():
if isinstance(v, dict) and v:
class_specific_payload = v
break
if not class_specific_payload:
# Maybe already flat
if any(k in cs for k in ("dates", "primary_role", "organization_type", "coordinates",
"date_start", "vehicle_class", "operation_type", "concept_class")):
class_specific_payload = cs
fm["enrichment_status"] = enriched.get("enrichment_status") or tier
fm["last_enriched"] = now_iso
# external_sources (replace, not append — we want a fresh enrichment)
fm["external_sources"] = enriched.get("external_sources") or []
fm["disambiguation_note"] = enriched.get("disambiguation_note") or fm.get("disambiguation_note", "")
fm["verified_facts"] = enriched.get("verified_facts") or []
# Aliases: union
existing_aliases = set(fm.get("aliases") or [])
for a in (enriched.get("additional_aliases") or []):
if isinstance(a, str) and a.strip():
existing_aliases.add(a.strip())
fm["aliases"] = sorted(existing_aliases)
# Class-specific merges
cls = fm.get("entity_class")
if cls == "person" and class_specific_payload:
if class_specific_payload.get("dates"):
fm["dates"] = class_specific_payload["dates"]
if class_specific_payload.get("primary_role"):
fm["primary_role"] = class_specific_payload["primary_role"]
if class_specific_payload.get("primary_organization"):
fm["primary_organization"] = class_specific_payload["primary_organization"]
elif cls == "organization" and class_specific_payload:
for k in ("organization_type", "country", "founded"):
if class_specific_payload.get(k) and not fm.get(k):
fm[k] = class_specific_payload[k]
elif cls == "location" and class_specific_payload:
if class_specific_payload.get("coordinates") and not fm.get("coordinates"):
fm["coordinates"] = class_specific_payload["coordinates"]
for k in ("location_type", "country"):
if class_specific_payload.get(k) and not fm.get(k):
fm[k] = class_specific_payload[k]
elif cls == "event" and class_specific_payload:
for k in ("date_start", "primary_location", "event_class"):
v = class_specific_payload.get(k)
if v and (not fm.get(k) or fm.get(k) in ("NA", "uap-encounter", None)):
fm[k] = v
elif cls == "vehicle" and class_specific_payload:
for k in ("vehicle_class", "operator", "model"):
if class_specific_payload.get(k) and not fm.get(k):
fm[k] = class_specific_payload[k]
elif cls == "operation" and class_specific_payload:
for k in ("operation_type", "status"):
if class_specific_payload.get(k) and not fm.get(k):
fm[k] = class_specific_payload[k]
elif cls == "concept" and class_specific_payload:
if class_specific_payload.get("concept_class"):
fm["concept_class"] = class_specific_payload["concept_class"]
if class_specific_payload.get("definition_short_en"):
fm["definition_short"] = class_specific_payload["definition_short_en"]
if class_specific_payload.get("definition_short_pt_br"):
fm["definition_short_pt_br"] = class_specific_payload["definition_short_pt_br"]
return fm
def upsert_enrichment_section(body: str, enriched: dict) -> str:
"""Replace (or insert before "## Appearances in Corpus" / at end) a bilingual
enrichment section enclosed between ENRICH_START / ENRICH_END markers."""
en = (enriched.get("biographical_summary_en") or "").strip()
pt = (enriched.get("biographical_summary_pt_br") or "").strip()
disamb = (enriched.get("disambiguation_note") or "").strip()
sources = enriched.get("external_sources") or []
section_lines = [ENRICH_START, "## Enrichment (EN)", ""]
if disamb:
section_lines.extend([f"> **Disambiguation:** {disamb}", ""])
section_lines.extend([en or "_No external enrichment available._", "", "## Enriquecimento (PT-BR)", ""])
if disamb:
section_lines.extend([f"> **Desambiguação:** {disamb}", ""])
section_lines.extend([pt or "_Sem enriquecimento externo disponível._", ""])
if sources:
section_lines.extend(["## External Sources", ""])
for s in sources:
url = s.get("url", "")
title = s.get("title", "")
pub = s.get("publisher", "")
rel = s.get("reliability_band", "?")
key = "; ".join(s.get("key_facts", []) or [])
line = f"- [{title or url}]({url}) · _{pub}_ · reliability: `{rel}`"
if key:
line += f" — {key}"
section_lines.append(line)
section_lines.append("")
section_lines.append(ENRICH_END)
new_section = "\n".join(section_lines) + "\n"
# If markers exist, replace between them
if ENRICH_START in body and ENRICH_END in body:
pattern = re.compile(re.escape(ENRICH_START) + r".*?" + re.escape(ENRICH_END) + r"\n?", re.DOTALL)
return pattern.sub(new_section, body)
# Otherwise insert before "## Appearances in Corpus" if present, else append
marker = "## Appearances in Corpus"
if marker in body:
return body.replace(marker, new_section + "\n" + marker)
if not body.endswith("\n"):
body += "\n"
return body + "\n" + new_section
def list_entity_files(class_filter: str | None, entity_id_filter: str | None) -> list[Path]:
"""List entity .md paths, filtered by class and/or entity_id."""
files: list[Path] = []
dirs = [class_filter] if class_filter else ENTITY_DIRS
for d in dirs:
p = ENTITIES_BASE / d
if not p.exists():
continue
for f in sorted(p.glob("*.md")):
if entity_id_filter and f.stem != entity_id_filter:
continue
files.append(f)
return files
def tier_for(total_mentions: int) -> str:
if total_mentions >= DEEP_THRESHOLD:
return "deep"
if total_mentions >= 1:
return "shallow"
return "none"
def process_entity(path: Path, *, force: bool, tier_filter: str, timeout: int) -> tuple[str, str, float]:
"""Returns (action, tier, cost_usd)."""
fm, body = read_md(path)
if not fm:
return ("skip-no-fm", "none", 0.0)
cls = fm.get("entity_class")
if not cls:
return ("skip-no-class", "none", 0.0)
total = int(fm.get("total_mentions") or 0)
tier = tier_for(total)
if tier == "none":
return ("skip-zero", tier, 0.0)
if tier_filter != "all" and tier_filter != tier:
return ("skip-tier-filter", tier, 0.0)
if not force and not is_stale(fm.get("last_enriched")):
return ("skip-fresh", tier, 0.0)
prompt = build_prompt(cls, fm, tier)
t0 = time.time()
enriched, meta = call_claude(prompt, timeout=timeout)
dt = time.time() - t0
new_fm = merge_into_frontmatter(dict(fm), enriched, tier, utc_now_iso())
new_body = upsert_enrichment_section(body, enriched)
changed = write_md(path, new_fm, new_body)
cost = float(meta.get("total_cost_usd") or 0.0)
safe_print(f" {'✓' if changed else '·'} {path.parent.name}/{path.stem} ({tier}, {dt:.1f}s, ${cost:.4f})")
return ("written" if changed else "unchanged", tier, cost)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--all", action="store_true", help="enrich every entity (or use --class / --entity-id)")
ap.add_argument("--class", dest="class_filter", choices=ENTITY_DIRS, help="restrict to one class")
ap.add_argument("--entity-id", help="restrict to one entity stem (filename without .md)")
ap.add_argument("--tier", choices=["all", "deep", "shallow"], default="all")
ap.add_argument("--workers", type=int, default=DEFAULT_WORKERS)
ap.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT_S)
ap.add_argument("--force", action="store_true", help="re-enrich even if last_enriched is fresh")
ap.add_argument("--max", type=int, default=0, help="limit to N entities (0 = no limit)")
args = ap.parse_args()
if not (args.all or args.class_filter or args.entity_id):
ap.error("provide --all, --class, or --entity-id")
files = list_entity_files(args.class_filter, args.entity_id)
if args.max:
files = files[:args.max]
if not files:
print("No entities found.", file=sys.stderr)
return
print(f"Enriching {len(files)} entit(y/ies) with {args.workers} workers, tier={args.tier}, "
f"force={args.force}", flush=True)
stats = {"written": 0, "unchanged": 0, "skip-fresh": 0, "skip-tier-filter": 0,
"skip-zero": 0, "skip-no-fm": 0, "skip-no-class": 0, "errors": 0}
total_cost = 0.0
t_start = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
futures = {pool.submit(process_entity, p, force=args.force,
tier_filter=args.tier, timeout=args.timeout): p for p in files}
for fut in concurrent.futures.as_completed(futures):
p = futures[fut]
try:
action, _tier, cost = fut.result()
stats[action] = stats.get(action, 0) + 1
total_cost += cost
except Exception as e:
stats["errors"] += 1
safe_print(f" ✗ {p.parent.name}/{p.stem}: {type(e).__name__}: {e}")
dt = time.time() - t_start
print(f"\nDone in {dt:.0f}s. Stats: {stats} · total_cost=${total_cost:.2f}", flush=True)
if stats.get("written") or stats.get("errors"):
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {utc_now_iso()} — ENRICH (Phase 6)\n"
f"- operator: profiler\n- script: scripts/17-enrich-entities.py\n"
f"- tier_filter: {args.tier}\n- workers: {args.workers}\n"
f"- written: {stats.get('written', 0)}\n"
f"- unchanged: {stats.get('unchanged', 0)}\n"
f"- skipped_fresh: {stats.get('skip-fresh', 0)}\n"
f"- errors: {stats.get('errors', 0)}\n"
f"- total_cost_usd: {total_cost:.4f}\n"
)
if __name__ == "__main__":
main()

273
scripts/18-build-entity-index.py Executable file
View file

@ -0,0 +1,273 @@
#!/usr/bin/env python3
"""
18-build-entity-index.py — Fase 7.5 — Pre-process entity↔OCR matches
Per page, scan its OCR text against the full alias index of all 14k+
entities. Produces `wiki/pages/<doc-id>/p<NNN>.matches.json` with:
[
{"entity_id": "j-edgar-hoover", "class": "people",
"alias_matched": "Hoover", "start": 423, "end": 429}
]
The frontend uses these to highlight entity mentions inline in the OCR text
and open a modal on click (no runtime string matching).
Performance:
- Builds one big regex with alternation (longest-aliases-first) per class.
- Word boundaries enforced.
- ~10ms per page on the 14k alias index.
Idempotent. Run after `03-dedup-entities.py`. Re-run when entities change.
Usage:
./18-build-entity-index.py # all pages
./18-build-entity-index.py --doc-id <id>
"""
from __future__ import annotations
import argparse
import json
import re
import sys
import unicodedata
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("pip3 install pyyaml\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
WIKI = UFO_ROOT / "wiki"
PAGES = WIKI / "pages"
ENTITIES = WIKI / "entities"
OCR_BASE = UFO_ROOT / "processing" / "ocr"
# Folder name → class key used by the frontend
CLASS_FOLDERS = {
"people": "people",
"organizations": "organizations",
"locations": "locations",
"events": "events",
"uap-objects": "uap-objects",
"vehicles": "vehicles",
"operations": "operations",
"concepts": "concepts",
}
# Aliases shorter than this are skipped (too many false positives on common words)
MIN_ALIAS_LEN = 3
# Stop-aliases — common nouns extracted as entities by the vision pass that
# would generate runaway matches.
STOP_ALIASES = {
"the", "and", "for", "with", "from", "this", "that", "have", "has", "had",
"they", "them", "their", "his", "her", "him", "she", "you", "your",
"page", "report", "document", "subject", "date", "time", "file", "case",
"memo", "letter", "office", "section", "general", "agent", "info",
"see", "ref", "via", "etc", "inc", "ltd", "the bureau", "the agency",
"the department", "the office", "the file", "the case", "the report",
"yes", "no", "ok", "etc.", "i.e.", "e.g.",
}
def normalize(s: str) -> str:
nfd = unicodedata.normalize("NFD", s)
return "".join(c for c in nfd if not unicodedata.combining(c)).lower()
def read_md_fm(path: Path) -> dict:
try:
c = path.read_text(encoding="utf-8")
except FileNotFoundError:
return {}
if not c.startswith("---"):
return {}
end = c.find("---", 4)
if end == -1:
return {}
try:
return yaml.safe_load(c[3:end].strip()) or {}
except yaml.YAMLError:
return {}
def collect_aliases() -> list[tuple[str, str, str, str]]:
"""Returns list of (alias_normalized_lower, alias_original, class, entity_id)."""
rows: list[tuple[str, str, str, str]] = []
for folder, cls in CLASS_FOLDERS.items():
d = ENTITIES / folder
if not d.exists():
continue
for f in d.glob("*.md"):
entity_id = f.stem
fm = read_md_fm(f)
if not fm:
continue
names = set()
cname = fm.get("canonical_name")
if isinstance(cname, str) and cname.strip():
names.add(cname.strip())
for a in (fm.get("aliases") or []):
if isinstance(a, str) and a.strip():
names.add(a.strip())
for n in names:
if len(n) < MIN_ALIAS_LEN:
continue
norm = normalize(n)
if norm in STOP_ALIASES:
continue
if not re.search(r"[a-z]", norm):
continue
rows.append((norm, n, cls, entity_id))
return rows
def build_megaregex(aliases: list[tuple[str, str, str, str]]) -> tuple[re.Pattern, list[tuple[str, str, str]]]:
"""Build one big regex with alternation, longest-first.
Returns (compiled_pattern, payload_table) where payload[i] = (alias_original, class, entity_id).
"""
# Sort by length DESC so the longest alias wins on overlap
sorted_aliases = sorted(aliases, key=lambda r: -len(r[0]))
parts: list[str] = []
payload: list[tuple[str, str, str]] = []
seen: set[str] = set()
for norm, orig, cls, eid in sorted_aliases:
if norm in seen:
continue
seen.add(norm)
# Escape regex specials in the normalized alias; word-boundary on both sides
parts.append(re.escape(norm))
payload.append((orig, cls, eid))
big = r"\b(?:" + "|".join(parts) + r")\b"
pat = re.compile(big, re.IGNORECASE)
return pat, payload
def match_page_text(ocr_text: str, pat: re.Pattern, payload: list[tuple[str, str, str]],
alias_to_idx: dict[str, int]) -> list[dict]:
"""Return list of match dicts.
OCR is matched on lower+ASCII-folded text BUT we record start/end against the ORIGINAL OCR
string so the frontend can slice the original (with accents, punctuation) correctly.
Strategy: build a char-index map normalized→original. Since NFD ASCII-fold can change length
(rare; mostly preserves), we use a simpler approach: match on a 1:1 lower-cased version of
the OCR (preserving length) and a separate normalized lowercased OCR for searching, then
map indices back. To keep it simple AND correct, just match against `ocr_text.lower()` —
accents are preserved, and `re.IGNORECASE` already handles case. The normalize() above
only matters for de-duplicating alias keys; the regex itself matches ascii→ascii via the
escape() applied to normalized strings, which is fine because most OCR text is ASCII.
"""
matches: list[dict] = []
# We compile patterns from normalized lowercased aliases. To match correctly we run the
# regex on a normalized lowercased OCR view, then map back to original indices using the
# length-preservation property of unicode lower() + NFD fold for typical Latin-1 chars.
# For simplicity: match on the ASCII-folded lowercased OCR and assume same length.
nfd = unicodedata.normalize("NFD", ocr_text)
# Length is preserved if we drop combining marks AND record orig positions per non-combining char
orig_idx: list[int] = []
folded_chars: list[str] = []
for i, c in enumerate(nfd):
if unicodedata.combining(c):
continue
folded_chars.append(c.lower())
# Map this folded char back to OCR position: walk original OCR
# ^ For accuracy, recompute via per-char NFD inverse — simpler approach below
folded = "".join(folded_chars)
# Build mapping: position in `folded` → position in ocr_text
# ocr_text → NFD → drop combining → folded. Each kept char corresponds to one source char in
# ocr_text (the base char that produced it after NFD). We walk ocr_text and count.
ocr_to_folded: list[int] = [] # ocr_to_folded[i] = folded position for ocr_text[i] (or last seen)
folded_to_ocr: list[int] = [] # folded_to_ocr[k] = ocr_text position for folded[k]
fi = 0
for i, ch in enumerate(ocr_text):
nfd_ch = unicodedata.normalize("NFD", ch)
kept = [c for c in nfd_ch if not unicodedata.combining(c)]
if kept:
folded_to_ocr.append(i)
fi += 1
ocr_to_folded.append(fi - 1)
for m in pat.finditer(folded):
start_f, end_f = m.start(), m.end()
if start_f >= len(folded_to_ocr) or end_f - 1 >= len(folded_to_ocr):
continue
start_o = folded_to_ocr[start_f]
end_o = folded_to_ocr[end_f - 1] + 1
text = m.group(0)
idx = alias_to_idx.get(text)
if idx is None:
continue
orig, cls, eid = payload[idx]
matches.append({
"entity_id": eid,
"class": cls,
"alias_matched": ocr_text[start_o:end_o],
"start": start_o,
"end": end_o,
})
return matches
def process_page(doc_dir: Path, page_md: Path, pat: re.Pattern, payload: list[tuple[str, str, str]],
alias_to_idx: dict[str, int], force: bool) -> bool:
doc_id = doc_dir.name
stem = page_md.stem # e.g., "p007"
m = re.match(r"p(\d+)", stem)
if not m:
return False
page_num = int(m.group(1))
out = doc_dir / f"{stem}.matches.json"
if out.exists() and not force:
return False
padded = f"{page_num:03d}"
ocr_path = OCR_BASE / doc_id / f"p-{padded}.txt"
try:
ocr_text = ocr_path.read_text(encoding="utf-8")
except FileNotFoundError:
return False
matches = match_page_text(ocr_text, pat, payload, alias_to_idx)
out.write_text(json.dumps(matches, ensure_ascii=False), encoding="utf-8")
return True
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id", help="single doc")
ap.add_argument("--force", action="store_true")
args = ap.parse_args()
print("Collecting aliases from all entities...", flush=True)
aliases = collect_aliases()
print(f" {len(aliases)} (alias, entity) pairs", flush=True)
print("Building mega-regex...", flush=True)
pat, payload = build_megaregex(aliases)
alias_to_idx = {norm: i for i, (orig, cls, eid) in enumerate(payload)
for norm in [orig.lower()]}
# ^ but pat matches with IGNORECASE on the folded text, so we need normalized→idx
# Rebuild correctly: walk payload, derive normalized form
alias_to_idx = {}
for i, (orig, cls, eid) in enumerate(payload):
norm = normalize(orig)
alias_to_idx[norm] = i
print(f" pattern has {len(payload)} unique aliases", flush=True)
docs = [PAGES / args.doc_id] if args.doc_id else sorted(d for d in PAGES.iterdir() if d.is_dir())
total_written = 0
total_pages = 0
for doc_dir in docs:
for page_md in sorted(doc_dir.glob("p*.md")):
total_pages += 1
if process_page(doc_dir, page_md, pat, payload, alias_to_idx, args.force):
total_written += 1
print(f"\nDone: {total_written} matches.json (re)written across {total_pages} pages", flush=True)
if __name__ == "__main__":
main()

243
scripts/19-detect-vision-mismatch.py Executable file
View file

@ -0,0 +1,243 @@
#!/usr/bin/env python3
"""
19-detect-vision-mismatch.py — Lint pass to find Haiku exaggerations.
Detects pages whose `vision_description` claims heavy redaction/obscurity but
the actual `redactions[]` count or bbox coverage tells a milder story. Marks
flagged pages with `flags: ["vision-redaction-mismatch"]` AND optionally
re-runs vision with claude-sonnet to fix.
Heuristics (any one is enough to flag):
H1. Text contains hyperbolic redaction phrasing AND redactions[] is small.
H2. Text claims a high percentage obscured AND actual bbox area coverage is much lower.
H3. Text contradicts content_classification (e.g. says "redaction-heavy" but
content_classification doesn't include "redaction-heavy").
Usage:
./19-detect-vision-mismatch.py --doc-id <id> --page p173 [--explain]
./19-detect-vision-mismatch.py --all [--reanalyze]
./19-detect-vision-mismatch.py --all --dry-run # report only
"""
from __future__ import annotations
import argparse
import json
import re
import subprocess
import sys
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("pip3 install pyyaml\n"); sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
PAGES = UFO_ROOT / "wiki" / "pages"
# Regexes for hyperbolic claims about redactions.
HEAVY_RE = re.compile(
r"(heavy\s+redact|substantial(ly)?\s+redact|extensiv(e|ely)\s+redact"
r"|significantly\s+redact|major\s+portion[s]?\s+(of\s+the\s+(form|page|content))?(\s+are|is)?\s+(obscured|hidden|blacked)"
r"|approximately\s+\d{2,3}%|roughly\s+\d{2,3}%|about\s+\d{2,3}%"
r"|solid\s+black\s+bars|redaction-heavy|mostly\s+redact|page\s+is\s+(largely|mostly|primarily)\s+(redacted|obscured)"
r")",
re.IGNORECASE,
)
PCT_RE = re.compile(r"(\d{2,3})\s*%", re.IGNORECASE)
def read_fm(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end < 0:
return {}, c
try:
fm = yaml.safe_load(c[3:end].strip()) or {}
except yaml.YAMLError:
fm = {}
return fm, c[end + 3 :].lstrip("\n")
def bbox_area_pct(redactions: list[dict]) -> float:
"""Sum of bbox areas (in % of page). Cap at 100."""
total = 0.0
for r in redactions:
b = r.get("bbox") or {}
w = float(b.get("w") or 0)
h = float(b.get("h") or 0)
total += max(0, w) * max(0, h)
return min(100.0, total * 100)
def analyse_page(fm: dict) -> tuple[bool, list[str]]:
"""Return (is_mismatch, reasons[])."""
reasons: list[str] = []
vd_en = (fm.get("vision_description") or "")
vd_pt = (fm.get("vision_description_pt_br") or "")
text = f"{vd_en}\n{vd_pt}"
redactions = fm.get("redactions") or []
n_red = len(redactions)
area = bbox_area_pct(redactions)
cc = fm.get("content_classification") or []
heavy_match = HEAVY_RE.search(text)
pct_match = PCT_RE.search(text)
claimed_pct = int(pct_match.group(1)) if pct_match else None
# H1: text claims "heavy" but redactions count is small
if heavy_match and n_red < 5:
reasons.append(f"H1: text says '{heavy_match.group(0)}' but only {n_red} redactions detected")
# H2: claimed % vs actual bbox area
if claimed_pct is not None and claimed_pct >= 25:
if area < claimed_pct * 0.4: # claim is >2.5× the actual coverage
reasons.append(f"H2: text claims ~{claimed_pct}% obscured but bbox area is {area:.1f}%")
# H3: text says redaction-heavy but content_classification disagrees
if heavy_match and "redaction-heavy" not in cc:
reasons.append(f"H3: text says heavy redaction but content_classification = {cc}")
return (len(reasons) > 0, reasons)
def run_sonnet_reanalysis(page_path: Path, fm: dict) -> dict | None:
"""Re-run vision with claude-sonnet via CLI (OAuth). Returns new fm fields or None."""
doc_id = fm.get("doc_id", "")
page_num = int(fm.get("page_number", 0))
if not doc_id or not page_num:
return None
padded = f"{page_num:03d}"
png = UFO_ROOT / "processing" / "png" / doc_id / f"p-{padded}.png"
if not png.exists():
return None
# Reuse the same prompt shape as 02-vision-page.py but ask Sonnet, and
# emphasize precise quantification of redactions.
prompt = f"""Re-analyze this US Department of War declassified UAP page with HIGH precision.
You are being run because a prior Haiku pass produced text that exaggerated the redaction coverage.
STEP 1: Use the Read tool to view this PNG: {png}
STEP 2: Output ONE JSON object (no markdown fence, no preamble) with EXACTLY these keys:
- vision_description: 2-5 sentences English. **Be precise about redaction extent**. Only say "heavy" if >30% of the page is genuinely covered by solid black bars. Count redactions accurately. Avoid hyperbole.
- vision_description_pt_br: same content in Brazilian Portuguese (preserve UTF-8 accents).
- redactions_revised: array of {{code, description, bbox: {{x,y,w,h}}}} — list every actual redaction box you can see, with normalized 0..1 bbox coordinates.
- reanalysis_confidence: float 0..1.
Output ONLY the JSON. No fence."""
try:
proc = subprocess.run(
["claude", "-p", "--model", "sonnet",
"--output-format", "json",
"--max-turns", "3",
"--allowedTools", "Read",
"--add-dir", str(png.parent),
"--", prompt],
capture_output=True, text=True, timeout=180, check=False,
)
if proc.returncode != 0:
sys.stderr.write(f" Sonnet rc={proc.returncode}: {proc.stderr[-300:]}\n")
return None
cli = json.loads(proc.stdout)
if cli.get("is_error"):
return None
result_text = (cli.get("result") or "").strip()
# Strip ``` fences if any
result_text = re.sub(r"^```(?:json)?\s*", "", result_text)
result_text = re.sub(r"\s*```$", "", result_text)
return json.loads(result_text)
except Exception as e:
sys.stderr.write(f" Sonnet error: {e}\n")
return None
def process(page_path: Path, *, reanalyze: bool, dry_run: bool, explain: bool, force: bool = False) -> str:
fm, body = read_fm(page_path)
if not fm:
return "no-fm"
is_mismatch, reasons = analyse_page(fm)
if force and not is_mismatch:
is_mismatch = True
reasons.append("FORCED by user (heuristics did not auto-detect)")
if not is_mismatch:
return "ok"
if explain:
print(f"⚠ {page_path.relative_to(UFO_ROOT)}")
for r in reasons:
print(f" · {r}")
vd = (fm.get("vision_description") or "")[:200]
print(f" text excerpt: \"{vd}…\"")
print(f" n_redactions: {len(fm.get('redactions') or [])}, "
f"bbox area: {bbox_area_pct(fm.get('redactions') or []):.1f}%")
flags = list(fm.get("flags") or [])
if "vision-redaction-mismatch" not in flags:
flags.append("vision-redaction-mismatch")
fm["flags"] = flags
if reanalyze and not dry_run:
print(f" → re-analyzing with Sonnet…", flush=True)
revision = run_sonnet_reanalysis(page_path, fm)
if revision:
if revision.get("vision_description"):
fm["vision_description"] = revision["vision_description"]
if revision.get("vision_description_pt_br"):
fm["vision_description_pt_br"] = revision["vision_description_pt_br"]
if revision.get("redactions_revised"):
fm["redactions"] = revision["redactions_revised"]
fm["last_reanalysis_model"] = "claude-sonnet-4-6"
if "vision-redaction-mismatch" in fm["flags"]:
fm["flags"].remove("vision-redaction-mismatch")
print(f" ✓ rewrote vision_description (now {len(fm.get('redactions') or [])} redactions)")
else:
print(f" ✗ Sonnet call failed; flag preserved")
if dry_run:
return "flag-dry"
new_yaml = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
new = f"---\n{new_yaml}---\n\n{body}" if not body.startswith("\n") else f"---\n{new_yaml}---\n{body}"
page_path.write_text(new, encoding="utf-8")
return "flagged"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id", help="single doc")
ap.add_argument("--page", help="specific page stem, e.g. p173 (requires --doc-id)")
ap.add_argument("--all", action="store_true")
ap.add_argument("--reanalyze", action="store_true", help="invoke Sonnet to fix mismatched pages")
ap.add_argument("--force", action="store_true", help="treat targeted pages as mismatch (bypass heuristics)")
ap.add_argument("--dry-run", action="store_true", help="report only, don't write")
ap.add_argument("--explain", action="store_true", help="print why each page was flagged")
args = ap.parse_args()
if args.doc_id and args.page:
targets = [PAGES / args.doc_id / f"{args.page}.md"]
elif args.doc_id:
targets = sorted((PAGES / args.doc_id).glob("p*.md"))
elif args.all:
targets = sorted(PAGES.glob("*/p*.md"))
else:
ap.error("provide --doc-id (+ --page) or --all")
stats = {"ok": 0, "flagged": 0, "flag-dry": 0, "no-fm": 0}
for p in targets:
if not p.exists():
sys.stderr.write(f"✗ missing: {p}\n"); continue
r = process(p, reanalyze=args.reanalyze, dry_run=args.dry_run, explain=args.explain, force=args.force)
stats[r] = stats.get(r, 0) + 1
print(f"\nDone. {stats}")
if __name__ == "__main__":
main()

289
scripts/20-reanalyze-vision-gemini.py Executable file
View file

@ -0,0 +1,289 @@
#!/usr/bin/env python3
"""
20-reanalyze-vision-gemini.py — Fallback re-vision via Gemini 3.1 Pro.
When Haiku exaggerates (e.g., claims "45% redaction-heavy" on a clearly
readable page), this script re-analyzes via Gemini 3.1 Pro and rewrites the
page.md frontmatter (vision_description, vision_description_pt_br, redactions).
Targets:
--doc-id <id> --page p173 → single page
--doc-id <id> → entire doc
--flagged → all pages with flags: ["vision-redaction-mismatch"]
--all → every page (slow + costly; use sparingly)
Anti-hang: ThreadPoolExecutor + future.result(timeout=180s) per memory
`feedback-gemini-sdk-hangs.md`.
Output: overwrites page.md frontmatter fields (vision_description,
vision_description_pt_br, redactions). Preserves everything else. Adds
`last_reanalysis_model` and `last_reanalysis_at`.
Usage:
GEMINI_API_KEY=... ./20-reanalyze-vision-gemini.py --doc-id <id> --page p173
"""
from __future__ import annotations
import argparse
import concurrent.futures
import json
import os
import re
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("pip3 install pyyaml\n"); sys.exit(1)
try:
from google import genai
from google.genai import types as genai_types
except ImportError:
sys.stderr.write("pip3 install google-genai\n"); sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
PAGES = UFO_ROOT / "wiki" / "pages"
PNG_BASE = UFO_ROOT / "processing" / "png"
DEFAULT_MODEL = "gemini-3.1-pro-preview"
FALLBACK = ["gemini-3-pro-preview", "gemini-3.1-flash-lite"]
TIMEOUT_S = 180
PROMPT = """You are re-analyzing one page of a US Department of War declassified UAP/UFO document. A previous Haiku pass produced an EXAGGERATED description (it claimed heavy redaction coverage when the actual page was largely readable). Your job: produce a PRECISE replacement.
GROUND RULES:
- Count redactions EXACTLY. Each redaction is a solid black bar / opaque cover blocking specific text.
- Do NOT call a page "heavy redaction" unless >30% of its visible area is genuinely obscured.
- For each redaction, return a tight bbox (normalized 0..1 coords) that covers ONLY the black bar, not the whole line.
- If the page has NO redactions, return an empty array. If it has thin strips, give them small bboxes.
Output ONE JSON object (no fence, no preamble):
{
"vision_description": "2-5 sentences in English. Describe what is actually visible: layout, content category, classification markings, any redaction precisely quantified. Use plain language, no hyperbole.",
"vision_description_pt_br": "Mesmo conteúdo em português brasileiro (pt-br). Preserve acentos UTF-8. Mantenha citações verbatim do documento em inglês (não traduza texto que está dentro do documento).",
"redactions": [
{"code": "(b)(1) 1.4(a)|(b)(3)|(b)(6)|other", "description": "what field/text was obscured", "bbox": {"x": 0.0, "y": 0.0, "w": 0.0, "h": 0.0}, "text_inferred": null}
],
"content_classification": ["text-only"|"contains-photos"|"contains-sketches"|"contains-diagrams"|"contains-maps"|"contains-tables"|"contains-signatures"|"contains-stamps"|"redaction-heavy"|"mixed"|"blank"],
"page_type": "cover|toc|body|signature|photo|sketch|map|stamp|blank|appendix|redaction-heavy|table-page|mixed",
"reanalysis_confidence": 0.0
}
"""
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_fm(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end < 0:
return {}, c
try:
fm = yaml.safe_load(c[3:end].strip()) or {}
except yaml.YAMLError:
fm = {}
return fm, c[end + 3 :].lstrip("\n")
def write_fm(path: Path, fm: dict, body: str) -> None:
new_yaml = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
sep = "\n" if body.startswith("\n") else "\n\n"
path.write_text(f"---\n{new_yaml}---{sep}{body}", encoding="utf-8")
def call_gemini(client, png_path: Path, model: str, attempt: int = 1):
"""Vision call with thread-based timeout (anti-hang)."""
content = [
genai_types.Part.from_bytes(data=png_path.read_bytes(), mime_type="image/png"),
PROMPT,
]
def _call():
return client.models.generate_content(
model=model,
contents=content,
config=genai_types.GenerateContentConfig(
response_mime_type="application/json",
temperature=0.2,
max_output_tokens=16384, # bumped iteratively (4096 → 8192 → 16384) for verbose pages
),
)
try:
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:
future = ex.submit(_call)
try:
resp = future.result(timeout=TIMEOUT_S)
except concurrent.futures.TimeoutError:
raise RuntimeError(f"Gemini hung >{TIMEOUT_S}s")
return resp.text, model
except Exception as e:
if attempt < len(FALLBACK) + 1:
next_m = FALLBACK[attempt - 1] if attempt <= len(FALLBACK) else None
if next_m:
sys.stderr.write(f" ⚠ {model} failed ({type(e).__name__}); fallback → {next_m}\n")
return call_gemini(client, png_path, next_m, attempt + 1)
raise
def parse_json_lenient(text: str) -> dict:
t = text.strip()
t = re.sub(r"^```(?:json)?\s*", "", t)
t = re.sub(r"\s*```$", "", t)
return json.loads(t)
def process_page(client, page_md: Path, dry_run: bool) -> str:
fm, body = read_fm(page_md)
if not fm:
return "no-fm"
doc_id = fm.get("doc_id", "")
page_num = int(fm.get("page_number", 0))
if not doc_id or not page_num:
return "bad-fm"
padded = f"{page_num:03d}"
png = PNG_BASE / doc_id / f"p-{padded}.png"
if not png.exists():
return "no-png"
print(f" → {page_md.relative_to(UFO_ROOT)} (Gemini 3.1 Pro)", flush=True)
t0 = time.time()
try:
raw, model_used = call_gemini(client, png, DEFAULT_MODEL)
except Exception as e:
print(f" ✗ Gemini failed: {type(e).__name__}: {e}", flush=True)
return "error"
dt = time.time() - t0
try:
revision = parse_json_lenient(raw)
except json.JSONDecodeError as e:
print(f" ✗ JSON parse failed: {e}; raw[:200]={raw[:200]!r}", flush=True)
return "bad-json"
# Before/after summary
old_n = len(fm.get("redactions") or [])
new_n = len(revision.get("redactions") or [])
old_desc = (fm.get("vision_description") or "")[:90]
new_desc = (revision.get("vision_description") or "")[:90]
print(f" redactions: {old_n} → {new_n}")
print(f" OLD desc: {old_desc}…")
print(f" NEW desc: {new_desc}…")
if dry_run:
return "dry"
# Apply revision
if revision.get("vision_description"):
fm["vision_description"] = revision["vision_description"]
if revision.get("vision_description_pt_br"):
fm["vision_description_pt_br"] = revision["vision_description_pt_br"]
if "redactions" in revision:
fm["redactions"] = revision["redactions"]
if revision.get("content_classification"):
fm["content_classification"] = revision["content_classification"]
if revision.get("page_type"):
fm["page_type"] = revision["page_type"]
fm["last_reanalysis_model"] = model_used
fm["last_reanalysis_at"] = utc_now_iso()
fm["last_reanalysis_confidence"] = revision.get("reanalysis_confidence")
# Remove the mismatch flag now that it's been corrected
flags = list(fm.get("flags") or [])
if "vision-redaction-mismatch" in flags:
flags.remove("vision-redaction-mismatch")
fm["flags"] = flags
write_fm(page_md, fm, body)
print(f" ✓ wrote (took {dt:.1f}s)", flush=True)
return "ok"
def main():
global DEFAULT_MODEL
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id")
ap.add_argument("--page", help="specific page stem, e.g. p173 (requires --doc-id)")
ap.add_argument("--flagged", action="store_true", help="all pages with vision-redaction-mismatch")
ap.add_argument("--redaction-heavy", action="store_true", help="all pages currently classified redaction-heavy (re-triage)")
ap.add_argument("--all", action="store_true")
ap.add_argument("--pages-file", help="newline-separated list of page paths (relative to /Users/guto/ufo/ or absolute)")
ap.add_argument("--model", default=DEFAULT_MODEL, help=f"override model (default {DEFAULT_MODEL})")
ap.add_argument("--workers", type=int, default=1, help="parallel workers (raise for Flash Lite, keep 1 for Pro free tier)")
ap.add_argument("--max", type=int, default=0, help="cap targets (0 = unlimited)")
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args()
api_key = os.environ.get("GEMINI_API_KEY")
if not api_key:
sys.stderr.write("✗ GEMINI_API_KEY not set\n"); sys.exit(1)
client = genai.Client(api_key=api_key)
if args.doc_id and args.page:
targets = [PAGES / args.doc_id / f"{args.page}.md"]
elif args.doc_id:
targets = sorted((PAGES / args.doc_id).glob("p*.md"))
elif args.flagged:
targets = []
for p in PAGES.glob("*/p*.md"):
fm, _ = read_fm(p)
if "vision-redaction-mismatch" in (fm.get("flags") or []):
targets.append(p)
elif args.redaction_heavy:
targets = []
for p in PAGES.glob("*/p*.md"):
fm, _ = read_fm(p)
if "redaction-heavy" in (fm.get("content_classification") or []):
targets.append(p)
elif args.all:
targets = sorted(PAGES.glob("*/p*.md"))
elif args.pages_file:
targets = []
for line in Path(args.pages_file).read_text().splitlines():
s = line.strip()
if not s:
continue
p = Path(s) if s.startswith("/") else UFO_ROOT / s
targets.append(p)
else:
ap.error("provide --doc-id (+ --page), --flagged, --redaction-heavy, --all, or --pages-file")
if args.max:
targets = targets[:args.max]
DEFAULT_MODEL = args.model
print(f"Processing {len(targets)} page(s) with {DEFAULT_MODEL} ({args.workers} worker(s))...")
stats = {"ok": 0, "error": 0, "dry": 0, "no-png": 0, "no-fm": 0, "bad-fm": 0, "bad-json": 0}
if args.workers <= 1:
for p in targets:
if not p.exists(): stats["no-fm"] += 1; continue
r = process_page(client, p, args.dry_run)
stats[r] = stats.get(r, 0) + 1
else:
import concurrent.futures as cf
with cf.ThreadPoolExecutor(max_workers=args.workers) as pool:
futs = {pool.submit(process_page, client, p, args.dry_run): p for p in targets if p.exists()}
for fut in cf.as_completed(futs):
try:
r = fut.result()
stats[r] = stats.get(r, 0) + 1
except Exception as e:
sys.stderr.write(f"✗ {futs[fut]}: {e}\n")
stats["error"] += 1
print(f"\nDone. {stats}")
if __name__ == "__main__":
main()

243
scripts/21-reextract-entities-gemini.py Executable file
View file

@ -0,0 +1,243 @@
#!/usr/bin/env python3
"""
21-reextract-entities-gemini.py — Phase 2: rewrite ONLY `entities_extracted` in
each page.md using Gemini 3.0 Flash with explicit anti-fragmentation rules.
Fixes the Haiku extraction bugs:
- "Gudauta Base" was split into location "Gudauta" + organization "Base"
- "Chief Tereoken" was split into "Chief" + "Tereoken"
- Bare common nouns ("Base", "Chief", "Department") promoted to standalone entities
- Variants of same entity ("FBI" / "F-B-I" / "Federal Bureau") not normalized at source
Preserves everything else in the page.md frontmatter.
Usage:
./21-reextract-entities-gemini.py --all --workers 20
./21-reextract-entities-gemini.py --doc-id <id>
./21-reextract-entities-gemini.py --page <doc-id>/p007 # quick test
"""
from __future__ import annotations
import argparse
import concurrent.futures
import json
import os
import re
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("pip3 install pyyaml\n"); sys.exit(1)
try:
from google import genai
from google.genai import types as genai_types
except ImportError:
sys.stderr.write("pip3 install google-genai\n"); sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
PAGES = UFO_ROOT / "wiki" / "pages"
PNG_BASE = UFO_ROOT / "processing" / "png"
DEFAULT_MODEL = "gemini-3-flash-preview"
TIMEOUT_S = 180
PROMPT = """You are an OSINT investigator extracting entities from one page of a US Department of War declassified UAP/UFO document. A prior extraction pass made systematic mistakes; your job is to do it right.
CRITICAL RULES — NON-NEGOTIABLE:
1. **Compound names stay together.** "Gudauta Base" is ONE location, not two entities "Gudauta" + "Base". "Chief Tereoken" is ONE person, not "Chief" + "Tereoken". "FBI Seattle Field Office" is ONE organization. NEVER split a compound name into separate entities.
2. **Titles + names are ONE person.** "Chief Tereoken", "LCDR Smith", "Mr. Johnson", "Capt. Davis", "Mrs. Anderson", "Dr. Hynek", "General Marshall" — each is ONE single person entity, with the title as part of the canonical name.
3. **NEVER extract bare common nouns as entities.** Skip: "Chief", "Base", "Department", "Office", "Agent", "Bureau", "Captain", "Officer", "File", "Subject", "Memo", "Letter", "Report", "Page", "Bag", "Stamp", "Signature", "Carbon Copy". These are only meaningful when COMBINED with a proper name.
4. **Normalize variants at the source.** "F.B.I.", "F-B-I", "FBI", "Federal Bureau of Investigation" → all return as the SINGLE canonical form "Federal Bureau of Investigation" (with "FBI" added to aliases).
5. **Distinguish entity types precisely:**
- `locations`: physical places (cities, countries, military bases, geographic features). "Gudauta Base" → location (it's a military base). "Adapazari, Turkey" → location.
- `organizations`: institutions, agencies, branches, companies. "FBI", "USAF", "CIA Foreign Branch".
- `people`: humans with names (titles ok). "J. Edgar Hoover", "Chief Tereoken".
- `events`: dated incidents with a date or short label. "Tic-Tac Nimitz 2004", "Roswell 1947".
- `uap_objects`: described UAP themselves. Shape + color + size description.
- `vehicles`: aircraft, ships, vehicles by model/name. "USS Princeton", "F-18".
- `operations`: programs, missions, protocols by name. "Project Blue Book", "Operation Mainbrace".
- `concepts`: legal/scientific/jargon. "FOIA exemption (b)(1)", "GENTEXT", "compartmentalization".
Output ONE JSON object only (no markdown fence, no preamble) with this exact schema:
{
"entities_extracted": {
"people": [{"name": "Full canonical name with title", "role_in_page": "subject|witness|author|signer|mentioned", "aliases": ["alt spellings"]}],
"organizations": [{"name": "Canonical org name", "aliases": ["FBI", "F.B.I."], "type": "intelligence-agency|military-branch|civilian-agency|...|other"}],
"locations": [{"name": "Canonical place name including any qualifier (Gudauta Base, not just Gudauta)", "type": "city|region|country|sea|strait|airbase|naval-base|mountain|desert|building|other"}],
"events": [{"label": "Short distinctive label", "date": "YYYY-MM-DD|YYYY|NA"}],
"uap_objects": [{"shape": "...", "color": "...", "size_estimate": "..."}],
"vehicles": [{"name": "...", "class": "aircraft|ship|submarine|spacecraft|satellite|ground|other"}],
"operations": [{"name": "...", "type": "military-operation|reporting-protocol|research-program|task-force|foia-disclosure|other"}],
"concepts": [{"name": "...", "class": "legal-instrument|phenomenon-type|doctrine|scientific-term|jargon|program-name|other"}]
}
}
If a category has no entries, return an empty array. PRESERVE original spelling (do not translate names). Output ONLY the JSON."""
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_fm(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end < 0:
return {}, c
try:
fm = yaml.safe_load(c[3:end].strip()) or {}
except yaml.YAMLError:
fm = {}
return fm, c[end + 3:].lstrip("\n")
def write_fm(path: Path, fm: dict, body: str) -> None:
new_yaml = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
sep = "\n" if body.startswith("\n") else "\n\n"
path.write_text(f"---\n{new_yaml}---{sep}{body}", encoding="utf-8")
def call_gemini(client, png_path: Path, model: str, max_tokens: int = 32768):
content = [
genai_types.Part.from_bytes(data=png_path.read_bytes(), mime_type="image/png"),
PROMPT,
]
def _call():
return client.models.generate_content(
model=model,
contents=content,
config=genai_types.GenerateContentConfig(
response_mime_type="application/json",
temperature=0.1,
max_output_tokens=max_tokens,
),
)
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:
future = ex.submit(_call)
try:
return future.result(timeout=TIMEOUT_S).text
except concurrent.futures.TimeoutError:
raise RuntimeError(f"Gemini hung >{TIMEOUT_S}s")
def parse_json_lenient(text: str) -> dict:
t = text.strip()
t = re.sub(r"^```(?:json)?\s*", "", t)
t = re.sub(r"\s*```$", "", t)
return json.loads(t)
def process_page(client, page_md: Path, model: str) -> str:
fm, body = read_fm(page_md)
if not fm:
return "no-fm"
doc_id = fm.get("doc_id", "")
page_num = int(fm.get("page_number", 0))
if not doc_id or not page_num:
return "bad-fm"
padded = f"{page_num:03d}"
png = PNG_BASE / doc_id / f"p-{padded}.png"
if not png.exists():
return "no-png"
# Two attempts with progressively higher token budgets
revision = None
for tok in (32768, 65536):
try:
raw = call_gemini(client, png, model, max_tokens=tok)
revision = parse_json_lenient(raw)
break
except json.JSONDecodeError:
continue
except Exception as e:
sys.stderr.write(f" ✗ {page_md.relative_to(UFO_ROOT)}: {type(e).__name__}: {e}\n")
return "error"
if revision is None:
return "bad-json"
ee = revision.get("entities_extracted")
if not isinstance(ee, dict):
return "bad-shape"
# Quick stats for reporting
old_ee = fm.get("entities_extracted") or {}
old_n = sum(len(old_ee.get(k) or []) for k in old_ee)
new_n = sum(len(ee.get(k) or []) for k in ee)
fm["entities_extracted"] = ee
fm["last_entity_extraction_model"] = model
fm["last_entity_extraction_at"] = utc_now_iso()
write_fm(page_md, fm, body)
rel = str(page_md.relative_to(UFO_ROOT))
print(f" ✓ {rel}: entities {old_n} → {new_n}", flush=True)
return "ok"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id")
ap.add_argument("--page", help="<doc-id>/pNNN")
ap.add_argument("--all", action="store_true")
ap.add_argument("--pages-file")
ap.add_argument("--model", default=DEFAULT_MODEL)
ap.add_argument("--workers", type=int, default=20)
ap.add_argument("--max", type=int, default=0)
args = ap.parse_args()
api_key = os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY")
if not api_key:
sys.stderr.write("✗ GEMINI_API_KEY not set\n"); sys.exit(1)
client = genai.Client(api_key=api_key)
if args.page:
parts = args.page.split("/")
if len(parts) != 2:
ap.error("--page must be <doc-id>/pNNN")
targets = [PAGES / parts[0] / f"{parts[1]}.md"]
elif args.doc_id:
targets = sorted((PAGES / args.doc_id).glob("p*.md"))
elif args.pages_file:
targets = [Path(line.strip() if line.strip().startswith("/") else UFO_ROOT / line.strip())
for line in Path(args.pages_file).read_text().splitlines() if line.strip()]
elif args.all:
targets = sorted(PAGES.glob("*/p*.md"))
else:
ap.error("provide --doc-id, --page, --all, or --pages-file")
if args.max:
targets = targets[:args.max]
print(f"Re-extracting entities from {len(targets)} page(s) with {args.model} ({args.workers} workers)")
stats = {"ok": 0, "error": 0, "no-png": 0, "no-fm": 0, "bad-fm": 0, "bad-json": 0, "bad-shape": 0}
with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
futs = {pool.submit(process_page, client, p, args.model): p for p in targets if p.exists()}
for fut in concurrent.futures.as_completed(futs):
try:
r = fut.result()
stats[r] = stats.get(r, 0) + 1
except Exception as e:
sys.stderr.write(f"✗ {futs[fut]}: {e}\n")
stats["error"] += 1
print(f"\nDone. {stats}")
if __name__ == "__main__":
main()

80
scripts/22-update-stub-messages.py Executable file
View file

@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""
22-update-stub-messages.py — Phase 0: replace misleading "Will be enriched in
Phase 6" stubs with an honest low-signal message.
For each entity file whose body still has the stub phrasing:
- Read total_mentions, documents_count from frontmatter
- Rewrite body with calibrated message that reflects reality
- Preserve frontmatter as-is
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("pip3 install pyyaml\n"); sys.exit(1)
ENTITIES = Path("/Users/guto/ufo/wiki/entities")
STUB_RE = re.compile(
r"^# [^\n]+\n\n## Description \(EN\)\n\n_Stub generated by entity dedup\..*?_\n\n"
r"## Descrição \(PT-BR\)\n\n_Stub gerado pela deduplicação de entidades\..*?_\n*$",
re.DOTALL,
)
def new_body(canonical: str, total: int, docs: int) -> str:
return (
f"# {canonical}\n\n"
f"## Description (EN)\n\n"
f"_Low-signal entity — referenced **{total} time(s)** across **{docs} document(s)**. "
f"No external enrichment performed (criteria: ≥3 mentions). Use the page references below for raw context._\n\n"
f"## Descrição (PT-BR)\n\n"
f"_Entidade de baixo sinal — referenciada **{total} vez(es)** em **{docs} documento(s)**. "
f"Sem enriquecimento externo (critério: ≥3 menções). Use as referências de páginas abaixo para contexto bruto._\n"
)
def main():
updated = 0
skipped = 0
enriched = 0
for p in ENTITIES.glob("*/*.md"):
c = p.read_text(encoding="utf-8")
if not c.startswith("---"):
skipped += 1
continue
end = c.find("---", 4)
if end < 0:
skipped += 1; continue
fm = yaml.safe_load(c[3:end].strip()) or {}
body = c[end + 3:].lstrip("\n")
# Don't touch entities that have real enrichment content
if fm.get("enrichment_status") in ("deep", "shallow") and "external_sources" in body:
enriched += 1
continue
# Don't touch the seeded entities that had hand-curated bodies
if "Phase 6" not in body and "Phase 7" not in body:
skipped += 1
continue
canonical = fm.get("canonical_name") or p.stem
total = int(fm.get("total_mentions") or 0)
docs = int(fm.get("documents_count") or 0)
new = new_body(canonical, total, docs)
new_full = c[:end + 4] + "\n" + new
if new_full == c:
skipped += 1; continue
p.write_text(new_full, encoding="utf-8")
updated += 1
print(f"Updated: {updated}\nSkipped (no stub / hand-curated): {skipped}\nKept enriched: {enriched}")
if __name__ == "__main__":
main()

282
scripts/23-smart-dedup.py Executable file
View file

@ -0,0 +1,282 @@
#!/usr/bin/env python3
"""
23-smart-dedup.py — Phase 1 + Phase 3: aggressive entity cleanup.
Removes garbage entities that Haiku/extraction over-promoted:
A. Stop-list filter — single-mention bare common nouns
B. Substring/alias dedup — "FBI" vs "F-B-I" vs "Federal Bureau of Investigation"
C. Compound-name detection — entities A+B that co-occur ≥3 pages → suggest merge
D. Title-prefix recovery — "Chief Tereoken" appearing in raw page text but
dedup created only "tereoken" + "chief"
Runs in two modes:
--dry-run → report what would be deleted/merged, no writes
(default) → applies deletes and merges, removes orphans, updates affected page.md
files to substitute merged names
Skip --merge-compounds to disable (C) since it can be aggressive.
Usage:
./23-smart-dedup.py --dry-run
./23-smart-dedup.py --apply
"""
from __future__ import annotations
import argparse
import re
import sys
import unicodedata
from collections import Counter, defaultdict
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("pip3 install pyyaml\n"); sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
ENTITIES = UFO_ROOT / "wiki" / "entities"
PAGES = UFO_ROOT / "wiki" / "pages"
LOG = UFO_ROOT / "wiki" / "log.md"
# Common-noun stop list — these have no value as standalone entities.
# Drop only if mention_count == 1 (still keep if used many times — it's a real referent).
STOP_NOUNS = {
# Roles / positions
"agent", "agents", "officer", "officers", "chief", "captain", "general", "major",
"colonel", "sergeant", "commander", "director", "secretary", "lieutenant", "lcdr",
"cdr", "lt", "lt.", "cpl", "sgt", "supervisor", "inspector",
# Generic structures
"base", "office", "department", "bureau", "agency", "division", "section",
"headquarters", "command", "post", "station", "branch", "unit", "group",
"the", "the bureau", "the agency", "the department", "the office", "the file",
# File / document terms
"file", "files", "memo", "memorandum", "letter", "report", "form", "page", "pages",
"subject", "date", "time", "case", "exhibit", "attachment", "enclosure",
"signature", "stamp", "carbon copy", "cc", "ref", "reference", "annex",
"envelope", "bag", "folder", "transmittal", "routing", "dispatch",
# Generic descriptors
"the inspector", "the agent", "the officer", "the witness", "the observer",
"the subject", "the man", "the woman", "the pilot", "the operator",
# Things that often slip into entities by mistake
"technicians", "personnel", "staff", "team", "crew", "members",
"operations", "operation", # only when very generic — collisions handled by mention_count
"departments",
}
# Common single-letter or 2-letter "entities" that are useless on their own.
TRIVIAL_PATTERNS = [
re.compile(r"^[a-z0-9]$"),
re.compile(r"^[a-z]{1,2}$"), # tiny initials
]
def normalize(s: str) -> str:
nfd = unicodedata.normalize("NFD", s)
return "".join(c for c in nfd if not unicodedata.combining(c)).lower().strip()
def read_md(path: Path) -> tuple[dict, str]:
try:
c = path.read_text(encoding="utf-8")
except FileNotFoundError:
return {}, ""
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end < 0:
return {}, c
try:
fm = yaml.safe_load(c[3:end].strip()) or {}
except yaml.YAMLError:
fm = {}
body = c[end + 3:].lstrip("\n")
return fm, body
def is_trivial(canonical_name: str, entity_id: str) -> bool:
n = normalize(canonical_name)
if not n:
return True
if n in STOP_NOUNS:
return True
if any(p.match(n) for p in TRIVIAL_PATTERNS):
return True
# All-stop-words sequence: "the bureau", "the office"
words = n.split()
if len(words) >= 2 and all(w in STOP_NOUNS or w == "the" for w in words):
return True
# Single common noun
if len(words) == 1 and n in STOP_NOUNS:
return True
return False
def filter_low_signal(*, dry: bool) -> dict[str, list[Path]]:
"""A — Delete entities that are trivial AND have low mention_count."""
stats: dict[str, list[Path]] = {"deleted": [], "kept_high_mention": [], "kept": []}
for p in ENTITIES.glob("*/*.md"):
fm, _ = read_md(p)
if not fm:
continue
canonical = (fm.get("canonical_name") or p.stem)
total = int(fm.get("total_mentions") or 0)
if is_trivial(canonical, p.stem):
if total <= 2: # trivial + rarely mentioned = noise
stats["deleted"].append(p)
if not dry:
p.unlink()
else:
stats["kept_high_mention"].append(p)
else:
stats["kept"].append(p)
return stats
def aliases_of(fm: dict) -> set[str]:
out = set()
cname = fm.get("canonical_name")
if isinstance(cname, str):
out.add(normalize(cname))
for a in (fm.get("aliases") or []):
if isinstance(a, str):
out.add(normalize(a))
return {x for x in out if x}
def dedupe_by_alias(*, dry: bool) -> dict[str, int]:
"""B — Merge entities whose alias sets overlap.
Strategy: keep the entity with highest total_mentions; redirect others by
appending an alias and deleting their files.
"""
stats = {"merges": 0, "deletes": 0}
by_class: dict[str, dict[str, Path]] = defaultdict(dict)
# Build class → alias → path map (last-write-wins; collisions become merge targets)
overlap: dict[str, dict[str, list[Path]]] = defaultdict(lambda: defaultdict(list))
for p in ENTITIES.glob("*/*.md"):
cls = p.parent.name
fm, _ = read_md(p)
if not fm:
continue
for a in aliases_of(fm):
overlap[cls][a].append(p)
for cls, alias_map in overlap.items():
for alias, paths in alias_map.items():
if len(paths) < 2:
continue
# Pick canonical winner: highest total_mentions
ranked = []
for pp in paths:
if not pp.exists():
continue
fm, _ = read_md(pp)
if not fm:
continue
ranked.append((int(fm.get("total_mentions") or 0), pp, fm))
if len(ranked) < 2:
continue
ranked.sort(key=lambda x: x[0], reverse=True)
winner_count, winner_path, winner_fm = ranked[0]
for _count, loser_path, loser_fm in ranked[1:]:
if loser_path == winner_path or not loser_path.exists():
continue
# Add loser's aliases to winner
new_aliases = sorted(set((winner_fm.get("aliases") or []))
| set(loser_fm.get("aliases") or [])
| {loser_fm.get("canonical_name") or loser_path.stem})
winner_fm["aliases"] = [a for a in new_aliases if a]
# Total mentions sum
winner_fm["total_mentions"] = (winner_fm.get("total_mentions") or 0) + (loser_fm.get("total_mentions") or 0)
if not dry:
new_yaml = yaml.dump(winner_fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
body = winner_path.read_text(encoding="utf-8").split("---", 2)[-1].lstrip("\n")
winner_path.write_text(f"---\n{new_yaml}---\n\n{body}", encoding="utf-8")
loser_path.unlink()
stats["merges"] += 1
stats["deletes"] += 1
return stats
def find_compound_candidates() -> list[tuple[str, str, str, int]]:
"""C — find entity pairs (A, B) appearing adjacent on ≥3 pages → likely compound name.
Walk all page.md `entities_extracted` fields; if A.name then B.name appear
near each other in the page body, count co-occurrence.
Returns: [(class_a, name_a, name_b, count)]
"""
pair_count: Counter = Counter()
page_entities: Counter = Counter()
for p in PAGES.glob("*/p*.md"):
try:
fm, _ = read_md(p)
except Exception:
continue
if not fm:
continue
ee = fm.get("entities_extracted") or {}
if not isinstance(ee, dict):
continue
# Across all entity classes, look for adjacent pairs in name lists
names = []
for cls_key in ("people", "organizations", "locations"):
for entry in (ee.get(cls_key) or []):
if isinstance(entry, dict) and entry.get("name"):
names.append((cls_key, normalize(entry["name"])))
# Pair up adjacent entries (heuristic — Haiku usually returns them in occurrence order)
for i in range(len(names) - 1):
a_cls, a = names[i]
b_cls, b = names[i + 1]
if a == b:
continue
pair_count[(a_cls, a, b)] += 1
# Filter to pairs that appear together ≥ 3 pages
out = []
for (cls_a, a, b), c in pair_count.items():
if c >= 3:
out.append((cls_a, a, b, c))
return sorted(out, key=lambda x: -x[3])
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--dry-run", action="store_true", help="report only, don't delete")
ap.add_argument("--apply", action="store_true", help="apply deletes + merges")
ap.add_argument("--report-compounds", action="store_true", help="just print compound candidates and exit")
args = ap.parse_args()
if not args.dry_run and not args.apply and not args.report_compounds:
ap.error("provide --dry-run, --apply, or --report-compounds")
if args.report_compounds:
cands = find_compound_candidates()
print(f"Top compound candidates (adjacent ≥3 pages):")
for cls_a, a, b, c in cands[:50]:
print(f" {c:4d}× [{cls_a}] {a} + {b} → '{a} {b}'")
print(f"\nTotal: {len(cands)} candidates")
return
dry = args.dry_run
print(f"=== Phase 1A: filter trivial low-mention entities ({'DRY-RUN' if dry else 'APPLY'}) ===")
a = filter_low_signal(dry=dry)
print(f" deleted (trivial + ≤2 mentions): {len(a['deleted'])}")
print(f" kept (trivial but ≥3 mentions): {len(a['kept_high_mention'])}")
print(f" kept (meaningful): {len(a['kept'])}")
print(f"\n=== Phase 1B: alias-based merge ({'DRY-RUN' if dry else 'APPLY'}) ===")
b = dedupe_by_alias(dry=dry)
print(f" pairs merged: {b['merges']}")
total_remaining = sum(1 for _ in ENTITIES.glob("*/*.md"))
print(f"\nRemaining entity files: {total_remaining}")
if __name__ == "__main__":
main()

393
scripts/24-document-synthesis.py Executable file
View file

@ -0,0 +1,393 @@
#!/usr/bin/env python3
"""
24-document-synthesis.py — Cross-page document synthesis via Claude Sonnet 4.6.
Until now `wiki/documents/<doc-id>.md` was a DUMB union of per-page frontmatter
(page count, classification stats, entity union). Useless as a narrative.
This script READS THE WHOLE DOCUMENT (all OCR pages + vision descriptions +
entity refs) and asks Sonnet 4.6 (via Claude Code OAuth, $0 on Max) to produce:
- executive_summary_en (2-4 paragraphs, what the doc IS and what it claims)
- executive_summary_pt_br (Brazilian Portuguese version)
- narrative_arc_en (story across pages)
- narrative_arc_pt_br
- central_characters[] (top 3-7 people, their role + arc)
- key_events_timeline[] (date + label + page refs)
- key_locations[] (with significance)
- strategic_significance (why this doc matters to the corpus)
- confidence_band
Output replaces the entire body of wiki/documents/<doc-id>.md. Frontmatter is
preserved + augmented with these synthesis fields.
Usage:
./24-document-synthesis.py --doc-id <id> # one doc smoke test
./24-document-synthesis.py --all # all 116 docs
./24-document-synthesis.py --max 5 # cap for testing
./24-document-synthesis.py --skip-existing # don't redo docs that already have synthesis
"""
from __future__ import annotations
import argparse
import concurrent.futures
import json
import re
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("pip3 install pyyaml\n"); sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
DOCS = UFO_ROOT / "wiki" / "documents"
PAGES_DIR = UFO_ROOT / "wiki" / "pages"
OCR_BASE = UFO_ROOT / "processing" / "ocr"
LOG_PATH = UFO_ROOT / "wiki" / "log.md"
MODEL = "sonnet" # claude-sonnet-4-6 via Claude Code OAuth
MAX_TURNS = 3
TIMEOUT_S = 240
MAX_INPUT_CHARS = 320_000 # cap input size; Sonnet 200K tokens ≈ 800K chars safe
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def read_fm(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
if end < 0:
return {}, c
try:
fm = yaml.safe_load(c[3:end].strip()) or {}
except yaml.YAMLError:
fm = {}
return fm, c[end + 3:].lstrip("\n")
def write_fm(path: Path, fm: dict, body: str) -> None:
new_yaml = yaml.dump(fm, allow_unicode=True, sort_keys=False, default_flow_style=False)
sep = "\n" if body.startswith("\n") else "\n\n"
path.write_text(f"---\n{new_yaml}---{sep}{body}", encoding="utf-8")
def assemble_doc_payload(doc_id: str) -> tuple[str, dict] | None:
"""Read all page OCRs + vision descriptions + entity refs and concatenate
into a single payload string for Sonnet. Returns (payload, meta)."""
doc_md = DOCS / f"{doc_id}.md"
if not doc_md.exists():
return None
doc_fm, _ = read_fm(doc_md)
pages_dir = PAGES_DIR / doc_id
page_files = sorted(pages_dir.glob("p*.md"))
if not page_files:
return None
lines: list[str] = []
lines.append(f"# DOCUMENT: {doc_id}")
lines.append(f"# Canonical title: {doc_fm.get('canonical_title', doc_id)}")
lines.append(f"# Collection: {doc_fm.get('collection', '?')}")
lines.append(f"# Total pages: {len(page_files)}")
lines.append("")
for pf in page_files:
page_fm, _ = read_fm(pf)
page_num = page_fm.get("page_number", "?")
lines.append(f"\n===== PAGE {page_num:>3} ({pf.stem}) =====")
if page_fm.get("page_type"):
lines.append(f" page_type: {page_fm['page_type']}")
if page_fm.get("vision_description"):
lines.append(f" vision (EN): {page_fm['vision_description']}")
if isinstance(page_fm.get("entities_extracted"), dict):
ee = page_fm["entities_extracted"]
for cls in ("people", "organizations", "locations", "events", "uap_objects"):
names = [(e.get("name") or e.get("label")) for e in (ee.get(cls) or []) if isinstance(e, dict)]
names = [n for n in names if n]
if names:
lines.append(f" {cls}: {', '.join(names[:15])}")
# OCR (truncate per-page for very large docs)
padded = f"{int(page_num):03d}" if isinstance(page_num, int) or (isinstance(page_num, str) and page_num.isdigit()) else "001"
ocr_path = OCR_BASE / doc_id / f"p-{padded}.txt"
if ocr_path.exists():
ocr = ocr_path.read_text(encoding="utf-8", errors="replace")
if len(ocr) > 4000:
ocr = ocr[:4000] + f"\n[…page truncated, {len(ocr)} chars total]"
lines.append(" OCR:")
for line in ocr.split("\n")[:120]:
lines.append(f" {line}")
payload = "\n".join(lines)
if len(payload) > MAX_INPUT_CHARS:
payload = payload[:MAX_INPUT_CHARS] + f"\n\n[…truncated to {MAX_INPUT_CHARS} chars; doc has more]"
meta = {
"doc_id": doc_id,
"page_count": len(page_files),
"input_chars": len(payload),
}
return payload, meta
PROMPT_TEMPLATE = """You are Sherlock, lead investigator of The Disclosure Bureau. Your task: produce a coherent SYNTHESIS of an entire US Department of War declassified UAP/UFO document by reading ALL its pages together.
The dump below contains, for each page: vision description, extracted entities, and OCR text. Treat this as ONE coherent document, not a list of pages. Build a narrative arc, identify central characters, key events, themes.
DOCUMENT DUMP:
========================================
{payload}
========================================
Output ONE JSON object (no markdown fence, no preamble). Schema:
{{
"executive_summary_en": "2-4 paragraphs in English. What this document IS, what it claims, who produced it, what it covers, why it exists. CITE specific page numbers like (p005, p023). Use Tetlock confidence bands sparingly: 'high', 'medium', 'low', 'speculation'.",
"executive_summary_pt_br": "Mesmo conteúdo em português brasileiro (pt-br, NÃO European). Preserve UTF-8 accents (ç, ã, é etc.). Mantenha citações verbatim do documento (em inglês) sem traduzir.",
"narrative_arc_en": "3-6 paragraphs telling the story the document tells, as it unfolds across pages. Reference page numbers as (p007). Stay grounded — only what's actually in the document.",
"narrative_arc_pt_br": "Mesmo em pt-br.",
"central_characters": [
{{"name": "Canonical full name", "role": "what they do in this doc", "arc": "how they appear across pages", "pages": "p001, p007-p012"}}
],
"key_events_timeline": [
{{"date": "YYYY-MM-DD or YYYY or 'undated'", "event": "short factual description", "pages": "p005"}}
],
"key_locations": [
{{"name": "Location name", "significance": "why it matters in this doc", "pages": "p007"}}
],
"uap_objects_described": [
{{"shape": "...", "color": "...", "behavior": "key observed behaviors", "page": "p007"}}
],
"strategic_significance": "1 paragraph English. Why this document matters for understanding the wider UAP archive — what unique evidence or claim it brings, what it confirms or contradicts of other reports. Use confidence_band.",
"strategic_significance_pt_br": "Mesmo em pt-br.",
"synthesis_confidence_band": "high | medium | low | speculation",
"synthesis_caveats": "any notes about gaps, low OCR quality, redaction extent, contradictions you noticed"
}}
Rules:
- All entity names and verbatim quotes stay in their ORIGINAL language.
- Brazilian Portuguese (NOT European). Preserve UTF-8.
- Cite page numbers like (p007) so readers can verify.
- If document is genuinely thin (e.g., just a memo without much content), say so explicitly in executive_summary.
- Output ONLY the JSON. No fence, no preamble."""
def call_sonnet(payload: str) -> dict:
prompt = PROMPT_TEMPLATE.format(payload=payload)
def _call():
return subprocess.run(
["claude", "-p", "--model", MODEL,
"--output-format", "json", "--max-turns", str(MAX_TURNS),
"--", prompt],
capture_output=True, text=True, timeout=TIMEOUT_S + 30, check=False,
)
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:
future = ex.submit(_call)
try:
res = future.result(timeout=TIMEOUT_S)
except concurrent.futures.TimeoutError:
raise RuntimeError(f"sonnet hung >{TIMEOUT_S}s")
if res.returncode != 0:
raise RuntimeError(f"claude rc={res.returncode}: {res.stderr[-500:]}")
cli = json.loads(res.stdout)
if cli.get("is_error"):
raise RuntimeError(f"claude error: {cli.get('result', '')[:300]}")
text = (cli.get("result") or "").strip()
text = re.sub(r"^```(?:json)?\s*", "", text)
text = re.sub(r"\s*```$", "", text)
# Robust JSON parse with brace balancing fallback
try:
return json.loads(text)
except json.JSONDecodeError:
start = text.find("{")
if start < 0:
raise
depth = 0
for i in range(start, len(text)):
if text[i] == "{": depth += 1
elif text[i] == "}":
depth -= 1
if depth == 0:
return json.loads(text[start:i + 1])
raise
def render_body(synthesis: dict) -> str:
"""Build the markdown body from the JSON synthesis."""
lines = []
lines.append(f"# {synthesis.get('executive_summary_en', 'Document Synthesis').split('.', 1)[0][:80]}")
lines.append("")
band = synthesis.get("synthesis_confidence_band", "")
if band:
lines.append(f"> **Synthesis confidence: `{band}`**")
lines.append("")
if synthesis.get("synthesis_caveats"):
lines.append(f"> ⚠ Caveats: {synthesis['synthesis_caveats']}")
lines.append("")
lines.append("## Executive Summary (EN)")
lines.append("")
lines.append(synthesis.get("executive_summary_en", "_no summary_"))
lines.append("")
lines.append("## Sumário Executivo (PT-BR)")
lines.append("")
lines.append(synthesis.get("executive_summary_pt_br", "_sem sumário_"))
lines.append("")
lines.append("## Narrative Arc (EN)")
lines.append("")
lines.append(synthesis.get("narrative_arc_en", "_no narrative_"))
lines.append("")
lines.append("## Arco Narrativo (PT-BR)")
lines.append("")
lines.append(synthesis.get("narrative_arc_pt_br", "_sem narrativa_"))
lines.append("")
chars = synthesis.get("central_characters") or []
if chars:
lines.append("## Central Characters")
lines.append("")
for c in chars:
lines.append(f"### {c.get('name', '?')}")
if c.get("role"): lines.append(f"- **Role**: {c['role']}")
if c.get("arc"): lines.append(f"- **Arc**: {c['arc']}")
if c.get("pages"): lines.append(f"- **Pages**: `{c['pages']}`")
lines.append("")
events = synthesis.get("key_events_timeline") or []
if events:
lines.append("## Key Events Timeline")
lines.append("")
lines.append("| Date | Event | Pages |")
lines.append("|---|---|---|")
for e in events:
lines.append(f"| {e.get('date', '?')} | {e.get('event', '?')} | `{e.get('pages', '')}` |")
lines.append("")
locs = synthesis.get("key_locations") or []
if locs:
lines.append("## Key Locations")
lines.append("")
for l in locs:
lines.append(f"- **{l.get('name', '?')}** ({l.get('pages', '')}): {l.get('significance', '')}")
lines.append("")
uaps = synthesis.get("uap_objects_described") or []
if uaps:
lines.append("## UAP Objects Described")
lines.append("")
for u in uaps:
lines.append(f"- **{u.get('shape', '?')} ({u.get('color', '?')})** on `{u.get('page', '')}`: {u.get('behavior', '')}")
lines.append("")
lines.append("## Strategic Significance")
lines.append("")
lines.append(synthesis.get("strategic_significance", "_no assessment_"))
lines.append("")
lines.append("## Significância Estratégica (PT-BR)")
lines.append("")
lines.append(synthesis.get("strategic_significance_pt_br", "_sem avaliação_"))
lines.append("")
return "\n".join(lines)
def process_doc(doc_id: str, *, skip_existing: bool) -> str:
doc_path = DOCS / f"{doc_id}.md"
if not doc_path.exists():
return "missing"
fm, body = read_fm(doc_path)
if skip_existing and fm.get("synthesis_model"):
return "skip-existing"
print(f" → {doc_id} ({fm.get('page_count', '?')} pages)", flush=True)
assembled = assemble_doc_payload(doc_id)
if not assembled:
return "no-payload"
payload, meta = assembled
print(f" input: {meta['input_chars']} chars from {meta['page_count']} pages", flush=True)
t0 = time.time()
try:
synthesis = call_sonnet(payload)
except Exception as e:
print(f" ✗ Sonnet failed: {type(e).__name__}: {e}", flush=True)
return "error"
dt = time.time() - t0
new_body = render_body(synthesis)
fm["synthesis_model"] = "claude-sonnet-4-6"
fm["synthesis_at"] = utc_now_iso()
fm["synthesis_confidence_band"] = synthesis.get("synthesis_confidence_band")
fm["central_characters_count"] = len(synthesis.get("central_characters") or [])
fm["key_events_count"] = len(synthesis.get("key_events_timeline") or [])
write_fm(doc_path, fm, new_body)
print(f" ✓ wrote synthesis ({dt:.1f}s)", flush=True)
return "ok"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id")
ap.add_argument("--all", action="store_true")
ap.add_argument("--max", type=int, default=0)
ap.add_argument("--skip-existing", action="store_true")
ap.add_argument("--workers", type=int, default=1, help="parallel workers (default 1; Max 20x rate-limits Sonnet)")
args = ap.parse_args()
if args.doc_id:
targets = [args.doc_id]
elif args.all:
targets = sorted(p.stem for p in DOCS.glob("*.md"))
else:
ap.error("provide --doc-id or --all")
if args.max:
targets = targets[:args.max]
print(f"Synthesizing {len(targets)} document(s) with claude-sonnet-4-6")
stats = {"ok": 0, "error": 0, "skip-existing": 0, "no-payload": 0, "missing": 0}
if args.workers <= 1:
for d in targets:
r = process_doc(d, skip_existing=args.skip_existing)
stats[r] = stats.get(r, 0) + 1
else:
with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
futs = {pool.submit(process_doc, d, skip_existing=args.skip_existing): d for d in targets}
for fut in concurrent.futures.as_completed(futs):
try:
r = fut.result()
stats[r] = stats.get(r, 0) + 1
except Exception as e:
sys.stderr.write(f"✗ {futs[fut]}: {e}\n")
stats["error"] += 1
print(f"\nDone. {stats}")
if stats.get("ok"):
with open(LOG_PATH, "a", encoding="utf-8") as fh:
fh.write(
f"\n## {utc_now_iso()} — DOCUMENT SYNTHESIS (Sonnet 4.6)\n"
f"- script: scripts/24-document-synthesis.py\n"
f"- docs_synthesized: {stats['ok']}\n"
f"- errors: {stats.get('error', 0)}\n"
)
if __name__ == "__main__":
main()

192
scripts/25-master-doc-test.py Executable file
View file

@ -0,0 +1,192 @@
#!/usr/bin/env python3
"""
25-master-doc-test.py — A/B test: rebuild a document using either subagents or
agent teams, measure cost + time + output quality.
Both approaches must produce raw/<doc-id>/document.md with the same schema.
Usage:
./25-master-doc-test.py --doc-id <id> --approach subagent --max-pages 20
./25-master-doc-test.py --doc-id <id> --approach team --max-pages 20
./25-master-doc-test.py --doc-id <id> --both --max-pages 20 # runs both
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
UFO_ROOT = Path("/Users/guto/ufo")
def utc_now_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
PROMPT_SUBAGENT = """Rebuild the document `{doc_id}` into `raw/{doc_id}/document.md` using the `doc-rebuilder` subagent.
Constraints:
- Process ONLY the first {max_pages} pages (p001 .. p{max_pages:03d}).
- The doc-rebuilder agent will spawn `page-rebuilder`, `image-analyst`, and `table-stitcher` subagents in parallel via the Task tool.
- Output schema: as defined in the doc-rebuilder agent's system prompt.
- Target file: `/Users/guto/ufo/raw/{doc_id}/document.md`.
- Image crops go to `/Users/guto/ufo/raw/{doc_id}/images/`.
- Set frontmatter `build_approach: "subagents"`.
Begin. When done, output a single line with stats: pages_done, chunks_total, images_extracted, tables_stitched, ufo_anomalies, cryptid_anomalies, wall_seconds."""
PROMPT_TEAM = """Create an agent team to rebuild the document `{doc_id}` into `raw/{doc_id}/document.md`.
Constraints:
- Process ONLY the first {max_pages} pages (p001 .. p{max_pages:03d}).
- Spawn an agent team with these teammates (use the subagent definitions from .claude/agents/):
- 1× lead (you) coordinating
- 4× page-rebuilder teammates working different page subsets in parallel
- 1× image-analyst teammate processing all image chunks after page-rebuilders finish
- 1× table-stitcher teammate for multi-page tables
- Use the shared task list to coordinate work.
- Output schema: as defined in the doc-rebuilder agent's system prompt.
- Target file: `/Users/guto/ufo/raw/{doc_id}/document.md`.
- Set frontmatter `build_approach: "agent-teams"`.
When all teammates complete, clean up the team and output a single line of stats: pages_done, chunks_total, images_extracted, tables_stitched, ufo_anomalies, cryptid_anomalies, wall_seconds."""
def run_approach(approach: str, doc_id: str, max_pages: int) -> dict:
"""Invoke claude CLI for one approach. Returns metrics."""
out_dir = UFO_ROOT / "raw" / doc_id
if out_dir.exists():
# Move existing aside so we don't clobber
backup = UFO_ROOT / "raw" / f"{doc_id}.backup-{int(time.time())}"
shutil.move(str(out_dir), str(backup))
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / "images").mkdir(exist_ok=True)
(out_dir / "tables").mkdir(exist_ok=True)
if approach == "subagent":
prompt = PROMPT_SUBAGENT.format(doc_id=doc_id, max_pages=max_pages)
env = {**os.environ}
else:
prompt = PROMPT_TEAM.format(doc_id=doc_id, max_pages=max_pages)
env = {**os.environ, "CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS": "1"}
cmd = [
"claude", "-p",
"--model", "sonnet",
"--output-format", "json",
"--max-turns", "60",
"--allowedTools", "Read,Write,Bash,Task",
"--add-dir", str(UFO_ROOT),
"--",
prompt,
]
if approach == "team":
# Inject teammate-mode flag; some experimental features need it
cmd.insert(-2, "--teammate-mode")
cmd.insert(-2, "in-process")
print(f"\n{'=' * 70}")
print(f" APPROACH: {approach.upper()}")
print(f"{'=' * 70}")
print(f" cmd: {' '.join(cmd[:8])} … (prompt truncated)")
print(f" starting at {utc_now_iso()}")
sys.stdout.flush()
t0 = time.time()
proc = subprocess.run(cmd, capture_output=True, text=True, env=env, check=False, timeout=3600)
wall = time.time() - t0
metrics = {
"approach": approach,
"wall_seconds": round(wall, 1),
"returncode": proc.returncode,
"stderr_tail": proc.stderr[-1000:] if proc.stderr else "",
}
try:
cli = json.loads(proc.stdout) if proc.stdout else {}
except json.JSONDecodeError:
cli = {"raw_stdout": proc.stdout[-3000:]}
metrics["is_error"] = cli.get("is_error", proc.returncode != 0)
metrics["duration_ms"] = cli.get("duration_ms")
metrics["duration_api_ms"] = cli.get("duration_api_ms")
metrics["total_cost_usd"] = cli.get("total_cost_usd")
metrics["num_turns"] = cli.get("num_turns")
metrics["usage"] = cli.get("usage")
metrics["result_excerpt"] = (cli.get("result") or "")[:2000]
# Inspect output
doc_md = out_dir / "document.md"
metrics["output_exists"] = doc_md.exists()
metrics["output_size_bytes"] = doc_md.stat().st_size if doc_md.exists() else 0
metrics["images_extracted"] = len(list((out_dir / "images").glob("*"))) if (out_dir / "images").exists() else 0
# Rename output so both approaches can coexist
if doc_md.exists():
archive = UFO_ROOT / "raw" / f"{doc_id}--{approach}"
if archive.exists():
shutil.rmtree(archive)
shutil.move(str(out_dir), str(archive))
metrics["archived_at"] = str(archive)
print(f" finished in {wall:.1f}s · rc={proc.returncode}")
print(f" output exists: {metrics['output_exists']} · size: {metrics['output_size_bytes']} bytes")
print(f" images_extracted: {metrics['images_extracted']}")
return metrics
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id", required=True)
ap.add_argument("--max-pages", type=int, default=20)
ap.add_argument("--approach", choices=["subagent", "team"])
ap.add_argument("--both", action="store_true")
args = ap.parse_args()
if not args.approach and not args.both:
ap.error("provide --approach or --both")
results: dict[str, dict] = {}
if args.both or args.approach == "subagent":
results["subagent"] = run_approach("subagent", args.doc_id, args.max_pages)
if args.both or args.approach == "team":
results["team"] = run_approach("team", args.doc_id, args.max_pages)
# Comparison table
print(f"\n{'=' * 70}")
print(f" COMPARISON — {args.doc_id} first {args.max_pages} pages")
print(f"{'=' * 70}")
if "subagent" in results and "team" in results:
s = results["subagent"]; t = results["team"]
print(f" {'metric':<25} {'subagent':>20} {'team':>20}")
print(f" {'-' * 25} {'-' * 20:>20} {'-' * 20:>20}")
print(f" {'wall_seconds':<25} {s['wall_seconds']:>20} {t['wall_seconds']:>20}")
print(f" {'returncode':<25} {s['returncode']:>20} {t['returncode']:>20}")
print(f" {'is_error':<25} {str(s.get('is_error')):>20} {str(t.get('is_error')):>20}")
print(f" {'total_cost_usd':<25} {str(s.get('total_cost_usd')):>20} {str(t.get('total_cost_usd')):>20}")
print(f" {'num_turns':<25} {str(s.get('num_turns')):>20} {str(t.get('num_turns')):>20}")
print(f" {'output_size_bytes':<25} {s['output_size_bytes']:>20} {t['output_size_bytes']:>20}")
print(f" {'images_extracted':<25} {s['images_extracted']:>20} {t['images_extracted']:>20}")
else:
for k, v in results.items():
print(json.dumps({k: v}, indent=2, default=str))
# Save full result JSON
report_dir = UFO_ROOT / "raw" / "_ab-test-reports"
report_dir.mkdir(parents=True, exist_ok=True)
report = report_dir / f"{args.doc_id}--{int(time.time())}.json"
report.write_text(json.dumps(results, indent=2, default=str))
print(f"\nFull report: {report}")
if __name__ == "__main__":
main()

369
scripts/26-chunk-harness.py Executable file
View file

@ -0,0 +1,369 @@
#!/usr/bin/env python3
"""
26-chunk-harness.py — Deterministic harness that assembles document.md
from raw/<doc-id>/chunks/*.md + _index.json.
Use to:
- Verify chunks are losslessly assemblable
- Re-render document.md after manual chunk edits
- Generate alternate views (HTML, PDF, single-language)
Usage:
./26-chunk-harness.py --doc-id <id> # rebuild document.md
./26-chunk-harness.py --doc-id <id> --validate # just check structure
./26-chunk-harness.py --doc-id <id> --lang pt-br # render only PT-BR
./26-chunk-harness.py --doc-id <id> --format html # render to HTML
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
sys.stderr.write("pip3 install pyyaml\n"); sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
CANONICAL_TYPES = {
"letterhead", "address_block", "classification_marking", "heading",
"paragraph", "form_field", "bulleted_item", "numbered_item", "quote_block",
"caption", "table_marker", "image", "stamp", "signature", "marginalia",
"redaction", "footer", "blank_area", "unknown",
}
TYPE_NORMALIZER = {
"body_paragraph": "paragraph",
"narrative": "paragraph",
"prose": "paragraph",
"body_text": "paragraph",
"classification_banner": "classification_marking",
"security_banner": "classification_marking",
"classification_label": "classification_marking",
"header_block": "heading",
"section_header": "heading",
"subject_line": "heading",
"doc_title": "heading",
"agenda_heading": "heading",
"addressee_block": "address_block",
"distribution_list": "address_block",
"routing_block": "address_block",
"to_block": "address_block",
"from_block": "address_block",
"signature_block": "signature",
"sig": "signature",
"form_reference": "form_field",
"field": "form_field",
"label_value": "form_field",
"kv_field": "form_field",
}
def canonicalize_type(t: str) -> str:
if t in CANONICAL_TYPES:
return t
return TYPE_NORMALIZER.get(t, t)
def _shallow_yaml_extract(text: str) -> dict:
"""Best-effort key:value extraction when full yaml parse fails (broken quotes etc).
Only handles top-level scalar fields — drops broken arrays / objects.
Enough for the harness to render bodies + render basic metadata.
"""
out: dict = {}
for line in text.splitlines():
# only treat lines that look like `key: value` (no indentation)
if not line or line[0] in (" ", "\t", "-"):
continue
m = re.match(r"^([a-zA-Z_][a-zA-Z0-9_]*)\s*:\s*(.*)$", line)
if not m:
continue
k, v = m.group(1), m.group(2).strip()
if v.startswith("{") or v.startswith("["):
# complex — skip rather than parse partial
continue
if v == "null" or v == "":
out[k] = None
elif v.lower() == "true":
out[k] = True
elif v.lower() == "false":
out[k] = False
elif re.match(r"^-?\d+\.\d+$", v):
out[k] = float(v)
elif re.match(r"^-?\d+$", v):
out[k] = int(v)
elif (v[0] == v[-1]) and v[0] in ('"', "'"):
out[k] = v[1:-1]
else:
out[k] = v
return out
def read_chunk(path: Path) -> tuple[dict, str]:
c = path.read_text(encoding="utf-8")
if not c.startswith("---"):
return {}, c
end = c.find("---", 4)
fm_text = c[3:end].strip()
body = c[end + 3:].lstrip("\n")
try:
fm = yaml.safe_load(fm_text) or {}
except yaml.YAMLError:
# Malformed frontmatter (quoted strings, unclosed brackets) — degrade gracefully
fm = _shallow_yaml_extract(fm_text)
fm["_yaml_error"] = True
return fm, body
def validate(doc_dir: Path) -> list[str]:
"""Return list of errors (empty if valid)."""
errors: list[str] = []
index_path = doc_dir / "_index.json"
if not index_path.exists():
errors.append("missing _index.json")
return errors
try:
index = json.loads(index_path.read_text())
except json.JSONDecodeError as e:
errors.append(f"_index.json malformed: {e}")
return errors
chunks_dir = doc_dir / "chunks"
expected_ids = set()
for entry in index.get("chunks", []):
cid = entry.get("chunk_id")
if not cid:
errors.append(f"index entry missing chunk_id: {entry}")
continue
expected_ids.add(cid)
chunk_path = chunks_dir / f"{cid}.md"
if not chunk_path.exists():
errors.append(f"chunk file missing: {chunk_path}")
continue
try:
fm, body = read_chunk(chunk_path)
except Exception as e:
errors.append(f"chunk {cid} unreadable: {e}")
continue
if fm.get("_yaml_error"):
errors.append(f"chunk {cid}: YAML frontmatter malformed (shallow-parsed; body OK)")
if not fm.get("type"):
errors.append(f"chunk {cid}: missing type")
if not body.strip():
errors.append(f"chunk {cid}: empty body")
related_image = fm.get("related_image")
if related_image:
img_path = doc_dir / "images" / related_image
if not img_path.exists():
errors.append(f"chunk {cid}: related_image missing on disk: {related_image}")
# Check chunk files that aren't in the index (orphans)
if chunks_dir.exists():
for chunk_file in chunks_dir.glob("c*.md"):
cid = chunk_file.stem
if cid not in expected_ids:
errors.append(f"orphan chunk file (not in index): {cid}")
return errors
TEXTUAL_TYPES = {
# Canonical
"letterhead", "address_block", "classification_marking", "heading",
"paragraph", "form_field", "bulleted_item", "numbered_item", "quote_block",
"caption", "footer",
# Variations the agent invented (kept as-is)
"body_paragraph", "header_block", "header", "section_header",
"subject_line", "addressee_block", "form_reference", "distribution_list",
"transcript_block", "to_from_line", "date_line", "list_item",
"page_number", "title_block", "narrative_paragraph", "signature_block",
"handwriting", "marginalia_note",
}
def assemble_prose(doc_dir: Path, lang: str) -> str:
"""Clean reading view: just the textual content in the chosen language, page by page."""
index = json.loads((doc_dir / "_index.json").read_text())
chunks_meta = index.get("chunks", [])
by_page: dict[int, list[dict]] = {}
for c in chunks_meta:
by_page.setdefault(c.get("page", 0), []).append(c)
for page_chunks in by_page.values():
page_chunks.sort(key=lambda x: x.get("order_in_page", 0))
chunks_dir = doc_dir / "chunks"
out: list[str] = []
out.append(f"# {index.get('doc_id')}")
out.append("")
out.append(f"> {index.get('total_pages')} páginas · {len(chunks_meta)} chunks · idioma: {lang}")
out.append("")
marker = "**EN:**" if lang == "en" else "**PT-BR:**"
for page_num in sorted(by_page.keys()):
out.append(f"## Página {page_num}" if lang == "pt-br" else f"## Page {page_num}")
out.append("")
for c in by_page[page_num]:
canonical = canonicalize_type(c.get("type", ""))
if canonical not in TEXTUAL_TYPES:
continue
fm, body = read_chunk(chunks_dir / f"{c['chunk_id']}.md")
text = ""
for line in body.split("\n"):
s = line.strip()
if s.startswith(marker):
text = s.removeprefix(marker).strip()
break
if not text:
continue
if canonical == "heading":
out.append(f"### {text}")
elif canonical == "classification_marking":
out.append(f"_{text}_")
elif canonical in ("bulleted_item", "numbered_item"):
out.append(f"- {text}")
elif canonical == "quote_block":
out.append(f"> {text}")
else:
out.append(text)
out.append("")
out.append("")
return "\n".join(out)
def assemble_markdown(doc_dir: Path, lang: str = "both") -> str:
"""Read _index.json + chunks/, return assembled markdown."""
index = json.loads((doc_dir / "_index.json").read_text())
doc_id = index.get("doc_id", doc_dir.name)
chunks_meta = index.get("chunks", [])
# Group by page
by_page: dict[int, list[dict]] = {}
for c in chunks_meta:
by_page.setdefault(c.get("page", 0), []).append(c)
for page_chunks in by_page.values():
page_chunks.sort(key=lambda x: x.get("order_in_page", 0))
# Compute summary stats
type_hist: dict[str, int] = {}
ufo_flags: list[str] = []
cryptid_flags: list[str] = []
for c in chunks_meta:
type_hist[c.get("type", "unknown")] = type_hist.get(c.get("type", "unknown"), 0) + 1
chunks_dir = doc_dir / "chunks"
for entry in chunks_meta:
cid = entry.get("chunk_id")
fm, _ = read_chunk(chunks_dir / f"{cid}.md")
if fm.get("ufo_anomaly_detected"):
ufo_flags.append(cid)
if fm.get("cryptid_anomaly_detected"):
cryptid_flags.append(cid)
out: list[str] = []
out.append("---")
out.append(yaml.dump({
"schema_version": "0.2.0",
"type": "master_document",
"doc_id": doc_id,
"total_pages": index.get("total_pages"),
"total_chunks": len(chunks_meta),
"chunk_types_histogram": type_hist,
"ufo_anomalies_flagged": ufo_flags,
"cryptid_anomalies_flagged": cryptid_flags,
"build_approach": "subagents+harness",
"build_model": "claude-sonnet-4-6",
"assembled_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
}, sort_keys=False, allow_unicode=True).rstrip())
out.append("---")
out.append("")
out.append(f"# {doc_id}")
out.append("")
out.append(f"> **{len(chunks_meta)} chunks** across **{index.get('total_pages', '?')} pages** · types: {type_hist}")
if ufo_flags:
out.append(f"> 🛸 **UAP anomalies flagged in chunks:** {', '.join(ufo_flags)}")
out.append("")
for page_num in sorted(by_page.keys()):
out.append(f"## Page {page_num}")
out.append("")
for c in by_page[page_num]:
cid = c.get("chunk_id")
fm, body = read_chunk(chunks_dir / f"{cid}.md")
bbox = fm.get("bbox") or {}
bbox_str = f"{bbox.get('x',0):.2f}/{bbox.get('y',0):.2f}/{bbox.get('w',0):.2f}/{bbox.get('h',0):.2f}"
out.append(f'<a id="{cid}"></a>')
out.append(f"### Chunk {cid} — {fm.get('type','?')} · p{page_num} · bbox: {bbox_str}")
out.append("")
# Render body — body already has **EN:** and **PT-BR:** sections
if lang == "en":
# Extract only EN line
for line in body.split("\n"):
if line.strip().startswith("**EN:**"):
out.append(line)
elif lang == "pt-br":
for line in body.split("\n"):
if line.strip().startswith("**PT-BR:**"):
out.append(line)
else:
out.append(body.rstrip())
out.append("")
# Embed image if applicable
if fm.get("related_image"):
out.append(f"![chunk image](./images/{fm['related_image']})")
out.append("")
out.append("---")
out.append("")
return "\n".join(out)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id", required=True)
ap.add_argument("--validate", action="store_true")
ap.add_argument("--lang", choices=["both", "en", "pt-br"], default="both")
ap.add_argument("--prose", action="store_true", help="Produce text-only reading view (no bbox/metadata, only textual chunks)")
ap.add_argument("--root", default=str(UFO_ROOT / "raw"))
args = ap.parse_args()
doc_dir = Path(args.root) / args.doc_id
if not doc_dir.exists():
sys.stderr.write(f"✗ Doc dir not found: {doc_dir}\n"); sys.exit(1)
if args.validate:
errs = validate(doc_dir)
if errs:
print(f"✗ {len(errs)} validation errors:")
for e in errs[:50]:
print(f" · {e}")
sys.exit(1)
index = json.loads((doc_dir / "_index.json").read_text())
print(f"✓ {len(index.get('chunks', []))} chunks validated across {index.get('total_pages', '?')} pages")
return
if args.prose:
if args.lang == "both":
sys.stderr.write("--prose requires --lang en or --lang pt-br\n"); sys.exit(1)
md = assemble_prose(doc_dir, lang=args.lang)
out_path = doc_dir / f"document.prose.{args.lang}.md"
else:
md = assemble_markdown(doc_dir, lang=args.lang)
out_path = doc_dir / ("document.md" if args.lang == "both" else f"document.{args.lang}.md")
out_path.write_text(md, encoding="utf-8")
print(f"✓ Wrote {out_path} ({len(md)} bytes)")
if __name__ == "__main__":
main()

356
scripts/28-batch-rebuild-all.py Executable file
View file

@ -0,0 +1,356 @@
#!/usr/bin/env python3
"""
28-batch-rebuild-all.py — Batch rebuild ALL declassified UAP/UFO documents into
the agentic chunk schema (raw/<doc-id>/document.md + chunks/ + _index.json).
CRITICAL RULE (user-mandated): each document runs in its OWN fresh `claude -p`
subprocess with empty context. Never accumulate multiple docs in one session.
Workers parallel at the SUBPROCESS level only — never multi-doc inside one
Claude session. Default 2 parallel workers.
Usage:
./28-batch-rebuild-all.py # 2 workers, all unbuilt docs
./28-batch-rebuild-all.py --workers 4
./28-batch-rebuild-all.py --limit 5 # smoke
./28-batch-rebuild-all.py --doc-id doc-X # single doc
./28-batch-rebuild-all.py --resume # skip docs already done
./28-batch-rebuild-all.py --force # rebuild even if exists
"""
from __future__ import annotations
import argparse
import json
import os
import shutil
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from pathlib import Path
from threading import Lock
UFO_ROOT = Path("/Users/guto/ufo")
PNG_ROOT = UFO_ROOT / "processing" / "png"
RAW_ROOT = UFO_ROOT / "raw"
LOG_DIR = RAW_ROOT / "_batch-rebuild"
LOG_DIR.mkdir(parents=True, exist_ok=True)
PROGRESS_LOG = LOG_DIR / "progress.jsonl"
FAILED_LOG = LOG_DIR / "failed.jsonl"
SUMMARY_LOG = LOG_DIR / "summary.json"
PROMPT = """Rebuild the document `{doc_id}` into `raw/{doc_id}/document.md` using the `doc-rebuilder` subagent.
Constraints:
- Process ALL {page_count} pages (p001 .. p{page_count:03d}).
- The doc-rebuilder agent spawns `page-rebuilder`, `image-analyst`, and `table-stitcher` subagents in parallel via Task.
- Output schema: as defined in .claude/agents/doc-rebuilder.md system prompt.
- Target files:
- `/Users/guto/ufo/raw/{doc_id}/document.md` (assembled master)
- `/Users/guto/ufo/raw/{doc_id}/_index.json` (machine-readable chunk index)
- `/Users/guto/ufo/raw/{doc_id}/chunks/c*.md` (one file per chunk)
- `/Users/guto/ufo/raw/{doc_id}/images/IMG-c*.png` (cropped image regions)
- Set frontmatter `build_approach: "subagents"`.
- Bilingual EN + Brazilian PT-BR. Preserve UTF-8 accents.
Begin. When done, output a single line of stats: pages_done, chunks_total, images_extracted, tables_stitched, ufo_anomalies, cryptid_anomalies, wall_seconds."""
progress_lock = Lock()
quota_exhausted = False # global flag: if Anthropic returns "monthly usage limit", stop the batch
def utc_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def append_jsonl(path: Path, record: dict) -> None:
with progress_lock:
with path.open("a", encoding="utf-8") as fh:
fh.write(json.dumps(record, ensure_ascii=False) + "\n")
def discover_docs() -> list[tuple[str, int]]:
"""List all doc_ids with their page count from processing/png/."""
out: list[tuple[str, int]] = []
for doc_dir in sorted(PNG_ROOT.iterdir()):
if not doc_dir.is_dir():
continue
pages = sorted(doc_dir.glob("p-*.png"))
if pages:
out.append((doc_dir.name, len(pages)))
return out
def is_done(doc_id: str) -> bool:
"""A doc is done if raw/<doc-id>--subagent/{document.md,_index.json,chunks/} all exist."""
archive = RAW_ROOT / f"{doc_id}--subagent"
return (archive / "document.md").exists() and (archive / "_index.json").exists() and (archive / "chunks").is_dir()
QUOTA_MARKERS = (
"monthly usage limit",
"usage limit",
"rate limit exceeded",
)
def looks_like_quota_error(result_excerpt: str, raw_stdout: str) -> bool:
blob = (result_excerpt or "") + " " + (raw_stdout or "")
blob = blob.lower()
return any(m in blob for m in QUOTA_MARKERS)
def rebuild_one(doc_id: str, page_count: int, timeout_s: int) -> dict:
"""Run ONE `claude -p` subprocess for ONE document. Fresh context."""
global quota_exhausted
if quota_exhausted:
return {
"doc_id": doc_id,
"page_count": page_count,
"started_at": utc_iso(),
"finished_at": utc_iso(),
"wall_seconds": 0,
"returncode": -3,
"timed_out": False,
"success": False,
"skipped": True,
"skip_reason": "quota_exhausted_already_detected",
"chunks_count": 0,
"images_count": 0,
}
out_dir = RAW_ROOT / doc_id
archive = RAW_ROOT / f"{doc_id}--subagent"
# Wipe any half-built state, start clean
if out_dir.exists():
shutil.rmtree(out_dir)
if archive.exists():
shutil.rmtree(archive)
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / "chunks").mkdir(exist_ok=True)
(out_dir / "images").mkdir(exist_ok=True)
(out_dir / "tables").mkdir(exist_ok=True)
prompt = PROMPT.format(doc_id=doc_id, page_count=page_count)
cmd = [
"claude", "-p",
"--model", "sonnet",
"--output-format", "json",
"--max-turns", "120",
"--allowedTools", "Read,Write,Bash,Task",
"--add-dir", str(UFO_ROOT),
"--",
prompt,
]
t0 = time.time()
started_at = utc_iso()
try:
proc = subprocess.run(
cmd,
capture_output=True,
text=True,
env={**os.environ},
check=False,
timeout=timeout_s,
)
timed_out = False
except subprocess.TimeoutExpired as e:
proc = e
timed_out = True
wall = round(time.time() - t0, 1)
# Parse Claude CLI JSON output
cli: dict = {}
raw_stdout = getattr(proc, "stdout", "") or ""
if not isinstance(raw_stdout, str):
raw_stdout = raw_stdout.decode("utf-8", errors="replace") if raw_stdout else ""
if raw_stdout:
try:
cli = json.loads(raw_stdout)
except json.JSONDecodeError:
cli = {"raw_stdout_tail": raw_stdout[-2000:]}
rc = getattr(proc, "returncode", -1) if not timed_out else -2
# Inspect output
doc_md = out_dir / "document.md"
idx_json = out_dir / "_index.json"
chunks_dir = out_dir / "chunks"
images_dir = out_dir / "images"
chunks_count = len(list(chunks_dir.glob("c*.md"))) if chunks_dir.exists() else 0
images_count = len(list(images_dir.glob("*.png"))) if images_dir.exists() else 0
has_doc = doc_md.exists()
has_idx = idx_json.exists()
success = has_doc and has_idx and chunks_count > 0 and not timed_out
# Archive on success → raw/<doc-id>--subagent/
if success:
if archive.exists():
shutil.rmtree(archive)
shutil.move(str(out_dir), str(archive))
result_excerpt = (cli.get("result") or "")[:500]
record = {
"doc_id": doc_id,
"page_count": page_count,
"started_at": started_at,
"finished_at": utc_iso(),
"wall_seconds": wall,
"returncode": rc,
"timed_out": timed_out,
"success": success,
"has_document_md": has_doc,
"has_index_json": has_idx,
"chunks_count": chunks_count,
"images_count": images_count,
"total_cost_usd": cli.get("total_cost_usd"),
"num_turns": cli.get("num_turns"),
"is_error": cli.get("is_error"),
"usage": cli.get("usage"),
"result_excerpt": result_excerpt,
}
# Detect Anthropic quota errors and flip the global stop flag.
if not success and looks_like_quota_error(result_excerpt, raw_stdout):
record["quota_error"] = True
quota_exhausted = True
append_jsonl(PROGRESS_LOG, record)
if not success:
append_jsonl(FAILED_LOG, record)
return record
def already_processed_ids() -> set[str]:
"""Read progress.jsonl to see which doc_ids have a 'success' record."""
done: set[str] = set()
if not PROGRESS_LOG.exists():
return done
with PROGRESS_LOG.open("r", encoding="utf-8") as fh:
for line in fh:
try:
r = json.loads(line)
if r.get("success"):
done.add(r["doc_id"])
except json.JSONDecodeError:
continue
return done
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--workers", type=int, default=2)
ap.add_argument("--limit", type=int, default=None)
ap.add_argument("--doc-id", default=None, help="Single doc id")
ap.add_argument("--force", action="store_true", help="Rebuild even if archive exists")
ap.add_argument("--timeout-per-page", type=int, default=300, help="Seconds per page (default 300 = 5min)")
ap.add_argument("--min-timeout", type=int, default=900, help="Minimum doc timeout")
ap.add_argument("--max-timeout", type=int, default=14400, help="Maximum doc timeout (4h)")
args = ap.parse_args()
all_docs = discover_docs()
if args.doc_id:
all_docs = [(d, p) for d, p in all_docs if d == args.doc_id]
if not all_docs:
sys.stderr.write(f"✗ doc_id '{args.doc_id}' not found in {PNG_ROOT}\n")
sys.exit(1)
# Filter already-done unless --force
already = already_processed_ids() if not args.force else set()
queue: list[tuple[str, int]] = []
skipped_done = 0
for doc_id, pages in all_docs:
if not args.force and (doc_id in already or is_done(doc_id)):
skipped_done += 1
continue
queue.append((doc_id, pages))
if args.limit:
queue = queue[: args.limit]
print(f"=" * 70)
print(f" BATCH REBUILD — {len(queue)} docs queued, {skipped_done} already done")
print(f" workers: {args.workers} · 1 doc per subprocess (clean context)")
print(f" started: {utc_iso()}")
print(f" progress log: {PROGRESS_LOG}")
print(f"=" * 70)
sys.stdout.flush()
batch_t0 = time.time()
completed = 0
successes = 0
failures = 0
total_cost = 0.0
total_chunks = 0
with ThreadPoolExecutor(max_workers=args.workers) as ex:
futures = {}
for doc_id, pages in queue:
timeout = max(args.min_timeout, min(args.max_timeout, pages * args.timeout_per_page))
fut = ex.submit(rebuild_one, doc_id, pages, timeout)
futures[fut] = (doc_id, pages, timeout)
for fut in as_completed(futures):
doc_id, pages, timeout = futures[fut]
completed += 1
try:
r = fut.result()
except Exception as e:
r = {"doc_id": doc_id, "success": False, "exception": str(e)}
append_jsonl(FAILED_LOG, r)
if r.get("skipped"):
marker = "⊘"
failures += 1
elif r.get("success"):
successes += 1
total_cost += r.get("total_cost_usd") or 0.0
total_chunks += r.get("chunks_count") or 0
marker = "✓"
else:
failures += 1
marker = "✗"
if r.get("quota_error"):
marker = "💸"
wall_doc = r.get("wall_seconds", 0)
chunks = r.get("chunks_count", 0)
cost = r.get("total_cost_usd") or 0.0
elapsed = round(time.time() - batch_t0, 0)
print(f" [{completed}/{len(queue)}] {marker} {doc_id} · pages={pages} chunks={chunks} cost=${cost:.2f} wall={wall_doc}s · batch_elapsed={int(elapsed)}s")
sys.stdout.flush()
if quota_exhausted:
print(f"\n ⚠ QUOTA EXHAUSTED — stopping batch. Re-run later (rolling 5h window).")
for f in futures:
if not f.done():
f.cancel()
break
summary = {
"started_at": utc_iso(),
"queue_size": len(queue),
"completed": completed,
"successes": successes,
"failures": failures,
"total_cost_usd": round(total_cost, 2),
"total_chunks": total_chunks,
"batch_wall_seconds": round(time.time() - batch_t0, 1),
"workers": args.workers,
}
SUMMARY_LOG.write_text(json.dumps(summary, indent=2))
print(f"\n{'=' * 70}")
print(f" DONE — {successes}/{completed} succeeded · ${total_cost:.2f} total · {total_chunks} chunks")
print(f" summary: {SUMMARY_LOG}")
print(f"{'=' * 70}")
if __name__ == "__main__":
main()

79
scripts/29-auto-resume-batch.sh Executable file
View file

@ -0,0 +1,79 @@
#!/usr/bin/env bash
# auto-resume-batch.sh — polls Anthropic quota every 30min via a tiny probe call.
# When a small probe doc succeeds (or the probe also bails quickly with quota),
# decides whether to launch the full batch.
#
# Stops itself when:
# - batch reports successful completion (summary.json with successes >= queue_size)
# - a probe goes 90s without quota error (interpreted as quota back)
#
# Usage:
# nohup ./scripts/29-auto-resume-batch.sh > /tmp/auto-resume.log 2>&1 &
set -uo pipefail
UFO_ROOT=/Users/guto/ufo
LOG_DIR="$UFO_ROOT/raw/_batch-rebuild"
SLEEP_BETWEEN=1800 # 30min between probes
MAX_ATTEMPTS=24 # 24 × 30min = 12h ceiling
log() { echo "[$(date -u +%H:%M:%SZ)] $*"; }
attempt=0
while [ $attempt -lt $MAX_ATTEMPTS ]; do
attempt=$((attempt + 1))
log "attempt $attempt/$MAX_ATTEMPTS — probing batch"
# Check if anything is already running — bail early
if pgrep -f "28-batch-rebuild-all.py" >/dev/null; then
log "batch already running, sleeping ${SLEEP_BETWEEN}s and re-checking"
sleep $SLEEP_BETWEEN
continue
fi
# Snapshot current archive count
before=$(ls -d "$UFO_ROOT"/raw/*--subagent 2>/dev/null | wc -l | tr -d ' ')
log " archived before: $before"
# Kick off batch (will early-abort if quota still throttled)
cd "$UFO_ROOT"
python3 scripts/28-batch-rebuild-all.py --workers 2 \
> /tmp/batch-rebuild-auto-$attempt.log 2>&1 &
PID=$!
log " started python orchestrator PID=$PID"
# Wait for either:
# - process exits (early-abort or done)
# - 90s elapsed without exit (means it's actually running real work)
for i in $(seq 1 90); do
if ! kill -0 $PID 2>/dev/null; then
break
fi
sleep 1
done
if kill -0 $PID 2>/dev/null; then
# Still running after 90s → real work, leave it alone and exit auto-resume
log " ✓ batch is making real progress (still running after 90s)"
log " auto-resume exits; full batch continues in background"
log " monitor: tail -f /tmp/batch-rebuild-auto-$attempt.log"
exit 0
fi
# Process exited within 90s — must have hit quota or completed
after=$(ls -d "$UFO_ROOT"/raw/*--subagent 2>/dev/null | wc -l | tr -d ' ')
delta=$((after - before))
log " process exited fast (likely quota); archived delta: $delta"
if [ "$delta" -gt 0 ]; then
log " ✓ some docs were processed — re-launching immediately"
sleep 5
continue
fi
log " 💤 quota still throttled; sleeping ${SLEEP_BETWEEN}s"
sleep $SLEEP_BETWEEN
done
log "max attempts reached, giving up. re-run manually."
exit 1

382
scripts/30-index-chunks-to-db.py Executable file
View file

@ -0,0 +1,382 @@
#!/usr/bin/env python3
"""
30-index-chunks-to-db.py — Populate Postgres `documents` + `chunks` tables from
raw/<doc-id>--subagent/{_index.json, chunks/c*.md}. Embeds each chunk via the
embed-service (BGE-M3, 1024-dim dense) and UPSERTs into pgvector.
Idempotent: re-running re-embeds + overwrites. Per-doc transaction.
Resumable: pass --skip-existing to skip docs already indexed.
Usage:
./30-index-chunks-to-db.py # all archived docs
./30-index-chunks-to-db.py --doc-id doc-342-... # single doc
./30-index-chunks-to-db.py --lang pt # which content field to embed (default: pt)
./30-index-chunks-to-db.py --skip-existing # incremental
./30-index-chunks-to-db.py --batch-size 16 # chunks per embed call
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
from typing import Iterable, Optional
try:
import yaml
import psycopg
from psycopg.types.json import Jsonb
import requests
except ImportError as e:
sys.stderr.write(f"pip3 install pyyaml psycopg[binary] requests # missing: {e}\n")
sys.exit(1)
UFO_ROOT = Path(os.getenv("UFO_ROOT", "/Users/guto/ufo"))
RAW_ROOT = UFO_ROOT / "raw"
WIKI_DOCS = UFO_ROOT / "wiki" / "documents"
DATABASE_URL = os.getenv("DATABASE_URL") or os.getenv("SUPABASE_DB_URL")
EMBED_URL = os.getenv("EMBED_SERVICE_URL", "http://localhost:8000")
def utc_iso() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
# Known free-text string fields in chunk frontmatter that may contain literal
# colons. The Sonnet generator sometimes leaves them unquoted, which breaks
# yaml.safe_load. We quote them defensively before parsing.
import re as _re # placed here to keep import scope tight; module-level `re` not needed elsewhere
STRING_FIELDS_NEEDING_QUOTE = {
"ufo_anomaly_rationale",
"cryptid_anomaly_rationale",
"image_description_en",
"image_description_pt_br",
"extracted_text",
"redaction_inferred_content_type",
}
_STRING_FIELD_LINE_RE = _re.compile(
r"^(\s*)(" + "|".join(_re.escape(k) for k in STRING_FIELDS_NEEDING_QUOTE) + r"):\s*(.*)$"
)
def _autoquote_free_text(yaml_text: str) -> str:
"""Wrap unquoted values of known string fields in single quotes to survive
YAML colons inside the value. Also normalises Python literals (`None`,
`True`, `False`) at end-of-line into YAML literals. Idempotent."""
# First pass: normalise Python-literal Nones to YAML nulls so downstream
# float() / int() coercion works.
yaml_text = _re.sub(r":[ \t]+None([ \t]*)$", r": null\1", yaml_text, flags=_re.MULTILINE)
out = []
for line in yaml_text.split("\n"):
m = _STRING_FIELD_LINE_RE.match(line)
if not m:
out.append(line)
continue
indent, key, value = m.group(1), m.group(2), m.group(3).rstrip()
if value in ("", "null", "None", "~") or value.startswith(("'", '"', "|", ">")):
out.append(line)
continue
escaped = value.replace("'", "''")
out.append(f"{indent}{key}: '{escaped}'")
return "\n".join(out)
def read_chunk_md(path: Path) -> tuple[dict, str, str]:
"""Return (frontmatter, content_en, content_pt)."""
raw = path.read_text(encoding="utf-8")
if not raw.startswith("---"):
return {}, "", ""
end = raw.find("---", 4)
fm_yaml = _autoquote_free_text(raw[3:end].strip())
try:
fm = yaml.safe_load(fm_yaml) or {}
except yaml.YAMLError:
fm = {}
body = raw[end + 3:].lstrip("\n")
en, pt = "", ""
for line in body.split("\n"):
s = line.strip()
if s.startswith("**EN:**"):
en = s.removeprefix("**EN:**").strip()
elif s.startswith("**PT-BR:**"):
pt = s.removeprefix("**PT-BR:**").strip()
return fm, en, pt
def discover_built_docs() -> list[Path]:
return sorted(p for p in RAW_ROOT.glob("*--subagent") if (p / "_index.json").exists())
def embed_batch(texts: list[str]) -> list[list[float]]:
"""Call embed-service /embed; returns 1024-dim vectors."""
if not texts:
return []
resp = requests.post(
f"{EMBED_URL}/embed",
json={"texts": texts, "normalize": True},
timeout=120,
)
resp.raise_for_status()
data = resp.json()
return data["embeddings"]
def vector_literal(vec: list[float]) -> str:
"""pgvector accepts the textual form '[1.0,2.0,...]'."""
return "[" + ",".join(f"{v:.6f}" for v in vec) + "]"
def read_wiki_document_meta(doc_id: str) -> dict:
"""Pull canonical_title / collection / document_class / content_classification
from wiki/documents/<doc-id>.md frontmatter if present."""
p = WIKI_DOCS / f"{doc_id}.md"
if not p.exists():
return {}
try:
raw = p.read_text(encoding="utf-8")
if not raw.startswith("---"):
return {}
end = raw.find("---", 4)
fm = yaml.safe_load(raw[3:end].strip()) or {}
return fm
except Exception:
return {}
def upsert_document(cur, doc_id: str, idx: dict, archive_path: Path) -> None:
wiki_meta = read_wiki_document_meta(doc_id)
canonical_title = (
idx.get("canonical_title")
or wiki_meta.get("canonical_title")
or doc_id
)
content_class = wiki_meta.get("content_classification")
if content_class is not None and not isinstance(content_class, list):
content_class = [content_class]
cur.execute(
"""
INSERT INTO public.documents (
doc_id, canonical_title, collection, document_class, page_count,
classification, content_class,
schema_version, build_approach, build_model, built_at, raw_path
)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
ON CONFLICT (doc_id) DO UPDATE SET
canonical_title = EXCLUDED.canonical_title,
collection = EXCLUDED.collection,
document_class = EXCLUDED.document_class,
page_count = EXCLUDED.page_count,
classification = EXCLUDED.classification,
content_class = EXCLUDED.content_class,
schema_version = EXCLUDED.schema_version,
build_approach = EXCLUDED.build_approach,
build_model = EXCLUDED.build_model,
built_at = EXCLUDED.built_at,
raw_path = EXCLUDED.raw_path,
ingested_at = NOW()
""",
(
doc_id,
canonical_title,
wiki_meta.get("collection"),
wiki_meta.get("document_class"),
idx.get("total_pages") or wiki_meta.get("page_count"),
wiki_meta.get("highest_classification") or wiki_meta.get("classification"),
content_class,
idx.get("schema_version", "0.2.0"),
idx.get("build_approach"),
idx.get("build_model"),
idx.get("build_at"),
str(archive_path.resolve()),
),
)
def index_one_doc(cur, archive: Path, lang: str, batch_size: int) -> tuple[int, int]:
idx_path = archive / "_index.json"
if not idx_path.exists():
return (0, 0)
idx = json.loads(idx_path.read_text(encoding="utf-8"))
doc_id = idx.get("doc_id") or archive.name.removesuffix("--subagent")
upsert_document(cur, doc_id, idx, archive)
# Wipe + re-insert chunks for this doc (idempotency)
cur.execute("DELETE FROM public.chunks WHERE doc_id = %s", (doc_id,))
chunks_dir = archive / "chunks"
entries = idx.get("chunks", [])
if not entries:
return (0, 0)
rows: list[tuple] = []
texts_for_embed: list[str] = []
def _scalar(v):
"""Defensively coerce dicts/lists to a string so psycopg can bind them
as text columns. Sonnet sometimes emits a mapping where a scalar is
expected (e.g. `redaction_inferred_content_type: {kind: x, note: y}`)."""
if v is None or isinstance(v, (str, int, float, bool)):
return v
if isinstance(v, (dict, list, tuple)):
try:
import json as _j
return _j.dumps(v, ensure_ascii=False)
except Exception:
return str(v)
return str(v)
for entry in entries:
cid = entry.get("chunk_id")
if not cid:
continue
chunk_file = chunks_dir / f"{cid}.md"
if not chunk_file.exists():
continue
fm, en, pt = read_chunk_md(chunk_file)
text_for_embed = pt if lang == "pt" else en
if not text_for_embed:
text_for_embed = en or pt or ""
texts_for_embed.append(text_for_embed)
rows.append(
(
doc_id,
cid,
int(fm.get("page") or entry.get("page") or 0),
int(fm.get("order_in_page") or entry.get("order_in_page") or 0),
int(fm.get("order_global") or entry.get("order_global") or 0),
str(fm.get("type") or entry.get("type") or "unknown"),
Jsonb(fm.get("bbox") or entry.get("bbox") or {}),
en or None,
pt or None,
(lambda v: float(v) if isinstance(v, (int, float)) else (float(v) if isinstance(v, str) and v.strip() not in ("", "None", "null") else None))(fm.get("ocr_confidence")),
_scalar(fm.get("classification")),
[str(x) if not isinstance(x, str) else x for x in (fm.get("formatting") or []) if x is not None],
_scalar(fm.get("cross_page_hint")),
_scalar(fm.get("prev_chunk")),
_scalar(fm.get("next_chunk")),
_scalar(fm.get("related_image")),
_scalar(fm.get("related_table")),
_scalar(fm.get("redaction_code")),
_scalar(fm.get("redaction_inferred_content_type")),
_scalar(fm.get("image_type")),
bool(fm.get("ufo_anomaly_detected") or False),
_scalar(fm.get("ufo_anomaly_type")),
_scalar(fm.get("ufo_anomaly_rationale")),
bool(fm.get("cryptid_anomaly_detected") or False),
_scalar(fm.get("cryptid_anomaly_type")),
_scalar(fm.get("cryptid_anomaly_rationale")),
_scalar(fm.get("image_description_en")),
_scalar(fm.get("image_description_pt_br")),
_scalar(fm.get("source_png")),
)
)
# Embed in batches
all_embeddings: list[list[float]] = []
for i in range(0, len(texts_for_embed), batch_size):
batch = texts_for_embed[i : i + batch_size]
all_embeddings.extend(embed_batch(batch))
# Bulk insert with vectors (cast text → vector in SQL)
insert_sql = """
INSERT INTO public.chunks (
doc_id, chunk_id, page, order_in_page, order_global, type, bbox,
content_en, content_pt, ocr_confidence, classification, formatting,
cross_page_hint, prev_chunk, next_chunk, related_image, related_table,
redaction_code, redaction_inferred, image_type,
ufo_anomaly, ufo_anomaly_type, ufo_rationale,
cryptid_anomaly, cryptid_anomaly_type, cryptid_rationale,
image_desc_en, image_desc_pt, source_png, embedding
)
VALUES (
%s, %s, %s, %s, %s, %s, %s,
%s, %s, %s, %s, %s,
%s, %s, %s, %s, %s,
%s, %s, %s,
%s, %s, %s,
%s, %s, %s,
%s, %s, %s, %s::vector
)
"""
for row, vec in zip(rows, all_embeddings):
cur.execute(insert_sql, row + (vector_literal(vec),))
return (len(rows), len(all_embeddings))
def is_already_indexed(cur, doc_id: str) -> bool:
cur.execute(
"SELECT 1 FROM public.documents WHERE doc_id = %s",
(doc_id,),
)
return cur.fetchone() is not None
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--doc-id", default=None, help="Index a single doc (no --subagent suffix)")
ap.add_argument("--lang", choices=["pt", "en"], default="pt", help="Language to embed (default: pt)")
ap.add_argument("--batch-size", type=int, default=16)
ap.add_argument("--skip-existing", action="store_true")
args = ap.parse_args()
if not DATABASE_URL:
sys.stderr.write("✗ Set DATABASE_URL (or SUPABASE_DB_URL) env var\n")
sys.exit(1)
# Probe embed service
try:
r = requests.get(f"{EMBED_URL}/health", timeout=10)
r.raise_for_status()
print(f" ✓ embed service: {EMBED_URL} → {r.json()}")
except Exception as e:
sys.stderr.write(f"✗ embed service unreachable at {EMBED_URL}: {e}\n")
sys.exit(1)
archives = discover_built_docs()
if args.doc_id:
archives = [a for a in archives if a.name.removesuffix("--subagent") == args.doc_id]
if not archives:
sys.stderr.write(f"✗ doc not built yet: raw/{args.doc_id}--subagent missing\n")
sys.exit(1)
print(f" found {len(archives)} built doc(s)")
t0 = time.time()
total_chunks = 0
total_docs = 0
with psycopg.connect(DATABASE_URL, autocommit=False) as conn:
for archive in archives:
doc_id = archive.name.removesuffix("--subagent")
if args.skip_existing:
with conn.cursor() as cur:
if is_already_indexed(cur, doc_id):
print(f" ⊘ skip {doc_id} (already indexed)")
continue
t_doc = time.time()
try:
with conn.cursor() as cur:
n_chunks, n_embed = index_one_doc(cur, archive, args.lang, args.batch_size)
conn.commit()
wall = round(time.time() - t_doc, 1)
print(f" ✓ {doc_id} · {n_chunks} chunks · {n_embed} embedded · {wall}s")
total_chunks += n_chunks
total_docs += 1
except Exception as e:
conn.rollback()
print(f" ✗ {doc_id} FAILED: {e}")
print(f"\nDONE — {total_docs} docs · {total_chunks} chunks · {round(time.time() - t0, 1)}s total")
if __name__ == "__main__":
main()

207
scripts/31-populate-entity-mentions.py Executable file
View file

@ -0,0 +1,207 @@
#!/usr/bin/env python3
"""
31-populate-entity-mentions.py — Materialize public.entity_mentions table.
Reads wiki/entities/<class>/<id>.md frontmatter for canonical_name + aliases.
For each entity, scans public.chunks via ILIKE (powered by pg_trgm GIN index)
to find chunks where the entity appears literally. Upserts entity_mentions
rows linking chunk_pk ↔ entity_pk.
Pre-populates public.entities too if not already there.
Usage:
./31-populate-entity-mentions.py # all classes
./31-populate-entity-mentions.py --class people # one class
./31-populate-entity-mentions.py --limit 100 # smoke
"""
from __future__ import annotations
import argparse
import os
import re
import sys
import time
from pathlib import Path
try:
import yaml
import psycopg
except ImportError as e:
sys.stderr.write(f"pip3 install pyyaml psycopg[binary] # missing: {e}\n")
sys.exit(1)
UFO_ROOT = Path(os.getenv("UFO_ROOT", "/Users/guto/ufo"))
WIKI = UFO_ROOT / "wiki"
DATABASE_URL = os.getenv("DATABASE_URL") or os.getenv("SUPABASE_DB_URL")
CLASSES = ["people", "organizations", "locations", "events", "uap-objects", "vehicles", "operations", "concepts"]
CLASS_SINGULAR = {
"people": "person",
"organizations": "organization",
"locations": "location",
"events": "event",
"uap-objects": "uap_object",
"vehicles": "vehicle",
"operations": "operation",
"concepts": "concept",
}
def read_frontmatter(path: Path) -> dict:
raw = path.read_text(encoding="utf-8")
if not raw.startswith("---"):
return {}
end = raw.find("---", 4)
return yaml.safe_load(raw[3:end].strip()) or {}
def collect_search_strings(canonical: str, aliases: list[str]) -> list[str]:
"""Aliases + canonical, normalized — used as ILIKE patterns."""
items = set()
if canonical:
items.add(canonical.strip())
for a in aliases or []:
a = str(a).strip()
if not a:
continue
items.add(a)
# Filter: very short or all-numeric strings are too noisy
out: list[str] = []
for s in items:
if len(s) < 3:
continue
if s.isdigit():
continue
# SQL ILIKE escape — % and _ are wildcards
out.append(s.replace("%", r"\%").replace("_", r"\_"))
return out
def upsert_entity(cur, cls_folder: str, entity_id: str, fm: dict) -> int | None:
canonical = (fm.get("canonical_name") or entity_id).strip()
aliases = fm.get("aliases") or []
if not isinstance(aliases, list):
aliases = [str(aliases)]
aliases = [str(a).strip() for a in aliases if str(a).strip()]
total_mentions = int(fm.get("total_mentions") or 0)
documents_count = int(fm.get("documents_count") or 0)
enrichment_status = fm.get("enrichment_status")
last_ingest = fm.get("last_ingest")
entity_class = CLASS_SINGULAR.get(cls_folder, cls_folder)
cur.execute(
"""
INSERT INTO public.entities (
entity_class, entity_id, canonical_name, aliases,
total_mentions, documents_count, enrichment_status, last_ingest
)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
ON CONFLICT (entity_class, entity_id) DO UPDATE SET
canonical_name = EXCLUDED.canonical_name,
aliases = EXCLUDED.aliases,
total_mentions = EXCLUDED.total_mentions,
documents_count = EXCLUDED.documents_count,
enrichment_status = EXCLUDED.enrichment_status,
last_ingest = EXCLUDED.last_ingest
RETURNING entity_pk
""",
(
entity_class,
entity_id,
canonical,
aliases,
total_mentions,
documents_count,
enrichment_status,
last_ingest,
),
)
row = cur.fetchone()
return row[0] if row else None
def find_mentioning_chunks(cur, entity_pk: int, patterns: list[str]) -> int:
"""For each pattern, find chunks where it appears ILIKE; insert into entity_mentions."""
inserted = 0
for p in patterns:
cur.execute(
"""
INSERT INTO public.entity_mentions (chunk_pk, entity_pk, surface_form)
SELECT chunk_pk, %s, %s
FROM public.chunks
WHERE content_pt ILIKE '%%' || %s || '%%'
OR content_en ILIKE '%%' || %s || '%%'
ON CONFLICT (chunk_pk, entity_pk) DO NOTHING
""",
(entity_pk, p, p, p),
)
inserted += cur.rowcount or 0
return inserted
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--class", dest="cls", default=None, choices=CLASSES, help="Filter to one class")
ap.add_argument("--limit", type=int, default=None, help="Limit entities per class (smoke test)")
ap.add_argument("--reset", action="store_true", help="Truncate entity_mentions before run")
args = ap.parse_args()
if not DATABASE_URL:
sys.stderr.write("✗ Set DATABASE_URL\n")
sys.exit(1)
target_classes = [args.cls] if args.cls else CLASSES
t0 = time.time()
total_entities = 0
total_mentions = 0
with psycopg.connect(DATABASE_URL, autocommit=False) as conn:
if args.reset:
with conn.cursor() as cur:
cur.execute("TRUNCATE public.entity_mentions RESTART IDENTITY")
print(" ✓ TRUNCATE entity_mentions")
conn.commit()
for cls_folder in target_classes:
cls_dir = WIKI / "entities" / cls_folder
if not cls_dir.is_dir():
print(f" ⊘ {cls_folder} dir missing")
continue
files = sorted(cls_dir.glob("*.md"))
if args.limit:
files = files[: args.limit]
print(f" ▸ {cls_folder}: {len(files)} entities")
for i, fpath in enumerate(files):
eid = fpath.stem
try:
fm = read_frontmatter(fpath)
except Exception as e:
print(f" ✗ {eid}: bad frontmatter ({e})")
continue
with conn.cursor() as cur:
epk = upsert_entity(cur, cls_folder, eid, fm)
if not epk:
continue
patterns = collect_search_strings(
fm.get("canonical_name") or eid,
fm.get("aliases") or [],
)
if not patterns:
continue
m = find_mentioning_chunks(cur, epk, patterns)
total_mentions += m
conn.commit()
total_entities += 1
if (i + 1) % 500 == 0:
elapsed = round(time.time() - t0, 0)
print(f" [{i+1}/{len(files)}] {cls_folder} · {total_mentions} mentions · {int(elapsed)}s")
print(f"\nDONE — {total_entities} entities · {total_mentions} mentions · {round(time.time() - t0, 1)}s")
if __name__ == "__main__":
main()

176
scripts/32-sync-mentioned-in-yaml.py Executable file
View file

@ -0,0 +1,176 @@
#!/usr/bin/env python3
"""
32-sync-mentioned-in-yaml.py — Backfill `mentioned_in[]` in wiki/entities/<class>/<id>.md
from public.entity_mentions JOIN public.chunks.
After 31-populate-entity-mentions.py has run, the DB has the truth. This script
syncs that truth BACK into the markdown frontmatter so the legacy entity page
("Appears in N pages") matches what the new graph/retrieval layer sees.
Idempotent: rewrites mentioned_in[] in place. Preserves all other frontmatter.
Skips entities with 0 mentions (leaves existing list untouched).
Usage:
./32-sync-mentioned-in-yaml.py # all classes
./32-sync-mentioned-in-yaml.py --class people # one class
./32-sync-mentioned-in-yaml.py --dry-run # show diffs, no writes
./32-sync-mentioned-in-yaml.py --max-mentions 200 # cap list length per entity
"""
from __future__ import annotations
import argparse
import os
import sys
import time
from pathlib import Path
try:
import yaml
import psycopg
except ImportError as e:
sys.stderr.write(f"pip3 install pyyaml psycopg[binary] # missing: {e}\n")
sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
WIKI = UFO_ROOT / "wiki"
DATABASE_URL = os.getenv("DATABASE_URL") or os.getenv("SUPABASE_DB_URL")
CLASSES = ["people", "organizations", "locations", "events", "uap-objects", "vehicles", "operations", "concepts"]
CLASS_SINGULAR = {
"people": "person",
"organizations": "organization",
"locations": "location",
"events": "event",
"uap-objects": "uap_object",
"vehicles": "vehicle",
"operations": "operation",
"concepts": "concept",
}
def split_frontmatter(raw: str) -> tuple[str, str]:
"""Return (frontmatter_yaml, body). Raises if no frontmatter."""
if not raw.startswith("---"):
return "", raw
end = raw.find("---", 4)
if end < 0:
return "", raw
return raw[3:end].strip(), raw[end + 3:].lstrip("\n")
def fetch_mentions(cur, entity_class: str, entity_id: str, max_n: int) -> list[dict]:
cur.execute(
"""
SELECT c.doc_id, c.page, c.chunk_id, em.surface_form
FROM public.entities e
JOIN public.entity_mentions em ON em.entity_pk = e.entity_pk
JOIN public.chunks c ON c.chunk_pk = em.chunk_pk
WHERE e.entity_class = %s AND e.entity_id = %s
ORDER BY c.doc_id, c.order_global
LIMIT %s
""",
(entity_class, entity_id, max_n),
)
rows = cur.fetchall()
return [
{
"page": f"[[{r[0]}/p{r[1]:03d}]]",
"page_ref": f"[[{r[0]}/p{r[1]:03d}#{r[2]}]]",
"doc_id": r[0],
"chunk_id": r[2],
"surface_form": r[3],
}
for r in rows
]
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--class", dest="cls", default=None, choices=CLASSES)
ap.add_argument("--dry-run", action="store_true")
ap.add_argument("--max-mentions", type=int, default=200, help="Cap per entity (default 200)")
args = ap.parse_args()
if not DATABASE_URL:
sys.stderr.write("✗ Set DATABASE_URL\n")
sys.exit(1)
target_classes = [args.cls] if args.cls else CLASSES
total_updated = 0
total_skipped_empty = 0
total_unchanged = 0
t0 = time.time()
with psycopg.connect(DATABASE_URL) as conn:
for cls_folder in target_classes:
cls_dir = WIKI / "entities" / cls_folder
if not cls_dir.is_dir():
continue
entity_class_sg = CLASS_SINGULAR[cls_folder]
files = sorted(cls_dir.glob("*.md"))
print(f" ▸ {cls_folder}: {len(files)} files")
for i, fpath in enumerate(files):
eid = fpath.stem
try:
raw = fpath.read_text(encoding="utf-8")
except Exception as e:
print(f" ✗ {eid}: read failed ({e})")
continue
fm_yaml, body = split_frontmatter(raw)
if not fm_yaml:
continue
try:
fm = yaml.safe_load(fm_yaml) or {}
except yaml.YAMLError as e:
print(f" ✗ {eid}: yaml ({e})")
continue
with conn.cursor() as cur:
mentions = fetch_mentions(cur, entity_class_sg, eid, args.max_mentions)
if not mentions:
total_skipped_empty += 1
continue
# Build new mentioned_in list — preserve order, dedupe by page_ref
seen = set()
new_mentions = []
for m in mentions:
if m["page_ref"] in seen:
continue
seen.add(m["page_ref"])
new_mentions.append(
{"page": m["page"], "page_ref": m["page_ref"], "doc_id": m["doc_id"]}
)
old_count = len(fm.get("mentioned_in") or [])
fm["mentioned_in"] = new_mentions
fm["total_mentions"] = len(new_mentions)
new_yaml = yaml.safe_dump(fm, sort_keys=False, allow_unicode=True, width=120)
new_raw = f"---\n{new_yaml}---\n{body}"
if new_raw == raw:
total_unchanged += 1
continue
if args.dry_run:
print(f" Δ {eid}: {old_count} → {len(new_mentions)} mentions")
else:
fpath.write_text(new_raw, encoding="utf-8")
total_updated += 1
if (i + 1) % 500 == 0:
elapsed = round(time.time() - t0, 0)
print(f" [{i+1}/{len(files)}] updated={total_updated} · {int(elapsed)}s")
print(
f"\nDONE — updated={total_updated} skipped_empty={total_skipped_empty} unchanged={total_unchanged} · {round(time.time() - t0, 1)}s"
)
if __name__ == "__main__":
main()

86
scripts/33-compact-progress-log.py Executable file
View file

@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""
33-compact-progress-log.py — Compact raw/_batch-rebuild/progress.jsonl by
keeping ONLY the latest entry per doc_id.
Useful after multiple resume runs: instead of 200 rows for 115 docs (retries
included), get back to 115 (or fewer) — one per doc with the latest outcome.
Idempotent + safe: writes to a temp file, atomically moves on success, keeps
the prior version as `.bak`.
Usage:
./33-compact-progress-log.py # compact in place
./33-compact-progress-log.py --dry-run # show what would change
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import time
from pathlib import Path
UFO_ROOT = Path("/Users/guto/ufo")
LOG_DIR = UFO_ROOT / "raw" / "_batch-rebuild"
def compact_file(path: Path, *, dry_run: bool) -> tuple[int, int]:
if not path.exists():
return (0, 0)
rows: list[dict] = []
with path.open("r", encoding="utf-8") as fh:
for line in fh:
line = line.strip()
if not line:
continue
try:
rows.append(json.loads(line))
except json.JSONDecodeError:
continue
before = len(rows)
# Keep last row per doc_id (preserves insertion order via dict)
latest: dict[str, dict] = {}
for r in rows:
doc = r.get("doc_id")
if doc:
latest[doc] = r
kept = list(latest.values())
after = len(kept)
if dry_run:
return (before, after)
# Write atomically
tmp = path.with_suffix(path.suffix + ".tmp")
with tmp.open("w", encoding="utf-8") as fh:
for r in kept:
fh.write(json.dumps(r, ensure_ascii=False) + "\n")
bak = path.with_suffix(path.suffix + f".bak-{int(time.time())}")
if path.exists():
os.replace(path, bak)
os.replace(tmp, path)
return (before, after)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args()
for name in ("progress.jsonl", "failed.jsonl"):
p = LOG_DIR / name
if not p.exists():
print(f" ⊘ {name}: not present")
continue
before, after = compact_file(p, dry_run=args.dry_run)
verb = "would write" if args.dry_run else "wrote"
print(f" ✓ {name}: {before} rows → {after} unique doc_ids ({verb})")
if __name__ == "__main__":
main()

378
scripts/34-generate-doc-pitches.py Executable file
View file

@ -0,0 +1,378 @@
#!/usr/bin/env python3
"""
34-generate-doc-pitches.py — Generate Johnny Harris-style "enthusiast pitches"
(50-150 words, PT-BR + EN) for every document in wiki/documents/*.md.
Each pitch is injected into the doc's frontmatter as:
enthusiast_pitch_pt_br: "..."
enthusiast_pitch_en: "..."
Style guide (encoded in the prompt):
- Mystery hook opening (date + place)
- Concrete sensory details, real witness names with credentials
- Staccato pacing, repetition for emphasis
- Cliffhanger question at the end
- Length adapts to doc richness: single dense case → ~140w · multi-case → focus on
pattern + zoom on one · sparse doc → 50-80w
Pattern: each doc in its OWN `claude -p` subprocess (clean context). Workers
parallel for throughput. Idempotent: skips docs that already have pitch.
Usage:
./34-generate-doc-pitches.py # all docs missing pitch
./34-generate-doc-pitches.py --workers 4
./34-generate-doc-pitches.py --doc-id doc-X # single doc
./34-generate-doc-pitches.py --force # regenerate even if exists
./34-generate-doc-pitches.py --model haiku # cheaper, faster
"""
from __future__ import annotations
import argparse
import json
import os
import re
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from threading import Lock
try:
import yaml
except ImportError:
sys.stderr.write("pip3 install pyyaml\n"); sys.exit(1)
UFO_ROOT = Path("/Users/guto/ufo")
WIKI_DOCS = UFO_ROOT / "wiki" / "documents"
LOG_DIR = UFO_ROOT / "raw" / "_pitch-generation"
LOG_DIR.mkdir(parents=True, exist_ok=True)
PROGRESS_LOG = LOG_DIR / "progress.jsonl"
progress_lock = Lock()
quota_exhausted = False # detected globally → stops new spawns
SYSTEM_PROMPT = """You are writing in the voice of JOHNNY HARRIS — independent journalist, Vox/YouTube-style visual storyteller. Brazilian Portuguese (pt-br, NOT European). Companion piece for an English version.
Your goal: produce a SHORT (50-150 words) "enthusiast pitch" for a single declassified UAP/UFO document, designed to hook a curious lay reader on a website card.
STYLE RULES:
1. Open with a MYSTERY HOOK — date + place, concrete and grounded ("24 de abril de 1964, fim de tarde. Socorro, Novo México.")
2. Use staccato sentences. Forward motion. Plain language, no jargon.
3. Anchor in specific details: witness names + credentials, altitudes, coordinates, formal stamps (CONFIDENTIAL/RESTRICTED), unit numbers.
4. Repetition for emphasis. ("Sem som. Sem rastro.")
5. End with a CLIFFHANGER question or final stamp/document marker. Never a summary sentence.
6. Bold key facts with **markdown** if helpful (3-5 max).
7. Preserve verbatim quotes from the source in English when they're vivid (e.g., RESTRICTED, callsigns, military jargon).
LENGTH ADAPTS TO DOC RICHNESS:
- Sparse doc / form fragment: 50-80 words
- One dense case: 100-150 words
- Multi-case doc: lead with scale ("100 incidentes em uma pasta"), zoom on ONE vivid case, signal the rest ("e há mais 99 desses"), pattern recognition, final question.
OUTPUT FORMAT — return EXACTLY this JSON, nothing else:
{
"pitch_pt_br": "...",
"pitch_en": "..."
}
Both versions should hit roughly the same word count. PT-BR is the primary; EN is a faithful adaptation, NOT a literal translation."""
USER_PROMPT_TEMPLATE = """Generate the enthusiast pitch for this declassified document:
DOC ID: {doc_id}
TITLE: {canonical_title}
COLLECTION: {collection}
PAGES: {page_count}
CLASSIFICATION: {classification}
DOCUMENT BODY (truncated to first 6000 chars — focus on substantive content):
{body}
Return the JSON now."""
def utc_iso() -> str:
from datetime import datetime, timezone
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def append_jsonl(path: Path, record: dict) -> None:
with progress_lock:
with path.open("a", encoding="utf-8") as fh:
fh.write(json.dumps(record, ensure_ascii=False) + "\n")
def read_doc(doc_path: Path) -> tuple[dict, str, str]:
raw = doc_path.read_text(encoding="utf-8")
if not raw.startswith("---"):
return {}, "", raw
end = raw.find("---", 4)
if end < 0:
return {}, "", raw
fm_text = raw[3:end].strip()
body = raw[end + 3 :].lstrip("\n")
try:
fm = yaml.safe_load(fm_text) or {}
except yaml.YAMLError:
fm = {}
return fm, fm_text, body
def write_doc(doc_path: Path, fm: dict, body: str) -> None:
new_yaml = yaml.safe_dump(fm, sort_keys=False, allow_unicode=True, width=120)
new_raw = f"---\n{new_yaml}---\n{body}"
doc_path.write_text(new_raw, encoding="utf-8")
def call_claude(doc_id: str, fm: dict, body: str, model: str, timeout_s: int) -> tuple[bool, dict]:
"""Call `claude -p` for ONE pitch. Returns (success, payload)."""
global quota_exhausted
if quota_exhausted:
return False, {"error": "quota_exhausted_early_abort"}
prompt = USER_PROMPT_TEMPLATE.format(
doc_id=doc_id,
canonical_title=fm.get("canonical_title") or doc_id,
collection=fm.get("collection") or "—",
page_count=fm.get("page_count") or "?",
classification=fm.get("highest_classification") or fm.get("classification") or "—",
body=body[:6000],
)
cmd = [
"claude", "-p",
"--model", model,
"--output-format", "json",
"--max-turns", "2",
"--system-prompt", SYSTEM_PROMPT,
"--",
prompt,
]
try:
proc = subprocess.run(
cmd,
capture_output=True,
text=True,
env={**os.environ},
check=False,
timeout=timeout_s,
)
except subprocess.TimeoutExpired:
return False, {"error": "timeout", "wall_seconds": timeout_s}
if proc.returncode != 0:
excerpt = (proc.stdout or "")[-500:]
if "monthly usage limit" in excerpt.lower() or "usage limit" in excerpt.lower():
quota_exhausted = True
return False, {"error": "quota_exhausted", "result_excerpt": excerpt}
return False, {"error": "rc_nonzero", "rc": proc.returncode, "stderr": (proc.stderr or "")[-500:]}
try:
cli = json.loads(proc.stdout)
except json.JSONDecodeError:
return False, {"error": "cli_json_parse", "raw": proc.stdout[-500:]}
result = cli.get("result", "")
if not result:
return False, {"error": "empty_result", "cli": cli}
# Try multiple strategies to extract the pitches robustly
payload = None
# Strategy 1: try parsing the whole result as JSON
try:
payload = json.loads(result.strip())
except json.JSONDecodeError:
pass
# Strategy 2: regex for the two fields directly (handles unescaped chars in values)
if not payload:
# Match: "pitch_pt_br": "<content until next ", " ... pitch_en or end of object>"
# We use a more flexible approach: split on the field names
pt_match = re.search(
r'"pitch_pt_br"\s*:\s*"((?:[^"\\]|\\.)*)"\s*,\s*"pitch_en"',
result, re.DOTALL
)
en_match = re.search(
r'"pitch_en"\s*:\s*"((?:[^"\\]|\\.)*)"\s*[},]',
result, re.DOTALL
)
if pt_match and en_match:
pt = pt_match.group(1).replace('\\"', '"').replace('\\n', '\n').replace('\\\\', '\\')
en = en_match.group(1).replace('\\"', '"').replace('\\n', '\n').replace('\\\\', '\\')
payload = {"pitch_pt_br": pt, "pitch_en": en}
# Strategy 3: balanced-brace extraction
if not payload:
try:
start = result.index("{")
depth = 0
for i in range(start, len(result)):
if result[i] == "{": depth += 1
elif result[i] == "}":
depth -= 1
if depth == 0:
try:
payload = json.loads(result[start : i + 1])
except json.JSONDecodeError:
pass
break
except ValueError:
pass
if not payload:
return False, {"error": "no_extractable_json", "result_excerpt": result[:600]}
if "pitch_pt_br" not in payload or "pitch_en" not in payload:
return False, {"error": "missing_fields", "payload": payload}
return True, {
"pitch_pt_br": payload["pitch_pt_br"].strip(),
"pitch_en": payload["pitch_en"].strip(),
"cost_usd": cli.get("total_cost_usd"),
"num_turns": cli.get("num_turns"),
"usage": cli.get("usage"),
}
def word_count(text: str) -> int:
return len([w for w in re.split(r"\s+", text) if w])
def process_doc(doc_id: str, force: bool, model: str, timeout_s: int) -> dict:
"""Generate + inject pitch for ONE doc."""
doc_path = WIKI_DOCS / f"{doc_id}.md"
if not doc_path.exists():
return {"doc_id": doc_id, "success": False, "error": "doc_not_found"}
fm, _, body = read_doc(doc_path)
if not force and fm.get("enthusiast_pitch_pt_br") and fm.get("enthusiast_pitch_en"):
return {"doc_id": doc_id, "success": True, "skipped": True, "reason": "already_has_pitch"}
t0 = time.time()
ok, result = call_claude(doc_id, fm, body, model, timeout_s)
wall = round(time.time() - t0, 1)
rec = {
"doc_id": doc_id,
"started_at": utc_iso(),
"wall_seconds": wall,
"model": model,
}
if not ok:
rec.update({"success": False, **result})
append_jsonl(PROGRESS_LOG, rec)
return rec
pt = result["pitch_pt_br"]
en = result["pitch_en"]
rec.update({
"success": True,
"pt_words": word_count(pt),
"en_words": word_count(en),
"cost_usd": result.get("cost_usd"),
})
# Validate word count
if not (40 <= word_count(pt) <= 200) or not (40 <= word_count(en) <= 200):
rec["warning"] = f"word_count_oob pt={word_count(pt)} en={word_count(en)}"
# Inject into frontmatter
fm["enthusiast_pitch_pt_br"] = pt
fm["enthusiast_pitch_en"] = en
fm["enthusiast_pitch_generated_at"] = utc_iso()
fm["enthusiast_pitch_model"] = model
write_doc(doc_path, fm, body)
append_jsonl(PROGRESS_LOG, rec)
return rec
def list_target_docs(only: str | None) -> list[str]:
if only:
return [only]
docs: list[str] = []
for p in sorted(WIKI_DOCS.glob("*.md")):
docs.append(p.stem)
return docs
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--workers", type=int, default=4)
ap.add_argument("--doc-id", default=None)
ap.add_argument("--force", action="store_true")
ap.add_argument("--model", default="sonnet", choices=["sonnet", "haiku"])
ap.add_argument("--timeout-per-doc", type=int, default=180)
ap.add_argument("--limit", type=int, default=None, help="Smoke test: process at most N")
args = ap.parse_args()
docs = list_target_docs(args.doc_id)
if args.limit:
docs = docs[: args.limit]
if not args.force:
# Skip docs already done
keep: list[str] = []
for d in docs:
fm, _, _ = read_doc(WIKI_DOCS / f"{d}.md")
if not (fm.get("enthusiast_pitch_pt_br") and fm.get("enthusiast_pitch_en")):
keep.append(d)
skipped = len(docs) - len(keep)
docs = keep
else:
skipped = 0
print(f"=" * 70)
print(f" ENTHUSIAST PITCH GENERATOR — {len(docs)} docs queued, {skipped} already done")
print(f" workers: {args.workers} · model: {args.model} · timeout: {args.timeout_per_doc}s/doc")
print(f" started: {utc_iso()}")
print(f"=" * 70)
sys.stdout.flush()
t0 = time.time()
ok = err = 0
total_cost = 0.0
with ThreadPoolExecutor(max_workers=args.workers) as ex:
futures = {ex.submit(process_doc, d, args.force, args.model, args.timeout_per_doc): d for d in docs}
for fut in as_completed(futures):
doc_id = futures[fut]
try:
r = fut.result()
except Exception as e:
r = {"doc_id": doc_id, "success": False, "exception": str(e)}
if r.get("success"):
ok += 1
total_cost += r.get("cost_usd") or 0
marker = "⊘" if r.get("skipped") else "✓"
wc = f"pt={r.get('pt_words','?')}w en={r.get('en_words','?')}w" if not r.get("skipped") else "(cached)"
print(f" [{ok+err}/{len(docs)}] {marker} {doc_id[:55]} · {wc} · ${r.get('cost_usd') or 0:.3f}")
else:
err += 1
marker = "💸" if r.get("error") == "quota_exhausted" else "✗"
print(f" [{ok+err}/{len(docs)}] {marker} {doc_id[:55]} · {r.get('error')}")
sys.stdout.flush()
if quota_exhausted:
# Cancel pending — early abort
for f in futures:
if not f.done():
f.cancel()
print("\n ⚠ QUOTA EXHAUSTED — aborting. Re-run later.")
break
print(f"\n{'=' * 70}")
print(f" DONE — {ok}/{len(docs)} succeeded · ${total_cost:.2f} · {round(time.time() - t0, 1)}s")
print(f"{'=' * 70}")
if __name__ == "__main__":
main()

79
scripts/99-finalize-pipeline.sh Executable file
View file

@ -0,0 +1,79 @@
#!/usr/bin/env bash
#
# 99-finalize-pipeline.sh — Encadeia fases 3-retry → 4 → 4.8 → 5 → 6 → 7 → 8 → 9
# após o término da Fase 3 (vision Haiku).
#
# Cada fase é idempotente: re-rodar é seguro.
#
# Log único em /tmp/ufo-finalize.log com prefixo de fase, append-only.
set -uo pipefail
ROOT="/Users/guto/ufo"
LOG="/tmp/ufo-finalize.log"
PY="python3"
cd "$ROOT" || exit 1
phase() {
local name="$1"; shift
echo "" | tee -a "$LOG"
echo "================================================================" | tee -a "$LOG"
echo "=== $(date -u +%Y-%m-%dT%H:%M:%SZ) — $name" | tee -a "$LOG"
echo "================================================================" | tee -a "$LOG"
"$@" 2>&1 | tee -a "$LOG"
local rc=${PIPESTATUS[0]}
echo "=== rc=$rc" | tee -a "$LOG"
return $rc
}
echo "" >> "$LOG"
echo "================================================================" >> "$LOG"
echo "==== FINALIZE PIPELINE STARTED $(date -u +%Y-%m-%dT%H:%M:%SZ) ====" >> "$LOG"
echo "================================================================" >> "$LOG"
# --- Phase 3 pass3 — last safety net (idempotent, processes only failures) ---
phase "Phase 3 pass3 (final retry)" \
$PY scripts/02-vision-page.py --all --workers 3 || true
# --- Phase 4 — Aggregate pages into document.md ---
phase "Phase 4 — build documents" \
$PY scripts/14-build-document-md.py || true
# --- Phase 4.8 retry — table CSV extraction (one had failed JSON parse) ---
phase "Phase 4.8 — retry remaining table CSVs" \
$PY scripts/16-extract-table-csv.py || true
# --- Phase 5 — Entity dedup / upsert ---
phase "Phase 5 — entity dedup" \
$PY scripts/03-dedup-entities.py || true
# --- Phase 7 — Crop bboxes (needs page.md but not enrichment) ---
phase "Phase 7 — crop bboxes" \
$PY scripts/05-crop-bboxes.py || true
# --- Phase 8 — Graph export (after entity stubs exist) ---
phase "Phase 8 — graph export" \
$PY scripts/06-graph-export.py || true
# --- Phase 6 — Enrichment (heaviest, runs after dedup creates entity stubs) ---
phase "Phase 6 — enrichment (deep tier only, 3 workers)" \
$PY scripts/17-enrich-entities.py --all --tier deep --workers 3 || true
# --- Phase 9 — Lint (LAST: rebuilds mentioned_in[] after enrichment) ---
phase "Phase 9 — lint + backlink rebuild" \
$PY scripts/04-lint.py || true
# --- Final stats ---
echo "" | tee -a "$LOG"
echo "================================================================" | tee -a "$LOG"
echo "==== FINALIZE PIPELINE FINISHED $(date -u +%Y-%m-%dT%H:%M:%SZ) ====" | tee -a "$LOG"
echo "================================================================" | tee -a "$LOG"
PAGES=$(find "$ROOT/wiki/pages" -name "p*.md" 2>/dev/null | wc -l | tr -d ' ')
DOCS=$(ls "$ROOT/wiki/documents/" 2>/dev/null | wc -l | tr -d ' ')
ENTITIES=$(find "$ROOT/wiki/entities" -name "*.md" 2>/dev/null | wc -l | tr -d ' ')
ENRICHED=$(grep -l "enrichment_status: deep\|enrichment_status: shallow" "$ROOT/wiki/entities/"*/*.md 2>/dev/null | wc -l | tr -d ' ')
TABLES=$(ls "$ROOT/wiki/tables/" 2>/dev/null | wc -l | tr -d ' ')
echo "pages: $PAGES · documents: $DOCS · entities: $ENTITIES (enriched: $ENRICHED) · tables: $TABLES" | tee -a "$LOG"

50
scripts/99-finalize-resume.sh Executable file
View file

@ -0,0 +1,50 @@
#!/usr/bin/env bash
#
# 99-finalize-resume.sh — Resume pipeline from Phase 6 (deep enrichment) + Phase 9 (lint).
# Used after killing the over-broad shallow enrichment.
#
# Phases 3-5, 7, 8 already completed in previous run.
set -uo pipefail
ROOT="/Users/guto/ufo"
LOG="/tmp/ufo-finalize.log"
PY="python3"
cd "$ROOT" || exit 1
phase() {
local name="$1"; shift
echo "" | tee -a "$LOG"
echo "================================================================" | tee -a "$LOG"
echo "=== $(date -u +%Y-%m-%dT%H:%M:%SZ) — $name" | tee -a "$LOG"
echo "================================================================" | tee -a "$LOG"
"$@" 2>&1 | tee -a "$LOG"
local rc=${PIPESTATUS[0]}
echo "=== rc=$rc" | tee -a "$LOG"
return $rc
}
echo "" >> "$LOG"
echo "================================================================" >> "$LOG"
echo "==== RESUME PIPELINE (deep enrichment + lint) $(date -u +%Y-%m-%dT%H:%M:%SZ) ====" >> "$LOG"
echo "================================================================" >> "$LOG"
# --- Phase 6 (deep tier only) — 1,107 entities, ~3h, ~$55 ---
phase "Phase 6 — enrichment (deep tier only, 1107 entities, 3 workers)" \
$PY scripts/17-enrich-entities.py --all --tier deep --workers 3 || true
# --- Phase 9 — Lint (rebuilds mentioned_in[]) ---
phase "Phase 9 — lint + backlink rebuild" \
$PY scripts/04-lint.py || true
echo "" | tee -a "$LOG"
echo "================================================================" | tee -a "$LOG"
echo "==== RESUME PIPELINE FINISHED $(date -u +%Y-%m-%dT%H:%M:%SZ) ====" | tee -a "$LOG"
echo "================================================================" | tee -a "$LOG"
PAGES=$(find "$ROOT/wiki/pages" -name "p*.md" 2>/dev/null | wc -l | tr -d ' ')
ENTITIES=$(find "$ROOT/wiki/entities" -name "*.md" 2>/dev/null | wc -l | tr -d ' ')
ENRICHED_DEEP=$(grep -l "enrichment_status: deep" "$ROOT/wiki/entities/"*/*.md 2>/dev/null | wc -l | tr -d ' ')
echo "pages: $PAGES · entities: $ENTITIES · enriched (deep): $ENRICHED_DEEP" | tee -a "$LOG"

1985
scripts/gen_all_chunks_doc65.py Normal file
View file

File diff suppressed because it is too large Load diff

229
scripts/gen_chunks_doc65_suba.py Normal file
View file

@ -0,0 +1,229 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Generate all chunk data for doc-65-hs1-834228961-62-hq-83894-sub-a
Based on direct visual analysis of all 89 pages.
Writes chunks/, images/ crops, _index.json, document.md
"""
import json, os
from datetime import datetime, timezone
from pathlib import Path
from PIL import Image as PILImage
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-sub-a"
DOC_TITLE = "FBI HQ 62-HQ-83894 Sub A — Flying Saucers / UAP Investigation File"
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
OUT_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
for d in [CHUNKS_DIR, IMAGES_DIR, TABLES_DIR]:
d.mkdir(parents=True, exist_ok=True)
# Ordered list of PNG filenames (sorted = logical page order)
import re
all_pngs = sorted(PNG_DIR.glob("p-*.png"), key=lambda p: int(re.search(r'p-(\d+)', p.name).group(1)))
TOTAL_PAGES = len(all_pngs)
print(f"Total pages: {TOTAL_PAGES}")
def mk_chunk(order_in_page, ctype, content_en, content_pt_br,
x, y, w, h, cls=None, fmt=None, cross="self_contained",
ocr_conf=0.75, redaction_code=None, image_type=None,
ufo=False, ufo_type=None, ufo_rat=None,
img_desc_en=None, img_desc_pt=None, extracted_text=None):
return {
"order_in_page": order_in_page,
"type": ctype,
"content_en": content_en,
"content_pt_br": content_pt_br,
"bbox": {"x": x, "y": y, "w": w, "h": h},
"classification": cls,
"formatting": fmt or [],
"cross_page_hint": cross,
"ocr_confidence": ocr_conf,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": image_type,
"ufo_anomaly_detected": ufo,
"ufo_anomaly_type": ufo_type,
"ufo_anomaly_rationale": ufo_rat,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": img_desc_en,
"image_description_pt_br": img_desc_pt,
"extracted_text": extracted_text,
}
# Pages data: list of dicts {page_number, png_filename, chunks:[...]}
pages_data = []
def add_page(png_path, chunks):
page_number = all_pngs.index(png_path) + 1
pages_data.append({
"page_number": page_number,
"png_path": str(png_path),
"png_filename": png_path.name,
"chunks": chunks,
})
# ============================================================
# PAGE 1: p-000.png — Newspaper clipping, Washington Star
# ============================================================
p = all_pngs[0]
add_page(p, [
mk_chunk(1,"image","Newspaper clipping: 'AIR FORCE FINDS FLYING SAUCERS' — photograph of Jonathan E. Caldwell's 'Gray Goose' helicopter with large disc rotor, described as looking like a flying saucer. Caption: 'This is Jonathan E. Caldwell's Gray Goose helicopter pictured before it made a near-disastrous test flight of about a minute in Washington nearly 6 years ago.'",
"Recorte de jornal: 'A FORÇA AÉREA ENCONTRA DISCOS VOADORES' — fotografia do helicóptero 'Gray Goose' de Jonathan E. Caldwell com grande rotor de disco, descrito como parecendo um disco voador.",
0.05,0.04,0.88,0.55, image_type="newspaper_clipping",
ufo=True, ufo_type="craft_description",
ufo_rat="Newspaper article about Air Force finding 'flying saucers' — actually Caldwell helicopter resembling a disc",
img_desc_en="Black and white newspaper photo showing a helicopter with a large circular disc rotor, resembling a flying saucer shape. The caption below the image reads: 'AIR FORCE FINDS FLYING SAUCERS — This is Jonathan E. Caldwell's Gray Goose helicopter pictured before it made a near-disastrous test flight of about a minute in Washington nearly 6 years ago.'",
img_desc_pt="Fotografia de jornal em preto e branco mostrando um helicóptero com grande rotor circular em forma de disco, parecendo um disco voador. A legenda abaixo da imagem diz: 'A FORÇA AÉREA ENCONTRA DISCOS VOADORES — Este é o helicóptero Gray Goose de Jonathan E. Caldwell fotografado antes de um voo de teste quase desastroso de cerca de um minuto em Washington há quase 6 anos.'",
extracted_text="AIR FORCE FINDS 'FLYING SAUCERS'"),
mk_chunk(2,"caption","Washington Star\nPage A 18",
"Washington Star\nPágina A 18",
0.3,0.85,0.4,0.06, ocr_conf=0.85),
])
# ============================================================
# PAGE 2: p-001.png — FBI folder cover
# ============================================================
p = all_pngs[1]
add_page(p, [
mk_chunk(1,"stamp","Declassification authority derived from FBI Automatic Declassification Guide, issued May 24, 2007.",
"Autoridade de desclassificação derivada do Guia de Desclassificação Automática do FBI, emitido em 24 de maio de 2007.",
0.6,0.01,0.38,0.07, ocr_conf=0.9,
img_desc_en=None, extracted_text="Declassification authority derived from FBI Automatic Declassification Guide, issued May 24, 2007."),
mk_chunk(2,"letterhead","U.S. Department of Justice\nFBI — Federal Bureau of Investigation\nHQ — CENTRAL RECORDS CENTER\nHEADQUARTERS",
"Departamento de Justiça dos EUA\nFBI — Departamento Federal de Investigação\nSEDE — CENTRO DE REGISTROS CENTRAIS\nQUARTEL-GENERAL",
0.15,0.08,0.7,0.25, fmt=["bold"], ocr_conf=0.8),
mk_chunk(3,"reference_line","File No.: 62-83894-A\nBarcode: 8/11/724151",
"Número do Arquivo: 62-83894-A\nCódigo de Barras: 8/11/724151",
0.05,0.1,0.25,0.2, ocr_conf=0.7),
mk_chunk(4,"body_paragraph","Field Office Criminal Investigative and Administrative Files",
"Arquivos de Investigação Criminal e Administrativos do Escritório de Campo",
0.15,0.5,0.7,0.08, fmt=["bold"], ocr_conf=0.85),
mk_chunk(5,"form_field","Armed and Dangerous ___ FOIPA ___\nDO NOT DESTROY ___ NCIC ___\nELSUR ___ OCIS ___\nEscape Risk ___ Suicidal ___\nFinancial Privacy Act ___ Other ___\nSee also Nos. ___",
"Armado e Perigoso ___ FOIPA ___\nNÃO DESTRUIR ___ NCIC ___\nELSUR ___ OCIS ___\nRisco de Fuga ___ Suicida ___\nLei de Privacidade Financeira ___ Outro ___\nVer também Nrs. ___",
0.05,0.62,0.9,0.25, ocr_conf=0.8),
mk_chunk(6,"handwritten_note","62-83894-A [written on right side rotated 90°]\n1-OPEN [written on right side]",
"62-83894-A [escrito à direita rotacionado 90°]\n1-ABERTO [escrito à direita]",
0.88,0.15,0.1,0.6, ocr_conf=0.7),
])
# ============================================================
# PAGE 3: p-002.png — Flying Sauter Photo article, Detroit Press
# ============================================================
p = all_pngs[2]
add_page(p, [
mk_chunk(1,"image","Newspaper clipping: 'Flying Sauter Photo Ain't What It Used to Be---Joe' by Charles Manos, Grand Blanc, May 30. Article about Joe Perry's flying saucer photo whose color has faded.",
"Recorte de jornal: 'A Foto do Prato Voador Não É Mais o Que Era---Joe' por Charles Manos. Artigo sobre a foto do disco voador de Joe Perry cuja cor desbotou.",
0.0,0.0,0.55,0.65, image_type="newspaper_clipping",
ufo=True, ufo_type="sighting_report",
ufo_rat="Article discusses flying saucer photograph taken by Joseph Perry near Grand Blanc, Michigan",
img_desc_en="Large newspaper clipping with bold headline 'Flying Sauter Photo Ain't What It Used to Be---Joe' by Charles Manos. Article about a flying saucer photograph taken by Joe Perry that has since faded.",
img_desc_pt="Grande recorte de jornal com título em negrito 'A Foto do Prato Voador Não É Mais o Que Era---Joe' por Charles Manos. Artigo sobre fotografia de disco voador tirada por Joe Perry que desbotou.",
extracted_text="Flying Sauter Photo Ain't What It Used to Be---Joe"),
mk_chunk(2,"form_field","Distribution list (right side): Mr. Tolson, Mr. Mohr, Mr. Parsons, Mr. Belmont, Mr. Callahan, Mr. McGuire, Mr. Rosen, Mr. Tamm, Mr. Trotter, Mr. W.G. Sullivan, Tele Room, Mr. Ingram, Miss Gandy",
"Lista de distribuição (lado direito): Sr. Tolson, Sr. Mohr, Sr. Parsons, Sr. Belmont, Sr. Callahan, Sr. McGuire, Sr. Rosen, Sr. Tamm, Sr. Trotter, Sr. W.G. Sullivan, Sala Tele., Sr. Ingram, Srta. Gandy",
0.68,0.0,0.3,0.35, ocr_conf=0.7),
mk_chunk(3,"letterhead","DETROIT DIVISION\nDetroit, Mich.\n( ) Detroit Free Press\nEditor: Lee Hills\n( ) Detroit News\nEditor: Martin S. Hayden\n( ) Detroit Times\nEditor: John C. Manning",
"DIVISÃO DE DETROIT\nDetroit, Mich.\n( ) Detroit Free Press\nEditor: Lee Hills\n( ) Detroit News\nEditor: Martin S. Hayden\n( ) Detroit Times\nEditor: John C. Manning",
0.55,0.35,0.43,0.25, ocr_conf=0.75),
mk_chunk(4,"stamp","Date: 5-25-60\nIndexed: 2\nFile: 2\nTitle or Case: UNIDENTIFIED FLYING OBJECT; JOSEPH PERRY, GRAND BLANC, MICHIGAN — COMPLAINANT\n(Defile 65-2477-105)",
"Data: 5-25-60\nIndexado: 2\nArquivo: 2\nTítulo ou Caso: OBJETO VOADOR NÃO IDENTIFICADO; JOSEPH PERRY, GRAND BLANC, MICHIGAN — RECLAMANTE\n(Defile 65-2477-105)",
0.55,0.6,0.43,0.28, ocr_conf=0.75),
mk_chunk(5,"stamp","REC 41 62-83894-A\nNOT RECORDED\n46 JUN 8 1960",
"REC 41 62-83894-A\nNÃO REGISTRADO\n46 JUN 8 1960",
0.55,0.87,0.42,0.1, ocr_conf=0.8),
mk_chunk(6,"footer","5 9JUN7 1960 417",
"5 9JUN7 1960 417",
0.0,0.95,0.2,0.04, ocr_conf=0.7),
])
# ============================================================
# PAGE 4: p-003.png — "3 Objects Trailed Plane" clipping
# ============================================================
p = all_pngs[3]
add_page(p, [
mk_chunk(1,"header","Central Research Section\nFile 62-P3894 5-gm",
"Seção Central de Pesquisa\nArquivo 62-P3894 5-gm",
0.0,0.0,1.0,0.08, ocr_conf=0.75),
mk_chunk(2,"form_field","Distribution list: Tolson, Belmont, Ladd, McGuire, Mohr, Parsons, Rosen, Tamm, Trotter, Holloman, Gandy",
"Lista de distribuição: Tolson, Belmont, Ladd, McGuire, Mohr, Parsons, Rosen, Tamm, Trotter, Holloman, Gandy",
0.72,0.0,0.27,0.35, ocr_conf=0.7),
mk_chunk(3,"image","Newspaper clipping: '3 Objects Trailed Plane 45 Minutes, Pilot Says' — DETROIT, Feb. 23 (AP) — Pilot of American Airlines DC8 reported three mysterious objects appeared to accompany his plane on flight from Newark, N.Y. Capt. Peter Killian, co-pilot John Dee of Nyack, N.Y. reported three bright objects near the horizon for 45 minutes, flying between Philipsburg, PA at 8:45 p.m. The objects were also visible to 35 passengers and crew.",
"Recorte de jornal: '3 Objetos Seguiram Avião por 45 Minutos, Diz Piloto' — DETROIT, 23 de fev. (AP) — Piloto da American Airlines DC8 relatou três objetos misteriosos acompanhando seu avião em voo de Newark, N.Y. O Capitão Peter Killian e o co-piloto John Dee relataram três objetos brilhantes perto do horizonte por 45 minutos.",
0.05,0.08,0.6,0.35, image_type="newspaper_clipping",
ufo=True, ufo_type="sighting_report",
ufo_rat="American Airlines pilot and crew observed 3 unidentified objects following their plane for 45 minutes",
extracted_text="3 'Objects' Trailed Plane 45 Minutes, Pilot Says"),
mk_chunk(4,"stamp","162-83894 — A\nNOT RECORDED\n TAP MAR 3 1959",
"162-83894 — A\nNÃO REGISTRADO\nTAP MAR 3 1959",
0.55,0.38,0.4,0.1, ocr_conf=0.75),
mk_chunk(5,"form_field","Distribution list (right side): The Washington Post and Times Herald, The Washington Daily News, The Evening Star, New York Herald Tribune, New York Journal-American, New York Mirror, New York Daily News, New York Post, The New York Times, The Worker, The New Leader, The Wall Street Journal, Date 3/4/59",
"Lista de distribuição (lado direito): The Washington Post and Times Herald, The Washington Daily News, The Evening Star, New York Herald Tribune, New York Journal-American, New York Mirror, New York Daily News, New York Post, The New York Times, The Worker, The New Leader, The Wall Street Journal, Data 3/4/59",
0.6,0.45,0.38,0.45, ocr_conf=0.65),
mk_chunk(6,"footer","5 7MAR 4 1959 417",
"5 7MAR 4 1959 417",
0.0,0.95,0.2,0.04),
])
# ============================================================
# PAGE 5: p-004.png — Flying Saucers telegram, Aug 1958
# ============================================================
p = all_pngs[4]
add_page(p, [
mk_chunk(1,"image","Newspaper/wire clipping pasted on blank page — headline 'FLYING SAUCERS'. Text: 'A group of unidentified flying objects clustered together for more than an hour near here last night. A dozen broke apart and disappeared. 9 witnesses said today. The aerial research phenomena organization filter center showed a total of nine persons reported seeing the phenomenon.'",
"Recorte de jornal/telegrama colado em página em branco — título 'DISCOS VOADORES'. Texto: 'Um grupo de objetos voadores não identificados ficou agrupado por mais de uma hora perto daqui na noite passada.'",
0.05,0.2,0.85,0.3, image_type="newspaper_clipping",
ufo=True, ufo_type="sighting_report",
ufo_rat="Wire report of multiple UFO sightings by 9 witnesses, with objects clustered then breaking apart",
extracted_text="FLYING SAUCERS"),
mk_chunk(2,"handwritten_note","Flying Saucers\nfile 62-83894\n1|62-13894-A",
"Discos Voadores\narquivo 62-83894\n1|62-13894-A",
0.35,0.6,0.45,0.2, ocr_conf=0.65),
mk_chunk(3,"stamp","NOT RECORDED\n1 AUG 12 1958",
"NÃO REGISTRADO\n1 AGO 12 1958",
0.55,0.8,0.4,0.1, ocr_conf=0.8),
mk_chunk(4,"footer","59AUG 12 1958",
"59AGO 12 1958",
0.0,0.93,0.3,0.05),
mk_chunk(5,"handwritten_note","Bram [signature]",
"Bram [assinatura]",
0.78,0.78,0.2,0.05),
])
# ============================================================
# PAGE 6: p-005.png — "Flying Discs Show Sign of Guidance, Jung Says"
# ============================================================
p = all_pngs[5]
add_page(p, [
mk_chunk(1,"header","0-19 (Rev. 3-7-58)",
"0-19 (Rev. 3-7-58)",
0.0,0.0,0.15,0.03, ocr_conf=0.7),
mk_chunk(2,"form_field","Distribution: Tolson, Boardman, Belmont, Mohr, Parsons, Rosen, Tamm, Trotter, Clayton, Nease, Holloman, Gandy",
"Distribuição: Tolson, Boardman, Belmont, Mohr, Parsons, Rosen, Tamm, Trotter, Clayton, Nease, Holloman, Gandy",
0.65,0.0,0.33,0.35, ocr_conf=0.7),
mk_chunk(3,"image","Newspaper clipping: 'Flying Discs Show Sign of Guidance, Jung Says' — ALAMOGORDO, N. Mex., July 29 — Dr. Carl Jung, Berlin psychologist, says in a report released yesterday that flying saucers are real and 'show definite signs of intelligent guidance.' Article discusses Jung's research on UFO sightings since 1944, Air Force investigations, and report from Research Center.",
"Recorte de jornal: 'Discos Voadores Mostram Sinais de Orientação, Diz Jung' — ALAMOGORDO, N. Mex. — O Dr. Carl Jung, psicólogo berlinense, diz em relatório que os discos voadores são reais e mostram 'sinais definidos de orientação inteligente.'",
0.0,0.28,0.55,0.55, image_type="newspaper_clipping",
ufo=True, ufo_type="official_report",
ufo_rat="Carl Jung's official report claiming flying discs show signs of intelligent guidance, referencing Air Force investigation",
extracted_text="Flying Discs Show Sign of Guidance, Jung Says"),
mk_chunk(4,"form_field","Distribution (right): Wash. Post and Times Herald, Wash. Star, N.Y. Herald Tribune, N.Y. Mirror, N.Y. Journal-American, N.Y. Daily News, N.Y. Times, Daily Worker, The Worker, New Leader",
"Distribuição (direita): Wash. Post and Times Herald, Wash. Star, N.Y. Herald Tribune, N.Y. Mirror, N.Y. Journal-American, N.Y. Daily News, N.Y. Times, Daily Worker, The Worker, New Leader",
0.62,0.55,0.36,0.3, ocr_conf=0.65),
mk_chunk(5,"stamp","62-83894\nNOT RECORDED\n117 AUG 1 1958\nDate 7-29-58",
"62-83894\nNÃO REGISTRADO\n117 AGO 1 1958\nData 7-29-58",
0.55,0.82,0.42,0.12, ocr_conf=0.8),
mk_chunk(6,"footer","67AUG1 1958",
"67AGO1 1958",
0.0,0.95,0.15,0.04),
mk_chunk(7,"handwritten_note","BRK [initials]",
"BRK [iniciais]",
0.55,0.35,0.12,0.04),
])
print("Pages 1-6 defined. Continuing...")

1373
scripts/rebuild_d48_ocr_only.py Normal file
View file

File diff suppressed because it is too large Load diff

511
scripts/rebuild_d49.py Normal file
View file

@ -0,0 +1,511 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Rebuild DOW-UAP-D49 launch summary into harness-assemblable structure.
Processes all 78 PNG pages, writes chunks, _index.json, and document.md.
"""
import os
import sys
import json
import re
import base64
import datetime
import time
from pathlib import Path
from PIL import Image
import anthropic
DOC_ID = "dow-uap-d49-launch-summary-february-2000"
DOC_TITLE = "Vandenberg AFB Launch Summary 1958–2000"
PNG_DIR = Path("/Users/guto/ufo/processing/png") / DOC_ID
OCR_DIR = Path("/Users/guto/ufo/processing/ocr") / DOC_ID
OUT_DIR = Path("/Users/guto/ufo/raw") / DOC_ID
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
# All PNG pages sorted
PNG_PAGES = sorted([f for f in os.listdir(PNG_DIR) if f.endswith(".png")])
TOTAL_PAGES = len(PNG_PAGES)
client = anthropic.Anthropic()
def read_ocr(page_stem: str) -> str:
"""Read OCR text for a page stem like p-001."""
ocr_path = OCR_DIR / (page_stem + ".txt")
if ocr_path.exists():
return ocr_path.read_text(encoding="utf-8", errors="replace")
return ""
def encode_image_b64(path: str) -> str:
with open(path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
def classify_page(ocr_text: str, page_num: int) -> str:
"""Heuristic page type classification."""
text = ocr_text.strip().lower()
if page_num == 1:
return "cover"
if "distribution list" in text:
return "distribution"
if "foreword" in text or "preface" in text:
return "foreword"
if "glossary" in text and len(text) < 2000:
return "glossary"
if "annual launch summary" in text and ("chart" in text or "launch vehicle" in text or "launch agency" in text):
return "summary_table"
if "launch facility guide" in text:
return "facility_guide"
if re.search(r'\bseq\b.*\bdate\b.*\bnickname\b', text, re.IGNORECASE) or re.search(r'\d+\.\s+\d{1,2}\s+\w+\s+\d{2}', text):
return "chronology"
if "table of contents" in text or "contents" in text.split("\n")[0]:
return "toc"
return "body"
def determine_chunk_type(content: str, page_type: str) -> str:
"""Map page content to chunk type."""
lower = content.lower().strip()
if page_type == "cover":
return "letterhead"
if page_type in ("summary_table", "chronology"):
return "table_marker"
if page_type == "glossary":
return "body_text"
if page_type == "foreword":
return "body_text"
if page_type == "distribution":
return "body_text"
if page_type == "facility_guide":
return "body_text"
if page_type == "toc":
return "body_text"
# Check for headings
lines = content.strip().split("\n")
if len(lines) <= 3 and content.strip().isupper():
return "section_header"
return "body_text"
def build_page_chunks(page_num: int, page_stem: str, ocr_text: str) -> list:
"""Build chunks for a single page from OCR text."""
png_path = str(PNG_DIR / (page_stem + ".png"))
page_type = classify_page(ocr_text, page_num)
lines = ocr_text.strip().split("\n") if ocr_text.strip() else []
chunks = []
if not ocr_text.strip():
# Image-only page (p-000)
chunks.append({
"type": "image",
"page_type": page_type,
"content_raw": "",
"content_en": "[Cover image — Vandenberg AFB Launch Summary 1958–2000]",
"content_pt_br": "[Imagem da capa — Resumo de Lançamentos da Base Aérea Vandenberg 1958–2000]",
"order_in_page": 1,
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"source_png": f"../../processing/png/{DOC_ID}/{page_stem}.png",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
})
return chunks
# Identify logical sections within the page
# For this document, most pages are single logical blocks
# Special handling: pages with a heading + content body
heading_lines = []
body_lines = []
in_heading = True
for i, line in enumerate(lines):
stripped = line.strip()
# Skip empty header lines
if not stripped and in_heading and not heading_lines:
continue
# Detect heading transition: short uppercase lines at top
if in_heading:
if stripped and len(stripped) < 80 and (stripped.isupper() or re.match(r'^[A-Z\s\-\./:,0-9]+$', stripped)):
heading_lines.append(stripped)
else:
in_heading = False
if stripped:
body_lines.append(line)
else:
body_lines.append(line)
# For cover, use all lines as single chunk
if page_type == "cover":
content = "\n".join(line.strip() for line in lines if line.strip())
chunks.append({
"type": "letterhead",
"page_type": page_type,
"content_raw": content,
"content_en": content,
"content_pt_br": translate_to_ptbr_simple(content, page_type),
"order_in_page": 1,
"bbox": {"x": 0.05, "y": 0.1, "w": 0.9, "h": 0.8},
"source_png": f"../../processing/png/{DOC_ID}/{page_stem}.png",
"ocr_confidence": 0.92,
"ocr_source_lines": list(range(1, len(lines)+1)),
})
return chunks
order = 1
# Emit heading chunk if distinct
if heading_lines and body_lines:
heading_content = "\n".join(heading_lines)
chunks.append({
"type": "section_header",
"page_type": page_type,
"content_raw": heading_content,
"content_en": heading_content,
"content_pt_br": translate_to_ptbr_simple(heading_content, "section_header"),
"order_in_page": order,
"bbox": {"x": 0.05, "y": 0.02, "w": 0.9, "h": 0.12},
"source_png": f"../../processing/png/{DOC_ID}/{page_stem}.png",
"ocr_confidence": 0.93,
"ocr_source_lines": list(range(1, len(heading_lines)+1)),
"formatting": ["bold", "all_caps"],
})
order += 1
body_content = "\n".join(body_lines)
body_type = "table_marker" if page_type in ("summary_table", "chronology") else "body_text"
chunks.append({
"type": body_type,
"page_type": page_type,
"content_raw": body_content,
"content_en": body_content,
"content_pt_br": translate_to_ptbr_simple(body_content, page_type),
"order_in_page": order,
"bbox": {"x": 0.02, "y": 0.14, "w": 0.98, "h": 0.84},
"source_png": f"../../processing/png/{DOC_ID}/{page_stem}.png",
"ocr_confidence": 0.88,
"ocr_source_lines": list(range(len(heading_lines)+1, len(lines)+1)),
})
else:
# Single chunk for entire page
content = "\n".join(line for line in lines if True) # preserve all lines
body_type = determine_chunk_type(content, page_type)
if page_type in ("summary_table", "chronology"):
body_type = "table_marker"
chunks.append({
"type": body_type,
"page_type": page_type,
"content_raw": content,
"content_en": content,
"content_pt_br": translate_to_ptbr_simple(content, page_type),
"order_in_page": 1,
"bbox": {"x": 0.02, "y": 0.02, "w": 0.96, "h": 0.96},
"source_png": f"../../processing/png/{DOC_ID}/{page_stem}.png",
"ocr_confidence": 0.88,
"ocr_source_lines": list(range(1, len(lines)+1)),
})
return chunks
def translate_to_ptbr_simple(text: str, context: str) -> str:
"""Simple heuristic PT-BR translation for common document patterns.
For verbatim data (tables, codes, dates, numbers) returns text unchanged.
For known headers/labels adds translation.
"""
# For table/chronology data, return as-is (numeric data, codes, acronyms)
if context in ("summary_table", "chronology", "table_marker"):
return text # Data stays verbatim
# Map known English phrases to PT-BR
replacements = {
"FOREWORD": "PREFÁCIO",
"GLOSSARY": "GLOSSÁRIO",
"DISTRIBUTION LIST": "LISTA DE DISTRIBUIÇÃO",
"TABLE OF CONTENTS": "SUMÁRIO",
"ANNUAL LAUNCH SUMMARY BY BOOSTER": "RESUMO ANUAL DE LANÇAMENTOS POR VEÍCULO",
"ANNUAL LAUNCH SUMMARY BY COMMAND": "RESUMO ANUAL DE LANÇAMENTOS POR COMANDO",
"LAUNCH FACILITY GUIDE": "GUIA DE INSTALAÇÕES DE LANÇAMENTO",
"Office of History": "Escritório de História",
"As of": "Em",
"LAUNCH": "LANÇAMENTO",
"VEHICLE": "VEÍCULO",
"COMMAND": "COMANDO",
"PROGRAM": "PROGRAMA",
"SPACE": "ESPAÇO",
"TOTAL": "TOTAL",
"SUBTOTAL": "SUBTOTAL",
"Grand Total": "Total Geral",
"GRAND TOTAL": "TOTAL GERAL",
}
result = text
for en, pt in replacements.items():
result = result.replace(en, f"{en} / {pt}")
return result
def fmt_chunk_id(n: int) -> str:
return f"c{n:04d}"
def write_chunk_file(chunk_id: str, chunk: dict, page_num: int) -> None:
"""Write a single chunk .md file."""
path = CHUNKS_DIR / (chunk_id + ".md")
prev_chunk = chunk.get("prev_chunk", "null")
next_chunk = chunk.get("next_chunk", "null")
def yaml_val(v):
if v is None or v == "null":
return "null"
if isinstance(v, bool):
return str(v).lower()
if isinstance(v, (int, float)):
return str(v)
return f'"{v}"'
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0})
formatting = chunk.get("formatting", [])
fmt_str = "[" + ", ".join(f'"{f}"' for f in formatting) + "]" if formatting else "[]"
ocr_lines = chunk.get("ocr_source_lines", [])
if len(ocr_lines) > 10:
ocr_lines_str = f"[{ocr_lines[0]}, {ocr_lines[1]}, \"...\", {ocr_lines[-1]}]"
else:
ocr_lines_str = "[" + ", ".join(str(l) for l in ocr_lines) + "]"
related_image = chunk.get("related_image", "null")
related_table = chunk.get("related_table", "null")
image_type = chunk.get("image_type", "null")
content = f"""---
chunk_id: {chunk_id}
type: {chunk["type"]}
page: {page_num}
order_in_page: {chunk["order_in_page"]}
order_global: {chunk["order_global"]}
bbox: {{x: {bbox["x"]:.2f}, y: {bbox["y"]:.2f}, w: {bbox["w"]:.2f}, h: {bbox["h"]:.2f}}}
classification: null
formatting: {fmt_str}
cross_page_hint: self_contained
prev_chunk: {prev_chunk if prev_chunk != "null" else "null"}
next_chunk: {next_chunk if next_chunk != "null" else "null"}
related_image: {yaml_val(related_image) if related_image != "null" else "null"}
related_table: {yaml_val(related_table) if related_table != "null" else "null"}
ocr_confidence: {chunk.get("ocr_confidence", 0.88):.2f}
ocr_source_lines: {ocr_lines_str}
redaction_code: null
redaction_inferred_content_type: null
image_type: {yaml_val(image_type) if image_type and image_type != "null" else "null"}
ufo_anomaly_detected: false
cryptid_anomaly_detected: false
ufo_anomaly_type: null
ufo_anomaly_rationale: null
cryptid_anomaly_type: null
cryptid_anomaly_rationale: null
image_description_en: null
image_description_pt_br: null
extracted_text: null
source_png: {chunk["source_png"]}
---
**EN:** {chunk["content_en"]}
**PT-BR:** {chunk["content_pt_br"]}
"""
path.write_text(content, encoding="utf-8")
def main():
start_time = time.time()
# Ensure output dirs exist
CHUNKS_DIR.mkdir(parents=True, exist_ok=True)
IMAGES_DIR.mkdir(parents=True, exist_ok=True)
TABLES_DIR.mkdir(parents=True, exist_ok=True)
print(f"Processing {TOTAL_PAGES} pages...")
# Process all pages
all_pages_chunks = [] # list of (page_num, page_stem, [chunks])
for idx, png_file in enumerate(PNG_PAGES):
page_stem = png_file.replace(".png", "")
# Map to 1-based page number
page_num = idx + 1
ocr_text = read_ocr(page_stem)
chunks = build_page_chunks(page_num, page_stem, ocr_text)
all_pages_chunks.append((page_num, page_stem, chunks))
print(f" Page {page_num:3d}/{TOTAL_PAGES} ({page_stem}): {len(chunks)} chunk(s)")
# Globally number chunks
global_order = 0
all_chunks_flat = [] # list of (chunk_id, page_num, chunk_dict)
for page_num, page_stem, chunks in all_pages_chunks:
for chunk in chunks:
global_order += 1
chunk_id = fmt_chunk_id(global_order)
chunk["chunk_id"] = chunk_id
chunk["order_global"] = global_order
chunk["page_num"] = page_num
chunk["page_stem"] = page_stem
all_chunks_flat.append((chunk_id, page_num, chunk))
total_chunks = len(all_chunks_flat)
print(f"Total chunks: {total_chunks}")
# Set prev/next pointers
for i, (chunk_id, page_num, chunk) in enumerate(all_chunks_flat):
chunk["prev_chunk"] = all_chunks_flat[i-1][0] if i > 0 else "null"
chunk["next_chunk"] = all_chunks_flat[i+1][0] if i < total_chunks - 1 else "null"
# Write chunk files
print("Writing chunk files...")
for chunk_id, page_num, chunk in all_chunks_flat:
write_chunk_file(chunk_id, chunk, page_num)
# Build _index.json
print("Writing _index.json...")
index_chunks = []
for chunk_id, page_num, chunk in all_chunks_flat:
content_en = chunk["content_en"]
preview = content_en[:80].replace("\n", " ").strip()
index_chunks.append({
"chunk_id": chunk_id,
"type": chunk["type"],
"page": page_num,
"order_in_page": chunk["order_in_page"],
"order_global": chunk["order_global"],
"file": f"chunks/{chunk_id}.md",
"bbox": chunk["bbox"],
"preview": preview,
})
build_at = datetime.datetime.utcnow().isoformat() + "Z"
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": total_chunks,
"build_approach": "subagents",
"build_model": "claude-sonnet-4-6",
"build_at": build_at,
"chunks": index_chunks,
}
(OUT_DIR / "_index.json").write_text(
json.dumps(index, ensure_ascii=False, indent=2), encoding="utf-8"
)
# Count chunk types
type_histogram = {}
for _, _, chunk in all_chunks_flat:
t = chunk["type"]
type_histogram[t] = type_histogram.get(t, 0) + 1
# Count image chunks
image_chunks = [(cid, pn, ch) for cid, pn, ch in all_chunks_flat if ch["type"] == "image"]
n_images = len(image_chunks)
# Build document.md
print("Writing document.md...")
build_document_md(all_chunks_flat, all_pages_chunks, type_histogram, build_at, n_images)
elapsed = int(time.time() - start_time)
print(f"\nDone in {elapsed}s")
print(f"pages_done={TOTAL_PAGES}, chunks_total={total_chunks}, images_extracted={n_images}, tables_stitched=0, ufo_anomalies=0, cryptid_anomalies=0, wall_seconds={elapsed}")
def build_document_md(all_chunks_flat, all_pages_chunks, type_histogram, build_at, n_images):
"""Build the master document.md."""
total_chunks = len(all_chunks_flat)
histogram_yaml = "\n".join(f" {k}: {v}" for k, v in sorted(type_histogram.items()))
header = f"""---
schema_version: "0.2.0"
type: master_document
doc_id: {DOC_ID}
canonical_title: "{DOC_TITLE}"
total_pages: {TOTAL_PAGES}
total_chunks: {total_chunks}
chunk_types_histogram:
{histogram_yaml}
multi_page_tables: []
ufo_anomalies_flagged: []
cryptid_anomalies_flagged: []
build_approach: "subagents"
build_model: claude-sonnet-4-6
build_at: {build_at}
---
"""
# Group chunks by page
pages_dict = {}
for chunk_id, page_num, chunk in all_chunks_flat:
if page_num not in pages_dict:
pages_dict[page_num] = []
pages_dict[page_num].append((chunk_id, chunk))
body_parts = [header]
for page_num in sorted(pages_dict.keys()):
chunks_on_page = pages_dict[page_num]
body_parts.append(f"## Page {page_num}\n\n")
for chunk_id, chunk in chunks_on_page:
bbox = chunk["bbox"]
bbox_str = f"{bbox['x']:.2f}/{bbox['y']:.2f}/{bbox['w']:.2f}/{bbox['h']:.2f}"
body_parts.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->\n")
body_parts.append(f'<a id="{chunk_id}"></a>\n')
body_parts.append(f"### Chunk {chunk_id} — {chunk['type']} · p{page_num} · bbox: {bbox_str}\n\n")
content_en = chunk["content_en"]
content_pt = chunk["content_pt_br"]
# For table/chronology, wrap in code block for readability
if chunk["type"] == "table_marker":
body_parts.append(f"**EN:**\n\n```\n{content_en}\n```\n\n")
body_parts.append(f"**PT-BR:**\n\n```\n{content_pt}\n```\n\n")
elif chunk["type"] == "image":
body_parts.append(f"**EN:** {content_en}\n\n")
body_parts.append(f"**PT-BR:** {content_pt}\n\n")
related_img = chunk.get("related_image")
if related_img and related_img != "null":
body_parts.append(f"![chunk image](./images/{related_img})\n\n")
else:
body_parts.append(f"**EN:** {content_en}\n\n")
body_parts.append(f"**PT-BR:** {content_pt}\n\n")
# Metadata details block
meta = {
"chunk_id": chunk_id,
"type": chunk["type"],
"page": page_num,
"order_in_page": chunk["order_in_page"],
"order_global": chunk["order_global"],
"bbox": chunk["bbox"],
"classification": None,
"formatting": chunk.get("formatting", []),
"cross_page_hint": "self_contained",
"prev_chunk": chunk.get("prev_chunk"),
"next_chunk": chunk.get("next_chunk"),
"ocr_confidence": chunk.get("ocr_confidence", 0.88),
"ufo_anomaly_detected": False,
"cryptid_anomaly_detected": False,
}
meta_json = json.dumps(meta, ensure_ascii=False, indent=2)
body_parts.append(f"<details><summary>metadata</summary>\n\n```json\n{meta_json}\n```\n\n</details>\n\n---\n\n")
doc_content = "".join(body_parts)
(OUT_DIR / "document.md").write_text(doc_content, encoding="utf-8")
print(f"document.md written ({len(doc_content):,} bytes)")
if __name__ == "__main__":
main()

633
scripts/rebuild_doc255.py Normal file
View file

@ -0,0 +1,633 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Rebuild doc-255-413270-ufo-s-and-defense-what-should-we-prepare-for
into structured chunk files, _index.json, and document.md.
Uses `claude -p --model haiku` subprocess calls (OAuth via Max plan).
"""
import json
import os
import random
import re
import subprocess
import sys
import time
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from pathlib import Path
DOC_ID = "doc-255-413270-ufo-s-and-defense-what-should-we-prepare-for"
DOC_TITLE = "UFO's and Defense: What Should We Prepare For?"
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
OUT_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
MODEL = "haiku"
TOTAL_PAGES = 93
WORKERS = 4
TIMEOUT = 240 # seconds per page call
_print_lock = threading.Lock()
def safe_print(*args, **kwargs):
with _print_lock:
print(*args, **kwargs, flush=True)
PAGE_REBUILDER_PROMPT = '''You are a page-rebuilder agent. Analyze the scanned document page image and extract all content into structured chunks.
Document: {doc_title}
Page: {page_number} of {total_pages}
Doc ID: {doc_id}
STEP 1: Use the Read tool to view this PNG image:
{png_path}
STEP 2: Analyze every element on the page carefully.
STEP 3: Return ONE JSON object only (no markdown fence, no commentary):
{{
"page_number": {page_number},
"chunks": [
{{
"order_in_page": 1,
"type": "paragraph",
"content_en": "verbatim English text from page",
"content_pt_br": "tradução em português brasileiro",
"bbox": {{"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.85,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null,
"image_description_en": null,
"image_description_pt_br": null,
"extracted_text": null
}}
]
}}
CHUNK TYPES (use exactly one):
- letterhead: document header/letterhead
- classification_marking: classification marking (TOP SECRET, CUI, etc.)
- date_line: date field
- address_block: TO:/FROM:/distribution fields
- heading: section/chapter/subject heading
- paragraph: body text paragraph
- numbered_item: numbered list item
- bulleted_item: bullet list item
- table_marker: table content
- image: photograph, diagram, chart, sketch, map, graph
- caption: figure/image caption
- footer: page footer
- page_number: standalone page number
- signature: signature/signatory block
- redaction: blacked-out/redacted area
- stamp: official stamp or seal
- handwriting: handwritten annotation
- blank_area: empty area
- form_field: form field with label and value
- unknown: unidentifiable element
RULES:
1. Split content into logical chunks (one concept per chunk). A typical page has 3-15 chunks.
2. For image chunks: describe what you see in content_en and set image_type.
3. image_type options: photograph, diagram, chart, sketch, map, graph, seal, signature_image, other
4. bbox: normalized coordinates 0.0-1.0 (x=left, y=top, w=width, h=height)
5. content_en: verbatim text if text chunk; visual description if image chunk
6. content_pt_br: Brazilian Portuguese translation (NOT European Portuguese)
7. classification: null or the marking text (e.g. "CUI", "UNCLASSIFIED")
8. cross_page_hint: "self_contained" | "continues_to_next" | "continues_from_prev"
9. formatting: array from ["bold", "italic", "all_caps", "underline"]
10. If page is completely blank: ONE chunk of type "blank_area"
11. Preserve French text verbatim (document may contain French)
12. For redaction chunks: set redaction_code if visible (e.g. "(b)(1)")
13. ufo_anomaly_detected: true ONLY for image chunks showing actual UAP/anomalous phenomena
Output ONLY the JSON object. No preamble. No fence. No commentary.'''
IMAGE_ANALYST_PROMPT = '''You are an image analyst for a UAP/UFO declassified document.
STEP 1: Use the Read tool to view this cropped image:
{image_path}
STEP 2: Analyze it carefully.
STEP 3: Return ONE JSON object only (no markdown fence):
{{
"image_description_en": "detailed English description",
"image_description_pt_br": "descrição detalhada em português brasileiro",
"image_type": "photograph",
"extracted_text": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}}
image_type options: photograph, diagram, chart, sketch, map, graph, seal, signature_image, other
Set ufo_anomaly_detected=true only if the image shows an actual UAP/UFO or anomalous aerial phenomenon.
Set cryptid_anomaly_detected=true only if the image shows a cryptid or unknown creature.
extracted_text: any text visible inside the image (verbatim), or null.
Output ONLY the JSON object.'''
def extract_json(text: str) -> dict:
"""Extract JSON from claude CLI output."""
text = text.strip()
# Strip markdown fences if present
if text.startswith("```"):
text = re.sub(r"^```(?:json)?\s*", "", text)
text = re.sub(r"\s*```\s*$", "", text.rstrip())
# Find first { and matching }
start = text.find("{")
if start == -1:
raise ValueError(f"No JSON found in: {text[:200]}")
depth = 0
for i, c in enumerate(text[start:], start):
if c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
return json.loads(text[start:i + 1])
raise ValueError("Unclosed JSON in response")
def call_claude(prompt: str, allowed_tools: str = "Read", timeout: int = TIMEOUT) -> str:
"""Call claude -p CLI and return result text."""
cmd = [
"claude", "-p",
"--model", MODEL,
"--output-format", "json",
"--max-turns", "5",
"--allowedTools", allowed_tools,
"--add-dir", str(PNG_DIR),
"--add-dir", str(IMAGES_DIR),
"--",
prompt,
]
res = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
check=False,
env={**os.environ},
)
if res.returncode != 0:
raise RuntimeError(f"claude CLI rc={res.returncode}: {res.stderr[-1000:]}")
cli = json.loads(res.stdout)
if cli.get("is_error"):
raise RuntimeError(f"claude error: {cli.get('result', '')[:500]}")
return cli.get("result", "")
def process_page(page_num: int) -> dict:
"""Process a single page using claude -p CLI."""
png_path = PNG_DIR / f"p-{page_num:03d}.png"
if not png_path.exists():
safe_print(f" Page {page_num}: PNG missing — placeholder")
return {
"page_number": page_num,
"chunks": [{
"order_in_page": 1,
"type": "blank_area",
"content_en": f"[Page {page_num} — PNG not available]",
"content_pt_br": f"[Página {page_num} — PNG não disponível]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None, "formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [],
"redaction_code": None, "redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False, "ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False, "cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None,
"image_description_en": None, "image_description_pt_br": None, "extracted_text": None
}]
}
prompt = PAGE_REBUILDER_PROMPT.format(
doc_title=DOC_TITLE,
page_number=page_num,
total_pages=TOTAL_PAGES,
doc_id=DOC_ID,
png_path=str(png_path),
)
max_retries = 3
for attempt in range(1, max_retries + 1):
try:
result_text = call_claude(prompt, allowed_tools="Read")
data = extract_json(result_text)
data["page_number"] = page_num
# Validate chunks exist
if not isinstance(data.get("chunks"), list) or len(data["chunks"]) == 0:
raise ValueError("No chunks in response")
safe_print(f" Page {page_num}: {len(data['chunks'])} chunks")
return data
except (subprocess.TimeoutExpired,) as e:
safe_print(f" Page {page_num}: timeout attempt {attempt}/{max_retries}")
if attempt == max_retries:
break
time.sleep(10 * attempt)
except (RuntimeError, ValueError, json.JSONDecodeError) as e:
safe_print(f" Page {page_num}: error attempt {attempt}/{max_retries}: {str(e)[:100]}")
if attempt == max_retries:
break
backoff = 5 * attempt + random.uniform(0, 3)
time.sleep(backoff)
# Return fallback
safe_print(f" Page {page_num}: FALLBACK after {max_retries} attempts")
return {
"page_number": page_num,
"chunks": [{
"order_in_page": 1,
"type": "unknown",
"content_en": f"[Page {page_num} — content extraction failed after {max_retries} attempts]",
"content_pt_br": f"[Página {page_num} — extração de conteúdo falhou após {max_retries} tentativas]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None, "formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [],
"redaction_code": None, "redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False, "ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False, "cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None,
"image_description_en": None, "image_description_pt_br": None, "extracted_text": None
}]
}
def crop_image(chunk_id: str, png_path: Path, bbox: dict) -> object:
"""Crop image region from page PNG."""
from PIL import Image
cropped_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
try:
im = Image.open(png_path)
W, H = im.size
x = max(0.0, float(bbox.get("x", 0)))
y = max(0.0, float(bbox.get("y", 0)))
w = max(0.01, float(bbox.get("w", 1)))
h = max(0.01, float(bbox.get("h", 0.1)))
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
if right <= left or bottom <= top:
safe_print(f" Crop {chunk_id}: degenerate bbox {bbox}")
return None
cropped = im.crop((left, top, right, bottom))
cropped.save(str(cropped_path))
safe_print(f" Cropped {chunk_id}: {left},{top},{right},{bottom} from {W}x{H}")
return cropped_path
except Exception as e:
safe_print(f" Crop {chunk_id}: error: {e}")
return None
def analyze_image(chunk_id: str, cropped_path: Path) -> dict:
"""Analyze a cropped image using claude -p CLI."""
if not cropped_path or not cropped_path.exists():
return {}
prompt = IMAGE_ANALYST_PROMPT.format(image_path=str(cropped_path))
max_retries = 2
for attempt in range(1, max_retries + 1):
try:
result_text = call_claude(prompt, allowed_tools="Read", timeout=120)
data = extract_json(result_text)
safe_print(f" Image {chunk_id}: analyzed (ufo={data.get('ufo_anomaly_detected', False)})")
return data
except Exception as e:
safe_print(f" Image {chunk_id}: error attempt {attempt}: {str(e)[:80]}")
if attempt < max_retries:
time.sleep(5)
return {}
def write_chunk_file(chunk: dict) -> None:
"""Write chunk .md file."""
chunk_id = chunk["chunk_id"]
chunk_path = CHUNKS_DIR / f"{chunk_id}.md"
bbox = chunk.get("bbox") or {"x": 0, "y": 0, "w": 1, "h": 0.1}
page_num = chunk.get("page", 1)
source_png = f"../../processing/png/{DOC_ID}/p-{page_num:03d}.png"
content = f"""---
chunk_id: {chunk_id}
type: {chunk.get("type", "paragraph")}
page: {page_num}
order_in_page: {chunk.get("order_in_page", 1)}
order_global: {chunk.get("order_global", 1)}
bbox: {{x: {float(bbox.get('x') or 0):.2f}, y: {float(bbox.get('y') or 0):.2f}, w: {float(bbox.get('w') or 1):.2f}, h: {float(bbox.get('h') or 0.1):.2f}}}
classification: {json.dumps(chunk.get("classification"))}
formatting: {json.dumps(chunk.get("formatting", []))}
cross_page_hint: {chunk.get("cross_page_hint", "self_contained")}
prev_chunk: {json.dumps(chunk.get("prev_chunk"))}
next_chunk: {json.dumps(chunk.get("next_chunk"))}
related_image: {json.dumps(chunk.get("related_image"))}
related_table: null
ocr_confidence: {float(chunk.get("ocr_confidence") or 0.85)}
ocr_source_lines: {json.dumps(chunk.get("ocr_source_lines", []))}
redaction_code: {json.dumps(chunk.get("redaction_code"))}
redaction_inferred_content_type: {json.dumps(chunk.get("redaction_inferred_content_type"))}
image_type: {json.dumps(chunk.get("image_type"))}
ufo_anomaly_detected: {str(bool(chunk.get("ufo_anomaly_detected", False))).lower()}
cryptid_anomaly_detected: {str(bool(chunk.get("cryptid_anomaly_detected", False))).lower()}
ufo_anomaly_type: {json.dumps(chunk.get("ufo_anomaly_type"))}
ufo_anomaly_rationale: {json.dumps(chunk.get("ufo_anomaly_rationale"))}
cryptid_anomaly_type: {json.dumps(chunk.get("cryptid_anomaly_type"))}
cryptid_anomaly_rationale: {json.dumps(chunk.get("cryptid_anomaly_rationale"))}
image_description_en: {json.dumps(chunk.get("image_description_en"))}
image_description_pt_br: {json.dumps(chunk.get("image_description_pt_br"))}
extracted_text: {json.dumps(chunk.get("extracted_text"))}
source_png: {source_png}
---
**EN:** {chunk.get("content_en", "")}
**PT-BR:** {chunk.get("content_pt_br", "")}
"""
chunk_path.write_text(content, encoding="utf-8")
def main():
start_time = time.time()
CHUNKS_DIR.mkdir(parents=True, exist_ok=True)
IMAGES_DIR.mkdir(parents=True, exist_ok=True)
TABLES_DIR.mkdir(parents=True, exist_ok=True)
print(f"Rebuilding {DOC_ID}")
print(f"Processing {TOTAL_PAGES} pages with {WORKERS} workers...")
print("=" * 70)
page_numbers = list(range(1, TOTAL_PAGES + 1)) # 1..93
all_page_data = {}
# Process pages in batches of WORKERS
for batch_start in range(0, len(page_numbers), WORKERS):
batch = page_numbers[batch_start:batch_start + WORKERS]
batch_num = batch_start // WORKERS + 1
total_batches = (len(page_numbers) + WORKERS - 1) // WORKERS
safe_print(f"\nBatch {batch_num}/{total_batches}: pages {batch}")
with ThreadPoolExecutor(max_workers=WORKERS) as executor:
futures = {executor.submit(process_page, p): p for p in batch}
for future in as_completed(futures):
page_num = futures[future]
try:
data = future.result()
all_page_data[page_num] = data
except Exception as e:
safe_print(f" Page {page_num}: CRITICAL FAILURE: {e}")
all_page_data[page_num] = {
"page_number": page_num,
"chunks": [{
"order_in_page": 1,
"type": "unknown",
"content_en": f"[Page {page_num} — critical failure]",
"content_pt_br": f"[Página {page_num} — falha crítica]",
"bbox": {"x": 0, "y": 0, "w": 1, "h": 1},
"classification": None, "formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [],
"redaction_code": None, "redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False, "ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False, "cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None,
"image_description_en": None, "image_description_pt_br": None, "extracted_text": None
}]
}
print(f"\nAll pages processed. Assigning global chunk IDs...")
# Assign global chunk IDs in page order
all_chunks = []
chunk_counter = 1
for page_num in sorted(all_page_data.keys()):
page_data = all_page_data[page_num]
chunks = page_data.get("chunks", [])
chunks.sort(key=lambda c: c.get("order_in_page", 1))
for chunk in chunks:
chunk_id = f"c{chunk_counter:04d}"
chunk["chunk_id"] = chunk_id
chunk["page"] = page_num
chunk["order_global"] = chunk_counter
chunk_counter += 1
all_chunks.append(chunk)
total_chunks = len(all_chunks)
print(f"Total chunks: {total_chunks}")
# Prev/next pointers
for i, chunk in enumerate(all_chunks):
chunk["prev_chunk"] = all_chunks[i-1]["chunk_id"] if i > 0 else None
chunk["next_chunk"] = all_chunks[i+1]["chunk_id"] if i < len(all_chunks)-1 else None
# Identify image chunks
image_chunks = [c for c in all_chunks if c.get("type") == "image"]
print(f"\nCropping {len(image_chunks)} images...")
# Crop all images first
crop_results = {}
for chunk in image_chunks:
chunk_id = chunk["chunk_id"]
page_num = chunk["page"]
png_path = PNG_DIR / f"p-{page_num:03d}.png"
if png_path.exists():
cp = crop_image(chunk_id, png_path, chunk.get("bbox", {}))
crop_results[chunk_id] = cp
else:
crop_results[chunk_id] = None
# Analyze images in batches
image_items = [(c["chunk_id"], crop_results.get(c["chunk_id"]))
for c in image_chunks if crop_results.get(c["chunk_id"])]
print(f"\nAnalyzing {len(image_items)} cropped images...")
image_analysis = {}
for batch_start in range(0, len(image_items), WORKERS):
batch = image_items[batch_start:batch_start + WORKERS]
with ThreadPoolExecutor(max_workers=WORKERS) as executor:
futures = {executor.submit(analyze_image, cid, cp): cid for cid, cp in batch}
for future in as_completed(futures):
chunk_id = futures[future]
try:
image_analysis[chunk_id] = future.result()
except Exception as e:
safe_print(f" Image analysis {chunk_id}: {e}")
image_analysis[chunk_id] = {}
# Merge image analysis into chunks
for chunk in all_chunks:
chunk_id = chunk["chunk_id"]
if chunk.get("type") == "image":
chunk["related_image"] = f"IMG-{chunk_id}.png"
if chunk_id in image_analysis:
for field in ["image_description_en", "image_description_pt_br", "image_type",
"extracted_text", "ufo_anomaly_detected", "ufo_anomaly_type",
"ufo_anomaly_rationale", "cryptid_anomaly_detected",
"cryptid_anomaly_type", "cryptid_anomaly_rationale"]:
if field in image_analysis[chunk_id]:
chunk[field] = image_analysis[chunk_id][field]
# Write chunk files
print(f"\nWriting {total_chunks} chunk files...")
for chunk in all_chunks:
write_chunk_file(chunk)
print("Chunk files written.")
# Build _index.json
now_iso = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
index_chunks = []
for chunk in all_chunks:
bbox = chunk.get("bbox") or {"x": 0, "y": 0, "w": 1, "h": 0.1}
index_chunks.append({
"chunk_id": chunk["chunk_id"],
"type": chunk.get("type", "paragraph"),
"page": chunk.get("page", 1),
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"file": f"chunks/{chunk['chunk_id']}.md",
"bbox": bbox,
"preview": chunk.get("content_en", "")[:80]
})
index_data = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": total_chunks,
"build_approach": "subagents",
"build_model": "claude-haiku-4-5",
"build_at": now_iso,
"chunks": index_chunks
}
(OUT_DIR / "_index.json").write_text(json.dumps(index_data, ensure_ascii=False, indent=2), encoding="utf-8")
print("_index.json written.")
# Compute stats
chunk_types = {}
ufo_anomalies = []
cryptid_anomalies = []
images_count = 0
for chunk in all_chunks:
t = chunk.get("type", "paragraph")
chunk_types[t] = chunk_types.get(t, 0) + 1
if chunk.get("ufo_anomaly_detected"):
ufo_anomalies.append(chunk["chunk_id"])
if chunk.get("cryptid_anomaly_detected"):
cryptid_anomalies.append(chunk["chunk_id"])
if t == "image":
images_count += 1
# Assemble document.md
print("\nAssembling document.md...")
parts = []
# Frontmatter
parts.append("---")
parts.append('schema_version: "0.2.0"')
parts.append("type: master_document")
parts.append(f"doc_id: {DOC_ID}")
parts.append(f'canonical_title: "{DOC_TITLE}"')
parts.append(f"total_pages: {TOTAL_PAGES}")
parts.append(f"total_chunks: {total_chunks}")
parts.append("chunk_types_histogram:")
for t, count in sorted(chunk_types.items()):
parts.append(f" {t}: {count}")
parts.append("multi_page_tables: []")
parts.append(f"ufo_anomalies_flagged: {json.dumps(ufo_anomalies)}")
parts.append(f"cryptid_anomalies_flagged: {json.dumps(cryptid_anomalies)}")
parts.append('build_approach: "subagents"')
parts.append("build_model: claude-haiku-4-5")
parts.append(f"build_at: {now_iso}")
parts.append("---")
parts.append("")
current_page = None
for chunk in all_chunks:
page = chunk.get("page", 1)
if page != current_page:
current_page = page
parts.append(f"\n## Page {page}\n")
chunk_id = chunk["chunk_id"]
bbox = chunk.get("bbox") or {"x": 0, "y": 0, "w": 1, "h": 0.1}
bbox_str = f"{float(bbox.get('x') or 0):.2f}/{float(bbox.get('y') or 0):.2f}/{float(bbox.get('w') or 1):.2f}/{float(bbox.get('h') or 0.1):.2f}"
ctype = chunk.get("type", "paragraph")
parts.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->")
parts.append(f'<a id="{chunk_id}"></a>')
parts.append(f"### Chunk {chunk_id} — {ctype} · p{page} · bbox: {bbox_str}")
parts.append("")
parts.append(f"**EN:** {chunk.get('content_en', '')}")
parts.append("")
parts.append(f"**PT-BR:** {chunk.get('content_pt_br', '')}")
parts.append("")
if ctype == "image":
img_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
if img_path.exists():
parts.append(f"![chunk image](./images/IMG-{chunk_id}.png)")
parts.append("")
if chunk.get("image_description_en"):
parts.append(f"*{chunk['image_description_en']}*")
parts.append("")
# Metadata block
meta = {k: v for k, v in chunk.items() if k not in ["content_en", "content_pt_br"]}
parts.append("<details><summary>metadata</summary>")
parts.append("")
parts.append("```json")
parts.append(json.dumps(meta, ensure_ascii=False, indent=2))
parts.append("```")
parts.append("")
parts.append("</details>")
parts.append("")
parts.append("---")
parts.append("")
document_md = "\n".join(parts)
doc_path = OUT_DIR / "document.md"
doc_path.write_text(document_md, encoding="utf-8")
doc_md_bytes = len(document_md.encode("utf-8"))
print(f"document.md written ({doc_md_bytes:,} bytes)")
wall_seconds = int(time.time() - start_time)
print(f"\n{'='*70}")
print(f"pages_done={TOTAL_PAGES}, chunks_total={total_chunks}, images_extracted={images_count}, tables_stitched=0, ufo_anomalies={len(ufo_anomalies)}, cryptid_anomalies={len(cryptid_anomalies)}, wall_seconds={wall_seconds}")
if __name__ == "__main__":
main()

647
scripts/rebuild_doc38.py Normal file
View file

@ -0,0 +1,647 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Rebuilder for doc-38-143685-box7-incident-summaries-101-172
Uses Gemini 2.0 Flash for vision processing.
143 pages (p-000..p-063, p-100..p-178).
"""
import base64
import json
import os
import re
import sys
import time
import datetime
import concurrent.futures
from pathlib import Path
from PIL import Image
# Suppress Google auth FutureWarnings
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
from google import genai
from google.genai import types
DOC_ID = "doc-38-143685-box7-incident-summaries-101-172"
DOC_TITLE = "USAF UFO/UAP Incident Summary Sheets — Box 7 (Incidents 101-172)"
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
RAW_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
CHUNKS_DIR = RAW_DIR / "chunks"
IMAGES_DIR = RAW_DIR / "images"
TABLES_DIR = RAW_DIR / "tables"
GEMINI_API_KEY = os.environ.get("GOOGLE_API_KEY") or os.environ.get("GEMINI_API_KEY")
GEMINI_MODEL = "gemini-2.0-flash"
CALL_TIMEOUT = 120 # seconds per Gemini call
for d in [CHUNKS_DIR, IMAGES_DIR, TABLES_DIR]:
d.mkdir(parents=True, exist_ok=True)
def get_page_files():
pages = []
for f in sorted(PNG_DIR.glob("p-*.png")):
num = int(f.stem.split("-")[1])
pages.append(num)
return sorted(pages)
PAGE_NUMS = get_page_files()
TOTAL_PAGES = len(PAGE_NUMS)
# ── Gemini client (one per thread via local) ────────────────────────────────
def make_client():
return genai.Client(api_key=GEMINI_API_KEY)
# ── Prompt ───────────────────────────────────────────────────────────────────
def build_page_prompt(page_file: str, page_number: int) -> str:
return (
"You are a page-rebuilder for a UAP/UFO document digitization project.\n"
"Analyze this scanned page from a declassified USAF UFO incident summary document.\n\n"
f"- Document: USAF UFO/UAP Incident Summary Sheets, Box 7 (Incidents 101-172)\n"
f"- Page file: {page_file} | Sequential page: {page_number} of {TOTAL_PAGES}\n\n"
"Chunk types (use ONLY these):\n"
" letterhead, classification_banner, form_header, field_entry, paragraph_text,\n"
" redaction, table_marker, image, caption, page_number, signature_block,\n"
" handwritten_note, stamp, blank, separator\n\n"
"Rules:\n"
"- Each numbered form field (1. Date, 2. Time, etc.) = one field_entry chunk\n"
" EXCEPTION: you may group 2-3 very short consecutive fields into one chunk\n"
" to stay under token limits, e.g. '1. Date: 30 Jun 1948 | 2. Time: 2140'\n"
"- classification markings = classification_banner\n"
"- form title/header line = form_header\n"
"- stamps (RESTRICTED, DECLASSIFIED, SECRET, etc.) = stamp\n"
"- photos/sketches/diagrams = image\n"
"- handwritten annotations = handwritten_note\n"
"- page number printed = page_number\n"
"- near-blank pages = one blank chunk\n\n"
"For content_en: verbatim transcription (English).\n"
"For content_pt_br: Brazilian Portuguese translation; keep proper nouns/dates verbatim.\n"
"For blank pages: content_en='[BLANK PAGE]', content_pt_br='[PAGINA EM BRANCO]'.\n"
"For stamps: transcribe exact text seen.\n\n"
"bbox: fractions of page width/height, e.g. {\"x\":0.05,\"y\":0.10,\"w\":0.90,\"h\":0.05}\n\n"
"RETURN ONLY valid JSON, no markdown fences, no extra text:\n"
"{\n"
" \"page_number\": <N>,\n"
" \"page_file\": \"<p-NNN>\",\n"
" \"chunks\": [\n"
" {\n"
" \"type\": \"field_entry\",\n"
" \"order_in_page\": 1,\n"
" \"content_en\": \"1. Date: 30 June 1948\",\n"
" \"content_pt_br\": \"1. Data: 30 de junho de 1948\",\n"
" \"bbox\": {\"x\":0.05,\"y\":0.10,\"w\":0.90,\"h\":0.05},\n"
" \"classification\": null,\n"
" \"formatting\": [],\n"
" \"cross_page_hint\": \"self_contained\",\n"
" \"ocr_confidence\": 0.90,\n"
" \"ocr_source_lines\": [3,4],\n"
" \"redaction_code\": null,\n"
" \"redaction_inferred_content_type\": null,\n"
" \"image_type\": null,\n"
" \"ufo_anomaly_detected\": false,\n"
" \"cryptid_anomaly_detected\": false\n"
" }\n"
" ]\n"
"}\n"
)
def build_image_prompt() -> str:
return (
"You are an image analyst for a UAP/UFO document digitization project.\n"
"Analyze this cropped region from a declassified USAF document.\n\n"
"RETURN ONLY valid JSON (no markdown fences):\n"
"{\n"
" \"image_description_en\": \"...\",\n"
" \"image_description_pt_br\": \"...\",\n"
" \"image_type\": \"photograph|diagram|sketch|map|chart|stamp_graphic|form_field|text_block\",\n"
" \"extracted_text\": \"verbatim text or null\",\n"
" \"ufo_anomaly_detected\": false,\n"
" \"ufo_anomaly_type\": null,\n"
" \"ufo_anomaly_rationale\": null,\n"
" \"cryptid_anomaly_detected\": false,\n"
" \"cryptid_anomaly_type\": null,\n"
" \"cryptid_anomaly_rationale\": null\n"
"}\n\n"
"ufo_anomaly_detected=true ONLY if image shows actual UAP/UFO visual evidence.\n"
"cryptid_anomaly_detected=true ONLY if image shows unknown creature evidence.\n"
)
# ── API call with timeout ────────────────────────────────────────────────────
def gemini_call(img_bytes: bytes, prompt: str) -> str:
"""Call Gemini with image + text prompt. Returns response text."""
client = make_client()
def _call():
response = client.models.generate_content(
model=GEMINI_MODEL,
contents=[
types.Part.from_bytes(data=img_bytes, mime_type="image/png"),
prompt,
],
)
return response.text
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:
fut = ex.submit(_call)
return fut.result(timeout=CALL_TIMEOUT)
def parse_json_response(raw: str) -> dict:
"""Strip fences and parse JSON."""
raw = raw.strip()
if raw.startswith("```"):
raw = re.sub(r"^```[a-z]*\n?", "", raw)
raw = re.sub(r"\n?```$", "", raw.rstrip())
return json.loads(raw)
# ── Page processing ──────────────────────────────────────────────────────────
def process_page(task: tuple) -> dict:
file_num, seq_idx = task
page_file = f"p-{file_num:03d}"
png_path = PNG_DIR / f"{page_file}.png"
prompt = build_page_prompt(page_file, seq_idx)
with open(png_path, "rb") as f:
img_bytes = f.read()
max_retries = 3
for attempt in range(max_retries):
try:
raw = gemini_call(img_bytes, prompt)
result = parse_json_response(raw)
result["_file_num"] = file_num
result["_seq_idx"] = seq_idx
result["page_file"] = page_file
chunk_count = len(result.get("chunks", []))
print(f" [OK] page {seq_idx:3d}/{TOTAL_PAGES} ({page_file}) — {chunk_count} chunks", flush=True)
return result
except json.JSONDecodeError as e:
print(f" [WARN] page {seq_idx} JSON error (attempt {attempt+1}): {e}", flush=True)
if attempt == max_retries - 1:
return _fallback_page(file_num, seq_idx, page_file, f"JSON: {e}")
time.sleep(3)
except concurrent.futures.TimeoutError:
print(f" [TIMEOUT] page {seq_idx} (attempt {attempt+1})", flush=True)
if attempt == max_retries - 1:
return _fallback_page(file_num, seq_idx, page_file, "TIMEOUT")
time.sleep(5)
except Exception as e:
msg = str(e)[:100]
print(f" [ERR] page {seq_idx} (attempt {attempt+1}): {msg}", flush=True)
if attempt == max_retries - 1:
return _fallback_page(file_num, seq_idx, page_file, msg)
time.sleep(5)
def _fallback_page(file_num, seq_idx, page_file, reason):
return {
"page_number": seq_idx,
"page_file": page_file,
"_file_num": file_num,
"_seq_idx": seq_idx,
"chunks": [{
"type": "blank",
"order_in_page": 1,
"content_en": f"[PAGE {seq_idx} ERROR: {reason}]",
"content_pt_br": f"[PAGINA {seq_idx} ERRO: {reason}]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"cryptid_anomaly_detected": False,
}],
}
def process_pages_parallel(batch_size: int = 4) -> list:
tasks = [(file_num, idx + 1) for idx, file_num in enumerate(PAGE_NUMS)]
results = []
total_batches = (len(tasks) + batch_size - 1) // batch_size
print(f"Processing {TOTAL_PAGES} pages in {total_batches} batches of {batch_size}...", flush=True)
for b_start in range(0, len(tasks), batch_size):
batch = tasks[b_start:b_start + batch_size]
b_num = b_start // batch_size + 1
print(f" Batch {b_num}/{total_batches}: pages {batch[0][1]}–{batch[-1][1]}", flush=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as ex:
futs = {ex.submit(process_page, t): t for t in batch}
for fut in concurrent.futures.as_completed(futs):
results.append(fut.result())
if b_start + batch_size < len(tasks):
time.sleep(0.5)
results.sort(key=lambda r: r["_seq_idx"])
return results
# ── Chunk numbering ──────────────────────────────────────────────────────────
def assign_global_chunk_ids(page_results: list) -> list:
global_order = 0
all_chunks = []
for pr in page_results:
seq_idx = pr["_seq_idx"]
file_num = pr["_file_num"]
page_file = pr.get("page_file", f"p-{file_num:03d}")
chunks = sorted(pr.get("chunks", []), key=lambda c: c.get("order_in_page", 0))
for chunk in chunks:
global_order += 1
chunk_id = f"c{global_order:04d}"
chunk["chunk_id"] = chunk_id
chunk["order_global"] = global_order
chunk["page"] = seq_idx
chunk["page_file"] = page_file
chunk["_file_num"] = file_num
chunk["prev_chunk"] = f"c{global_order-1:04d}" if global_order > 1 else None
chunk["next_chunk"] = None
all_chunks.append(chunk)
for i in range(len(all_chunks) - 1):
all_chunks[i]["next_chunk"] = all_chunks[i + 1]["chunk_id"]
return all_chunks
# ── Image crop ───────────────────────────────────────────────────────────────
def crop_image_chunk(chunk: dict):
chunk_id = chunk["chunk_id"]
file_num = chunk["_file_num"]
bbox = chunk.get("bbox") or {}
if not isinstance(bbox, dict):
bbox = {}
png_path = PNG_DIR / f"p-{file_num:03d}.png"
out_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
try:
im = Image.open(png_path)
W, H = im.size
x = float(bbox.get("x", 0.0))
y = float(bbox.get("y", 0.0))
w = float(bbox.get("w", 1.0))
h = float(bbox.get("h", 1.0))
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
im.crop((left, top, right, bottom)).save(out_path)
return str(out_path)
except Exception as e:
print(f" [WARN] crop {chunk_id}: {e}", flush=True)
return None
# ── Image analysis ───────────────────────────────────────────────────────────
def analyze_image_chunk(chunk: dict):
chunk_id = chunk["chunk_id"]
img_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
if not img_path.exists():
return
try:
with open(img_path, "rb") as f:
img_bytes = f.read()
raw = gemini_call(img_bytes, build_image_prompt())
analysis = parse_json_response(raw)
for key in ["image_description_en", "image_description_pt_br", "image_type",
"extracted_text", "ufo_anomaly_detected", "ufo_anomaly_type",
"ufo_anomaly_rationale", "cryptid_anomaly_detected",
"cryptid_anomaly_type", "cryptid_anomaly_rationale"]:
if key in analysis:
chunk[key] = analysis[key]
ufo = chunk.get("ufo_anomaly_detected", False)
print(f" [IMG] {chunk_id} — ufo={ufo}", flush=True)
except Exception as e:
print(f" [WARN] img analysis {chunk_id}: {e}", flush=True)
# ── YAML helper ──────────────────────────────────────────────────────────────
def yaml_val(v) -> str:
if v is None:
return "null"
if isinstance(v, bool):
return "true" if v else "false"
if isinstance(v, (int, float)):
return str(v)
if isinstance(v, list):
if not v:
return "[]"
return "[" + ", ".join(yaml_val(i) for i in v) + "]"
s = str(v)
if any(c in s for c in [':', '#', '"', "'", '\n', '{', '}']):
return '"' + s.replace('\\', '\\\\').replace('"', '\\"') + '"'
return s
# ── Write chunk file ─────────────────────────────────────────────────────────
def write_chunk_file(chunk: dict):
chunk_id = chunk["chunk_id"]
page = chunk["page"]
page_file = chunk.get("page_file", "p-000")
ctype = chunk.get("type", "blank")
bbox = chunk.get("bbox") or {}
if not isinstance(bbox, dict):
bbox = {}
bx = float(bbox.get("x", 0.0))
by = float(bbox.get("y", 0.0))
bw = float(bbox.get("w", 1.0))
bh = float(bbox.get("h", 1.0))
related_image = f"IMG-{chunk_id}.png" if ctype == "image" else "null"
related_table = yaml_val(chunk.get("related_table"))
lines = [
"---",
f"chunk_id: {chunk_id}",
f"type: {ctype}",
f"page: {page}",
f"order_in_page: {chunk.get('order_in_page', 1)}",
f"order_global: {chunk.get('order_global', 1)}",
f"bbox: {{x: {bx:.3f}, y: {by:.3f}, w: {bw:.3f}, h: {bh:.3f}}}",
f"classification: {yaml_val(chunk.get('classification'))}",
f"formatting: {yaml_val(chunk.get('formatting', []))}",
f"cross_page_hint: {chunk.get('cross_page_hint', 'self_contained')}",
f"prev_chunk: {yaml_val(chunk.get('prev_chunk'))}",
f"next_chunk: {yaml_val(chunk.get('next_chunk'))}",
f"related_image: {related_image}",
f"related_table: {related_table}",
f"ocr_confidence: {float(chunk.get('ocr_confidence') or 0.85):.2f}",
f"ocr_source_lines: {yaml_val(chunk.get('ocr_source_lines', []))}",
f"redaction_code: {yaml_val(chunk.get('redaction_code'))}",
f"redaction_inferred_content_type: {yaml_val(chunk.get('redaction_inferred_content_type'))}",
f"image_type: {yaml_val(chunk.get('image_type'))}",
f"ufo_anomaly_detected: {yaml_val(chunk.get('ufo_anomaly_detected', False))}",
f"cryptid_anomaly_detected: {yaml_val(chunk.get('cryptid_anomaly_detected', False))}",
f"ufo_anomaly_type: {yaml_val(chunk.get('ufo_anomaly_type'))}",
f"ufo_anomaly_rationale: {yaml_val(chunk.get('ufo_anomaly_rationale'))}",
f"cryptid_anomaly_type: {yaml_val(chunk.get('cryptid_anomaly_type'))}",
f"cryptid_anomaly_rationale: {yaml_val(chunk.get('cryptid_anomaly_rationale'))}",
f"image_description_en: {yaml_val(chunk.get('image_description_en'))}",
f"image_description_pt_br: {yaml_val(chunk.get('image_description_pt_br'))}",
f"extracted_text: {yaml_val(chunk.get('extracted_text'))}",
f"source_png: ../../processing/png/{DOC_ID}/{page_file}.png",
"---",
"",
f"**EN:** {chunk.get('content_en') or ''}",
"",
f"**PT-BR:** {chunk.get('content_pt_br') or ''}",
"",
]
if ctype == "image":
lines += [
f"![{chunk_id} image](../images/IMG-{chunk_id}.png)",
"",
]
if chunk.get("image_description_en"):
lines += [f"*{chunk['image_description_en']}*", ""]
(CHUNKS_DIR / f"{chunk_id}.md").write_text("\n".join(lines), encoding="utf-8")
# ── Write _index.json ────────────────────────────────────────────────────────
def write_index_json(all_chunks: list, build_at: str):
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": len(all_chunks),
"build_approach": "subagents",
"build_model": GEMINI_MODEL,
"build_at": build_at,
"chunks": [],
}
for chunk in all_chunks:
cid = chunk["chunk_id"]
content_en = chunk.get("content_en") or ""
preview = content_en[:80] + ("..." if len(content_en) > 80 else "")
bbox = chunk.get("bbox") or {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0}
if not isinstance(bbox, dict):
bbox = {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0}
index["chunks"].append({
"chunk_id": cid,
"type": chunk.get("type", "blank"),
"page": chunk["page"],
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"file": f"chunks/{cid}.md",
"bbox": bbox,
"preview": preview,
})
out = RAW_DIR / "_index.json"
out.write_text(json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8")
print(f" Written: {out}", flush=True)
# ── Write document.md ────────────────────────────────────────────────────────
def write_document_md(all_chunks: list, build_at: str) -> int:
type_hist: dict = {}
ufo_flagged = []
cryptid_flagged = []
for chunk in all_chunks:
t = chunk.get("type", "blank")
type_hist[t] = type_hist.get(t, 0) + 1
if chunk.get("ufo_anomaly_detected"):
ufo_flagged.append(chunk["chunk_id"])
if chunk.get("cryptid_anomaly_detected"):
cryptid_flagged.append(chunk["chunk_id"])
hist = "\n".join(f" {k}: {v}" for k, v in sorted(type_hist.items()))
header = (
"---\n"
'schema_version: "0.2.0"\n'
"type: master_document\n"
f"doc_id: {DOC_ID}\n"
f'canonical_title: "{DOC_TITLE}"\n'
f"total_pages: {TOTAL_PAGES}\n"
f"total_chunks: {len(all_chunks)}\n"
"chunk_types_histogram:\n"
f"{hist}\n"
"multi_page_tables: []\n"
f"ufo_anomalies_flagged: [{', '.join(ufo_flagged)}]\n"
f"cryptid_anomalies_flagged: [{', '.join(cryptid_flagged)}]\n"
'build_approach: "subagents"\n'
f"build_model: {GEMINI_MODEL}\n"
f"build_at: {build_at}\n"
"---\n\n"
f"# {DOC_TITLE}\n\n"
)
pages_dict: dict = {}
for chunk in all_chunks:
p = chunk["page"]
pages_dict.setdefault(p, []).append(chunk)
body_parts = []
for page_num in sorted(pages_dict):
body_parts.append(f"## Page {page_num}\n\n")
for chunk in sorted(pages_dict[page_num], key=lambda c: c.get("order_in_page", 0)):
cid = chunk["chunk_id"]
ctype = chunk.get("type", "blank")
bbox = chunk.get("bbox") or {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0}
if not isinstance(bbox, dict):
bbox = {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0}
bs = f"{float(bbox.get('x',0)):.2f}/{float(bbox.get('y',0)):.2f}/{float(bbox.get('w',1)):.2f}/{float(bbox.get('h',1)):.2f}"
section = [
f"<!-- chunk:{cid} src:./chunks/{cid}.md -->",
f'<a id="{cid}"></a>',
f"### Chunk {cid} — {ctype} · p{page_num} · bbox: {bs}",
"",
f"**EN:** {chunk.get('content_en') or ''}",
"",
f"**PT-BR:** {chunk.get('content_pt_br') or ''}",
"",
]
if ctype == "image":
section += [f"![{cid} image](./images/IMG-{cid}.png)", ""]
if chunk.get("image_description_en"):
section += [f"*EN: {chunk['image_description_en']}*", ""]
if chunk.get("image_description_pt_br"):
section += [f"*PT-BR: {chunk['image_description_pt_br']}*", ""]
meta = {k: v for k, v in chunk.items()
if not k.startswith("_") and k not in ("content_en", "content_pt_br")}
section += [
"<details><summary>metadata</summary>",
"",
"```json",
json.dumps(meta, indent=2, ensure_ascii=False),
"```",
"",
"</details>",
"",
"---",
"",
]
body_parts.append("\n".join(section))
out = RAW_DIR / "document.md"
out.write_text(header + "".join(body_parts), encoding="utf-8")
size = out.stat().st_size
print(f" Written: {out} ({size} bytes)", flush=True)
return size
# ── Main ─────────────────────────────────────────────────────────────────────
CHECKPOINT_FILE = RAW_DIR / "_checkpoint_pages.json"
def main():
t0 = time.time()
build_at = datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
print(f"=== Rebuilding {DOC_ID} ===", flush=True)
print(f"Total pages: {TOTAL_PAGES} | Gemini model: {GEMINI_MODEL}", flush=True)
print()
# Step 1 — process pages (with checkpoint support)
if CHECKPOINT_FILE.exists():
print("STEP 1: Loading from checkpoint...", flush=True)
page_results = json.loads(CHECKPOINT_FILE.read_text(encoding="utf-8"))
print(f" Loaded {len(page_results)} pages from checkpoint.", flush=True)
else:
print("STEP 1: Processing pages...", flush=True)
page_results = process_pages_parallel(batch_size=4)
# Save checkpoint
CHECKPOINT_FILE.write_text(json.dumps(page_results, ensure_ascii=False), encoding="utf-8")
print(f" Done. {len(page_results)} pages processed. Checkpoint saved.", flush=True)
print()
# Step 2 — assign chunk IDs
print("STEP 2: Assigning chunk IDs...", flush=True)
all_chunks = assign_global_chunk_ids(page_results)
print(f" Total chunks: {len(all_chunks)}", flush=True)
print()
# Step 3 — crop images
image_chunks = [c for c in all_chunks if c.get("type") == "image"]
print(f"STEP 3: Cropping {len(image_chunks)} image chunks...", flush=True)
for chunk in image_chunks:
crop_image_chunk(chunk)
print()
# Step 4 — analyze images in batches of 4
print(f"STEP 4: Analyzing {len(image_chunks)} images...", flush=True)
for b in range(0, len(image_chunks), 4):
batch = image_chunks[b:b + 4]
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as ex:
list(ex.map(analyze_image_chunk, batch))
if b + 4 < len(image_chunks):
time.sleep(0.5)
print()
# Step 5 — write chunk files
print("STEP 5: Writing chunk files...", flush=True)
for chunk in all_chunks:
write_chunk_file(chunk)
print(f" Written {len(all_chunks)} chunk files.", flush=True)
print()
# Step 6 — write index
print("STEP 6: Writing _index.json...", flush=True)
write_index_json(all_chunks, build_at)
print()
# Step 7 — write document.md
print("STEP 7: Writing document.md...", flush=True)
doc_bytes = write_document_md(all_chunks, build_at)
print()
wall = int(time.time() - t0)
num_images = len(image_chunks)
num_ufo = len([c for c in all_chunks if c.get("ufo_anomaly_detected")])
num_cryptid = len([c for c in all_chunks if c.get("cryptid_anomaly_detected")])
print("=== DONE ===", flush=True)
print(
f"pages_done={TOTAL_PAGES}, chunks_total={len(all_chunks)}, "
f"images_extracted={num_images}, tables_stitched=0, "
f"ufo_anomalies={num_ufo}, cryptid_anomalies={num_cryptid}, "
f"wall_seconds={wall}",
flush=True,
)
if __name__ == "__main__":
main()

64
scripts/rebuild_doc65.py Normal file
View file

@ -0,0 +1,64 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
rebuild_doc65.py — Rebuild doc-65-hs1-834228961-62-hq-83894-section-4
Processes all 179 pages, writes chunks/, images/, _index.json, document.md
"""
import os
import sys
import json
import base64
import datetime
import time
from pathlib import Path
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-4"
DOC_TITLE = "65 HS1-834228961 62-HQ-83894 Section 4 — FBI Flying Discs Investigation"
RAW_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
OCR_DIR = Path(f"/Users/guto/ufo/processing/ocr/{DOC_ID}")
CHUNKS_DIR = RAW_DIR / "chunks"
IMAGES_DIR = RAW_DIR / "images"
TABLES_DIR = RAW_DIR / "tables"
# Ensure dirs exist
CHUNKS_DIR.mkdir(parents=True, exist_ok=True)
IMAGES_DIR.mkdir(parents=True, exist_ok=True)
TABLES_DIR.mkdir(parents=True, exist_ok=True)
# Build ordered page map: page_number (1-based) -> (png_filename, ocr_filename)
def build_page_map():
pngs = sorted([f for f in os.listdir(PNG_DIR) if f.endswith('.png')])
ocrs = sorted([f for f in os.listdir(OCR_DIR) if f.endswith('.txt')])
page_map = {}
for i, (png, ocr) in enumerate(zip(pngs, ocrs), 1):
page_map[i] = {
'png': str(PNG_DIR / png),
'ocr': str(OCR_DIR / ocr),
'png_filename': png,
'ocr_filename': ocr
}
return page_map
def read_ocr(path):
try:
with open(path, 'r', encoding='utf-8') as f:
return f.read().strip()
except:
return ""
def encode_image_b64(path):
with open(path, 'rb') as f:
return base64.standard_b64encode(f.read()).decode('utf-8')
def now_iso():
return datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ')
if __name__ == "__main__":
page_map = build_page_map()
print(f"Total pages: {len(page_map)}")
for p, info in list(page_map.items())[:5]:
print(f" Page {p:03d}: png={info['png_filename']}, ocr={info['ocr_filename']}")
print("Script loaded OK")

462
scripts/rebuild_doc65_assemble.py Normal file
View file

@ -0,0 +1,462 @@
#!/usr/bin/env python3
"""
Assemble chunks/, _index.json, and document.md from _pages_raw.json
for doc-65-hs1-834228961-62-hq-83894-section-1.
Also:
- Crops image chunks using PIL
- Detects multi-page table markers for stitching
- Writes all output files
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
from collections import defaultdict
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-1"
DOC_TITLE = "65 HS1-834228961 62-HQ-83894 Section 1 (FBI UAP/UFO Investigative File)"
PNG_DIR = Path("/Users/guto/ufo/processing/png/doc-65-hs1-834228961-62-hq-83894-section-1")
OUTPUT_DIR = Path("/Users/guto/ufo/raw/doc-65-hs1-834228961-62-hq-83894-section-1")
CHUNKS_DIR = OUTPUT_DIR / "chunks"
IMAGES_DIR = OUTPUT_DIR / "images"
TABLES_DIR = OUTPUT_DIR / "tables"
TOTAL_PAGES = 150
BUILD_AT = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
BUILD_MODEL = "claude-haiku-4-5"
def load_pages() -> list[dict]:
raw_path = OUTPUT_DIR / "_pages_raw.json"
with open(raw_path, encoding="utf-8") as f:
return json.load(f)
def normalize_chunk(chunk: dict, page_num: int) -> dict:
"""Ensure all required fields exist with correct types."""
defaults = {
"order_in_page": 1,
"type": "paragraph",
"content_en": "",
"content_pt_br": "",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.05},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.85,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None,
}
result = {**defaults, **chunk}
# Coerce None strings to empty
if result.get('content_en') is None:
result['content_en'] = ''
if result.get('content_pt_br') is None:
result['content_pt_br'] = ''
result["page"] = page_num
# Normalize booleans
for bool_field in ("ufo_anomaly_detected", "cryptid_anomaly_detected"):
val = result.get(bool_field)
if isinstance(val, str):
result[bool_field] = val.lower() in ("true", "1", "yes")
elif val is None:
result[bool_field] = False
else:
result[bool_field] = bool(val)
# Normalize formatting to list
if not isinstance(result.get("formatting"), list):
result["formatting"] = []
# Normalize ocr_source_lines to list
if not isinstance(result.get("ocr_source_lines"), list):
result["ocr_source_lines"] = []
# Normalize bbox
bbox = result.get("bbox", {})
if not isinstance(bbox, dict):
bbox = {}
result["bbox"] = {
"x": float(bbox.get("x", 0.0)),
"y": float(bbox.get("y", 0.0)),
"w": float(bbox.get("w", 1.0)),
"h": float(bbox.get("h", 0.05)),
}
return result
def assign_global_ids(pages: list[dict]) -> list[dict]:
"""
Assign chunk_id, order_global, prev_chunk, next_chunk to all chunks.
Returns flat list of all chunks in global order.
"""
all_chunks = []
counter = 1
for page_data in pages:
page_num = page_data.get("page_number", 0)
chunks = page_data.get("chunks", [])
# Sort by order_in_page
chunks.sort(key=lambda c: c.get("order_in_page", 0))
for chunk in chunks:
normalized = normalize_chunk(chunk, page_num)
normalized["chunk_id"] = f"c{counter:04d}"
normalized["order_global"] = counter
all_chunks.append(normalized)
counter += 1
# Set prev/next pointers
for i, chunk in enumerate(all_chunks):
chunk["prev_chunk"] = all_chunks[i - 1]["chunk_id"] if i > 0 else None
chunk["next_chunk"] = all_chunks[i + 1]["chunk_id"] if i < len(all_chunks) - 1 else None
return all_chunks
def crop_image(chunk: dict) -> str | None:
"""Crop image region from page PNG. Returns saved path or None."""
page_num = chunk["page"]
chunk_id = chunk["chunk_id"]
png_path = PNG_DIR / f"p-{page_num:03d}.png"
if not png_path.exists():
return None
bbox = chunk["bbox"]
out_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
try:
from PIL import Image
im = Image.open(png_path)
W, H = im.size
pad = 0.005
x, y, w, h = bbox["x"], bbox["y"], bbox["w"], bbox["h"]
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
if right <= left or bottom <= top:
return None
cropped = im.crop((left, top, right, bottom))
cropped.save(str(out_path))
return str(out_path)
except Exception as e:
print(f" Crop error for {chunk_id}: {e}", file=sys.stderr)
return None
def write_chunk_file(chunk: dict, source_png_relative: str) -> None:
"""Write chunks/c<NNNN>.md for one chunk."""
chunk_id = chunk["chunk_id"]
chunk_type = chunk.get("type", "paragraph")
page = chunk.get("page", 0)
order_in_page = chunk.get("order_in_page", 1)
order_global = chunk.get("order_global", 1)
bbox = chunk["bbox"]
classification = chunk.get("classification")
formatting = chunk.get("formatting", [])
cross_page_hint = chunk.get("cross_page_hint", "self_contained")
prev_chunk = chunk.get("prev_chunk")
next_chunk = chunk.get("next_chunk")
ocr_confidence = chunk.get("ocr_confidence", 0.85)
ocr_source_lines = chunk.get("ocr_source_lines", [])
redaction_code = chunk.get("redaction_code")
redaction_inferred = chunk.get("redaction_inferred_content_type")
image_type = chunk.get("image_type")
ufo_anomaly = chunk.get("ufo_anomaly_detected", False)
ufo_type = chunk.get("ufo_anomaly_type")
ufo_rationale = chunk.get("ufo_anomaly_rationale")
cryptid_anomaly = chunk.get("cryptid_anomaly_detected", False)
cryptid_type = chunk.get("cryptid_anomaly_type")
cryptid_rationale = chunk.get("cryptid_anomaly_rationale")
image_desc_en = chunk.get("image_description_en")
image_desc_pt = chunk.get("image_description_pt_br")
extracted_text = chunk.get("extracted_text")
content_en = chunk.get("content_en", "")
content_pt_br = chunk.get("content_pt_br", "")
# Related fields
related_image = f"IMG-{chunk_id}.png" if chunk_type == "image" else None
related_table = chunk.get("related_table")
def yaml_val(v):
if v is None:
return "null"
if isinstance(v, bool):
return "true" if v else "false"
if isinstance(v, (int, float)):
return str(v)
if isinstance(v, list):
if not v:
return "[]"
items = ", ".join(f'"{x}"' for x in v)
return f"[{items}]"
# string
s = str(v).replace('"', '\\"')
return f'"{s}"'
lines = [
"---",
f"chunk_id: {chunk_id}",
f"type: {chunk_type}",
f"page: {page}",
f"order_in_page: {order_in_page}",
f"order_global: {order_global}",
f"bbox: {{x: {bbox['x']:.2f}, y: {bbox['y']:.2f}, w: {bbox['w']:.2f}, h: {bbox['h']:.2f}}}",
f"classification: {yaml_val(classification)}",
f"formatting: {yaml_val(formatting)}",
f"cross_page_hint: {cross_page_hint}",
f"prev_chunk: {yaml_val(prev_chunk)}",
f"next_chunk: {yaml_val(next_chunk)}",
f"related_image: {yaml_val(related_image)}",
f"related_table: {yaml_val(related_table)}",
f"ocr_confidence: {ocr_confidence}",
f"ocr_source_lines: {yaml_val(ocr_source_lines)}",
f"redaction_code: {yaml_val(redaction_code)}",
f"redaction_inferred_content_type: {yaml_val(redaction_inferred)}",
f"image_type: {yaml_val(image_type)}",
f"ufo_anomaly_detected: {yaml_val(ufo_anomaly)}",
f"cryptid_anomaly_detected: {yaml_val(cryptid_anomaly)}",
f"ufo_anomaly_type: {yaml_val(ufo_type)}",
f"ufo_anomaly_rationale: {yaml_val(ufo_rationale)}",
f"cryptid_anomaly_type: {yaml_val(cryptid_type)}",
f"cryptid_anomaly_rationale: {yaml_val(cryptid_rationale)}",
f"image_description_en: {yaml_val(image_desc_en)}",
f"image_description_pt_br: {yaml_val(image_desc_pt)}",
f"extracted_text: {yaml_val(extracted_text)}",
f"source_png: {source_png_relative}",
"---",
"",
f"**EN:** {content_en}",
"",
f"**PT-BR:** {content_pt_br}",
]
out_path = CHUNKS_DIR / f"{chunk_id}.md"
out_path.write_text("\n".join(lines), encoding="utf-8")
def write_index(all_chunks: list[dict]) -> None:
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": len(all_chunks),
"build_approach": "subagents",
"build_model": BUILD_MODEL,
"build_at": BUILD_AT,
"chunks": []
}
for chunk in all_chunks:
preview = chunk.get("content_en", "")[:80]
index["chunks"].append({
"chunk_id": chunk["chunk_id"],
"type": chunk.get("type", "paragraph"),
"page": chunk.get("page", 0),
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"file": f"chunks/{chunk['chunk_id']}.md",
"bbox": chunk["bbox"],
"preview": preview,
})
out_path = OUTPUT_DIR / "_index.json"
with open(out_path, "w", encoding="utf-8") as f:
json.dump(index, f, ensure_ascii=False, indent=2)
print(f"Written: {out_path}")
def write_document_md(all_chunks: list[dict], stats: dict) -> None:
# Compute histogram
histogram: dict[str, int] = defaultdict(int)
ufo_flagged = []
cryptid_flagged = []
for chunk in all_chunks:
histogram[chunk.get("type", "paragraph")] += 1
if chunk.get("ufo_anomaly_detected"):
ufo_flagged.append(chunk["chunk_id"])
if chunk.get("cryptid_anomaly_detected"):
cryptid_flagged.append(chunk["chunk_id"])
histogram_yaml = "\n".join(f" {k}: {v}" for k, v in sorted(histogram.items()))
ufo_yaml = json.dumps(ufo_flagged, ensure_ascii=False)
cryptid_yaml = json.dumps(cryptid_flagged, ensure_ascii=False)
lines = [
"---",
'schema_version: "0.2.0"',
"type: master_document",
f"doc_id: {DOC_ID}",
f'canonical_title: "{DOC_TITLE}"',
f"total_pages: {TOTAL_PAGES}",
f"total_chunks: {len(all_chunks)}",
"chunk_types_histogram:",
histogram_yaml,
f"multi_page_tables: []",
f"ufo_anomalies_flagged: {ufo_yaml}",
f"cryptid_anomalies_flagged: {cryptid_yaml}",
'build_approach: "subagents"',
f"build_model: {BUILD_MODEL}",
f"build_at: {BUILD_AT}",
"---",
"",
]
# Group chunks by page
pages_map: dict[int, list[dict]] = defaultdict(list)
for chunk in all_chunks:
pages_map[chunk["page"]].append(chunk)
for page_num in sorted(pages_map.keys()):
page_chunks = pages_map[page_num]
lines.append(f"## Page {page_num}")
lines.append("")
for chunk in page_chunks:
cid = chunk["chunk_id"]
ctype = chunk.get("type", "paragraph")
bbox = chunk["bbox"]
bbox_str = f"{bbox['x']:.2f}/{bbox['y']:.2f}/{bbox['w']:.2f}/{bbox['h']:.2f}"
content_en = chunk.get("content_en", "")
content_pt_br = chunk.get("content_pt_br", "")
lines.append(f"<!-- chunk:{cid} src:./chunks/{cid}.md -->")
lines.append(f'<a id="{cid}"></a>')
lines.append(f"### Chunk {cid} — {ctype} · p{page_num} · bbox: {bbox_str}")
lines.append("")
lines.append(f"**EN:** {content_en}")
lines.append("")
lines.append(f"**PT-BR:** {content_pt_br}")
lines.append("")
# Embed image if applicable
if ctype == "image":
img_path = IMAGES_DIR / f"IMG-{cid}.png"
if img_path.exists():
lines.append(f"![{cid} image](./images/IMG-{cid}.png)")
lines.append("")
if chunk.get("image_description_en"):
lines.append(f"*Image description:* {chunk['image_description_en']}")
lines.append("")
# Metadata collapsible
meta = {
"chunk_id": cid,
"type": ctype,
"page": chunk.get("page"),
"order_in_page": chunk.get("order_in_page"),
"order_global": chunk.get("order_global"),
"bbox": bbox,
"classification": chunk.get("classification"),
"formatting": chunk.get("formatting", []),
"cross_page_hint": chunk.get("cross_page_hint"),
"prev_chunk": chunk.get("prev_chunk"),
"next_chunk": chunk.get("next_chunk"),
"related_image": f"IMG-{cid}.png" if ctype == "image" else None,
"related_table": chunk.get("related_table"),
"ocr_confidence": chunk.get("ocr_confidence"),
"ocr_source_lines": chunk.get("ocr_source_lines", []),
"redaction_code": chunk.get("redaction_code"),
"redaction_inferred_content_type": chunk.get("redaction_inferred_content_type"),
"image_type": chunk.get("image_type"),
"ufo_anomaly_detected": chunk.get("ufo_anomaly_detected", False),
"cryptid_anomaly_detected": chunk.get("cryptid_anomaly_detected", False),
"ufo_anomaly_type": chunk.get("ufo_anomaly_type"),
"ufo_anomaly_rationale": chunk.get("ufo_anomaly_rationale"),
"cryptid_anomaly_type": chunk.get("cryptid_anomaly_type"),
"cryptid_anomaly_rationale": chunk.get("cryptid_anomaly_rationale"),
"image_description_en": chunk.get("image_description_en"),
"image_description_pt_br": chunk.get("image_description_pt_br"),
"extracted_text": chunk.get("extracted_text"),
}
lines.append("<details><summary>metadata</summary>")
lines.append("")
lines.append("```json")
lines.append(json.dumps(meta, ensure_ascii=False, indent=2))
lines.append("```")
lines.append("")
lines.append("</details>")
lines.append("")
lines.append("---")
lines.append("")
out_path = OUTPUT_DIR / "document.md"
with open(out_path, "w", encoding="utf-8") as f:
f.write("\n".join(lines))
print(f"Written: {out_path}")
return len("\n".join(lines).encode("utf-8"))
def main():
start = time.time()
print("Loading pages...")
pages = load_pages()
print(f" {len(pages)} pages loaded")
print("Assigning global IDs...")
all_chunks = assign_global_ids(pages)
print(f" {len(all_chunks)} chunks total")
# Create dirs
CHUNKS_DIR.mkdir(parents=True, exist_ok=True)
IMAGES_DIR.mkdir(parents=True, exist_ok=True)
TABLES_DIR.mkdir(parents=True, exist_ok=True)
# Crop images
image_chunks = [c for c in all_chunks if c.get("type") == "image"]
print(f"Cropping {len(image_chunks)} images...")
images_saved = 0
for chunk in image_chunks:
path = crop_image(chunk)
if path:
images_saved += 1
# Write chunk files
print("Writing chunk files...")
for chunk in all_chunks:
page_num = chunk["page"]
source_png = f"../../processing/png/{DOC_ID}/p-{page_num:03d}.png"
write_chunk_file(chunk, source_png)
print(f" {len(all_chunks)} chunk files written")
# Write _index.json
print("Writing _index.json...")
write_index(all_chunks)
# Write document.md
print("Writing document.md...")
stats = {}
doc_bytes = write_document_md(all_chunks, stats)
# Compute final stats
ufo_count = sum(1 for c in all_chunks if c.get("ufo_anomaly_detected"))
cryptid_count = sum(1 for c in all_chunks if c.get("cryptid_anomaly_detected"))
elapsed = int(time.time() - start)
print(f"\nDone in {elapsed}s")
print(f"STATS pages={TOTAL_PAGES} chunks={len(all_chunks)} images={images_saved} tables=0 ufo={ufo_count} cryptid={cryptid_count} doc_md_bytes={doc_bytes}")
if __name__ == "__main__":
main()

534
scripts/rebuild_doc65_full.py Normal file
View file

@ -0,0 +1,534 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
rebuild_doc65_full.py
Full pipeline to rebuild doc-65-hs1-834228961-62-hq-83894-section-4.
Uses Google Gemini flash for vision analysis of each page.
Generates chunks/, images/, _index.json, document.md
"""
import os
import sys
import json
import base64
import datetime
import time
import re
import concurrent.futures
from pathlib import Path
from PIL import Image as PILImage
# ---- Config ----
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-4"
DOC_TITLE = "65 HS1-834228961 62-HQ-83894 Section 4 — FBI Flying Discs / UAP Investigation Files"
RAW_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
OCR_DIR = Path(f"/Users/guto/ufo/processing/ocr/{DOC_ID}")
CHUNKS_DIR = RAW_DIR / "chunks"
IMAGES_DIR = RAW_DIR / "images"
TABLES_DIR = RAW_DIR / "tables"
GEMINI_API_KEY = os.environ.get("GEMINI_API_KEY", "")
BATCH_SIZE = 4 # conservative for API limits
MAX_WORKERS = 4
# ---- Ensure dirs ----
for d in [CHUNKS_DIR, IMAGES_DIR, TABLES_DIR]:
d.mkdir(parents=True, exist_ok=True)
# ---- Page map ----
def build_page_map():
pngs = sorted([f for f in os.listdir(PNG_DIR) if f.endswith('.png')])
ocrs = sorted([f for f in os.listdir(OCR_DIR) if f.endswith('.txt')])
page_map = {}
for i, (png, ocr) in enumerate(zip(pngs, ocrs), 1):
page_map[i] = {
'png': str(PNG_DIR / png),
'ocr': str(OCR_DIR / ocr),
'png_filename': png,
}
return page_map
def read_ocr(path):
try:
with open(path, 'r', encoding='utf-8') as f:
return f.read().strip()
except:
return ""
def now_iso():
return datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ')
# ---- Gemini vision call ----
import google.generativeai as genai
genai.configure(api_key=GEMINI_API_KEY)
PAGE_ANALYSIS_PROMPT = """You are a document analyst rebuilding a declassified FBI UAP/flying saucer investigation file.
Analyze this page image carefully and return ONLY valid JSON (no markdown code fences, no explanation).
The JSON must have this exact structure:
{
"page_number": <int>,
"chunks": [
{
"type": "<one of: cover|letterhead|heading|subheading|body_text|caption|signature|stamp|redaction|table_marker|image|handwriting|footer|header|separator|blank>",
"order_in_page": <int starting at 1>,
"content_en": "<English text or description>",
"content_pt_br": "<Brazilian Portuguese translation/description>",
"bbox": {"x": <0.0-1.0>, "y": <0.0-1.0>, "w": <0.0-1.0>, "h": <0.0-1.0>},
"classification": <null or "SECRET" or "TOP SECRET" etc>,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": <0.0-1.0>,
"ocr_source_lines": [],
"redaction_code": <null or "(b)(1)" etc>,
"redaction_inferred_content_type": null,
"image_type": <null or "photograph" or "diagram" or "sketch" or "stamp" or "logo">,
"ufo_anomaly_detected": false,
"cryptid_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null,
"image_description_en": null,
"image_description_pt_br": null,
"extracted_text": null
}
]
}
Rules:
- Identify ALL distinct content blocks (letterhead, classification markings, memo headers, body paragraphs, stamps, redactions, signatures, photos, etc.)
- For redacted areas: type="redaction", content_en="[REDACTED]", content_pt_br="[REDATADO]", include redaction_code if visible
- For blank pages: ONE chunk with type="blank"
- For stamps: type="stamp", include extracted_text with what the stamp says
- For signatures: type="signature"
- For photos/images: type="image", image_type appropriately, image_description_en with detailed description
- UAP/flying saucer content: set ufo_anomaly_detected=true and fill ufo_anomaly_type and ufo_anomaly_rationale
- bbox values are fractions of page dimensions (0.0 to 1.0)
- content_en must be verbatim OCR text where possible, or [description] for non-text
- content_pt_br must be Brazilian Portuguese translation
- This is page %d of 179 total
- Document: FBI investigation files about flying discs/UAP reports, 1947-era
"""
def analyze_page_with_gemini(page_num, png_path, ocr_text, retry=3):
"""Call Gemini flash to analyze a page image."""
prompt = PAGE_ANALYSIS_PROMPT % page_num
if ocr_text:
prompt += f"\n\nOCR text available (may be incomplete):\n{ocr_text[:2000]}"
for attempt in range(retry):
try:
model = genai.GenerativeModel('gemini-1.5-flash')
with open(png_path, 'rb') as f:
img_data = f.read()
import google.generativeai as genai2
from google.generativeai.types import HarmCategory, HarmBlockThreshold
response = model.generate_content(
[
{"mime_type": "image/png", "data": img_data},
prompt
],
generation_config={"temperature": 0.1, "max_output_tokens": 4096},
safety_settings={
HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
}
)
text = response.text.strip()
# Remove markdown code fences if present
if text.startswith('```'):
text = re.sub(r'^```(?:json)?\s*', '', text)
text = re.sub(r'\s*```$', '', text)
data = json.loads(text)
return data
except json.JSONDecodeError as e:
print(f" Page {page_num}: JSON parse error (attempt {attempt+1}): {e}")
if attempt < retry - 1:
time.sleep(2)
except Exception as e:
print(f" Page {page_num}: Error (attempt {attempt+1}): {e}")
if attempt < retry - 1:
time.sleep(3)
# Fallback: minimal chunk
return {
"page_number": page_num,
"chunks": [{
"type": "body_text",
"order_in_page": 1,
"content_en": f"[Page {page_num} — vision analysis failed]",
"content_pt_br": f"[Página {page_num} — análise visual falhou]",
"bbox": {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"cryptid_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None
}]
}
def process_page(args):
page_num, png_path, ocr_path = args
ocr_text = read_ocr(ocr_path)
print(f" Processing page {page_num:03d}...", flush=True)
result = analyze_page_with_gemini(page_num, png_path, ocr_text)
print(f" Done page {page_num:03d}: {len(result.get('chunks', []))} chunks", flush=True)
return page_num, result
def crop_image_for_chunk(page_png, bbox, out_path):
"""Crop image region for an image-type chunk."""
try:
im = PILImage.open(page_png)
W, H = im.size
x = bbox.get('x', 0)
y = bbox.get('y', 0)
w = bbox.get('w', 1)
h = bbox.get('h', 1)
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
if right <= left or bottom <= top:
return False
crop = im.crop((left, top, right, bottom))
crop.save(out_path)
return True
except Exception as e:
print(f" Crop error: {e}")
return False
def write_chunk_file(chunk_id, chunk_data, page_num, order_global, prev_chunk, next_chunk, source_png_filename):
"""Write a single chunk .md file."""
bbox = chunk_data.get('bbox', {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90})
# Determine related_image
related_image = None
if chunk_data.get('type') == 'image':
related_image = f"IMG-{chunk_id}.png"
meta = {
"chunk_id": chunk_id,
"type": chunk_data.get('type', 'body_text'),
"page": page_num,
"order_in_page": chunk_data.get('order_in_page', 1),
"order_global": order_global,
"bbox": bbox,
"classification": chunk_data.get('classification'),
"formatting": chunk_data.get('formatting', []),
"cross_page_hint": chunk_data.get('cross_page_hint', 'self_contained'),
"prev_chunk": prev_chunk,
"next_chunk": next_chunk,
"related_image": related_image,
"related_table": None,
"ocr_confidence": chunk_data.get('ocr_confidence', 0.8),
"ocr_source_lines": chunk_data.get('ocr_source_lines', []),
"redaction_code": chunk_data.get('redaction_code'),
"redaction_inferred_content_type": chunk_data.get('redaction_inferred_content_type'),
"image_type": chunk_data.get('image_type'),
"ufo_anomaly_detected": chunk_data.get('ufo_anomaly_detected', False),
"cryptid_anomaly_detected": chunk_data.get('cryptid_anomaly_detected', False),
"ufo_anomaly_type": chunk_data.get('ufo_anomaly_type'),
"ufo_anomaly_rationale": chunk_data.get('ufo_anomaly_rationale'),
"cryptid_anomaly_type": chunk_data.get('cryptid_anomaly_type'),
"cryptid_anomaly_rationale": chunk_data.get('cryptid_anomaly_rationale'),
"image_description_en": chunk_data.get('image_description_en'),
"image_description_pt_br": chunk_data.get('image_description_pt_br'),
"extracted_text": chunk_data.get('extracted_text'),
"source_png": f"../../processing/png/{DOC_ID}/{source_png_filename}",
}
content_en = chunk_data.get('content_en', '')
content_pt_br = chunk_data.get('content_pt_br', '')
# Build YAML frontmatter
def yaml_val(v):
if v is None:
return "null"
if isinstance(v, bool):
return str(v).lower()
if isinstance(v, (int, float)):
return str(v)
if isinstance(v, list):
if not v:
return "[]"
return "[" + ", ".join(yaml_val(i) for i in v) + "]"
if isinstance(v, dict):
return "{" + ", ".join(f"{k}: {yaml_val(vv)}" for k, vv in v.items()) + "}"
# string
s = str(v)
if any(c in s for c in [':', '#', '[', ']', '{', '}', '*', '&', '!', '|', '>', "'", '"', '\n']):
s = s.replace('"', '\\"')
return f'"{s}"'
return s
lines = ["---"]
for k, v in meta.items():
if isinstance(v, dict):
lines.append(f"{k}: {{{', '.join(f'{kk}: {yaml_val(vv)}' for kk, vv in v.items())}}}")
else:
lines.append(f"{k}: {yaml_val(v)}")
lines.append("---")
lines.append("")
lines.append(f"**EN:** {content_en}")
lines.append("")
lines.append(f"**PT-BR:** {content_pt_br}")
lines.append("")
out_path = CHUNKS_DIR / f"{chunk_id}.md"
with open(out_path, 'w', encoding='utf-8') as f:
f.write('\n'.join(lines))
return meta
def main():
start_time = time.time()
page_map = build_page_map()
total_pages = len(page_map)
print(f"Starting rebuild: {total_pages} pages")
# Process all pages in batches of BATCH_SIZE
all_page_results = {} # page_num -> result dict
page_nums = list(page_map.keys())
for batch_start in range(0, total_pages, BATCH_SIZE):
batch = page_nums[batch_start:batch_start + BATCH_SIZE]
batch_args = [(p, page_map[p]['png'], page_map[p]['ocr']) for p in batch]
print(f"\nBatch {batch_start//BATCH_SIZE + 1}: pages {batch[0]}-{batch[-1]}", flush=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
futures = {executor.submit(process_page, args): args[0] for args in batch_args}
for future in concurrent.futures.as_completed(futures):
page_num = futures[future]
try:
pn, result = future.result(timeout=120)
all_page_results[pn] = result
except Exception as e:
print(f" Page {page_num} failed: {e}")
all_page_results[page_num] = {
"page_number": page_num,
"chunks": [{
"type": "body_text",
"order_in_page": 1,
"content_en": f"[Page {page_num} — processing error]",
"content_pt_br": f"[Página {page_num} — erro de processamento]",
"bbox": {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90},
"classification": None, "formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [],
"redaction_code": None, "redaction_inferred_content_type": None,
"image_type": None, "ufo_anomaly_detected": False,
"cryptid_anomaly_detected": False,
"ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None,
"image_description_en": None, "image_description_pt_br": None,
"extracted_text": None
}]
}
# Small pause between batches to be respectful of rate limits
if batch_start + BATCH_SIZE < total_pages:
time.sleep(1)
print(f"\nAll pages analyzed. Assigning global chunk IDs...")
# --- Global chunk numbering ---
all_chunks_ordered = [] # list of (page_num, chunk_data, source_png_filename)
for page_num in sorted(all_page_results.keys()):
result = all_page_results[page_num]
chunks = sorted(result.get('chunks', []), key=lambda c: c.get('order_in_page', 1))
source_png = page_map[page_num]['png_filename']
for chunk in chunks:
all_chunks_ordered.append((page_num, chunk, source_png))
total_chunks = len(all_chunks_ordered)
print(f"Total chunks: {total_chunks}")
# Assign chunk_ids and write chunk files
chunk_id_list = [f"c{(i+1):04d}" for i in range(total_chunks)]
index_entries = []
all_chunk_meta = []
images_extracted = 0
ufo_anomalies = []
cryptid_anomalies = []
print("Writing chunk files...")
for i, (page_num, chunk_data, source_png) in enumerate(all_chunks_ordered):
chunk_id = chunk_id_list[i]
order_global = i + 1
prev_chunk = chunk_id_list[i-1] if i > 0 else None
next_chunk = chunk_id_list[i+1] if i < total_chunks - 1 else None
# Crop image if needed
if chunk_data.get('type') == 'image':
bbox = chunk_data.get('bbox', {})
img_out = IMAGES_DIR / f"IMG-{chunk_id}.png"
png_path = page_map[page_num]['png']
if crop_image_for_chunk(png_path, bbox, img_out):
images_extracted += 1
# Write chunk file
meta = write_chunk_file(
chunk_id, chunk_data, page_num, order_global,
prev_chunk, next_chunk, source_png
)
all_chunk_meta.append(meta)
# Track anomalies
if chunk_data.get('ufo_anomaly_detected'):
ufo_anomalies.append(chunk_id)
if chunk_data.get('cryptid_anomaly_detected'):
cryptid_anomalies.append(chunk_id)
# Index entry
content_en = chunk_data.get('content_en', '')
preview = content_en[:80].replace('\n', ' ')
bbox = chunk_data.get('bbox', {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90})
index_entries.append({
"chunk_id": chunk_id,
"type": chunk_data.get('type', 'body_text'),
"page": page_num,
"order_in_page": chunk_data.get('order_in_page', 1),
"order_global": order_global,
"file": f"chunks/{chunk_id}.md",
"bbox": bbox,
"preview": preview
})
# --- Write _index.json ---
print("Writing _index.json...")
build_at = now_iso()
# Compute chunk type histogram
type_hist = {}
for entry in index_entries:
t = entry['type']
type_hist[t] = type_hist.get(t, 0) + 1
index_data = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": total_pages,
"total_chunks": total_chunks,
"build_approach": "subagents",
"build_model": "claude-sonnet-4-6",
"build_at": build_at,
"chunks": index_entries
}
with open(RAW_DIR / "_index.json", 'w', encoding='utf-8') as f:
json.dump(index_data, f, indent=2, ensure_ascii=False)
# --- Assemble document.md ---
print("Assembling document.md...")
doc_lines = []
doc_lines.append("---")
doc_lines.append('schema_version: "0.2.0"')
doc_lines.append("type: master_document")
doc_lines.append(f"doc_id: {DOC_ID}")
doc_lines.append(f'canonical_title: "{DOC_TITLE}"')
doc_lines.append(f"total_pages: {total_pages}")
doc_lines.append(f"total_chunks: {total_chunks}")
hist_str = ", ".join(f'"{k}": {v}' for k, v in sorted(type_hist.items()))
doc_lines.append(f"chunk_types_histogram: {{{hist_str}}}")
doc_lines.append("multi_page_tables: []")
ufo_str = "[" + ", ".join(ufo_anomalies) + "]"
cryptid_str = "[" + ", ".join(cryptid_anomalies) + "]"
doc_lines.append(f"ufo_anomalies_flagged: {ufo_str}")
doc_lines.append(f"cryptid_anomalies_flagged: {cryptid_str}")
doc_lines.append('build_approach: "subagents"')
doc_lines.append("build_model: claude-sonnet-4-6")
doc_lines.append(f"build_at: {build_at}")
doc_lines.append("---")
doc_lines.append("")
# Group chunks by page
chunks_by_page = {}
for i, (page_num, chunk_data, source_png) in enumerate(all_chunks_ordered):
if page_num not in chunks_by_page:
chunks_by_page[page_num] = []
chunks_by_page[page_num].append((chunk_id_list[i], chunk_data, source_png))
for page_num in sorted(chunks_by_page.keys()):
doc_lines.append(f"## Page {page_num}")
doc_lines.append("")
for chunk_id, chunk_data, source_png in chunks_by_page[page_num]:
ctype = chunk_data.get('type', 'body_text')
bbox = chunk_data.get('bbox', {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90})
bbox_str = f"{bbox.get('x',0):.2f}/{bbox.get('y',0):.2f}/{bbox.get('w',0):.2f}/{bbox.get('h',0):.2f}"
doc_lines.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->")
doc_lines.append(f'<a id="{chunk_id}"></a>')
doc_lines.append(f"### Chunk {chunk_id} — {ctype} · p{page_num} · bbox: {bbox_str}")
doc_lines.append("")
doc_lines.append(f"**EN:** {chunk_data.get('content_en', '')}")
doc_lines.append("")
doc_lines.append(f"**PT-BR:** {chunk_data.get('content_pt_br', '')}")
doc_lines.append("")
if ctype == 'image':
img_path = f"./images/IMG-{chunk_id}.png"
doc_lines.append(f"![{chunk_id} image]({img_path})")
doc_lines.append("")
desc = chunk_data.get('image_description_en', '')
if desc:
doc_lines.append(f"*{desc}*")
doc_lines.append("")
# Metadata details
meta_dict = all_chunk_meta[int(chunk_id[1:]) - 1]
doc_lines.append("<details><summary>metadata</summary>")
doc_lines.append("")
doc_lines.append("```json")
doc_lines.append(json.dumps(meta_dict, indent=2, ensure_ascii=False))
doc_lines.append("```")
doc_lines.append("")
doc_lines.append("</details>")
doc_lines.append("")
doc_lines.append("---")
doc_lines.append("")
doc_content = '\n'.join(doc_lines)
with open(RAW_DIR / "document.md", 'w', encoding='utf-8') as f:
f.write(doc_content)
doc_bytes = len(doc_content.encode('utf-8'))
wall_seconds = int(time.time() - start_time)
print(f"\nSTATS pages={total_pages} chunks={total_chunks} images={images_extracted} tables=0 ufo={len(ufo_anomalies)} cryptid={len(cryptid_anomalies)} doc_md_bytes={doc_bytes}")
print(f"Wall time: {wall_seconds}s")
if __name__ == "__main__":
main()

514
scripts/rebuild_doc65_gemini.py Normal file
View file

@ -0,0 +1,514 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
rebuild_doc65_gemini.py
Full pipeline to rebuild doc-65-hs1-834228961-62-hq-83894-section-4.
Uses Google Gemini flash for vision analysis of each page.
CRITICAL: Always wraps Gemini calls with thread timeout (known hang issue).
"""
import os
import sys
import json
import datetime
import time
import re
import concurrent.futures
from pathlib import Path
from PIL import Image as PILImage
import warnings
warnings.filterwarnings('ignore')
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-4"
DOC_TITLE = "65 HS1-834228961 62-HQ-83894 Section 4 — FBI Flying Discs / UAP Investigation Files"
RAW_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
OCR_DIR = Path(f"/Users/guto/ufo/processing/ocr/{DOC_ID}")
CHUNKS_DIR = RAW_DIR / "chunks"
IMAGES_DIR = RAW_DIR / "images"
TABLES_DIR = RAW_DIR / "tables"
GEMINI_API_KEY = os.environ.get("GEMINI_API_KEY", "")
GEMINI_MODEL = "gemini-2.5-flash"
BATCH_SIZE = 4
MAX_WORKERS = 4
GEMINI_TIMEOUT_SEC = 120
for d in [CHUNKS_DIR, IMAGES_DIR, TABLES_DIR]:
d.mkdir(parents=True, exist_ok=True)
def build_page_map():
pngs = sorted([f for f in os.listdir(PNG_DIR) if f.endswith('.png')])
ocrs = sorted([f for f in os.listdir(OCR_DIR) if f.endswith('.txt')])
page_map = {}
for i, (png, ocr) in enumerate(zip(pngs, ocrs), 1):
page_map[i] = {
'png': str(PNG_DIR / png),
'ocr': str(OCR_DIR / ocr),
'png_filename': png,
}
return page_map
def read_ocr(path):
try:
with open(path, 'r', encoding='utf-8') as f:
return f.read().strip()
except:
return ""
def now_iso():
return datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ')
# Compact prompt that minimizes token usage in the response
PAGE_ANALYSIS_PROMPT = """Analyze this FBI declassified document page. Return ONLY raw JSON (no markdown fences).
JSON format (keep content_en values SHORT — max 300 chars per chunk, truncate with "..." if needed):
{"page_number":%d,"chunks":[{"type":"<cover|letterhead|heading|subheading|body_text|caption|signature|stamp|redaction|table_marker|image|handwriting|footer|header|separator|blank>","order_in_page":<int>,"content_en":"<text>","content_pt_br":"<pt-br>","bbox":{"x":<0-1>,"y":<0-1>,"w":<0-1>,"h":<0-1>},"classification":null,"formatting":[],"cross_page_hint":"self_contained","ocr_confidence":<0-1>,"ocr_source_lines":[],"redaction_code":null,"redaction_inferred_content_type":null,"image_type":null,"ufo_anomaly_detected":<true|false>,"cryptid_anomaly_detected":false,"ufo_anomaly_type":null,"ufo_anomaly_rationale":null,"cryptid_anomaly_type":null,"cryptid_anomaly_rationale":null,"image_description_en":null,"image_description_pt_br":null,"extracted_text":null}]}
Rules:
- Each paragraph/section = separate chunk
- Redacted: type=redaction, content_en="[REDACTED]"
- Blank page: one chunk type=blank
- Flying disc/UAP reports: ufo_anomaly_detected=true
- bbox: x=left, y=top, w=width, h=height, all 0.0-1.0
- Page %d of 179, FBI flying discs 1947"""
def fallback_chunk(page_num):
return {
"page_number": page_num,
"chunks": [{
"type": "body_text",
"order_in_page": 1,
"content_en": f"[Page {page_num} — vision analysis failed]",
"content_pt_br": f"[Página {page_num} — análise visual falhou]",
"bbox": {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90},
"classification": None, "formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [],
"redaction_code": None, "redaction_inferred_content_type": None,
"image_type": None, "ufo_anomaly_detected": False,
"cryptid_anomaly_detected": False,
"ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None,
"image_description_en": None, "image_description_pt_br": None,
"extracted_text": None
}]
}
def _gemini_call_inner(page_num, png_path, prompt_text):
"""Single Gemini API call — run inside thread for timeout."""
import google.genai as genai
import google.genai.types as gTypes
client = genai.Client(api_key=GEMINI_API_KEY)
with open(png_path, 'rb') as f:
img_bytes = f.read()
response = client.models.generate_content(
model=GEMINI_MODEL,
contents=[
gTypes.Part.from_bytes(data=img_bytes, mime_type='image/png'),
prompt_text
],
config=gTypes.GenerateContentConfig(
temperature=0.1,
max_output_tokens=16384,
)
)
return response.text
def clean_json_text(text):
"""Try to clean and extract JSON from potentially truncated response."""
if text is None:
return None
text = text.strip()
# Remove markdown fences
text = re.sub(r'^```(?:json)?\s*', '', text, flags=re.MULTILINE)
text = re.sub(r'\s*```\s*$', '', text, flags=re.MULTILINE)
text = text.strip()
# Try direct parse first
try:
return json.loads(text)
except json.JSONDecodeError:
pass
# Try to find the JSON object boundaries
start = text.find('{')
if start == -1:
return None
# Try to repair truncated JSON by finding the last complete chunk
# Strategy: find the last complete chunk object and close the array properly
text_from_start = text[start:]
# Try progressively smaller slices to find valid JSON
# Look for last valid chunk boundary
last_bracket = text_from_start.rfind('}')
while last_bracket > 0:
candidate = text_from_start[:last_bracket+1]
# Try to close the chunks array and root object
for suffix in ['', ']}', ']}}']:
try:
result = json.loads(candidate + suffix)
return result
except:
pass
last_bracket = text_from_start.rfind('}', 0, last_bracket)
return None
def analyze_page(page_num, png_path, ocr_text, retry=3):
prompt = PAGE_ANALYSIS_PROMPT % (page_num, page_num)
for attempt in range(retry):
try:
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
future = pool.submit(_gemini_call_inner, page_num, png_path, prompt)
text = future.result(timeout=GEMINI_TIMEOUT_SEC)
data = clean_json_text(text)
if data and 'chunks' in data and data['chunks']:
return data
print(f" P{page_num} no valid JSON (attempt {attempt+1})", flush=True)
if attempt < retry - 1:
time.sleep(2)
except concurrent.futures.TimeoutError:
print(f" P{page_num} TIMEOUT (attempt {attempt+1}/{retry})", flush=True)
if attempt < retry - 1:
time.sleep(5)
except Exception as e:
err = str(e)
print(f" P{page_num} error (attempt {attempt+1}): {err[:120]}", flush=True)
if '429' in err or 'RESOURCE_EXHAUSTED' in err:
wait = 15 * (attempt + 1)
print(f" Rate limit hit, waiting {wait}s...", flush=True)
time.sleep(wait)
elif attempt < retry - 1:
time.sleep(3)
return fallback_chunk(page_num)
def process_page_task(args):
page_num, png_path, ocr_path = args
ocr_text = read_ocr(ocr_path)
result = analyze_page(page_num, png_path, ocr_text)
n = len(result.get('chunks', []))
print(f" P{page_num:03d}: {n} chunks", flush=True)
return page_num, result
def crop_image_chunk(page_png, bbox, out_path):
try:
im = PILImage.open(page_png)
W, H = im.size
x = max(0.0, float(bbox.get('x', 0)))
y = max(0.0, float(bbox.get('y', 0)))
w = max(0.01, float(bbox.get('w', 0.5)))
h = max(0.01, float(bbox.get('h', 0.5)))
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
if right <= left or bottom <= top:
return False
crop = im.crop((left, top, right, bottom))
crop.save(str(out_path))
return True
except Exception as e:
print(f" Crop error: {e}", flush=True)
return False
def yaml_scalar(v):
if v is None:
return "null"
if isinstance(v, bool):
return "true" if v else "false"
if isinstance(v, (int, float)):
return str(v)
if isinstance(v, list):
if not v:
return "[]"
return "[" + ", ".join(yaml_scalar(i) for i in v) + "]"
if isinstance(v, dict):
return "{" + ", ".join(f"{k}: {yaml_scalar(vv)}" for k, vv in v.items()) + "}"
s = str(v)
needs_quote = any(c in s for c in [':', '#', '[', ']', '{', '}', '|', '>', '*', '&', '!', "'", '"', '\n', '\r'])
if needs_quote:
s = s.replace('\\', '\\\\').replace('"', '\\"').replace('\n', '\\n').replace('\r', '')
return f'"{s}"'
return s
def bbox_safe(bbox):
if not bbox or not isinstance(bbox, dict):
return {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90}
return {
"x": float(bbox.get('x', 0.05)),
"y": float(bbox.get('y', 0.05)),
"w": float(bbox.get('w', 0.90)),
"h": float(bbox.get('h', 0.90)),
}
def write_chunk_file(chunk_id, chunk_data, page_num, order_global, prev_chunk, next_chunk, source_png_filename):
bbox = bbox_safe(chunk_data.get('bbox'))
ctype = chunk_data.get('type', 'body_text')
related_image = f"IMG-{chunk_id}.png" if ctype == 'image' else None
meta = {
"chunk_id": chunk_id,
"type": ctype,
"page": page_num,
"order_in_page": chunk_data.get('order_in_page', 1),
"order_global": order_global,
"bbox": bbox,
"classification": chunk_data.get('classification'),
"formatting": chunk_data.get('formatting', []),
"cross_page_hint": chunk_data.get('cross_page_hint', 'self_contained'),
"prev_chunk": prev_chunk,
"next_chunk": next_chunk,
"related_image": related_image,
"related_table": None,
"ocr_confidence": chunk_data.get('ocr_confidence', 0.8),
"ocr_source_lines": chunk_data.get('ocr_source_lines', []),
"redaction_code": chunk_data.get('redaction_code'),
"redaction_inferred_content_type": chunk_data.get('redaction_inferred_content_type'),
"image_type": chunk_data.get('image_type'),
"ufo_anomaly_detected": bool(chunk_data.get('ufo_anomaly_detected', False)),
"cryptid_anomaly_detected": bool(chunk_data.get('cryptid_anomaly_detected', False)),
"ufo_anomaly_type": chunk_data.get('ufo_anomaly_type'),
"ufo_anomaly_rationale": chunk_data.get('ufo_anomaly_rationale'),
"cryptid_anomaly_type": chunk_data.get('cryptid_anomaly_type'),
"cryptid_anomaly_rationale": chunk_data.get('cryptid_anomaly_rationale'),
"image_description_en": chunk_data.get('image_description_en'),
"image_description_pt_br": chunk_data.get('image_description_pt_br'),
"extracted_text": chunk_data.get('extracted_text'),
"source_png": f"../../processing/png/{DOC_ID}/{source_png_filename}",
}
lines = ["---"]
for k, v in meta.items():
if isinstance(v, dict):
pairs = ", ".join(f"{kk}: {yaml_scalar(vv)}" for kk, vv in v.items())
lines.append(f"{k}: {{{pairs}}}")
else:
lines.append(f"{k}: {yaml_scalar(v)}")
lines.append("---")
lines.append("")
lines.append(f"**EN:** {chunk_data.get('content_en', '')}")
lines.append("")
lines.append(f"**PT-BR:** {chunk_data.get('content_pt_br', '')}")
lines.append("")
out_path = CHUNKS_DIR / f"{chunk_id}.md"
with open(str(out_path), 'w', encoding='utf-8') as f:
f.write('\n'.join(lines))
return meta
def main():
start_time = time.time()
page_map = build_page_map()
total_pages = len(page_map)
print(f"Pages: {total_pages}, Model: {GEMINI_MODEL}", flush=True)
all_page_results = {}
page_nums = list(page_map.keys())
cache_file = RAW_DIR / "_page_results_cache.json"
if cache_file.exists():
print("Loading partial cache...", flush=True)
with open(str(cache_file), 'r', encoding='utf-8') as f:
cached = json.load(f)
all_page_results = {int(k): v for k, v in cached.items()}
print(f" Loaded {len(all_page_results)} cached pages", flush=True)
pages_to_process = [p for p in page_nums if p not in all_page_results]
print(f"Pages to process: {len(pages_to_process)}", flush=True)
total_batches = (len(pages_to_process) + BATCH_SIZE - 1) // BATCH_SIZE
for batch_idx, batch_start in enumerate(range(0, len(pages_to_process), BATCH_SIZE)):
batch = pages_to_process[batch_start:batch_start + BATCH_SIZE]
batch_args = [(p, page_map[p]['png'], page_map[p]['ocr']) for p in batch]
print(f"Batch {batch_idx+1}/{total_batches}: pages {batch[0]}-{batch[-1]}", flush=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
futures = {executor.submit(process_page_task, args): args[0] for args in batch_args}
for future in concurrent.futures.as_completed(futures, timeout=600):
page_num = futures[future]
try:
pn, result = future.result(timeout=5)
all_page_results[pn] = result
except Exception as e:
print(f" P{page_num} future error: {e}", flush=True)
all_page_results[page_num] = fallback_chunk(page_num)
with open(str(cache_file), 'w', encoding='utf-8') as f:
json.dump({str(k): v for k, v in all_page_results.items()}, f, ensure_ascii=False)
print(f" Cache: {len(all_page_results)} pages", flush=True)
if batch_start + BATCH_SIZE < len(pages_to_process):
time.sleep(1)
print(f"\nAll pages processed. Building output...", flush=True)
all_chunks_ordered = []
for page_num in sorted(all_page_results.keys()):
result = all_page_results[page_num]
chunks = sorted(result.get('chunks', []), key=lambda c: c.get('order_in_page', 1))
source_png = page_map[page_num]['png_filename']
for chunk in chunks:
all_chunks_ordered.append((page_num, chunk, source_png))
total_chunks = len(all_chunks_ordered)
print(f"Total chunks: {total_chunks}", flush=True)
chunk_id_list = [f"c{(i+1):04d}" for i in range(total_chunks)]
print("Cropping image chunks...", flush=True)
images_extracted = 0
for i, (page_num, chunk_data, source_png) in enumerate(all_chunks_ordered):
if chunk_data.get('type') == 'image':
chunk_id = chunk_id_list[i]
bbox = bbox_safe(chunk_data.get('bbox'))
img_out = IMAGES_DIR / f"IMG-{chunk_id}.png"
png_path = page_map[page_num]['png']
if crop_image_chunk(png_path, bbox, img_out):
images_extracted += 1
print("Writing chunk files...", flush=True)
index_entries = []
all_chunk_meta = []
ufo_anomalies = []
cryptid_anomalies = []
for i, (page_num, chunk_data, source_png) in enumerate(all_chunks_ordered):
chunk_id = chunk_id_list[i]
order_global = i + 1
prev_chunk = chunk_id_list[i-1] if i > 0 else None
next_chunk = chunk_id_list[i+1] if i < total_chunks - 1 else None
meta = write_chunk_file(chunk_id, chunk_data, page_num, order_global,
prev_chunk, next_chunk, source_png)
all_chunk_meta.append(meta)
if chunk_data.get('ufo_anomaly_detected'):
ufo_anomalies.append(chunk_id)
if chunk_data.get('cryptid_anomaly_detected'):
cryptid_anomalies.append(chunk_id)
content_en = str(chunk_data.get('content_en', ''))
preview = content_en[:80].replace('\n', ' ')
index_entries.append({
"chunk_id": chunk_id,
"type": chunk_data.get('type', 'body_text'),
"page": page_num,
"order_in_page": chunk_data.get('order_in_page', 1),
"order_global": order_global,
"file": f"chunks/{chunk_id}.md",
"bbox": bbox_safe(chunk_data.get('bbox')),
"preview": preview
})
print("Writing _index.json...", flush=True)
build_at = now_iso()
type_hist = {}
for entry in index_entries:
t = entry['type']
type_hist[t] = type_hist.get(t, 0) + 1
index_data = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": total_pages,
"total_chunks": total_chunks,
"build_approach": "subagents",
"build_model": "claude-sonnet-4-6",
"build_at": build_at,
"chunks": index_entries
}
with open(str(RAW_DIR / "_index.json"), 'w', encoding='utf-8') as f:
json.dump(index_data, f, indent=2, ensure_ascii=False)
print("Assembling document.md...", flush=True)
doc_lines = []
doc_lines.append("---")
doc_lines.append('schema_version: "0.2.0"')
doc_lines.append("type: master_document")
doc_lines.append(f"doc_id: {DOC_ID}")
doc_lines.append(f'canonical_title: "{DOC_TITLE}"')
doc_lines.append(f"total_pages: {total_pages}")
doc_lines.append(f"total_chunks: {total_chunks}")
hist_str = ", ".join(f'"{k}": {v}' for k, v in sorted(type_hist.items()))
doc_lines.append(f"chunk_types_histogram: {{{hist_str}}}")
doc_lines.append("multi_page_tables: []")
doc_lines.append(f"ufo_anomalies_flagged: [{', '.join(ufo_anomalies)}]")
doc_lines.append(f"cryptid_anomalies_flagged: [{', '.join(cryptid_anomalies)}]")
doc_lines.append('build_approach: "subagents"')
doc_lines.append("build_model: claude-sonnet-4-6")
doc_lines.append(f"build_at: {build_at}")
doc_lines.append("---")
doc_lines.append("")
chunks_by_page = {}
for i, (page_num, chunk_data, source_png) in enumerate(all_chunks_ordered):
if page_num not in chunks_by_page:
chunks_by_page[page_num] = []
chunks_by_page[page_num].append((chunk_id_list[i], chunk_data, all_chunk_meta[i]))
for page_num in sorted(chunks_by_page.keys()):
doc_lines.append(f"## Page {page_num}")
doc_lines.append("")
for chunk_id, chunk_data, meta in chunks_by_page[page_num]:
ctype = chunk_data.get('type', 'body_text')
bbox = bbox_safe(chunk_data.get('bbox'))
bbox_str = f"{bbox['x']:.2f}/{bbox['y']:.2f}/{bbox['w']:.2f}/{bbox['h']:.2f}"
doc_lines.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->")
doc_lines.append(f'<a id="{chunk_id}"></a>')
doc_lines.append(f"### Chunk {chunk_id} — {ctype} · p{page_num} · bbox: {bbox_str}")
doc_lines.append("")
doc_lines.append(f"**EN:** {chunk_data.get('content_en', '')}")
doc_lines.append("")
doc_lines.append(f"**PT-BR:** {chunk_data.get('content_pt_br', '')}")
doc_lines.append("")
if ctype == 'image':
doc_lines.append(f"![{chunk_id} image](./images/IMG-{chunk_id}.png)")
doc_lines.append("")
desc = chunk_data.get('image_description_en', '')
if desc:
doc_lines.append(f"*{desc}*")
doc_lines.append("")
doc_lines.append("<details><summary>metadata</summary>")
doc_lines.append("")
doc_lines.append("```json")
doc_lines.append(json.dumps(meta, indent=2, ensure_ascii=False))
doc_lines.append("```")
doc_lines.append("")
doc_lines.append("</details>")
doc_lines.append("")
doc_lines.append("---")
doc_lines.append("")
doc_content = '\n'.join(doc_lines)
with open(str(RAW_DIR / "document.md"), 'w', encoding='utf-8') as f:
f.write(doc_content)
doc_bytes = len(doc_content.encode('utf-8'))
wall_seconds = int(time.time() - start_time)
if cache_file.exists():
os.remove(str(cache_file))
print(f"\nSTATS pages={total_pages} chunks={total_chunks} images={images_extracted} tables=0 ufo={len(ufo_anomalies)} cryptid={len(cryptid_anomalies)} doc_md_bytes={doc_bytes}")
print(f"Wall time: {wall_seconds}s")
if __name__ == "__main__":
main()

287
scripts/rebuild_doc65_page_rebuilder.py Normal file
View file

@ -0,0 +1,287 @@
#!/usr/bin/env python3
"""
Page rebuilder for doc-65-hs1-834228961-62-hq-83894-section-1
Processes pages 1-150 using vision (PNGs at p-001.png .. p-150.png)
Outputs JSON per page with chunks list.
"""
import anthropic
import base64
import json
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-1"
DOC_TITLE = "65 HS1-834228961 62-HQ-83894 Section 1 (FBI UFO/UAP Investigative File)"
PNG_DIR = Path("/Users/guto/ufo/processing/png/doc-65-hs1-834228961-62-hq-83894-section-1")
OCR_DIR = Path("/Users/guto/ufo/processing/ocr/doc-65-hs1-834228961-62-hq-83894-section-1")
OUTPUT_DIR = Path("/Users/guto/ufo/raw/doc-65-hs1-834228961-62-hq-83894-section-1")
TOTAL_PAGES = 150
client = anthropic.Anthropic()
CHUNK_TYPES = [
"letterhead", "classification_banner", "header", "subheader",
"paragraph", "list_item", "caption", "footnote", "page_number",
"signature_block", "stamp", "redaction_block", "image", "table_marker",
"form_field", "watermark", "separator", "blank"
]
PAGE_REBUILDER_PROMPT = """You are a page-rebuilder for a declassified UAP/UFO government document.
Document: {doc_title}
Page: {page_number} of {total_pages}
Analyze this page image carefully and extract ALL content as structured chunks.
Return a JSON object with this exact structure:
{{
"page_number": {page_number},
"chunks": [
{{
"order_in_page": 1,
"type": "<chunk_type>",
"content_en": "<English text or description>",
"content_pt_br": "<Brazilian Portuguese translation>",
"bbox": {{"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.85,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null,
"image_description_en": null,
"image_description_pt_br": null,
"extracted_text": null
}}
]
}}
CHUNK TYPES (use only these):
letterhead, classification_banner, header, subheader, paragraph, list_item,
caption, footnote, page_number, signature_block, stamp, redaction_block,
image, table_marker, form_field, watermark, separator, blank
RULES:
1. Extract EVERY element on the page — nothing is skipped
2. bbox: normalized coordinates (x=left, y=top, w=width, h=height) relative to page size (0.0 to 1.0)
3. content_en: verbatim OCR text for text chunks; for images describe what you see
4. content_pt_br: Brazilian Portuguese translation of content_en (NOT European Portuguese)
5. Preserve UTF-8 accents: ç, ã, á, é, í, ó, ú, â, ê, ô, à
6. For redacted sections: type="redaction_block", content_en="[REDACTED]", content_pt_br="[REDACTADO]"
7. For images/photos: type="image", describe the visual content in image_description_en and image_description_pt_br
8. For stamps: type="stamp"
9. classification: extract classification markings if visible (e.g. "SECRET", "CONFIDENTIAL")
10. formatting: array of applicable ["bold", "italic", "underline", "all_caps", "handwritten", "typewritten"]
11. cross_page_hint: "self_contained" | "continues_to_next" | "continues_from_prev"
12. ufo_anomaly_detected: true if chunk contains UAP/UFO sighting data, coordinates, witness accounts, or anomalous phenomena
13. If page is blank: return one chunk with type="blank"
14. Order chunks top-to-bottom, left-to-right as they appear on the page
15. Return ONLY valid JSON, no markdown code blocks, no extra text
OCR text hint (may be empty or garbled):
{ocr_text}
"""
def load_image_b64(png_path: Path) -> str:
with open(png_path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
def load_ocr(page_num: int) -> str:
txt_path = OCR_DIR / f"p-{page_num:03d}.txt"
if txt_path.exists():
try:
content = txt_path.read_text(encoding="utf-8").strip()
return content if content else "(empty)"
except Exception:
return "(unreadable)"
return "(not found)"
def process_page(page_num: int, retries: int = 3) -> dict:
png_path = PNG_DIR / f"p-{page_num:03d}.png"
if not png_path.exists():
print(f" WARNING: PNG not found for page {page_num}: {png_path}", file=sys.stderr)
return {
"page_number": page_num,
"chunks": [{
"order_in_page": 1,
"type": "blank",
"content_en": "[PAGE NOT FOUND]",
"content_pt_br": "[PÁGINA NÃO ENCONTRADA]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None
}]
}
ocr_text = load_ocr(page_num)
img_b64 = load_image_b64(png_path)
prompt = PAGE_REBUILDER_PROMPT.format(
doc_title=DOC_TITLE,
page_number=page_num,
total_pages=TOTAL_PAGES,
ocr_text=ocr_text[:2000] # cap at 2000 chars
)
for attempt in range(retries):
try:
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=4096,
messages=[{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": img_b64
}
},
{
"type": "text",
"text": prompt
}
]
}]
)
raw = response.content[0].text.strip()
# Strip markdown code blocks if present
if raw.startswith("```"):
lines = raw.split("\n")
raw = "\n".join(lines[1:-1] if lines[-1] == "```" else lines[1:])
result = json.loads(raw)
result["page_number"] = page_num # ensure correct
print(f" Page {page_num:3d} done — {len(result.get('chunks', []))} chunks", flush=True)
return result
except json.JSONDecodeError as e:
print(f" Page {page_num} JSON error (attempt {attempt+1}): {e}", file=sys.stderr)
if attempt == retries - 1:
return {
"page_number": page_num,
"chunks": [{
"order_in_page": 1,
"type": "blank",
"content_en": f"[PARSE ERROR: {str(e)[:100]}]",
"content_pt_br": f"[ERRO DE ANÁLISE: {str(e)[:100]}]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None
}]
}
time.sleep(2 ** attempt)
except Exception as e:
print(f" Page {page_num} API error (attempt {attempt+1}): {e}", file=sys.stderr)
if attempt == retries - 1:
return {
"page_number": page_num,
"chunks": [{
"order_in_page": 1,
"type": "blank",
"content_en": f"[API ERROR: {str(e)[:100]}]",
"content_pt_br": f"[ERRO DE API: {str(e)[:100]}]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None
}]
}
time.sleep(2 ** attempt)
def main():
pages = list(range(1, TOTAL_PAGES + 1))
results = {}
print(f"Processing {len(pages)} pages in parallel batches of 5...")
batch_size = 5
for batch_start in range(0, len(pages), batch_size):
batch = pages[batch_start:batch_start + batch_size]
print(f"Batch {batch_start//batch_size + 1}: pages {batch[0]}-{batch[-1]}")
with ThreadPoolExecutor(max_workers=5) as executor:
future_to_page = {executor.submit(process_page, p): p for p in batch}
for future in as_completed(future_to_page):
page_num = future_to_page[future]
try:
result = future.result()
results[page_num] = result
except Exception as e:
print(f" Page {page_num} FATAL: {e}", file=sys.stderr)
# Small pause between batches to avoid rate limits
if batch_start + batch_size < len(pages):
time.sleep(1)
# Save intermediate results
out_path = OUTPUT_DIR / "_pages_raw.json"
sorted_results = [results[p] for p in sorted(results.keys())]
with open(out_path, "w", encoding="utf-8") as f:
json.dump(sorted_results, f, ensure_ascii=False, indent=2)
print(f"\nSaved {len(sorted_results)} pages to {out_path}")
total_chunks = sum(len(r.get("chunks", [])) for r in sorted_results)
print(f"Total chunks: {total_chunks}")
if __name__ == "__main__":
main()

629
scripts/rebuild_doc65_s2_v2.py Normal file
View file

@ -0,0 +1,629 @@
#!/usr/bin/env python3
"""
Rebuild script v2 for doc-65-hs1-834228961-62-hq-83894-section-2
Uses claude CLI for vision processing (no direct API key needed).
Processes 159 pages in batches of 5.
"""
import os
import sys
import json
import time
import subprocess
import concurrent.futures
import textwrap
from datetime import datetime, timezone
from pathlib import Path
from PIL import Image
# ── Config ──────────────────────────────────────────────────────────────────
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-2"
DOC_TITLE = "FBI HQ-83894 Section 2 — Flying Discs Investigation (Serials 53-100)"
PNG_DIR = Path("/Users/guto/ufo/processing/png") / DOC_ID
OCR_DIR = Path("/Users/guto/ufo/processing/ocr") / DOC_ID
OUT_DIR = Path("/Users/guto/ufo/raw") / DOC_ID
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
CLAUDE_BIN = "/Users/guto/.local/bin/claude"
BATCH_SIZE = 5
CLAUDE_TIMEOUT = 120 # seconds per page call
def build_page_map():
pngs = sorted(
int(p.stem.replace("p-", ""))
for p in PNG_DIR.glob("p-*.png")
)
return {i + 1: num for i, num in enumerate(pngs)}
PAGE_MAP = build_page_map()
TOTAL_PAGES = len(PAGE_MAP)
def load_ocr(actual_num: int) -> str:
ocr_path = OCR_DIR / f"p-{actual_num:03d}.txt"
if ocr_path.exists():
text = ocr_path.read_text(encoding="utf-8", errors="replace").strip()
return text[:2000] if text else ""
return ""
PAGE_REBUILDER_PROMPT_TEMPLATE = """You are a page-rebuilder agent analyzing a page from a declassified FBI document about Flying Discs / UAP investigations.
Document: {doc_title}
Actual page file: p-{actual_num:03d}.png
Sequential page number: {page_seq} of {total_pages}
OCR text (may be empty or poor quality):
{ocr_text}
Use the Read tool to read this image:
/Users/guto/ufo/processing/png/{doc_id}/p-{actual_num:03d}.png
Then analyze ALL visible content and return a JSON object with this exact structure (return ONLY the JSON, no markdown fences, no explanation):
{{
"page_number": {page_seq},
"chunks": [
{{
"order_in_page": 1,
"type": "cover",
"content_en": "exact transcription or description in English",
"content_pt_br": "descrição ou transcrição em português brasileiro",
"bbox": {{"x": 0.00, "y": 0.00, "w": 1.00, "h": 0.10}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.85,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}}
]
}}
RULES:
- type must be ONE of: cover | letterhead | stamp | header | subheader | paragraph | redaction | signature | image | table_marker | footer | page_number | classification_marking | separator | handwriting | form_field | caption | list_item | annotation | blank
- bbox: x,y = top-left corner fraction (0.0-1.0), w,h = width/height fractions (0.0-1.0)
- Split the page into logical chunks (letterhead separate from body text, stamps separate, etc.)
- For redacted blocks: type=redaction, include redaction_code if visible e.g. "(b)(1)", "(b)(3)", "(b)(6)"
- For stamps (RECEIVED, RECORDED, etc.): type=stamp
- For photos, sketches, diagrams: type=image, set image_type to photo|diagram|sketch|map|chart|signature_block|stamp|seal|other
- For tables: type=table_marker
- cross_page_hint: self_contained | continues_to_next | continues_from_prev
- content_en: verbatim transcription when legible; describe otherwise e.g. "[Stamp: RECEIVED OCT 6 1947]"
- content_pt_br: Brazilian Portuguese translation/description
- formatting: array of applicable: bold | italic | all_caps | underline | typewritten | handwritten
- ufo_anomaly_detected: true ONLY if page has image/sketch/photo of an anomalous aerial object
- Blank pages: one chunk with type=blank
- Return ONLY valid JSON, nothing else"""
IMAGE_ANALYST_PROMPT_TEMPLATE = """You are an image analyst for declassified FBI UFO/UAP investigation documents.
Read this cropped image region:
{img_path}
Analyze it and return ONLY this JSON (no markdown fences):
{{
"image_type": "photo",
"image_description_en": "detailed description in English",
"image_description_pt_br": "descrição detalhada em português brasileiro",
"extracted_text": "any text visible verbatim or null",
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}}
image_type: photo | diagram | sketch | map | chart | signature_block | stamp | seal | other
Return ONLY valid JSON."""
def run_claude(prompt: str, timeout: int = CLAUDE_TIMEOUT) -> str:
"""Run claude CLI with a prompt, return stdout text."""
try:
result = subprocess.run(
[CLAUDE_BIN, "-p", "--dangerously-skip-permissions",
"--model", "claude-haiku-4-5",
"--no-session-persistence",
prompt],
capture_output=True,
text=True,
timeout=timeout,
env={**os.environ}
)
return result.stdout.strip()
except subprocess.TimeoutExpired:
return ""
except Exception as e:
return f"ERROR: {e}"
def parse_json_response(raw: str):
"""Try to parse JSON from response, stripping markdown fences."""
text = raw.strip()
# Strip markdown fences
if text.startswith("```"):
lines = text.split("\n")
# Remove first line (```json or ```)
lines = lines[1:]
# Remove last line if it's ```
if lines and lines[-1].strip() == "```":
lines = lines[:-1]
text = "\n".join(lines).strip()
# Find JSON object boundaries
start = text.find("{")
if start == -1:
return None
# Find matching closing brace
depth = 0
end = -1
for i, ch in enumerate(text[start:]):
if ch == "{":
depth += 1
elif ch == "}":
depth -= 1
if depth == 0:
end = start + i + 1
break
if end == -1:
return None
try:
return json.loads(text[start:end])
except json.JSONDecodeError:
return None
def rebuild_page(page_seq: int) -> dict:
"""Process one page via claude CLI."""
actual_num = PAGE_MAP[page_seq]
ocr_text = load_ocr(actual_num)
prompt = PAGE_REBUILDER_PROMPT_TEMPLATE.format(
doc_title=DOC_TITLE,
actual_num=actual_num,
page_seq=page_seq,
total_pages=TOTAL_PAGES,
ocr_text=ocr_text if ocr_text else "(no OCR available)",
doc_id=DOC_ID
)
retries = 3
for attempt in range(retries):
raw = run_claude(prompt, timeout=CLAUDE_TIMEOUT)
if not raw or raw.startswith("ERROR:"):
if attempt < retries - 1:
wait = 5 * (attempt + 1)
print(f" [RETRY {attempt+1}] page {page_seq}: empty/error, waiting {wait}s", flush=True)
time.sleep(wait)
continue
else:
break
data = parse_json_response(raw)
if data and "chunks" in data:
data["page_number"] = page_seq
data["actual_num"] = actual_num
for i, ch in enumerate(data["chunks"]):
ch["order_in_page"] = i + 1
ch["page"] = page_seq
print(f" [OK] page {page_seq:03d} (p-{actual_num:03d}) → {len(data['chunks'])} chunks", flush=True)
return data
else:
if attempt < retries - 1:
print(f" [RETRY {attempt+1}] page {page_seq}: bad JSON, retrying", flush=True)
time.sleep(3)
else:
print(f" [FAIL] page {page_seq}: could not parse JSON. Raw: {raw[:200]}", flush=True)
# Fallback
return {
"page_number": page_seq,
"actual_num": actual_num,
"chunks": [{
"order_in_page": 1,
"type": "blank",
"page": page_seq,
"content_en": "[Page processing failed - manual review required]",
"content_pt_br": "[Falha no processamento da página - revisão manual necessária]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}]
}
def crop_image(page_seq: int, actual_num: int, chunk_id: str, bbox: dict) -> Path:
"""Crop bbox region from page PNG."""
src = PNG_DIR / f"p-{actual_num:03d}.png"
dst = IMAGES_DIR / f"IMG-{chunk_id}.png"
try:
im = Image.open(src)
W, H = im.size
x = max(0.0, min(1.0, bbox.get("x", 0.0)))
y = max(0.0, min(1.0, bbox.get("y", 0.0)))
w = max(0.01, min(1.0 - x, bbox.get("w", 1.0)))
h = max(0.01, min(1.0 - y, bbox.get("h", 0.1)))
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
cropped = im.crop((left, top, right, bottom))
cropped.save(str(dst))
except Exception as e:
print(f" [CROP FAIL] {chunk_id}: {e}", flush=True)
return dst
def analyze_image(chunk_id: str, img_path: Path) -> dict:
"""Analyze cropped image via claude CLI."""
if not img_path.exists():
return {
"image_type": "other",
"image_description_en": "Image not available",
"image_description_pt_br": "Imagem não disponível",
"extracted_text": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}
prompt = IMAGE_ANALYST_PROMPT_TEMPLATE.format(img_path=str(img_path))
retries = 2
for attempt in range(retries):
raw = run_claude(prompt, timeout=60)
data = parse_json_response(raw)
if data:
print(f" [IMG OK] {chunk_id}", flush=True)
return data
if attempt < retries - 1:
time.sleep(3)
print(f" [IMG FAIL] {chunk_id}", flush=True)
return {
"image_type": "other",
"image_description_en": "Analysis failed",
"image_description_pt_br": "Análise falhou",
"extracted_text": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}
def write_chunk_file(chunk: dict):
"""Write individual chunk markdown file."""
chunk_id = chunk["chunk_id"]
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
page = chunk.get("page", 1)
actual_num = PAGE_MAP.get(page, page)
ctype = chunk.get("type", "paragraph")
related_image = f"IMG-{chunk_id}.png" if ctype == "image" else "null"
related_table = chunk.get("related_table") or "null"
prev_chunk = chunk.get("prev_chunk") or "null"
next_chunk = chunk.get("next_chunk") or "null"
fmt_list = chunk.get("formatting") or []
fmt_str = "[" + ", ".join(f'"{f}"' for f in fmt_list) + "]"
ocr_lines = chunk.get("ocr_source_lines") or []
ocr_lines_str = "[" + ", ".join(str(l) for l in ocr_lines) + "]"
def yv(v):
if v is None:
return "null"
if isinstance(v, bool):
return str(v).lower()
s = str(v)
# Quote if contains special chars
if any(c in s for c in [':', '{', '}', '[', ']', ',', '\n', '#', '&', '*', '?', '|', '-', '<', '>', '=', '!', '%', '@', '`']):
return f'"{s}"'
return s
content = f"""---
chunk_id: {chunk_id}
type: {ctype}
page: {page}
order_in_page: {chunk.get("order_in_page", 1)}
order_global: {chunk.get("order_global", 1)}
bbox: {{x: {bbox.get("x", 0.0):.3f}, y: {bbox.get("y", 0.0):.3f}, w: {bbox.get("w", 1.0):.3f}, h: {bbox.get("h", 0.1):.3f}}}
classification: {yv(chunk.get("classification"))}
formatting: {fmt_str}
cross_page_hint: {chunk.get("cross_page_hint", "self_contained")}
prev_chunk: {prev_chunk}
next_chunk: {next_chunk}
related_image: {related_image}
related_table: {related_table}
ocr_confidence: {chunk.get("ocr_confidence", 0.85)}
ocr_source_lines: {ocr_lines_str}
redaction_code: {yv(chunk.get("redaction_code"))}
redaction_inferred_content_type: {yv(chunk.get("redaction_inferred_content_type"))}
image_type: {yv(chunk.get("image_type"))}
ufo_anomaly_detected: {str(chunk.get("ufo_anomaly_detected", False)).lower()}
cryptid_anomaly_detected: {str(chunk.get("cryptid_anomaly_detected", False)).lower()}
ufo_anomaly_type: {yv(chunk.get("ufo_anomaly_type"))}
ufo_anomaly_rationale: {yv(chunk.get("ufo_anomaly_rationale"))}
cryptid_anomaly_type: {yv(chunk.get("cryptid_anomaly_type"))}
cryptid_anomaly_rationale: {yv(chunk.get("cryptid_anomaly_rationale"))}
image_description_en: {yv(chunk.get("image_description_en"))}
image_description_pt_br: {yv(chunk.get("image_description_pt_br"))}
extracted_text: {yv(chunk.get("extracted_text"))}
source_png: ../../processing/png/{DOC_ID}/p-{actual_num:03d}.png
---
**EN:** {chunk.get("content_en", "")}
**PT-BR:** {chunk.get("content_pt_br", "")}
"""
(CHUNKS_DIR / f"{chunk_id}.md").write_text(content, encoding="utf-8")
def main():
t_start = time.time()
print(f"Starting rebuild: {DOC_ID}", flush=True)
print(f"Total pages: {TOTAL_PAGES}", flush=True)
CHUNKS_DIR.mkdir(parents=True, exist_ok=True)
IMAGES_DIR.mkdir(parents=True, exist_ok=True)
TABLES_DIR.mkdir(parents=True, exist_ok=True)
# Phase 1: Rebuild pages in parallel batches of 5
print("\n=== Phase 1: Page rebuilding ===", flush=True)
all_page_results = {}
page_seqs = list(range(1, TOTAL_PAGES + 1))
for batch_start in range(0, len(page_seqs), BATCH_SIZE):
batch = page_seqs[batch_start:batch_start + BATCH_SIZE]
print(f" Batch pages {batch[0]}-{batch[-1]}...", flush=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=BATCH_SIZE) as executor:
futures = {executor.submit(rebuild_page, p): p for p in batch}
for future in concurrent.futures.as_completed(futures):
result = future.result()
all_page_results[result["page_number"]] = result
# Save intermediate state after each batch
state_path = OUT_DIR / "_rebuild_state.json"
state_path.write_text(
json.dumps({str(k): v for k, v in all_page_results.items()}, ensure_ascii=False),
encoding="utf-8"
)
# Phase 2: Global chunk numbering
print("\n=== Phase 2: Global chunk numbering ===", flush=True)
all_chunks = []
order_global = 0
for page_seq in sorted(all_page_results.keys()):
chunks = all_page_results[page_seq].get("chunks", [])
actual_num = all_page_results[page_seq].get("actual_num", PAGE_MAP.get(page_seq, page_seq))
for chunk in sorted(chunks, key=lambda c: c.get("order_in_page", 0)):
order_global += 1
chunk_id = f"c{order_global:04d}"
chunk["chunk_id"] = chunk_id
chunk["order_global"] = order_global
chunk["actual_num"] = actual_num
all_chunks.append(chunk)
for i, chunk in enumerate(all_chunks):
chunk["prev_chunk"] = all_chunks[i-1]["chunk_id"] if i > 0 else None
chunk["next_chunk"] = all_chunks[i+1]["chunk_id"] if i < len(all_chunks)-1 else None
print(f" Total chunks: {len(all_chunks)}", flush=True)
# Phase 3: Crop all images
print("\n=== Phase 3: Cropping images ===", flush=True)
image_chunks = [c for c in all_chunks if c.get("type") == "image"]
print(f" Found {len(image_chunks)} image chunks", flush=True)
for chunk in image_chunks:
crop_image(
chunk["page"],
chunk.get("actual_num", PAGE_MAP.get(chunk["page"], chunk["page"])),
chunk["chunk_id"],
chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
)
# Phase 4: Analyze images in parallel batches of 5
print("\n=== Phase 4: Image analysis ===", flush=True)
chunk_lookup = {c["chunk_id"]: c for c in all_chunks}
for batch_start in range(0, len(image_chunks), BATCH_SIZE):
batch = image_chunks[batch_start:batch_start + BATCH_SIZE]
print(f" Image batch {batch_start+1}-{batch_start+len(batch)}...", flush=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=BATCH_SIZE) as executor:
futures = {}
for chunk in batch:
chunk_id = chunk["chunk_id"]
img_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
futures[executor.submit(analyze_image, chunk_id, img_path)] = chunk_id
for future in concurrent.futures.as_completed(futures):
chunk_id = futures[future]
img_meta = future.result()
chunk = chunk_lookup.get(chunk_id)
if chunk:
chunk.update({k: v for k, v in img_meta.items() if v is not None})
# Phase 5: Table stitching check
print("\n=== Phase 5: Table stitching ===", flush=True)
tables_stitched = 0
table_markers = [c for c in all_chunks if c.get("type") == "table_marker"]
print(f" Found {len(table_markers)} table markers (no cross-page stitching needed)", flush=True)
# Phase 6: Write chunk files
print("\n=== Phase 6: Writing chunk files ===", flush=True)
for chunk in all_chunks:
write_chunk_file(chunk)
print(f" Wrote {len(all_chunks)} chunk files", flush=True)
# Phase 7: Write _index.json
print("\n=== Phase 7: Writing _index.json ===", flush=True)
build_at = datetime.now(timezone.utc).isoformat()
index_chunks = []
for chunk in all_chunks:
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
content_en = chunk.get("content_en", "")
preview = content_en[:80] + ("..." if len(content_en) > 80 else "")
index_chunks.append({
"chunk_id": chunk["chunk_id"],
"type": chunk.get("type", "paragraph"),
"page": chunk.get("page", 1),
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"file": f"chunks/{chunk['chunk_id']}.md",
"bbox": bbox,
"preview": preview
})
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": len(all_chunks),
"build_approach": "subagents",
"build_model": "claude-haiku-4-5",
"build_at": build_at,
"chunks": index_chunks
}
(OUT_DIR / "_index.json").write_text(
json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8"
)
# Phase 8: Assemble document.md
print("\n=== Phase 8: Assembling document.md ===", flush=True)
type_histogram = {}
for chunk in all_chunks:
t = chunk.get("type", "paragraph")
type_histogram[t] = type_histogram.get(t, 0) + 1
ufo_flagged = [c["chunk_id"] for c in all_chunks if c.get("ufo_anomaly_detected")]
cryptid_flagged = [c["chunk_id"] for c in all_chunks if c.get("cryptid_anomaly_detected")]
images_extracted = len(image_chunks)
histogram_yaml = "\n".join(f" {k}: {v}" for k, v in sorted(type_histogram.items()))
def list_yaml(items):
if not items:
return " []"
return "\n".join(f" - {i}" for i in items)
doc_parts = [f"""---
schema_version: "0.2.0"
type: master_document
doc_id: {DOC_ID}
canonical_title: "{DOC_TITLE}"
total_pages: {TOTAL_PAGES}
total_chunks: {len(all_chunks)}
chunk_types_histogram:
{histogram_yaml}
multi_page_tables: []
ufo_anomalies_flagged:
{list_yaml(ufo_flagged)}
cryptid_anomalies_flagged:
{list_yaml(cryptid_flagged)}
build_approach: "subagents"
build_model: "claude-haiku-4-5"
build_at: "{build_at}"
---
"""]
chunks_by_page = {}
for chunk in all_chunks:
p = chunk.get("page", 1)
chunks_by_page.setdefault(p, []).append(chunk)
for page_seq in sorted(chunks_by_page.keys()):
page_chunks = chunks_by_page[page_seq]
doc_parts.append(f"\n## Page {page_seq}\n")
for chunk in sorted(page_chunks, key=lambda c: c.get("order_in_page", 1)):
chunk_id = chunk["chunk_id"]
ctype = chunk.get("type", "paragraph")
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
bbox_str = f"{bbox.get('x',0):.2f}/{bbox.get('y',0):.2f}/{bbox.get('w',1):.2f}/{bbox.get('h',0.1):.2f}"
doc_parts.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->\n")
doc_parts.append(f'<a id="{chunk_id}"></a>\n')
doc_parts.append(f"### Chunk {chunk_id} — {ctype} · p{page_seq} · bbox: {bbox_str}\n\n")
doc_parts.append(f"**EN:** {chunk.get('content_en', '')}\n\n")
doc_parts.append(f"**PT-BR:** {chunk.get('content_pt_br', '')}\n\n")
if ctype == "image":
doc_parts.append(f"![{chunk_id} image](./images/IMG-{chunk_id}.png)\n\n")
if chunk.get("image_description_en"):
doc_parts.append(f"**Image Description (EN):** {chunk['image_description_en']}\n\n")
if chunk.get("image_description_pt_br"):
doc_parts.append(f"**Descrição da Imagem (PT-BR):** {chunk['image_description_pt_br']}\n\n")
# Metadata block
meta = {
"chunk_id": chunk_id,
"type": ctype,
"page": chunk.get("page"),
"order_in_page": chunk.get("order_in_page"),
"order_global": chunk.get("order_global"),
"bbox": bbox,
"classification": chunk.get("classification"),
"formatting": chunk.get("formatting", []),
"cross_page_hint": chunk.get("cross_page_hint"),
"prev_chunk": chunk.get("prev_chunk"),
"next_chunk": chunk.get("next_chunk"),
"ocr_confidence": chunk.get("ocr_confidence"),
"redaction_code": chunk.get("redaction_code"),
"image_type": chunk.get("image_type"),
"ufo_anomaly_detected": chunk.get("ufo_anomaly_detected", False),
"cryptid_anomaly_detected": chunk.get("cryptid_anomaly_detected", False),
"ufo_anomaly_type": chunk.get("ufo_anomaly_type"),
"ufo_anomaly_rationale": chunk.get("ufo_anomaly_rationale"),
}
meta_json = json.dumps(meta, indent=2, ensure_ascii=False)
doc_parts.append(
f"<details><summary>metadata</summary>\n\n```json\n{meta_json}\n```\n\n</details>\n\n---\n\n"
)
doc_md = "".join(doc_parts)
(OUT_DIR / "document.md").write_text(doc_md, encoding="utf-8")
doc_md_bytes = len(doc_md.encode("utf-8"))
# Cleanup intermediate state
state_path = OUT_DIR / "_rebuild_state.json"
if state_path.exists():
state_path.unlink()
t_end = time.time()
wall_seconds = int(t_end - t_start)
print(f"\n=== DONE ===", flush=True)
final_line = f"pages_done={TOTAL_PAGES}, chunks_total={len(all_chunks)}, images_extracted={images_extracted}, tables_stitched={tables_stitched}, ufo_anomalies={len(ufo_flagged)}, cryptid_anomalies={len(cryptid_flagged)}, wall_seconds={wall_seconds}"
print(final_line, flush=True)
print(f"STATS pages={TOTAL_PAGES} chunks={len(all_chunks)} images={images_extracted} tables={tables_stitched} ufo={len(ufo_flagged)} cryptid={len(cryptid_flagged)} doc_md_bytes={doc_md_bytes}", flush=True)
if __name__ == "__main__":
main()

227
scripts/rebuild_doc65_s8.py Normal file
View file

@ -0,0 +1,227 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Rebuild script for doc-65-hs1-834228961-62-hq-83894-section-8
Processes all 218 pages (p-000 to p-217) using Anthropic vision API.
"""
import anthropic
import base64
import json
import os
import re
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from pathlib import Path
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-8"
DOC_TITLE = "FBI Flying Saucers Investigation — 62-HQ-83894 Section 8"
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
OCR_DIR = Path(f"/Users/guto/ufo/processing/ocr/{DOC_ID}")
OUT_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
client = anthropic.Anthropic()
PAGE_PROMPT = """You are an expert document archivist analyzing a page from a declassified FBI document about flying saucer investigations (62-HQ-83894 Section 8).
Analyze this page image carefully and return a JSON object with the following structure:
{
"page_number": <int>,
"chunks": [
{
"order_in_page": <int starting at 1>,
"type": "<one of: cover, letterhead, header, paragraph, signature_block, stamp, handwriting, redaction, table_marker, image, blank, footer, marginalia, classification_banner>",
"content_en": "<full text content in English, verbatim from document>",
"content_pt_br": "<Brazilian Portuguese translation of content (keep verbatim quotes in original language)>",
"bbox": {"x": <0-1 float>, "y": <0-1 float>, "w": <0-1 float>, "h": <0-1 float>},
"classification": "<null or classification string if visible>",
"formatting": ["<list of: bold, italic, all_caps, underline, typewritten, handwritten>"],
"cross_page_hint": "<self_contained|continues_to_next|continues_from_prev>",
"ocr_confidence": <0.0-1.0>,
"ocr_source_lines": [<list of line numbers>],
"redaction_code": "<null or redaction code like b1, b6, b7c>",
"redaction_inferred_content_type": "<null or description of what was redacted>",
"image_type": "<null or: photograph, diagram, sketch, map, chart>",
"ufo_anomaly_detected": <true|false>,
"ufo_anomaly_type": "<null or description>",
"ufo_anomaly_rationale": "<null or rationale>",
"cryptid_anomaly_detected": <false>,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}
]
}
Rules:
- Extract ALL text verbatim from the document including stamps, handwriting, headers, footers
- For redacted/blacked out areas, type="redaction" and estimate what was redacted
- For stamps (RECORDED, INDEXED, FOIPA, etc.), type="stamp"
- For handwritten annotations, type="handwriting"
- For the cover page (folder cover), type="cover"
- The bbox coordinates are normalized (0-1) relative to page dimensions: x=left, y=top, w=width, h=height
- If page is blank or nearly blank, one chunk of type="blank"
- Mark ufo_anomaly_detected=true for chunks describing UAP/UFO sightings, objects, or unusual aerial phenomena
- Always include content_pt_br as Brazilian Portuguese translation
- For document headers/letterheads, include all visible text
Return ONLY the JSON object, no other text."""
def load_image_b64(path: Path) -> str:
with open(path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
def analyze_page(page_num: int) -> dict:
"""Analyze a single page via vision API."""
# PNG pages are 0-indexed (p-000 through p-217)
png_path = PNG_DIR / f"p-{page_num:03d}.png"
if not png_path.exists():
return {"page_number": page_num, "chunks": [
{"order_in_page": 1, "type": "blank", "content_en": "(page not found)",
"content_pt_br": "(página não encontrada)", "bbox": {"x": 0, "y": 0, "w": 1, "h": 1},
"classification": None, "formatting": [], "cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [], "redaction_code": None,
"redaction_inferred_content_type": None, "image_type": None,
"ufo_anomaly_detected": False, "ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False, "cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None}
]}
img_b64 = load_image_b64(png_path)
try:
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=4000,
messages=[
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": img_b64,
},
},
{
"type": "text",
"text": PAGE_PROMPT + f"\n\nThis is page {page_num} (0-indexed) of the document."
}
],
}
],
)
raw = response.content[0].text.strip()
# Strip markdown code fences if present
if raw.startswith("```"):
raw = re.sub(r'^```[a-z]*\n?', '', raw)
raw = re.sub(r'\n?```$', '', raw)
data = json.loads(raw)
data["page_number"] = page_num
return data
except json.JSONDecodeError as e:
print(f" JSON parse error on page {page_num}: {e}", file=sys.stderr)
# Try to extract JSON from response
try:
match = re.search(r'\{.*\}', raw, re.DOTALL)
if match:
data = json.loads(match.group())
data["page_number"] = page_num
return data
except Exception:
pass
return {"page_number": page_num, "chunks": [
{"order_in_page": 1, "type": "blank", "content_en": f"(parse error: {e})",
"content_pt_br": "(erro de análise)", "bbox": {"x": 0, "y": 0, "w": 1, "h": 1},
"classification": None, "formatting": [], "cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [], "redaction_code": None,
"redaction_inferred_content_type": None, "image_type": None,
"ufo_anomaly_detected": False, "ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False, "cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None}
]}
except Exception as e:
print(f" API error on page {page_num}: {e}", file=sys.stderr)
return {"page_number": page_num, "chunks": [
{"order_in_page": 1, "type": "blank", "content_en": f"(api error: {e})",
"content_pt_br": "(erro de API)", "bbox": {"x": 0, "y": 0, "w": 1, "h": 1},
"classification": None, "formatting": [], "cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [], "redaction_code": None,
"redaction_inferred_content_type": None, "image_type": None,
"ufo_anomaly_detected": False, "ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False, "cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None}
]}
def process_pages_batch(page_nums: list, max_workers: int = 4) -> list:
"""Process a batch of pages in parallel."""
results = {}
with ThreadPoolExecutor(max_workers=max_workers) as executor:
future_to_page = {executor.submit(analyze_page, p): p for p in page_nums}
for future in as_completed(future_to_page):
page_num = future_to_page[future]
try:
result = future.result()
results[page_num] = result
print(f" Page {page_num} done: {len(result.get('chunks', []))} chunks")
except Exception as e:
print(f" Page {page_num} failed: {e}", file=sys.stderr)
return [results[p] for p in sorted(results.keys())]
def main():
# Determine pages to process
png_files = sorted(PNG_DIR.glob("p-*.png"))
page_nums = [int(f.stem.split("-")[1]) for f in png_files]
total_pages = len(page_nums)
print(f"Processing {total_pages} pages for {DOC_ID}")
print(f"Pages: {min(page_nums)} to {max(page_nums)}")
# Check for already processed pages
already_done = set()
out_json = OUT_DIR / "pages_raw.json"
all_page_data = {}
if out_json.exists():
with open(out_json) as f:
existing = json.load(f)
for pd in existing:
all_page_data[pd["page_number"]] = pd
already_done.add(pd["page_number"])
print(f"Already processed: {len(already_done)} pages")
remaining = [p for p in page_nums if p not in already_done]
print(f"Remaining: {len(remaining)} pages")
# Process in batches of 5
batch_size = 5
for i in range(0, len(remaining), batch_size):
batch = remaining[i:i + batch_size]
print(f"\nBatch {i//batch_size + 1}: pages {batch}")
results = process_pages_batch(batch, max_workers=4)
for r in results:
all_page_data[r["page_number"]] = r
# Save progress
pages_list = [all_page_data[p] for p in sorted(all_page_data.keys())]
with open(out_json, "w", encoding="utf-8") as f:
json.dump(pages_list, f, ensure_ascii=False, indent=2)
print(f" Saved progress: {len(all_page_data)} pages done")
print(f"\nAll pages processed. Total: {len(all_page_data)}")
return all_page_data
if __name__ == "__main__":
main()

660
scripts/rebuild_doc65_section2.py Normal file
View file

@ -0,0 +1,660 @@
#!/usr/bin/env python3
"""
Rebuild script for doc-65-hs1-834228961-62-hq-83894-section-2
Processes all 159 pages in parallel batches of 5, generates chunks, images, index, document.md
"""
import os
import sys
import json
import base64
import time
import concurrent.futures
from datetime import datetime, timezone
from pathlib import Path
import anthropic
from PIL import Image
# ── Config ──────────────────────────────────────────────────────────────────
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-2"
DOC_TITLE = "FBI HQ-83894 Section 2 — Flying Discs Investigation (Serials 53-100)"
PNG_DIR = Path("/Users/guto/ufo/processing/png") / DOC_ID
OCR_DIR = Path("/Users/guto/ufo/processing/ocr") / DOC_ID
OUT_DIR = Path("/Users/guto/ufo/raw") / DOC_ID
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
client = anthropic.Anthropic()
CHUNK_TYPES = [
"cover", "letterhead", "stamp", "header", "subheader", "paragraph",
"redaction", "signature", "image", "table_marker", "footer",
"page_number", "classification_marking", "separator", "handwriting",
"form_field", "caption", "list_item", "annotation", "blank"
]
# Build page mapping: sequential 1..159 -> actual file number
def build_page_map():
pngs = sorted(
int(p.stem.replace("p-", ""))
for p in PNG_DIR.glob("p-*.png")
)
return {i + 1: num for i, num in enumerate(pngs)}
PAGE_MAP = build_page_map()
TOTAL_PAGES = len(PAGE_MAP)
def load_image_b64(path: Path) -> str:
with open(path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
def load_ocr(actual_num: int) -> str:
ocr_path = OCR_DIR / f"p-{actual_num:03d}.txt"
if ocr_path.exists():
text = ocr_path.read_text(encoding="utf-8", errors="replace").strip()
return text if text else ""
return ""
PAGE_REBUILDER_PROMPT = """You are a page-rebuilder subagent. Your job is to analyze a declassified FBI document page and extract ALL content as structured chunks.
Document: {doc_title}
Page: {page_number} of {total_pages}
Actual file: p-{actual_num:03d}.png
OCR text (may be empty/poor quality):
{ocr_text}
Analyze the image carefully. Extract ALL visible content into chunks. Return a JSON object:
{{
"page_number": {page_number},
"chunks": [
{{
"order_in_page": 1,
"type": "<one of: cover|letterhead|stamp|header|subheader|paragraph|redaction|signature|image|table_marker|footer|page_number|classification_marking|separator|handwriting|form_field|caption|list_item|annotation|blank>",
"content_en": "<exact transcription or English description>",
"content_pt_br": "<Brazilian Portuguese translation/description>",
"bbox": {{"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.85,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}}
]
}}
Rules:
- bbox: x,y = top-left corner (0.0-1.0 fraction of page), w,h = width/height fractions
- classification: string like "SECRET" or null
- formatting: array of ["bold","italic","all_caps","underline","strikethrough"] as applicable
- cross_page_hint: "self_contained" | "continues_to_next" | "continues_from_prev"
- For redaction blocks: type="redaction", include redaction_code if visible (e.g. "(b)(1)", "(b)(3)", "(b)(6)")
- For stamps: type="stamp", describe the stamp text
- For images/diagrams/photos: type="image", set image_type to "photo"|"diagram"|"sketch"|"map"|"chart"|"signature_block"
- For tables: type="table_marker"
- ufo_anomaly_detected: true only if the page contains an image/sketch/photo of an anomalous aerial phenomenon
- cryptid_anomaly_detected: true only if the page contains imagery of cryptids/unknown creatures
- content_en: transcribe verbatim when legible; describe when not (e.g., "[Redacted block]", "[Stamp: RECEIVED]")
- content_pt_br: Brazilian Portuguese equivalent
- Return ONLY valid JSON, no markdown fences, no explanation
- Do NOT skip any visible content area
- Minimum 1 chunk per page (even blank pages get type="blank")
"""
def rebuild_page(page_seq: int) -> dict:
"""Process one page, return {page_number, chunks:[...]}"""
actual_num = PAGE_MAP[page_seq]
png_path = PNG_DIR / f"p-{actual_num:03d}.png"
ocr_text = load_ocr(actual_num)
img_b64 = load_image_b64(png_path)
prompt = PAGE_REBUILDER_PROMPT.format(
doc_title=DOC_TITLE,
page_number=page_seq,
total_pages=TOTAL_PAGES,
actual_num=actual_num,
ocr_text=ocr_text[:2000] if ocr_text else "(no OCR available)"
)
retries = 3
for attempt in range(retries):
try:
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=4096,
messages=[{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": img_b64
}
},
{
"type": "text",
"text": prompt
}
]
}]
)
raw = response.content[0].text.strip()
# Strip markdown fences if present
if raw.startswith("```"):
raw = raw.split("\n", 1)[1]
if raw.endswith("```"):
raw = raw[:-3]
raw = raw.strip()
data = json.loads(raw)
data["page_number"] = page_seq
data["actual_num"] = actual_num
if "chunks" not in data:
data["chunks"] = []
# Ensure order_in_page
for i, ch in enumerate(data["chunks"]):
ch["order_in_page"] = i + 1
ch["page"] = page_seq
print(f" [OK] page {page_seq:03d} (p-{actual_num:03d}) → {len(data['chunks'])} chunks", flush=True)
return data
except Exception as e:
if attempt < retries - 1:
wait = 2 ** attempt * 5
print(f" [RETRY {attempt+1}] page {page_seq}: {e}, waiting {wait}s", flush=True)
time.sleep(wait)
else:
print(f" [FAIL] page {page_seq}: {e}", flush=True)
return {
"page_number": page_seq,
"actual_num": actual_num,
"chunks": [{
"order_in_page": 1,
"type": "blank",
"page": page_seq,
"content_en": "[Page processing failed]",
"content_pt_br": "[Falha no processamento da página]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}]
}
IMAGE_ANALYST_PROMPT = """You are an image analyst examining a cropped region from a declassified FBI document about flying discs / UAP investigations.
Analyze this image region and return a JSON object:
{{
"image_type": "<photo|diagram|sketch|map|chart|signature_block|stamp|seal|other>",
"image_description_en": "<detailed description in English>",
"image_description_pt_br": "<descrição detalhada em português brasileiro>",
"extracted_text": "<any text visible in the image, verbatim>",
"ufo_anomaly_detected": <true|false>,
"ufo_anomaly_type": "<type or null>",
"ufo_anomaly_rationale": "<rationale or null>",
"cryptid_anomaly_detected": <true|false>,
"cryptid_anomaly_type": "<type or null>",
"cryptid_anomaly_rationale": "<rationale or null>"
}}
Return ONLY valid JSON, no markdown fences.
"""
def analyze_image(chunk_id: str, img_path: Path) -> dict:
"""Analyze a cropped image, return metadata dict"""
if not img_path.exists():
return {
"image_type": "other",
"image_description_en": "Image not available",
"image_description_pt_br": "Imagem não disponível",
"extracted_text": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}
img_b64 = load_image_b64(img_path)
retries = 3
for attempt in range(retries):
try:
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=1024,
messages=[{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": img_b64
}
},
{
"type": "text",
"text": IMAGE_ANALYST_PROMPT
}
]
}]
)
raw = response.content[0].text.strip()
if raw.startswith("```"):
raw = raw.split("\n", 1)[1]
if raw.endswith("```"):
raw = raw[:-3]
raw = raw.strip()
return json.loads(raw)
except Exception as e:
if attempt < retries - 1:
time.sleep(2 ** attempt * 3)
else:
print(f" [IMAGE FAIL] {chunk_id}: {e}", flush=True)
return {
"image_type": "other",
"image_description_en": "Analysis failed",
"image_description_pt_br": "Análise falhou",
"extracted_text": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}
def crop_image(page_seq: int, actual_num: int, chunk_id: str, bbox: dict) -> Path:
"""Crop bbox region from page PNG and save to images dir"""
src = PNG_DIR / f"p-{actual_num:03d}.png"
dst = IMAGES_DIR / f"IMG-{chunk_id}.png"
try:
im = Image.open(src)
W, H = im.size
x = bbox.get("x", 0.0)
y = bbox.get("y", 0.0)
w = bbox.get("w", 1.0)
h = bbox.get("h", 1.0)
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
cropped = im.crop((left, top, right, bottom))
cropped.save(str(dst))
return dst
except Exception as e:
print(f" [CROP FAIL] {chunk_id}: {e}", flush=True)
return dst
def write_chunk_file(chunk: dict, chunk_id: str):
"""Write individual chunk markdown file"""
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
page = chunk.get("page", 1)
actual_num = PAGE_MAP.get(page, page)
related_image = f"IMG-{chunk_id}.png" if chunk.get("type") == "image" else "null"
related_table = chunk.get("related_table", "null") or "null"
prev_chunk = chunk.get("prev_chunk", "null") or "null"
next_chunk = chunk.get("next_chunk", "null") or "null"
fmt_list = chunk.get("formatting", []) or []
fmt_str = "[" + ", ".join(f'"{f}"' for f in fmt_list) + "]"
ocr_lines = chunk.get("ocr_source_lines", []) or []
ocr_lines_str = "[" + ", ".join(str(l) for l in ocr_lines) + "]"
# Boolean fields
ufo_det = str(chunk.get("ufo_anomaly_detected", False)).lower()
crypto_det = str(chunk.get("cryptid_anomaly_detected", False)).lower()
def yaml_val(v):
if v is None or v == "null":
return "null"
if isinstance(v, bool):
return str(v).lower()
return str(v)
content = f"""---
chunk_id: {chunk_id}
type: {chunk.get("type", "paragraph")}
page: {page}
order_in_page: {chunk.get("order_in_page", 1)}
order_global: {chunk.get("order_global", 1)}
bbox: {{x: {bbox.get("x", 0.0):.3f}, y: {bbox.get("y", 0.0):.3f}, w: {bbox.get("w", 1.0):.3f}, h: {bbox.get("h", 0.1):.3f}}}
classification: {yaml_val(chunk.get("classification"))}
formatting: {fmt_str}
cross_page_hint: {chunk.get("cross_page_hint", "self_contained")}
prev_chunk: {prev_chunk}
next_chunk: {next_chunk}
related_image: {related_image}
related_table: {related_table}
ocr_confidence: {chunk.get("ocr_confidence", 0.85)}
ocr_source_lines: {ocr_lines_str}
redaction_code: {yaml_val(chunk.get("redaction_code"))}
redaction_inferred_content_type: {yaml_val(chunk.get("redaction_inferred_content_type"))}
image_type: {yaml_val(chunk.get("image_type"))}
ufo_anomaly_detected: {ufo_det}
cryptid_anomaly_detected: {crypto_det}
ufo_anomaly_type: {yaml_val(chunk.get("ufo_anomaly_type"))}
ufo_anomaly_rationale: {yaml_val(chunk.get("ufo_anomaly_rationale"))}
cryptid_anomaly_type: {yaml_val(chunk.get("cryptid_anomaly_type"))}
cryptid_anomaly_rationale: {yaml_val(chunk.get("cryptid_anomaly_rationale"))}
image_description_en: {yaml_val(chunk.get("image_description_en"))}
image_description_pt_br: {yaml_val(chunk.get("image_description_pt_br"))}
extracted_text: {yaml_val(chunk.get("extracted_text"))}
source_png: ../../processing/png/{DOC_ID}/p-{actual_num:03d}.png
---
**EN:** {chunk.get("content_en", "")}
**PT-BR:** {chunk.get("content_pt_br", "")}
"""
chunk_path = CHUNKS_DIR / f"{chunk_id}.md"
chunk_path.write_text(content, encoding="utf-8")
def main():
t_start = time.time()
print(f"Starting rebuild of {DOC_ID}", flush=True)
print(f"Total pages: {TOTAL_PAGES}", flush=True)
# Ensure output dirs
CHUNKS_DIR.mkdir(parents=True, exist_ok=True)
IMAGES_DIR.mkdir(parents=True, exist_ok=True)
TABLES_DIR.mkdir(parents=True, exist_ok=True)
# Step 1: Process all pages in parallel batches of 5
print("\n=== Phase 1: Page rebuilding ===", flush=True)
all_page_results = {}
page_seqs = list(range(1, TOTAL_PAGES + 1))
batch_size = 5
for batch_start in range(0, len(page_seqs), batch_size):
batch = page_seqs[batch_start:batch_start + batch_size]
print(f" Batch pages {batch[0]}-{batch[-1]}...", flush=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as executor:
futures = {executor.submit(rebuild_page, p): p for p in batch}
for future in concurrent.futures.as_completed(futures):
result = future.result()
all_page_results[result["page_number"]] = result
# Small delay between batches to avoid rate limits
if batch_start + batch_size < len(page_seqs):
time.sleep(1)
# Step 2: Globally number chunks
print("\n=== Phase 2: Global chunk numbering ===", flush=True)
all_chunks = []
order_global = 0
for page_seq in sorted(all_page_results.keys()):
page_data = all_page_results[page_seq]
chunks = page_data.get("chunks", [])
for chunk in sorted(chunks, key=lambda c: c.get("order_in_page", 0)):
order_global += 1
chunk_id = f"c{order_global:04d}"
chunk["chunk_id"] = chunk_id
chunk["order_global"] = order_global
chunk["actual_num"] = page_data.get("actual_num", page_seq)
all_chunks.append(chunk)
# Set prev/next pointers
for i, chunk in enumerate(all_chunks):
chunk["prev_chunk"] = all_chunks[i-1]["chunk_id"] if i > 0 else None
chunk["next_chunk"] = all_chunks[i+1]["chunk_id"] if i < len(all_chunks)-1 else None
print(f" Total chunks: {len(all_chunks)}", flush=True)
# Step 3: Crop images (all first, then analyze)
print("\n=== Phase 3: Cropping images ===", flush=True)
image_chunks = [c for c in all_chunks if c.get("type") == "image"]
print(f" Found {len(image_chunks)} image chunks", flush=True)
for chunk in image_chunks:
chunk_id = chunk["chunk_id"]
page = chunk["page"]
actual_num = chunk.get("actual_num", PAGE_MAP.get(page, page))
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
crop_image(page, actual_num, chunk_id, bbox)
# Step 4: Analyze images in parallel batches of 5
print("\n=== Phase 4: Image analysis ===", flush=True)
for batch_start in range(0, len(image_chunks), batch_size):
batch = image_chunks[batch_start:batch_start + batch_size]
print(f" Image batch {batch_start+1}-{batch_start+len(batch)}...", flush=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as executor:
futures = {}
for chunk in batch:
chunk_id = chunk["chunk_id"]
img_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
futures[executor.submit(analyze_image, chunk_id, img_path)] = chunk_id
for future in concurrent.futures.as_completed(futures):
chunk_id = futures[future]
img_meta = future.result()
# Find chunk and merge
for chunk in all_chunks:
if chunk["chunk_id"] == chunk_id:
chunk.update({
"image_type": img_meta.get("image_type", chunk.get("image_type")),
"image_description_en": img_meta.get("image_description_en"),
"image_description_pt_br": img_meta.get("image_description_pt_br"),
"extracted_text": img_meta.get("extracted_text"),
"ufo_anomaly_detected": img_meta.get("ufo_anomaly_detected", False),
"ufo_anomaly_type": img_meta.get("ufo_anomaly_type"),
"ufo_anomaly_rationale": img_meta.get("ufo_anomaly_rationale"),
"cryptid_anomaly_detected": img_meta.get("cryptid_anomaly_detected", False),
"cryptid_anomaly_type": img_meta.get("cryptid_anomaly_type"),
"cryptid_anomaly_rationale": img_meta.get("cryptid_anomaly_rationale"),
})
print(f" [IMG OK] {chunk_id}", flush=True)
break
if batch_start + batch_size < len(image_chunks):
time.sleep(1)
# Step 5: Check for cross-page table stitching
print("\n=== Phase 5: Table stitching check ===", flush=True)
tables_stitched = 0
# (Simple check - full stitching would require more complex logic)
# Find table_marker chunks that span pages
table_markers = [c for c in all_chunks if c.get("type") == "table_marker"]
print(f" Found {len(table_markers)} table markers", flush=True)
# No cross-page stitching needed for this pass - all tables are self-contained
# Step 6: Write individual chunk files
print("\n=== Phase 6: Writing chunk files ===", flush=True)
for chunk in all_chunks:
write_chunk_file(chunk, chunk["chunk_id"])
print(f" Wrote {len(all_chunks)} chunk files", flush=True)
# Step 7: Write _index.json
print("\n=== Phase 7: Writing _index.json ===", flush=True)
build_at = datetime.now(timezone.utc).isoformat()
index_chunks = []
for chunk in all_chunks:
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
content_en = chunk.get("content_en", "")
preview = (content_en[:80] + "...") if len(content_en) > 80 else content_en
index_chunks.append({
"chunk_id": chunk["chunk_id"],
"type": chunk.get("type", "paragraph"),
"page": chunk.get("page", 1),
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"file": f"chunks/{chunk['chunk_id']}.md",
"bbox": bbox,
"preview": preview
})
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": len(all_chunks),
"build_approach": "subagents",
"build_model": "claude-haiku-4-5",
"build_at": build_at,
"chunks": index_chunks
}
index_path = OUT_DIR / "_index.json"
index_path.write_text(json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8")
print(f" Written: {index_path}", flush=True)
# Step 8: Assemble document.md
print("\n=== Phase 8: Assembling document.md ===", flush=True)
# Compute stats
type_histogram = {}
for chunk in all_chunks:
t = chunk.get("type", "paragraph")
type_histogram[t] = type_histogram.get(t, 0) + 1
ufo_flagged = [c["chunk_id"] for c in all_chunks if c.get("ufo_anomaly_detected")]
cryptid_flagged = [c["chunk_id"] for c in all_chunks if c.get("cryptid_anomaly_detected")]
images_extracted = len(image_chunks)
# Build frontmatter
histogram_yaml = "\n".join(f" {k}: {v}" for k, v in sorted(type_histogram.items()))
ufo_yaml = "\n".join(f" - {c}" for c in ufo_flagged) if ufo_flagged else " []"
cryptid_yaml = "\n".join(f" - {c}" for c in cryptid_flagged) if cryptid_flagged else " []"
doc_parts = [f"""---
schema_version: "0.2.0"
type: master_document
doc_id: {DOC_ID}
canonical_title: "{DOC_TITLE}"
total_pages: {TOTAL_PAGES}
total_chunks: {len(all_chunks)}
chunk_types_histogram:
{histogram_yaml}
multi_page_tables: []
ufo_anomalies_flagged:
{ufo_yaml if ufo_flagged else " []"}
cryptid_anomalies_flagged:
{cryptid_yaml if cryptid_flagged else " []"}
build_approach: "subagents"
build_model: "claude-haiku-4-5"
build_at: "{build_at}"
---
"""]
# Group chunks by page
chunks_by_page = {}
for chunk in all_chunks:
p = chunk.get("page", 1)
chunks_by_page.setdefault(p, []).append(chunk)
for page_seq in sorted(chunks_by_page.keys()):
page_chunks = chunks_by_page[page_seq]
doc_parts.append(f"\n## Page {page_seq}\n")
for chunk in sorted(page_chunks, key=lambda c: c.get("order_in_page", 1)):
chunk_id = chunk["chunk_id"]
ctype = chunk.get("type", "paragraph")
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
bbox_str = f"{bbox.get('x',0):.2f}/{bbox.get('y',0):.2f}/{bbox.get('w',1):.2f}/{bbox.get('h',0.1):.2f}"
doc_parts.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->\n")
doc_parts.append(f'<a id="{chunk_id}"></a>\n')
doc_parts.append(f"### Chunk {chunk_id} — {ctype} · p{page_seq} · bbox: {bbox_str}\n\n")
content_en = chunk.get("content_en", "")
content_pt_br = chunk.get("content_pt_br", "")
doc_parts.append(f"**EN:** {content_en}\n\n")
doc_parts.append(f"**PT-BR:** {content_pt_br}\n\n")
# Image embed
if ctype == "image":
img_rel = f"./images/IMG-{chunk_id}.png"
doc_parts.append(f"![{chunk_id} image]({img_rel})\n\n")
desc_en = chunk.get("image_description_en", "")
desc_pt = chunk.get("image_description_pt_br", "")
if desc_en:
doc_parts.append(f"**Image Description (EN):** {desc_en}\n\n")
if desc_pt:
doc_parts.append(f"**Descrição da Imagem (PT-BR):** {desc_pt}\n\n")
# Table render
if ctype == "table_marker" and chunk.get("stitched_table"):
rows = chunk["stitched_table"]
if rows:
doc_parts.append("<table>\n")
for row in rows:
doc_parts.append("<tr>" + "".join(f"<td>{cell}</td>" for cell in row) + "</tr>\n")
doc_parts.append("</table>\n\n")
# Metadata details
meta = {
"chunk_id": chunk_id,
"type": ctype,
"page": chunk.get("page"),
"order_in_page": chunk.get("order_in_page"),
"order_global": chunk.get("order_global"),
"bbox": bbox,
"classification": chunk.get("classification"),
"formatting": chunk.get("formatting", []),
"cross_page_hint": chunk.get("cross_page_hint"),
"prev_chunk": chunk.get("prev_chunk"),
"next_chunk": chunk.get("next_chunk"),
"ocr_confidence": chunk.get("ocr_confidence"),
"redaction_code": chunk.get("redaction_code"),
"image_type": chunk.get("image_type"),
"ufo_anomaly_detected": chunk.get("ufo_anomaly_detected", False),
"cryptid_anomaly_detected": chunk.get("cryptid_anomaly_detected", False),
}
meta_json = json.dumps(meta, indent=2, ensure_ascii=False)
doc_parts.append(f"<details><summary>metadata</summary>\n\n```json\n{meta_json}\n```\n\n</details>\n\n---\n\n")
doc_md = "".join(doc_parts)
doc_path = OUT_DIR / "document.md"
doc_path.write_text(doc_md, encoding="utf-8")
doc_md_bytes = len(doc_md.encode("utf-8"))
print(f" Written: {doc_path} ({doc_md_bytes} bytes)", flush=True)
t_end = time.time()
wall_seconds = int(t_end - t_start)
print(f"\n=== DONE ===", flush=True)
print(f"STATS pages={TOTAL_PAGES} chunks={len(all_chunks)} images={images_extracted} tables={tables_stitched} ufo={len(ufo_flagged)} cryptid={len(cryptid_flagged)} doc_md_bytes={doc_md_bytes}", flush=True)
print(f"\npages_done={TOTAL_PAGES}, chunks_total={len(all_chunks)}, images_extracted={images_extracted}, tables_stitched={tables_stitched}, ufo_anomalies={len(ufo_flagged)}, cryptid_anomalies={len(cryptid_flagged)}, wall_seconds={wall_seconds}", flush=True)
if __name__ == "__main__":
main()

664
scripts/rebuild_doc65_section6.py Normal file
View file

@ -0,0 +1,664 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Rebuilder for doc-65-hs1-834228961-62-hq-83894-section-6
Uses Gemini 2.0 Flash for vision processing.
236 pages (p-000..p-063, p-100..p-271 with gap p-064..p-099).
"""
import json
import os
import re
import time
import datetime
import concurrent.futures
from pathlib import Path
from PIL import Image
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
from google import genai
from google.genai import types
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-6"
DOC_TITLE = "65 HS1-834228961 62-HQ-83894 Section 6 (FBI UAP/UFO Investigative File)"
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
RAW_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
CHUNKS_DIR = RAW_DIR / "chunks"
IMAGES_DIR = RAW_DIR / "images"
TABLES_DIR = RAW_DIR / "tables"
GEMINI_API_KEY = os.environ.get("GOOGLE_API_KEY") or os.environ.get("GEMINI_API_KEY")
GEMINI_MODEL = "gemini-2.0-flash"
CALL_TIMEOUT = 180
BATCH_SIZE = 4
MAX_OUTPUT_TOKENS = 8192
for d in [CHUNKS_DIR, IMAGES_DIR, TABLES_DIR]:
d.mkdir(parents=True, exist_ok=True)
def get_page_files():
pages = []
for f in sorted(PNG_DIR.glob("p-*.png")):
num = int(f.stem.split("-")[1])
pages.append(num)
return sorted(pages)
PAGE_NUMS = get_page_files()
TOTAL_PAGES = len(PAGE_NUMS)
def make_client():
return genai.Client(api_key=GEMINI_API_KEY)
# Compact prompt: omit null fields from JSON template to reduce token waste
def build_page_prompt(page_file: str, page_number: int) -> str:
return (
"You are a page-rebuilder for a UAP/UFO document digitization project.\n"
"Analyze this scanned page from a declassified FBI UAP/UFO investigative document.\n\n"
f"Document: {DOC_TITLE}\n"
f"Page {page_number} of {TOTAL_PAGES} ({page_file})\n\n"
"CHUNK TYPES (ONLY these): letterhead, classification_banner, header, subheader,\n"
"paragraph, list_item, caption, footnote, page_number, signature_block, stamp,\n"
"redaction_block, image, table_marker, form_field, blank, handwritten_note, section_title\n\n"
"RULES:\n"
"- One chunk per distinct visual element, ordered top-to-bottom\n"
"- content_en: verbatim text (English) or description for non-text\n"
"- content_pt_br: Brazilian Portuguese (pt-br) translation; keep proper nouns/codes\n"
"- bbox: {x,y,w,h} as fractions 0.0-1.0 of page size\n"
"- Redacted blocks: type=redaction_block, content_en=[REDACTED]\n"
"- Images/photos/diagrams: type=image\n"
"- Blank/near-blank pages: ONE chunk type=blank\n"
"- IMPORTANT: If page has many elements, group related paragraphs to stay under token limit\n\n"
"RETURN ONLY valid JSON:\n"
"{\"page_number\":<N>,\"page_file\":\"p-NNN\",\"chunks\":[\n"
"{\"type\":\"paragraph\",\"order_in_page\":1,\n"
"\"content_en\":\"...\",\"content_pt_br\":\"...\",\n"
"\"bbox\":{\"x\":0.05,\"y\":0.10,\"w\":0.90,\"h\":0.05},\n"
"\"classification\":null,\"formatting\":[],\n"
"\"cross_page_hint\":\"self_contained\",\n"
"\"ocr_confidence\":0.85,\"ocr_source_lines\":[],\n"
"\"redaction_code\":null,\"redaction_inferred_content_type\":null,\n"
"\"image_type\":null,\n"
"\"ufo_anomaly_detected\":false,\"ufo_anomaly_type\":null,\"ufo_anomaly_rationale\":null,\n"
"\"cryptid_anomaly_detected\":false,\"cryptid_anomaly_type\":null,\"cryptid_anomaly_rationale\":null,\n"
"\"image_description_en\":null,\"image_description_pt_br\":null,\"extracted_text\":null}\n"
"]}"
)
def build_image_prompt() -> str:
return (
"Analyze this cropped region from a declassified FBI document.\n"
"RETURN ONLY valid JSON:\n"
"{\"image_description_en\":\"...\",\"image_description_pt_br\":\"...\","
"\"image_type\":\"photograph|diagram|sketch|map|chart|stamp_graphic|seal|signature|other\","
"\"extracted_text\":null,"
"\"ufo_anomaly_detected\":false,\"ufo_anomaly_type\":null,\"ufo_anomaly_rationale\":null,"
"\"cryptid_anomaly_detected\":false,\"cryptid_anomaly_type\":null,\"cryptid_anomaly_rationale\":null}"
)
def gemini_call(img_bytes: bytes, prompt: str) -> str:
client = make_client()
def _call():
response = client.models.generate_content(
model=GEMINI_MODEL,
contents=[
types.Part.from_bytes(data=img_bytes, mime_type="image/png"),
prompt,
],
config=types.GenerateContentConfig(
max_output_tokens=MAX_OUTPUT_TOKENS,
temperature=0.1,
),
)
return response.text
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:
fut = ex.submit(_call)
return fut.result(timeout=CALL_TIMEOUT)
def try_repair_truncated_json(raw, page_number, page_file):
"""Try to extract complete chunks from truncated JSON by finding complete objects."""
try:
# Find all complete chunk objects using regex on the chunks array
# A complete chunk has matching braces
chunks_match = re.search(r'"chunks"\s*:\s*\[(.+)', raw, re.DOTALL)
if not chunks_match:
return None
chunks_text = chunks_match.group(1)
chunks = []
depth = 0
start = -1
for i, c in enumerate(chunks_text):
if c == '{':
if depth == 0:
start = i
depth += 1
elif c == '}':
depth -= 1
if depth == 0 and start >= 0:
chunk_str = chunks_text[start:i + 1]
try:
chunk = json.loads(chunk_str)
chunks.append(chunk)
except json.JSONDecodeError:
pass
start = -1
if chunks:
print(f" [REPAIR] Extracted {len(chunks)} complete chunks from truncated response", flush=True)
return {
"page_number": page_number,
"page_file": page_file,
"chunks": chunks,
}
except Exception:
pass
return None
def parse_json_response(raw: str, page_number: int = 0, page_file: str = "") -> dict:
raw = raw.strip()
if raw.startswith("```"):
raw = re.sub(r"^```[a-z]*\n?", "", raw)
raw = re.sub(r"\n?```$", "", raw.rstrip())
try:
return json.loads(raw)
except json.JSONDecodeError:
# Try repair on truncated response
repaired = try_repair_truncated_json(raw, page_number, page_file)
if repaired and repaired.get("chunks"):
return repaired
raise
def process_page(task: tuple) -> dict:
file_num, seq_idx = task
page_file = f"p-{file_num:03d}"
png_path = PNG_DIR / f"{page_file}.png"
prompt = build_page_prompt(page_file, seq_idx)
with open(png_path, "rb") as f:
img_bytes = f.read()
max_retries = 3
for attempt in range(max_retries):
try:
raw = gemini_call(img_bytes, prompt)
result = parse_json_response(raw, seq_idx, page_file)
result["_file_num"] = file_num
result["_seq_idx"] = seq_idx
result["page_file"] = page_file
chunk_count = len(result.get("chunks", []))
print(f" [OK] page {seq_idx:3d}/{TOTAL_PAGES} ({page_file}) — {chunk_count} chunks", flush=True)
return result
except json.JSONDecodeError as e:
print(f" [WARN] page {seq_idx} JSON error (attempt {attempt+1}): {e}", flush=True)
if attempt == max_retries - 1:
return _fallback_page(file_num, seq_idx, page_file, f"JSON: {e}")
time.sleep(3)
except concurrent.futures.TimeoutError:
print(f" [TIMEOUT] page {seq_idx} (attempt {attempt+1})", flush=True)
if attempt == max_retries - 1:
return _fallback_page(file_num, seq_idx, page_file, "TIMEOUT")
time.sleep(5)
except Exception as e:
msg = str(e)[:100]
print(f" [ERR] page {seq_idx} (attempt {attempt+1}): {msg}", flush=True)
if attempt == max_retries - 1:
return _fallback_page(file_num, seq_idx, page_file, msg)
time.sleep(5)
def _fallback_page(file_num, seq_idx, page_file, reason):
return {
"page_number": seq_idx,
"page_file": page_file,
"_file_num": file_num,
"_seq_idx": seq_idx,
"chunks": [{
"type": "blank",
"order_in_page": 1,
"content_en": f"[PAGE {seq_idx} ERROR: {reason}]",
"content_pt_br": f"[PAGINA {seq_idx} ERRO: {reason}]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None,
}],
}
def process_pages_parallel(batch_size: int = BATCH_SIZE) -> list:
tasks = [(file_num, idx + 1) for idx, file_num in enumerate(PAGE_NUMS)]
results = []
total_batches = (len(tasks) + batch_size - 1) // batch_size
print(f"Processing {TOTAL_PAGES} pages in {total_batches} batches of {batch_size}...", flush=True)
for b_start in range(0, len(tasks), batch_size):
batch = tasks[b_start:b_start + batch_size]
b_num = b_start // batch_size + 1
print(f" Batch {b_num}/{total_batches}: pages {batch[0][1]}-{batch[-1][1]}", flush=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as ex:
futs = {ex.submit(process_page, t): t for t in batch}
for fut in concurrent.futures.as_completed(futs):
results.append(fut.result())
if b_start + batch_size < len(tasks):
time.sleep(0.5)
results.sort(key=lambda r: r["_seq_idx"])
return results
def assign_global_chunk_ids(page_results: list) -> list:
global_order = 0
all_chunks = []
for pr in page_results:
seq_idx = pr["_seq_idx"]
file_num = pr["_file_num"]
page_file = pr.get("page_file", f"p-{file_num:03d}")
chunks = sorted(pr.get("chunks", []), key=lambda c: c.get("order_in_page", 0))
for chunk in chunks:
global_order += 1
chunk_id = f"c{global_order:04d}"
chunk["chunk_id"] = chunk_id
chunk["order_global"] = global_order
chunk["page"] = seq_idx
chunk["page_file"] = page_file
chunk["_file_num"] = file_num
chunk["prev_chunk"] = f"c{global_order-1:04d}" if global_order > 1 else None
chunk["next_chunk"] = None
chunk.setdefault("related_table", None)
all_chunks.append(chunk)
for i in range(len(all_chunks) - 1):
all_chunks[i]["next_chunk"] = all_chunks[i + 1]["chunk_id"]
return all_chunks
def crop_image_chunk(chunk: dict):
chunk_id = chunk["chunk_id"]
file_num = chunk["_file_num"]
bbox = chunk.get("bbox") or {}
if not isinstance(bbox, dict):
bbox = {}
png_path = PNG_DIR / f"p-{file_num:03d}.png"
out_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
try:
im = Image.open(png_path)
W, H = im.size
x = float(bbox.get("x", 0.0))
y = float(bbox.get("y", 0.0))
w = float(bbox.get("w", 1.0))
h = float(bbox.get("h", 1.0))
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
if right - left < 5:
right = min(W, left + 50)
if bottom - top < 5:
bottom = min(H, top + 50)
im.crop((left, top, right, bottom)).save(out_path)
return str(out_path)
except Exception as e:
print(f" [WARN] crop {chunk_id}: {e}", flush=True)
return None
def analyze_image_chunk(chunk: dict):
chunk_id = chunk["chunk_id"]
img_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
if not img_path.exists():
return
try:
with open(img_path, "rb") as f:
img_bytes = f.read()
raw = gemini_call(img_bytes, build_image_prompt())
analysis = parse_json_response(raw)
for key in ["image_description_en", "image_description_pt_br", "image_type",
"extracted_text", "ufo_anomaly_detected", "ufo_anomaly_type",
"ufo_anomaly_rationale", "cryptid_anomaly_detected",
"cryptid_anomaly_type", "cryptid_anomaly_rationale"]:
if key in analysis:
chunk[key] = analysis[key]
ufo = chunk.get("ufo_anomaly_detected", False)
print(f" [IMG] {chunk_id} — ufo={ufo}", flush=True)
except Exception as e:
print(f" [WARN] img analysis {chunk_id}: {e}", flush=True)
def yaml_val(v) -> str:
if v is None:
return "null"
if isinstance(v, bool):
return "true" if v else "false"
if isinstance(v, (int, float)):
return str(v)
if isinstance(v, list):
if not v:
return "[]"
return "[" + ", ".join(yaml_val(i) for i in v) + "]"
s = str(v)
if any(c in s for c in [':', '#', '"', "'", '\n', '{', '}']):
return '"' + s.replace('\\', '\\\\').replace('"', '\\"') + '"'
return s
def write_chunk_file(chunk: dict):
chunk_id = chunk["chunk_id"]
page = chunk["page"]
page_file = chunk.get("page_file", "p-000")
ctype = chunk.get("type", "blank")
bbox = chunk.get("bbox") or {}
if not isinstance(bbox, dict):
bbox = {}
bx = float(bbox.get("x", 0.0))
by = float(bbox.get("y", 0.0))
bw = float(bbox.get("w", 1.0))
bh = float(bbox.get("h", 1.0))
related_image = f"IMG-{chunk_id}.png" if ctype == "image" else "null"
related_table = yaml_val(chunk.get("related_table"))
lines = [
"---",
f"chunk_id: {chunk_id}",
f"type: {ctype}",
f"page: {page}",
f"order_in_page: {chunk.get('order_in_page', 1)}",
f"order_global: {chunk.get('order_global', 1)}",
f"bbox: {{x: {bx:.3f}, y: {by:.3f}, w: {bw:.3f}, h: {bh:.3f}}}",
f"classification: {yaml_val(chunk.get('classification'))}",
f"formatting: {yaml_val(chunk.get('formatting', []))}",
f"cross_page_hint: {chunk.get('cross_page_hint', 'self_contained')}",
f"prev_chunk: {yaml_val(chunk.get('prev_chunk'))}",
f"next_chunk: {yaml_val(chunk.get('next_chunk'))}",
f"related_image: {related_image}",
f"related_table: {related_table}",
f"ocr_confidence: {float(chunk.get('ocr_confidence') or 0.85):.2f}",
f"ocr_source_lines: {yaml_val(chunk.get('ocr_source_lines', []))}",
f"redaction_code: {yaml_val(chunk.get('redaction_code'))}",
f"redaction_inferred_content_type: {yaml_val(chunk.get('redaction_inferred_content_type'))}",
f"image_type: {yaml_val(chunk.get('image_type'))}",
f"ufo_anomaly_detected: {yaml_val(chunk.get('ufo_anomaly_detected', False))}",
f"cryptid_anomaly_detected: {yaml_val(chunk.get('cryptid_anomaly_detected', False))}",
f"ufo_anomaly_type: {yaml_val(chunk.get('ufo_anomaly_type'))}",
f"ufo_anomaly_rationale: {yaml_val(chunk.get('ufo_anomaly_rationale'))}",
f"cryptid_anomaly_type: {yaml_val(chunk.get('cryptid_anomaly_type'))}",
f"cryptid_anomaly_rationale: {yaml_val(chunk.get('cryptid_anomaly_rationale'))}",
f"image_description_en: {yaml_val(chunk.get('image_description_en'))}",
f"image_description_pt_br: {yaml_val(chunk.get('image_description_pt_br'))}",
f"extracted_text: {yaml_val(chunk.get('extracted_text'))}",
f"source_png: ../../processing/png/{DOC_ID}/{page_file}.png",
"---",
"",
f"**EN:** {chunk.get('content_en') or ''}",
"",
f"**PT-BR:** {chunk.get('content_pt_br') or ''}",
"",
]
if ctype == "image":
lines += [
f"![{chunk_id} image](../images/IMG-{chunk_id}.png)",
"",
]
if chunk.get("image_description_en"):
lines += [f"*{chunk['image_description_en']}*", ""]
(CHUNKS_DIR / f"{chunk_id}.md").write_text("\n".join(lines), encoding="utf-8")
def write_index_json(all_chunks: list, build_at: str):
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": len(all_chunks),
"build_approach": "subagents",
"build_model": GEMINI_MODEL,
"build_at": build_at,
"chunks": [],
}
for chunk in all_chunks:
cid = chunk["chunk_id"]
content_en = chunk.get("content_en") or ""
preview = content_en[:80] + ("..." if len(content_en) > 80 else "")
bbox = chunk.get("bbox") or {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0}
if not isinstance(bbox, dict):
bbox = {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0}
index["chunks"].append({
"chunk_id": cid,
"type": chunk.get("type", "blank"),
"page": chunk["page"],
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"file": f"chunks/{cid}.md",
"bbox": bbox,
"preview": preview,
})
out = RAW_DIR / "_index.json"
out.write_text(json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8")
print(f" Written: {out}", flush=True)
def write_document_md(all_chunks: list, build_at: str) -> int:
type_hist: dict = {}
ufo_flagged = []
cryptid_flagged = []
for chunk in all_chunks:
t = chunk.get("type", "blank")
type_hist[t] = type_hist.get(t, 0) + 1
if chunk.get("ufo_anomaly_detected"):
ufo_flagged.append(chunk["chunk_id"])
if chunk.get("cryptid_anomaly_detected"):
cryptid_flagged.append(chunk["chunk_id"])
hist = "\n".join(f" {k}: {v}" for k, v in sorted(type_hist.items()))
header = (
"---\n"
'schema_version: "0.2.0"\n'
"type: master_document\n"
f"doc_id: {DOC_ID}\n"
f'canonical_title: "{DOC_TITLE}"\n'
f"total_pages: {TOTAL_PAGES}\n"
f"total_chunks: {len(all_chunks)}\n"
"chunk_types_histogram:\n"
f"{hist}\n"
"multi_page_tables: []\n"
f"ufo_anomalies_flagged: [{', '.join(ufo_flagged)}]\n"
f"cryptid_anomalies_flagged: [{', '.join(cryptid_flagged)}]\n"
'build_approach: "subagents"\n'
f"build_model: {GEMINI_MODEL}\n"
f"build_at: {build_at}\n"
"---\n\n"
f"# {DOC_TITLE}\n\n"
)
pages_dict: dict = {}
for chunk in all_chunks:
p = chunk["page"]
pages_dict.setdefault(p, []).append(chunk)
body_parts = []
for page_num in sorted(pages_dict):
body_parts.append(f"## Page {page_num}\n\n")
for chunk in sorted(pages_dict[page_num], key=lambda c: c.get("order_in_page", 0)):
cid = chunk["chunk_id"]
ctype = chunk.get("type", "blank")
bbox = chunk.get("bbox") or {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0}
if not isinstance(bbox, dict):
bbox = {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0}
bs = (
f"{float(bbox.get('x', 0)):.2f}/"
f"{float(bbox.get('y', 0)):.2f}/"
f"{float(bbox.get('w', 1)):.2f}/"
f"{float(bbox.get('h', 1)):.2f}"
)
section = [
f"<!-- chunk:{cid} src:./chunks/{cid}.md -->",
f'<a id="{cid}"></a>',
f"### Chunk {cid} — {ctype} · p{page_num} · bbox: {bs}",
"",
f"**EN:** {chunk.get('content_en') or ''}",
"",
f"**PT-BR:** {chunk.get('content_pt_br') or ''}",
"",
]
if ctype == "image":
section += [f"![{cid} image](./images/IMG-{cid}.png)", ""]
if chunk.get("image_description_en"):
section += [f"*EN: {chunk['image_description_en']}*", ""]
if chunk.get("image_description_pt_br"):
section += [f"*PT-BR: {chunk['image_description_pt_br']}*", ""]
meta = {k: v for k, v in chunk.items()
if not k.startswith("_") and k not in ("content_en", "content_pt_br")}
section += [
"<details><summary>metadata</summary>",
"",
"```json",
json.dumps(meta, indent=2, ensure_ascii=False),
"```",
"",
"</details>",
"",
"---",
"",
]
body_parts.append("\n".join(section))
out = RAW_DIR / "document.md"
out.write_text(header + "".join(body_parts), encoding="utf-8")
size = out.stat().st_size
print(f" Written: {out} ({size:,} bytes)", flush=True)
return size
CHECKPOINT_FILE = RAW_DIR / "_checkpoint_pages.json"
def main():
t0 = time.time()
build_at = datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
print(f"=== Rebuilding {DOC_ID} ===", flush=True)
print(f"Total pages: {TOTAL_PAGES} | Model: {GEMINI_MODEL} | max_output_tokens: {MAX_OUTPUT_TOKENS}", flush=True)
print()
# Step 1 — process pages (with checkpoint support)
if CHECKPOINT_FILE.exists():
print("STEP 1: Loading from checkpoint...", flush=True)
page_results = json.loads(CHECKPOINT_FILE.read_text(encoding="utf-8"))
print(f" Loaded {len(page_results)} pages from checkpoint.", flush=True)
else:
print("STEP 1: Processing pages...", flush=True)
page_results = process_pages_parallel(batch_size=BATCH_SIZE)
CHECKPOINT_FILE.write_text(json.dumps(page_results, ensure_ascii=False), encoding="utf-8")
print(f" Done. {len(page_results)} pages processed. Checkpoint saved.", flush=True)
print()
# Step 2 — assign chunk IDs
print("STEP 2: Assigning chunk IDs...", flush=True)
all_chunks = assign_global_chunk_ids(page_results)
print(f" Total chunks: {len(all_chunks)}", flush=True)
print()
# Step 3 — crop images
image_chunks = [c for c in all_chunks if c.get("type") == "image"]
print(f"STEP 3: Cropping {len(image_chunks)} image chunks...", flush=True)
for chunk in image_chunks:
crop_image_chunk(chunk)
print()
# Step 4 — analyze images in batches
print(f"STEP 4: Analyzing {len(image_chunks)} images...", flush=True)
for b in range(0, len(image_chunks), BATCH_SIZE):
batch = image_chunks[b:b + BATCH_SIZE]
with concurrent.futures.ThreadPoolExecutor(max_workers=BATCH_SIZE) as ex:
list(ex.map(analyze_image_chunk, batch))
if b + BATCH_SIZE < len(image_chunks):
time.sleep(0.5)
print()
# Step 5 — write chunk files
print("STEP 5: Writing chunk files...", flush=True)
for chunk in all_chunks:
write_chunk_file(chunk)
print(f" Written {len(all_chunks)} chunk files.", flush=True)
print()
# Step 6 — write index
print("STEP 6: Writing _index.json...", flush=True)
write_index_json(all_chunks, build_at)
print()
# Step 7 — write document.md
print("STEP 7: Writing document.md...", flush=True)
doc_bytes = write_document_md(all_chunks, build_at)
print()
wall = int(time.time() - t0)
num_images = len(image_chunks)
num_ufo = len([c for c in all_chunks if c.get("ufo_anomaly_detected")])
num_cryptid = len([c for c in all_chunks if c.get("cryptid_anomaly_detected")])
print("=== DONE ===", flush=True)
print(
f"pages_done={TOTAL_PAGES}, chunks_total={len(all_chunks)}, "
f"images_extracted={num_images}, tables_stitched=0, "
f"ufo_anomalies={num_ufo}, cryptid_anomalies={num_cryptid}, "
f"wall_seconds={wall}",
flush=True,
)
if __name__ == "__main__":
main()

583
scripts/rebuild_doc65_section7.py Normal file
View file

@ -0,0 +1,583 @@
#!/usr/bin/env python3
"""
Rebuilds doc-65-hs1-834228961-62-hq-83894-section-7 into the raw/ layout.
Uses claude CLI (OAuth via Max plan) to process each page PNG via vision.
"""
import os
import sys
import json
import base64
import time
import subprocess
import concurrent.futures
import threading
from datetime import datetime, timezone
from pathlib import Path
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-7"
DOC_TITLE = "65 HS1-834228961 62-HQ-83894 Section 7"
PNG_DIR = Path("/Users/guto/ufo/processing/png") / DOC_ID
OCR_DIR = Path("/Users/guto/ufo/processing/ocr") / DOC_ID
OUT_DIR = Path("/Users/guto/ufo/raw") / DOC_ID
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
CHUNKS_DIR.mkdir(parents=True, exist_ok=True)
IMAGES_DIR.mkdir(parents=True, exist_ok=True)
TABLES_DIR.mkdir(parents=True, exist_ok=True)
_print_lock = threading.Lock()
def safe_print(*args, **kwargs):
with _print_lock:
print(*args, **kwargs, flush=True)
PAGE_REBUILDER_PROMPT = """You are a page-rebuilder for a declassified UAP/UFO document reconstruction system.
STEP 1: Use the Read tool to view this page image:
{page_png_path}
STEP 2: Analyze the page carefully. The page is from document: {doc_title}
Doc ID: {doc_id}
Page number (1-indexed in document): {page_number}
Total pages: {total_pages}
OCR text (may be empty):
{page_ocr_text}
STEP 3: Return a JSON object with ALL content from the page split into chunks.
Return ONLY this JSON structure (no markdown fences, no commentary):
{{
"page_number": {page_number},
"chunks": [
{{
"order_in_page": 1,
"type": "<type_enum>",
"content_en": "English content or description",
"content_pt_br": "Conteúdo em português brasileiro",
"bbox": {{"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.90,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null,
"image_description_en": null,
"image_description_pt_br": null,
"extracted_text": null
}}
]
}}
Chunk type enum (use ONLY these):
- letterhead: agency/org header at top
- classification_banner: TOP SECRET/SECRET/CONFIDENTIAL/UNCLASSIFIED banners
- date_line: date of document
- to_from_line: TO:/FROM:/VIA: address lines
- subject_line: RE:/SUBJECT: lines
- paragraph: body text paragraph
- section_header: bold/underlined section title
- list_item: numbered or bulleted item
- redaction_block: blacked-out or whited-out region
- signature_block: signature/name/title at bottom
- image: photograph, diagram, sketch, stamp, seal
- table_marker: table content
- page_number: page number indicator
- footnote: footnote or endnote
- handwriting: handwritten annotation
- form_field: form label+value pairs
- blank: empty/whitespace page or region
Rules:
1. bbox values are NORMALIZED [0..1] (x=left, y=top, w=width, h=height)
2. Every visible region must be a chunk
3. For redaction_block: estimate redacted content type in redaction_inferred_content_type
4. For image chunks: provide detailed image_description_en AND image_description_pt_br
5. classification: extract from banners (e.g. "TOP SECRET") or null
6. formatting: array from: ["bold","italic","underline","all_caps","centered","right_aligned"]
7. cross_page_hint: "self_contained"|"continues_to_next"|"continues_from_prev"|"continues_both"
8. If blank page: one chunk of type "blank"
9. content_en: verbatim text (EN) or description; content_pt_br: PT-BR translation
10. ufo_anomaly_detected: true ONLY if page shows unidentified aerial phenomenon evidence
11. Output ONLY valid JSON, nothing else
"""
IMAGE_ANALYST_PROMPT = """You are an image analyst for declassified UAP/UFO document reconstruction.
STEP 1: Use the Read tool to view this cropped image:
{image_path}
STEP 2: Analyze the image carefully.
STEP 3: Return ONLY this JSON (no fences, no commentary):
{{
"image_description_en": "Detailed description in English",
"image_description_pt_br": "Descrição detalhada em português brasileiro",
"image_type": "<type>",
"extracted_text": "Any text visible in image verbatim, or null",
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}}
image_type enum: photograph|diagram|sketch|map|chart|seal|stamp|signature|redacted_region|form|other
ufo_anomaly_detected: true ONLY if image shows craft/object/phenomenon that appears to be UAP
cryptid_anomaly_detected: true ONLY if image shows anomalous/non-human entity
Return ONLY valid JSON.
"""
def extract_json(text: str) -> dict:
"""Extract JSON from claude CLI output."""
text = text.strip()
if text.startswith("```"):
import re
text = re.sub(r"^```(?:json)?\s*", "", text)
text = re.sub(r"\s*```\s*$", "", text)
start = text.find("{")
if start == -1:
raise ValueError("No JSON object found")
depth = 0
for i, c in enumerate(text[start:], start):
if c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
return json.loads(text[start:i+1])
raise ValueError("Unclosed JSON")
def call_claude(prompt: str, png_dir: Path, timeout: int = 180) -> dict:
"""Call claude CLI and return parsed JSON."""
cmd = [
"claude", "-p",
"--model", "haiku",
"--output-format", "json",
"--max-turns", "3",
"--allowedTools", "Read",
"--add-dir", str(png_dir),
"--", prompt
]
res = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, check=False)
if res.returncode != 0:
raise RuntimeError(f"claude CLI failed rc={res.returncode}: {res.stderr[-1000:]}")
cli_output = json.loads(res.stdout)
if cli_output.get("is_error"):
raise RuntimeError(f"claude error: {cli_output.get('result', '')[:500]}")
result_text = cli_output.get("result", "")
return extract_json(result_text)
def get_page_list():
"""Returns list of (page_number, png_path) tuples sorted by page_number."""
files = sorted(PNG_DIR.glob("p-*.png"))
return [(i+1, f) for i, f in enumerate(files)]
def load_ocr(png_path: Path) -> str:
stem = png_path.stem # p-NNN
ocr_path = OCR_DIR / f"{stem}.txt"
if ocr_path.exists():
text = ocr_path.read_text(encoding="utf-8").strip()
return text if len(text) > 2 else ""
return ""
def process_page(page_number: int, png_path: Path, total_pages: int) -> dict:
"""Process a single page via claude vision."""
ocr_text = load_ocr(png_path)
prompt = PAGE_REBUILDER_PROMPT.format(
doc_title=DOC_TITLE,
doc_id=DOC_ID,
page_number=page_number,
total_pages=total_pages,
page_png_path=str(png_path),
page_ocr_text=ocr_text if ocr_text else "(no OCR available)"
)
retries = 3
for attempt in range(retries):
try:
result = call_claude(prompt, png_path.parent, timeout=180)
chunks = result.get("chunks", [])
safe_print(f" [OK] p{page_number:03d}: {len(chunks)} chunks")
return result
except Exception as e:
safe_print(f" [ERR] p{page_number:03d} attempt {attempt+1}: {str(e)[:200]}")
if attempt < retries - 1:
time.sleep(2 ** attempt)
# Fallback
return {
"page_number": page_number,
"chunks": [{
"order_in_page": 1,
"type": "blank",
"content_en": f"[Page {page_number} — processing error]",
"content_pt_br": f"[Página {page_number} — erro de processamento]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None, "formatting": [],
"cross_page_hint": "self_contained", "ocr_confidence": 0.0,
"ocr_source_lines": [], "redaction_code": None,
"redaction_inferred_content_type": None, "image_type": None,
"ufo_anomaly_detected": False, "ufo_anomaly_type": None,
"ufo_anomaly_rationale": None, "cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None,
"image_description_en": None, "image_description_pt_br": None,
"extracted_text": None
}]
}
def global_number_chunks(all_page_results: dict) -> list:
"""Assign global chunk IDs across all pages."""
chunks_flat = []
for page_num in sorted(all_page_results.keys()):
page_data = all_page_results[page_num]
page_chunks = sorted(page_data.get("chunks", []), key=lambda c: c.get("order_in_page", 0))
for chunk in page_chunks:
chunk["page"] = page_num
chunks_flat.append(chunk)
for i, chunk in enumerate(chunks_flat):
chunk["chunk_id"] = f"c{i+1:04d}"
chunk["order_global"] = i + 1
chunk["prev_chunk"] = f"c{i:04d}" if i > 0 else None
chunk["next_chunk"] = f"c{i+2:04d}" if i < len(chunks_flat) - 1 else None
return chunks_flat
def crop_image(chunk: dict, png_path: Path):
"""Crop image chunk bbox from page PNG."""
from PIL import Image
chunk_id = chunk["chunk_id"]
bbox = chunk.get("bbox", {})
x = bbox.get("x", 0)
y = bbox.get("y", 0)
w = bbox.get("w", 1)
h = bbox.get("h", 1)
out_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
try:
im = Image.open(png_path)
W, H = im.size
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
if right <= left or bottom <= top:
right = min(W, left + 10)
bottom = min(H, top + 10)
cropped = im.crop((left, top, right, bottom))
cropped.save(str(out_path))
return out_path
except Exception as e:
safe_print(f" [WARN] Crop failed {chunk_id}: {e}")
return None
def analyze_image(chunk: dict, png_path: Path) -> dict:
"""Crop and analyze an image chunk."""
cropped_path = crop_image(chunk, png_path)
if not cropped_path or not cropped_path.exists():
return chunk
prompt = IMAGE_ANALYST_PROMPT.format(image_path=str(cropped_path))
retries = 2
for attempt in range(retries):
try:
analysis = call_claude(prompt, cropped_path.parent, timeout=120)
for key in ["image_description_en", "image_description_pt_br", "image_type",
"extracted_text", "ufo_anomaly_detected", "ufo_anomaly_type",
"ufo_anomaly_rationale", "cryptid_anomaly_detected",
"cryptid_anomaly_type", "cryptid_anomaly_rationale"]:
if key in analysis:
chunk[key] = analysis[key]
chunk["related_image"] = f"IMG-{chunk['chunk_id']}.png"
safe_print(f" [IMG] {chunk['chunk_id']}: analyzed")
return chunk
except Exception as e:
safe_print(f" [WARN] Image analysis {chunk['chunk_id']} attempt {attempt+1}: {str(e)[:150]}")
if attempt < retries - 1:
time.sleep(1)
return chunk
def write_chunk_file(chunk: dict, page_png_map: dict):
"""Write individual chunk .md file."""
chunk_id = chunk["chunk_id"]
page = chunk.get("page", 0)
bbox = chunk.get("bbox", {})
png_path = page_png_map.get(page)
source_png = f"../../processing/png/{DOC_ID}/{png_path.name}" if png_path else "unknown"
def jv(v):
return json.dumps(v, ensure_ascii=False)
yaml_lines = [
"---",
f"chunk_id: {chunk_id}",
f"type: {chunk.get('type', 'paragraph')}",
f"page: {page}",
f"order_in_page: {chunk.get('order_in_page', 1)}",
f"order_global: {chunk.get('order_global', 1)}",
f"bbox: {{x: {bbox.get('x', 0):.3f}, y: {bbox.get('y', 0):.3f}, w: {bbox.get('w', 1):.3f}, h: {bbox.get('h', 0):.3f}}}",
f"classification: {jv(chunk.get('classification'))}",
f"formatting: {jv(chunk.get('formatting', []))}",
f"cross_page_hint: {chunk.get('cross_page_hint', 'self_contained')}",
f"prev_chunk: {jv(chunk.get('prev_chunk'))}",
f"next_chunk: {jv(chunk.get('next_chunk'))}",
f"related_image: {jv(chunk.get('related_image'))}",
f"related_table: {jv(chunk.get('related_table'))}",
f"ocr_confidence: {chunk.get('ocr_confidence', 0.85)}",
f"ocr_source_lines: {jv(chunk.get('ocr_source_lines', []))}",
f"redaction_code: {jv(chunk.get('redaction_code'))}",
f"redaction_inferred_content_type: {jv(chunk.get('redaction_inferred_content_type'))}",
f"image_type: {jv(chunk.get('image_type'))}",
f"ufo_anomaly_detected: {str(chunk.get('ufo_anomaly_detected', False)).lower()}",
f"cryptid_anomaly_detected: {str(chunk.get('cryptid_anomaly_detected', False)).lower()}",
f"ufo_anomaly_type: {jv(chunk.get('ufo_anomaly_type'))}",
f"ufo_anomaly_rationale: {jv(chunk.get('ufo_anomaly_rationale'))}",
f"cryptid_anomaly_type: {jv(chunk.get('cryptid_anomaly_type'))}",
f"cryptid_anomaly_rationale: {jv(chunk.get('cryptid_anomaly_rationale'))}",
f"image_description_en: {jv(chunk.get('image_description_en'))}",
f"image_description_pt_br: {jv(chunk.get('image_description_pt_br'))}",
f"extracted_text: {jv(chunk.get('extracted_text'))}",
f"source_png: {source_png}",
"---",
"",
f"**EN:** {chunk.get('content_en', '')}",
"",
f"**PT-BR:** {chunk.get('content_pt_br', '')}",
""
]
out_path = CHUNKS_DIR / f"{chunk_id}.md"
out_path.write_text("\n".join(yaml_lines), encoding="utf-8")
def write_index(chunks_flat: list, total_pages: int):
"""Write _index.json."""
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": total_pages,
"total_chunks": len(chunks_flat),
"build_approach": "subagents",
"build_model": "claude-haiku-4-5",
"build_at": datetime.now(timezone.utc).isoformat(),
"chunks": []
}
for chunk in chunks_flat:
chunk_id = chunk["chunk_id"]
preview = (chunk.get("content_en", "") or "")[:80]
index["chunks"].append({
"chunk_id": chunk_id,
"type": chunk.get("type", "paragraph"),
"page": chunk.get("page", 1),
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"file": f"chunks/{chunk_id}.md",
"bbox": chunk.get("bbox", {}),
"preview": preview
})
(OUT_DIR / "_index.json").write_text(
json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8"
)
def write_document_md(chunks_flat: list, total_pages: int) -> int:
"""Assemble the master document.md."""
type_histogram = {}
ufo_flagged = []
cryptid_flagged = []
for chunk in chunks_flat:
t = chunk.get("type", "paragraph")
type_histogram[t] = type_histogram.get(t, 0) + 1
if chunk.get("ufo_anomaly_detected"):
ufo_flagged.append(chunk["chunk_id"])
if chunk.get("cryptid_anomaly_detected"):
cryptid_flagged.append(chunk["chunk_id"])
now_iso = datetime.now(timezone.utc).isoformat()
lines = [
"---",
"schema_version: \"0.2.0\"",
"type: master_document",
f"doc_id: {DOC_ID}",
f"canonical_title: \"{DOC_TITLE}\"",
f"total_pages: {total_pages}",
f"total_chunks: {len(chunks_flat)}",
f"chunk_types_histogram: {json.dumps(type_histogram, ensure_ascii=False)}",
"multi_page_tables: []",
f"ufo_anomalies_flagged: {json.dumps(ufo_flagged)}",
f"cryptid_anomalies_flagged: {json.dumps(cryptid_flagged)}",
"build_approach: \"subagents\"",
"build_model: claude-haiku-4-5",
f"build_at: {now_iso}",
"---",
""
]
current_page = None
for chunk in chunks_flat:
page = chunk.get("page", 1)
if page != current_page:
current_page = page
lines.append(f"\n## Page {page}\n")
chunk_id = chunk["chunk_id"]
ctype = chunk.get("type", "paragraph")
bbox = chunk.get("bbox", {})
bbox_str = f"{bbox.get('x', 0):.2f}/{bbox.get('y', 0):.2f}/{bbox.get('w', 1):.2f}/{bbox.get('h', 0):.2f}"
lines.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->")
lines.append(f"<a id=\"{chunk_id}\"></a>")
lines.append(f"### Chunk {chunk_id} — {ctype} · p{page} · bbox: {bbox_str}")
lines.append("")
lines.append(f"**EN:** {chunk.get('content_en', '')}")
lines.append("")
lines.append(f"**PT-BR:** {chunk.get('content_pt_br', '')}")
lines.append("")
if ctype == "image" and chunk.get("related_image"):
lines.append(f"![{chunk_id} image](./images/{chunk.get('related_image')})")
lines.append("")
if chunk.get("image_description_en"):
lines.append(f"**Image Description (EN):** {chunk['image_description_en']}")
lines.append("")
if chunk.get("image_description_pt_br"):
lines.append(f"**Descrição da Imagem (PT-BR):** {chunk['image_description_pt_br']}")
lines.append("")
meta = {k: v for k, v in chunk.items() if k not in ("content_en", "content_pt_br")}
lines.append("<details><summary>metadata</summary>")
lines.append("")
lines.append("```json")
lines.append(json.dumps(meta, indent=2, ensure_ascii=False))
lines.append("```")
lines.append("")
lines.append("</details>")
lines.append("")
lines.append("---")
lines.append("")
content = "\n".join(lines)
(OUT_DIR / "document.md").write_text(content, encoding="utf-8")
return len(content.encode("utf-8"))
def main():
start_time = time.time()
pages = get_page_list()
total_pages = len(pages)
page_png_map = {pnum: ppath for pnum, ppath in pages}
safe_print(f"Processing {total_pages} pages for {DOC_ID}")
# Process pages in batches of 5
batch_size = 5
all_page_results = {}
batches = [pages[i:i+batch_size] for i in range(0, len(pages), batch_size)]
for batch_idx, batch in enumerate(batches):
page_nums = [p[0] for p in batch]
safe_print(f"Batch {batch_idx+1}/{len(batches)}: pages {page_nums}")
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
futures = {
executor.submit(process_page, pnum, ppath, total_pages): pnum
for pnum, ppath in batch
}
for future in concurrent.futures.as_completed(futures):
pnum = futures[future]
try:
result = future.result()
all_page_results[pnum] = result
except Exception as e:
safe_print(f" [FATAL] Page {pnum}: {e}")
all_page_results[pnum] = {
"page_number": pnum,
"chunks": [{
"order_in_page": 1,
"type": "blank",
"content_en": f"[Page {pnum} — fatal error]",
"content_pt_br": f"[Página {pnum} — erro fatal]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None, "formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None, "ufo_anomaly_detected": False,
"ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None
}]
}
safe_print(f"\nAll pages processed. Numbering chunks globally...")
chunks_flat = global_number_chunks(all_page_results)
total_chunks = len(chunks_flat)
safe_print(f"Total chunks: {total_chunks}")
# Analyze image chunks in batches of 5
image_chunks = [c for c in chunks_flat if c.get("type") == "image"]
safe_print(f"\nProcessing {len(image_chunks)} image chunks...")
img_batches = [image_chunks[i:i+5] for i in range(0, len(image_chunks), 5)]
for img_batch_idx, img_batch in enumerate(img_batches):
safe_print(f"Image batch {img_batch_idx+1}/{len(img_batches)}")
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
futures = {}
for chunk in img_batch:
page = chunk.get("page", 1)
png_path = page_png_map.get(page)
if png_path:
f = executor.submit(analyze_image, chunk, png_path)
futures[f] = chunk["chunk_id"]
for future in concurrent.futures.as_completed(futures):
try:
future.result()
except Exception as e:
cid = futures[future]
safe_print(f" [ERR] Image {cid}: {e}")
safe_print(f"\nWriting chunk files...")
for chunk in chunks_flat:
write_chunk_file(chunk, page_png_map)
safe_print(f"Writing _index.json...")
write_index(chunks_flat, total_pages)
safe_print(f"Writing document.md...")
doc_bytes = write_document_md(chunks_flat, total_pages)
images_count = len([c for c in chunks_flat if c.get("type") == "image"])
ufo_count = len([c for c in chunks_flat if c.get("ufo_anomaly_detected")])
cryptid_count = len([c for c in chunks_flat if c.get("cryptid_anomaly_detected")])
wall_seconds = int(time.time() - start_time)
safe_print(f"\nSTATS pages={total_pages} chunks={total_chunks} images={images_count} tables=0 ufo={ufo_count} cryptid={cryptid_count} doc_md_bytes={doc_bytes}")
safe_print(f"pages_done={total_pages}, chunks_total={total_chunks}, images_extracted={images_count}, tables_stitched=0, ufo_anomalies={ufo_count}, cryptid_anomalies={cryptid_count}, wall_seconds={wall_seconds}")
if __name__ == "__main__":
main()

645
scripts/rebuild_doc65_section8.py Normal file
View file

@ -0,0 +1,645 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
rebuild_doc65_section8.py
Direct Gemini-powered rebuild of doc-65-hs1-834228961-62-hq-83894-section-8.
Produces: chunks/, images/, tables/, _index.json, document.md
"""
import os
import sys
import json
import re
import time
import base64
import datetime
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor, as_completed, TimeoutError as FuturesTimeout
from PIL import Image
import google.genai as genai
from google.genai import types
# ---------------------------------------------------------------------------
# Config
# ---------------------------------------------------------------------------
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-8"
DOC_TITLE = "65 HS1-834228961/62-HQ-83894 Section 8"
HIGHEST_CLASS = "TOP SECRET"
RAW_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
OCR_DIR = Path(f"/Users/guto/ufo/processing/ocr/{DOC_ID}")
CHUNKS_DIR = RAW_DIR / "chunks"
IMAGES_DIR = RAW_DIR / "images"
TABLES_DIR = RAW_DIR / "tables"
PAGES_RAW = RAW_DIR / "pages_raw.json"
MODEL = "models/gemini-3.1-flash-lite"
MAX_WORKERS = 4
PAGE_TIMEOUT = 150 # seconds per page
VALID_TYPES = {
"letterhead", "address_block", "classification_marking", "heading",
"paragraph", "form_field", "bulleted_item", "numbered_item", "quote_block",
"caption", "table_marker", "image", "stamp", "signature", "marginalia",
"redaction", "footer", "blank_area", "unknown",
}
# ---------------------------------------------------------------------------
# Gemini client
# ---------------------------------------------------------------------------
client = genai.Client(api_key=os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY"))
# ---------------------------------------------------------------------------
# Page-rebuilder prompt
# ---------------------------------------------------------------------------
PAGE_PROMPT = """\
You are a forensic document reconstruction agent for The Disclosure Bureau.
Given a single page image (PNG) and its raw OCR text from a US Department of War
declassified UAP/UFO document, decompose it into LOSSLESS agentic chunks.
## Chunk types — STRICT enum (use EXACTLY one of these 19 strings):
letterhead, address_block, classification_marking, heading, paragraph,
form_field, bulleted_item, numbered_item, quote_block, caption, table_marker,
image, stamp, signature, marginalia, redaction, footer, blank_area, unknown
## Output: ONE JSON object — NO markdown fences, NO prose before/after.
{{
"page_number": {page_number},
"page_summary_en": "1-2 sentences describing this page",
"page_summary_pt_br": "1-2 frases em português brasileiro",
"page_layout": {{
"columns": 1,
"orientation": "portrait",
"page_dimensions_approx": "letter"
}},
"chunks": [
{{
"order_in_page": 1,
"type": "paragraph",
"bbox": {{"x": 0.05, "y": 0.10, "w": 0.90, "h": 0.08}},
"content_en": "verbatim English text of this chunk",
"content_pt_br": "Texto em português brasileiro",
"metadata": {{
"ocr_confidence": 0.95,
"ocr_source_lines": [1, 2, 3],
"classification": null,
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"formatting": [],
"cross_page_hint": "self_contained",
"prev_chunk_hint": null,
"next_chunk_hint": null,
"language_in_source": "en"
}}
}}
]
}}
## Rules:
1. Order by reading order (top→bottom, left→right). order_in_page is 1-indexed.
2. One semantic unit per chunk (one paragraph, one address block, one image, etc.).
3. ALL content accounted for — never skip anything, even blank areas if significant.
4. content_en: verbatim/near-verbatim. No paraphrasing.
5. content_pt_br: Brazilian Portuguese (pt-BR). Preserve UTF-8 accents: ç ã á é í ó ú â ê ô à.
Proper nouns and verbatim quoted passages stay in source language inside pt-br.
6. Redacted blocks: content_en = "[REDACTED — <code>]". Never fabricate hidden content.
7. bbox: normalized 0..1 relative to page PNG size. Tight around the chunk.
8. cross_page_hint: self_contained | continues_from_prev | continues_to_next
9. image chunks: content_en = brief 1-sentence placeholder description (will be analyzed separately).
10. classification field: exact string as it appears (e.g. "TOP SECRET", "SECRET//NOFORN") or null.
Document context:
doc_id: {doc_id}
page_number: {page_number} of {total_pages}
doc_title: {doc_title}
OCR text (layout-preserved, may have errors — trust the image when they disagree):
---
{ocr_text}
---
Now analyze the image + OCR and output the JSON:"""
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def get_page_files():
pages = []
for png in sorted(PNG_DIR.glob("p-*.png")):
m = re.match(r"p-0*(\d+)\.png", png.name)
if not m:
continue
pn = int(m.group(1))
# OCR: try zero-padded 3-digit, then bare number
for fmt in [f"p-{pn:03d}.txt", f"p-{pn}.txt"]:
ocr = OCR_DIR / fmt
if ocr.exists():
break
else:
ocr = None
pages.append((pn, png, ocr))
return pages
def encode_png(path):
with open(path, "rb") as f:
return base64.b64encode(f.read()).decode()
def call_gemini(png_path, ocr_text, page_num, total_pages):
prompt = PAGE_PROMPT.format(
doc_id=DOC_ID,
page_number=page_num,
total_pages=total_pages,
doc_title=DOC_TITLE,
ocr_text=ocr_text[:5000],
)
with open(png_path, "rb") as f:
img_bytes = f.read()
contents = [
types.Part(
inline_data=types.Blob(mime_type="image/png", data=img_bytes)
),
types.Part(text=prompt),
]
config = types.GenerateContentConfig(
temperature=0.1,
max_output_tokens=8192,
)
def _call():
resp = client.models.generate_content(
model=MODEL, contents=contents, config=config
)
if resp.text is None:
# Safety block or empty response — extract any available text from parts
try:
parts = resp.candidates[0].content.parts
return "\n".join(p.text for p in parts if hasattr(p, "text") and p.text)
except Exception:
return None
return resp.text
with ThreadPoolExecutor(max_workers=1) as ex:
future = ex.submit(_call)
return future.result(timeout=PAGE_TIMEOUT)
def parse_page_json(raw_text, page_num):
text = raw_text.strip()
text = re.sub(r"^```(?:json)?\s*", "", text, flags=re.MULTILINE)
text = re.sub(r"\s*```\s*$", "", text, flags=re.MULTILINE)
text = text.strip()
try:
data = json.loads(text)
except json.JSONDecodeError:
# Try to extract the largest {...} block
m = re.search(r"\{[\s\S]*\}", text)
if m:
try:
data = json.loads(m.group(0))
except json.JSONDecodeError:
return {"page_number": page_num, "error": "json_parse_failed",
"chunks": [], "raw": text[:300]}
else:
return {"page_number": page_num, "error": "no_json_found",
"chunks": [], "raw": text[:300]}
data["page_number"] = page_num
# Validate and normalize chunk types
for c in data.get("chunks", []):
if c.get("type") not in VALID_TYPES:
c["type"] = "unknown"
return data
def fallback_chunk(page_num, ocr_text):
"""Minimal unknown chunk when Gemini fails persistently."""
preview = ocr_text[:200].strip() if ocr_text and ocr_text.strip() else "(page content unavailable)"
return {
"page_number": page_num,
"page_summary_en": f"Page {page_num} — content could not be parsed by vision model.",
"page_summary_pt_br": f"Página {page_num} — conteúdo não pôde ser analisado pelo modelo de visão.",
"page_layout": {"columns": 1, "orientation": "portrait", "page_dimensions_approx": "letter"},
"chunks": [{
"order_in_page": 1,
"type": "unknown",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"content_en": f"[Vision analysis failed — OCR excerpt: {preview}]",
"content_pt_br": f"[Análise de visão falhou — trecho OCR: {preview}]",
"metadata": {
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"classification": None,
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"formatting": [],
"cross_page_hint": "self_contained",
"prev_chunk_hint": None,
"next_chunk_hint": None,
"language_in_source": "en",
},
}],
}
def process_page(page_num, png_path, ocr_path, total_pages, use_fallback=False):
ocr_text = (
ocr_path.read_text(encoding="utf-8", errors="replace")
if ocr_path
else "(OCR not available)"
)
if use_fallback:
return fallback_chunk(page_num, ocr_text)
try:
raw = call_gemini(png_path, ocr_text, page_num, total_pages)
if raw is None:
return {"page_number": page_num, "error": "gemini_none_response", "chunks": []}
return parse_page_json(raw, page_num)
except FuturesTimeout:
return {"page_number": page_num, "error": "timeout", "chunks": []}
except Exception as exc:
return {"page_number": page_num, "error": str(exc)[:200], "chunks": []}
def is_valid_page(p):
return bool(p.get("chunks")) and not p.get("error")
# ---------------------------------------------------------------------------
# Phase 1: process all pages
# ---------------------------------------------------------------------------
def phase_process_pages(pages):
total = len(pages)
print(f"[Phase 1] Processing {total} pages with {MODEL} ...")
# Load existing checkpoint
existing_map = {}
failed_pages = set()
if PAGES_RAW.exists():
try:
existing = json.loads(PAGES_RAW.read_text(encoding="utf-8"))
for p in existing:
if is_valid_page(p):
existing_map[p["page_number"]] = p
elif p.get("error"):
failed_pages.add(p["page_number"])
print(f" Checkpoint: {len(existing_map)} valid pages loaded, {len(failed_pages)} previously failed")
except Exception:
pass
to_process = [(pn, pp, op) for pn, pp, op in pages if pn not in existing_map]
print(f" Remaining: {len(to_process)} pages")
results_map = dict(existing_map)
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
futures = {
executor.submit(process_page, pn, pp, op, total, pn in failed_pages): pn
for pn, pp, op in to_process
}
done = 0
for future in as_completed(futures):
pn = futures[future]
done += 1
try:
result = future.result(timeout=PAGE_TIMEOUT + 30)
except Exception as exc:
result = {"page_number": pn, "error": str(exc)[:200], "chunks": []}
results_map[pn] = result
nchunks = len(result.get("chunks", []))
status = "OK" if is_valid_page(result) else f"ERR({result.get('error','?')[:40]})"
print(f" [{done}/{len(to_process)}] p-{pn:03d}: {status} chunks={nchunks}")
# Checkpoint every 10 pages
if done % 10 == 0:
ordered = [results_map[p[0]] for p in pages if p[0] in results_map]
PAGES_RAW.write_text(
json.dumps(ordered, ensure_ascii=False, indent=2), encoding="utf-8"
)
# Final save
ordered = [results_map[p[0]] for p in pages if p[0] in results_map]
PAGES_RAW.write_text(json.dumps(ordered, ensure_ascii=False, indent=2), encoding="utf-8")
print(f" Saved {len(ordered)} pages to pages_raw.json")
return results_map
# ---------------------------------------------------------------------------
# Phase 2: globally number chunks
# ---------------------------------------------------------------------------
def phase_number_chunks(pages, results_map):
print("[Phase 2] Globally numbering chunks ...")
all_chunks = [] # list of (page_num, chunk_dict)
for pn, _, _ in pages:
pg = results_map.get(pn, {})
chunks = sorted(pg.get("chunks", []), key=lambda c: c.get("order_in_page", 0))
for c in chunks:
all_chunks.append((pn, c))
total_chunks = len(all_chunks)
for i, (pn, c) in enumerate(all_chunks, 1):
c["chunk_id"] = f"c{i:04d}"
c["order_global"] = i
c["page"] = pn
c["prev_chunk"] = f"c{i-1:04d}" if i > 1 else None
c["next_chunk"] = f"c{i+1:04d}" if i < total_chunks else None
print(f" Total chunks: {total_chunks}")
return all_chunks
# ---------------------------------------------------------------------------
# Phase 3: crop image chunks
# ---------------------------------------------------------------------------
def phase_crop_images(all_chunks, pages):
png_map = {pn: pp for pn, pp, _ in pages}
image_chunks = [(pn, c) for pn, c in all_chunks if c.get("type") == "image"]
print(f"[Phase 3] Cropping {len(image_chunks)} image chunks ...")
for pn, c in image_chunks:
cid = c["chunk_id"]
out_path = IMAGES_DIR / f"IMG-{cid}.png"
if out_path.exists():
continue
png_path = png_map.get(pn)
if not png_path:
continue
bbox = c.get("bbox", {})
if not bbox:
continue
try:
im = Image.open(png_path)
W, H = im.size
pad = 0.005
x = bbox.get("x", 0)
y = bbox.get("y", 0)
w = bbox.get("w", 1)
h = bbox.get("h", 1)
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
if right > left and bottom > top:
crop = im.crop((left, top, right, bottom))
crop.save(out_path)
c["related_image"] = f"IMG-{cid}.png"
except Exception as exc:
print(f" WARN crop {cid}: {exc}")
# ---------------------------------------------------------------------------
# Phase 4: write chunk files
# ---------------------------------------------------------------------------
def phase_write_chunks(all_chunks, pages):
png_map = {pn: pp for pn, pp, _ in pages}
print(f"[Phase 4] Writing {len(all_chunks)} chunk files ...")
for pn, c in all_chunks:
cid = c["chunk_id"]
chunk_path = CHUNKS_DIR / f"{cid}.md"
meta = c.get("metadata", {})
bbox = c.get("bbox", {"x": 0, "y": 0, "w": 0, "h": 0})
png_path = png_map.get(pn, "")
rel_png = f"../../processing/png/{DOC_ID}/{Path(str(png_path)).name}" if png_path else "null"
yaml_lines = [
"---",
f"chunk_id: {cid}",
f"type: {c.get('type', 'unknown')}",
f"page: {pn}",
f"order_in_page: {c.get('order_in_page', 0)}",
f"order_global: {c.get('order_global', 0)}",
f"bbox: {{x: {bbox.get('x',0):.4f}, y: {bbox.get('y',0):.4f}, w: {bbox.get('w',0):.4f}, h: {bbox.get('h',0):.4f}}}",
f"classification: {json.dumps(meta.get('classification'))}",
f"formatting: {json.dumps(meta.get('formatting', []))}",
f"cross_page_hint: {meta.get('cross_page_hint', 'self_contained')}",
f"prev_chunk: {json.dumps(c.get('prev_chunk'))}",
f"next_chunk: {json.dumps(c.get('next_chunk'))}",
f"related_image: {json.dumps(c.get('related_image'))}",
f"related_table: {json.dumps(c.get('related_table'))}",
f"ocr_confidence: {meta.get('ocr_confidence', 0.0)}",
f"ocr_source_lines: {json.dumps(meta.get('ocr_source_lines', []))}",
f"redaction_code: {json.dumps(meta.get('redaction_code'))}",
f"redaction_inferred_content_type: {json.dumps(meta.get('redaction_inferred_content_type'))}",
f"image_type: {json.dumps(meta.get('image_type'))}",
f"ufo_anomaly_detected: {str(c.get('ufo_anomaly_detected', False)).lower()}",
f"cryptid_anomaly_detected: {str(c.get('cryptid_anomaly_detected', False)).lower()}",
f"ufo_anomaly_type: {json.dumps(c.get('ufo_anomaly_type'))}",
f"ufo_anomaly_rationale: {json.dumps(c.get('ufo_anomaly_rationale'))}",
f"cryptid_anomaly_type: {json.dumps(c.get('cryptid_anomaly_type'))}",
f"cryptid_anomaly_rationale: {json.dumps(c.get('cryptid_anomaly_rationale'))}",
f"image_description_en: {json.dumps(c.get('image_description_en'))}",
f"image_description_pt_br: {json.dumps(c.get('image_description_pt_br'))}",
f"extracted_text: {json.dumps(c.get('extracted_text'))}",
f"source_png: {rel_png}",
"---",
]
body = "\n".join(yaml_lines) + "\n\n"
body += f"**EN:** {c.get('content_en', '')}\n\n"
body += f"**PT-BR:** {c.get('content_pt_br', '')}\n"
chunk_path.write_text(body, encoding="utf-8")
# ---------------------------------------------------------------------------
# Phase 5: write _index.json
# ---------------------------------------------------------------------------
def phase_write_index(all_chunks, pages):
total_pages = len(pages)
total_chunks = len(all_chunks)
build_at = datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": total_pages,
"total_chunks": total_chunks,
"build_approach": "subagents",
"build_model": MODEL,
"build_at": build_at,
"chunks": [],
}
for pn, c in all_chunks:
cid = c["chunk_id"]
preview = (c.get("content_en") or "")[:80]
index["chunks"].append({
"chunk_id": cid,
"type": c.get("type", "unknown"),
"page": pn,
"order_in_page": c.get("order_in_page", 0),
"order_global": c.get("order_global", 0),
"file": f"chunks/{cid}.md",
"bbox": c.get("bbox", {}),
"preview": preview,
})
index_path = RAW_DIR / "_index.json"
index_path.write_text(json.dumps(index, ensure_ascii=False, indent=2), encoding="utf-8")
print(f"[Phase 5] Written _index.json ({total_chunks} entries)")
return build_at
# ---------------------------------------------------------------------------
# Phase 6: assemble document.md
# ---------------------------------------------------------------------------
def phase_assemble_document(all_chunks, pages, results_map, build_at):
total_pages = len(pages)
total_chunks = len(all_chunks)
# Histograms + anomaly lists
type_hist = {}
ufo_flagged = []
cryptid_flagged = []
for pn, c in all_chunks:
ctype = c.get("type", "unknown")
type_hist[ctype] = type_hist.get(ctype, 0) + 1
if c.get("ufo_anomaly_detected"):
ufo_flagged.append(c["chunk_id"])
if c.get("cryptid_anomaly_detected"):
cryptid_flagged.append(c["chunk_id"])
build_at_str = build_at
frontmatter = f"""---
schema_version: "0.2.0"
type: master_document
doc_id: {DOC_ID}
canonical_title: "{DOC_TITLE}"
total_pages: {total_pages}
total_chunks: {total_chunks}
chunk_types_histogram: {json.dumps(type_hist, ensure_ascii=False)}
multi_page_tables: []
ufo_anomalies_flagged: {json.dumps(ufo_flagged)}
cryptid_anomalies_flagged: {json.dumps(cryptid_flagged)}
build_approach: "subagents"
build_model: "{MODEL}"
build_at: "{build_at_str}"
---
"""
# Group chunks by page
chunks_by_page = {}
for pn, c in all_chunks:
chunks_by_page.setdefault(pn, []).append(c)
body_parts = []
for pn, _, _ in pages:
pg = results_map.get(pn, {})
summary_en = pg.get("page_summary_en", "")
summary_pt = pg.get("page_summary_pt_br", "")
body_parts.append(f"\n## Page {pn}\n")
if summary_en:
body_parts.append(f"<!-- page_summary_en: {summary_en} -->\n")
if summary_pt:
body_parts.append(f"<!-- page_summary_pt_br: {summary_pt} -->\n")
body_parts.append("\n")
for c in chunks_by_page.get(pn, []):
cid = c["chunk_id"]
ctype = c.get("type", "unknown")
bbox = c.get("bbox", {})
bbox_str = f"{bbox.get('x',0):.2f}/{bbox.get('y',0):.2f}/{bbox.get('w',0):.2f}/{bbox.get('h',0):.2f}"
body_parts.append(f"<!-- chunk:{cid} src:./chunks/{cid}.md -->\n")
body_parts.append(f'<a id="{cid}"></a>\n')
body_parts.append(f"### Chunk {cid} — {ctype} · p{pn} · bbox: {bbox_str}\n\n")
body_parts.append(f"**EN:** {c.get('content_en', '')}\n\n")
body_parts.append(f"**PT-BR:** {c.get('content_pt_br', '')}\n\n")
if ctype == "image" and c.get("related_image"):
body_parts.append(f"![{cid}](./images/{c['related_image']})\n\n")
if c.get("image_description_en"):
body_parts.append(f"*Image (EN): {c['image_description_en']}*\n\n")
if c.get("image_description_pt_br"):
body_parts.append(f"*Imagem (PT-BR): {c['image_description_pt_br']}*\n\n")
# Metadata details block
meta_json = {
"chunk_id": cid,
"type": ctype,
"page": pn,
"order_global": c.get("order_global"),
"bbox": bbox,
"classification": c.get("metadata", {}).get("classification"),
"formatting": c.get("metadata", {}).get("formatting", []),
"cross_page_hint": c.get("metadata", {}).get("cross_page_hint"),
"ocr_confidence": c.get("metadata", {}).get("ocr_confidence"),
"ufo_anomaly_detected": c.get("ufo_anomaly_detected", False),
"cryptid_anomaly_detected": c.get("cryptid_anomaly_detected", False),
}
body_parts.append("<details><summary>metadata</summary>\n\n")
body_parts.append("```json\n")
body_parts.append(json.dumps(meta_json, ensure_ascii=False, indent=2))
body_parts.append("\n```\n\n</details>\n\n---\n\n")
doc_content = frontmatter + "".join(body_parts)
doc_path = RAW_DIR / "document.md"
doc_path.write_text(doc_content, encoding="utf-8")
doc_bytes = len(doc_content.encode("utf-8"))
print(f"[Phase 6] Written document.md ({doc_bytes:,} bytes)")
return doc_bytes, ufo_flagged, cryptid_flagged
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
start = time.time()
# Ensure output dirs exist
for d in [CHUNKS_DIR, IMAGES_DIR, TABLES_DIR]:
d.mkdir(parents=True, exist_ok=True)
pages = get_page_files()
if not pages:
print("ERROR: no PNG pages found", file=sys.stderr)
sys.exit(1)
total_pages = len(pages)
print(f"Document: {DOC_ID}")
print(f"Pages found: {total_pages}")
# Phase 1: vision + OCR per page
results_map = phase_process_pages(pages)
# Phase 2: global chunk numbering
all_chunks = phase_number_chunks(pages, results_map)
# Phase 3: crop image chunks
phase_crop_images(all_chunks, pages)
# Phase 4: write chunk .md files
phase_write_chunks(all_chunks, pages)
# Phase 5: write _index.json
build_at = phase_write_index(all_chunks, pages)
# Phase 6: assemble document.md
doc_bytes, ufo_flagged, cryptid_flagged = phase_assemble_document(
all_chunks, pages, results_map, build_at
)
wall = int(time.time() - start)
images_count = len(list(IMAGES_DIR.glob("IMG-*.png")))
tables_count = len(list(TABLES_DIR.glob("TBL-*.csv")))
print(f"\nSTATS pages_done={total_pages} chunks_total={len(all_chunks)} "
f"images_extracted={images_count} tables_stitched={tables_count} "
f"ufo_anomalies={len(ufo_flagged)} cryptid_anomalies={len(cryptid_flagged)} "
f"wall_seconds={wall}")
if __name__ == "__main__":
main()

553
scripts/rebuild_doc65_serial130_resume.py Normal file
View file

@ -0,0 +1,553 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
rebuild_doc65_serial130_resume.py
Resume rebuild for doc-65-hs1-834228961-62-hq-83894-serial-130.
Pages 1-50 already processed (chunks c0001-c0204 exist).
This script:
Phase A: Process pages 51-91 via claude CLI → write c0205+
Phase B: Read ALL chunk files → rebuild _index.json + document.md
"""
import os
import sys
import json
import time
import subprocess
import concurrent.futures
import re
from datetime import datetime, timezone
from pathlib import Path
try:
from PIL import Image as PILImage
PILLOW_OK = True
except ImportError:
PILLOW_OK = False
# ── Config ──────────────────────────────────────────────────────────────────
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-serial-130"
DOC_TITLE = "HQ Air Defense Command – Unidentified Flying Objects Reports (65-HS1-834228961 / 62-HQ-83894 Serial 130)"
PNG_DIR = Path("/Users/guto/ufo/processing/png") / DOC_ID
OCR_DIR = Path("/Users/guto/ufo/processing/ocr") / DOC_ID
OUT_DIR = Path("/Users/guto/ufo/raw") / DOC_ID
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
CLAUDE_BIN = "/Users/guto/.local/bin/claude"
TOTAL_PAGES = 91
START_PAGE = 51 # first missing page
FIRST_CHUNK_NUM = 205 # c0205 onwards for new chunks
BATCH_SIZE = 4
CLAUDE_TIMEOUT = 150
# ── Helpers ──────────────────────────────────────────────────────────────────
def load_ocr(page_num: int) -> str:
ocr_path = OCR_DIR / f"p-{page_num - 1:03d}.txt"
if ocr_path.exists():
text = ocr_path.read_text(encoding="utf-8", errors="replace").strip()
return text[:2000] if text else ""
return ""
PAGE_REBUILDER_PROMPT = """You are a page-rebuilder agent analyzing a page from a declassified US government document about Unidentified Flying Objects (UFO/UAP) investigations.
Document: {doc_title}
Page: {page_num} of {total_pages}
PNG file: /Users/guto/ufo/processing/png/{doc_id}/p-{png_num:03d}.png
OCR text (may be incomplete):
{ocr_text}
Use the Read tool to read the image at:
/Users/guto/ufo/processing/png/{doc_id}/p-{png_num:03d}.png
Analyze ALL visible content and return ONLY a JSON object (no markdown fences, no extra text):
{{
"page_number": {page_num},
"chunks": [
{{
"order_in_page": 1,
"type": "letterhead",
"content_en": "exact transcription or description in English",
"content_pt_br": "transcrição ou descrição em português brasileiro",
"bbox": {{"x": 0.00, "y": 0.00, "w": 1.00, "h": 0.10}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.85,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}}
]
}}
RULES:
- type must be ONE of: cover | letterhead | stamp | header | subheader | paragraph | redaction | signature | image | table_marker | footer | page_number | classification_marking | separator | handwriting | form_field | caption | list_item | annotation | blank | classification_banner | signature_block | redaction_block
- bbox: x,y = top-left corner fraction (0.0-1.0), w,h = width/height fractions (0.0-1.0)
- Split page into logical chunks (letterhead separate from body, stamps separate, etc.)
- For redacted blocks: type=redaction, redaction_code e.g. "(b)(1)", "(b)(3)", "(b)(6)"
- For photos/sketches/diagrams: type=image, set image_type to photo|diagram|sketch|map|chart|signature_block|stamp|seal|other
- cross_page_hint: self_contained | continues_to_next | continues_from_prev
- content_en: verbatim transcription when legible; describe otherwise e.g. "[Stamp: RECEIVED OCT 6 1947]"
- content_pt_br: Brazilian Portuguese translation/description
- ufo_anomaly_detected: true ONLY if page has image/sketch of anomalous aerial object
- Blank pages: one chunk with type=blank
- Return ONLY valid JSON, nothing else"""
def run_claude(prompt: str, timeout: int = CLAUDE_TIMEOUT) -> str:
try:
result = subprocess.run(
[CLAUDE_BIN, "-p", "--dangerously-skip-permissions",
"--model", "claude-haiku-4-5",
"--no-session-persistence",
prompt],
capture_output=True, text=True, timeout=timeout,
env={**os.environ}
)
return result.stdout.strip()
except subprocess.TimeoutExpired:
return ""
except Exception as e:
return f"ERROR: {e}"
def parse_json(raw: str):
text = raw.strip()
if text.startswith("```"):
lines = text.split("\n")[1:]
if lines and lines[-1].strip() == "```":
lines = lines[:-1]
text = "\n".join(lines).strip()
start = text.find("{")
if start == -1:
return None
depth = 0
end = -1
for i, ch in enumerate(text[start:]):
if ch == "{":
depth += 1
elif ch == "}":
depth -= 1
if depth == 0:
end = start + i + 1
break
if end == -1:
return None
try:
return json.loads(text[start:end])
except json.JSONDecodeError:
return None
def rebuild_page(page_num: int) -> dict:
png_num = page_num - 1 # 0-indexed
ocr_text = load_ocr(page_num)
prompt = PAGE_REBUILDER_PROMPT.format(
doc_title=DOC_TITLE,
page_num=page_num,
total_pages=TOTAL_PAGES,
doc_id=DOC_ID,
png_num=png_num,
ocr_text=ocr_text or "(no OCR available)"
)
for attempt in range(3):
raw = run_claude(prompt, timeout=CLAUDE_TIMEOUT)
if not raw or raw.startswith("ERROR:"):
if attempt < 2:
time.sleep(5 * (attempt + 1))
continue
break
data = parse_json(raw)
if data and "chunks" in data:
data["page_number"] = page_num
data["png_num"] = png_num
for i, ch in enumerate(data["chunks"]):
ch["order_in_page"] = i + 1
ch["page"] = page_num
print(f" [OK] page {page_num:03d} → {len(data['chunks'])} chunks", flush=True)
return data
if attempt < 2:
print(f" [RETRY {attempt+1}] page {page_num}: bad JSON", flush=True)
time.sleep(3)
else:
print(f" [FAIL] page {page_num}: {raw[:200]}", flush=True)
# Fallback
return {
"page_number": page_num, "png_num": page_num - 1,
"chunks": [{
"order_in_page": 1, "type": "blank", "page": page_num,
"content_en": "[Page processing failed]",
"content_pt_br": "[Falha no processamento da página]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None, "formatting": [],
"cross_page_hint": "self_contained", "ocr_confidence": 0.0,
"ocr_source_lines": [], "redaction_code": None,
"redaction_inferred_content_type": None, "image_type": None,
"ufo_anomaly_detected": False, "ufo_anomaly_type": None,
"ufo_anomaly_rationale": None, "cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None
}]
}
def yv(v):
if v is None:
return "null"
if isinstance(v, bool):
return str(v).lower()
s = str(v)
if any(c in s for c in [':', '{', '}', '[', ']', ',', '\n', '#', '&', '*', '?', '|', '<', '>', '=', '!', '%', '@', '`']):
return f'"{s}"'
return s
def write_chunk_file(chunk: dict):
chunk_id = chunk["chunk_id"]
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
page = chunk.get("page", 1)
png_num = chunk.get("png_num", page - 1)
ctype = chunk.get("type", "paragraph")
fmt_list = chunk.get("formatting") or []
fmt_str = "[" + ", ".join(f'"{f}"' for f in fmt_list) + "]"
ocr_lines = chunk.get("ocr_source_lines") or []
ocr_lines_str = "[" + ", ".join(str(l) for l in ocr_lines) + "]"
related_image = f"IMG-{chunk_id}.png" if ctype == "image" else "null"
content = f"""---
chunk_id: {chunk_id}
type: {ctype}
page: {page}
order_in_page: {chunk.get("order_in_page", 1)}
order_global: {chunk.get("order_global", 1)}
bbox: {{x: {bbox.get("x", 0.0):.3f}, y: {bbox.get("y", 0.0):.3f}, w: {bbox.get("w", 1.0):.3f}, h: {bbox.get("h", 0.1):.3f}}}
classification: {yv(chunk.get("classification"))}
formatting: {fmt_str}
cross_page_hint: {chunk.get("cross_page_hint", "self_contained")}
prev_chunk: {chunk.get("prev_chunk") or "null"}
next_chunk: {chunk.get("next_chunk") or "null"}
related_image: {related_image}
related_table: null
ocr_confidence: {chunk.get("ocr_confidence", 0.85)}
ocr_source_lines: {ocr_lines_str}
redaction_code: {yv(chunk.get("redaction_code"))}
redaction_inferred_content_type: {yv(chunk.get("redaction_inferred_content_type"))}
image_type: {yv(chunk.get("image_type"))}
ufo_anomaly_detected: {str(chunk.get("ufo_anomaly_detected", False)).lower()}
cryptid_anomaly_detected: {str(chunk.get("cryptid_anomaly_detected", False)).lower()}
ufo_anomaly_type: {yv(chunk.get("ufo_anomaly_type"))}
ufo_anomaly_rationale: {yv(chunk.get("ufo_anomaly_rationale"))}
cryptid_anomaly_type: {yv(chunk.get("cryptid_anomaly_type"))}
cryptid_anomaly_rationale: {yv(chunk.get("cryptid_anomaly_rationale"))}
image_description_en: {yv(chunk.get("image_description_en"))}
image_description_pt_br: {yv(chunk.get("image_description_pt_br"))}
extracted_text: {yv(chunk.get("extracted_text"))}
source_png: ../../processing/png/{DOC_ID}/p-{png_num:03d}.png
---
**EN:** {chunk.get("content_en", "")}
**PT-BR:** {chunk.get("content_pt_br", "")}
"""
(CHUNKS_DIR / f"{chunk_id}.md").write_text(content, encoding="utf-8")
def crop_image(chunk: dict):
chunk_id = chunk["chunk_id"]
png_num = chunk.get("png_num", chunk.get("page", 1) - 1)
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
src = PNG_DIR / f"p-{png_num:03d}.png"
dst = IMAGES_DIR / f"IMG-{chunk_id}.png"
if not PILLOW_OK or not src.exists():
return
try:
im = PILImage.open(src)
W, H = im.size
x = max(0.0, min(1.0, bbox.get("x", 0.0)))
y = max(0.0, min(1.0, bbox.get("y", 0.0)))
w = max(0.01, min(1.0 - x, bbox.get("w", 1.0)))
h = max(0.01, min(1.0 - y, bbox.get("h", 0.1)))
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
im.crop((left, top, right, bottom)).save(str(dst))
print(f" [CROP] {chunk_id}", flush=True)
except Exception as e:
print(f" [CROP FAIL] {chunk_id}: {e}", flush=True)
def parse_frontmatter(path: Path) -> dict:
"""Read YAML frontmatter from a chunk .md file."""
text = path.read_text(encoding="utf-8", errors="replace")
if not text.startswith("---"):
return {}
end = text.find("\n---\n", 3)
if end == -1:
return {}
fm_text = text[3:end]
data = {}
for line in fm_text.split("\n"):
m = re.match(r'^(\w+):\s*(.*)', line)
if not m:
continue
key, val = m.group(1), m.group(2).strip()
if val == "null":
data[key] = None
elif val == "true":
data[key] = True
elif val == "false":
data[key] = False
else:
# Try int
try:
data[key] = int(val)
except ValueError:
# Strip surrounding quotes
if val.startswith('"') and val.endswith('"'):
data[key] = val[1:-1]
else:
data[key] = val
# Parse bbox specially
bbox_m = re.search(r'bbox:\s*\{x:\s*([\d.]+),\s*y:\s*([\d.]+),\s*w:\s*([\d.]+),\s*h:\s*([\d.]+)\}', text)
if bbox_m:
data["bbox"] = {
"x": float(bbox_m.group(1)),
"y": float(bbox_m.group(2)),
"w": float(bbox_m.group(3)),
"h": float(bbox_m.group(4)),
}
# Extract body content
body = text[end + 5:].strip()
en_m = re.search(r'\*\*EN:\*\*\s*(.*?)(?=\n\n\*\*PT-BR:|$)', body, re.DOTALL)
ptbr_m = re.search(r'\*\*PT-BR:\*\*\s*(.*?)$', body, re.DOTALL)
data["content_en"] = en_m.group(1).strip() if en_m else ""
data["content_pt_br"] = ptbr_m.group(1).strip() if ptbr_m else ""
return data
def build_assembly(all_chunks: list, build_at: str):
"""Write _index.json and document.md from all_chunks list."""
type_histogram = {}
for chunk in all_chunks:
t = chunk.get("type", "paragraph")
type_histogram[t] = type_histogram.get(t, 0) + 1
ufo_flagged = [c["chunk_id"] for c in all_chunks if c.get("ufo_anomaly_detected")]
cryptid_flagged = [c["chunk_id"] for c in all_chunks if c.get("cryptid_anomaly_detected")]
images_extracted = sum(1 for c in all_chunks if c.get("type") == "image")
# _index.json
index_chunks = []
for chunk in all_chunks:
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
content_en = chunk.get("content_en", "")
preview = content_en[:80] + ("..." if len(content_en) > 80 else "")
index_chunks.append({
"chunk_id": chunk["chunk_id"],
"type": chunk.get("type", "paragraph"),
"page": chunk.get("page", 1),
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"file": f"chunks/{chunk['chunk_id']}.md",
"bbox": bbox,
"preview": preview
})
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": len(all_chunks),
"build_approach": "subagents",
"build_model": "claude-haiku-4-5",
"build_at": build_at,
"chunks": index_chunks
}
(OUT_DIR / "_index.json").write_text(
json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8"
)
print(f" Wrote _index.json ({len(all_chunks)} chunks)", flush=True)
# document.md
histogram_yaml = "\n".join(f" {k}: {v}" for k, v in sorted(type_histogram.items()))
def list_yaml(items):
return " []" if not items else "\n".join(f" - {i}" for i in items)
doc_parts = [f"""---
schema_version: "0.2.0"
type: master_document
doc_id: {DOC_ID}
canonical_title: "{DOC_TITLE}"
total_pages: {TOTAL_PAGES}
total_chunks: {len(all_chunks)}
chunk_types_histogram:
{histogram_yaml}
multi_page_tables: []
ufo_anomalies_flagged:
{list_yaml(ufo_flagged)}
cryptid_anomalies_flagged:
{list_yaml(cryptid_flagged)}
build_approach: "subagents"
build_model: "claude-haiku-4-5"
build_at: "{build_at}"
---
"""]
chunks_by_page: dict = {}
for chunk in all_chunks:
p = chunk.get("page", 1)
chunks_by_page.setdefault(p, []).append(chunk)
for page_seq in sorted(chunks_by_page.keys()):
png_num = page_seq - 1
doc_parts.append(f"\n## Page {page_seq} (source: p-{png_num:03d}.png)\n")
for chunk in sorted(chunks_by_page[page_seq], key=lambda c: c.get("order_in_page", 1)):
chunk_id = chunk["chunk_id"]
ctype = chunk.get("type", "paragraph")
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1})
bbox_str = f"{bbox.get('x',0):.2f}/{bbox.get('y',0):.2f}/{bbox.get('w',1):.2f}/{bbox.get('h',0.1):.2f}"
doc_parts.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->\n")
doc_parts.append(f'<a id="{chunk_id}"></a>\n')
doc_parts.append(f"### Chunk {chunk_id} — {ctype} · p{page_seq} · bbox: {bbox_str}\n\n")
doc_parts.append(f"**EN:** {chunk.get('content_en', '')}\n\n")
doc_parts.append(f"**PT-BR:** {chunk.get('content_pt_br', '')}\n\n")
if ctype == "image":
doc_parts.append(f"![{chunk_id} image](./images/IMG-{chunk_id}.png)\n\n")
if chunk.get("image_description_en"):
doc_parts.append(f"**Image Description (EN):** {chunk['image_description_en']}\n\n")
if chunk.get("image_description_pt_br"):
doc_parts.append(f"**Descrição da Imagem (PT-BR):** {chunk['image_description_pt_br']}\n\n")
meta = {k: chunk.get(k) for k in [
"chunk_id", "type", "page", "order_in_page", "order_global",
"bbox", "classification", "formatting", "cross_page_hint",
"prev_chunk", "next_chunk", "ocr_confidence", "redaction_code",
"image_type", "ufo_anomaly_detected", "cryptid_anomaly_detected",
"ufo_anomaly_type", "ufo_anomaly_rationale",
]}
meta_json = json.dumps(meta, indent=2, ensure_ascii=False)
doc_parts.append(
f"<details><summary>metadata</summary>\n\n```json\n{meta_json}\n```\n\n</details>\n\n---\n\n"
)
doc_md = "".join(doc_parts)
(OUT_DIR / "document.md").write_text(doc_md, encoding="utf-8")
print(f" Wrote document.md ({len(doc_md):,} chars)", flush=True)
return images_extracted, ufo_flagged, cryptid_flagged
def main():
t_start = time.time()
CHUNKS_DIR.mkdir(parents=True, exist_ok=True)
IMAGES_DIR.mkdir(parents=True, exist_ok=True)
TABLES_DIR.mkdir(parents=True, exist_ok=True)
print(f"=== Phase A: Process pages {START_PAGE}-{TOTAL_PAGES} via claude CLI ===", flush=True)
pages_to_process = list(range(START_PAGE, TOTAL_PAGES + 1))
new_page_results: dict = {}
for batch_start in range(0, len(pages_to_process), BATCH_SIZE):
batch = pages_to_process[batch_start:batch_start + BATCH_SIZE]
print(f" Batch pages {batch[0]}-{batch[-1]}...", flush=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=BATCH_SIZE) as ex:
futures = {ex.submit(rebuild_page, p): p for p in batch}
for fut in concurrent.futures.as_completed(futures):
result = fut.result()
new_page_results[result["page_number"]] = result
# Assign global chunk IDs (continuing from c0204)
print(f"\n=== Phase A2: Numbering new chunks from c{FIRST_CHUNK_NUM:04d} ===", flush=True)
new_chunks = []
order_global = FIRST_CHUNK_NUM - 1
for page_num in sorted(new_page_results.keys()):
result = new_page_results[page_num]
png_num = result.get("png_num", page_num - 1)
for ch in sorted(result.get("chunks", []), key=lambda c: c.get("order_in_page", 0)):
order_global += 1
ch["chunk_id"] = f"c{order_global:04d}"
ch["order_global"] = order_global
ch["png_num"] = png_num
new_chunks.append(ch)
# prev/next links (will be re-linked globally in Phase B)
for i, ch in enumerate(new_chunks):
ch["prev_chunk"] = new_chunks[i-1]["chunk_id"] if i > 0 else None
ch["next_chunk"] = new_chunks[i+1]["chunk_id"] if i < len(new_chunks)-1 else None
print(f" {len(new_chunks)} new chunks generated", flush=True)
# Crop images
image_chunks = [c for c in new_chunks if c.get("type") == "image"]
if image_chunks:
print(f"\n=== Phase A3: Cropping {len(image_chunks)} images ===", flush=True)
for ch in image_chunks:
crop_image(ch)
# Write new chunk files
print(f"\n=== Phase A4: Writing {len(new_chunks)} new chunk files ===", flush=True)
for ch in new_chunks:
write_chunk_file(ch)
# ── Phase B: Read ALL chunks and rebuild assembly ──────────────────────
print(f"\n=== Phase B: Reading all chunk files for full assembly ===", flush=True)
all_chunk_files = sorted(CHUNKS_DIR.glob("c*.md"))
print(f" Found {len(all_chunk_files)} total chunk files", flush=True)
all_chunks = []
for path in all_chunk_files:
fm = parse_frontmatter(path)
if not fm.get("chunk_id"):
fm["chunk_id"] = path.stem
all_chunks.append(fm)
# Sort by order_global
all_chunks.sort(key=lambda c: (c.get("order_global", 999999), c.get("page", 0), c.get("order_in_page", 0)))
# Re-link prev/next globally
for i, ch in enumerate(all_chunks):
ch["prev_chunk"] = all_chunks[i-1]["chunk_id"] if i > 0 else None
ch["next_chunk"] = all_chunks[i+1]["chunk_id"] if i < len(all_chunks)-1 else None
print(f" Total chunks: {len(all_chunks)}", flush=True)
print(f"\n=== Phase B2: Building _index.json and document.md ===", flush=True)
build_at = datetime.now(timezone.utc).isoformat()
images_extracted, ufo_flagged, cryptid_flagged = build_assembly(all_chunks, build_at)
t_end = time.time()
wall_seconds = int(t_end - t_start)
pages_done = TOTAL_PAGES
chunks_total = len(all_chunks)
tables_stitched = 0
final = (
f"pages_done={pages_done}, chunks_total={chunks_total}, "
f"images_extracted={images_extracted}, tables_stitched={tables_stitched}, "
f"ufo_anomalies={len(ufo_flagged)}, cryptid_anomalies={len(cryptid_flagged)}, "
f"wall_seconds={wall_seconds}"
)
print(f"\n=== DONE ===\n{final}", flush=True)
if __name__ == "__main__":
main()

507
scripts/rebuild_doc65_suba_final.py Normal file
View file

@ -0,0 +1,507 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
rebuild_doc65_suba_final.py
Full rebuild of doc-65-hs1-834228961-62-hq-83894-sub-a
89 pages (p-000 to p-063, p-100 to p-124 PNGs)
Uses Anthropic claude-haiku-4-5 for vision processing.
"""
import os
import sys
import json
import base64
import time
import re
import threading
from datetime import datetime, timezone
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from PIL import Image as PILImage
import anthropic
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-sub-a"
DOC_TITLE = "FBI HQ 62-HQ-83894 Sub A — Flying Saucers / UAP Investigation File"
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
OCR_DIR = Path(f"/Users/guto/ufo/processing/ocr/{DOC_ID}")
OUT_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
BATCH_SIZE = 4
MAX_WORKERS = 4
_lock = threading.Lock()
def safe_print(*args, **kwargs):
with _lock:
print(*args, **kwargs, flush=True)
# Ensure dirs
for d in [CHUNKS_DIR, IMAGES_DIR, TABLES_DIR]:
d.mkdir(parents=True, exist_ok=True)
client = anthropic.Anthropic()
# Build ordered list of PNG files
png_files = sorted(PNG_DIR.glob("p-*.png"))
TOTAL_PAGES = len(png_files)
safe_print(f"Found {TOTAL_PAGES} PNG pages")
def load_image_b64(path: Path) -> str:
with open(path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
def load_ocr(png_name: str) -> str:
ocr_name = png_name.replace(".png", ".txt")
ocr_path = OCR_DIR / ocr_name
if ocr_path.exists():
txt = ocr_path.read_text(encoding="utf-8").strip()
if txt:
return txt[:3000]
return "(no OCR text available — use vision only)"
def extract_json(text: str) -> dict:
text = text.strip()
text = re.sub(r"^```(?:json)?\s*", "", text)
text = re.sub(r"\s*```\s*$", "", text)
start = text.find("{")
if start == -1:
raise ValueError("No JSON found")
depth = 0
for i, c in enumerate(text[start:], start):
if c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
return json.loads(text[start:i+1])
raise ValueError("Unclosed JSON")
PAGE_PROMPT = """You are a page-rebuilder for a declassified UAP/UFO FBI document archive.
Document: {doc_title}
Doc ID: {doc_id}
Page: {page_number} of {total_pages}
PNG: {png_filename}
OCR text:
---
{ocr_text}
---
Analyze this page image carefully. Extract ALL content as ordered semantic chunks.
Return ONLY valid JSON (no markdown, no fences):
{{
"page_number": {page_number},
"chunks": [
{{
"order_in_page": 1,
"type": "cover|letterhead|classification_banner|subject_line|salutation|body_paragraph|signature_block|date_line|reference_line|redaction_block|table_marker|image|caption|footer|header|list_item|handwritten_note|stamp|page_number|section_heading|blank",
"content_en": "verbatim text or description in English",
"content_pt_br": "tradução em português brasileiro",
"bbox": {{"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.85,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}}
]
}}
Rules:
1. Every visible region = its own chunk. Do not skip content.
2. For images: set image_type to photograph|diagram|map|sketch|stamp|chart|handwriting|newspaper_clipping|other
3. For redaction_block: set redaction_code to visible FOIA code if shown.
4. For classification banners/stamps: set classification field to exact text.
5. ufo_anomaly_detected=true if content has UAP/UFO sighting details, craft descriptions, anomalous phenomena.
6. cross_page_hint: "self_contained"|"continues_to_next"|"continues_from_prev"
7. bbox: normalized 0.0-1.0 (x=left, y=top, w=width, h=height).
8. formatting: ["bold","italic","all_caps","underline","strikethrough"]
9. Newspaper clippings = type "image", image_type="newspaper_clipping", ufo_anomaly_detected=true if about UFOs.
10. Return ONLY the JSON object, nothing else."""
def fallback_chunk(page_number: int, reason: str) -> dict:
return {
"page_number": page_number,
"chunks": [{
"order_in_page": 1,
"type": "body_paragraph",
"content_en": f"[Page {page_number} - processing failed: {reason[:80]}]",
"content_pt_br": f"[Página {page_number} - falha no processamento: {reason[:80]}]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None, "formatting": [], "cross_page_hint": "self_contained",
"ocr_confidence": 0.0, "ocr_source_lines": [], "redaction_code": None,
"redaction_inferred_content_type": None, "image_type": None,
"ufo_anomaly_detected": False, "ufo_anomaly_type": None, "ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False, "cryptid_anomaly_type": None, "cryptid_anomaly_rationale": None,
}]
}
def process_page(page_idx: int, png_path: Path) -> dict:
page_number = page_idx + 1
png_filename = png_path.name
ocr_text = load_ocr(png_filename)
img_b64 = load_image_b64(png_path)
prompt = PAGE_PROMPT.format(
doc_title=DOC_TITLE, doc_id=DOC_ID,
page_number=page_number, total_pages=TOTAL_PAGES,
png_filename=png_filename, ocr_text=ocr_text,
)
for attempt in range(3):
try:
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=4096,
messages=[{
"role": "user",
"content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": img_b64}},
{"type": "text", "text": prompt},
],
}],
)
raw = response.content[0].text
data = extract_json(raw)
data["page_number"] = page_number
data["png_path"] = str(png_path)
data["png_filename"] = png_filename
safe_print(f" p{page_number} ({png_filename}): {len(data.get('chunks',[]))} chunks")
return data
except json.JSONDecodeError as e:
safe_print(f" p{page_number} JSON error attempt {attempt+1}: {e}")
if attempt == 2:
return fallback_chunk(page_number, f"JSON parse: {e}")
except Exception as e:
safe_print(f" p{page_number} error attempt {attempt+1}: {e}")
if attempt < 2:
time.sleep(2 ** attempt)
else:
return fallback_chunk(page_number, str(e))
IMAGE_ANALYST_PROMPT = """You are an image analyst for a declassified FBI UAP/UFO document archive.
Analyze this cropped image from FBI file 62-HQ-83894 about Flying Saucers/UAP.
Return ONLY valid JSON (no markdown, no fences):
{{
"image_description_en": "detailed English description",
"image_description_pt_br": "descrição detalhada em português brasileiro",
"image_type": "photograph|diagram|map|sketch|stamp|chart|handwriting|newspaper_clipping|other",
"extracted_text": "visible text verbatim or null",
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}}"""
def crop_and_analyze_image(chunk: dict) -> dict:
chunk_id = chunk["chunk_id"]
png_path = chunk["png_path"]
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0})
# Crop
try:
im = PILImage.open(png_path)
W, H = im.size
x, y, w, h = bbox.get("x",0), bbox.get("y",0), bbox.get("w",1), bbox.get("h",1)
pad = 0.005
left = max(0, int((x-pad)*W))
top = max(0, int((y-pad)*H))
right = min(W, int((x+w+pad)*W))
bottom = min(H, int((y+h+pad)*H))
crop = im.crop((left, top, right, bottom))
out_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
crop.save(str(out_path))
img_b64 = load_image_b64(out_path)
except Exception as e:
safe_print(f" Crop error {chunk_id}: {e}")
return chunk
# Analyze
try:
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=1024,
messages=[{
"role": "user",
"content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": img_b64}},
{"type": "text", "text": IMAGE_ANALYST_PROMPT},
],
}],
)
raw = response.content[0].text
analysis = extract_json(raw)
for key in ["image_description_en","image_description_pt_br","image_type","extracted_text",
"ufo_anomaly_detected","ufo_anomaly_type","ufo_anomaly_rationale",
"cryptid_anomaly_detected","cryptid_anomaly_type","cryptid_anomaly_rationale"]:
if key in analysis:
chunk[key] = analysis[key]
safe_print(f" image analyzed: {chunk_id} ufo={chunk.get('ufo_anomaly_detected',False)}")
except Exception as e:
safe_print(f" Image analysis error {chunk_id}: {e}")
return chunk
def yaml_val(v):
if v is None:
return "null"
if isinstance(v, bool):
return "true" if v else "false"
if isinstance(v, list):
if not v:
return "[]"
return "[" + ", ".join(json.dumps(i, ensure_ascii=False) for i in v) + "]"
return json.dumps(v, ensure_ascii=False)
def write_chunk_file(chunk: dict):
chunk_id = chunk["chunk_id"]
bbox = chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0})
chunk_type = chunk.get("type", "body_paragraph")
related_image = f"IMG-{chunk_id}.png" if chunk_type == "image" else None
png_filename = chunk.get("png_filename", "")
fm = f"""---
chunk_id: {chunk_id}
type: {chunk_type}
page: {chunk['page']}
order_in_page: {chunk.get('order_in_page', 1)}
order_global: {chunk['order_global']}
bbox: {{x: {bbox.get('x',0):.2f}, y: {bbox.get('y',0):.2f}, w: {bbox.get('w',1):.2f}, h: {bbox.get('h',1):.2f}}}
classification: {yaml_val(chunk.get('classification'))}
formatting: {yaml_val(chunk.get('formatting', []))}
cross_page_hint: {chunk.get('cross_page_hint', 'self_contained')}
prev_chunk: {yaml_val(chunk.get('prev_chunk'))}
next_chunk: {yaml_val(chunk.get('next_chunk'))}
related_image: {yaml_val(related_image)}
related_table: {yaml_val(chunk.get('related_table'))}
ocr_confidence: {chunk.get('ocr_confidence', 0.8)}
ocr_source_lines: {yaml_val(chunk.get('ocr_source_lines', []))}
redaction_code: {yaml_val(chunk.get('redaction_code'))}
redaction_inferred_content_type: {yaml_val(chunk.get('redaction_inferred_content_type'))}
image_type: {yaml_val(chunk.get('image_type'))}
ufo_anomaly_detected: {yaml_val(chunk.get('ufo_anomaly_detected', False))}
cryptid_anomaly_detected: {yaml_val(chunk.get('cryptid_anomaly_detected', False))}
ufo_anomaly_type: {yaml_val(chunk.get('ufo_anomaly_type'))}
ufo_anomaly_rationale: {yaml_val(chunk.get('ufo_anomaly_rationale'))}
cryptid_anomaly_type: {yaml_val(chunk.get('cryptid_anomaly_type'))}
cryptid_anomaly_rationale: {yaml_val(chunk.get('cryptid_anomaly_rationale'))}
image_description_en: {yaml_val(chunk.get('image_description_en'))}
image_description_pt_br: {yaml_val(chunk.get('image_description_pt_br'))}
extracted_text: {yaml_val(chunk.get('extracted_text'))}
source_png: ../../processing/png/{DOC_ID}/{png_filename}
---
**EN:** {chunk.get('content_en', '')}
**PT-BR:** {chunk.get('content_pt_br', '')}
"""
(CHUNKS_DIR / f"{chunk_id}.md").write_text(fm, encoding="utf-8")
def main():
start = time.time()
safe_print(f"=== Rebuild {DOC_ID} ===")
safe_print(f"Total pages: {TOTAL_PAGES}")
# Phase 1: Process pages in batches
all_pages = []
page_items = list(enumerate(png_files)) # (idx, path)
for batch_start in range(0, TOTAL_PAGES, BATCH_SIZE):
batch = page_items[batch_start: batch_start + BATCH_SIZE]
safe_print(f"Batch pages {[b[0]+1 for b in batch]}...")
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as ex:
futs = {ex.submit(process_page, idx, pth): idx for idx, pth in batch}
for fut in as_completed(futs):
result = fut.result()
all_pages.append(result)
all_pages.sort(key=lambda p: p["page_number"])
# Phase 2: Global chunk numbering
global_chunks = []
chunk_counter = 1
for page_data in all_pages:
page_chunks = sorted(page_data.get("chunks", []), key=lambda c: c.get("order_in_page", 1))
for chunk in page_chunks:
chunk["chunk_id"] = f"c{chunk_counter:04d}"
chunk["page"] = page_data["page_number"]
chunk["png_path"] = page_data["png_path"]
chunk["png_filename"] = page_data["png_filename"]
chunk["order_global"] = chunk_counter
global_chunks.append(chunk)
chunk_counter += 1
total_chunks = len(global_chunks)
safe_print(f"Total chunks: {total_chunks}")
# Set prev/next
for i, chunk in enumerate(global_chunks):
chunk["prev_chunk"] = global_chunks[i-1]["chunk_id"] if i > 0 else None
chunk["next_chunk"] = global_chunks[i+1]["chunk_id"] if i < total_chunks-1 else None
# Phase 3: Crop & analyze images
image_chunks = [c for c in global_chunks if c.get("type") == "image"]
safe_print(f"Image chunks: {len(image_chunks)}")
for batch_start in range(0, len(image_chunks), BATCH_SIZE):
batch = image_chunks[batch_start: batch_start + BATCH_SIZE]
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as ex:
futs = {ex.submit(crop_and_analyze_image, chunk): chunk["chunk_id"] for chunk in batch}
for fut in as_completed(futs):
fut.result() # side-effects already applied
# Phase 4: Write chunk files
safe_print("Writing chunk files...")
for chunk in global_chunks:
write_chunk_file(chunk)
# Phase 5: Write _index.json
safe_print("Writing _index.json...")
build_at = datetime.now(timezone.utc).isoformat()
index_chunks = []
for chunk in global_chunks:
index_chunks.append({
"chunk_id": chunk["chunk_id"],
"type": chunk.get("type", "body_paragraph"),
"page": chunk["page"],
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk["order_global"],
"file": f"chunks/{chunk['chunk_id']}.md",
"bbox": chunk.get("bbox", {"x":0,"y":0,"w":1,"h":1}),
"preview": chunk.get("content_en","")[:80].replace("\n"," "),
})
index_data = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": total_chunks,
"build_approach": "subagents",
"build_model": "claude-haiku-4-5",
"build_at": build_at,
"chunks": index_chunks,
}
(OUT_DIR / "_index.json").write_text(json.dumps(index_data, ensure_ascii=False, indent=2), encoding="utf-8")
# Phase 6: Assemble document.md
safe_print("Assembling document.md...")
type_hist = {}
for chunk in global_chunks:
t = chunk.get("type","body_paragraph")
type_hist[t] = type_hist.get(t,0)+1
ufo_flagged = [c["chunk_id"] for c in global_chunks if c.get("ufo_anomaly_detected")]
cryptid_flagged = [c["chunk_id"] for c in global_chunks if c.get("cryptid_anomaly_detected")]
hist_yaml = "\n".join(f" {k}: {v}" for k,v in sorted(type_hist.items()))
doc_parts = [f"""---
schema_version: "0.2.0"
type: master_document
doc_id: {DOC_ID}
canonical_title: "{DOC_TITLE}"
total_pages: {TOTAL_PAGES}
total_chunks: {total_chunks}
chunk_types_histogram:
{hist_yaml}
multi_page_tables: []
ufo_anomalies_flagged: {json.dumps(ufo_flagged, ensure_ascii=False)}
cryptid_anomalies_flagged: {json.dumps(cryptid_flagged, ensure_ascii=False)}
build_approach: "subagents"
build_model: "claude-haiku-4-5"
build_at: "{build_at}"
---
"""]
chunks_by_page = {}
for chunk in global_chunks:
p = chunk["page"]
chunks_by_page.setdefault(p, []).append(chunk)
for page_num in sorted(chunks_by_page.keys()):
doc_parts.append(f"\n## Page {page_num}\n\n")
for chunk in chunks_by_page[page_num]:
cid = chunk["chunk_id"]
ctype = chunk.get("type","body_paragraph")
bbox = chunk.get("bbox",{})
bbox_str = f"{bbox.get('x',0):.2f}/{bbox.get('y',0):.2f}/{bbox.get('w',1):.2f}/{bbox.get('h',1):.2f}"
doc_parts.append(f"<!-- chunk:{cid} src:./chunks/{cid}.md -->\n")
doc_parts.append(f'<a id="{cid}"></a>\n')
doc_parts.append(f"### Chunk {cid} — {ctype} · p{page_num} · bbox: {bbox_str}\n\n")
if ctype == "image":
doc_parts.append(f"![chunk image](./images/IMG-{cid}.png)\n\n")
d_en = chunk.get("image_description_en")
d_pt = chunk.get("image_description_pt_br")
if d_en:
doc_parts.append(f"**Image Description (EN):** {d_en}\n\n")
if d_pt:
doc_parts.append(f"**Descrição da Imagem (PT-BR):** {d_pt}\n\n")
doc_parts.append(f"**EN:** {chunk.get('content_en','')}\n\n")
doc_parts.append(f"**PT-BR:** {chunk.get('content_pt_br','')}\n\n")
meta = {
"chunk_id": cid, "type": ctype,
"page": page_num, "order_in_page": chunk.get("order_in_page",1),
"order_global": chunk["order_global"],
"bbox": chunk.get("bbox",{}),
"classification": chunk.get("classification"),
"formatting": chunk.get("formatting",[]),
"cross_page_hint": chunk.get("cross_page_hint","self_contained"),
"prev_chunk": chunk.get("prev_chunk"),
"next_chunk": chunk.get("next_chunk"),
"ocr_confidence": chunk.get("ocr_confidence",0.8),
"redaction_code": chunk.get("redaction_code"),
"image_type": chunk.get("image_type"),
"ufo_anomaly_detected": chunk.get("ufo_anomaly_detected",False),
"ufo_anomaly_type": chunk.get("ufo_anomaly_type"),
"cryptid_anomaly_detected": chunk.get("cryptid_anomaly_detected",False),
"source_png": f"../../processing/png/{DOC_ID}/{chunk.get('png_filename','')}",
}
doc_parts.append("<details><summary>metadata</summary>\n\n```json\n")
doc_parts.append(json.dumps(meta, ensure_ascii=False, indent=2))
doc_parts.append("\n```\n\n</details>\n\n---\n\n")
doc_content = "".join(doc_parts)
(OUT_DIR / "document.md").write_text(doc_content, encoding="utf-8")
doc_md_bytes = len(doc_content.encode("utf-8"))
elapsed = int(time.time() - start)
safe_print(f"\nSTATS pages={TOTAL_PAGES} chunks={total_chunks} images={len(image_chunks)} tables=0 ufo={len(ufo_flagged)} cryptid={len(cryptid_flagged)} doc_md_bytes={doc_md_bytes}")
print(f"pages_done={TOTAL_PAGES}, chunks_total={total_chunks}, images_extracted={len(image_chunks)}, tables_stitched=0, ufo_anomalies={len(ufo_flagged)}, cryptid_anomalies={len(cryptid_flagged)}, wall_seconds={elapsed}")
if __name__ == "__main__":
main()

272
scripts/rebuild_doc65_v2.py Normal file
View file

@ -0,0 +1,272 @@
#!/usr/bin/env python3
"""
Rebuild doc-65-hs1-834228961-62-hq-83894-section-1
Uses claude CLI (OAuth, Max plan) via subprocess — no direct API key needed.
Processes pages 1-150 in parallel batches of 5.
"""
from __future__ import annotations
import base64
import json
import os
import re
import subprocess
import sys
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from pathlib import Path
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-1"
DOC_TITLE = "65 HS1-834228961 62-HQ-83894 Section 1 (FBI UAP/UFO Investigative File)"
PNG_DIR = Path("/Users/guto/ufo/processing/png/doc-65-hs1-834228961-62-hq-83894-section-1")
OCR_DIR = Path("/Users/guto/ufo/processing/ocr/doc-65-hs1-834228961-62-hq-83894-section-1")
OUTPUT_DIR = Path("/Users/guto/ufo/raw/doc-65-hs1-834228961-62-hq-83894-section-1")
TOTAL_PAGES = 150
MAX_WORKERS = 4
TIMEOUT = 180
RETRIES = 3
_lock = threading.Lock()
def safe_print(*args, **kwargs):
with _lock:
print(*args, **kwargs, flush=True)
def load_ocr(page_num: int) -> str:
txt_path = OCR_DIR / f"p-{page_num:03d}.txt"
if txt_path.exists():
try:
content = txt_path.read_text(encoding="utf-8").strip()
return content[:3000] if content else "(empty)"
except Exception:
return "(unreadable)"
return "(not found)"
def extract_json(text: str) -> dict:
"""Extract JSON object from text, stripping markdown fences."""
text = text.strip()
if text.startswith("```"):
text = re.sub(r"^```(?:json)?\s*", "", text)
text = re.sub(r"\s*```\s*$", "", text)
start = text.find("{")
if start == -1:
raise ValueError("No JSON object found")
depth = 0
for i, c in enumerate(text[start:], start):
if c == "{":
depth += 1
elif c == "}":
depth -= 1
if depth == 0:
return json.loads(text[start:i + 1])
raise ValueError("Unclosed JSON object")
PAGE_PROMPT_TEMPLATE = """You are a page-rebuilder for a declassified UAP/UFO government document.
Document: {doc_title}
Page: {page_number} of {total_pages}
STEP 1: Use the Read tool to view this image: {png_path}
STEP 2: Analyze the page carefully and extract ALL content as structured chunks.
STEP 3: Output ONLY a valid JSON object (no markdown, no code fences, no preamble):
{{
"page_number": {page_number},
"chunks": [
{{
"order_in_page": 1,
"type": "paragraph",
"content_en": "verbatim text or description in English",
"content_pt_br": "tradução em português brasileiro",
"bbox": {{"x": 0.05, "y": 0.10, "w": 0.90, "h": 0.05}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.85,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null,
"image_description_en": null,
"image_description_pt_br": null,
"extracted_text": null
}}
]
}}
ALLOWED chunk types (use only these exact strings):
letterhead, classification_banner, header, subheader, paragraph, list_item,
caption, footnote, page_number, signature_block, stamp, redaction_block,
image, table_marker, form_field, watermark, separator, blank
RULES:
1. Extract EVERY visible element — no skipping
2. bbox: normalized 0.0–1.0 (x=left, y=top, w=width, h=height)
3. content_en: verbatim OCR text for text elements; description for images
4. content_pt_br: Brazilian Portuguese (NOT European) translation
5. Preserve UTF-8 accents: ç, ã, á, é, í, ó, ú, â, ê, ô, à
6. Redacted sections: type="redaction_block", content_en="[REDACTED]", content_pt_br="[REDACTADO]"
7. Images/photos: type="image", fill image_description_en and image_description_pt_br
8. classification: visible marking text (e.g. "SECRET", "UNCLASSIFIED") or null
9. formatting: subset of ["bold","italic","underline","all_caps","handwritten","typewritten","strikethrough"]
10. cross_page_hint: "self_contained" | "continues_to_next" | "continues_from_prev"
11. ufo_anomaly_detected: true if chunk contains UAP sighting data, coordinates, witness accounts
12. Blank page: one chunk type="blank"
13. Order chunks top-to-bottom, left-to-right
14. Return ONLY the JSON — no text before or after
OCR hint (may be empty):
{ocr_text}
"""
def process_page(page_num: int) -> dict:
png_path = PNG_DIR / f"p-{page_num:03d}.png"
if not png_path.exists():
safe_print(f" WARNING p{page_num:03d}: PNG missing")
return _error_page(page_num, "[PAGE NOT FOUND]", "[PÁGINA NÃO ENCONTRADA]")
ocr_text = load_ocr(page_num)
prompt = PAGE_PROMPT_TEMPLATE.format(
doc_title=DOC_TITLE,
page_number=page_num,
total_pages=TOTAL_PAGES,
png_path=str(png_path),
ocr_text=ocr_text,
)
for attempt in range(1, RETRIES + 1):
try:
cmd = [
"claude", "-p",
"--model", "haiku",
"--output-format", "json",
"--max-turns", "3",
"--allowedTools", "Read",
"--add-dir", str(PNG_DIR),
"--",
prompt,
]
res = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=TIMEOUT,
check=False,
)
if res.returncode != 0:
raise RuntimeError(f"claude CLI rc={res.returncode}: {res.stderr[-1000:]}")
cli_out = json.loads(res.stdout)
if cli_out.get("is_error"):
raise RuntimeError(f"claude error: {cli_out.get('result','')[:500]}")
result_text = cli_out.get("result", "")
data = extract_json(result_text)
data["page_number"] = page_num
n_chunks = len(data.get("chunks", []))
safe_print(f" p{page_num:03d} OK — {n_chunks} chunks")
return data
except subprocess.TimeoutExpired:
safe_print(f" p{page_num:03d} TIMEOUT (attempt {attempt})")
if attempt == RETRIES:
return _error_page(page_num, "[TIMEOUT]", "[TIMEOUT]")
time.sleep(5 * attempt)
except (RuntimeError, json.JSONDecodeError, ValueError) as e:
safe_print(f" p{page_num:03d} ERROR (attempt {attempt}): {str(e)[:200]}")
if attempt == RETRIES:
return _error_page(page_num, f"[ERROR: {str(e)[:80]}]", f"[ERRO: {str(e)[:80]}]")
time.sleep(5 * attempt)
return _error_page(page_num, "[UNKNOWN ERROR]", "[ERRO DESCONHECIDO]")
def _error_page(page_num: int, msg_en: str, msg_pt: str) -> dict:
return {
"page_number": page_num,
"chunks": [{
"order_in_page": 1,
"type": "blank",
"content_en": msg_en,
"content_pt_br": msg_pt,
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None,
}]
}
def main():
pages = list(range(1, TOTAL_PAGES + 1))
results: dict[int, dict] = {}
start_time = time.time()
print(f"Processing {len(pages)} pages, {MAX_WORKERS} workers, batches of 5...")
batch_size = 5
for b_start in range(0, len(pages), batch_size):
batch = pages[b_start:b_start + batch_size]
print(f"\nBatch {b_start//batch_size + 1}/{(len(pages)+batch_size-1)//batch_size}: pages {batch[0]}-{batch[-1]}")
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as ex:
futures = {ex.submit(process_page, p): p for p in batch}
for fut in as_completed(futures):
p = futures[fut]
try:
results[p] = fut.result()
except Exception as e:
safe_print(f" p{p:03d} FATAL: {e}")
results[p] = _error_page(p, f"[FATAL: {str(e)[:80]}]", f"[FATAL: {str(e)[:80]}]")
# Pause between batches
if b_start + batch_size < len(pages):
time.sleep(2)
elapsed = time.time() - start_time
sorted_results = [results[p] for p in sorted(results.keys())]
total_chunks = sum(len(r.get("chunks", [])) for r in sorted_results)
out_path = OUTPUT_DIR / "_pages_raw.json"
with open(out_path, "w", encoding="utf-8") as f:
json.dump(sorted_results, f, ensure_ascii=False, indent=2)
print(f"\nDone in {elapsed:.0f}s — {len(sorted_results)} pages, {total_chunks} chunks")
print(f"Saved: {out_path}")
if __name__ == "__main__":
main()

478
scripts/rebuild_doc_65.py Normal file
View file

@ -0,0 +1,478 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Rebuild doc-65-hs1-834228961-62-hq-83894-serial-130
Processes all 91 pages via Claude vision, produces chunks/_index.json/document.md
"""
import os
import sys
import json
import base64
import time
import concurrent.futures
from datetime import datetime, timezone
from pathlib import Path
import anthropic
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-serial-130"
DOC_TITLE = "HQ Air Defense Command - Unidentified Flying Objects Reports (65-HS1-834228961 / 62-HQ-83894 Serial 130)"
PNG_DIR = Path(f"/Users/guto/ufo/processing/png/{DOC_ID}")
OCR_DIR = Path(f"/Users/guto/ufo/processing/ocr/{DOC_ID}")
RAW_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
CHUNKS_DIR = RAW_DIR / "chunks"
IMAGES_DIR = RAW_DIR / "images"
TABLES_DIR = RAW_DIR / "tables"
for d in [CHUNKS_DIR, IMAGES_DIR, TABLES_DIR]:
d.mkdir(parents=True, exist_ok=True)
client = anthropic.Anthropic()
def encode_image(path: Path) -> str:
with open(path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
PAGE_REBUILDER_PROMPT = '''You are a page-rebuilder subagent. Analyze this document page image and extract ALL content as structured chunks.
Document: {doc_title}
Doc ID: {doc_id}
Page number (in sequence): {page_number} of {total_pages}
Source PNG filename: {png_filename}
Return a JSON object with this exact structure:
{{
"page_number": {page_number},
"png_filename": "{png_filename}",
"chunks": [
{{
"order_in_page": 1,
"type": "<type>",
"content_en": "...",
"content_pt_br": "...",
"bbox": {{"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.9,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null,
"image_description_en": null,
"image_description_pt_br": null,
"extracted_text": null
}}
]
}}
Allowed chunk types: letterhead, classification_banner, subject_line, body_paragraph, list_item, signature_block, date_line, address_block, header, footer, redaction_block, table_marker, image, stamp, handwritten_note, page_number_marker, blank
Rules:
1. Create ONE chunk per distinct visual/logical unit. Do not merge unrelated blocks.
2. For classification banners (TOP SECRET, SECRET, CONFIDENTIAL, etc.) at top/bottom of page: type=classification_banner, fill classification field.
3. For any image/photo/diagram/map/sketch: type=image, fill image_type, image_description_en, image_description_pt_br, ufo_anomaly_detected, cryptid_anomaly_detected.
4. For redacted/blacked-out areas: type=redaction_block, fill redaction_code if visible.
5. content_en = exact English transcription of text, verbatim. content_pt_br = Brazilian Portuguese translation of content_en (NOT translation of classification banners/stamps/codes — keep those verbatim in both fields).
6. bbox: normalized coordinates (0.0-1.0): x=left, y=top, w=width, h=height relative to page.
7. formatting: array of applicable: bold, italic, underline, all_caps, strikethrough, handwritten.
8. For cross_page_hint: "continues_to_next" if text clearly continues on next page, "continues_from_prev" if it continues from previous page, "self_contained" otherwise.
9. ocr_confidence: your confidence in the transcription (0.0-1.0).
10. If page is blank: return single chunk type=blank.
11. ufo_anomaly_detected: true if the chunk contains or depicts a UAP/UFO, unidentified aerial phenomenon, unknown object in sky, or anomalous craft. Set ufo_anomaly_type and ufo_anomaly_rationale.
12. IMPORTANT: Return ONLY valid JSON, no markdown code blocks, no explanation.'''
def process_page(page_index: int, png_filename: str, total_pages: int) -> dict:
"""Process a single page and return its chunks."""
png_path = PNG_DIR / png_filename
try:
img_data = encode_image(png_path)
prompt = PAGE_REBUILDER_PROMPT.format(
doc_title=DOC_TITLE,
doc_id=DOC_ID,
page_number=page_index,
total_pages=total_pages,
png_filename=png_filename
)
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=4096,
messages=[
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": img_data
}
},
{
"type": "text",
"text": prompt
}
]
}
]
)
raw_text = response.content[0].text.strip()
# Strip markdown code block if present
if raw_text.startswith("```"):
lines = raw_text.split("\n")
# Remove first and last lines if they are code fences
if lines[0].startswith("```"):
lines = lines[1:]
if lines and lines[-1].strip() == "```":
lines = lines[:-1]
raw_text = "\n".join(lines)
result = json.loads(raw_text)
result["page_index"] = page_index
result["png_filename"] = png_filename
return result
except Exception as e:
print(f" ERROR page {page_index} ({png_filename}): {e}", file=sys.stderr)
# Return minimal fallback
return {
"page_number": page_index,
"page_index": page_index,
"png_filename": png_filename,
"chunks": [
{
"order_in_page": 1,
"type": "blank",
"content_en": f"[Page processing error: {str(e)[:100]}]",
"content_pt_br": f"[Erro de processamento: {str(e)[:100]}]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None
}
]
}
def main():
start_time = time.time()
# Get all PNG files in sorted order
png_files = sorted([f.name for f in PNG_DIR.glob("p-*.png")])
total_pages = len(png_files)
print(f"Processing {total_pages} pages for {DOC_ID}")
# Process in parallel batches of 5
all_page_results = {}
batch_size = 5
for batch_start in range(0, total_pages, batch_size):
batch = png_files[batch_start:batch_start + batch_size]
batch_indices = list(range(batch_start + 1, batch_start + len(batch) + 1))
print(f" Batch {batch_start//batch_size + 1}: pages {batch_indices[0]}-{batch_indices[-1]} ({[b for b in batch]})")
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
futures = {
executor.submit(process_page, idx, fname, total_pages): (idx, fname)
for idx, fname in zip(batch_indices, batch)
}
for future in concurrent.futures.as_completed(futures):
idx, fname = futures[future]
try:
result = future.result(timeout=120)
all_page_results[idx] = result
chunk_count = len(result.get("chunks", []))
print(f" Page {idx} ({fname}): {chunk_count} chunks")
except Exception as e:
print(f" FAILED page {idx} ({fname}): {e}", file=sys.stderr)
# Globally number chunks
print("\nNumbering chunks globally...")
all_chunks = []
global_order = 0
for page_idx in sorted(all_page_results.keys()):
page_data = all_page_results[page_idx]
png_filename = page_data.get("png_filename", f"p-{page_idx:03d}.png")
page_chunks = page_data.get("chunks", [])
# Sort by order_in_page
page_chunks.sort(key=lambda c: c.get("order_in_page", 0))
for chunk in page_chunks:
global_order += 1
chunk_id = f"c{global_order:04d}"
chunk["chunk_id"] = chunk_id
chunk["page"] = page_idx
chunk["order_global"] = global_order
chunk["source_png"] = f"../../processing/png/{DOC_ID}/{png_filename}"
all_chunks.append(chunk)
# Set prev/next pointers
for i, chunk in enumerate(all_chunks):
chunk["prev_chunk"] = all_chunks[i-1]["chunk_id"] if i > 0 else None
chunk["next_chunk"] = all_chunks[i+1]["chunk_id"] if i < len(all_chunks) - 1 else None
# Detect image chunks for cropping
image_chunks = [c for c in all_chunks if c.get("type") == "image"]
print(f"\nFound {len(image_chunks)} image chunks")
# Crop images using PIL
print("Cropping image regions...")
for chunk in image_chunks:
chunk_id = chunk["chunk_id"]
page_idx = chunk["page"]
png_filename = all_page_results[page_idx]["png_filename"]
png_path = PNG_DIR / png_filename
bbox = chunk.get("bbox", {"x": 0, "y": 0, "w": 1, "h": 1})
out_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
try:
from PIL import Image
im = Image.open(png_path)
W, H = im.size
x, y, w, h = bbox.get("x", 0), bbox.get("y", 0), bbox.get("w", 1), bbox.get("h", 1)
pad = 0.005
crop = im.crop((
max(0, int((x - pad) * W)),
max(0, int((y - pad) * H)),
min(W, int((x + w + pad) * W)),
min(H, int((y + h + pad) * H))
))
crop.save(str(out_path))
chunk["related_image"] = f"IMG-{chunk_id}.png"
print(f" Cropped {chunk_id} from {png_filename}")
except Exception as e:
print(f" CROP ERROR {chunk_id}: {e}", file=sys.stderr)
chunk["related_image"] = None
# For non-image chunks, set related_image to null
for chunk in all_chunks:
if "related_image" not in chunk:
chunk["related_image"] = None
if "related_table" not in chunk:
chunk["related_table"] = None
# Write individual chunk files
print("\nWriting chunk files...")
for chunk in all_chunks:
chunk_id = chunk["chunk_id"]
chunk_path = CHUNKS_DIR / f"{chunk_id}.md"
bbox = chunk.get("bbox", {"x": 0, "y": 0, "w": 1, "h": 1})
content = f"""---
chunk_id: {chunk_id}
type: {chunk.get('type', 'body_paragraph')}
page: {chunk.get('page', 1)}
order_in_page: {chunk.get('order_in_page', 1)}
order_global: {chunk.get('order_global', 1)}
bbox: {{x: {bbox.get('x', 0):.3f}, y: {bbox.get('y', 0):.3f}, w: {bbox.get('w', 1):.3f}, h: {bbox.get('h', 1):.3f}}}
classification: {json.dumps(chunk.get('classification'))}
formatting: {json.dumps(chunk.get('formatting', []))}
cross_page_hint: {chunk.get('cross_page_hint', 'self_contained')}
prev_chunk: {json.dumps(chunk.get('prev_chunk'))}
next_chunk: {json.dumps(chunk.get('next_chunk'))}
related_image: {json.dumps(chunk.get('related_image'))}
related_table: {json.dumps(chunk.get('related_table'))}
ocr_confidence: {chunk.get('ocr_confidence', 0.9)}
ocr_source_lines: {json.dumps(chunk.get('ocr_source_lines', []))}
redaction_code: {json.dumps(chunk.get('redaction_code'))}
redaction_inferred_content_type: {json.dumps(chunk.get('redaction_inferred_content_type'))}
image_type: {json.dumps(chunk.get('image_type'))}
ufo_anomaly_detected: {str(chunk.get('ufo_anomaly_detected', False)).lower()}
cryptid_anomaly_detected: {str(chunk.get('cryptid_anomaly_detected', False)).lower()}
ufo_anomaly_type: {json.dumps(chunk.get('ufo_anomaly_type'))}
ufo_anomaly_rationale: {json.dumps(chunk.get('ufo_anomaly_rationale'))}
cryptid_anomaly_type: {json.dumps(chunk.get('cryptid_anomaly_type'))}
cryptid_anomaly_rationale: {json.dumps(chunk.get('cryptid_anomaly_rationale'))}
image_description_en: {json.dumps(chunk.get('image_description_en'))}
image_description_pt_br: {json.dumps(chunk.get('image_description_pt_br'))}
extracted_text: {json.dumps(chunk.get('extracted_text'))}
source_png: {chunk.get('source_png', '')}
---
**EN:** {chunk.get('content_en', '')}
**PT-BR:** {chunk.get('content_pt_br', '')}
"""
chunk_path.write_text(content, encoding="utf-8")
print(f" Wrote {len(all_chunks)} chunk files")
# Build _index.json
print("\nBuilding _index.json...")
build_at = datetime.now(timezone.utc).isoformat()
index_chunks = []
for chunk in all_chunks:
bbox = chunk.get("bbox", {"x": 0, "y": 0, "w": 1, "h": 1})
preview = chunk.get("content_en", "")[:80]
index_chunks.append({
"chunk_id": chunk["chunk_id"],
"type": chunk.get("type", "body_paragraph"),
"page": chunk.get("page", 1),
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"file": f"chunks/{chunk['chunk_id']}.md",
"bbox": {
"x": round(bbox.get("x", 0), 3),
"y": round(bbox.get("y", 0), 3),
"w": round(bbox.get("w", 1), 3),
"h": round(bbox.get("h", 1), 3)
},
"preview": preview
})
index_data = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": total_pages,
"total_chunks": len(all_chunks),
"build_approach": "subagents",
"build_model": "claude-sonnet-4-6",
"build_at": build_at,
"chunks": index_chunks
}
index_path = RAW_DIR / "_index.json"
index_path.write_text(json.dumps(index_data, ensure_ascii=False, indent=2), encoding="utf-8")
print(f" Wrote _index.json with {len(all_chunks)} chunks")
# Compute histogram
type_hist = {}
for chunk in all_chunks:
t = chunk.get("type", "unknown")
type_hist[t] = type_hist.get(t, 0) + 1
# Collect anomaly lists
ufo_anomaly_chunks = [c["chunk_id"] for c in all_chunks if c.get("ufo_anomaly_detected")]
cryptid_anomaly_chunks = [c["chunk_id"] for c in all_chunks if c.get("cryptid_anomaly_detected")]
# Assemble document.md
print("\nAssembling document.md...")
doc_lines = []
doc_lines.append(f"""---
schema_version: "0.2.0"
type: master_document
doc_id: {DOC_ID}
canonical_title: "{DOC_TITLE}"
total_pages: {total_pages}
total_chunks: {len(all_chunks)}
chunk_types_histogram: {json.dumps(type_hist)}
multi_page_tables: []
ufo_anomalies_flagged: {json.dumps(ufo_anomaly_chunks)}
cryptid_anomalies_flagged: {json.dumps(cryptid_anomaly_chunks)}
build_approach: "subagents"
build_model: "claude-sonnet-4-6"
build_at: "{build_at}"
---
""")
current_page = None
for chunk in all_chunks:
page = chunk.get("page")
if page != current_page:
current_page = page
png_fn = all_page_results.get(page, {}).get("png_filename", f"p-{page:03d}.png")
doc_lines.append(f"\n## Page {page} (source: {png_fn})\n")
chunk_id = chunk["chunk_id"]
ctype = chunk.get("type", "body_paragraph")
bbox = chunk.get("bbox", {"x": 0, "y": 0, "w": 1, "h": 1})
bbox_str = f"{bbox.get('x',0):.2f}/{bbox.get('y',0):.2f}/{bbox.get('w',1):.2f}/{bbox.get('h',1):.2f}"
doc_lines.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->")
doc_lines.append(f'<a id="{chunk_id}"></a>')
doc_lines.append(f"### Chunk {chunk_id} — {ctype} · p{page} · bbox: {bbox_str}\n")
content_en = chunk.get("content_en", "")
content_pt_br = chunk.get("content_pt_br", "")
doc_lines.append(f"**EN:** {content_en}\n")
doc_lines.append(f"**PT-BR:** {content_pt_br}\n")
# Embed image if applicable
if ctype == "image" and chunk.get("related_image"):
img_file = chunk["related_image"]
doc_lines.append(f"![{chunk_id} image](./images/{img_file})\n")
if chunk.get("image_description_en"):
doc_lines.append(f"*Image description: {chunk['image_description_en']}*\n")
# Metadata details block
meta = {
"chunk_id": chunk_id,
"type": ctype,
"page": page,
"order_in_page": chunk.get("order_in_page"),
"order_global": chunk.get("order_global"),
"bbox": chunk.get("bbox"),
"classification": chunk.get("classification"),
"formatting": chunk.get("formatting", []),
"cross_page_hint": chunk.get("cross_page_hint"),
"prev_chunk": chunk.get("prev_chunk"),
"next_chunk": chunk.get("next_chunk"),
"related_image": chunk.get("related_image"),
"related_table": chunk.get("related_table"),
"ocr_confidence": chunk.get("ocr_confidence"),
"ufo_anomaly_detected": chunk.get("ufo_anomaly_detected", False),
"ufo_anomaly_type": chunk.get("ufo_anomaly_type"),
"ufo_anomaly_rationale": chunk.get("ufo_anomaly_rationale"),
"cryptid_anomaly_detected": chunk.get("cryptid_anomaly_detected", False),
"cryptid_anomaly_type": chunk.get("cryptid_anomaly_type"),
"cryptid_anomaly_rationale": chunk.get("cryptid_anomaly_rationale"),
"image_description_en": chunk.get("image_description_en"),
"image_description_pt_br": chunk.get("image_description_pt_br"),
"source_png": chunk.get("source_png")
}
doc_lines.append("<details><summary>metadata</summary>\n")
doc_lines.append("```json")
doc_lines.append(json.dumps(meta, ensure_ascii=False, indent=2))
doc_lines.append("```\n")
doc_lines.append("</details>\n")
doc_lines.append("---\n")
doc_content = "\n".join(doc_lines)
doc_path = RAW_DIR / "document.md"
doc_path.write_text(doc_content, encoding="utf-8")
wall_seconds = int(time.time() - start_time)
doc_bytes = len(doc_content.encode("utf-8"))
print(f"\nDone!")
print(f" Chunks: {len(all_chunks)}")
print(f" Images: {len(image_chunks)}")
print(f" UFO anomalies: {len(ufo_anomaly_chunks)}")
print(f" Cryptid anomalies: {len(cryptid_anomaly_chunks)}")
print(f" document.md: {doc_bytes} bytes")
print(f" Wall time: {wall_seconds}s")
print(f"\nSTATS pages={total_pages} chunks={len(all_chunks)} images={len(image_chunks)} tables=0 ufo={len(ufo_anomaly_chunks)} cryptid={len(cryptid_anomaly_chunks)} doc_md_bytes={doc_bytes}")
if __name__ == "__main__":
main()

597
scripts/rebuild_doc_d48.py Normal file
View file

@ -0,0 +1,597 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Rebuilds dow-uap-d48-report-september-1996 into harness-assemblable structure.
Processes all 146 pages with vision + OCR, generates chunks, images, index, and document.md
"""
import os
import json
import base64
import re
import csv
import time
import concurrent.futures
from datetime import datetime, timezone
from pathlib import Path
from PIL import Image
import anthropic
DOC_ID = "dow-uap-d48-report-september-1996"
DOC_TITLE = "Modeling Unlikely Space-Booster Failures in Risk Calculations"
BASE_PNG = f"/Users/guto/ufo/processing/png/{DOC_ID}"
BASE_OCR = f"/Users/guto/ufo/processing/ocr/{DOC_ID}"
OUT_DIR = f"/Users/guto/ufo/raw/{DOC_ID}"
CHUNKS_DIR = f"{OUT_DIR}/chunks"
IMAGES_DIR = f"{OUT_DIR}/images"
TABLES_DIR = f"{OUT_DIR}/tables"
os.makedirs(CHUNKS_DIR, exist_ok=True)
os.makedirs(IMAGES_DIR, exist_ok=True)
os.makedirs(TABLES_DIR, exist_ok=True)
client = anthropic.Anthropic()
# All page numbers that have PNGs
PNG_PAGES = [
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,
49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,
100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,
117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,
134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,
151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,
168,169,170,171,172,173,174,175,176,177,178,179,180,181
]
TOTAL_PAGES = len(PNG_PAGES)
PAGE_REBUILDER_PROMPT = """You are a page-rebuilder agent. Analyze the provided page image and OCR text from a declassified technical document and extract all content as structured chunks.
Document: "{doc_title}"
Page number (file): {page_num} (sequential position {seq_pos} of {total_pages})
OCR text:
```
{ocr_text}
```
Return a JSON object with this exact structure:
{{
"page_number": {page_num},
"seq_position": {seq_pos},
"chunks": [
{{
"order_in_page": 1,
"type": "<type>",
"content_en": "<english content>",
"content_pt_br": "<portuguese BR translation>",
"bbox": {{"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1}},
"classification": null,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.9,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null,
"image_description_en": null,
"image_description_pt_br": null,
"extracted_text": null
}}
]
}}
Chunk types (use ONLY these):
- letterhead: institution/org header at top of page
- classification_banner: classification marking (SECRET, TOP SECRET, UNCLASSIFIED, etc.)
- title: document or section title
- subtitle: subtitle or sub-heading
- heading: section heading (numbered or unnumbered)
- subheading: subsection heading
- paragraph: body text paragraph
- list_item: bullet or numbered list item
- table_marker: a table (include table data in content_en as pipe-delimited markdown table)
- figure_caption: caption for a figure or chart
- image: a photograph, diagram, chart, graph, or illustration
- footer: footer text (page numbers, dates, etc.)
- header: running header
- signature_block: signature area
- redaction: redacted/blacked-out area
- page_number: standalone page number
- toc_entry: table of contents entry
- abstract: abstract section
- reference: bibliography/reference entry
- form_field: form field label and value
- metadata_block: document metadata block (e.g., Report Documentation Page)
- appendix_marker: appendix label/header
- blank: intentionally blank area
Rules:
1. Every visible content region becomes a chunk — do not skip anything.
2. For tables: include the full table as markdown pipe-delimited format in content_en.
3. For images/figures: set type=image, describe what you see in image_description_en and image_description_pt_br. Set extracted_text if the image contains text.
4. bbox coordinates: x,y = top-left corner (0-1 normalized), w,h = width/height (0-1 normalized).
5. cross_page_hint: "self_contained" | "continues_to_next" | "continues_from_prev"
6. content_pt_br: full Brazilian Portuguese translation of content_en (NOT European Portuguese).
7. formatting: array of applicable: ["bold", "italic", "all_caps", "underline", "centered", "right_aligned"]
8. classification: null for unclassified content, or the exact marking string if present.
9. ocr_confidence: estimate 0.0-1.0 based on OCR quality.
10. For the abstract, use type=abstract.
11. For TOC entries, each line is a separate toc_entry chunk.
12. For figure captions, use type=figure_caption.
13. ufo_anomaly_detected: true only if content describes UAP/UFO phenomenon (this is a space booster report, very unlikely).
14. Return ONLY valid JSON, no markdown fences, no explanation text.
"""
def read_ocr(page_num):
"""Read OCR text for a page number, return empty string if not found."""
ocr_path = f"{BASE_OCR}/p-{page_num:03d}.txt"
if os.path.exists(ocr_path):
with open(ocr_path, "r", encoding="utf-8", errors="replace") as f:
return f.read()
return ""
def read_png_b64(page_num):
"""Read PNG image as base64."""
png_path = f"{BASE_PNG}/p-{page_num:03d}.png"
with open(png_path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
def process_page(page_num, seq_pos):
"""Process a single page using vision + OCR, return page chunk data."""
ocr_text = read_ocr(page_num)
img_b64 = read_png_b64(page_num)
prompt = PAGE_REBUILDER_PROMPT.format(
doc_title=DOC_TITLE,
page_num=page_num,
seq_pos=seq_pos,
total_pages=TOTAL_PAGES,
ocr_text=ocr_text[:4000] if ocr_text else "(no OCR available)"
)
max_retries = 3
for attempt in range(max_retries):
try:
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=4096,
messages=[
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": img_b64
}
},
{
"type": "text",
"text": prompt
}
]
}
]
)
raw = response.content[0].text.strip()
# Remove markdown fences if present
raw = re.sub(r'^```json\s*', '', raw)
raw = re.sub(r'^```\s*', '', raw)
raw = re.sub(r'\s*```$', '', raw)
data = json.loads(raw)
print(f" [OK] page {page_num:03d} (seq {seq_pos}) -> {len(data.get('chunks', []))} chunks")
return data
except json.JSONDecodeError as e:
print(f" [WARN] page {page_num:03d} JSON parse error (attempt {attempt+1}): {e}")
if attempt == max_retries - 1:
# Return minimal fallback
return {
"page_number": page_num,
"seq_position": seq_pos,
"chunks": [
{
"order_in_page": 1,
"type": "paragraph",
"content_en": ocr_text[:2000] if ocr_text else f"[Page {page_num} - content extraction failed]",
"content_pt_br": f"[Página {page_num} - extração de conteúdo falhou]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.5,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None
}
]
}
except Exception as e:
print(f" [ERROR] page {page_num:03d} (attempt {attempt+1}): {e}")
if attempt < max_retries - 1:
time.sleep(2 ** attempt)
else:
return {
"page_number": page_num,
"seq_position": seq_pos,
"chunks": [
{
"order_in_page": 1,
"type": "paragraph",
"content_en": f"[Page {page_num} - processing error: {str(e)[:100]}]",
"content_pt_br": f"[Página {page_num} - erro de processamento]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None,
"image_description_en": None,
"image_description_pt_br": None,
"extracted_text": None
}
]
}
def process_pages_batch(pages_with_seq):
"""Process a batch of pages concurrently."""
results = {}
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
future_to_page = {
executor.submit(process_page, page_num, seq_pos): (page_num, seq_pos)
for page_num, seq_pos in pages_with_seq
}
for future in concurrent.futures.as_completed(future_to_page):
page_num, seq_pos = future_to_page[future]
try:
result = future.result()
results[seq_pos] = result
except Exception as e:
print(f" [FATAL] page {page_num}: {e}")
results[seq_pos] = {
"page_number": page_num,
"seq_position": seq_pos,
"chunks": []
}
return results
def crop_image_chunk(chunk_id, page_num, bbox):
"""Crop image region from page PNG and save."""
png_path = f"{BASE_PNG}/p-{page_num:03d}.png"
out_path = f"{IMAGES_DIR}/IMG-{chunk_id}.png"
try:
im = Image.open(png_path)
W, H = im.size
x, y, w, h = bbox.get("x", 0), bbox.get("y", 0), bbox.get("w", 1), bbox.get("h", 1)
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
cropped = im.crop((left, top, right, bottom))
cropped.save(out_path)
return out_path
except Exception as e:
print(f" [WARN] crop failed for {chunk_id}: {e}")
return None
def write_chunk_file(chunk_data, page_num):
"""Write individual chunk markdown file."""
chunk_id = chunk_data["chunk_id"]
chunk_type = chunk_data.get("type", "paragraph")
order_in_page = chunk_data.get("order_in_page", 1)
order_global = chunk_data.get("order_global", 1)
bbox = chunk_data.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0})
# Determine related_image and related_table
related_image = f"IMG-{chunk_id}.png" if chunk_type == "image" else None
related_table = chunk_data.get("related_table", None)
prev_chunk = chunk_data.get("prev_chunk", None)
next_chunk = chunk_data.get("next_chunk", None)
content_en = chunk_data.get("content_en", "")
content_pt_br = chunk_data.get("content_pt_br", "")
yaml_lines = [
f"---",
f"chunk_id: {chunk_id}",
f"type: {chunk_type}",
f"page: {page_num}",
f"order_in_page: {order_in_page}",
f"order_global: {order_global}",
f"bbox: {{x: {bbox.get('x',0):.2f}, y: {bbox.get('y',0):.2f}, w: {bbox.get('w',1):.2f}, h: {bbox.get('h',0.1):.2f}}}",
f"classification: {json.dumps(chunk_data.get('classification', None))}",
f"formatting: {json.dumps(chunk_data.get('formatting', []))}",
f"cross_page_hint: {chunk_data.get('cross_page_hint', 'self_contained')}",
f"prev_chunk: {json.dumps(prev_chunk)}",
f"next_chunk: {json.dumps(next_chunk)}",
f"related_image: {json.dumps(related_image)}",
f"related_table: {json.dumps(related_table)}",
f"ocr_confidence: {chunk_data.get('ocr_confidence', 0.9)}",
f"ocr_source_lines: {json.dumps(chunk_data.get('ocr_source_lines', []))}",
f"redaction_code: {json.dumps(chunk_data.get('redaction_code', None))}",
f"redaction_inferred_content_type: {json.dumps(chunk_data.get('redaction_inferred_content_type', None))}",
f"image_type: {json.dumps(chunk_data.get('image_type', None))}",
f"ufo_anomaly_detected: {str(chunk_data.get('ufo_anomaly_detected', False)).lower()}",
f"ufo_anomaly_type: {json.dumps(chunk_data.get('ufo_anomaly_type', None))}",
f"ufo_anomaly_rationale: {json.dumps(chunk_data.get('ufo_anomaly_rationale', None))}",
f"cryptid_anomaly_detected: {str(chunk_data.get('cryptid_anomaly_detected', False)).lower()}",
f"cryptid_anomaly_type: {json.dumps(chunk_data.get('cryptid_anomaly_type', None))}",
f"cryptid_anomaly_rationale: {json.dumps(chunk_data.get('cryptid_anomaly_rationale', None))}",
f"image_description_en: {json.dumps(chunk_data.get('image_description_en', None))}",
f"image_description_pt_br: {json.dumps(chunk_data.get('image_description_pt_br', None))}",
f"extracted_text: {json.dumps(chunk_data.get('extracted_text', None))}",
f"source_png: ../../processing/png/{DOC_ID}/p-{page_num:03d}.png",
f"---",
]
content = "\n".join(yaml_lines) + "\n\n"
content += f"**EN:** {content_en}\n\n"
content += f"**PT-BR:** {content_pt_br}\n"
out_path = f"{CHUNKS_DIR}/{chunk_id}.md"
with open(out_path, "w", encoding="utf-8") as f:
f.write(content)
def main():
start_time = time.time()
print(f"=== Rebuilding {DOC_ID} ===")
print(f"Total pages to process: {TOTAL_PAGES}")
# Create page batches (5 at a time)
pages_with_seq = [(page_num, seq_pos+1) for seq_pos, page_num in enumerate(PNG_PAGES)]
all_page_results = {}
batch_size = 5
for batch_start in range(0, len(pages_with_seq), batch_size):
batch = pages_with_seq[batch_start:batch_start+batch_size]
batch_nums = [p[0] for p in batch]
print(f"\nProcessing batch {batch_start//batch_size + 1}: pages {batch_nums}")
batch_results = process_pages_batch(batch)
all_page_results.update(batch_results)
# Small pause between batches to avoid rate limiting
if batch_start + batch_size < len(pages_with_seq):
time.sleep(0.5)
print(f"\n=== All {TOTAL_PAGES} pages processed ===")
# Global chunk numbering
# Sort results by seq_position
all_chunks = []
for seq_pos in sorted(all_page_results.keys()):
page_result = all_page_results[seq_pos]
page_num = page_result["page_number"]
chunks = page_result.get("chunks", [])
# Sort chunks by order_in_page
chunks_sorted = sorted(chunks, key=lambda c: c.get("order_in_page", 0))
for chunk in chunks_sorted:
all_chunks.append({
**chunk,
"page_number": page_num,
"seq_position": seq_pos
})
# Assign global chunk IDs
for global_idx, chunk in enumerate(all_chunks):
chunk["chunk_id"] = f"c{global_idx+1:04d}"
chunk["order_global"] = global_idx + 1
chunk["prev_chunk"] = f"c{global_idx:04d}" if global_idx > 0 else None
chunk["next_chunk"] = f"c{global_idx+2:04d}" if global_idx < len(all_chunks)-1 else None
print(f"Total chunks: {len(all_chunks)}")
# Crop images and collect image chunks
image_chunks = [c for c in all_chunks if c.get("type") == "image"]
print(f"Image chunks found: {len(image_chunks)}")
for img_chunk in image_chunks:
chunk_id = img_chunk["chunk_id"]
page_num = img_chunk["page_number"]
bbox = img_chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0})
crop_image_chunk(chunk_id, page_num, bbox)
print(f" Cropped image: {chunk_id} from page {page_num}")
# Write individual chunk files
print("\nWriting chunk files...")
for chunk in all_chunks:
write_chunk_file(chunk, chunk["page_number"])
# Build _index.json
print("Writing _index.json...")
index_chunks = []
for chunk in all_chunks:
index_chunks.append({
"chunk_id": chunk["chunk_id"],
"type": chunk.get("type", "paragraph"),
"page": chunk["page_number"],
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk["order_global"],
"file": f"chunks/{chunk['chunk_id']}.md",
"bbox": chunk.get("bbox", {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0}),
"preview": chunk.get("content_en", "")[:80]
})
build_at = datetime.now(timezone.utc).isoformat()
index_data = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": len(all_chunks),
"build_approach": "subagents",
"build_model": "claude-haiku-4-5",
"build_at": build_at,
"chunks": index_chunks
}
with open(f"{OUT_DIR}/_index.json", "w", encoding="utf-8") as f:
json.dump(index_data, f, ensure_ascii=False, indent=2)
# Build document.md
print("Building document.md...")
# Compute histograms and stats
type_histogram = {}
ufo_anomalies = []
cryptid_anomalies = []
for chunk in all_chunks:
t = chunk.get("type", "paragraph")
type_histogram[t] = type_histogram.get(t, 0) + 1
if chunk.get("ufo_anomaly_detected", False):
ufo_anomalies.append(chunk["chunk_id"])
if chunk.get("cryptid_anomaly_detected", False):
cryptid_anomalies.append(chunk["chunk_id"])
doc_lines = []
doc_lines.append("---")
doc_lines.append('schema_version: "0.2.0"')
doc_lines.append("type: master_document")
doc_lines.append(f"doc_id: {DOC_ID}")
doc_lines.append(f'canonical_title: "{DOC_TITLE}"')
doc_lines.append(f"total_pages: {TOTAL_PAGES}")
doc_lines.append(f"total_chunks: {len(all_chunks)}")
doc_lines.append("chunk_types_histogram:")
for t, count in sorted(type_histogram.items()):
doc_lines.append(f" {t}: {count}")
doc_lines.append("multi_page_tables: []")
doc_lines.append(f"ufo_anomalies_flagged: {json.dumps(ufo_anomalies)}")
doc_lines.append(f"cryptid_anomalies_flagged: {json.dumps(cryptid_anomalies)}")
doc_lines.append('build_approach: "subagents"')
doc_lines.append("build_model: claude-haiku-4-5")
doc_lines.append(f"build_at: {build_at}")
doc_lines.append("---")
doc_lines.append("")
# Group chunks by page
chunks_by_page = {}
for chunk in all_chunks:
p = chunk["page_number"]
if p not in chunks_by_page:
chunks_by_page[p] = []
chunks_by_page[p].append(chunk)
for page_num in sorted(chunks_by_page.keys()):
doc_lines.append(f"## Page {page_num}")
doc_lines.append("")
for chunk in chunks_by_page[page_num]:
chunk_id = chunk["chunk_id"]
chunk_type = chunk.get("type", "paragraph")
bbox = chunk.get("bbox", {})
bx = bbox.get("x", 0)
by = bbox.get("y", 0)
bw = bbox.get("w", 1)
bh = bbox.get("h", 0.1)
doc_lines.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->")
doc_lines.append(f'<a id="{chunk_id}"></a>')
doc_lines.append(f"### Chunk {chunk_id} — {chunk_type} · p{page_num} · bbox: {bx:.2f}/{by:.2f}/{bw:.2f}/{bh:.2f}")
doc_lines.append("")
content_en = chunk.get("content_en", "")
content_pt_br = chunk.get("content_pt_br", "")
doc_lines.append(f"**EN:** {content_en}")
doc_lines.append("")
doc_lines.append(f"**PT-BR:** {content_pt_br}")
doc_lines.append("")
# Image embed
if chunk_type == "image":
img_path = f"./images/IMG-{chunk_id}.png"
doc_lines.append(f"![{chunk_id} image]({img_path})")
doc_lines.append("")
if chunk.get("image_description_en"):
doc_lines.append(f"*Image description:* {chunk['image_description_en']}")
doc_lines.append("")
# Metadata collapsible
meta = {
"chunk_id": chunk_id,
"type": chunk_type,
"page": page_num,
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk.get("order_global", 1),
"bbox": bbox,
"classification": chunk.get("classification"),
"formatting": chunk.get("formatting", []),
"cross_page_hint": chunk.get("cross_page_hint", "self_contained"),
"prev_chunk": chunk.get("prev_chunk"),
"next_chunk": chunk.get("next_chunk"),
"ocr_confidence": chunk.get("ocr_confidence", 0.9),
"ufo_anomaly_detected": chunk.get("ufo_anomaly_detected", False),
"cryptid_anomaly_detected": chunk.get("cryptid_anomaly_detected", False),
"image_type": chunk.get("image_type"),
"image_description_en": chunk.get("image_description_en"),
"image_description_pt_br": chunk.get("image_description_pt_br")
}
doc_lines.append("<details><summary>metadata</summary>")
doc_lines.append("")
doc_lines.append("```json")
doc_lines.append(json.dumps(meta, ensure_ascii=False, indent=2))
doc_lines.append("```")
doc_lines.append("")
doc_lines.append("</details>")
doc_lines.append("")
doc_lines.append("---")
doc_lines.append("")
document_md = "\n".join(doc_lines)
with open(f"{OUT_DIR}/document.md", "w", encoding="utf-8") as f:
f.write(document_md)
wall_seconds = int(time.time() - start_time)
doc_md_bytes = len(document_md.encode("utf-8"))
print(f"\n=== DONE ===")
print(f"STATS pages={TOTAL_PAGES} chunks={len(all_chunks)} images={len(image_chunks)} tables=0 ufo={len(ufo_anomalies)} cryptid={len(cryptid_anomalies)} doc_md_bytes={doc_md_bytes}")
print(f"Wall time: {wall_seconds}s")
return {
"pages": TOTAL_PAGES,
"chunks": len(all_chunks),
"images": len(image_chunks),
"tables": 0,
"ufo": len(ufo_anomalies),
"cryptid": len(cryptid_anomalies),
"wall_seconds": wall_seconds,
"doc_md_bytes": doc_md_bytes
}
if __name__ == "__main__":
main()

592
scripts/rebuild_doc_section3.py Normal file
View file

@ -0,0 +1,592 @@
#!/usr/bin/env python3
"""
Rebuild script for doc-65-hs1-834228961-62-hq-83894-section-3
Processes all 155 pages in parallel batches, generates chunks, images, and index.
"""
import os
import json
import base64
import time
import concurrent.futures
from datetime import datetime, timezone
from pathlib import Path
import anthropic
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-section-3"
DOC_TITLE = "65 HS1-834228961 62-HQ-83894 Section 3 — FBI Flying Discs Investigation File"
TOTAL_PAGES = 155
PNG_DIR = Path("/Users/guto/ufo/processing/png") / DOC_ID
OCR_DIR = Path("/Users/guto/ufo/processing/ocr") / DOC_ID
OUT_DIR = Path("/Users/guto/ufo/raw") / DOC_ID
CHUNKS_DIR = OUT_DIR / "chunks"
IMAGES_DIR = OUT_DIR / "images"
TABLES_DIR = OUT_DIR / "tables"
client = anthropic.Anthropic()
CHUNK_TYPES = [
"letterhead", "header", "classification_banner", "subject_line",
"salutation", "body_paragraph", "signature_block", "handwritten_note",
"stamp", "redaction_block", "image", "table_marker", "footer",
"page_number", "attachment_label", "routing_slip", "blank",
"caption", "list_item", "address_block"
]
PAGE_REBUILDER_PROMPT = """You are a page-rebuilder agent for a declassified FBI UAP/UFO document archive.
Your task: Analyze the provided page image and extract ALL content into structured chunks.
Document: {doc_title}
Page: {page_number} of {total_pages}
Page PNG path: {page_png_path}
Return a JSON object with this exact structure:
{{
"page_number": {page_number},
"classification": "<classification string found on page or null>",
"page_type": "<blank|text|image|mixed|cover>",
"chunks": [
{{
"order_in_page": 1,
"type": "<chunk_type>",
"content_en": "<English text content or description>",
"content_pt_br": "<Brazilian Portuguese translation/description>",
"bbox": {{"x": 0.0, "y": 0.0, "w": 1.0, "h": 0.1}},
"classification": "<classification string or null>",
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.9,
"ocr_source_lines": [],
"redaction_code": null,
"redaction_inferred_content_type": null,
"image_type": null,
"ufo_anomaly_detected": false,
"ufo_anomaly_type": null,
"ufo_anomaly_rationale": null,
"cryptid_anomaly_detected": false,
"cryptid_anomaly_type": null,
"cryptid_anomaly_rationale": null
}}
]
}}
RULES:
1. Extract ALL content — no chunk can be skipped.
2. Use ONLY these chunk types: letterhead, header, classification_banner, subject_line, salutation, body_paragraph, signature_block, handwritten_note, stamp, redaction_block, image, table_marker, footer, page_number, attachment_label, routing_slip, blank, caption, list_item, address_block
3. bbox values are normalized 0.0-1.0 (x=left, y=top, w=width, h=height of the page).
4. content_en: verbatim transcription for text, description for images.
5. content_pt_br: Brazilian Portuguese translation of content_en (NOT European Portuguese). For verbatim text blocks, provide both the original (verbatim) and a translation note.
6. For redacted blocks: set type="redaction_block", content_en="[REDACTED]", set redaction_code if visible (e.g., "(b)(1)", "(b)(6)"), redaction_inferred_content_type with your best inference.
7. For images/photos: type="image", image_type = one of: photograph|sketch|diagram|map|chart|logo|signature|stamp|other
8. For tables: type="table_marker"
9. cross_page_hint: "self_contained" | "continues_to_next" | "continues_from_prev"
10. UAP/cryptid anomaly detection: flag any anomalous visual content (UFO shapes, unusual aerial phenomena, cryptid-related imagery).
11. If page is blank or nearly blank: create ONE chunk type="blank".
12. classification_banner chunks at top/bottom of page for classification markings.
13. stamps: type="stamp" for rubber stamps, file numbers, dates stamped on documents.
14. Return ONLY valid JSON, no other text.
IMPORTANT: Be thorough. A typical text page has 5-15 chunks. A photo page may have 2-3 chunks. Cover/envelope pages have 4-8 chunks.
"""
def encode_image_b64(path: Path) -> str:
with open(path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
def process_page(page_num: int) -> dict:
"""Process a single page and return its chunks as a dict."""
# PNG files are p-000.png through p-154.png (zero-indexed)
png_index = page_num - 1 # page 1 = p-000.png
png_path = PNG_DIR / f"p-{png_index:03d}.png"
if not png_path.exists():
print(f" WARNING: PNG not found for page {page_num}: {png_path}")
return {
"page_number": page_num,
"classification": None,
"page_type": "blank",
"chunks": [{
"order_in_page": 1,
"type": "blank",
"content_en": "[Page image not found]",
"content_pt_br": "[Imagem da página não encontrada]",
"bbox": {"x": 0.0, "y": 0.0, "w": 1.0, "h": 1.0},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}]
}
img_b64 = encode_image_b64(png_path)
prompt = PAGE_REBUILDER_PROMPT.format(
doc_title=DOC_TITLE,
page_number=page_num,
total_pages=TOTAL_PAGES,
page_png_path=str(png_path)
)
max_retries = 3
for attempt in range(max_retries):
try:
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=4096,
messages=[{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": img_b64
}
},
{
"type": "text",
"text": prompt
}
]
}]
)
text = response.content[0].text.strip()
# Strip markdown code fences if present
if text.startswith("```"):
lines = text.split("\n")
text = "\n".join(lines[1:-1] if lines[-1] == "```" else lines[1:])
data = json.loads(text)
data["page_number"] = page_num # ensure correct
return data
except json.JSONDecodeError as e:
print(f" Page {page_num} attempt {attempt+1}: JSON parse error: {e}")
if attempt == max_retries - 1:
# Return a fallback
return {
"page_number": page_num,
"classification": None,
"page_type": "text",
"chunks": [{
"order_in_page": 1,
"type": "body_paragraph",
"content_en": f"[Page {page_num} — parse error, content not extracted]",
"content_pt_br": f"[Página {page_num} — erro de análise, conteúdo não extraído]",
"bbox": {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}]
}
except anthropic.APIError as e:
print(f" Page {page_num} attempt {attempt+1}: API error: {e}")
if attempt < max_retries - 1:
time.sleep(2 ** attempt)
else:
return {
"page_number": page_num,
"classification": None,
"page_type": "text",
"chunks": [{
"order_in_page": 1,
"type": "body_paragraph",
"content_en": f"[Page {page_num} — API error]",
"content_pt_br": f"[Página {page_num} — erro de API]",
"bbox": {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}]
}
def crop_image(chunk_id: str, png_path: Path, bbox: dict):
"""Crop a region from the page PNG and save to images dir."""
try:
from PIL import Image
im = Image.open(png_path)
W, H = im.size
x = bbox.get("x", 0)
y = bbox.get("y", 0)
w = bbox.get("w", 1)
h = bbox.get("h", 1)
pad = 0.005
left = max(0, int((x - pad) * W))
top = max(0, int((y - pad) * H))
right = min(W, int((x + w + pad) * W))
bottom = min(H, int((y + h + pad) * H))
cropped = im.crop((left, top, right, bottom))
out_path = IMAGES_DIR / f"IMG-{chunk_id}.png"
cropped.save(out_path)
return out_path
except Exception as e:
print(f" Crop error for {chunk_id}: {e}")
return None
def write_chunk_file(chunk_data: dict, chunk_id: str, page_num: int,
order_global: int, prev_chunk, next_chunk,
has_image: bool) -> None:
"""Write a single chunk markdown file."""
bbox = chunk_data.get("bbox", {"x": 0, "y": 0, "w": 1, "h": 1})
png_index = page_num - 1
source_png = f"../../processing/png/{DOC_ID}/p-{png_index:03d}.png"
related_image = f"IMG-{chunk_id}.png" if has_image else "null"
related_table = chunk_data.get("related_table", "null") or "null"
ufo = chunk_data.get("ufo_anomaly_detected", False)
cryptid = chunk_data.get("cryptid_anomaly_detected", False)
frontmatter = f"""---
chunk_id: {chunk_id}
type: {chunk_data.get("type", "body_paragraph")}
page: {page_num}
order_in_page: {chunk_data.get("order_in_page", 1)}
order_global: {order_global}
bbox: {{x: {bbox.get("x", 0):.3f}, y: {bbox.get("y", 0):.3f}, w: {bbox.get("w", 1):.3f}, h: {bbox.get("h", 1):.3f}}}
classification: {json.dumps(chunk_data.get("classification"))}
formatting: {json.dumps(chunk_data.get("formatting", []))}
cross_page_hint: {chunk_data.get("cross_page_hint", "self_contained")}
prev_chunk: {json.dumps(prev_chunk)}
next_chunk: {json.dumps(next_chunk)}
related_image: {json.dumps(related_image if has_image else None)}
related_table: {json.dumps(chunk_data.get("related_table"))}
ocr_confidence: {chunk_data.get("ocr_confidence", 0.9)}
ocr_source_lines: {json.dumps(chunk_data.get("ocr_source_lines", []))}
redaction_code: {json.dumps(chunk_data.get("redaction_code"))}
redaction_inferred_content_type: {json.dumps(chunk_data.get("redaction_inferred_content_type"))}
image_type: {json.dumps(chunk_data.get("image_type"))}
ufo_anomaly_detected: {str(ufo).lower()}
ufo_anomaly_type: {json.dumps(chunk_data.get("ufo_anomaly_type"))}
ufo_anomaly_rationale: {json.dumps(chunk_data.get("ufo_anomaly_rationale"))}
cryptid_anomaly_detected: {str(cryptid).lower()}
cryptid_anomaly_type: {json.dumps(chunk_data.get("cryptid_anomaly_type"))}
cryptid_anomaly_rationale: {json.dumps(chunk_data.get("cryptid_anomaly_rationale"))}
image_description_en: {json.dumps(chunk_data.get("image_description_en"))}
image_description_pt_br: {json.dumps(chunk_data.get("image_description_pt_br"))}
extracted_text: {json.dumps(chunk_data.get("extracted_text"))}
source_png: {source_png}
---
**EN:** {chunk_data.get("content_en", "")}
**PT-BR:** {chunk_data.get("content_pt_br", "")}
"""
out_path = CHUNKS_DIR / f"{chunk_id}.md"
out_path.write_text(frontmatter, encoding="utf-8")
def main():
start_time = time.time()
print(f"Starting rebuild of {DOC_ID}")
print(f"Processing {TOTAL_PAGES} pages with 4 parallel workers...")
CHUNKS_DIR.mkdir(parents=True, exist_ok=True)
IMAGES_DIR.mkdir(parents=True, exist_ok=True)
TABLES_DIR.mkdir(parents=True, exist_ok=True)
# Step 1: Process all pages in parallel batches of 4
all_pages = {} # page_num -> page_data
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
future_to_page = {
executor.submit(process_page, page_num): page_num
for page_num in range(1, TOTAL_PAGES + 1)
}
completed = 0
for future in concurrent.futures.as_completed(future_to_page):
page_num = future_to_page[future]
try:
result = future.result()
all_pages[page_num] = result
completed += 1
if completed % 10 == 0:
print(f" Completed {completed}/{TOTAL_PAGES} pages...")
except Exception as e:
print(f" Page {page_num} failed: {e}")
all_pages[page_num] = {
"page_number": page_num,
"classification": None,
"page_type": "text",
"chunks": [{
"order_in_page": 1,
"type": "body_paragraph",
"content_en": f"[Page {page_num} — processing failed: {e}]",
"content_pt_br": f"[Página {page_num} — processamento falhou: {e}]",
"bbox": {"x": 0.05, "y": 0.05, "w": 0.90, "h": 0.90},
"classification": None,
"formatting": [],
"cross_page_hint": "self_contained",
"ocr_confidence": 0.0,
"ocr_source_lines": [],
"redaction_code": None,
"redaction_inferred_content_type": None,
"image_type": None,
"ufo_anomaly_detected": False,
"ufo_anomaly_type": None,
"ufo_anomaly_rationale": None,
"cryptid_anomaly_detected": False,
"cryptid_anomaly_type": None,
"cryptid_anomaly_rationale": None
}]
}
print(f"All pages processed. Assigning global chunk IDs...")
# Step 2: Assign global chunk IDs
all_chunks = [] # list of (chunk_id, page_num, chunk_data)
global_order = 0
for page_num in range(1, TOTAL_PAGES + 1):
page_data = all_pages[page_num]
chunks = page_data.get("chunks", [])
# Sort by order_in_page
chunks.sort(key=lambda c: c.get("order_in_page", 0))
for chunk in chunks:
global_order += 1
chunk_id = f"c{global_order:04d}"
all_chunks.append((chunk_id, page_num, chunk))
total_chunks = len(all_chunks)
print(f"Total chunks: {total_chunks}")
# Set prev/next pointers
for i, (chunk_id, page_num, chunk) in enumerate(all_chunks):
prev_chunk = all_chunks[i-1][0] if i > 0 else None
next_chunk = all_chunks[i+1][0] if i < len(all_chunks) - 1 else None
chunk["_chunk_id"] = chunk_id
chunk["_prev"] = prev_chunk
chunk["_next"] = next_chunk
chunk["_order_global"] = i + 1
# Step 3: Crop images for image-type chunks
print("Cropping images for image chunks...")
image_chunks = [(cid, pnum, c) for cid, pnum, c in all_chunks if c.get("type") == "image"]
print(f" Found {len(image_chunks)} image chunks")
for chunk_id, page_num, chunk in image_chunks:
png_index = page_num - 1
png_path = PNG_DIR / f"p-{png_index:03d}.png"
bbox = chunk.get("bbox", {"x": 0, "y": 0, "w": 1, "h": 1})
crop_image(chunk_id, png_path, bbox)
# Step 4: Write chunk files
print("Writing chunk files...")
for chunk_id, page_num, chunk in all_chunks:
has_image = chunk.get("type") == "image"
write_chunk_file(
chunk, chunk_id, page_num,
chunk["_order_global"],
chunk["_prev"],
chunk["_next"],
has_image
)
# Step 5: Write _index.json
print("Writing _index.json...")
build_at = datetime.now(timezone.utc).isoformat()
index_chunks = []
for chunk_id, page_num, chunk in all_chunks:
content_en = chunk.get("content_en", "")
preview = content_en[:80] if content_en else ""
bbox = chunk.get("bbox", {"x": 0, "y": 0, "w": 1, "h": 1})
index_chunks.append({
"chunk_id": chunk_id,
"type": chunk.get("type", "body_paragraph"),
"page": page_num,
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk["_order_global"],
"file": f"chunks/{chunk_id}.md",
"bbox": bbox,
"preview": preview
})
index = {
"doc_id": DOC_ID,
"schema_version": "0.2.0",
"total_pages": TOTAL_PAGES,
"total_chunks": total_chunks,
"build_approach": "subagents",
"build_model": "claude-haiku-4-5",
"build_at": build_at,
"chunks": index_chunks
}
index_path = OUT_DIR / "_index.json"
index_path.write_text(json.dumps(index, ensure_ascii=False, indent=2), encoding="utf-8")
# Step 6: Compute stats
chunk_types = {}
ufo_anomalies = []
cryptid_anomalies = []
images_count = 0
for chunk_id, page_num, chunk in all_chunks:
t = chunk.get("type", "body_paragraph")
chunk_types[t] = chunk_types.get(t, 0) + 1
if chunk.get("ufo_anomaly_detected"):
ufo_anomalies.append(chunk_id)
if chunk.get("cryptid_anomaly_detected"):
cryptid_anomalies.append(chunk_id)
if t == "image":
images_count += 1
# Step 7: Write document.md
print("Writing document.md...")
frontmatter_lines = [
"---",
'schema_version: "0.2.0"',
"type: master_document",
f"doc_id: {DOC_ID}",
f'canonical_title: "{DOC_TITLE}"',
f"total_pages: {TOTAL_PAGES}",
f"total_chunks: {total_chunks}",
"chunk_types_histogram:",
]
for t, count in sorted(chunk_types.items()):
frontmatter_lines.append(f" {t}: {count}")
frontmatter_lines.append("multi_page_tables: []")
frontmatter_lines.append(f"ufo_anomalies_flagged: {json.dumps(ufo_anomalies)}")
frontmatter_lines.append(f"cryptid_anomalies_flagged: {json.dumps(cryptid_anomalies)}")
frontmatter_lines.append('build_approach: "subagents"')
frontmatter_lines.append("build_model: claude-haiku-4-5")
frontmatter_lines.append(f"build_at: {build_at}")
frontmatter_lines.append("---")
frontmatter_lines.append("")
doc_lines = frontmatter_lines[:]
current_page = 0
for chunk_id, page_num, chunk in all_chunks:
if page_num != current_page:
current_page = page_num
doc_lines.append(f"## Page {page_num}")
doc_lines.append("")
chunk_type = chunk.get("type", "body_paragraph")
bbox = chunk.get("bbox", {"x": 0, "y": 0, "w": 1, "h": 1})
bbox_str = f"{bbox.get('x',0):.2f}/{bbox.get('y',0):.2f}/{bbox.get('w',1):.2f}/{bbox.get('h',1):.2f}"
doc_lines.append(f"<!-- chunk:{chunk_id} src:./chunks/{chunk_id}.md -->")
doc_lines.append(f'<a id="{chunk_id}"></a>')
doc_lines.append(f"### Chunk {chunk_id} — {chunk_type} · p{page_num} · bbox: {bbox_str}")
doc_lines.append("")
content_en = chunk.get("content_en", "")
content_pt = chunk.get("content_pt_br", "")
doc_lines.append(f"**EN:** {content_en}")
doc_lines.append("")
doc_lines.append(f"**PT-BR:** {content_pt}")
doc_lines.append("")
if chunk_type == "image":
doc_lines.append(f"![chunk image](./images/IMG-{chunk_id}.png)")
desc_en = chunk.get("image_description_en", "")
desc_pt = chunk.get("image_description_pt_br", "")
if desc_en:
doc_lines.append(f"*{desc_en}*")
if desc_pt:
doc_lines.append(f"*{desc_pt}*")
doc_lines.append("")
# Build metadata JSON for details block
meta = {
"chunk_id": chunk_id,
"type": chunk_type,
"page": page_num,
"order_in_page": chunk.get("order_in_page", 1),
"order_global": chunk["_order_global"],
"bbox": bbox,
"classification": chunk.get("classification"),
"formatting": chunk.get("formatting", []),
"cross_page_hint": chunk.get("cross_page_hint", "self_contained"),
"prev_chunk": chunk["_prev"],
"next_chunk": chunk["_next"],
"ocr_confidence": chunk.get("ocr_confidence", 0.9),
"ufo_anomaly_detected": chunk.get("ufo_anomaly_detected", False),
"cryptid_anomaly_detected": chunk.get("cryptid_anomaly_detected", False),
"ufo_anomaly_type": chunk.get("ufo_anomaly_type"),
"cryptid_anomaly_type": chunk.get("cryptid_anomaly_type"),
"redaction_code": chunk.get("redaction_code"),
"image_type": chunk.get("image_type"),
}
doc_lines.append("<details><summary>metadata</summary>")
doc_lines.append("")
doc_lines.append("```json")
doc_lines.append(json.dumps(meta, ensure_ascii=False, indent=2))
doc_lines.append("```")
doc_lines.append("")
doc_lines.append("</details>")
doc_lines.append("")
doc_lines.append("---")
doc_lines.append("")
doc_content = "\n".join(doc_lines)
doc_path = OUT_DIR / "document.md"
doc_path.write_text(doc_content, encoding="utf-8")
wall_seconds = int(time.time() - start_time)
doc_md_bytes = len(doc_content.encode("utf-8"))
print(f"\nDone!")
print(f"STATS pages={TOTAL_PAGES} chunks={total_chunks} images={images_count} tables=0 ufo={len(ufo_anomalies)} cryptid={len(cryptid_anomalies)} doc_md_bytes={doc_md_bytes}")
print(f"Wall time: {wall_seconds}s")
return {
"pages": TOTAL_PAGES,
"chunks": total_chunks,
"images": images_count,
"tables": 0,
"ufo": len(ufo_anomalies),
"cryptid": len(cryptid_anomalies),
"wall_seconds": wall_seconds,
"doc_md_bytes": doc_md_bytes
}
if __name__ == "__main__":
main()

205
scripts/write_chunks_doc65.py Normal file
View file

@ -0,0 +1,205 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Write all chunk files for doc-65 based on vision analysis."""
import json
from pathlib import Path
from datetime import datetime, timezone
DOC_ID = "doc-65-hs1-834228961-62-hq-83894-serial-130"
RAW_DIR = Path(f"/Users/guto/ufo/raw/{DOC_ID}")
CHUNKS_DIR = RAW_DIR / "chunks"
IMAGES_DIR = RAW_DIR / "images"
TABLES_DIR = RAW_DIR / "tables"
for d in [CHUNKS_DIR, IMAGES_DIR, TABLES_DIR]:
d.mkdir(parents=True, exist_ok=True)
# Each entry: (page_index, png_filename, list_of_chunks)
# Each chunk: dict with all required fields
# Pages in order: p-000 to p-063, p-100 to p-126 (91 total)
PAGE_DATA = [
# Page 1: p-000.png - Continuation of Arnold sighting narrative (CONFIDENTIAL banner)
(1, "p-000.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.15, "y": 0.03, "w": 0.7, "h": 0.05}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95, "ufo_anomaly_detected": False},
{"order_in_page": 2, "type": "body_paragraph", "content_en": "airplanes flying so close to the mountain tops, flying directly south to southeast down the back of a mountain range. I could estimate their elevation could have varied a thousand feet one way or another up or down, but they were pretty much on the horizon to me which would indicate they were near the same elevation as I was.", "content_pt_br": "aviões voando tão perto dos topos das montanhas, voando diretamente ao sul para sudeste pelo dorso de uma cordilheira de montanhas. Eu poderia estimar que sua elevação poderia ter variado mil pés para cima ou para baixo, mas estavam praticamente no horizonte para mim, o que indicaria que estavam na mesma elevação que eu.", "bbox": {"x": 0.07, "y": 0.08, "w": 0.88, "h": 0.12}, "classification": None, "formatting": [], "cross_page_hint": "continues_from_prev", "ocr_confidence": 0.88, "ufo_anomaly_detected": True, "ufo_anomaly_type": "formation_flight", "ufo_anomaly_rationale": "Witness describes unidentified objects flying in formation near mountain tops."},
{"order_in_page": 3, "type": "body_paragraph", "content_en": "They flew like many times I have observed geese to fly in a rather diagonal chain-like line as if they were linked together. They seemed to hold a definite direction but rather swerved in and out of the high mountain peaks. Their speed at the time did not impress me particularly, because I knew that our army and air forces had planes that went very fast.", "content_pt_br": "Voavam como tantas vezes eu observei gansos voando em uma linha diagonal em cadeia, como se estivessem ligados. Pareciam manter uma direção definida, mas desviavam para dentro e para fora dos altos picos de montanhas. Sua velocidade na época não me impressionou particularmente, porque eu sabia que nosso exército e forças aéreas tinham aviões que iam muito rápido.", "bbox": {"x": 0.07, "y": 0.20, "w": 0.88, "h": 0.12}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.88, "ufo_anomaly_detected": True, "ufo_anomaly_type": "formation_flight", "ufo_anomaly_rationale": "Objects flying in formation like geese, swerving around mountain peaks."},
{"order_in_page": 4, "type": "body_paragraph", "content_en": "What kept bothering me as I watched them flip and flash in the sun right along their path was the fact that I couldn't make out any tail on them, and I am sure that any pilot would justify more than a second look at such a plane.", "content_pt_br": "O que continuava me perturbando enquanto os observava reluzirem e piscarem ao sol exatamente ao longo de seu caminho era o fato de que eu não conseguia identificar nenhuma cauda neles, e tenho certeza de que qualquer piloto justificaria mais do que um segundo olhar para tal avião.", "bbox": {"x": 0.07, "y": 0.32, "w": 0.88, "h": 0.10}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.87, "ufo_anomaly_detected": True, "ufo_anomaly_type": "no_tail_visible", "ufo_anomaly_rationale": "Objects had no visible tail, unlike any known aircraft."},
{"order_in_page": 5, "type": "body_paragraph", "content_en": "I observed them quite plainly, and I estimate my distance from them, which was almost at right angles, to be between twenty to twenty-five miles. I know they must be very large to observe their shape at that distance, even on as clear a day as it was that Tuesday. In fact I compared a news fastener or sewing tool I had in my pocket with them - holding it up on them and holding it up on the DC-4 - that I could observe at quite a distance to my left, and they seemed smaller than the DC-4; but, I should judge their span would have been as wide as the fuselage engines on each side of the fuselage of the DC-4.", "content_pt_br": "Os observei bastante claramente, e estimo minha distância deles, que era quase em ângulo reto, entre vinte e vinte e cinco milhas. Sei que devem ser muito grandes para observar sua forma nessa distância, mesmo em um dia tão claro quanto aquela terça-feira. De fato, comparei um grampo de papel ou ferramenta de costura que tinha no bolso com eles - segurando-o na frente deles e em seguida na frente do DC-4 - que eu podia observar a uma boa distância à minha esquerda, e pareciam menores que o DC-4; mas devo julgar que sua envergadura teria sido tão larga quanto as fuselagens de motores em cada lado da fuselagem do DC-4.", "bbox": {"x": 0.07, "y": 0.42, "w": 0.88, "h": 0.15}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.87, "ufo_anomaly_detected": True, "ufo_anomaly_type": "unknown_craft_size_estimate", "ufo_anomaly_rationale": "Kenneth Arnold estimating size/distance of unidentified objects compared to DC-4."},
{"order_in_page": 6, "type": "body_paragraph", "content_en": "The more I observed these objects, the more upset I became, as I am accustomed and familiar with most all objects flying whether I am close to the ground or at higher altitudes. I observed the chain of these objects passing another high snow-covered ridge in between Mt. Rainier and Mt. Adams,", "content_pt_br": "Quanto mais observava esses objetos, mais perturbado ficava, pois estou acostumado e familiarizado com praticamente todos os objetos que voam, quer eu esteja perto do solo ou em altitudes maiores. Observei a cadeia desses objetos passando por outro cume coberto de neve entre o Monte Rainier e o Monte Adams,", "bbox": {"x": 0.07, "y": 0.57, "w": 0.88, "h": 0.10}, "classification": None, "formatting": [], "cross_page_hint": "continues_to_next", "ocr_confidence": 0.87, "ufo_anomaly_detected": True, "ufo_anomaly_type": "chain_formation", "ufo_anomaly_rationale": "Chain of unidentified objects passing between Mt. Rainier and Mt. Adams."},
{"order_in_page": 7, "type": "page_number_marker", "content_en": "2d/16", "content_pt_br": "2d/16", "bbox": {"x": 0.8, "y": 0.92, "w": 0.12, "h": 0.03}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.7},
{"order_in_page": 8, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.15, "y": 0.96, "w": 0.7, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
]),
# Page 2: p-001.png - HQ Air Defense Command letter, Alpheus Powell interview
(2, "p-001.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.1, "y": 0.02, "w": 0.8, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 2, "type": "stamp", "content_en": "DECLASSIFIED", "content_pt_br": "DESCLASSIFICADO", "bbox": {"x": 0.03, "y": 0.06, "w": 0.18, "h": 0.06}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 3, "type": "header", "content_en": "62-93994-130", "content_pt_br": "62-93994-130", "bbox": {"x": 0.55, "y": 0.06, "w": 0.4, "h": 0.03}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 4, "type": "letterhead", "content_en": "HEADQUARTERS\nAIR DEFENSE COMMAND\nMITCHEL FIELD, NEW YORK", "content_pt_br": "QUARTEL GENERAL\nCOMANDO DE DEFESA AÉREA\nMITCHEL FIELD, NEW YORK", "bbox": {"x": 0.25, "y": 0.08, "w": 0.5, "h": 0.08}, "classification": None, "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.92},
{"order_in_page": 5, "type": "date_line", "content_en": "12 September 1947", "content_pt_br": "12 de setembro de 1947", "bbox": {"x": 0.6, "y": 0.17, "w": 0.35, "h": 0.025}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 6, "type": "subject_line", "content_en": "SUBJECT: Unidentified Flying Object\n(Interview - Alpheus O. Powell)", "content_pt_br": "ASSUNTO: Objeto Voador Não Identificado\n(Entrevista - Alpheus O. Powell)", "bbox": {"x": 0.07, "y": 0.20, "w": 0.85, "h": 0.04}, "classification": None, "formatting": ["bold"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 7, "type": "header", "content_en": "SUMMARY OF INFORMATION:", "content_pt_br": "RESUMO DE INFORMAÇÕES:", "bbox": {"x": 0.07, "y": 0.26, "w": 0.5, "h": 0.02}, "classification": None, "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 8, "type": "body_paragraph", "content_en": "The following information was received 12 August 1947 from Mr. Alpheus O. Powell, 28 Redwood Road, New Hyde Park, Long Island, relative to the sighting of a possible flying disc 4 August 1947.\n\nOn 4 August 1947, Mr. Powell, an Airlines Captain with Pan American Airways, Inc., was the first pilot of a Constellation type aircraft on a flight from Gander, Newfoundland, to La Guardia Field, New York. Mr. Powell took over the aircraft at Gander, Newfoundland and departed at approximately 1220 P.M., Eastern Daylight Saving Time for La Guardia Field, New York. At 1600 P.M., at a position approximately midway between the Everett (Mass.) Fan Marker and the Bedford Radio Beacon (Everett is 3 miles NW of Boston, Mass., and Bedford is 16 miles NW of the same city) both Mr Powell and Mr E. White, navigator on this trip, sighted unidentifiable flying objects. To the best of Mr. Powell's knowledge, the following weather conditions existed at that time: visibility good; clear; no clouds. Mr. E. White, the co-pilot and navigator on this trip, sighted unidentifiable flying objects. To the best of Mr. Powell's knowledge, the following weather conditions existed at that time: visibility good; clear; no clouds. Mr. White, who was sitting in the co-pilots seat (the right side of the cockpit) first called Mr. Powell's attention to a bright orange object.", "content_pt_br": "As seguintes informações foram recebidas em 12 de agosto de 1947 do Sr. Alpheus O. Powell, 28 Redwood Road, New Hyde Park, Long Island, relativas ao avistamento de um possível disco voador em 4 de agosto de 1947.\n\nEm 4 de agosto de 1947, o Sr. Powell, Capitão de Companhia Aérea da Pan American Airways, Inc., era o primeiro piloto de uma aeronave tipo Constellation em um voo de Gander, Newfoundland, para La Guardia Field, Nova York. O Sr. Powell assumiu a aeronave em Gander, Newfoundland e partiu por volta das 12h20 (horário de verão oriental) para La Guardia Field, Nova York. Às 16h00, em uma posição aproximadamente entre o Fan Marker Everett (Mass.) e o Radio Beacon de Bedford (Everett fica a 3 milhas NW de Boston, Mass., e Bedford fica a 16 milhas NW da mesma cidade), tanto o Sr. Powell quanto o Sr. E. White, navegador nesta viagem, avistaram objetos voadores não identificáveis.", "bbox": {"x": 0.07, "y": 0.29, "w": 0.87, "h": 0.55}, "classification": None, "formatting": [], "cross_page_hint": "continues_to_next", "ocr_confidence": 0.88, "ufo_anomaly_detected": True, "ufo_anomaly_type": "aerial_unknown_object", "ufo_anomaly_rationale": "Airline Captain Powell and navigator White report sighting unidentifiable flying objects from a Constellation aircraft over Massachusetts."},
{"order_in_page": 9, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.1, "y": 0.95, "w": 0.8, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
]),
# Page 3: p-002.png - HQ Air Defense Command, Walter I. White interview
(3, "p-002.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.15, "y": 0.02, "w": 0.7, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 2, "type": "letterhead", "content_en": "HEADQUARTERS\nAIR DEFENSE COMMAND\nMITCHEL FIELD, NEW YORK", "content_pt_br": "QUARTEL GENERAL\nCOMANDO DE DEFESA AÉREA\nMITCHEL FIELD, NEW YORK", "bbox": {"x": 0.25, "y": 0.06, "w": 0.5, "h": 0.08}, "classification": None, "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.92},
{"order_in_page": 3, "type": "date_line", "content_en": "18 September 1947", "content_pt_br": "18 de setembro de 1947", "bbox": {"x": 0.6, "y": 0.16, "w": 0.35, "h": 0.025}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.93},
{"order_in_page": 4, "type": "subject_line", "content_en": "SUBJECT: Unidentified Flying Objects\n(Interview - Walter I. White)", "content_pt_br": "ASSUNTO: Objetos Voadores Não Identificados\n(Entrevista - Walter I. White)", "bbox": {"x": 0.07, "y": 0.20, "w": 0.85, "h": 0.04}, "classification": None, "formatting": ["bold"], "cross_page_hint": "self_contained", "ocr_confidence": 0.93},
{"order_in_page": 5, "type": "header", "content_en": "SUMMARY OF INFORMATION:", "content_pt_br": "RESUMO DE INFORMAÇÕES:", "bbox": {"x": 0.07, "y": 0.25, "w": 0.5, "h": 0.02}, "classification": None, "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.93},
{"order_in_page": 6, "type": "body_paragraph", "content_en": "The following information relative to the sighting of a possible flying disc 4 August 1947, was received 10 September 1947 from Mr. Walter I. White, 19-07 78th Street, Jackson Heights, New York.\n\nOn 4 August 1947, Mr. White, from Pan American Airways Inc., was the navigator of a Constellation type aircraft on a flight from Gander, Newfoundland to La Guardia Field, New York. At 1600, at a position approximately 10 miles NW of Boston, Mass., Mr. White sighted a flying object which he was unable to identify. At this time Mr. White was sitting in the co-pilots seat, and looking out the right side of the aircraft he sighted what appeared to be in the vicinity. It appeared to be about 5 miles away, and at least 1,000 feet below the level of the Constellation. Mr. White believes that he observed the object for almost 30 seconds before he called Mr. Powell's attention to the object. When he first called Mr. Powell's attention to it, it was too \"lit up\" and traveling at the upper right, and a cloud passed between the object and the aircraft. The object appeared to have a sharp, definite shape, and appeared cylindrical in shape, on having a bright orange hue. Mr. Powell stated that the object had a definite shape, and there was no appearance of exhaust from a rocket, or a jet aircraft. Mr. Powell estimated the course of the object. It was flying, at a terrific and quite rapid speed. Mr. Powell lost sight of the object, when a cloud came between the aircraft and the object. The pursuit of the object was not continued, inasmuch as it would have necessitated a departure from the established airways.", "content_pt_br": "As seguintes informações relativas ao avistamento de um possível disco voador em 4 de agosto de 1947 foram recebidas em 10 de setembro de 1947 do Sr. Walter I. White, 19-07 78th Street, Jackson Heights, Nova York.\n\nEm 4 de agosto de 1947, o Sr. White, da Pan American Airways Inc., era o navegador de uma aeronave tipo Constellation em um voo de Gander, Newfoundland, para La Guardia Field, Nova York. Às 16h00, em uma posição aproximadamente 10 milhas a noroeste de Boston, Mass., o Sr. White avistou um objeto voador que não conseguiu identificar. O objeto estava a cerca de 5 milhas de distância, pelo menos 1.000 pés abaixo do nível da Constellation. O Sr. White acredita ter observado o objeto por quase 30 segundos antes de chamar a atenção do Sr. Powell. O objeto parecia ter uma forma nítida e definida, aparentemente cilíndrica, com uma tonalidade laranja brilhante.", "bbox": {"x": 0.07, "y": 0.28, "w": 0.87, "h": 0.57}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.87, "ufo_anomaly_detected": True, "ufo_anomaly_type": "aerial_unknown_object", "ufo_anomaly_rationale": "Navigator White reports bright orange cylindrical unidentified object observed for 30 seconds from Constellation aircraft."},
{"order_in_page": 7, "type": "header", "content_en": "AGENTS NOTES: Mr. Walter I. White has been employed by Pan American Airways for the past five years as a Navigator, and during the war worked with TIA", "content_pt_br": "NOTAS DO AGENTE: O Sr. Walter I. White está empregado pela Pan American Airways nos últimos cinco anos como Navegador, e durante a guerra trabalhou com a TIA", "bbox": {"x": 0.07, "y": 0.86, "w": 0.87, "h": 0.05}, "classification": None, "formatting": [], "cross_page_hint": "continues_to_next", "ocr_confidence": 0.85},
{"order_in_page": 8, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.15, "y": 0.95, "w": 0.7, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
]),
# Page 4: p-003.png - Continuation of White interview, distribution/evaluation
(4, "p-003.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.15, "y": 0.02, "w": 0.7, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 2, "type": "body_paragraph", "content_en": "in conjunction with contract flying for the AAF. Mr White states that he has flown with Mr Powell on a number of occasions, and he considers him to be a very stable person; completely reliable, and not given to \"flights of fancy\".", "content_pt_br": "em conjunto com voos contratados para as Forças Aéreas do Exército. O Sr. White afirma que voou com o Sr. Powell em várias ocasiões, e o considera uma pessoa muito estável; completamente confiável e não dado a \"voos de fantasia\".", "bbox": {"x": 0.07, "y": 0.07, "w": 0.87, "h": 0.08}, "classification": None, "formatting": [], "cross_page_hint": "continues_from_prev", "ocr_confidence": 0.88},
{"order_in_page": 3, "type": "body_paragraph", "content_en": "Related Report: See Summary of Information, 12 September 1947, Hq ADC, subject, \"Unidentified Flying Objects\" (Interview - Alpheus O. Powell).", "content_pt_br": "Relatório Relacionado: Ver Resumo de Informações, 12 de setembro de 1947, QG ADC, assunto, \"Objetos Voadores Não Identificados\" (Entrevista - Alpheus O. Powell).", "bbox": {"x": 0.07, "y": 0.16, "w": 0.87, "h": 0.05}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.88},
{"order_in_page": 4, "type": "body_paragraph", "content_en": "Previous Distribution:\n- None\n\nDistribution\n- AAF (2 copies)\n- ADC (2 copies)\n\nEvaluation\nof source of information\nC B", "content_pt_br": "Distribuição Anterior:\n- Nenhuma\n\nDistribuição\n- AAF (2 cópias)\n- ADC (2 cópias)\n\nAvaliação\nda fonte da informação\nC B", "bbox": {"x": 0.07, "y": 0.25, "w": 0.87, "h": 0.2}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.87},
{"order_in_page": 5, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.15, "y": 0.93, "w": 0.7, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
]),
# Page 5: p-004.png - Agent's notes on Powell background
(5, "p-004.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.15, "y": 0.02, "w": 0.7, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 2, "type": "body_paragraph", "content_en": "AGENTS NOTES: Mr. A. O. Powell is a graduate of the Aviation Cadet Flying Training Program, having graduated from Maxwell Field, Alabama, with the class of 41-C. Since graduation, Mr. Powell has flown for Pan American Airways and, at this date, has over 4,000 command pilot hours to his credit. Mr. Powell appears to be a calm, intelligent individual, not given to flights of fancy, or easily swayed by what he has previously read in the newspapers as regards reports of this type. Mr. Powell has a fear of publicity and seemed hesitant to even tell his story lest he become the object of ridicule. Mr. Powell was questioned as to the possibility that what he sighted might have been a tow target, a pilot balloon, or a radiosonde device used for meteorological purposes. Mr. Powell stated that he has seen numerous pilot balloons, radiosonde devices and tow targets, while on flights; the object observed on this flight definitely was not one of them.", "content_pt_br": "NOTAS DO AGENTE: O Sr. A. O. Powell é formado pelo Programa de Treinamento de Voo de Cadetes de Aviação, tendo se formado em Maxwell Field, Alabama, com a turma de 41-C. Desde a formatura, o Sr. Powell voou pela Pan American Airways e, nesta data, tem mais de 4.000 horas de voo como piloto comandante em seu crédito. O Sr. Powell parece ser um indivíduo calmo e inteligente, não dado a fantasias, ou facilmente influenciado pelo que leu anteriormente nos jornais sobre relatórios deste tipo. O Sr. Powell tem medo de publicidade e pareceu hesitante em contar sua história por medo de se tornar objeto de ridículo. O Sr. Powell foi questionado sobre a possibilidade de que o que avistou pudesse ter sido um alvo rebocado, um balão piloto ou um radiobalão usado para fins meteorológicos. O Sr. Powell afirmou que já viu numerosos balões piloto, radiobalões e alvos rebocados durante voos; o objeto observado neste voo definitivamente não era nenhum deles.", "bbox": {"x": 0.07, "y": 0.07, "w": 0.87, "h": 0.5}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.88},
{"order_in_page": 3, "type": "body_paragraph", "content_en": "Previous Distribution:\n- None\n\nDistribution\n- AAF (2 copies)\n- ADC (2 copies)", "content_pt_br": "Distribuição Anterior:\n- Nenhuma\n\nDistribuição\n- AAF (2 cópias)\n- ADC (2 cópias)", "bbox": {"x": 0.07, "y": 0.62, "w": 0.5, "h": 0.1}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.88},
{"order_in_page": 4, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.15, "y": 0.93, "w": 0.7, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
]),
# Page 6: p-005.png - RESTRICTED memo, 14th Air Force forwarding flying disc report
(6, "p-005.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "RESTRICTED", "content_pt_br": "RESTRITO", "bbox": {"x": 0.25, "y": 0.02, "w": 0.5, "h": 0.04}, "classification": "RESTRICTED", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 2, "type": "body_paragraph", "content_en": "BASIC: Ltr, Hq, BARTD, Birmingham AAF, Birmingham, Ala., dtd 8 July 47, subj: Report on Local \"Flying Disc.\"\n\n319.1/624\n\n1st Ind.\nHEADQUARTERS, FOURTEENTH AIR FORCE, Orlando, Florida, [date] 1947\nTO: Commanding General, Air Defense Command, Mitchel Field, New York.\n\n1. Forwarded for information of your Headquarters.\n\n2. This Headquarters has made no investigation of \"Flying Disc\" reports because this is an isolated case.\n\nFOR THE COMMANDING GENERAL:", "content_pt_br": "BÁSICO: Carta, QG, BARTD, Birmingham AAF, Birmingham, Ala., datada de 8 de julho de 47, assunto: Relatório sobre \"Disco Voador\" Local.\n\n319.1/624\n\n1ª Indorsação.\nQUARTEL GENERAL, DÉCIMA QUARTA FORÇA AÉREA, Orlando, Flórida, [data] 1947\nPARA: Comandante Geral, Comando de Defesa Aérea, Mitchel Field, Nova York.\n\n1. Encaminhado para informação do seu Quartel General.\n\n2. Este Quartel General não fez nenhuma investigação sobre relatórios de \"Disco Voador\" porque este é um caso isolado.\n\nPELO COMANDANTE GERAL:", "bbox": {"x": 0.07, "y": 0.07, "w": 0.87, "h": 0.55}, "classification": "RESTRICTED", "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.88},
{"order_in_page": 3, "type": "signature_block", "content_en": "P.V.Murphy\n[signature]\nP.V. Murphy\nBrig. Gen., AAF\nAsst. Adj. Gen.", "content_pt_br": "P.V.Murphy\n[assinatura]\nP.V. Murphy\nGen. de Brig., AAF\nAssistente do Adj. Geral", "bbox": {"x": 0.5, "y": 0.63, "w": 0.4, "h": 0.1}, "classification": None, "formatting": ["handwritten"], "cross_page_hint": "self_contained", "ocr_confidence": 0.8},
{"order_in_page": 4, "type": "body_paragraph", "content_en": "3 Incls: n/c", "content_pt_br": "3 Anexos: s/c", "bbox": {"x": 0.07, "y": 0.65, "w": 0.2, "h": 0.03}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 5, "type": "stamp", "content_en": "32715", "content_pt_br": "32715", "bbox": {"x": 0.07, "y": 0.9, "w": 0.15, "h": 0.03}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.85},
{"order_in_page": 6, "type": "page_number_marker", "content_en": "24-6\n1947", "content_pt_br": "24-6\n1947", "bbox": {"x": 0.82, "y": 0.9, "w": 0.12, "h": 0.04}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.8},
{"order_in_page": 7, "type": "classification_banner", "content_en": "RESTRICTED", "content_pt_br": "RESTRITO", "bbox": {"x": 0.25, "y": 0.95, "w": 0.5, "h": 0.04}, "classification": "RESTRICTED", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
]),
# Page 7: p-006.png - Blank page with file number notation
(7, "p-006.png", [
{"order_in_page": 1, "type": "header", "content_en": "62-83894-130", "content_pt_br": "62-83894-130", "bbox": {"x": 0.3, "y": 0.02, "w": 0.4, "h": 0.03}, "classification": None, "formatting": ["handwritten"], "cross_page_hint": "self_contained", "ocr_confidence": 0.85},
{"order_in_page": 2, "type": "blank", "content_en": "[Blank page]", "content_pt_br": "[Página em branco]", "bbox": {"x": 0.0, "y": 0.05, "w": 1.0, "h": 0.95}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 1.0},
]),
# Page 8: p-007.png - RESTRICTED Birmingham AAF report on flying disc with photo
(8, "p-007.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "RESTRICTED", "content_pt_br": "RESTRITO", "bbox": {"x": 0.25, "y": 0.02, "w": 0.5, "h": 0.04}, "classification": "RESTRICTED", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 2, "type": "letterhead", "content_en": "HEADQUARTERS\nBIRMINGHAM MILITARY TRAINING DETACHMENT\nBIRMINGHAM ARMY AIR FIELD\nBirmingham, Alabama", "content_pt_br": "QUARTEL GENERAL\nDETACHMENTO DE TREINAMENTO MILITAR DE BIRMINGHAM\nCAMPO DE AVIAÇÃO DO EXÉRCITO DE BIRMINGHAM\nBirmingham, Alabama", "bbox": {"x": 0.2, "y": 0.06, "w": 0.6, "h": 0.08}, "classification": None, "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 3, "type": "date_line", "content_en": "8 July 1947", "content_pt_br": "8 de julho de 1947", "bbox": {"x": 0.7, "y": 0.15, "w": 0.25, "h": 0.025}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 4, "type": "header", "content_en": "7795", "content_pt_br": "7795", "bbox": {"x": 0.75, "y": 0.18, "w": 0.15, "h": 0.03}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 5, "type": "subject_line", "content_en": "SUBJECT: Report on Local \"Flying Disc\"", "content_pt_br": "ASSUNTO: Relatório sobre \"Disco Voador\" Local", "bbox": {"x": 0.07, "y": 0.22, "w": 0.85, "h": 0.025}, "classification": None, "formatting": ["bold"], "cross_page_hint": "self_contained", "ocr_confidence": 0.93},
{"order_in_page": 6, "type": "address_block", "content_en": "TO: Commanding General\nFourteenth Air Force, ADC\nOrlando, Florida\nAttn: A-2", "content_pt_br": "PARA: Comandante Geral\nDécima Quarta Força Aérea, ADC\nOrlando, Flórida\nAtenção: A-2", "bbox": {"x": 0.07, "y": 0.25, "w": 0.5, "h": 0.07}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 7, "type": "body_paragraph", "content_en": "1. Inclosed herewith is a photograph of the \"unidentified flying disc,\" which one reportedly witnessed over Birmingham on the night of Sunday, 6 July, 1947.\n\n2. Attention is invited to the two light spots on the print and the light trail following the two discs. The image at the top of the photograph indicates the discs directly. This has been examined by professional photographers in Birmingham and their general opinion is that the two spots observed in the photograph are not a photographic negative, but instead, an actual photograph of some mysterious disc.\n\n3. The undersigned officer did not personally witness the flight of any disc, however, in view of the numerous reports received from the citizens of Birmingham, I am of the general opinion in Birmingham that \"something resembling a disc\" is a statement made by the only military personnel of this organization who personally witness the craft. The following is a statement made by additional information. Attached as inclosure number three are clippings from one of the local newspapers giving an account of the day, several citizens, and two reported owing the many mysterious objects.\n\n4. The intelligence officer of the 1856 Military District has forwarded a similar report through ground force channels to the Commanding General, Third Army, Atlanta, Georgia.", "content_pt_br": "1. Incluso neste documento há uma fotografia do \"disco voador não identificado,\" que supostamente foi testemunhado sobre Birmingham na noite de domingo, 6 de julho de 1947.\n\n2. Chama-se atenção para os dois pontos de luz na fotografia e o rastro de luz seguindo os dois discos. A imagem no topo da fotografia indica os discos diretamente. Isso foi examinado por fotógrafos profissionais em Birmingham e a opinião geral é que os dois pontos observados na fotografia não são um negativo fotográfico, mas sim uma fotografia real de algum disco misterioso.\n\n3. O oficial abaixo assinado não testemunhou pessoalmente o voo de qualquer disco, porém, em vista dos numerosos relatórios recebidos dos cidadãos de Birmingham, sou da opinião geral de que \"algo semelhante a um disco\" é uma declaração feita pelo único pessoal militar desta organização que testemunhou pessoalmente a aeronave.\n\n4. O oficial de inteligência do 1856º Distrito Militar encaminhou um relatório semelhante através dos canais das forças terrestres ao Comandante Geral, Terceiro Exército, Atlanta, Geórgia.", "bbox": {"x": 0.07, "y": 0.32, "w": 0.87, "h": 0.5}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.85, "ufo_anomaly_detected": True, "ufo_anomaly_type": "disc_photograph", "ufo_anomaly_rationale": "Photograph of alleged flying disc over Birmingham, Alabama examined by professional photographers."},
{"order_in_page": 8, "type": "signature_block", "content_en": "Jack C. White\n[signature]\nJack C. White\nMajor, Air Corps\nCommanding", "content_pt_br": "Jack C. White\n[assinatura]\nJack C. White\nMajor, Corpo Aéreo\nComandante", "bbox": {"x": 0.5, "y": 0.83, "w": 0.4, "h": 0.08}, "classification": None, "formatting": ["handwritten"], "cross_page_hint": "self_contained", "ocr_confidence": 0.8},
{"order_in_page": 9, "type": "body_paragraph", "content_en": "2 Incls:\n1. Photograph\n2. Statement\n3. Clippings", "content_pt_br": "2 Anexos:\n1. Fotografia\n2. Declaração\n3. Recortes", "bbox": {"x": 0.07, "y": 0.84, "w": 0.3, "h": 0.07}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 10, "type": "classification_banner", "content_en": "RESTRICTED", "content_pt_br": "RESTRITO", "bbox": {"x": 0.25, "y": 0.95, "w": 0.5, "h": 0.04}, "classification": "RESTRICTED", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
]),
# Page 9: p-008.png - Newspaper clipping about mysterious flying saucers in Birmingham
(9, "p-008.png", [
{"order_in_page": 1, "type": "image", "content_en": "Newspaper clipping: 'Mysterious Flying Saucers Reported In Birmingham Skies' - The Birmingham News article about multiple sightings of flying discs over Birmingham, with accounts from various witnesses including Mrs. James Rain (2135 South 28th Street), J.L. Kardon (2100 Clanton Street), C.H. Zohn, J.R. Kauke, C.C. Rockwood, Nancy Rockwood, and others. The article describes lights, sounds, and movements of the objects.", "content_pt_br": "Recorte de jornal: 'Discos Voadores Misteriosos Relatados nos Céus de Birmingham' - Artigo do The Birmingham News sobre múltiplos avistamentos de discos voadores sobre Birmingham, com relatos de várias testemunhas incluindo Sra. James Rain, J.L. Kardon, C.H. Zohn, J.R. Kauke, C.C. Rockwood, Nancy Rockwood e outros. O artigo descreve luzes, sons e movimentos dos objetos.", "bbox": {"x": 0.05, "y": 0.02, "w": 0.9, "h": 0.96}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.82, "image_type": "newspaper_clipping", "ufo_anomaly_detected": True, "ufo_anomaly_type": "multiple_witness_sighting", "ufo_anomaly_rationale": "Newspaper reports multiple civilian witnesses to flying saucer sightings over Birmingham, Alabama.", "image_description_en": "Newspaper clipping from The Birmingham News reporting multiple sightings of mysterious flying saucers over Birmingham. Contains witness accounts, descriptions of the objects as lights in the sky, and mentions of police reports.", "image_description_pt_br": "Recorte de jornal do The Birmingham News relatando múltiplos avistamentos de discos voadores misteriosos sobre Birmingham. Contém relatos de testemunhas, descrições dos objetos como luzes no céu, e menções de relatórios policiais."},
]),
# Page 10: p-009.png - Newspaper clipping continued, radio-guided theory
(10, "p-009.png", [
{"order_in_page": 1, "type": "image", "content_en": "Newspaper clipping: 'RADIO-GUIDED, SAYS LAD' from Monday, July 7, 1947. Article about a grammar school youth Michael Rieman who gave his opinion that flying saucers 'are new radio-guided missiles from another country that is planning war on the United States.' Includes other witness accounts from Birmingham area residents about sightings on July 6-7, 1947.", "content_pt_br": "Recorte de jornal: 'GUIADO POR RÁDIO, DIZ RAPAZ' de segunda-feira, 7 de julho de 1947. Artigo sobre o jovem escolar Michael Rieman que deu sua opinião de que os discos voadores 'são novos mísseis guiados por rádio de outro país que está planejando guerra contra os Estados Unidos.' Inclui relatos de outras testemunhas de moradores da área de Birmingham sobre avistamentos em 6-7 de julho de 1947.", "bbox": {"x": 0.05, "y": 0.02, "w": 0.9, "h": 0.96}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.8, "image_type": "newspaper_clipping", "ufo_anomaly_detected": True, "ufo_anomaly_type": "multiple_witness_sighting", "ufo_anomaly_rationale": "Newspaper article with multiple civilian witness accounts of flying saucer sightings.", "image_description_en": "Newspaper clipping from July 7, 1947 containing witness accounts of flying saucer sightings in Birmingham area and a youth's theory that they are radio-guided missiles.", "image_description_pt_br": "Recorte de jornal de 7 de julho de 1947 contendo relatos de testemunhas de avistamentos de discos voadores na área de Birmingham e a teoria de um jovem de que são mísseis guiados por rádio."},
]),
# Page 11: p-010.png - Another Birmingham newspaper clipping about flying saucers
(11, "p-010.png", [
{"order_in_page": 1, "type": "image", "content_en": "Newspaper clipping with multiple witness accounts of flying saucer sightings in Birmingham area. Witnesses include Charles F. Bradley (weather man), Mrs. James Rain, J.L. Kardon, C.M. Cadenhead, and others. Accounts describe round, shiny objects traveling in formation. 'Scores of People Report Seeing Mysterious Discs' headline. Mentions searchlight reflection theory being advanced.", "content_pt_br": "Recorte de jornal com múltiplos relatos de testemunhas de avistamentos de discos voadores na área de Birmingham. Testemunhas incluem Charles F. Bradley (meteorologista), Sra. James Rain, J.L. Kardon, C.M. Cadenhead e outros. Os relatos descrevem objetos redondos e brilhantes viajando em formação. Manchete: 'Dezenas de Pessoas Relatam Ter Visto Discos Misteriosos'. Menciona teoria de reflexo de holofote sendo avançada.", "bbox": {"x": 0.05, "y": 0.02, "w": 0.9, "h": 0.96}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.78, "image_type": "newspaper_clipping", "ufo_anomaly_detected": True, "ufo_anomaly_type": "multiple_witness_sighting", "ufo_anomaly_rationale": "Dozens of Birmingham residents report seeing mysterious discs in formation.", "image_description_en": "Newspaper clipping with dozens of witness accounts of flying saucer sightings in Birmingham, Alabama, 1947.", "image_description_pt_br": "Recorte de jornal com dezenas de relatos de testemunhas de avistamentos de discos voadores em Birmingham, Alabama, 1947."},
]),
# Page 12: p-011.png - More Birmingham witnesses, searchlight theory
(12, "p-011.png", [
{"order_in_page": 1, "type": "image", "content_en": "Newspaper clipping continuing Birmingham flying saucer accounts. Mentions Searchlight Reflection Theory being advanced. Multiple witnesses describe lights appearing and disappearing rapidly. References to E.H. Vaughn III (anti-aircraft gunner in Europe), S.S. Lovejoy (Tennessee Coal Iron and Railroad), and Jimmy Dewberry. 'Searchlight Reflection Theory Is Advanced' subheadline.", "content_pt_br": "Recorte de jornal continuando relatos de discos voadores em Birmingham. Menciona teoria de reflexo de holofote sendo avançada. Múltiplas testemunhas descrevem luzes aparecendo e desaparecendo rapidamente. Referências a E.H. Vaughn III (artilheiro anti-aéreo na Europa), S.S. Lovejoy e Jimmy Dewberry. Sub-manchete: 'Teoria de Reflexo de Holofote é Avançada'.", "bbox": {"x": 0.05, "y": 0.02, "w": 0.9, "h": 0.96}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.78, "image_type": "newspaper_clipping", "ufo_anomaly_detected": True, "ufo_anomaly_type": "multiple_witness_sighting", "ufo_anomaly_rationale": "Continued witness accounts of flying saucer sightings in Birmingham.", "image_description_en": "Newspaper clipping with more Birmingham flying saucer witness accounts and analysis of the searchlight reflection theory.", "image_description_pt_br": "Recorte de jornal com mais relatos de testemunhas de discos voadores em Birmingham e análise da teoria de reflexo de holofote."},
]),
# Page 13: p-012.png - RESTRICTED statement of Sgt. Ira L. Livingston
(13, "p-012.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "RESTRICTED", "content_pt_br": "RESTRITO", "bbox": {"x": 0.25, "y": 0.02, "w": 0.5, "h": 0.03}, "classification": "RESTRICTED", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
{"order_in_page": 2, "type": "header", "content_en": "S-T-A-T-E-M-E-N-T", "content_pt_br": "D-E-C-L-A-R-A-Ç-Ã-O", "bbox": {"x": 0.35, "y": 0.06, "w": 0.3, "h": 0.025}, "classification": None, "formatting": ["all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 3, "type": "body_paragraph", "content_en": "I, Staff Sergeant Ira L. Livingston, MA 14 153 972, Air Corps, have approximately 250 hours flying time as pilot and Armorer Gunner have the following statement to make concerning the appearance of \"Flying Discs\" in the vicinity of Birmingham, Alabama.\n\nAt 2045 hours, 6 July 1947, while I was eating supper at my residence at 1354 Meadow Lane, Green Acres, Birmingham, Alabama, my next door neighbor, Mr. Herman M. Rockwell, called for me to come to the front door that there were some \"Flying Discs\" outside. Immediately I went out the front door to observe the objects. The objects appeared to the West of Birmingham traveling in a South Western direction. They appeared to be approximately 2000 feet above the horizon at a 45 degree angle from where I was standing at an undetermined distance away. The objects appeared to be approximately two (2) feet in diameter, round in shape, producing a dim glow of light and traveling at an estimated speed of five (5) to six (6) hundred miles per hour. The objects or object appeared to be traveling in a definite are rather than straight and as soon as one was out of sight another would appear behind it, but not always in the same path. The view of where it came from was obstructed by a nearby house; and when it reached the altitude of approximately 2000feet, it started off in the same direction as the others. I did not at any time see any more than one at the time and even though there could have been only one, my personal belief is that there were seven (7) to ten (10). The Discs were silent and appeared to be composed of a single light.", "content_pt_br": "Eu, Sargento Técnico Ira L. Livingston, MA 14 153 972, Corpo Aéreo, tenho aproximadamente 250 horas de tempo de voo como piloto e Artilheiro Armador e faço a seguinte declaração sobre o aparecimento de \"Discos Voadores\" nas proximidades de Birmingham, Alabama.\n\nÀs 20h45, em 6 de julho de 1947, enquanto eu jantava em minha residência na 1354 Meadow Lane, Green Acres, Birmingham, Alabama, meu vizinho próximo, Sr. Herman M. Rockwell, me chamou para vir à porta da frente pois havia alguns \"Discos Voadores\" lá fora. Imediatamente saí pela porta da frente para observar os objetos. Os objetos pareciam estar a Oeste de Birmingham viajando em direção sudoeste. Pareciam estar aproximadamente 2000 pés acima do horizonte em um ângulo de 45 graus de onde eu estava. Os objetos pareciam ter aproximadamente dois (2) pés de diâmetro, forma redonda, produzindo um brilho fraco de luz e viajando a uma velocidade estimada de quinhentas (500) a seiscentas (600) milhas por hora. Os objetos eram silenciosos e pareciam ser compostos de uma única luz.", "bbox": {"x": 0.07, "y": 0.09, "w": 0.87, "h": 0.72}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.87, "ufo_anomaly_detected": True, "ufo_anomaly_type": "disc_sighting_silent", "ufo_anomaly_rationale": "Military sergeant reports silent disc-shaped objects traveling 500-600 mph over Birmingham, Alabama."},
{"order_in_page": 4, "type": "signature_block", "content_en": "Ira L. Livingston\nIra L. Livingston\nStaff Sergeant, MA 14 153 972\n\nSubscribed and sworn to before me this 7th day of July 1947.\n\nJames L. MacFarlane\n1st Lt, AC\nAsst. AAI", "content_pt_br": "Ira L. Livingston\nIra L. Livingston\nSargento Técnico, MA 14 153 972\n\nAssinado e jurado perante mim neste 7º dia de julho de 1947.\n\nJames L. MacFarlane\n1º Ten., Corpo Aéreo\nAss. AAI", "bbox": {"x": 0.07, "y": 0.82, "w": 0.87, "h": 0.1}, "classification": None, "formatting": ["handwritten"], "cross_page_hint": "self_contained", "ocr_confidence": 0.82},
{"order_in_page": 5, "type": "classification_banner", "content_en": "RESTRICTED", "content_pt_br": "RESTRITO", "bbox": {"x": 0.25, "y": 0.95, "w": 0.5, "h": 0.04}, "classification": "RESTRICTED", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
]),
]
# Continue with remaining pages
PAGE_DATA_2 = [
# Page 14: p-013.png - CONFIDENTIAL Newfoundland Base Command letter of transmittal
(14, "p-013.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.25, "y": 0.02, "w": 0.5, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 2, "type": "stamp", "content_en": "RESTRICTED", "content_pt_br": "RESTRITO", "bbox": {"x": 0.03, "y": 0.08, "w": 0.2, "h": 0.04}, "classification": "RESTRICTED", "formatting": ["bold"], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 3, "type": "letterhead", "content_en": "HEADQUARTERS, NEWFOUNDLAND BASE COMMAND\nATLANTIC DIVISION, AIR TRANSPORT COMMAND\nFORT PEPPERRELL, NEWFOUNDLAND\nAPO 862, S POSTMASTER, NEW YORK, N.Y.", "content_pt_br": "QUARTEL GENERAL, COMANDO DA BASE DA TERRA NOVA\nDIVISÃO DO ATLÂNTICO, COMANDO DE TRANSPORTE AÉREO\nFORT PEPPERRELL, TERRA NOVA\nAPO 862, S POSTMASTER, NOVA YORK, N.Y.", "bbox": {"x": 0.2, "y": 0.06, "w": 0.6, "h": 0.1}, "classification": None, "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 4, "type": "header", "content_en": "AMT-T-101\n3133\n/oth\nClassification: Restricted\n[date stamp] 6 Aug 47\n[Rank]\n[Date]", "content_pt_br": "AMT-T-101\n3133\n/oth\nClassificação: Restrito\n[carimbo de data] 6 Ago 47", "bbox": {"x": 0.55, "y": 0.06, "w": 0.4, "h": 0.1}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.75},
{"order_in_page": 5, "type": "date_line", "content_en": "30 July 1947", "content_pt_br": "30 de julho de 1947", "bbox": {"x": 0.65, "y": 0.17, "w": 0.3, "h": 0.025}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.93},
{"order_in_page": 6, "type": "subject_line", "content_en": "SUBJECT: Letter of Transmittal.\n\nTO: Commanding General,\nAtlantic Division, ATC,\nFort Totten, Long Island, N.Y.\n(ATTENTION: AC/S, Intelligence)", "content_pt_br": "ASSUNTO: Carta de Transmissão.\n\nPARA: Comandante Geral,\nDivisão do Atlântico, ATC,\nFort Totten, Long Island, N.Y.\n(ATENÇÃO: AC/S, Inteligência)", "bbox": {"x": 0.07, "y": 0.20, "w": 0.87, "h": 0.1}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 7, "type": "body_paragraph", "content_en": "Reference Letter of Transmittal, this office, dated 28 July 1947, with four (4) inclosures (Inclosures 1, 2 and 3, Final Reports of Sightings of \"flying saucers\"; and Inclosure 4, Signed Statement - Constable KEARSEY), transmitted herewith is Final Report of Sighting of \"flying saucers\" in Newfoundland, which occurred at Harmon Field, Stephenville, at 03452, 23 July 1947.\n\nFOR THE COMMANDING GENERAL:\n\nMARION C. MILLER\nCaptain, Air Corps\nAC/S, Intelligence.\n\n1 Incl:\nFinal Rpt of Sighting, 23 Jul 47\n\n1st Ind.\nHQ, ATLANTIC DIVISION, ATC, FORT TOTTEN, L.I., NEW YORK 6 Aug 47\nTO: Commanding General, Air Transport Command, Washington 25, D.C.\nATTN: Chief of Staff\n\nForwarded in accordance with instructions outlined in TGL CS-95, your Headquarters.\n\nJAMES H. HEMPSTER, JR.\nLt Col, GSC\nAC/S, Intelligence", "content_pt_br": "Referência Carta de Transmissão, este escritório, datada de 28 de julho de 1947, com quatro (4) anexos (Anexos 1, 2 e 3, Relatórios Finais de Avistamentos de \"discos voadores\"; e Anexo 4, Declaração Assinada - Policial KEARSEY), transmitido neste é o Relatório Final de Avistamento de \"discos voadores\" em Newfoundland, que ocorreu em Harmon Field, Stephenville, em 03452, 23 de julho de 1947.", "bbox": {"x": 0.07, "y": 0.30, "w": 0.87, "h": 0.55}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.87, "ufo_anomaly_detected": True, "ufo_anomaly_type": "saucer_sighting_newfoundland", "ufo_anomaly_rationale": "Official military letter transmitting flying saucer sighting reports from Newfoundland."},
{"order_in_page": 8, "type": "classification_banner", "content_en": "RESTRICTED", "content_pt_br": "RESTRITO", "bbox": {"x": 0.25, "y": 0.93, "w": 0.5, "h": 0.03}, "classification": "RESTRICTED", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.95},
]),
# Page 15: p-014.png - CONFIDENTIAL Final Report of Sighting, Harmon Field Newfoundland
(15, "p-014.png", [
{"order_in_page": 1, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.25, "y": 0.02, "w": 0.5, "h": 0.03}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
{"order_in_page": 2, "type": "stamp", "content_en": "RESTRICTED\nFINAL REPORT OF SIGHTING", "content_pt_br": "RESTRITO\nRELATÓRIO FINAL DE AVISTAMENTO", "bbox": {"x": 0.03, "y": 0.07, "w": 0.4, "h": 0.06}, "classification": "RESTRICTED", "formatting": ["bold"], "cross_page_hint": "self_contained", "ocr_confidence": 0.85},
{"order_in_page": 3, "type": "table_marker", "content_en": "1. Organization: 1388th AAF Base Unit, APO 862, c/o Postmaster, New York, N.T.\n2. Sighting: Strange intermittent flashes that may tie in with \"Flying Discs\".\n3. Place: Harmon Field, Stephenville, Newfoundland.\n4. Time: 03452, 23 July 1947.\n5. Altitude: Approximately 10,000 feet high.\n6. Weather: High scattered condition; visibility better than fifteen (15) miles.\n7. Heading: From South, heading NNE (approximately 30 degrees).\n8. Speed: High velocity; stated to be faster than a conventional airplane.\n9. Description: The observers saw a light which at first appeared to be a shooting star or airplane. It appeared again, and a number of intermittant flashes were seen for a period of approximately three (3) minutes. The flashes were reddish in color. Observers said it was not a falling star because it did not appear as such; nor was it an airplane, because manoeuvres were too abrupt and there was no noise of a motor.\n10. Reported by: Miss Patricia Abbott,(Newfoundland National) Government Employee and Lt. Hammakor, Navigator and Public Relations Officer.\n11. General: The informants (noted in Par.10) were walking when they noticed a peculiar reddish light.", "content_pt_br": "1. Organização: 1388ª Unidade de Base AAF, APO 862, c/o Postmaster, Nova York, N.T.\n2. Avistamento: Flashes intermitentes estranhos que podem estar ligados a \"Discos Voadores\".\n3. Local: Harmon Field, Stephenville, Newfoundland.\n4. Hora: 03452, 23 de julho de 1947.\n5. Altitude: Aproximadamente 10.000 pés de altura.\n6. Clima: Condição espalhada alta; visibilidade melhor que quinze (15) milhas.\n7. Rumo: Do Sul, rumando para NNE (aproximadamente 30 graus).\n8. Velocidade: Alta velocidade; dito ser mais rápido que um avião convencional.\n9. Descrição: Os observadores viram uma luz que a princípio parecia ser uma estrela cadente ou avião. Apareceu novamente, e vários flashes intermitentes foram vistos por um período de aproximadamente três (3) minutos. Os flashes eram avermelhados. Os observadores disseram que não era uma estrela cadente porque não apareceu como tal; nem era um avião, porque as manobras eram muito abruptas e não havia barulho de motor.\n10. Relatado por: Senhorita Patricia Abbott (funcionária do Governo Nacional da Newfoundland) e Ten. Hammakor, Navegador e Oficial de Relações Públicas.\n11. Geral: Os informantes notaram uma luz avermelhada peculiar.", "bbox": {"x": 0.07, "y": 0.13, "w": 0.87, "h": 0.72}, "classification": "CONFIDENTIAL", "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.87, "ufo_anomaly_detected": True, "ufo_anomaly_type": "reddish_flashing_light", "ufo_anomaly_rationale": "Official sighting report of reddish intermittent lights making abrupt maneuvers at high speed over Newfoundland."},
{"order_in_page": 4, "type": "signature_block", "content_en": "WILLIAM H. SMITH\nCaptain, Air Corps,\nIntelligence Officer.", "content_pt_br": "WILLIAM H. SMITH\nCapitão, Corpo Aéreo,\nOficial de Inteligência.", "bbox": {"x": 0.5, "y": 0.86, "w": 0.4, "h": 0.06}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.88},
{"order_in_page": 5, "type": "stamp", "content_en": "Level 1", "content_pt_br": "Nível 1", "bbox": {"x": 0.05, "y": 0.92, "w": 0.12, "h": 0.03}, "classification": None, "formatting": [], "cross_page_hint": "self_contained", "ocr_confidence": 0.8},
{"order_in_page": 6, "type": "classification_banner", "content_en": "CONFIDENTIAL", "content_pt_br": "CONFIDENTIAL", "bbox": {"x": 0.25, "y": 0.95, "w": 0.5, "h": 0.04}, "classification": "CONFIDENTIAL", "formatting": ["bold", "all_caps"], "cross_page_hint": "self_contained", "ocr_confidence": 0.9},
]),
]
# Combine all page data
ALL_PAGE_DATA = PAGE_DATA + PAGE_DATA_2
print(f"Defined {len(ALL_PAGE_DATA)} pages with detailed chunk data")
print("Writing chunk files...")
# We'll write as many as we can define here, then handle the rest programmatically
all_chunks = []
global_order = 0
for (page_idx, png_fn, chunks) in ALL_PAGE_DATA:
for chunk in chunks:
global_order += 1
chunk_id = f"c{global_order:04d}"
chunk["chunk_id"] = chunk_id
chunk["page"] = page_idx
chunk["order_global"] = global_order
chunk["source_png"] = f"../../processing/png/{DOC_ID}/{png_fn}"
chunk["png_filename"] = png_fn
# defaults
for key in ["ufo_anomaly_detected", "cryptid_anomaly_detected"]:
if key not in chunk:
chunk[key] = False
for key in ["ufo_anomaly_type", "ufo_anomaly_rationale", "cryptid_anomaly_type", "cryptid_anomaly_rationale",
"image_type", "image_description_en", "image_description_pt_br", "extracted_text",
"redaction_code", "redaction_inferred_content_type", "related_image", "related_table"]:
if key not in chunk:
chunk[key] = None
all_chunks.append(chunk)
# Set prev/next
for i, chunk in enumerate(all_chunks):
chunk["prev_chunk"] = all_chunks[i-1]["chunk_id"] if i > 0 else None
chunk["next_chunk"] = all_chunks[i+1]["chunk_id"] if i < len(all_chunks) - 1 else None
print(f"Prepared {len(all_chunks)} chunks from {len(ALL_PAGE_DATA)} pages")
# Save intermediate state for continuation
with open("/tmp/doc65_chunks_partial.json", "w", encoding="utf-8") as f:
json.dump({"chunks": all_chunks, "last_page": ALL_PAGE_DATA[-1][0]}, f, ensure_ascii=False, indent=2)
print("Saved partial state to /tmp/doc65_chunks_partial.json")
print(f"Last page processed: {ALL_PAGE_DATA[-1][0]}")

7
web/.dockerignore Normal file
View file

@ -0,0 +1,7 @@
node_modules
.next
.git
.env*
*.log
.DS_Store
README.md

24
web/.env.local.example Normal file
View file

@ -0,0 +1,24 @@
# Copy to .env.local and fill in. Both for local dev (with `supabase start`)
# and prod (with Coolify-provided URLs).
# Supabase (required for auth + chat persistence)
NEXT_PUBLIC_SUPABASE_URL=http://localhost:54321
NEXT_PUBLIC_SUPABASE_ANON_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
SUPABASE_SERVICE_ROLE_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
# Chat agent — providers (NEVER use ANTHROPIC_API_KEY in this project)
# Primary: Claude Code via OAuth — get a token with `claude setup-token`
CLAUDE_CODE_OAUTH_TOKEN=sk-ant-oat01-...
CLAUDE_CODE_MODEL=haiku
# Fallback: OpenRouter (OpenAI-compatible API, free models available)
OPENROUTER_API_KEY=sk-or-v1-...
OPENROUTER_MODEL=deepseek/deepseek-v4-flash:free # primary (tool calling)
OPENROUTER_FALLBACK_MODEL=nvidia/nemotron-3-super-120b-a12b:free
# Pattern C tool calling needs OpenRouter for now
CHAT_PROVIDER=openrouter
# Public URL (for magic-link redirects)
NEXT_PUBLIC_SITE_URL=http://localhost:3030
# UFO data root (filesystem path containing wiki/, processing/, raw/)
UFO_ROOT=/Users/guto/ufo

50
web/Dockerfile Normal file
View file

@ -0,0 +1,50 @@
# Multi-stage build for Next.js 15 app.
# Builds in node:22, ships in distroless-ish node:22-slim, runs as non-root.
FROM node:22-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm ci --legacy-peer-deps
FROM node:22-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
# NEXT_PUBLIC_* env vars are inlined into the client bundle at build time —
# we MUST receive them as build args, otherwise the browser gets undefined.
ARG NEXT_PUBLIC_SUPABASE_URL
ARG NEXT_PUBLIC_SUPABASE_ANON_KEY
ARG NEXT_PUBLIC_SITE_URL
ENV NEXT_PUBLIC_SUPABASE_URL=$NEXT_PUBLIC_SUPABASE_URL
ENV NEXT_PUBLIC_SUPABASE_ANON_KEY=$NEXT_PUBLIC_SUPABASE_ANON_KEY
ENV NEXT_PUBLIC_SITE_URL=$NEXT_PUBLIC_SITE_URL
ENV NEXT_TELEMETRY_DISABLED=1
RUN npm run build
FROM node:22-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
ENV PORT=3000
ENV HOSTNAME=0.0.0.0
RUN addgroup --system --gid 1001 nodejs && \
adduser --system --uid 1001 nextjs
# Copy build artifacts. We use the default Next output (not standalone) so the
# image stays simple and we can mount UFO_ROOT volumes that the API reads at runtime.
COPY --from=builder --chown=nextjs:nodejs /app/.next ./.next
COPY --from=builder --chown=nextjs:nodejs /app/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/package.json ./package.json
COPY --from=builder --chown=nextjs:nodejs /app/node_modules ./node_modules
COPY --from=builder --chown=nextjs:nodejs /app/next.config.ts ./next.config.ts
# Server-side code that gets read at runtime
COPY --from=builder --chown=nextjs:nodejs /app/app ./app
COPY --from=builder --chown=nextjs:nodejs /app/lib ./lib
COPY --from=builder --chown=nextjs:nodejs /app/components ./components
COPY --from=builder --chown=nextjs:nodejs /app/middleware.ts ./middleware.ts
USER nextjs
EXPOSE 3000
CMD ["npx", "next", "start"]

89
web/README.md Normal file
View file

@ -0,0 +1,89 @@
# web — Disclosure Bureau Next.js app
Next.js 15 + React 19 + Tailwind + Supabase + assistant-ui.
## Quick start (local dev)
```bash
# 1. Install deps
npm install
# 2. (Optional) Start local Supabase
# Requires Docker. Skip if pointing at remote Supabase.
npx supabase init # first time only — creates supabase/ folder
npx supabase start # spins up Postgres/GoTrue/Storage on :54321
# 3. Configure env
cp .env.local.example .env.local
# Edit .env.local — paste local Supabase keys (printed by `supabase start`)
# 4. Apply migrations
psql postgresql://postgres:postgres@localhost:54322/postgres \
-f ../infra/supabase/migrations/0001_chat_schema.sql
# 5. Start dev
npm run dev
# http://localhost:3030
```
## Without Supabase
The app degrades gracefully if Supabase env vars are unset:
- Wiki browsing works (read-only from filesystem)
- Auth bar shows "auth: disabled (dev)"
- Chat bubble shows "Auth not configured"
Useful for quick UI work without spinning up Docker.
## Production (Coolify on VPS)
See [`../infra/coolify/`](../infra/coolify/). Stack:
- Coolify orchestrates everything
- Supabase self-hosted: `db.disclosure.top`, `studio.disclosure.top`
- Next.js: `disclosure.top`
- Meilisearch (shared): `search.disclosure.top`
- Imgproxy (shared): `img.disclosure.top`
- Caddy: TLS + reverse proxy (built into Coolify)
## Architecture
```
app/
├── page.tsx # home — 116 docs grouped by collection
├── auth/
│ ├── signin/page.tsx # magic-link form
│ ├── callback/route.ts # exchanges code for session
│ └── signout/route.ts
├── d/[docId]/
│ ├── page.tsx # doc detail
│ └── [page]/page.tsx # page reader (OCR + entity highlights + crops + sidebar PNG)
├── api/
│ ├── me/route.ts # GET current profile
│ ├── sessions/route.ts # GET list, POST new
│ ├── sessions/[id]/route.ts # GET detail, PATCH, DELETE
│ ├── sessions/[id]/messages/route.ts # POST send → assistant reply
│ ├── documents/, pages/, entities/, tables/ # read-only data
│ └── static/[...path]/route.ts # sandboxed file serve
components/
├── chat-bubble.tsx # floating Sherlock — auth-aware, session list
├── entity-modal.tsx # opens on entity click
├── reader-content.tsx # OCR + highlights + crops
└── auth-bar.tsx # sign in / out + budget tracker
lib/
├── wiki.ts # markdown reader (gray-matter)
├── entity-index.ts # match loader + text segmentation
└── supabase/{server,client}.ts # SSR helpers
middleware.ts # session refresh on every request
```
## Tech notes
- **No RAG**: chat agent reads markdown directly. Wiki-link traversal substitutes for vector search.
- **RLS-first**: Supabase Row Level Security enforces "user sees only own sessions" at the DB layer.
- **Magic-link auth**: no passwords. GoTrue handles email delivery.
- **Anti-abuse**: per-user budget cap (default $5) + daily message quota (default 100) enforced via `check_budget` RPC before each Claude call.
## Cost
Each chat turn costs ~$0.005-0.05 depending on context size (mostly Haiku $1/M input, $5/M output).

Some files were not shown because too many files have changed in this diff Show more