Closes a UX gap the user surfaced: W3.5-3.8 built 8 detectives, 4 new
URL endpoints (/jobs/[id], /h/[id], /c/[slug], /api/h/[id]/red-team)
and a chat tool, but the homepage was unchanged — the bureau was
invisible unless you knew the URL or asked the chat to invoke
request_investigation.
Homepage (web/app/page.tsx):
- Title `▍ war.gov/ufo — Investigative Wiki` → `▍ The Disclosure Bureau`
- Subtitle expanded from "Holmes · Poirot · Dupin · Locard" to all 8
detectives (Holmes · Locard · Dupin · Schneier · Poirot · Taleb ·
Tetlock · Case-Writer)
- New `🔎 bureau` topbar link (gold, between graph/stats and batch)
- BureauSnapshot inserted right after the header
BureauSnapshot (web/components/bureau-snapshot.tsx) — server component:
- 8 detective tiles with role labels (each in its tone color)
- 6 clickable counters (evidence / hypotheses / contradictions /
witnesses / outliers / case reports) — anchor to /bureau#section
- 6 "recent artefacts" columns surfacing the last 3-4 of each kind:
hypotheses with prior→posterior + band + ↳reviewed_by marker,
contradictions with topic + resolution_status, evidence with
Grade badge + verbatim quote, outliers with title + scope.kind,
witness analyses with canonical_name + credibility + verdict,
case reports with slug + link to /c/<slug>
- "Recent jobs" strip linking to /jobs/[id] color-coded by status
- Reports read from /data/ufo/case/reports/ via fs.readdir + stat,
sorted by mtime — no DB round-trip needed for that section
/bureau (web/app/bureau/page.tsx) — full hub:
- Header with full counts
- 7 sections (anchored to homepage counter links): Case reports,
Hypotheses, Evidence, Contradictions, Outliers, Witnesses,
Recent jobs table — each rendering up to 100 rows
- Reports section parses frontmatter from each .md to surface topic
+ n_hypotheses + n_evidence on the card
Runtime fixes batched in:
- Poirot: coerce entity_pk via Number() — node-postgres returns
BIGINT as string by default; writer's Number.isFinite() rejected
it as "person_entity_pk required" (j-edgar-hoover retry path)
- Tetlock: write_calibration rationale cap 600 → 1200 chars. Prompt
still asks ≤ 600 but a 2× slack beats failing the job on honest
analysis. Observed live: Tetlock emitted ~620 chars on H-0003 and
the writer rejected the entire calibration.
- Case-Writer: Promise.all of 5 queries × max_parallel=2 jobs
demanded up to 10 connections against the investigator role's
rolconnlimit=4 → "too many connections for role investigator".
Sequentialized — the LLM call is the hot path, not these queries.
Smoke results visible now on the homepage:
- 3 hypotheses (H-0001/2/3) about green fireballs origin
- 3 contradictions (R-0001/2/3) about color, geographic confinement,
exclusive-green vs multicolored
- 2 evidence cards (E-0002/3) Grade B
- 3 outliers (G-0001/2/3) — including Taleb's deliberate
meteor-shower-camouflage flag
- 1 case report at /c/green-fireballs-sandia (Watson 13.4 KB,
five-act narrative, fully cited)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the bureau from 4 → 8 detectives. All eight run as Bun + claude-CLI
subprocesses against the same Supabase + investigation_jobs LISTEN/NOTIFY
queue, sharing search.ts hybridSearch and writer-side validators that
gate writes against schema + FK.
New detectives:
Poirot (witness_analysis)
- prompts/poirot.md — credibility / access / bias / corroboration /
verdict; uses entity_mentions JOIN chunks to pull 12 chunks per
person; resolves corroboration_refs chunk_ids defensively (accepts
bare cNNNN even when the model emits pNNN/cNNNN).
- INSERT into public.witnesses with W-NNNN naming.
- Tone: purple (#9b5de5).
Taleb (outlier_scan)
- prompts/taleb.md — "surprise is relative to a model"; at most 3
outliers; each requires explicit dominant_model + why_surprising +
what_it_implies; fan-out into public.gaps with scope.kind="outlier".
- Same unscoped-fallback as Dupin (Pass 1 with doc_id, Pass 2 widens
to corpus if hits < 3).
- Tone: yellow (#ffd23f).
Tetlock (calibrate_hypothesis)
- prompts/tetlock.md — honest Bayesian update; emits new_posterior +
Δ + recommended_action ∈ {keep, downgrade, upgrade, supersede}.
- write_calibration UPDATEs public.hypotheses + APPENDS a
"## Calibration history" section to the H-NNNN.md case file
(calibration is append-only — each datapoint matters). Posterior
band auto-corrected to match Tetlock thresholds.
- NO_NEW_EVIDENCE sentinel handled; pure 'keep' with |Δ|<0.005 only
touches updated_at + reviewed_by.
- Tone: teal (#26d4cc).
Case-Writer (case_report)
- prompts/case-writer.md — Dr. Watson assembles all artefacts
(E-NNNN, H-NNNN, R-NNNN, W-NNNN, G-NNNN) into a five-act narrative.
ILIKE filter on topic; doc_id optional scope.
- Larger budget cap (≥ $0.50) + longer timeout for prose generation.
- Writes case/reports/<slug>.md with frontmatter (topic + counts);
no DB table for v0.
- New page /c/[slug] renders the report via MarkdownBody + stat chips.
- Tone: gold (#e0c080).
Hardening across the bureau:
- Sentinel parsing now accepts backticked AND prose-trailing forms
(Holmes NO_HYPOTHESES, Dupin NO_CONTRADICTIONS, Schneier
INSUFFICIENT_HYPOTHESIS, Poirot INSUFFICIENT_TESTIMONY, Taleb
NO_OUTLIERS, Tetlock NO_NEW_EVIDENCE, Case-Writer
INSUFFICIENT_ARTEFACTS). Avoids the failure mode where the model
refuses honestly but the runtime treated it as a parse error
(observed live with Poirot+Hoover identifying the DIRECTOR
false-positive disambiguation issue in entity_mentions).
Chat tool extensions (web/lib/chat/tools.ts):
- request_investigation now accepts 7 kinds. Each routes to its
detective with appropriate validation (hypothesis_id regex,
person_id kebab-case, topic non-empty, doc_id for evidence_chain).
- ETA per kind: Holmes/Dupin 60s, Poirot 45s, Schneier/Tetlock 30s,
Taleb 50s, Case-Writer 180s (longer prose), Locard 30×n_chunks.
UI integration:
- chat-bubble inline card paints each detective in its tone color.
- /jobs/[id] page header swaps name/subtitle/tone per detective;
question label adapts ("Topic" / "Hypothesis under attack" /
"Witness under analysis" / "Topic to outlier-scan" / "Hypothesis
under recalibration" / "Case to assemble").
- job-status-poller renders: case-report link card (gold), outlier
cards (yellow), witness cards (purple) — alongside existing
hypothesis, evidence, contradiction cards.
- /api/jobs/[id] hydrates witnesses (JOIN entities for canonical_name)
+ gaps (with scope JSONB).
- /c/[slug] page reads /data/ufo/case/reports/<slug>.md and renders
with MarkdownBody, frontmatter parsed for stat chips.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the fourth AI detective in the Investigation Bureau runtime: Bruce
Schneier, who attacks an existing hypothesis as a red-team operator.
Runtime:
- prompts/schneier.md — discipline (don't disprove, just attack;
structured output with hidden_assumptions, failure_modes,
alternative_explanations, recommended_tests, verdict_one_sentence;
severity ∈ {low, medium, high}; emit INSUFFICIENT_HYPOTHESIS when
the input is too thin)
- src/detectives/schneier.ts — reads the hypothesis row + evidence
chain (joined via evidence_refs FK), feeds Claude with the
arguments + verbatim quotes, parses strict JSON object
- src/tools/write_red_team_review.ts — UPDATEs hypotheses.reviewed_by
+ updated_at; APPENDS (or replaces if re-reviewed) a structured
"## Red-team review (Schneier · X severity)" section to
case/hypotheses/H-NNNN.md. Caps each list at 5 entries × 240 chars,
validates verdict ≤ 280 chars.
- orchestrator: new `red_team_review` kind dispatching to runSchneier
Chat + UI:
- request_investigation gains kind=red_team_review + hypothesis_id arg
(validated against H-NNNN regex); detective auto-resolves to schneier
- chat-bubble inline card paints Schneier in red (#ff3344)
- /jobs/[id] page swaps title/subtitle/tone per detective; the
"Question" label becomes "Hypothesis under attack" for red_team_review
New /h/[hypothesisId] page (hypothesis dossier):
- Server-rendered from public.hypotheses + public.evidence (joined
via evidence_refs FK + chunk lookup)
- Header: ID + creator + reviewer (highlighted when Schneier has
visited), position as headline, question subtitle, Tetlock band
- Prior + posterior bars with Δ-delta indicator
- Argument grid: argument_for (green) vs argument_against (pink)
side-by-side with [[wiki-link]] auto-linking to source chunks
- Evidence chain: each E-NNNN with Grade A/B/C badge, verbatim
blockquote, link to source page
- Red-team review panel: parses the markdown section in the case
file (severity badge, verdict, 4 bullet panels for
hidden_assumptions / failure_modes / alternative_explanations /
recommended_tests). Empty state when not yet reviewed.
RedTeamRequestButton client component + POST /api/h/[id]/red-team —
authenticated user can trigger Schneier in one click; UI swaps to
"acompanhar" link to /jobs/[id] once queued.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the third AI detective in the Investigation Bureau runtime: C. Auguste
Dupin, who scans a corpus shortlist for pairs (or small groups) of chunks
that cannot both be true under any ordinary reading.
Runtime:
- prompts/dupin.md — discipline (no contradiction without ≥2 distinct
chunk_ids; reject same-vocabulary near-misses; FEW high-confidence
over MANY weak ones; emit `NO_CONTRADICTIONS` when corpus is silent)
- src/detectives/dupin.ts — hybridSearch with k=18 (more chunks than
Holmes because contradictions emerge from comparing dispersed
claims), strict JSON-array parsing, AT MOST 3 contradictions per call
- src/tools/write_contradiction.ts — validates topic + ≥2 positions
drawn from ≥2 distinct chunks, resolves chunk_pk via DB lookup
(rejects positions citing unknown chunks), INSERTs into
public.contradictions + writes case/contradictions/R-NNNN.md
- orchestrator: new `contradiction_scan` kind dispatching to runDupin;
payload { topic, doc_id?, lang?, context_chunks? }
Chat + UI:
- request_investigation gains kind=contradiction_scan + topic arg;
triggered detective auto-resolves to dupin
- chat-bubble inline card renders dupin in orange (#ff8a4d) to
distinguish from holmes (cyan) and locard (green)
- /jobs/[id] page swaps title + subtitle + tone per detective;
"Question" label becomes "Topic" for contradiction_scan
- /api/jobs/[id] hydrates public.contradictions when outputs[] surfaces
contradiction_ids
- job-status-poller renders ContradictionCard: topic + N positions
(verbatim statements quoted, stance label optional, link to source
chunk) + optional notes panel, with resolution_status badge
(open/resolved/irreconcilable)
R-NNNN shares the contradiction_id_seq slot with relation per
CLAUDE.md naming — same conceptual class (a connection between two
pieces of evidence in tension).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the loop between the chat UI and the Investigation Bureau runtime.
Chat tool (web/lib/chat/tools.ts):
- request_investigation { kind, question, doc_id?, chunks?, claim? }
INSERTs a row in public.investigation_jobs and returns
{ job_id, kind, status, eta_seconds, status_url, detective }.
- kind=hypothesis_tournament → Holmes (1 question → 2-3 rival hypotheses)
- kind=evidence_chain → Locard (1 doc → grade-A/B/C evidence with chain
of custody, default top-5 anomaly chunks)
- Plumbed user.email through ToolHandlerContext so triggered_by audits
the requesting user.
Public job viewer:
- GET /api/jobs/[id] joins investigation_jobs → public.evidence +
public.hypotheses for the IDs surfaced in outputs[]. Returns one
payload the page can render without n+1 round-trips. Strips
triggered_by from the response (it carries the user's email).
- app/jobs/[id]/page.tsx server-renders the case-file shell:
detective lore header (Holmes blue or Locard green), question chip,
scope chip with link back to the document.
- components/job-status-poller.tsx client island that polls every 3 s
while non-terminal, then once on terminal to hydrate evidence +
hypotheses. Renders:
· Phase tracker (queued → running → complete | failed)
· Hypothesis cards w/ prior + posterior bars + Δ delta indicator
+ Tetlock band badge (high/medium/low/speculation)
· Argument-for / argument-against with [[wiki-link]] auto-linking
to /d/<doc>/p<NNN>#<cNNNN>
· Evidence cards w/ Grade A/B/C badge + verbatim blockquote +
bbox crop preview via /api/crop + custody-steps disclosure
· Empty/in-flight panel ("os detetives estão lendo o corpus")
· Failure panel surfacing error + partial outputs
Inline chat-bubble card (components/chat-bubble.tsx):
- ToolTrace.richRender recognises request_investigation results and
renders a detective banner with status + ETA + link to /jobs/[id]
(target=_blank). Error case renders a red strip with the message.
UX flow now: user asks Sherlock a question → request_investigation
queues the job → chat card shows "🔎 Holmes · hypothesis_tournament ·
ETA ~60s" → user clicks → /jobs/<id> live-updates → 60 s later, 2-3
rival hypotheses + their arguments + chunk citations are rendered with
Bayesian update visible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scanned docs are messy — duplicate transcriptions (typed + handwritten),
two classification variants of the same narrative, OCR noise, repeated
banners. The doc page showed raw chunks, so everything appeared twice.
40_reading_version.py generates ONE clean, deduplicated, well-structured
bilingual Markdown reading version per doc (Sonnet): merges duplicate versions
without losing unique lines, drops page furniture, formats transcripts as
dialogue. Faithful — invents nothing; redactions kept as markers.
/d/[docId] now defaults to a "📖 leitura" tab rendering this clean version,
with "🔍 trechos · scan original" preserving the faithful per-chunk + per-page
scan view. reading.md lives in raw/<doc>--subagent/ alongside the chunks.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Doc page (/d/[docId]/[page]) gains prev/next navigation bars (top + bottom):
within a doc it steps page-by-page; at the first/last page it jumps to the
previous/next document. Replaces the disabled-at-boundary links.
Indexer tooling for the VPS repopulation:
- 30-index-chunks-to-db.py: add --no-embed (fast BM25-only index; vectors
backfilled separately) so the app is usable in minutes, not hours of CPU
embedding.
- 57_load_relations_from_json.py: load typed relations into public.relations
from reextract structured fields (deterministic ids, no fuzzy guessing).
- 58_backfill_embeddings.py: async pass to fill chunks.embedding (NULL rows)
via the embed-service.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add reextract pipeline (scripts/reextract/) that rebuilds doc-level entity
JSON from Sonnet-vision chunks via Opus, replacing the noisy per-page
extraction. Add synthesize scripts to regenerate wiki/entities from the 116
_reextract.json (30), aggregate missing page.md from chunks (31), and reprocess
805 pages the doc-rebuilder agent dropped on context overflow (32). Add
maintain scripts 43-56 for chunk-page sync, dedup, generic-entity marking, and
typed relation extraction.
Web: wire relations API + entity-relations component; entity/timeline/doc
pages consume the rebuilt layer.
Note: raw/, processing/, wiki/ remain gitignored (bulk data managed
separately); the 116 reextract JSONs and 7,798 rebuilt entity files live on
disk only. The 27 curated anchor events under wiki/entities/events/ are
preserved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The corpus had two parallel reverse-reference signals: the wiki/pages
entities_extracted blocks (Haiku page-level) and public.entity_mentions
(Sonnet chunk-level, ILIKE-matched). The entity page only consulted the
DB, so it showed "0 menções" for thousands of entities that were anchored
in pages or in cross-entity links the DB never indexed.
Resolved by collapsing all signals into the YAML frontmatter, which is
now the single runtime source for entity metadata.
scripts/maintain/42_sync_entity_stats.py walks every entity and writes:
mentioned_in: [...] # consolidated page refs
total_mentions: max(db, pages)
documents_count: max(db_docs, distinct page docs)
signal_sources:
db_chunks: int
page_refs: int
cross_refs: int
signal_strength: strong | weak | orphan | unverified
referenced_by: [[class/id]] # cross-entity backlinks
Outgoing wikilinks (e.g. OBJ.observed_in_event → EV) count toward the
entity's own cross_refs so anchored-but-not-mentioned entities don't
register as orphan.
OBJ canonical names like "7m long, 1.3m high, two rocket motors,
smooth flow, rotary drive null UAP (OBJ-EV1945-PEYERLSHOTDOWN-01)"
are rewritten to "Peyerl shot down UAP" derived from observed_in_event,
preserving the full description as an alias. --fix-obj-names did this
for every OBJ-* with >80 char canonical_name.
Default behaviour is conservative: --archive-only-junk archives only
single/double-char names and pure-numeric noise. Everything else stays
on disk with signal_strength marked, so the user can review later.
web/lib/retrieval/entity-pages.ts swapped from db-first to yaml-first.
The /e/[cls]/[id] page now reads counts straight from YAML and renders
a "força do sinal" badge with the per-source breakdown. Orphan entities
get a banner explaining they have no cross-references.
DB is still queried for ONE thing: the chunk text for preview cards on
the entity page, so we don't re-parse 21k markdown files on every render.
First-pass result: 9020 strong / 14520 weak / 10814 orphan; OBJ-EV1945-
PEYERLSHOTDOWN-01 now reads "Peyerl shot down UAP · fraca · 1 backlink"
in the live UI.
Two bugs combined to make the chat reply with only cards and no prose:
1. SQL trigger rollup_session_stats was failing with "column reference
total_cost_usd is ambiguous" because the UPDATE on public.profiles had
a FROM public.chat_sessions clause and both tables expose that column.
Persistence of every user message died at this point — sessions were
created in the DB but had message_count=0 forever. Applied SQL fix
that qualifies columns with p./s. aliases (production DB updated;
ALTER FUNCTION run live, not yet codified in a migration file).
2. The free-tier model (nemotron-3-super:free) spent all 5 tool-loop
turns on hybrid_search calls and never wrote any prose, returning
content_len=0. Added a forced-synthesis pass in openrouter.ts: when
the loop exits with empty assembledText but the model did call tools,
we send ONE final turn with tools omitted from the request payload
and a user message instructing the model to answer in 3-8 sentences
citing chunks. openrouterStreamCall now accepts a `withTools` opt
so the synthesis call can disable tool calling entirely.
Verified end-to-end with the actual user query "O que os astronautas
viram? Quem foi que viu?" on /d/nasa-uap-d6-apollo-17-...:
- content_len: 0 → 947 chars (real synthesis citing Schmitt)
- artifacts: 44 preserved
- assistant message persisted with tool_calls + citations columns
34 of 116 docs were generated with 0-based PNG numbering (p-000.png …
p-008.png) but the Sonnet chunks reference 1-based page numbers in their
YAML frontmatter (page: 9 means the 9th sheet of paper). The /api/crop
handler built p-009.png and got a 500, the browser's Next/Image surfaced
400, and the chunk rendered as a black box on screen.
Fixes:
- web/app/api/crop/route.ts: try p-NNN.png first, fall back to
p-(NNN-1).png if the 1-based file is missing. Cheap insurance for any
doc that comes in with the old convention.
- scripts/01-convert-pdfs.sh: previously printf '%03d' "$num" with $num
starting at 0 (e.g. "008") raised "invalid number" because Bash
parsed it as octal. Wrap with $((10#$num)) to force decimal — this
was silently corrupting page sequences and producing gaps like p-001
... p-008, p-011 (missing p-009/p-010).
- All 34 affected docs re-converted from PDFs with the patched script;
every directory now has continuous 1-based PNGs.
- /processing/png/ rsync'd to VPS, web redeployed.
Smoke: /api/crop?doc=doc-341-…&page=9&… now returns 200 image/webp
instead of 500. Tested in browser: chunk c0026 (diagram, p9) renders
the real engineering diagram.
Fase 3 onda 2 — entity synthesis at scale:
- scripts/synthesize/20_entity_summary.py: queries DB for entities with
total_mentions ≥ threshold + top-K verbatim chunk snippets via
entity_mentions JOIN, prompts Sonnet (Holmes-Watson voice, bilingual),
writes narrative_summary EN+PT-BR + summary_status=synthesized.
Ran on 187 candidates (mentions ≥ 20) → 158 OK · 1 err · 29 skipped (no
snippets). Combined with anchor curation: 20 curated + 158 synthesized
= 178 entities with real narrative (vs 0 a day ago).
Fase 4 — chat with typed artifacts + persistence:
- lib/chat/agui.ts: AG-UI v1 typed Artifact union (citation, crop_image,
entity_card, evidence_card, hypothesis_card, case_card, navigation_offer)
alongside the existing event types.
- lib/chat/tools.ts + openrouter.ts: hybrid_search emits up to 6
citation + crop_image artifacts per query. Provider collects them and
returns in done.artifacts so the route can persist.
- api/sessions/[id]/messages: persist artifacts to messages.citations.
- components/chat-bubble.tsx: ArtifactCard renders inline cards (citation,
crop_image, entity_card, navigation_offer) for streamed and persisted
messages. activeId now persisted in localStorage so navigation between
pages keeps the same conversation. New sessions are lazy (only when user
has zero). loadMessages hydrates tools + artifacts from server. CRUD UI:
rename (✎) + archive (🗑) buttons per session in the list.
Home search:
- doc-list-filters: input now fires hybrid_search (rerank=0 for speed)
in parallel with the local title filter; chunk hits render above the doc
grid with snippet + score + classification.
- api/search/hybrid: accept ?rerank=0 to skip the cross-encoder (1.3s vs 60s).
Auth flow:
- infra: SMTP_HOST=mail.spacemail.com:587 + DMARC published; mail now lands
in inbox. GOTRUE_MAILER_AUTOCONFIRM=false (real email verification).
- kong.yml: proxy /auth/callback on api.disclosure.top → web:3000 so PKCE
email links don't 404 at the gateway.
- web/app/auth/callback: handle both ?code= (OAuth) and ?token=&type=
(PKCE); redirect to the public site host before verifyOtp so the session
cookie lands on the right domain.
Audit deliverables:
- .nirvana/outputs/disclosure-bureau/.../systems-atelier/: 5 docs (code
analysis, tech debt, discovery brief, system arch, 5 ADRs) authored by
sa-principal that produced this roadmap. Kept in-tree for traceability.
- scripts/03-dedup-entities.py: stop emitting placeholder narrative ("Stub. Will
be enriched in Phase 7"); write summary_status=none + null fields instead.
- scripts/maintain/41_strip_stubs.py: idempotent migration that cleaned the
22,096 entity .md files (now zero stub strings in wiki/).
- scripts/synthesize/01_anchor_events.py: curated 20 anchor UAP events
(Roswell, Nimitz Tic-Tac, Phoenix Lights, Operação Prato, AATIP, etc.) with
bilingual Holmes-Watson narrative via claude -p --model sonnet
(CLAUDE_CODE_OAUTH_TOKEN). All summary_status=curated, confidence=high.
- web/api/timeline + timeline-view: filter narrative-less events by default,
render "curado" badge for hand-vetted ones, drop the date display alone.
- CLAUDE-schema-full.md: document the summary_status enum and the four states.
- docker-compose.yml: SMTP_HOST=mail.spacemail.com configured;
GOTRUE_MAILER_AUTOCONFIRM flipped to false (real email confirmation working).
- .nirvana/outputs/.../systems-atelier/: 5 deliverables of the architecture
audit that produced this roadmap.
Luiz Gustavo