Live PT-BR smoke on j-edgar-hoover produced verdict_pt_br at 304 chars
(prompt says ≤ 280). The writer correctly rejected it ("verdict too long
(304 > 280)") but the job failed instead of trimming.
Fix: detective now trims each language field at the nearest sentence
boundary (period or semicolon) above 60% of the cap; falls back to a hard
cut at the cap. Applied to verdict / verdict_pt_br (≤280), and to
access_to_event*, bias_notes* (≤800) for defense in depth.
The contract with the writer stays strict; the detective just becomes
forgiving about the model going 5-10% over.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User flagged that the bureau was emitting English-only output, violating
the project's bilingual rule. Every narrative field now ships in both
languages: stored in sibling DB columns + rendered as adjacent markdown
sections per CLAUDE.md §3.
Migration 0007 (apply as supabase_admin):
- public.hypotheses +question_pt_br, +position_pt_br,
+argument_for_pt_br, +argument_against_pt_br
- public.contradictions +topic_pt_br, +notes_pt_br
- public.witnesses +access_to_event_pt_br, +bias_notes_pt_br,
+verdict_pt_br
- public.gaps +description_pt_br, +suggested_next_move_pt_br
- public.evidence: unchanged (verbatim_excerpt stays source-language)
- JSONB siblings inside contradictions.chunks + gaps.scope handled at
runtime (statement_pt_br, title_pt_br, dominant_model_pt_br,
why_surprising_pt_br, what_it_implies_pt_br).
Detective prompts (all 7) rewritten with explicit bilingual JSON contract:
- Output protocol section names every EN field + its _pt_br sibling
- "Bilingual is mandatory" warning in the task instruction
- Sentinel skip-states unchanged (NO_HYPOTHESES, NO_CONTRADICTIONS,
INSUFFICIENT_TESTIMONY, INSUFFICIENT_HYPOTHESIS, NO_OUTLIERS,
NO_NEW_EVIDENCE, INSUFFICIENT_ARTEFACTS)
- Schneier: parallel arrays — hidden_assumptions[i] matches
hidden_assumptions_pt_br[i], lengths must match
- Case-Writer: interleaved §1 (EN) / §1 (PT-BR) per act in the body
Writer-side validation (all 7 tools):
- Reject INSERT if PT-BR sibling missing when EN field is set
- Persist both languages atomically in one INSERT (no half-updates)
- Markdown renderers write adjacent EN+PT-BR sections in case files
(## Argument for (EN) followed by ## Argumento a favor (PT-BR), etc.)
Detective parse layer (all 7 detectives):
- Coerce both keys from JSON output
- "incomplete_bilingual_*" skip reason when either side missing
- Defensive: PT-BR fields trimmed + length-capped same as EN
Orchestrator propagates question_pt_br + topic_pt_br through job payload
to runHolmes / runCaseWriter, mirroring the chat-tool entry point.
Web (UI):
- /api/jobs/[id] hydrates _pt_br siblings from pg
- job-status-poller HypothesisCard: PT-BR primary, EN in <details>
fallback when both exist
- ContradictionCard: PT-BR statement primary + secondary EN quote
- WitnessCard: PT-BR verdict primary + secondary EN quote, panels in PT
- GapCard: PT-BR title/why/implies primary
- /bureau hub: SELECTs both columns, renders PT-BR primary
- /h/[id]: ArgumentPanel renders PT-BR primary with collapsible EN
fallback when both exist
- BureauSnapshot homepage: position_pt_br / topic_pt_br / verdict_pt_br
primary
- DocBureauPanel /d/[doc]: same primary-PT-BR pattern
- New web/lib/i18n/pick.ts helper (unused yet by chat/agents — kept
for future locale-driven switching when both languages are equally
full; current rule is PT-BR-first since the user is brasileiro)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Builds on top of W3.9 to turn the homepage Bureau from a read-only
dashboard into a working command center.
UI improvements (web/components/bureau-snapshot.tsx):
- Detective tiles are now <Link>s — each navigates to its primary
artefact section in /bureau (Holmes→#hypotheses, Locard→#evidence,
Dupin→#contradictions, Schneier→#hypotheses, Poirot→#witnesses,
Taleb→#outliers, Tetlock→#hypotheses, Case-Writer→#reports). Hover
bg matches the detective's tone color.
- <QuickLaunch /> form inserted right under the tiles.
New <QuickLaunch /> client component:
- Detective dropdown (7 active kinds; evidence_chain not yet exposed
here since it needs a doc_id better picked from the doc page).
- Single input swaps placeholder + aria-label by kind: question for
Holmes, topic for Dupin/Taleb/Case-Writer, hypothesis_id for
Schneier/Tetlock, person_id for Poirot.
- Submits to POST /api/bureau/launch and redirects to /jobs/[id]
via the next.js router.
- Loading state ("queueing…") + error display inline.
POST /api/bureau/launch (web/app/api/bureau/launch/route.ts):
- Same 8-kind validator as the chat tool's request_investigation.
- Auth required when Supabase is configured (triggered_by = user:email).
- Returns { job_id, kind, detective, status_url, eta_seconds }.
DocBureauPanel on /d/[docId] (web/components/doc-bureau-panel.tsx):
- Server component inserted between the doc header and
AnomalyHighlights.
- Surfaces every bureau artefact that touches the doc:
· Evidence whose source_page_id starts with docId/p
· Hypotheses citing any of those evidence_ids
· Contradictions whose chunks[] has any item with this doc_id
· Gaps/outliers with scope.doc_id == docId
· Case reports whose markdown body references docId (filesystem scan)
- Empty state shows "Investigation Bureau — untouched" with a CTA
linking back to the homepage to launch the first investigation.
- When non-empty, header counts total artefacts + links to /bureau
for the full view.
Metadata (web/app/layout.tsx):
- description rewritten from "Investigative wiki of the US Department
of War UAP/UFO archive (war.gov/ufo)" to one that names the bureau
+ the 8 detectives. Affects SERP previews + social-card defaults.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/c/[slug] returned 404 even after the W3.9 web rebuild because the web
container's volume list didn't include the case/ directory the
investigator-runtime writes to. The BureauSnapshot file-listing for
Case reports gracefully fell back to empty, but /c/<slug> can't fall
back: it has to read the markdown.
Fix:
- Mount ${CASE_ROOT:-/data/disclosure/case}:/data/ufo/case:ro (read-only,
same pattern as wiki/processing/raw).
- Set CASE_ROOT=/data/ufo/case env in the web container so the
/c/[slug] page and BureauSnapshot resolve the same path.
Verified live: /c/green-fireballs-sandia now serves HTTP 200 with the
Watson narrative parsed + rendered via MarkdownBody.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes a UX gap the user surfaced: W3.5-3.8 built 8 detectives, 4 new
URL endpoints (/jobs/[id], /h/[id], /c/[slug], /api/h/[id]/red-team)
and a chat tool, but the homepage was unchanged — the bureau was
invisible unless you knew the URL or asked the chat to invoke
request_investigation.
Homepage (web/app/page.tsx):
- Title `▍ war.gov/ufo — Investigative Wiki` → `▍ The Disclosure Bureau`
- Subtitle expanded from "Holmes · Poirot · Dupin · Locard" to all 8
detectives (Holmes · Locard · Dupin · Schneier · Poirot · Taleb ·
Tetlock · Case-Writer)
- New `🔎 bureau` topbar link (gold, between graph/stats and batch)
- BureauSnapshot inserted right after the header
BureauSnapshot (web/components/bureau-snapshot.tsx) — server component:
- 8 detective tiles with role labels (each in its tone color)
- 6 clickable counters (evidence / hypotheses / contradictions /
witnesses / outliers / case reports) — anchor to /bureau#section
- 6 "recent artefacts" columns surfacing the last 3-4 of each kind:
hypotheses with prior→posterior + band + ↳reviewed_by marker,
contradictions with topic + resolution_status, evidence with
Grade badge + verbatim quote, outliers with title + scope.kind,
witness analyses with canonical_name + credibility + verdict,
case reports with slug + link to /c/<slug>
- "Recent jobs" strip linking to /jobs/[id] color-coded by status
- Reports read from /data/ufo/case/reports/ via fs.readdir + stat,
sorted by mtime — no DB round-trip needed for that section
/bureau (web/app/bureau/page.tsx) — full hub:
- Header with full counts
- 7 sections (anchored to homepage counter links): Case reports,
Hypotheses, Evidence, Contradictions, Outliers, Witnesses,
Recent jobs table — each rendering up to 100 rows
- Reports section parses frontmatter from each .md to surface topic
+ n_hypotheses + n_evidence on the card
Runtime fixes batched in:
- Poirot: coerce entity_pk via Number() — node-postgres returns
BIGINT as string by default; writer's Number.isFinite() rejected
it as "person_entity_pk required" (j-edgar-hoover retry path)
- Tetlock: write_calibration rationale cap 600 → 1200 chars. Prompt
still asks ≤ 600 but a 2× slack beats failing the job on honest
analysis. Observed live: Tetlock emitted ~620 chars on H-0003 and
the writer rejected the entire calibration.
- Case-Writer: Promise.all of 5 queries × max_parallel=2 jobs
demanded up to 10 connections against the investigator role's
rolconnlimit=4 → "too many connections for role investigator".
Sequentialized — the LLM call is the hot path, not these queries.
Smoke results visible now on the homepage:
- 3 hypotheses (H-0001/2/3) about green fireballs origin
- 3 contradictions (R-0001/2/3) about color, geographic confinement,
exclusive-green vs multicolored
- 2 evidence cards (E-0002/3) Grade B
- 3 outliers (G-0001/2/3) — including Taleb's deliberate
meteor-shower-camouflage flag
- 1 case report at /c/green-fireballs-sandia (Watson 13.4 KB,
five-act narrative, fully cited)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the bureau from 4 → 8 detectives. All eight run as Bun + claude-CLI
subprocesses against the same Supabase + investigation_jobs LISTEN/NOTIFY
queue, sharing search.ts hybridSearch and writer-side validators that
gate writes against schema + FK.
New detectives:
Poirot (witness_analysis)
- prompts/poirot.md — credibility / access / bias / corroboration /
verdict; uses entity_mentions JOIN chunks to pull 12 chunks per
person; resolves corroboration_refs chunk_ids defensively (accepts
bare cNNNN even when the model emits pNNN/cNNNN).
- INSERT into public.witnesses with W-NNNN naming.
- Tone: purple (#9b5de5).
Taleb (outlier_scan)
- prompts/taleb.md — "surprise is relative to a model"; at most 3
outliers; each requires explicit dominant_model + why_surprising +
what_it_implies; fan-out into public.gaps with scope.kind="outlier".
- Same unscoped-fallback as Dupin (Pass 1 with doc_id, Pass 2 widens
to corpus if hits < 3).
- Tone: yellow (#ffd23f).
Tetlock (calibrate_hypothesis)
- prompts/tetlock.md — honest Bayesian update; emits new_posterior +
Δ + recommended_action ∈ {keep, downgrade, upgrade, supersede}.
- write_calibration UPDATEs public.hypotheses + APPENDS a
"## Calibration history" section to the H-NNNN.md case file
(calibration is append-only — each datapoint matters). Posterior
band auto-corrected to match Tetlock thresholds.
- NO_NEW_EVIDENCE sentinel handled; pure 'keep' with |Δ|<0.005 only
touches updated_at + reviewed_by.
- Tone: teal (#26d4cc).
Case-Writer (case_report)
- prompts/case-writer.md — Dr. Watson assembles all artefacts
(E-NNNN, H-NNNN, R-NNNN, W-NNNN, G-NNNN) into a five-act narrative.
ILIKE filter on topic; doc_id optional scope.
- Larger budget cap (≥ $0.50) + longer timeout for prose generation.
- Writes case/reports/<slug>.md with frontmatter (topic + counts);
no DB table for v0.
- New page /c/[slug] renders the report via MarkdownBody + stat chips.
- Tone: gold (#e0c080).
Hardening across the bureau:
- Sentinel parsing now accepts backticked AND prose-trailing forms
(Holmes NO_HYPOTHESES, Dupin NO_CONTRADICTIONS, Schneier
INSUFFICIENT_HYPOTHESIS, Poirot INSUFFICIENT_TESTIMONY, Taleb
NO_OUTLIERS, Tetlock NO_NEW_EVIDENCE, Case-Writer
INSUFFICIENT_ARTEFACTS). Avoids the failure mode where the model
refuses honestly but the runtime treated it as a parse error
(observed live with Poirot+Hoover identifying the DIRECTOR
false-positive disambiguation issue in entity_mentions).
Chat tool extensions (web/lib/chat/tools.ts):
- request_investigation now accepts 7 kinds. Each routes to its
detective with appropriate validation (hypothesis_id regex,
person_id kebab-case, topic non-empty, doc_id for evidence_chain).
- ETA per kind: Holmes/Dupin 60s, Poirot 45s, Schneier/Tetlock 30s,
Taleb 50s, Case-Writer 180s (longer prose), Locard 30×n_chunks.
UI integration:
- chat-bubble inline card paints each detective in its tone color.
- /jobs/[id] page header swaps name/subtitle/tone per detective;
question label adapts ("Topic" / "Hypothesis under attack" /
"Witness under analysis" / "Topic to outlier-scan" / "Hypothesis
under recalibration" / "Case to assemble").
- job-status-poller renders: case-report link card (gold), outlier
cards (yellow), witness cards (purple) — alongside existing
hypothesis, evidence, contradiction cards.
- /api/jobs/[id] hydrates witnesses (JOIN entities for canonical_name)
+ gaps (with scope JSONB).
- /c/[slug] page reads /data/ufo/case/reports/<slug>.md and renders
with MarkdownBody, frontmatter parsed for stat chips.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the fourth AI detective in the Investigation Bureau runtime: Bruce
Schneier, who attacks an existing hypothesis as a red-team operator.
Runtime:
- prompts/schneier.md — discipline (don't disprove, just attack;
structured output with hidden_assumptions, failure_modes,
alternative_explanations, recommended_tests, verdict_one_sentence;
severity ∈ {low, medium, high}; emit INSUFFICIENT_HYPOTHESIS when
the input is too thin)
- src/detectives/schneier.ts — reads the hypothesis row + evidence
chain (joined via evidence_refs FK), feeds Claude with the
arguments + verbatim quotes, parses strict JSON object
- src/tools/write_red_team_review.ts — UPDATEs hypotheses.reviewed_by
+ updated_at; APPENDS (or replaces if re-reviewed) a structured
"## Red-team review (Schneier · X severity)" section to
case/hypotheses/H-NNNN.md. Caps each list at 5 entries × 240 chars,
validates verdict ≤ 280 chars.
- orchestrator: new `red_team_review` kind dispatching to runSchneier
Chat + UI:
- request_investigation gains kind=red_team_review + hypothesis_id arg
(validated against H-NNNN regex); detective auto-resolves to schneier
- chat-bubble inline card paints Schneier in red (#ff3344)
- /jobs/[id] page swaps title/subtitle/tone per detective; the
"Question" label becomes "Hypothesis under attack" for red_team_review
New /h/[hypothesisId] page (hypothesis dossier):
- Server-rendered from public.hypotheses + public.evidence (joined
via evidence_refs FK + chunk lookup)
- Header: ID + creator + reviewer (highlighted when Schneier has
visited), position as headline, question subtitle, Tetlock band
- Prior + posterior bars with Δ-delta indicator
- Argument grid: argument_for (green) vs argument_against (pink)
side-by-side with [[wiki-link]] auto-linking to source chunks
- Evidence chain: each E-NNNN with Grade A/B/C badge, verbatim
blockquote, link to source page
- Red-team review panel: parses the markdown section in the case
file (severity badge, verdict, 4 bullet panels for
hidden_assumptions / failure_modes / alternative_explanations /
recommended_tests). Empty state when not yet reviewed.
RedTeamRequestButton client component + POST /api/h/[id]/red-team —
authenticated user can trigger Schneier in one click; UI swaps to
"acompanhar" link to /jobs/[id] once queued.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two regressions surfaced in the smoke test that put Dupin from
0/3 contradictions written → 3/3 in the next run.
1. Single-doc scope was too narrow for Dupin's task.
Holmes's question about Sandia returned 4 chunks scoped to one doc,
but Dupin's terser "topic" form yielded only 1. Solution: Pass-1
tries the requested doc_id; if the head is < 2 chunks, Pass-2
widens to the whole corpus. Audit event carries `scope_widened`
so the case-writer can later flag cross-doc contradictions
distinctly. The unscoped retry hit 9 chunks and produced 3
contradictions across 3 different docs.
2. Chunk-block header was ambiguous to the model.
`--- doc-id/p007#c0042 ---` led Claude to parse `chunk_id` as
"p007#c0042" or "p007/c0042" in the JSON output. write_contradiction
then refused the FK lookup with "chunk not found". Fix:
- Explicit `doc_id:` / `chunk_id:` / `page:` lines per chunk
in the rendered block (no slashes/hashes the model can fold).
- Defensive normalizeChunkId() in write_contradiction.ts strips
any pNNN prefix and keeps only the trailing cNNNN — so the
writer is forgiving without losing strictness on the topic +
statement validation.
Smoke now produces (job 6deddf4b):
R-0001 (3 chunks) — Color of the fireball(s) in incident summaries
R-0002 (2 chunks) — Geographic confinement of green-fireball sightings
R-0003 (3 chunks) — Whether the phenomenon was exclusively green or
also red/multicolored
R-0003 connects 3 different declassified documents: the Los Alamos
conference (exclusively-green category), a retrospective document
(red OR green), and Incident 229 (red, blue, yellow — no green).
Real cross-doc contradiction, fully cited.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the third AI detective in the Investigation Bureau runtime: C. Auguste
Dupin, who scans a corpus shortlist for pairs (or small groups) of chunks
that cannot both be true under any ordinary reading.
Runtime:
- prompts/dupin.md — discipline (no contradiction without ≥2 distinct
chunk_ids; reject same-vocabulary near-misses; FEW high-confidence
over MANY weak ones; emit `NO_CONTRADICTIONS` when corpus is silent)
- src/detectives/dupin.ts — hybridSearch with k=18 (more chunks than
Holmes because contradictions emerge from comparing dispersed
claims), strict JSON-array parsing, AT MOST 3 contradictions per call
- src/tools/write_contradiction.ts — validates topic + ≥2 positions
drawn from ≥2 distinct chunks, resolves chunk_pk via DB lookup
(rejects positions citing unknown chunks), INSERTs into
public.contradictions + writes case/contradictions/R-NNNN.md
- orchestrator: new `contradiction_scan` kind dispatching to runDupin;
payload { topic, doc_id?, lang?, context_chunks? }
Chat + UI:
- request_investigation gains kind=contradiction_scan + topic arg;
triggered detective auto-resolves to dupin
- chat-bubble inline card renders dupin in orange (#ff8a4d) to
distinguish from holmes (cyan) and locard (green)
- /jobs/[id] page swaps title + subtitle + tone per detective;
"Question" label becomes "Topic" for contradiction_scan
- /api/jobs/[id] hydrates public.contradictions when outputs[] surfaces
contradiction_ids
- job-status-poller renders ContradictionCard: topic + N positions
(verbatim statements quoted, stance label optional, link to source
chunk) + optional notes panel, with resolution_status badge
(open/resolved/irreconcilable)
R-NNNN shares the contradiction_id_seq slot with relation per
CLAUDE.md naming — same conceptual class (a connection between two
pieces of evidence in tension).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the loop between the chat UI and the Investigation Bureau runtime.
Chat tool (web/lib/chat/tools.ts):
- request_investigation { kind, question, doc_id?, chunks?, claim? }
INSERTs a row in public.investigation_jobs and returns
{ job_id, kind, status, eta_seconds, status_url, detective }.
- kind=hypothesis_tournament → Holmes (1 question → 2-3 rival hypotheses)
- kind=evidence_chain → Locard (1 doc → grade-A/B/C evidence with chain
of custody, default top-5 anomaly chunks)
- Plumbed user.email through ToolHandlerContext so triggered_by audits
the requesting user.
Public job viewer:
- GET /api/jobs/[id] joins investigation_jobs → public.evidence +
public.hypotheses for the IDs surfaced in outputs[]. Returns one
payload the page can render without n+1 round-trips. Strips
triggered_by from the response (it carries the user's email).
- app/jobs/[id]/page.tsx server-renders the case-file shell:
detective lore header (Holmes blue or Locard green), question chip,
scope chip with link back to the document.
- components/job-status-poller.tsx client island that polls every 3 s
while non-terminal, then once on terminal to hydrate evidence +
hypotheses. Renders:
· Phase tracker (queued → running → complete | failed)
· Hypothesis cards w/ prior + posterior bars + Δ delta indicator
+ Tetlock band badge (high/medium/low/speculation)
· Argument-for / argument-against with [[wiki-link]] auto-linking
to /d/<doc>/p<NNN>#<cNNNN>
· Evidence cards w/ Grade A/B/C badge + verbatim blockquote +
bbox crop preview via /api/crop + custody-steps disclosure
· Empty/in-flight panel ("os detetives estão lendo o corpus")
· Failure panel surfacing error + partial outputs
Inline chat-bubble card (components/chat-bubble.tsx):
- ToolTrace.richRender recognises request_investigation results and
renders a detective banner with status + ETA + link to /jobs/[id]
(target=_blank). Error case renders a red strip with the message.
UX flow now: user asks Sherlock a question → request_investigation
queues the job → chat card shows "🔎 Holmes · hypothesis_tournament ·
ETA ~60s" → user clicks → /jobs/<id> live-updates → 60 s later, 2-3
rival hypotheses + their arguments + chunk citations are rendered with
Bayesian update visible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the second AI detective in the Investigation Bureau runtime: Sherlock
Holmes, who builds 2-3 rival hypotheses with calibrated priors + posteriors
against a corpus shortlist.
Pipeline:
1. hybridSearch() grounds Holmes with 8-15 chunks via the same
hybrid_search_chunks RPC the web uses (BM25 + dense + RRF). Default
max_dense_dist=0.55 (runtime favors recall over precision; web's
/api/search/hybrid stays at 0.40 for chat).
2. claude-sonnet-4-6 emits a strict JSON array with position +
argument_for + argument_against + prior + posterior + confidence_band
+ evidence_refs. Citations use [[doc-id/pNNN#cNNNN]] wiki-links.
3. writeHypothesis() validates posterior ∈ [0,1], auto-corrects the
Tetlock band from the posterior (high ≥0.90, medium 0.60-0.89,
low 0.30-0.59, speculation <0.30), checks evidence_refs FK against
public.evidence, INSERTs into public.hypotheses + writes
case/hypotheses/H-NNNN.md.
Discipline guarantees (prompts/holmes.md):
- posteriors across rivals sum to ≈1.0
- no claim without chunk citation
- prefer lower band when ambiguous (anti-inflation)
- declarative one-sentence position, no hedging
- emit `NO_HYPOTHESES` when corpus is silent (refuses to fabricate)
Smoke test (Sandia green fireballs 1948-49):
- H-0001 prior 0.5 → posterior 0.2 (speculation): natural meteoric
- H-0002 prior 0.3 → posterior 0.4 (low): classified weapons / tests
- H-0003 prior 0.2 → posterior 0.4 (low): genuinely unidentified
Bayesian update visible: "natural meteoric" prior dropped 60%; both
rivals climbed. 4 unique chunk citations across the 3 hypotheses.
orchestrator dispatches `hypothesis_tournament` kind via runHolmes;
job marked `failed` if all rivals error, `complete` otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Token consolidation:
- docker-compose web service now reads ${CLAUDE_CODE_OAUTH_TOKEN} directly,
drop the W1-F8 CLAUDE_CODE_OAUTH_TOKEN_FOR_WEB indirection (user feedback:
one var name, no _FOR_WEB suffix).
investigator-runtime claude.ts:
- --system-prompt silently dropped by CLI v2.1.150 for multi-KB prompts;
inline the system content into the user prompt with a separator
(mirrors scripts/reextract/run.py pattern).
- Multi-line prompts via positional -- broke ("Input must be provided …");
pipe via stdin instead.
- --allowedTools "" is rejected; when no tools wanted, omit it and explicitly
--disallowedTools the writer/reader set so the model can't reach for any.
investigator-runtime locard.ts:
- Log the raw response (first 600 chars) to container stderr — saved hours
of debugging when the writer rejected.
- Grade fallback: when Locard omits `grade` but provides custody_steps,
infer the highest grade that fits (≥3 → A, ≥2 → B, ≥1 → C).
investigator-runtime write_evidence.ts:
- Filter related_hypotheses entries with empty/null hypothesis_id silently
(Locard sometimes emits [{}] when it knows no link yet) instead of
failing the whole write.
Migration 0006_investigator_serial_sequences.sql:
- BIGSERIAL on the 7 investigation tables created auto-sequences
(evidence_evidence_pk_seq etc) that 0004 forgot to GRANT to the
investigator role. Without those grants every INSERT failed with
"permission denied for sequence …". Grant USAGE/SELECT/UPDATE on each
auto-seq.
Verified live: Locard wrote E-0002 + E-0003 from real Sandia chunks
(green fireball Feb 1949; cobalt particle analysis). Grade B, confidence
high, custody chain of 3 steps with honest gaps. Cost $0.09 for both,
~70s wall.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scanned docs are messy — duplicate transcriptions (typed + handwritten),
two classification variants of the same narrative, OCR noise, repeated
banners. The doc page showed raw chunks, so everything appeared twice.
40_reading_version.py generates ONE clean, deduplicated, well-structured
bilingual Markdown reading version per doc (Sonnet): merges duplicate versions
without losing unique lines, drops page furniture, formats transcripts as
dialogue. Faithful — invents nothing; redactions kept as markers.
/d/[docId] now defaults to a "📖 leitura" tab rendering this clean version,
with "🔍 trechos · scan original" preserving the faithful per-chunk + per-page
scan view. reading.md lives in raw/<doc>--subagent/ alongside the chunks.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
~414 chunks have ufo_anomaly_detected=true but no type/rationale (extraction
left them null), so the flag rendered "UAP flag: anomaly —" with a dangling
em-dash. Build the label from the parts that exist: fall back to "anomalia"
for a missing type and omit the "—" when there's no rationale. The flag still
shows (the chunk genuinely contains UAP content), just without the noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
29 image chunks have an empty bbox {}, and `fm.bbox ?? default` doesn't catch
an empty object, so the crop URL got w=undefined → /api/crop 400 → broken-image
icon. Now validate bbox (w/h > 0); without it, render the image's textual
description instead of requesting an impossible crop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root-cause fix for "search returns garbage for absent terms". The hybrid RPC's
dense branch always returned its k nearest vectors regardless of distance, so a
query for a term not in the corpus (e.g. "varginha") surfaced unrelated chunks.
The cross-encoder reranker would filter these but costs 18-62s on CPU —
unusable for interactive search.
Add max_dense_dist (default 0.40) to hybrid_search_chunks: dense neighbours
beyond that cosine distance are dropped server-side. Calibrated from measured
distances — strong semantic match ~0.12-0.20, no real match ~0.46-0.53. BM25
full-text still matches literal terms; the reranker becomes opt-in refinement.
Verified live: varginha/abducao → 0, disco voador/roswell → relevant, all <1s.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The hybrid_search RPC always returns up to recall_k dense neighbours, so a
query for a term absent from the corpus (e.g. "varginha") returned its 12
nearest vectors — irrelevant chunks like PAGE_NUMBER "1". Two bugs:
the reranker was skipped whenever results <= top_k, and there was no relevance
floor.
Now always run the cross-encoder reranker (BGE-reranker-v2-m3, normalized
sigmoid) and drop hits below 0.02. Verified: "varginha" → 0 results;
"roswell"/"tic tac"/"disco voador" → relevant hits on top (reranker cleanly
separates 0.0001 garbage from 0.03-0.27 matches).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The reprocess pass added chunks for pages beyond the original total_pages but
never updated the field, so doc-page navigation thought docs ended early
(jumped to next document mid-doc) and the page counter was wrong. Now bump
total_pages to the real max chunk page on each integration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Doc page (/d/[docId]/[page]) gains prev/next navigation bars (top + bottom):
within a doc it steps page-by-page; at the first/last page it jumps to the
previous/next document. Replaces the disabled-at-boundary links.
Indexer tooling for the VPS repopulation:
- 30-index-chunks-to-db.py: add --no-embed (fast BM25-only index; vectors
backfilled separately) so the app is usable in minutes, not hours of CPU
embedding.
- 57_load_relations_from_json.py: load typed relations into public.relations
from reextract structured fields (deterministic ids, no fuzzy guessing).
- 58_backfill_embeddings.py: async pass to fill chunks.embedding (NULL rows)
via the embed-service.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add reextract pipeline (scripts/reextract/) that rebuilds doc-level entity
JSON from Sonnet-vision chunks via Opus, replacing the noisy per-page
extraction. Add synthesize scripts to regenerate wiki/entities from the 116
_reextract.json (30), aggregate missing page.md from chunks (31), and reprocess
805 pages the doc-rebuilder agent dropped on context overflow (32). Add
maintain scripts 43-56 for chunk-page sync, dedup, generic-entity marking, and
typed relation extraction.
Web: wire relations API + entity-relations component; entity/timeline/doc
pages consume the rebuilt layer.
Note: raw/, processing/, wiki/ remain gitignored (bulk data managed
separately); the 116 reextract JSONs and 7,798 rebuilt entity files live on
disk only. The 27 curated anchor events under wiki/entities/events/ are
preserved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The corpus had two parallel reverse-reference signals: the wiki/pages
entities_extracted blocks (Haiku page-level) and public.entity_mentions
(Sonnet chunk-level, ILIKE-matched). The entity page only consulted the
DB, so it showed "0 menções" for thousands of entities that were anchored
in pages or in cross-entity links the DB never indexed.
Resolved by collapsing all signals into the YAML frontmatter, which is
now the single runtime source for entity metadata.
scripts/maintain/42_sync_entity_stats.py walks every entity and writes:
mentioned_in: [...] # consolidated page refs
total_mentions: max(db, pages)
documents_count: max(db_docs, distinct page docs)
signal_sources:
db_chunks: int
page_refs: int
cross_refs: int
signal_strength: strong | weak | orphan | unverified
referenced_by: [[class/id]] # cross-entity backlinks
Outgoing wikilinks (e.g. OBJ.observed_in_event → EV) count toward the
entity's own cross_refs so anchored-but-not-mentioned entities don't
register as orphan.
OBJ canonical names like "7m long, 1.3m high, two rocket motors,
smooth flow, rotary drive null UAP (OBJ-EV1945-PEYERLSHOTDOWN-01)"
are rewritten to "Peyerl shot down UAP" derived from observed_in_event,
preserving the full description as an alias. --fix-obj-names did this
for every OBJ-* with >80 char canonical_name.
Default behaviour is conservative: --archive-only-junk archives only
single/double-char names and pure-numeric noise. Everything else stays
on disk with signal_strength marked, so the user can review later.
web/lib/retrieval/entity-pages.ts swapped from db-first to yaml-first.
The /e/[cls]/[id] page now reads counts straight from YAML and renders
a "força do sinal" badge with the per-source breakdown. Orphan entities
get a banner explaining they have no cross-references.
DB is still queried for ONE thing: the chunk text for preview cards on
the entity page, so we don't re-parse 21k markdown files on every render.
First-pass result: 9020 strong / 14520 weak / 10814 orphan; OBJ-EV1945-
PEYERLSHOTDOWN-01 now reads "Peyerl shot down UAP · fraca · 1 backlink"
in the live UI.
middleware.ts now checks profiles.role on every /admin/* request and
returns a plain 404 (not a redirect) for anonymous users and any
authenticated user without role='admin'. The 404 wording matches a
non-existent route, so we don't leak the existence of an admin area.
gutomec@gmail.com promoted to admin in the live DB.
openrouter.ts chat now collects artifacts silently during the tool-call
loop instead of streaming them to the SSE. After the model writes its
final prose (and after the forced-synthesis pass if needed), we scan
the assembled text for [[doc-id/p007#cNNNN]] citations and emit ONLY the
artifacts referenced. Duplicates are deduped by chunk_id. The persisted
citations column on messages now stores the filtered set too, so old
sessions reload with the same focused card grid.
Before: every hybrid_search hit (up to 6 per call × 5 calls = 30+ citation
cards plus crop images) flooded the chat regardless of what the model
ended up using. After: only the chunks actually woven into the answer.
Two bugs combined to make the chat reply with only cards and no prose:
1. SQL trigger rollup_session_stats was failing with "column reference
total_cost_usd is ambiguous" because the UPDATE on public.profiles had
a FROM public.chat_sessions clause and both tables expose that column.
Persistence of every user message died at this point — sessions were
created in the DB but had message_count=0 forever. Applied SQL fix
that qualifies columns with p./s. aliases (production DB updated;
ALTER FUNCTION run live, not yet codified in a migration file).
2. The free-tier model (nemotron-3-super:free) spent all 5 tool-loop
turns on hybrid_search calls and never wrote any prose, returning
content_len=0. Added a forced-synthesis pass in openrouter.ts: when
the loop exits with empty assembledText but the model did call tools,
we send ONE final turn with tools omitted from the request payload
and a user message instructing the model to answer in 3-8 sentences
citing chunks. openrouterStreamCall now accepts a `withTools` opt
so the synthesis call can disable tool calling entirely.
Verified end-to-end with the actual user query "O que os astronautas
viram? Quem foi que viu?" on /d/nasa-uap-d6-apollo-17-...:
- content_len: 0 → 947 chars (real synthesis citing Schmitt)
- artifacts: 44 preserved
- assistant message persisted with tool_calls + citations columns
34 of 116 docs were generated with 0-based PNG numbering (p-000.png …
p-008.png) but the Sonnet chunks reference 1-based page numbers in their
YAML frontmatter (page: 9 means the 9th sheet of paper). The /api/crop
handler built p-009.png and got a 500, the browser's Next/Image surfaced
400, and the chunk rendered as a black box on screen.
Fixes:
- web/app/api/crop/route.ts: try p-NNN.png first, fall back to
p-(NNN-1).png if the 1-based file is missing. Cheap insurance for any
doc that comes in with the old convention.
- scripts/01-convert-pdfs.sh: previously printf '%03d' "$num" with $num
starting at 0 (e.g. "008") raised "invalid number" because Bash
parsed it as octal. Wrap with $((10#$num)) to force decimal — this
was silently corrupting page sequences and producing gaps like p-001
... p-008, p-011 (missing p-009/p-010).
- All 34 affected docs re-converted from PDFs with the patched script;
every directory now has continuous 1-based PNGs.
- /processing/png/ rsync'd to VPS, web redeployed.
Smoke: /api/crop?doc=doc-341-…&page=9&… now returns 200 image/webp
instead of 500. Tested in browser: chunk c0026 (diagram, p9) renders
the real engineering diagram.
Fase 3 onda 2 — entity synthesis at scale:
- scripts/synthesize/20_entity_summary.py: queries DB for entities with
total_mentions ≥ threshold + top-K verbatim chunk snippets via
entity_mentions JOIN, prompts Sonnet (Holmes-Watson voice, bilingual),
writes narrative_summary EN+PT-BR + summary_status=synthesized.
Ran on 187 candidates (mentions ≥ 20) → 158 OK · 1 err · 29 skipped (no
snippets). Combined with anchor curation: 20 curated + 158 synthesized
= 178 entities with real narrative (vs 0 a day ago).
Fase 4 — chat with typed artifacts + persistence:
- lib/chat/agui.ts: AG-UI v1 typed Artifact union (citation, crop_image,
entity_card, evidence_card, hypothesis_card, case_card, navigation_offer)
alongside the existing event types.
- lib/chat/tools.ts + openrouter.ts: hybrid_search emits up to 6
citation + crop_image artifacts per query. Provider collects them and
returns in done.artifacts so the route can persist.
- api/sessions/[id]/messages: persist artifacts to messages.citations.
- components/chat-bubble.tsx: ArtifactCard renders inline cards (citation,
crop_image, entity_card, navigation_offer) for streamed and persisted
messages. activeId now persisted in localStorage so navigation between
pages keeps the same conversation. New sessions are lazy (only when user
has zero). loadMessages hydrates tools + artifacts from server. CRUD UI:
rename (✎) + archive (🗑) buttons per session in the list.
Home search:
- doc-list-filters: input now fires hybrid_search (rerank=0 for speed)
in parallel with the local title filter; chunk hits render above the doc
grid with snippet + score + classification.
- api/search/hybrid: accept ?rerank=0 to skip the cross-encoder (1.3s vs 60s).
Auth flow:
- infra: SMTP_HOST=mail.spacemail.com:587 + DMARC published; mail now lands
in inbox. GOTRUE_MAILER_AUTOCONFIRM=false (real email verification).
- kong.yml: proxy /auth/callback on api.disclosure.top → web:3000 so PKCE
email links don't 404 at the gateway.
- web/app/auth/callback: handle both ?code= (OAuth) and ?token=&type=
(PKCE); redirect to the public site host before verifyOtp so the session
cookie lands on the right domain.
Audit deliverables:
- .nirvana/outputs/disclosure-bureau/.../systems-atelier/: 5 docs (code
analysis, tech debt, discovery brief, system arch, 5 ADRs) authored by
sa-principal that produced this roadmap. Kept in-tree for traceability.
User reported a ~ Google Gemini bill against an expected ~ budget.
Permanent ban on Gemini for this project — all LLM inference goes through
claude -p --model sonnet (CLAUDE_CODE_OAUTH_TOKEN) or OPENROUTER_API_KEY as
fallback. The 10 scripts in scripts/ that import google-genai/-generativeai
are moved under scripts/_archived-gemini/ with a DO-NOT-RUN README. Project
memory updated (feedback-no-gemini-ever.md) and the older reference note that
recommended gemini-3.1-pro-preview is revoked.
- scripts/03-dedup-entities.py: stop emitting placeholder narrative ("Stub. Will
be enriched in Phase 7"); write summary_status=none + null fields instead.
- scripts/maintain/41_strip_stubs.py: idempotent migration that cleaned the
22,096 entity .md files (now zero stub strings in wiki/).
- scripts/synthesize/01_anchor_events.py: curated 20 anchor UAP events
(Roswell, Nimitz Tic-Tac, Phoenix Lights, Operação Prato, AATIP, etc.) with
bilingual Holmes-Watson narrative via claude -p --model sonnet
(CLAUDE_CODE_OAUTH_TOKEN). All summary_status=curated, confidence=high.
- web/api/timeline + timeline-view: filter narrative-less events by default,
render "curado" badge for hand-vetted ones, drop the date display alone.
- CLAUDE-schema-full.md: document the summary_status enum and the four states.
- docker-compose.yml: SMTP_HOST=mail.spacemail.com configured;
GOTRUE_MAILER_AUTOCONFIRM flipped to false (real email confirmation working).
- .nirvana/outputs/.../systems-atelier/: 5 deliverables of the architecture
audit that produced this roadmap.
