Case bodies hold EN and PT-BR scenes interleaved (## I. EN, ## I. PT, ...),
but the reader rendered the whole body, so an EN visitor saw every scene
twice — once in English, once in Portuguese — and vice-versa.
Add pickLocaleBody(): split by section-number occurrence (first = EN, second
= PT-BR, the narrator's mandated order), with language sniffing only as a
fallback for unpaired/unnumbered sections, and full-body fallback when a body
can't be split (English-only legacy files). Wire it into the case reader
(body, lead, OG description) and the homepage card openings.
Also fix extractBody: it discarded the English H1 by treating the second
(PT-BR) H1 as "the first real heading". Future narrations keep both titles.
Verified live: d44 and doc-65 render EN-only under locale=en and PT-only
under locale=pt-br. Bulk-checked 89 files: 80 clean, 8 structurally clean
(source-language verbatim quotes retained per spec), 1 English-only legacy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fetchDocChunks took the first N chunks by order_global. On long files the
opening chunks are cover pages, routing slips, classification stamps and
redaction boxes — administrative front matter, not narrative. A 1001-chunk
FBI dossier (doc-65 section-7) handed the narrator 6 form_fields + 4
redaction_blocks + 2 letterheads and only 3 prose paragraphs, so it
refused with INSUFFICIENT_ARTEFACTS — burying first-person testimony
(e.g. the Flatwoods photographs) that sat deeper in the file.
Now select the richest substantive chunks (by content length, redaction
boxes last), then re-sort the winners into reading order for narrative
coherence. Validated: section-7 goes from 3 prose paragraphs to 22
(avg 1926 chars). 32 wrongly-refused rich documents re-queued.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
generateMetadata still derived the page <title>/OG title from the generic
auto frontmatter topic ("Dow Uap D44 …"). Search engines, link cards, and
AI crawlers show that string, so it must be the human magazine headline the
narrator wrote as the body H1. Mirror the page component's H1 extraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A per-document case_report grounded itself via hybridSearch keyed on the
document's auto-derived topic ("Fbi Photo B20", "Doc 59 214434 …"). That
garbage topic has no semantic neighbours, so the dense gate rejected every
chunk and the narrator saw zero scenes — skipping 61 of 75 batch documents
that in fact had dozens of embedded chunks.
When doc_id is set, fetch the document's own substantive chunks in reading
order (fetchDocChunks) instead of searching the corpus. Corpus-wide topic
reports still use hybridSearch.
Verified post-deploy: fbi-photo-b19 0→10 scenes, doc-65 →24, dow-uap-d44 →11.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User wants every case in the archive documented in the narrated format.
Queued a case_report for all 122 documents (slug = doc_id, scoped to that
doc's chunks). The LISTEN worker drains 2 at a time (~5h, ~$18 OAuth).
5 reports already existed; ~122 new jobs queued, skipping any already
queued/running/complete by slug.
Title rendering: the batch seeds `topic` from the auto-derived
canonical_title ("Doc 18 100754 General 1946 7 Vol 2"), which is ugly as a
headline. The narrator writes a proper magazine H1 in the body, so:
- /c/[slug]: extract the two body H1s (EN + PT-BR), use the locale one
as the headline, strip both leading H1s from the rendered body so the
title isn't duplicated. Falls back to frontmatter topic.
- case-library.tsx + featured-case.tsx: same H1 extraction for card
titles, so /bureau and the homepage show real titles not "Doc 18…".
- featured-case: pin the flagship (green-fireballs-narrative, which has
a curated hero painting) to the featured slot instead of most-recent,
so the batch's random completions don't take over the homepage hero.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ran Sun-Tzu briefs + case-writer for the 4 corpus-eligible iconic cases.
Results:
- Kenneth Arnold 1947 → /c/kenneth-arnold-1947 (brief B-0002, 275s)
- Mantell 1948 → /c/mantell-1948 (brief B-0003, 282s)
- Roswell 1947 → skipped (INSUFFICIENT_ARTEFACTS, thin corpus)
- Maury Island 1947 → skipped (INSUFFICIENT_ARTEFACTS, thin corpus)
The two that landed got bold enthusiast-grade closers woven from the
Sun-Tzu strategic brief:
- Kenneth Arnold: "O objeto que descreveram: 6 metros de largura, cor
de chama nas laterais, 75 pés acima do Rio Snake a 1.000 milhas por
hora. O governo guardou esse relato. O que concluiu sobre as
características físicas que ele continha permanece ausente destes
arquivos."
- Mantell: "Qualquer que fosse a coisa no céu sobre o Kentucky em
janeiro de 1948, a Força Aérea a registrou como 'disco voador' e a
deixou assim."
iconic-cases.ts: kenneth-arnold-1947 + mantell-1948 links now point to
their /c/ narratives instead of /e/ entity pages. Hero art copied to the
slug filenames so the case pages render full-bleed illustrations.
Roswell + Maury Island keep /e/ links (no narrative — corpus too thin).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The enriched prompt (scenes + verbatim + anomalies + witnesses + Dupin
tensions + Sun-Tzu brief) produces a longer bilingual narrative. At 300s
the claude CLI timed out mid-generation (job 945c14f2 failed at 310s).
Raised the floor to 600s; reprocess then completed in 251s with a bolder
closer reflecting the strategic brief.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enthusiast-grade intrigue, not skeptic framing. When the corpus disagrees
with itself — one witness says the fireball was green, another says red;
one report dates an event, another contradicts it — that is suspense
material, not a debunk. The narrator now receives a "Tensions in the
record" section (Dupin's contradictions for the topic, top 4) and is told
to use them to build the question the record itself could not settle.
- New ContradictionRow type + renderTensions() (bilingual, cites chunks)
- Query: contradictions matching topic ILIKE, top 4 by recency
- buildPrompt gains a tensions param + "## Tensions in the record"
section framed as intrigue ("these are intrigue, not failure")
Deferred deploy: the bulk entity-enrichment run is still finishing the
organization class inside the live investigator container; rebuilding now
would kill it. Will deploy this + reprocess the flagship green-fireballs
case_report once the bulk completes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User: "quero que o usuário possa escolher o idioma na navbar, e então leia
o site todo em seu idioma."
The locale infrastructure already existed (cookie + getLocale() + every
page renders bilingual). The toggle only lived as a subtle floating
control bottom-left. This surfaces it in the navbars where users expect
it, and confirms whole-site language switching.
locale-toggle-client.tsx
- New variant="navbar": gold-amber pill matching the magazine theme
(PT | EN, active = solid gold). Retro green variant kept for the
floating fallback.
- Guard: clicking the already-active language is a no-op.
- aria-pressed on each button for a11y.
locale-toggle.tsx: LocaleToggle gains variant passthrough.
site-header.tsx: <LocaleToggle variant="navbar"> added to the main nav
(homepage + /sightings /witnesses /objects /locations /operations
/documents).
bureau-nav.tsx: optional `locale` prop renders the same toggle pushed
to the right (ml-auto) on /bureau, /c/[slug], /h/[id], /jobs/[id].
All four callers now pass locale (getLocale added to h + jobs pages).
Cookie-based: switching sets `locale` cookie + router.refresh(); the
server re-renders every component through getLocale(), so the entire
page — headers, hero, summaries, case narratives, entity cards — flips
language in one click. The floating bottom-left toggle stays as a global
fallback for pages without a navbar (/d, /e, /search, /graph).
Also this session: Sun-Tzu smoke succeeded on the refreshed
CLAUDE_CODE_OAUTH_TOKEN — brief B-0001 written for the green fireballs
("bolas de fogo verdes confinadas ao corredor nuclear... cor espectral
e trajetória quase horizontal que os próprios analistas consideraram
irreconciliáveis com meteoros naturais", 3 pillars). Bulk entity
enrichment resumed in the background.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A11y:
- Skip-link in <body> (focus-visible only) that jumps to #main.
Bilingual ("Skip to content" / "Pular para o conteúdo").
- <main id="main"> landmark wrapping the homepage body.
- prefers-reduced-motion media query disables the hover-scale + image
transitions for users with vestibular sensitivity.
- Skip-link styled with high contrast (gold-on-dark) + outline on
keyboard focus.
Performance:
- HeroBanner background image: fetchPriority="high" + explicit
width/height (1600x900) for zero CLS.
- FeaturedCase image: fetchPriority="high" + 1280x720 to prevent
layout shift while the 2.7MB painting loads.
- IconicCases tiles already have loading="lazy".
- prose blockquote: overflow-wrap:anywhere so verbatim quotes don't
bust the mobile viewport.
Open Graph:
- app/layout.tsx default OG image set to the green-fireballs
painting (any page without its own image card inherits this).
- app/c/[slug] OG image is the case's editorial illustration when
one exists. WhatsApp, Twitter, Telegram, Slack, ChatGPT search
all pull this when the link is shared. 2000x1125 for the
"summary_large_image" twitter card.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8 hand-picked UFO incidents that any enthusiast recognises:
- Kenneth Arnold 1947 (the genesis — "flying saucers" coined)
- Roswell 1947 (the original Army press release)
- Maury Island 1947 (Puget Sound, slag drop, FBI plane crash)
- Mantell 1948 (first known UFO casualty)
- Rendlesham Forest 1980 (USAF security police, deputy commander tape)
- Phoenix Lights 1997 (V-formation across Arizona, governor's reversal)
- Nimitz Tic-Tac 2004 (USS Nimitz F/A-18, gun-cam released 2017)
- Green Fireballs Sandia 1948 (copper salts, nuclear-site overflights)
Each case has:
- Bilingual hand-written editorial blurb (PT-BR + EN, no LLM)
- Painterly editorial hero illustration at 2K
- Year + tag chips (military / civilian / modern / early / naval /
aviation / mass-sighting)
- Link to /e/events/<id> when indexed in corpus, /search?q=... when not,
/c/<slug> for the green-fireballs narrative case
Components:
lib/iconic-cases.ts — IconicCase type + ICONIC_CASES editorial list
components/iconic-cases.tsx — magazine grid: first two as wide hero pair,
rest as 3-up tiles below. Hover scale on images, gradient overlays,
aspect-16/11 hero + 16/10 compact, lazy-loaded.
app/page.tsx — inserted between <FeaturedCase /> and <PortalGrid />
4 new hero illustrations generated this session:
- iconic-nimitz-tic-tac-2004.png (Nano Banana Pro)
- iconic-phoenix-lights-1997.png (Nano Banana Pro)
- iconic-rendlesham-forest-1980.png (gpt-image-1.5 via Codex)
- iconic-mantell-1948.png (Nano Banana Pro)
Per user direction: mixed generators (Nano Banana primary, Codex
co-pilot) so the homepage has stylistic variety while keeping the
painterly editorial register.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sun-Tzu (silent backend) — builds the strongest pro-anomaly brief the
corpus supports for any topic. Bilingual JSON: thesis + 2-4 pillars
(each with claim + citation-backed support) + honest residual
unexplained clause. NEVER surfaced reader-facing.
Migration 0009 (apply as supabase_admin):
public.pro_anomaly_briefs brief_pk BIGSERIAL PK
brief_id B-NNNN unique
topic + topic_pt_br
thesis + thesis_pt_br
pillars JSONB
unexplained + unexplained_pt_br
doc_id, job_id, created_by, created_at
+ brief_id_seq sequence
+ GIN trigram indexes on topic + topic_pt_br
+ RLS policies (investigator INSERT, public SELECT)
+ GRANTs on seq + table to investigator
prompts/sun-tzu.md
"Adversarial strategist who plays the pro-disclosure side with the
same rigour a red-team plays skeptic" — single thesis, 2-4 pillars,
honest residual. Every claim cites a chunk. No fabrication from
training-time knowledge. Output INTERNAL — case-writer pulls it.
Bilingual mandatory. NO_STRONG_CASE sentinel when corpus is thin.
detectives/sun_tzu.ts
Grounds with hybridSearch top 18 chunks, calls Sonnet, parses
JSON strict, calls writeProAnomalyBrief.
tools/write_pro_anomaly_brief.ts
Validates 2-4 pillars with bilingual claim+support, requires at
least one [[wiki-link]] citation per pillar, INSERTs.
orchestrator: new kind "anomaly_brief" dispatches Sun-Tzu.
Case-writer integration (detectives/case_writer.ts):
- Pulls most recent matching brief via ILIKE on topic or doc_id.
- Renders brief as a separate prompt section labelled
"Strategic brief (internal — do NOT cite or attribute)".
- Instructs the narrator to weave the thesis as a quiet through-
line, use pillar facts in scenes, let the unexplained clause
inform the closing paragraph. Forbidden to name "the analyst",
say "a brief argues", or use the words "thesis"/"pillar"
explicitly. Translate it into prose.
Entity hero illustrations:
- 3 painterly editorial illustrations generated via Nano Banana
Pro at 2K, stored under /data/disclosure/processing/case-art/:
* EV-1947-06-24-kenneth-arnold-sighting.png — cockpit POV of
Arnold in a CallAir A-2 over Mount Rainier, 9 chevron disc
objects in formation, 1947 Life-magazine register.
* EV-1947-07-08-roswell-incident.png — debris field in NM
desert, USAAF officer in 1947 uniform examining foil
fragments, period staff car.
* EV-1947-06-21-maury-island-incident.png — wooden patrol
boat on Puget Sound, 6 doughnut craft hovering, one
shedding glowing slag, Harold Dahl + son + dog watching.
- app/e/[cls]/[id]/page.tsx: full-bleed editorial hero replaces
the old gradient header card when an illustration exists for
that entity_id. Title sits over the painting with gradient
overlay. "Ilustração editorial" chip in the top-right.
Quota note: Claude OAuth still rate-limited as of this commit, so
Sun-Tzu hasn't been smoke-tested in production. Code is shipped and
ready; first brief will land when the weekly quota refreshes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hero illustration:
- Painterly 16:9 editorial illustration generated via Nano Banana Pro
for the featured case (green-fireballs-narrative): late-1940s
desert night, vivid emerald fireball over silhouetted Sandia mesas,
1948-era state-police sedan parked on US 66 shoulder with an
officer in period uniform looking up, faint green glow on his
face. Sandia Base 5 Miles roadsign. New Yorker-cover painterly
register, NOT photorealistic, NOT sci-fi.
- Stored at /data/disclosure/processing/case-art/<slug>.png, served
through the existing /api/static/processing/ route. 2.7MB at 2K.
- components/featured-case.tsx: prefers the illustration over the
declassified-page thumbnail when present. Tags it "Editorial
illustration" / "Ilustração editorial" so the reader knows it's
not a photograph.
- app/c/[slug]/page.tsx: full-bleed editorial hero at the top of
the article when an illustration exists for the slug. Title sits
on the image with gradient overlay; "Ilustração editorial" chip
in the top-right corner labels the art honestly. When no
illustration exists the page falls back to the plain title header.
Sitemap fix:
- Added export const dynamic = "force-dynamic" + revalidate = 3600
to app/sitemap.ts. Without these Next.js statically generated the
sitemap at build time, when the DB and case-files volume were
unreachable from the build container — which is why production
was serving only the 9 static URLs instead of ~3000.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GEO/SEO surface area:
app/robots.ts (new — Next.js dynamic robots)
Explicitly ALLOWS major AI crawlers: GPTBot, OAI-SearchBot,
ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot,
Perplexity-User, Google-Extended, Applebot-Extended, CCBot,
DuckAssistBot, YouBot, Bytespider, Amazonbot. The site exists to
be cited by LLMs answering UAP/UFO questions — we want them in.
/api/admin/, /admin/, /auth/ disallowed for everyone.
app/sitemap.ts (new — Next.js dynamic sitemap)
Lists 9 top-level routes + every /d/<doc> + every /c/<slug> from
the filesystem + up to 500 entity URLs per class
(event, person, uap_object, location, organization),
sorted with summary-enriched entities first. ~3000 URLs total at
current corpus size. lastModified honours summary_generated_at so
crawlers re-index when entities are re-enriched.
app/c/[slug]/page.tsx (rewritten — magazine reading view)
- generateMetadata: per-case title, description (auto-extracted
from the locale-preferred lead paragraph), canonical URL,
hreflang alternate, OpenGraph article type with publishedTime,
Twitter card.
- JSON-LD Article schema embedded at end of page: schema.org
Article + Organization publisher + inLanguage + isAccessibleForFree.
This is what makes the case appear as a citable source in
Google AI Overviews / Perplexity / ChatGPT search.
- Reading view rewritten: display-serif headline (Fraunces), italic
blockquotes with gold accent, prose-typography styling, no more
detective stats line, no more "written by case-writer@detective"
attribution. Locale-aware: PT-BR pulls topic_pt_br + lead in PT,
English mirror.
tailwind.config.ts
+ @tailwindcss/typography plugin
+ font-display family wired to var(--font-display) (Fraunces)
package.json
+ "@tailwindcss/typography" devDependency
Phase 3A note: bulk entity enrichment hit Claude OAuth weekly quota mid-run.
6 events + 3 uap_objects landed bilingual summaries before the quota
exhausted. UI gracefully splits enriched vs bare entities so /sightings
shows the magazine-grade cards (Kenneth Arnold 1947, Roswell, Maury Island,
Joseph Perry 1960 lunar photo, Civil Defense Director 1966, etc.) on top
of a compact table of the rest. Re-run when quota refreshes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Today /sightings, /witnesses, /objects, /locations and /operations show
a name + mention count and nothing else. After this each row carries a
60-100 word bilingual narrative summary written from the chunks where
the entity actually appears.
Migration 0008 (apply as supabase_admin):
public.entities +summary_en TEXT
+summary_pt_br TEXT
+summary_generated_at TIMESTAMPTZ
+summary_model TEXT
+summary_status TEXT
CHECK ('pending'|'ai_generated'|'curated'|'refused')
+ index on summary_status
+ GRANT UPDATE (summary_*) ON entities TO investigator
+ new policy entities_investigator_update_summary (RLS UPDATE for
investigator role)
Enrichment script (investigator-runtime/scripts/enrich_entity_summaries.ts):
- Per-class config (chunk_k, min_mentions, max_per_class)
- Path A: entity_mentions JOIN chunks (high-precision linker)
- Path B (fallback): hybridSearch on canonical_name + aliases when
entity_mentions returns zero. This is what unlocked Kenneth Arnold
and similar entities — their wiki YAML has high total_mentions
counted from frontmatter mentioned_in[], but the entity_mentions
extractor was silent because the matches came from the wiki text,
not the OCR chunks.
- Sonnet 4.6 via OAuth Max, ~$0.04 per entity, ~$10 for the full
260-entity bulk run.
- INSUFFICIENT skip when chunks can't sustain a 60-word summary —
refused entries get summary_status='refused' so they're not retried.
UI uplift:
- lib/retrieval/entity-pages.ts: getEntityCore now prefers the DB
summary (ai_generated or curated) over wiki YAML narrative.
- components/entity-list-page.tsx:
* SELECT now pulls summary_en, summary_pt_br, summary_status
* Sorted with summary-enriched rows first (so the magazine grid
lands on quality content immediately)
* MagazineGrid: 4-line summary preview replaces aliases line
* CompactGrid: enriched rows render as full editorial cards,
bare rows fall back to a compact table below
Smoke results:
- Kenneth Arnold sighting: "On June 24, 1947, pilot Kenneth Arnold
reported sighting unidentified objects over the Pacific Northwest,
and the account spread worldwide. It set off a run of similar
reports: County Commissioner Crankes saw comparable objects after
Arnold's account reached the press, and United Airlines pilot
Emil H. Smith spotted flying discs on July 4 during a routine
flight out of Boise, Idaho..."
- Roswell Incident: includes Colonel Corso's 1997 book + the 1995
GAO finding that radio messages from Oct 46–Feb 47 were destroyed
+ Senator Strom Thurmond's foreword. Real magazine-grade content.
Background bulk run kicked off across all 5 classes (event,
uap_object, person, location, organization) — populating live as
the homepage rebuilds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User: "shouldn't mention the names of the mind-clones, should merge all
analyses and write like a best-seller author would, about what happened."
Voice rewrite (prompts/case-writer.md):
- Reference voices: Erik Larson, Sam Kean, John McPhee, Mark Bowden.
Plainspoken non-fiction, scene-driven, fascinated.
- One narrator. NEVER say "Sherlock Holmes argues" / "Sun-Tzu builds
the case" / "the team concluded". No internal-process names reach
the reader.
- Hook the first paragraph. Open in a scene with a date, place, and
person doing something specific. NOT "This case investigates..."
- Show, don't argue. Verbatim quotes stay source-language in
blockquotes; the narration around them is the narrator's voice.
- Every claim cites a chunk with [[doc-id/pNNN#cNNNN]].
- Forbidden ceremony: "In summary…", "Em suma…", "Ultimately…",
"It is worth noting…", detective names, probability tables,
hypothesis tournaments.
- The honest unknown is the subject, not a failure: "Whatever was in
the sky over Sandia in December 1948, the government never said."
- 4-6 numbered scenes, each title-cased specifically ("The Green
Sphere Over Highway 60" not "Background").
- Bilingual EN + PT-BR per CLAUDE.md §3 — sections alternate, no
mid-paragraph language mixing.
- Refusal: emit INSUFFICIENT_ARTEFACTS rather than padding when the
corpus is thin.
Raw-material pipeline (src/detectives/case_writer.ts):
- hybridSearch(topic, lang, top_k=18) gives the narrator real corpus
scenes with verbatim text + chunk_id citations + bbox metadata.
This is what was missing — v1 only saw pre-digested hypothesis
artefacts, which is how the academic prose got there.
- Dropped the hypotheses + contradictions queries from the loader.
They were skeptic-framing scaffolding that doesn't belong in the
raw material a best-seller narrator works from.
- New buildPrompt sections: "Primary-source scenes", "Curated
verbatim quotes", "Anomalies and surprises", "Named witnesses".
Anomalies (Taleb's outlier gaps) reframed: drop dominant_model
skeptic baseline, keep title + why_surprising as gold material.
- Refusal floor: < 4 scenes from hybridSearch → skip with reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User explicit: "1 bilhão de entusiastas pelo mundo ovni" — site is for the
UFO-curious public, not for skeptics. The 8-detective scaffolding becomes
invisible plumbing; the reader sees stories about what was observed.
Reader-facing changes:
New homepage (web/app/page.tsx)
- SiteHeader: magazine-style top nav (no detective tiles)
- HeroBanner: full-bleed editorial opener with declassified-page art
background, display-serif headline, live stats row (122 docs,
2047 events, 1861 witnesses, 867 craft catalogued)
- FeaturedCase: cover-story treatment of the most recent case_report,
uses a real document page as hero image, links to /c/[slug]
- PortalGrid: 6 thematic doorways into the archive — Sightings,
Witnesses, Craft, Hot spots, Programs, Documents — each tile shows
a real entity count and short editorial blurb
- GreatestHits: top 9 most-cited events from the corpus
(Kenneth Arnold 1947, Mantell 1948, …) as a magazine grid
- Doc list kept but reframed as "the primary record"
New sub-pages (5)
- /sightings → events (2047), magazine grid
- /witnesses → people (1861), compact table
- /objects → uap_objects (867), magazine grid
- /locations → locations (1757), compact table
- /operations → organizations (1596), compact table
- /documents → full doc list with thumbnails (mirrors homepage section
for direct deep-link)
All share <EntityListPage> shell with per-page i18n + JSON-LD ItemList
Stripped detective surfacing
- /jobs/[id]: "Sherlock Holmes / Dr. Watson" → "Investigation in progress"
- chat-bubble: detective-named card → neutral "Investigação em andamento"
- quick-launch: 7-kind detective dropdown → single "investigar um caso"
input (kind=case_report hardcoded)
- /bureau: rewritten as the case-file library (no artefact dumps)
Typography + design
- Fraunces variable serif loaded for display headings
(`.font-display` class)
- Gold-amber accent (#e0c080) unified as the brand colour
- Asymmetric magazine grids (1+2+3 column, generous whitespace)
- Hover micro-interactions (image scale on featured case, translateX
on portal arrows)
SEO + GEO
- layout.tsx metadataBase + title.template + per-route Metadata exports
- Organization JSON-LD on root layout
- WebSite + SearchAction JSON-LD on homepage
- CollectionPage + ItemList JSON-LD on every entity list page
- openGraph + twitter cards, pt-BR primary + en-US alternate
- ai:purpose meta tag for Generative Engine Optimization — declares
the site as a citation-linked primary-source archive
- robots: index + follow with large image preview
The detectives themselves remain alive in the backend (runtime, DB, audit
log), but the reader never sees "Holmes / Sun-Tzu / Watson" in the UI. The
next phase will reorient case-writer to write as a single best-seller voice
synthesising all the internal sources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Live failure surfaced by user feedback: Poirot wrote a low-credibility
verdict on J. Edgar Hoover (W-0002) based on 1 actual chunk and 11
entity_mentions false positives where 'DIRECTOR'/'DIRETOR' was linked to
him by mistake. Poirot's own bias_notes correctly identified this — yet
still produced a verdict. Published on a 'Disclosure Bureau' site, that's
libellously misleading.
Deleted W-0001 (Donald Keyhoe) and W-0002 (J. Edgar Hoover) from
public.witnesses + their .md files.
Prompt rewrite (prompts/poirot.md):
- New "What counts as testimony" section up front, before discipline.
Direct testimony = the person AUTHORED, was QUOTED verbatim with
attribution, or GAVE testimony in a recorded hearing. Not: third-
party mentions, generic title appearances ('Director'/'Diretor'
that entity-extraction speculatively linked), CC lines.
- HARD FLOOR rule: emit `direct_testimony_chunk_ids[]`. If < 3, refuse
with INSUFFICIENT_TESTIMONY. For famous historical figures
(Wikipedia-worthy public figures) the floor is 5.
- Bias claims MUST cite a specific chunk; ungrounded bias claims drop.
- Tone: "careful prosecutor preparing a brief, not debunker scoring
points."
Defense in depth (poirot.ts):
- Detective enforces the same floor before calling writeWitnessAnalysis,
using a FAMOUS slug list (j-edgar-hoover, donald-keyhoe, j-allen-
hynek, curtis-lemay, vannevar-bush, eisenhower, truman, kennedy,
ted-bloecher, ...).
- When the floor isn't met, emit `poirot_refused_floor` audit event +
skip with reason like `insufficient_direct_testimony_1_of_5`.
- Sentinel parser now also catches INSUFFICIENT_TESTIMONY when it
appears on the first line of an otherwise-prose response.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two complaints in one wave:
(W4.1) User: "Não pode ter vícios de IA como uso excessivo de '-' que a IA
coloca geralmente no lugar de vírgulas por exemplo. Isso deve fazer parte
do prompt geral."
- New prompts/_house-style.md banning the 9 most common AI prose tells
in both EN and PT-BR:
1. Em dashes as comma replacements (—)
2. Rule-of-three lists ("concrete, rigorous, and grounded")
3. Conjunctive openers ("Moreover", "Notably", "Ademais")
4. Superficial -ing analyses ("marking a shift", "destacando")
5. Inflated symbolism + AI vocab (tapestry, navigate, delve,
underscore, robust, multifaceted, marco histórico, ...)
6. Negative parallelisms ("Not just X but Y")
7. Vague attribution ("Some scholars say...")
8. Summary closers ("In summary...", "Em suma...")
9. Hedging fluff ("It's important to note...")
Verbatim chunk quotes are explicitly exempt; preserve as-is.
- claude.ts callClaude() lazily loads _house-style.md once per process
and PREPENDS it to every detective's system prompt:
composedSystem = houseStyle + "---" + detective.systemPrompt
This means all 7 detectives + future ones get the rules without any
per-prompt change.
(W4.2) User: "Quando entra em uma página da investigação não tem como
voltar! UX terrível!"
- New <BureauNav> sticky topbar with explicit "← home" + "🔎 bureau"
buttons + clickable breadcrumb trail. Always visible at the top of
every bureau page so the user can escape in one click.
- Wired into /bureau, /h/[hypothesisId], /c/[slug], /jobs/[id]. Each
page passes its sensible parent crumb (/bureau#hypotheses,
/bureau#reports, /bureau#jobs).
- Replaces the previous plain-text "disclosure.top / hypothesis /
H-0004" line which had no visual affordance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Live PT-BR smoke on j-edgar-hoover produced verdict_pt_br at 304 chars
(prompt says ≤ 280). The writer correctly rejected it ("verdict too long
(304 > 280)") but the job failed instead of trimming.
Fix: detective now trims each language field at the nearest sentence
boundary (period or semicolon) above 60% of the cap; falls back to a hard
cut at the cap. Applied to verdict / verdict_pt_br (≤280), and to
access_to_event*, bias_notes* (≤800) for defense in depth.
The contract with the writer stays strict; the detective just becomes
forgiving about the model going 5-10% over.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User flagged that the bureau was emitting English-only output, violating
the project's bilingual rule. Every narrative field now ships in both
languages: stored in sibling DB columns + rendered as adjacent markdown
sections per CLAUDE.md §3.
Migration 0007 (apply as supabase_admin):
- public.hypotheses +question_pt_br, +position_pt_br,
+argument_for_pt_br, +argument_against_pt_br
- public.contradictions +topic_pt_br, +notes_pt_br
- public.witnesses +access_to_event_pt_br, +bias_notes_pt_br,
+verdict_pt_br
- public.gaps +description_pt_br, +suggested_next_move_pt_br
- public.evidence: unchanged (verbatim_excerpt stays source-language)
- JSONB siblings inside contradictions.chunks + gaps.scope handled at
runtime (statement_pt_br, title_pt_br, dominant_model_pt_br,
why_surprising_pt_br, what_it_implies_pt_br).
Detective prompts (all 7) rewritten with explicit bilingual JSON contract:
- Output protocol section names every EN field + its _pt_br sibling
- "Bilingual is mandatory" warning in the task instruction
- Sentinel skip-states unchanged (NO_HYPOTHESES, NO_CONTRADICTIONS,
INSUFFICIENT_TESTIMONY, INSUFFICIENT_HYPOTHESIS, NO_OUTLIERS,
NO_NEW_EVIDENCE, INSUFFICIENT_ARTEFACTS)
- Schneier: parallel arrays — hidden_assumptions[i] matches
hidden_assumptions_pt_br[i], lengths must match
- Case-Writer: interleaved §1 (EN) / §1 (PT-BR) per act in the body
Writer-side validation (all 7 tools):
- Reject INSERT if PT-BR sibling missing when EN field is set
- Persist both languages atomically in one INSERT (no half-updates)
- Markdown renderers write adjacent EN+PT-BR sections in case files
(## Argument for (EN) followed by ## Argumento a favor (PT-BR), etc.)
Detective parse layer (all 7 detectives):
- Coerce both keys from JSON output
- "incomplete_bilingual_*" skip reason when either side missing
- Defensive: PT-BR fields trimmed + length-capped same as EN
Orchestrator propagates question_pt_br + topic_pt_br through job payload
to runHolmes / runCaseWriter, mirroring the chat-tool entry point.
Web (UI):
- /api/jobs/[id] hydrates _pt_br siblings from pg
- job-status-poller HypothesisCard: PT-BR primary, EN in <details>
fallback when both exist
- ContradictionCard: PT-BR statement primary + secondary EN quote
- WitnessCard: PT-BR verdict primary + secondary EN quote, panels in PT
- GapCard: PT-BR title/why/implies primary
- /bureau hub: SELECTs both columns, renders PT-BR primary
- /h/[id]: ArgumentPanel renders PT-BR primary with collapsible EN
fallback when both exist
- BureauSnapshot homepage: position_pt_br / topic_pt_br / verdict_pt_br
primary
- DocBureauPanel /d/[doc]: same primary-PT-BR pattern
- New web/lib/i18n/pick.ts helper (unused yet by chat/agents — kept
for future locale-driven switching when both languages are equally
full; current rule is PT-BR-first since the user is brasileiro)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Builds on top of W3.9 to turn the homepage Bureau from a read-only
dashboard into a working command center.
UI improvements (web/components/bureau-snapshot.tsx):
- Detective tiles are now <Link>s — each navigates to its primary
artefact section in /bureau (Holmes→#hypotheses, Locard→#evidence,
Dupin→#contradictions, Schneier→#hypotheses, Poirot→#witnesses,
Taleb→#outliers, Tetlock→#hypotheses, Case-Writer→#reports). Hover
bg matches the detective's tone color.
- <QuickLaunch /> form inserted right under the tiles.
New <QuickLaunch /> client component:
- Detective dropdown (7 active kinds; evidence_chain not yet exposed
here since it needs a doc_id better picked from the doc page).
- Single input swaps placeholder + aria-label by kind: question for
Holmes, topic for Dupin/Taleb/Case-Writer, hypothesis_id for
Schneier/Tetlock, person_id for Poirot.
- Submits to POST /api/bureau/launch and redirects to /jobs/[id]
via the next.js router.
- Loading state ("queueing…") + error display inline.
POST /api/bureau/launch (web/app/api/bureau/launch/route.ts):
- Same 8-kind validator as the chat tool's request_investigation.
- Auth required when Supabase is configured (triggered_by = user:email).
- Returns { job_id, kind, detective, status_url, eta_seconds }.
DocBureauPanel on /d/[docId] (web/components/doc-bureau-panel.tsx):
- Server component inserted between the doc header and
AnomalyHighlights.
- Surfaces every bureau artefact that touches the doc:
· Evidence whose source_page_id starts with docId/p
· Hypotheses citing any of those evidence_ids
· Contradictions whose chunks[] has any item with this doc_id
· Gaps/outliers with scope.doc_id == docId
· Case reports whose markdown body references docId (filesystem scan)
- Empty state shows "Investigation Bureau — untouched" with a CTA
linking back to the homepage to launch the first investigation.
- When non-empty, header counts total artefacts + links to /bureau
for the full view.
Metadata (web/app/layout.tsx):
- description rewritten from "Investigative wiki of the US Department
of War UAP/UFO archive (war.gov/ufo)" to one that names the bureau
+ the 8 detectives. Affects SERP previews + social-card defaults.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/c/[slug] returned 404 even after the W3.9 web rebuild because the web
container's volume list didn't include the case/ directory the
investigator-runtime writes to. The BureauSnapshot file-listing for
Case reports gracefully fell back to empty, but /c/<slug> can't fall
back: it has to read the markdown.
Fix:
- Mount ${CASE_ROOT:-/data/disclosure/case}:/data/ufo/case:ro (read-only,
same pattern as wiki/processing/raw).
- Set CASE_ROOT=/data/ufo/case env in the web container so the
/c/[slug] page and BureauSnapshot resolve the same path.
Verified live: /c/green-fireballs-sandia now serves HTTP 200 with the
Watson narrative parsed + rendered via MarkdownBody.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes a UX gap the user surfaced: W3.5-3.8 built 8 detectives, 4 new
URL endpoints (/jobs/[id], /h/[id], /c/[slug], /api/h/[id]/red-team)
and a chat tool, but the homepage was unchanged — the bureau was
invisible unless you knew the URL or asked the chat to invoke
request_investigation.
Homepage (web/app/page.tsx):
- Title `▍ war.gov/ufo — Investigative Wiki` → `▍ The Disclosure Bureau`
- Subtitle expanded from "Holmes · Poirot · Dupin · Locard" to all 8
detectives (Holmes · Locard · Dupin · Schneier · Poirot · Taleb ·
Tetlock · Case-Writer)
- New `🔎 bureau` topbar link (gold, between graph/stats and batch)
- BureauSnapshot inserted right after the header
BureauSnapshot (web/components/bureau-snapshot.tsx) — server component:
- 8 detective tiles with role labels (each in its tone color)
- 6 clickable counters (evidence / hypotheses / contradictions /
witnesses / outliers / case reports) — anchor to /bureau#section
- 6 "recent artefacts" columns surfacing the last 3-4 of each kind:
hypotheses with prior→posterior + band + ↳reviewed_by marker,
contradictions with topic + resolution_status, evidence with
Grade badge + verbatim quote, outliers with title + scope.kind,
witness analyses with canonical_name + credibility + verdict,
case reports with slug + link to /c/<slug>
- "Recent jobs" strip linking to /jobs/[id] color-coded by status
- Reports read from /data/ufo/case/reports/ via fs.readdir + stat,
sorted by mtime — no DB round-trip needed for that section
/bureau (web/app/bureau/page.tsx) — full hub:
- Header with full counts
- 7 sections (anchored to homepage counter links): Case reports,
Hypotheses, Evidence, Contradictions, Outliers, Witnesses,
Recent jobs table — each rendering up to 100 rows
- Reports section parses frontmatter from each .md to surface topic
+ n_hypotheses + n_evidence on the card
Runtime fixes batched in:
- Poirot: coerce entity_pk via Number() — node-postgres returns
BIGINT as string by default; writer's Number.isFinite() rejected
it as "person_entity_pk required" (j-edgar-hoover retry path)
- Tetlock: write_calibration rationale cap 600 → 1200 chars. Prompt
still asks ≤ 600 but a 2× slack beats failing the job on honest
analysis. Observed live: Tetlock emitted ~620 chars on H-0003 and
the writer rejected the entire calibration.
- Case-Writer: Promise.all of 5 queries × max_parallel=2 jobs
demanded up to 10 connections against the investigator role's
rolconnlimit=4 → "too many connections for role investigator".
Sequentialized — the LLM call is the hot path, not these queries.
Smoke results visible now on the homepage:
- 3 hypotheses (H-0001/2/3) about green fireballs origin
- 3 contradictions (R-0001/2/3) about color, geographic confinement,
exclusive-green vs multicolored
- 2 evidence cards (E-0002/3) Grade B
- 3 outliers (G-0001/2/3) — including Taleb's deliberate
meteor-shower-camouflage flag
- 1 case report at /c/green-fireballs-sandia (Watson 13.4 KB,
five-act narrative, fully cited)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the bureau from 4 → 8 detectives. All eight run as Bun + claude-CLI
subprocesses against the same Supabase + investigation_jobs LISTEN/NOTIFY
queue, sharing search.ts hybridSearch and writer-side validators that
gate writes against schema + FK.
New detectives:
Poirot (witness_analysis)
- prompts/poirot.md — credibility / access / bias / corroboration /
verdict; uses entity_mentions JOIN chunks to pull 12 chunks per
person; resolves corroboration_refs chunk_ids defensively (accepts
bare cNNNN even when the model emits pNNN/cNNNN).
- INSERT into public.witnesses with W-NNNN naming.
- Tone: purple (#9b5de5).
Taleb (outlier_scan)
- prompts/taleb.md — "surprise is relative to a model"; at most 3
outliers; each requires explicit dominant_model + why_surprising +
what_it_implies; fan-out into public.gaps with scope.kind="outlier".
- Same unscoped-fallback as Dupin (Pass 1 with doc_id, Pass 2 widens
to corpus if hits < 3).
- Tone: yellow (#ffd23f).
Tetlock (calibrate_hypothesis)
- prompts/tetlock.md — honest Bayesian update; emits new_posterior +
Δ + recommended_action ∈ {keep, downgrade, upgrade, supersede}.
- write_calibration UPDATEs public.hypotheses + APPENDS a
"## Calibration history" section to the H-NNNN.md case file
(calibration is append-only — each datapoint matters). Posterior
band auto-corrected to match Tetlock thresholds.
- NO_NEW_EVIDENCE sentinel handled; pure 'keep' with |Δ|<0.005 only
touches updated_at + reviewed_by.
- Tone: teal (#26d4cc).
Case-Writer (case_report)
- prompts/case-writer.md — Dr. Watson assembles all artefacts
(E-NNNN, H-NNNN, R-NNNN, W-NNNN, G-NNNN) into a five-act narrative.
ILIKE filter on topic; doc_id optional scope.
- Larger budget cap (≥ $0.50) + longer timeout for prose generation.
- Writes case/reports/<slug>.md with frontmatter (topic + counts);
no DB table for v0.
- New page /c/[slug] renders the report via MarkdownBody + stat chips.
- Tone: gold (#e0c080).
Hardening across the bureau:
- Sentinel parsing now accepts backticked AND prose-trailing forms
(Holmes NO_HYPOTHESES, Dupin NO_CONTRADICTIONS, Schneier
INSUFFICIENT_HYPOTHESIS, Poirot INSUFFICIENT_TESTIMONY, Taleb
NO_OUTLIERS, Tetlock NO_NEW_EVIDENCE, Case-Writer
INSUFFICIENT_ARTEFACTS). Avoids the failure mode where the model
refuses honestly but the runtime treated it as a parse error
(observed live with Poirot+Hoover identifying the DIRECTOR
false-positive disambiguation issue in entity_mentions).
Chat tool extensions (web/lib/chat/tools.ts):
- request_investigation now accepts 7 kinds. Each routes to its
detective with appropriate validation (hypothesis_id regex,
person_id kebab-case, topic non-empty, doc_id for evidence_chain).
- ETA per kind: Holmes/Dupin 60s, Poirot 45s, Schneier/Tetlock 30s,
Taleb 50s, Case-Writer 180s (longer prose), Locard 30×n_chunks.
UI integration:
- chat-bubble inline card paints each detective in its tone color.
- /jobs/[id] page header swaps name/subtitle/tone per detective;
question label adapts ("Topic" / "Hypothesis under attack" /
"Witness under analysis" / "Topic to outlier-scan" / "Hypothesis
under recalibration" / "Case to assemble").
- job-status-poller renders: case-report link card (gold), outlier
cards (yellow), witness cards (purple) — alongside existing
hypothesis, evidence, contradiction cards.
- /api/jobs/[id] hydrates witnesses (JOIN entities for canonical_name)
+ gaps (with scope JSONB).
- /c/[slug] page reads /data/ufo/case/reports/<slug>.md and renders
with MarkdownBody, frontmatter parsed for stat chips.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the fourth AI detective in the Investigation Bureau runtime: Bruce
Schneier, who attacks an existing hypothesis as a red-team operator.
Runtime:
- prompts/schneier.md — discipline (don't disprove, just attack;
structured output with hidden_assumptions, failure_modes,
alternative_explanations, recommended_tests, verdict_one_sentence;
severity ∈ {low, medium, high}; emit INSUFFICIENT_HYPOTHESIS when
the input is too thin)
- src/detectives/schneier.ts — reads the hypothesis row + evidence
chain (joined via evidence_refs FK), feeds Claude with the
arguments + verbatim quotes, parses strict JSON object
- src/tools/write_red_team_review.ts — UPDATEs hypotheses.reviewed_by
+ updated_at; APPENDS (or replaces if re-reviewed) a structured
"## Red-team review (Schneier · X severity)" section to
case/hypotheses/H-NNNN.md. Caps each list at 5 entries × 240 chars,
validates verdict ≤ 280 chars.
- orchestrator: new `red_team_review` kind dispatching to runSchneier
Chat + UI:
- request_investigation gains kind=red_team_review + hypothesis_id arg
(validated against H-NNNN regex); detective auto-resolves to schneier
- chat-bubble inline card paints Schneier in red (#ff3344)
- /jobs/[id] page swaps title/subtitle/tone per detective; the
"Question" label becomes "Hypothesis under attack" for red_team_review
New /h/[hypothesisId] page (hypothesis dossier):
- Server-rendered from public.hypotheses + public.evidence (joined
via evidence_refs FK + chunk lookup)
- Header: ID + creator + reviewer (highlighted when Schneier has
visited), position as headline, question subtitle, Tetlock band
- Prior + posterior bars with Δ-delta indicator
- Argument grid: argument_for (green) vs argument_against (pink)
side-by-side with [[wiki-link]] auto-linking to source chunks
- Evidence chain: each E-NNNN with Grade A/B/C badge, verbatim
blockquote, link to source page
- Red-team review panel: parses the markdown section in the case
file (severity badge, verdict, 4 bullet panels for
hidden_assumptions / failure_modes / alternative_explanations /
recommended_tests). Empty state when not yet reviewed.
RedTeamRequestButton client component + POST /api/h/[id]/red-team —
authenticated user can trigger Schneier in one click; UI swaps to
"acompanhar" link to /jobs/[id] once queued.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two regressions surfaced in the smoke test that put Dupin from
0/3 contradictions written → 3/3 in the next run.
1. Single-doc scope was too narrow for Dupin's task.
Holmes's question about Sandia returned 4 chunks scoped to one doc,
but Dupin's terser "topic" form yielded only 1. Solution: Pass-1
tries the requested doc_id; if the head is < 2 chunks, Pass-2
widens to the whole corpus. Audit event carries `scope_widened`
so the case-writer can later flag cross-doc contradictions
distinctly. The unscoped retry hit 9 chunks and produced 3
contradictions across 3 different docs.
2. Chunk-block header was ambiguous to the model.
`--- doc-id/p007#c0042 ---` led Claude to parse `chunk_id` as
"p007#c0042" or "p007/c0042" in the JSON output. write_contradiction
then refused the FK lookup with "chunk not found". Fix:
- Explicit `doc_id:` / `chunk_id:` / `page:` lines per chunk
in the rendered block (no slashes/hashes the model can fold).
- Defensive normalizeChunkId() in write_contradiction.ts strips
any pNNN prefix and keeps only the trailing cNNNN — so the
writer is forgiving without losing strictness on the topic +
statement validation.
Smoke now produces (job 6deddf4b):
R-0001 (3 chunks) — Color of the fireball(s) in incident summaries
R-0002 (2 chunks) — Geographic confinement of green-fireball sightings
R-0003 (3 chunks) — Whether the phenomenon was exclusively green or
also red/multicolored
R-0003 connects 3 different declassified documents: the Los Alamos
conference (exclusively-green category), a retrospective document
(red OR green), and Incident 229 (red, blue, yellow — no green).
Real cross-doc contradiction, fully cited.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the third AI detective in the Investigation Bureau runtime: C. Auguste
Dupin, who scans a corpus shortlist for pairs (or small groups) of chunks
that cannot both be true under any ordinary reading.
Runtime:
- prompts/dupin.md — discipline (no contradiction without ≥2 distinct
chunk_ids; reject same-vocabulary near-misses; FEW high-confidence
over MANY weak ones; emit `NO_CONTRADICTIONS` when corpus is silent)
- src/detectives/dupin.ts — hybridSearch with k=18 (more chunks than
Holmes because contradictions emerge from comparing dispersed
claims), strict JSON-array parsing, AT MOST 3 contradictions per call
- src/tools/write_contradiction.ts — validates topic + ≥2 positions
drawn from ≥2 distinct chunks, resolves chunk_pk via DB lookup
(rejects positions citing unknown chunks), INSERTs into
public.contradictions + writes case/contradictions/R-NNNN.md
- orchestrator: new `contradiction_scan` kind dispatching to runDupin;
payload { topic, doc_id?, lang?, context_chunks? }
Chat + UI:
- request_investigation gains kind=contradiction_scan + topic arg;
triggered detective auto-resolves to dupin
- chat-bubble inline card renders dupin in orange (#ff8a4d) to
distinguish from holmes (cyan) and locard (green)
- /jobs/[id] page swaps title + subtitle + tone per detective;
"Question" label becomes "Topic" for contradiction_scan
- /api/jobs/[id] hydrates public.contradictions when outputs[] surfaces
contradiction_ids
- job-status-poller renders ContradictionCard: topic + N positions
(verbatim statements quoted, stance label optional, link to source
chunk) + optional notes panel, with resolution_status badge
(open/resolved/irreconcilable)
R-NNNN shares the contradiction_id_seq slot with relation per
CLAUDE.md naming — same conceptual class (a connection between two
pieces of evidence in tension).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the loop between the chat UI and the Investigation Bureau runtime.
Chat tool (web/lib/chat/tools.ts):
- request_investigation { kind, question, doc_id?, chunks?, claim? }
INSERTs a row in public.investigation_jobs and returns
{ job_id, kind, status, eta_seconds, status_url, detective }.
- kind=hypothesis_tournament → Holmes (1 question → 2-3 rival hypotheses)
- kind=evidence_chain → Locard (1 doc → grade-A/B/C evidence with chain
of custody, default top-5 anomaly chunks)
- Plumbed user.email through ToolHandlerContext so triggered_by audits
the requesting user.
Public job viewer:
- GET /api/jobs/[id] joins investigation_jobs → public.evidence +
public.hypotheses for the IDs surfaced in outputs[]. Returns one
payload the page can render without n+1 round-trips. Strips
triggered_by from the response (it carries the user's email).
- app/jobs/[id]/page.tsx server-renders the case-file shell:
detective lore header (Holmes blue or Locard green), question chip,
scope chip with link back to the document.
- components/job-status-poller.tsx client island that polls every 3 s
while non-terminal, then once on terminal to hydrate evidence +
hypotheses. Renders:
· Phase tracker (queued → running → complete | failed)
· Hypothesis cards w/ prior + posterior bars + Δ delta indicator
+ Tetlock band badge (high/medium/low/speculation)
· Argument-for / argument-against with [[wiki-link]] auto-linking
to /d/<doc>/p<NNN>#<cNNNN>
· Evidence cards w/ Grade A/B/C badge + verbatim blockquote +
bbox crop preview via /api/crop + custody-steps disclosure
· Empty/in-flight panel ("os detetives estão lendo o corpus")
· Failure panel surfacing error + partial outputs
Inline chat-bubble card (components/chat-bubble.tsx):
- ToolTrace.richRender recognises request_investigation results and
renders a detective banner with status + ETA + link to /jobs/[id]
(target=_blank). Error case renders a red strip with the message.
UX flow now: user asks Sherlock a question → request_investigation
queues the job → chat card shows "🔎 Holmes · hypothesis_tournament ·
ETA ~60s" → user clicks → /jobs/<id> live-updates → 60 s later, 2-3
rival hypotheses + their arguments + chunk citations are rendered with
Bayesian update visible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the second AI detective in the Investigation Bureau runtime: Sherlock
Holmes, who builds 2-3 rival hypotheses with calibrated priors + posteriors
against a corpus shortlist.
Pipeline:
1. hybridSearch() grounds Holmes with 8-15 chunks via the same
hybrid_search_chunks RPC the web uses (BM25 + dense + RRF). Default
max_dense_dist=0.55 (runtime favors recall over precision; web's
/api/search/hybrid stays at 0.40 for chat).
2. claude-sonnet-4-6 emits a strict JSON array with position +
argument_for + argument_against + prior + posterior + confidence_band
+ evidence_refs. Citations use [[doc-id/pNNN#cNNNN]] wiki-links.
3. writeHypothesis() validates posterior ∈ [0,1], auto-corrects the
Tetlock band from the posterior (high ≥0.90, medium 0.60-0.89,
low 0.30-0.59, speculation <0.30), checks evidence_refs FK against
public.evidence, INSERTs into public.hypotheses + writes
case/hypotheses/H-NNNN.md.
Discipline guarantees (prompts/holmes.md):
- posteriors across rivals sum to ≈1.0
- no claim without chunk citation
- prefer lower band when ambiguous (anti-inflation)
- declarative one-sentence position, no hedging
- emit `NO_HYPOTHESES` when corpus is silent (refuses to fabricate)
Smoke test (Sandia green fireballs 1948-49):
- H-0001 prior 0.5 → posterior 0.2 (speculation): natural meteoric
- H-0002 prior 0.3 → posterior 0.4 (low): classified weapons / tests
- H-0003 prior 0.2 → posterior 0.4 (low): genuinely unidentified
Bayesian update visible: "natural meteoric" prior dropped 60%; both
rivals climbed. 4 unique chunk citations across the 3 hypotheses.
orchestrator dispatches `hypothesis_tournament` kind via runHolmes;
job marked `failed` if all rivals error, `complete` otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Token consolidation:
- docker-compose web service now reads ${CLAUDE_CODE_OAUTH_TOKEN} directly,
drop the W1-F8 CLAUDE_CODE_OAUTH_TOKEN_FOR_WEB indirection (user feedback:
one var name, no _FOR_WEB suffix).
investigator-runtime claude.ts:
- --system-prompt silently dropped by CLI v2.1.150 for multi-KB prompts;
inline the system content into the user prompt with a separator
(mirrors scripts/reextract/run.py pattern).
- Multi-line prompts via positional -- broke ("Input must be provided …");
pipe via stdin instead.
- --allowedTools "" is rejected; when no tools wanted, omit it and explicitly
--disallowedTools the writer/reader set so the model can't reach for any.
investigator-runtime locard.ts:
- Log the raw response (first 600 chars) to container stderr — saved hours
of debugging when the writer rejected.
- Grade fallback: when Locard omits `grade` but provides custody_steps,
infer the highest grade that fits (≥3 → A, ≥2 → B, ≥1 → C).
investigator-runtime write_evidence.ts:
- Filter related_hypotheses entries with empty/null hypothesis_id silently
(Locard sometimes emits [{}] when it knows no link yet) instead of
failing the whole write.
Migration 0006_investigator_serial_sequences.sql:
- BIGSERIAL on the 7 investigation tables created auto-sequences
(evidence_evidence_pk_seq etc) that 0004 forgot to GRANT to the
investigator role. Without those grants every INSERT failed with
"permission denied for sequence …". Grant USAGE/SELECT/UPDATE on each
auto-seq.
Verified live: Locard wrote E-0002 + E-0003 from real Sandia chunks
(green fireball Feb 1949; cobalt particle analysis). Grade B, confidence
high, custody chain of 3 steps with honest gaps. Cost $0.09 for both,
~70s wall.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scanned docs are messy — duplicate transcriptions (typed + handwritten),
two classification variants of the same narrative, OCR noise, repeated
banners. The doc page showed raw chunks, so everything appeared twice.
40_reading_version.py generates ONE clean, deduplicated, well-structured
bilingual Markdown reading version per doc (Sonnet): merges duplicate versions
without losing unique lines, drops page furniture, formats transcripts as
dialogue. Faithful — invents nothing; redactions kept as markers.
/d/[docId] now defaults to a "📖 leitura" tab rendering this clean version,
with "🔍 trechos · scan original" preserving the faithful per-chunk + per-page
scan view. reading.md lives in raw/<doc>--subagent/ alongside the chunks.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
~414 chunks have ufo_anomaly_detected=true but no type/rationale (extraction
left them null), so the flag rendered "UAP flag: anomaly —" with a dangling
em-dash. Build the label from the parts that exist: fall back to "anomalia"
for a missing type and omit the "—" when there's no rationale. The flag still
shows (the chunk genuinely contains UAP content), just without the noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
29 image chunks have an empty bbox {}, and `fm.bbox ?? default` doesn't catch
an empty object, so the crop URL got w=undefined → /api/crop 400 → broken-image
icon. Now validate bbox (w/h > 0); without it, render the image's textual
description instead of requesting an impossible crop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root-cause fix for "search returns garbage for absent terms". The hybrid RPC's
dense branch always returned its k nearest vectors regardless of distance, so a
query for a term not in the corpus (e.g. "varginha") surfaced unrelated chunks.
The cross-encoder reranker would filter these but costs 18-62s on CPU —
unusable for interactive search.
Add max_dense_dist (default 0.40) to hybrid_search_chunks: dense neighbours
beyond that cosine distance are dropped server-side. Calibrated from measured
distances — strong semantic match ~0.12-0.20, no real match ~0.46-0.53. BM25
full-text still matches literal terms; the reranker becomes opt-in refinement.
Verified live: varginha/abducao → 0, disco voador/roswell → relevant, all <1s.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The hybrid_search RPC always returns up to recall_k dense neighbours, so a
query for a term absent from the corpus (e.g. "varginha") returned its 12
nearest vectors — irrelevant chunks like PAGE_NUMBER "1". Two bugs:
the reranker was skipped whenever results <= top_k, and there was no relevance
floor.
Now always run the cross-encoder reranker (BGE-reranker-v2-m3, normalized
sigmoid) and drop hits below 0.02. Verified: "varginha" → 0 results;
"roswell"/"tic tac"/"disco voador" → relevant hits on top (reranker cleanly
separates 0.0001 garbage from 0.03-0.27 matches).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The reprocess pass added chunks for pages beyond the original total_pages but
never updated the field, so doc-page navigation thought docs ended early
(jumped to next document mid-doc) and the page counter was wrong. Now bump
total_pages to the real max chunk page on each integration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Doc page (/d/[docId]/[page]) gains prev/next navigation bars (top + bottom):
within a doc it steps page-by-page; at the first/last page it jumps to the
previous/next document. Replaces the disabled-at-boundary links.
Indexer tooling for the VPS repopulation:
- 30-index-chunks-to-db.py: add --no-embed (fast BM25-only index; vectors
backfilled separately) so the app is usable in minutes, not hours of CPU
embedding.
- 57_load_relations_from_json.py: load typed relations into public.relations
from reextract structured fields (deterministic ids, no fuzzy guessing).
- 58_backfill_embeddings.py: async pass to fill chunks.embedding (NULL rows)
via the embed-service.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add reextract pipeline (scripts/reextract/) that rebuilds doc-level entity
JSON from Sonnet-vision chunks via Opus, replacing the noisy per-page
extraction. Add synthesize scripts to regenerate wiki/entities from the 116
_reextract.json (30), aggregate missing page.md from chunks (31), and reprocess
805 pages the doc-rebuilder agent dropped on context overflow (32). Add
maintain scripts 43-56 for chunk-page sync, dedup, generic-entity marking, and
typed relation extraction.
Web: wire relations API + entity-relations component; entity/timeline/doc
pages consume the rebuilt layer.
Note: raw/, processing/, wiki/ remain gitignored (bulk data managed
separately); the 116 reextract JSONs and 7,798 rebuilt entity files live on
disk only. The 27 curated anchor events under wiki/entities/events/ are
preserved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The corpus had two parallel reverse-reference signals: the wiki/pages
entities_extracted blocks (Haiku page-level) and public.entity_mentions
(Sonnet chunk-level, ILIKE-matched). The entity page only consulted the
DB, so it showed "0 menções" for thousands of entities that were anchored
in pages or in cross-entity links the DB never indexed.
Resolved by collapsing all signals into the YAML frontmatter, which is
now the single runtime source for entity metadata.
scripts/maintain/42_sync_entity_stats.py walks every entity and writes:
mentioned_in: [...] # consolidated page refs
total_mentions: max(db, pages)
documents_count: max(db_docs, distinct page docs)
signal_sources:
db_chunks: int
page_refs: int
cross_refs: int
signal_strength: strong | weak | orphan | unverified
referenced_by: [[class/id]] # cross-entity backlinks
Outgoing wikilinks (e.g. OBJ.observed_in_event → EV) count toward the
entity's own cross_refs so anchored-but-not-mentioned entities don't
register as orphan.
OBJ canonical names like "7m long, 1.3m high, two rocket motors,
smooth flow, rotary drive null UAP (OBJ-EV1945-PEYERLSHOTDOWN-01)"
are rewritten to "Peyerl shot down UAP" derived from observed_in_event,
preserving the full description as an alias. --fix-obj-names did this
for every OBJ-* with >80 char canonical_name.
Default behaviour is conservative: --archive-only-junk archives only
single/double-char names and pure-numeric noise. Everything else stays
on disk with signal_strength marked, so the user can review later.
web/lib/retrieval/entity-pages.ts swapped from db-first to yaml-first.
The /e/[cls]/[id] page now reads counts straight from YAML and renders
a "força do sinal" badge with the per-source breakdown. Orphan entities
get a banner explaining they have no cross-references.
DB is still queried for ONE thing: the chunk text for preview cards on
the entity page, so we don't re-parse 21k markdown files on every render.
First-pass result: 9020 strong / 14520 weak / 10814 orphan; OBJ-EV1945-
PEYERLSHOTDOWN-01 now reads "Peyerl shot down UAP · fraca · 1 backlink"
in the live UI.
middleware.ts now checks profiles.role on every /admin/* request and
returns a plain 404 (not a redirect) for anonymous users and any
authenticated user without role='admin'. The 404 wording matches a
non-existent route, so we don't leak the existence of an admin area.
gutomec@gmail.com promoted to admin in the live DB.
openrouter.ts chat now collects artifacts silently during the tool-call
loop instead of streaming them to the SSE. After the model writes its
final prose (and after the forced-synthesis pass if needed), we scan
the assembled text for [[doc-id/p007#cNNNN]] citations and emit ONLY the
artifacts referenced. Duplicates are deduped by chunk_id. The persisted
citations column on messages now stores the filtered set too, so old
sessions reload with the same focused card grid.
Before: every hybrid_search hit (up to 6 per call × 5 calls = 30+ citation
cards plus crop images) flooded the chat regardless of what the model
ended up using. After: only the chunks actually woven into the answer.
Two bugs combined to make the chat reply with only cards and no prose:
1. SQL trigger rollup_session_stats was failing with "column reference
total_cost_usd is ambiguous" because the UPDATE on public.profiles had
a FROM public.chat_sessions clause and both tables expose that column.
Persistence of every user message died at this point — sessions were
created in the DB but had message_count=0 forever. Applied SQL fix
that qualifies columns with p./s. aliases (production DB updated;
ALTER FUNCTION run live, not yet codified in a migration file).
2. The free-tier model (nemotron-3-super:free) spent all 5 tool-loop
turns on hybrid_search calls and never wrote any prose, returning
content_len=0. Added a forced-synthesis pass in openrouter.ts: when
the loop exits with empty assembledText but the model did call tools,
we send ONE final turn with tools omitted from the request payload
and a user message instructing the model to answer in 3-8 sentences
citing chunks. openrouterStreamCall now accepts a `withTools` opt
so the synthesis call can disable tool calling entirely.
Verified end-to-end with the actual user query "O que os astronautas
viram? Quem foi que viu?" on /d/nasa-uap-d6-apollo-17-...:
- content_len: 0 → 947 chars (real synthesis citing Schmitt)
- artifacts: 44 preserved
- assistant message persisted with tool_calls + citations columns
34 of 116 docs were generated with 0-based PNG numbering (p-000.png …
p-008.png) but the Sonnet chunks reference 1-based page numbers in their
YAML frontmatter (page: 9 means the 9th sheet of paper). The /api/crop
handler built p-009.png and got a 500, the browser's Next/Image surfaced
400, and the chunk rendered as a black box on screen.
Fixes:
- web/app/api/crop/route.ts: try p-NNN.png first, fall back to
p-(NNN-1).png if the 1-based file is missing. Cheap insurance for any
doc that comes in with the old convention.
- scripts/01-convert-pdfs.sh: previously printf '%03d' "$num" with $num
starting at 0 (e.g. "008") raised "invalid number" because Bash
parsed it as octal. Wrap with $((10#$num)) to force decimal — this
was silently corrupting page sequences and producing gaps like p-001
... p-008, p-011 (missing p-009/p-010).
- All 34 affected docs re-converted from PDFs with the patched script;
every directory now has continuous 1-based PNGs.
- /processing/png/ rsync'd to VPS, web redeployed.
Smoke: /api/crop?doc=doc-341-…&page=9&… now returns 200 image/webp
instead of 500. Tested in browser: chunk c0026 (diagram, p9) renders
the real engineering diagram.
Fase 3 onda 2 — entity synthesis at scale:
- scripts/synthesize/20_entity_summary.py: queries DB for entities with
total_mentions ≥ threshold + top-K verbatim chunk snippets via
entity_mentions JOIN, prompts Sonnet (Holmes-Watson voice, bilingual),
writes narrative_summary EN+PT-BR + summary_status=synthesized.
Ran on 187 candidates (mentions ≥ 20) → 158 OK · 1 err · 29 skipped (no
snippets). Combined with anchor curation: 20 curated + 158 synthesized
= 178 entities with real narrative (vs 0 a day ago).
Fase 4 — chat with typed artifacts + persistence:
- lib/chat/agui.ts: AG-UI v1 typed Artifact union (citation, crop_image,
entity_card, evidence_card, hypothesis_card, case_card, navigation_offer)
alongside the existing event types.
- lib/chat/tools.ts + openrouter.ts: hybrid_search emits up to 6
citation + crop_image artifacts per query. Provider collects them and
returns in done.artifacts so the route can persist.
- api/sessions/[id]/messages: persist artifacts to messages.citations.
- components/chat-bubble.tsx: ArtifactCard renders inline cards (citation,
crop_image, entity_card, navigation_offer) for streamed and persisted
messages. activeId now persisted in localStorage so navigation between
pages keeps the same conversation. New sessions are lazy (only when user
has zero). loadMessages hydrates tools + artifacts from server. CRUD UI:
rename (✎) + archive (🗑) buttons per session in the list.
Home search:
- doc-list-filters: input now fires hybrid_search (rerank=0 for speed)
in parallel with the local title filter; chunk hits render above the doc
grid with snippet + score + classification.
- api/search/hybrid: accept ?rerank=0 to skip the cross-encoder (1.3s vs 60s).
Auth flow:
- infra: SMTP_HOST=mail.spacemail.com:587 + DMARC published; mail now lands
in inbox. GOTRUE_MAILER_AUTOCONFIRM=false (real email verification).
- kong.yml: proxy /auth/callback on api.disclosure.top → web:3000 so PKCE
email links don't 404 at the gateway.
- web/app/auth/callback: handle both ?code= (OAuth) and ?token=&type=
(PKCE); redirect to the public site host before verifyOtp so the session
cookie lands on the right domain.
Audit deliverables:
- .nirvana/outputs/disclosure-bureau/.../systems-atelier/: 5 docs (code
analysis, tech debt, discovery brief, system arch, 5 ADRs) authored by
sa-principal that produced this roadmap. Kept in-tree for traceability.
User reported a ~ Google Gemini bill against an expected ~ budget.
Permanent ban on Gemini for this project — all LLM inference goes through
claude -p --model sonnet (CLAUDE_CODE_OAUTH_TOKEN) or OPENROUTER_API_KEY as
fallback. The 10 scripts in scripts/ that import google-genai/-generativeai
are moved under scripts/_archived-gemini/ with a DO-NOT-RUN README. Project
memory updated (feedback-no-gemini-ever.md) and the older reference note that
recommended gemini-3.1-pro-preview is revoked.
Luiz Gustavo