Sun-Tzu (silent backend) — builds the strongest pro-anomaly brief the
corpus supports for any topic. Bilingual JSON: thesis + 2-4 pillars
(each with claim + citation-backed support) + honest residual
unexplained clause. NEVER surfaced reader-facing.
Migration 0009 (apply as supabase_admin):
public.pro_anomaly_briefs brief_pk BIGSERIAL PK
brief_id B-NNNN unique
topic + topic_pt_br
thesis + thesis_pt_br
pillars JSONB
unexplained + unexplained_pt_br
doc_id, job_id, created_by, created_at
+ brief_id_seq sequence
+ GIN trigram indexes on topic + topic_pt_br
+ RLS policies (investigator INSERT, public SELECT)
+ GRANTs on seq + table to investigator
prompts/sun-tzu.md
"Adversarial strategist who plays the pro-disclosure side with the
same rigour a red-team plays skeptic" — single thesis, 2-4 pillars,
honest residual. Every claim cites a chunk. No fabrication from
training-time knowledge. Output INTERNAL — case-writer pulls it.
Bilingual mandatory. NO_STRONG_CASE sentinel when corpus is thin.
detectives/sun_tzu.ts
Grounds with hybridSearch top 18 chunks, calls Sonnet, parses
JSON strict, calls writeProAnomalyBrief.
tools/write_pro_anomaly_brief.ts
Validates 2-4 pillars with bilingual claim+support, requires at
least one [[wiki-link]] citation per pillar, INSERTs.
orchestrator: new kind "anomaly_brief" dispatches Sun-Tzu.
Case-writer integration (detectives/case_writer.ts):
- Pulls most recent matching brief via ILIKE on topic or doc_id.
- Renders brief as a separate prompt section labelled
"Strategic brief (internal — do NOT cite or attribute)".
- Instructs the narrator to weave the thesis as a quiet through-
line, use pillar facts in scenes, let the unexplained clause
inform the closing paragraph. Forbidden to name "the analyst",
say "a brief argues", or use the words "thesis"/"pillar"
explicitly. Translate it into prose.
Entity hero illustrations:
- 3 painterly editorial illustrations generated via Nano Banana
Pro at 2K, stored under /data/disclosure/processing/case-art/:
* EV-1947-06-24-kenneth-arnold-sighting.png — cockpit POV of
Arnold in a CallAir A-2 over Mount Rainier, 9 chevron disc
objects in formation, 1947 Life-magazine register.
* EV-1947-07-08-roswell-incident.png — debris field in NM
desert, USAAF officer in 1947 uniform examining foil
fragments, period staff car.
* EV-1947-06-21-maury-island-incident.png — wooden patrol
boat on Puget Sound, 6 doughnut craft hovering, one
shedding glowing slag, Harold Dahl + son + dog watching.
- app/e/[cls]/[id]/page.tsx: full-bleed editorial hero replaces
the old gradient header card when an illustration exists for
that entity_id. Title sits over the painting with gradient
overlay. "Ilustração editorial" chip in the top-right.
Quota note: Claude OAuth still rate-limited as of this commit, so
Sun-Tzu hasn't been smoke-tested in production. Code is shipped and
ready; first brief will land when the weekly quota refreshes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hero illustration:
- Painterly 16:9 editorial illustration generated via Nano Banana Pro
for the featured case (green-fireballs-narrative): late-1940s
desert night, vivid emerald fireball over silhouetted Sandia mesas,
1948-era state-police sedan parked on US 66 shoulder with an
officer in period uniform looking up, faint green glow on his
face. Sandia Base 5 Miles roadsign. New Yorker-cover painterly
register, NOT photorealistic, NOT sci-fi.
- Stored at /data/disclosure/processing/case-art/<slug>.png, served
through the existing /api/static/processing/ route. 2.7MB at 2K.
- components/featured-case.tsx: prefers the illustration over the
declassified-page thumbnail when present. Tags it "Editorial
illustration" / "Ilustração editorial" so the reader knows it's
not a photograph.
- app/c/[slug]/page.tsx: full-bleed editorial hero at the top of
the article when an illustration exists for the slug. Title sits
on the image with gradient overlay; "Ilustração editorial" chip
in the top-right corner labels the art honestly. When no
illustration exists the page falls back to the plain title header.
Sitemap fix:
- Added export const dynamic = "force-dynamic" + revalidate = 3600
to app/sitemap.ts. Without these Next.js statically generated the
sitemap at build time, when the DB and case-files volume were
unreachable from the build container — which is why production
was serving only the 9 static URLs instead of ~3000.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GEO/SEO surface area:
app/robots.ts (new — Next.js dynamic robots)
Explicitly ALLOWS major AI crawlers: GPTBot, OAI-SearchBot,
ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot,
Perplexity-User, Google-Extended, Applebot-Extended, CCBot,
DuckAssistBot, YouBot, Bytespider, Amazonbot. The site exists to
be cited by LLMs answering UAP/UFO questions — we want them in.
/api/admin/, /admin/, /auth/ disallowed for everyone.
app/sitemap.ts (new — Next.js dynamic sitemap)
Lists 9 top-level routes + every /d/<doc> + every /c/<slug> from
the filesystem + up to 500 entity URLs per class
(event, person, uap_object, location, organization),
sorted with summary-enriched entities first. ~3000 URLs total at
current corpus size. lastModified honours summary_generated_at so
crawlers re-index when entities are re-enriched.
app/c/[slug]/page.tsx (rewritten — magazine reading view)
- generateMetadata: per-case title, description (auto-extracted
from the locale-preferred lead paragraph), canonical URL,
hreflang alternate, OpenGraph article type with publishedTime,
Twitter card.
- JSON-LD Article schema embedded at end of page: schema.org
Article + Organization publisher + inLanguage + isAccessibleForFree.
This is what makes the case appear as a citable source in
Google AI Overviews / Perplexity / ChatGPT search.
- Reading view rewritten: display-serif headline (Fraunces), italic
blockquotes with gold accent, prose-typography styling, no more
detective stats line, no more "written by case-writer@detective"
attribution. Locale-aware: PT-BR pulls topic_pt_br + lead in PT,
English mirror.
tailwind.config.ts
+ @tailwindcss/typography plugin
+ font-display family wired to var(--font-display) (Fraunces)
package.json
+ "@tailwindcss/typography" devDependency
Phase 3A note: bulk entity enrichment hit Claude OAuth weekly quota mid-run.
6 events + 3 uap_objects landed bilingual summaries before the quota
exhausted. UI gracefully splits enriched vs bare entities so /sightings
shows the magazine-grade cards (Kenneth Arnold 1947, Roswell, Maury Island,
Joseph Perry 1960 lunar photo, Civil Defense Director 1966, etc.) on top
of a compact table of the rest. Re-run when quota refreshes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Today /sightings, /witnesses, /objects, /locations and /operations show
a name + mention count and nothing else. After this each row carries a
60-100 word bilingual narrative summary written from the chunks where
the entity actually appears.
Migration 0008 (apply as supabase_admin):
public.entities +summary_en TEXT
+summary_pt_br TEXT
+summary_generated_at TIMESTAMPTZ
+summary_model TEXT
+summary_status TEXT
CHECK ('pending'|'ai_generated'|'curated'|'refused')
+ index on summary_status
+ GRANT UPDATE (summary_*) ON entities TO investigator
+ new policy entities_investigator_update_summary (RLS UPDATE for
investigator role)
Enrichment script (investigator-runtime/scripts/enrich_entity_summaries.ts):
- Per-class config (chunk_k, min_mentions, max_per_class)
- Path A: entity_mentions JOIN chunks (high-precision linker)
- Path B (fallback): hybridSearch on canonical_name + aliases when
entity_mentions returns zero. This is what unlocked Kenneth Arnold
and similar entities — their wiki YAML has high total_mentions
counted from frontmatter mentioned_in[], but the entity_mentions
extractor was silent because the matches came from the wiki text,
not the OCR chunks.
- Sonnet 4.6 via OAuth Max, ~$0.04 per entity, ~$10 for the full
260-entity bulk run.
- INSUFFICIENT skip when chunks can't sustain a 60-word summary —
refused entries get summary_status='refused' so they're not retried.
UI uplift:
- lib/retrieval/entity-pages.ts: getEntityCore now prefers the DB
summary (ai_generated or curated) over wiki YAML narrative.
- components/entity-list-page.tsx:
* SELECT now pulls summary_en, summary_pt_br, summary_status
* Sorted with summary-enriched rows first (so the magazine grid
lands on quality content immediately)
* MagazineGrid: 4-line summary preview replaces aliases line
* CompactGrid: enriched rows render as full editorial cards,
bare rows fall back to a compact table below
Smoke results:
- Kenneth Arnold sighting: "On June 24, 1947, pilot Kenneth Arnold
reported sighting unidentified objects over the Pacific Northwest,
and the account spread worldwide. It set off a run of similar
reports: County Commissioner Crankes saw comparable objects after
Arnold's account reached the press, and United Airlines pilot
Emil H. Smith spotted flying discs on July 4 during a routine
flight out of Boise, Idaho..."
- Roswell Incident: includes Colonel Corso's 1997 book + the 1995
GAO finding that radio messages from Oct 46–Feb 47 were destroyed
+ Senator Strom Thurmond's foreword. Real magazine-grade content.
Background bulk run kicked off across all 5 classes (event,
uap_object, person, location, organization) — populating live as
the homepage rebuilds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User: "shouldn't mention the names of the mind-clones, should merge all
analyses and write like a best-seller author would, about what happened."
Voice rewrite (prompts/case-writer.md):
- Reference voices: Erik Larson, Sam Kean, John McPhee, Mark Bowden.
Plainspoken non-fiction, scene-driven, fascinated.
- One narrator. NEVER say "Sherlock Holmes argues" / "Sun-Tzu builds
the case" / "the team concluded". No internal-process names reach
the reader.
- Hook the first paragraph. Open in a scene with a date, place, and
person doing something specific. NOT "This case investigates..."
- Show, don't argue. Verbatim quotes stay source-language in
blockquotes; the narration around them is the narrator's voice.
- Every claim cites a chunk with [[doc-id/pNNN#cNNNN]].
- Forbidden ceremony: "In summary…", "Em suma…", "Ultimately…",
"It is worth noting…", detective names, probability tables,
hypothesis tournaments.
- The honest unknown is the subject, not a failure: "Whatever was in
the sky over Sandia in December 1948, the government never said."
- 4-6 numbered scenes, each title-cased specifically ("The Green
Sphere Over Highway 60" not "Background").
- Bilingual EN + PT-BR per CLAUDE.md §3 — sections alternate, no
mid-paragraph language mixing.
- Refusal: emit INSUFFICIENT_ARTEFACTS rather than padding when the
corpus is thin.
Raw-material pipeline (src/detectives/case_writer.ts):
- hybridSearch(topic, lang, top_k=18) gives the narrator real corpus
scenes with verbatim text + chunk_id citations + bbox metadata.
This is what was missing — v1 only saw pre-digested hypothesis
artefacts, which is how the academic prose got there.
- Dropped the hypotheses + contradictions queries from the loader.
They were skeptic-framing scaffolding that doesn't belong in the
raw material a best-seller narrator works from.
- New buildPrompt sections: "Primary-source scenes", "Curated
verbatim quotes", "Anomalies and surprises", "Named witnesses".
Anomalies (Taleb's outlier gaps) reframed: drop dominant_model
skeptic baseline, keep title + why_surprising as gold material.
- Refusal floor: < 4 scenes from hybridSearch → skip with reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User explicit: "1 bilhão de entusiastas pelo mundo ovni" — site is for the
UFO-curious public, not for skeptics. The 8-detective scaffolding becomes
invisible plumbing; the reader sees stories about what was observed.
Reader-facing changes:
New homepage (web/app/page.tsx)
- SiteHeader: magazine-style top nav (no detective tiles)
- HeroBanner: full-bleed editorial opener with declassified-page art
background, display-serif headline, live stats row (122 docs,
2047 events, 1861 witnesses, 867 craft catalogued)
- FeaturedCase: cover-story treatment of the most recent case_report,
uses a real document page as hero image, links to /c/[slug]
- PortalGrid: 6 thematic doorways into the archive — Sightings,
Witnesses, Craft, Hot spots, Programs, Documents — each tile shows
a real entity count and short editorial blurb
- GreatestHits: top 9 most-cited events from the corpus
(Kenneth Arnold 1947, Mantell 1948, …) as a magazine grid
- Doc list kept but reframed as "the primary record"
New sub-pages (5)
- /sightings → events (2047), magazine grid
- /witnesses → people (1861), compact table
- /objects → uap_objects (867), magazine grid
- /locations → locations (1757), compact table
- /operations → organizations (1596), compact table
- /documents → full doc list with thumbnails (mirrors homepage section
for direct deep-link)
All share <EntityListPage> shell with per-page i18n + JSON-LD ItemList
Stripped detective surfacing
- /jobs/[id]: "Sherlock Holmes / Dr. Watson" → "Investigation in progress"
- chat-bubble: detective-named card → neutral "Investigação em andamento"
- quick-launch: 7-kind detective dropdown → single "investigar um caso"
input (kind=case_report hardcoded)
- /bureau: rewritten as the case-file library (no artefact dumps)
Typography + design
- Fraunces variable serif loaded for display headings
(`.font-display` class)
- Gold-amber accent (#e0c080) unified as the brand colour
- Asymmetric magazine grids (1+2+3 column, generous whitespace)
- Hover micro-interactions (image scale on featured case, translateX
on portal arrows)
SEO + GEO
- layout.tsx metadataBase + title.template + per-route Metadata exports
- Organization JSON-LD on root layout
- WebSite + SearchAction JSON-LD on homepage
- CollectionPage + ItemList JSON-LD on every entity list page
- openGraph + twitter cards, pt-BR primary + en-US alternate
- ai:purpose meta tag for Generative Engine Optimization — declares
the site as a citation-linked primary-source archive
- robots: index + follow with large image preview
The detectives themselves remain alive in the backend (runtime, DB, audit
log), but the reader never sees "Holmes / Sun-Tzu / Watson" in the UI. The
next phase will reorient case-writer to write as a single best-seller voice
synthesising all the internal sources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Live failure surfaced by user feedback: Poirot wrote a low-credibility
verdict on J. Edgar Hoover (W-0002) based on 1 actual chunk and 11
entity_mentions false positives where 'DIRECTOR'/'DIRETOR' was linked to
him by mistake. Poirot's own bias_notes correctly identified this — yet
still produced a verdict. Published on a 'Disclosure Bureau' site, that's
libellously misleading.
Deleted W-0001 (Donald Keyhoe) and W-0002 (J. Edgar Hoover) from
public.witnesses + their .md files.
Prompt rewrite (prompts/poirot.md):
- New "What counts as testimony" section up front, before discipline.
Direct testimony = the person AUTHORED, was QUOTED verbatim with
attribution, or GAVE testimony in a recorded hearing. Not: third-
party mentions, generic title appearances ('Director'/'Diretor'
that entity-extraction speculatively linked), CC lines.
- HARD FLOOR rule: emit `direct_testimony_chunk_ids[]`. If < 3, refuse
with INSUFFICIENT_TESTIMONY. For famous historical figures
(Wikipedia-worthy public figures) the floor is 5.
- Bias claims MUST cite a specific chunk; ungrounded bias claims drop.
- Tone: "careful prosecutor preparing a brief, not debunker scoring
points."
Defense in depth (poirot.ts):
- Detective enforces the same floor before calling writeWitnessAnalysis,
using a FAMOUS slug list (j-edgar-hoover, donald-keyhoe, j-allen-
hynek, curtis-lemay, vannevar-bush, eisenhower, truman, kennedy,
ted-bloecher, ...).
- When the floor isn't met, emit `poirot_refused_floor` audit event +
skip with reason like `insufficient_direct_testimony_1_of_5`.
- Sentinel parser now also catches INSUFFICIENT_TESTIMONY when it
appears on the first line of an otherwise-prose response.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two complaints in one wave:
(W4.1) User: "Não pode ter vícios de IA como uso excessivo de '-' que a IA
coloca geralmente no lugar de vírgulas por exemplo. Isso deve fazer parte
do prompt geral."
- New prompts/_house-style.md banning the 9 most common AI prose tells
in both EN and PT-BR:
1. Em dashes as comma replacements (—)
2. Rule-of-three lists ("concrete, rigorous, and grounded")
3. Conjunctive openers ("Moreover", "Notably", "Ademais")
4. Superficial -ing analyses ("marking a shift", "destacando")
5. Inflated symbolism + AI vocab (tapestry, navigate, delve,
underscore, robust, multifaceted, marco histórico, ...)
6. Negative parallelisms ("Not just X but Y")
7. Vague attribution ("Some scholars say...")
8. Summary closers ("In summary...", "Em suma...")
9. Hedging fluff ("It's important to note...")
Verbatim chunk quotes are explicitly exempt; preserve as-is.
- claude.ts callClaude() lazily loads _house-style.md once per process
and PREPENDS it to every detective's system prompt:
composedSystem = houseStyle + "---" + detective.systemPrompt
This means all 7 detectives + future ones get the rules without any
per-prompt change.
(W4.2) User: "Quando entra em uma página da investigação não tem como
voltar! UX terrível!"
- New <BureauNav> sticky topbar with explicit "← home" + "🔎 bureau"
buttons + clickable breadcrumb trail. Always visible at the top of
every bureau page so the user can escape in one click.
- Wired into /bureau, /h/[hypothesisId], /c/[slug], /jobs/[id]. Each
page passes its sensible parent crumb (/bureau#hypotheses,
/bureau#reports, /bureau#jobs).
- Replaces the previous plain-text "disclosure.top / hypothesis /
H-0004" line which had no visual affordance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Live PT-BR smoke on j-edgar-hoover produced verdict_pt_br at 304 chars
(prompt says ≤ 280). The writer correctly rejected it ("verdict too long
(304 > 280)") but the job failed instead of trimming.
Fix: detective now trims each language field at the nearest sentence
boundary (period or semicolon) above 60% of the cap; falls back to a hard
cut at the cap. Applied to verdict / verdict_pt_br (≤280), and to
access_to_event*, bias_notes* (≤800) for defense in depth.
The contract with the writer stays strict; the detective just becomes
forgiving about the model going 5-10% over.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User flagged that the bureau was emitting English-only output, violating
the project's bilingual rule. Every narrative field now ships in both
languages: stored in sibling DB columns + rendered as adjacent markdown
sections per CLAUDE.md §3.
Migration 0007 (apply as supabase_admin):
- public.hypotheses +question_pt_br, +position_pt_br,
+argument_for_pt_br, +argument_against_pt_br
- public.contradictions +topic_pt_br, +notes_pt_br
- public.witnesses +access_to_event_pt_br, +bias_notes_pt_br,
+verdict_pt_br
- public.gaps +description_pt_br, +suggested_next_move_pt_br
- public.evidence: unchanged (verbatim_excerpt stays source-language)
- JSONB siblings inside contradictions.chunks + gaps.scope handled at
runtime (statement_pt_br, title_pt_br, dominant_model_pt_br,
why_surprising_pt_br, what_it_implies_pt_br).
Detective prompts (all 7) rewritten with explicit bilingual JSON contract:
- Output protocol section names every EN field + its _pt_br sibling
- "Bilingual is mandatory" warning in the task instruction
- Sentinel skip-states unchanged (NO_HYPOTHESES, NO_CONTRADICTIONS,
INSUFFICIENT_TESTIMONY, INSUFFICIENT_HYPOTHESIS, NO_OUTLIERS,
NO_NEW_EVIDENCE, INSUFFICIENT_ARTEFACTS)
- Schneier: parallel arrays — hidden_assumptions[i] matches
hidden_assumptions_pt_br[i], lengths must match
- Case-Writer: interleaved §1 (EN) / §1 (PT-BR) per act in the body
Writer-side validation (all 7 tools):
- Reject INSERT if PT-BR sibling missing when EN field is set
- Persist both languages atomically in one INSERT (no half-updates)
- Markdown renderers write adjacent EN+PT-BR sections in case files
(## Argument for (EN) followed by ## Argumento a favor (PT-BR), etc.)
Detective parse layer (all 7 detectives):
- Coerce both keys from JSON output
- "incomplete_bilingual_*" skip reason when either side missing
- Defensive: PT-BR fields trimmed + length-capped same as EN
Orchestrator propagates question_pt_br + topic_pt_br through job payload
to runHolmes / runCaseWriter, mirroring the chat-tool entry point.
Web (UI):
- /api/jobs/[id] hydrates _pt_br siblings from pg
- job-status-poller HypothesisCard: PT-BR primary, EN in <details>
fallback when both exist
- ContradictionCard: PT-BR statement primary + secondary EN quote
- WitnessCard: PT-BR verdict primary + secondary EN quote, panels in PT
- GapCard: PT-BR title/why/implies primary
- /bureau hub: SELECTs both columns, renders PT-BR primary
- /h/[id]: ArgumentPanel renders PT-BR primary with collapsible EN
fallback when both exist
- BureauSnapshot homepage: position_pt_br / topic_pt_br / verdict_pt_br
primary
- DocBureauPanel /d/[doc]: same primary-PT-BR pattern
- New web/lib/i18n/pick.ts helper (unused yet by chat/agents — kept
for future locale-driven switching when both languages are equally
full; current rule is PT-BR-first since the user is brasileiro)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Builds on top of W3.9 to turn the homepage Bureau from a read-only
dashboard into a working command center.
UI improvements (web/components/bureau-snapshot.tsx):
- Detective tiles are now <Link>s — each navigates to its primary
artefact section in /bureau (Holmes→#hypotheses, Locard→#evidence,
Dupin→#contradictions, Schneier→#hypotheses, Poirot→#witnesses,
Taleb→#outliers, Tetlock→#hypotheses, Case-Writer→#reports). Hover
bg matches the detective's tone color.
- <QuickLaunch /> form inserted right under the tiles.
New <QuickLaunch /> client component:
- Detective dropdown (7 active kinds; evidence_chain not yet exposed
here since it needs a doc_id better picked from the doc page).
- Single input swaps placeholder + aria-label by kind: question for
Holmes, topic for Dupin/Taleb/Case-Writer, hypothesis_id for
Schneier/Tetlock, person_id for Poirot.
- Submits to POST /api/bureau/launch and redirects to /jobs/[id]
via the next.js router.
- Loading state ("queueing…") + error display inline.
POST /api/bureau/launch (web/app/api/bureau/launch/route.ts):
- Same 8-kind validator as the chat tool's request_investigation.
- Auth required when Supabase is configured (triggered_by = user:email).
- Returns { job_id, kind, detective, status_url, eta_seconds }.
DocBureauPanel on /d/[docId] (web/components/doc-bureau-panel.tsx):
- Server component inserted between the doc header and
AnomalyHighlights.
- Surfaces every bureau artefact that touches the doc:
· Evidence whose source_page_id starts with docId/p
· Hypotheses citing any of those evidence_ids
· Contradictions whose chunks[] has any item with this doc_id
· Gaps/outliers with scope.doc_id == docId
· Case reports whose markdown body references docId (filesystem scan)
- Empty state shows "Investigation Bureau — untouched" with a CTA
linking back to the homepage to launch the first investigation.
- When non-empty, header counts total artefacts + links to /bureau
for the full view.
Metadata (web/app/layout.tsx):
- description rewritten from "Investigative wiki of the US Department
of War UAP/UFO archive (war.gov/ufo)" to one that names the bureau
+ the 8 detectives. Affects SERP previews + social-card defaults.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/c/[slug] returned 404 even after the W3.9 web rebuild because the web
container's volume list didn't include the case/ directory the
investigator-runtime writes to. The BureauSnapshot file-listing for
Case reports gracefully fell back to empty, but /c/<slug> can't fall
back: it has to read the markdown.
Fix:
- Mount ${CASE_ROOT:-/data/disclosure/case}:/data/ufo/case:ro (read-only,
same pattern as wiki/processing/raw).
- Set CASE_ROOT=/data/ufo/case env in the web container so the
/c/[slug] page and BureauSnapshot resolve the same path.
Verified live: /c/green-fireballs-sandia now serves HTTP 200 with the
Watson narrative parsed + rendered via MarkdownBody.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes a UX gap the user surfaced: W3.5-3.8 built 8 detectives, 4 new
URL endpoints (/jobs/[id], /h/[id], /c/[slug], /api/h/[id]/red-team)
and a chat tool, but the homepage was unchanged — the bureau was
invisible unless you knew the URL or asked the chat to invoke
request_investigation.
Homepage (web/app/page.tsx):
- Title `▍ war.gov/ufo — Investigative Wiki` → `▍ The Disclosure Bureau`
- Subtitle expanded from "Holmes · Poirot · Dupin · Locard" to all 8
detectives (Holmes · Locard · Dupin · Schneier · Poirot · Taleb ·
Tetlock · Case-Writer)
- New `🔎 bureau` topbar link (gold, between graph/stats and batch)
- BureauSnapshot inserted right after the header
BureauSnapshot (web/components/bureau-snapshot.tsx) — server component:
- 8 detective tiles with role labels (each in its tone color)
- 6 clickable counters (evidence / hypotheses / contradictions /
witnesses / outliers / case reports) — anchor to /bureau#section
- 6 "recent artefacts" columns surfacing the last 3-4 of each kind:
hypotheses with prior→posterior + band + ↳reviewed_by marker,
contradictions with topic + resolution_status, evidence with
Grade badge + verbatim quote, outliers with title + scope.kind,
witness analyses with canonical_name + credibility + verdict,
case reports with slug + link to /c/<slug>
- "Recent jobs" strip linking to /jobs/[id] color-coded by status
- Reports read from /data/ufo/case/reports/ via fs.readdir + stat,
sorted by mtime — no DB round-trip needed for that section
/bureau (web/app/bureau/page.tsx) — full hub:
- Header with full counts
- 7 sections (anchored to homepage counter links): Case reports,
Hypotheses, Evidence, Contradictions, Outliers, Witnesses,
Recent jobs table — each rendering up to 100 rows
- Reports section parses frontmatter from each .md to surface topic
+ n_hypotheses + n_evidence on the card
Runtime fixes batched in:
- Poirot: coerce entity_pk via Number() — node-postgres returns
BIGINT as string by default; writer's Number.isFinite() rejected
it as "person_entity_pk required" (j-edgar-hoover retry path)
- Tetlock: write_calibration rationale cap 600 → 1200 chars. Prompt
still asks ≤ 600 but a 2× slack beats failing the job on honest
analysis. Observed live: Tetlock emitted ~620 chars on H-0003 and
the writer rejected the entire calibration.
- Case-Writer: Promise.all of 5 queries × max_parallel=2 jobs
demanded up to 10 connections against the investigator role's
rolconnlimit=4 → "too many connections for role investigator".
Sequentialized — the LLM call is the hot path, not these queries.
Smoke results visible now on the homepage:
- 3 hypotheses (H-0001/2/3) about green fireballs origin
- 3 contradictions (R-0001/2/3) about color, geographic confinement,
exclusive-green vs multicolored
- 2 evidence cards (E-0002/3) Grade B
- 3 outliers (G-0001/2/3) — including Taleb's deliberate
meteor-shower-camouflage flag
- 1 case report at /c/green-fireballs-sandia (Watson 13.4 KB,
five-act narrative, fully cited)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the bureau from 4 → 8 detectives. All eight run as Bun + claude-CLI
subprocesses against the same Supabase + investigation_jobs LISTEN/NOTIFY
queue, sharing search.ts hybridSearch and writer-side validators that
gate writes against schema + FK.
New detectives:
Poirot (witness_analysis)
- prompts/poirot.md — credibility / access / bias / corroboration /
verdict; uses entity_mentions JOIN chunks to pull 12 chunks per
person; resolves corroboration_refs chunk_ids defensively (accepts
bare cNNNN even when the model emits pNNN/cNNNN).
- INSERT into public.witnesses with W-NNNN naming.
- Tone: purple (#9b5de5).
Taleb (outlier_scan)
- prompts/taleb.md — "surprise is relative to a model"; at most 3
outliers; each requires explicit dominant_model + why_surprising +
what_it_implies; fan-out into public.gaps with scope.kind="outlier".
- Same unscoped-fallback as Dupin (Pass 1 with doc_id, Pass 2 widens
to corpus if hits < 3).
- Tone: yellow (#ffd23f).
Tetlock (calibrate_hypothesis)
- prompts/tetlock.md — honest Bayesian update; emits new_posterior +
Δ + recommended_action ∈ {keep, downgrade, upgrade, supersede}.
- write_calibration UPDATEs public.hypotheses + APPENDS a
"## Calibration history" section to the H-NNNN.md case file
(calibration is append-only — each datapoint matters). Posterior
band auto-corrected to match Tetlock thresholds.
- NO_NEW_EVIDENCE sentinel handled; pure 'keep' with |Δ|<0.005 only
touches updated_at + reviewed_by.
- Tone: teal (#26d4cc).
Case-Writer (case_report)
- prompts/case-writer.md — Dr. Watson assembles all artefacts
(E-NNNN, H-NNNN, R-NNNN, W-NNNN, G-NNNN) into a five-act narrative.
ILIKE filter on topic; doc_id optional scope.
- Larger budget cap (≥ $0.50) + longer timeout for prose generation.
- Writes case/reports/<slug>.md with frontmatter (topic + counts);
no DB table for v0.
- New page /c/[slug] renders the report via MarkdownBody + stat chips.
- Tone: gold (#e0c080).
Hardening across the bureau:
- Sentinel parsing now accepts backticked AND prose-trailing forms
(Holmes NO_HYPOTHESES, Dupin NO_CONTRADICTIONS, Schneier
INSUFFICIENT_HYPOTHESIS, Poirot INSUFFICIENT_TESTIMONY, Taleb
NO_OUTLIERS, Tetlock NO_NEW_EVIDENCE, Case-Writer
INSUFFICIENT_ARTEFACTS). Avoids the failure mode where the model
refuses honestly but the runtime treated it as a parse error
(observed live with Poirot+Hoover identifying the DIRECTOR
false-positive disambiguation issue in entity_mentions).
Chat tool extensions (web/lib/chat/tools.ts):
- request_investigation now accepts 7 kinds. Each routes to its
detective with appropriate validation (hypothesis_id regex,
person_id kebab-case, topic non-empty, doc_id for evidence_chain).
- ETA per kind: Holmes/Dupin 60s, Poirot 45s, Schneier/Tetlock 30s,
Taleb 50s, Case-Writer 180s (longer prose), Locard 30×n_chunks.
UI integration:
- chat-bubble inline card paints each detective in its tone color.
- /jobs/[id] page header swaps name/subtitle/tone per detective;
question label adapts ("Topic" / "Hypothesis under attack" /
"Witness under analysis" / "Topic to outlier-scan" / "Hypothesis
under recalibration" / "Case to assemble").
- job-status-poller renders: case-report link card (gold), outlier
cards (yellow), witness cards (purple) — alongside existing
hypothesis, evidence, contradiction cards.
- /api/jobs/[id] hydrates witnesses (JOIN entities for canonical_name)
+ gaps (with scope JSONB).
- /c/[slug] page reads /data/ufo/case/reports/<slug>.md and renders
with MarkdownBody, frontmatter parsed for stat chips.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the fourth AI detective in the Investigation Bureau runtime: Bruce
Schneier, who attacks an existing hypothesis as a red-team operator.
Runtime:
- prompts/schneier.md — discipline (don't disprove, just attack;
structured output with hidden_assumptions, failure_modes,
alternative_explanations, recommended_tests, verdict_one_sentence;
severity ∈ {low, medium, high}; emit INSUFFICIENT_HYPOTHESIS when
the input is too thin)
- src/detectives/schneier.ts — reads the hypothesis row + evidence
chain (joined via evidence_refs FK), feeds Claude with the
arguments + verbatim quotes, parses strict JSON object
- src/tools/write_red_team_review.ts — UPDATEs hypotheses.reviewed_by
+ updated_at; APPENDS (or replaces if re-reviewed) a structured
"## Red-team review (Schneier · X severity)" section to
case/hypotheses/H-NNNN.md. Caps each list at 5 entries × 240 chars,
validates verdict ≤ 280 chars.
- orchestrator: new `red_team_review` kind dispatching to runSchneier
Chat + UI:
- request_investigation gains kind=red_team_review + hypothesis_id arg
(validated against H-NNNN regex); detective auto-resolves to schneier
- chat-bubble inline card paints Schneier in red (#ff3344)
- /jobs/[id] page swaps title/subtitle/tone per detective; the
"Question" label becomes "Hypothesis under attack" for red_team_review
New /h/[hypothesisId] page (hypothesis dossier):
- Server-rendered from public.hypotheses + public.evidence (joined
via evidence_refs FK + chunk lookup)
- Header: ID + creator + reviewer (highlighted when Schneier has
visited), position as headline, question subtitle, Tetlock band
- Prior + posterior bars with Δ-delta indicator
- Argument grid: argument_for (green) vs argument_against (pink)
side-by-side with [[wiki-link]] auto-linking to source chunks
- Evidence chain: each E-NNNN with Grade A/B/C badge, verbatim
blockquote, link to source page
- Red-team review panel: parses the markdown section in the case
file (severity badge, verdict, 4 bullet panels for
hidden_assumptions / failure_modes / alternative_explanations /
recommended_tests). Empty state when not yet reviewed.
RedTeamRequestButton client component + POST /api/h/[id]/red-team —
authenticated user can trigger Schneier in one click; UI swaps to
"acompanhar" link to /jobs/[id] once queued.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two regressions surfaced in the smoke test that put Dupin from
0/3 contradictions written → 3/3 in the next run.
1. Single-doc scope was too narrow for Dupin's task.
Holmes's question about Sandia returned 4 chunks scoped to one doc,
but Dupin's terser "topic" form yielded only 1. Solution: Pass-1
tries the requested doc_id; if the head is < 2 chunks, Pass-2
widens to the whole corpus. Audit event carries `scope_widened`
so the case-writer can later flag cross-doc contradictions
distinctly. The unscoped retry hit 9 chunks and produced 3
contradictions across 3 different docs.
2. Chunk-block header was ambiguous to the model.
`--- doc-id/p007#c0042 ---` led Claude to parse `chunk_id` as
"p007#c0042" or "p007/c0042" in the JSON output. write_contradiction
then refused the FK lookup with "chunk not found". Fix:
- Explicit `doc_id:` / `chunk_id:` / `page:` lines per chunk
in the rendered block (no slashes/hashes the model can fold).
- Defensive normalizeChunkId() in write_contradiction.ts strips
any pNNN prefix and keeps only the trailing cNNNN — so the
writer is forgiving without losing strictness on the topic +
statement validation.
Smoke now produces (job 6deddf4b):
R-0001 (3 chunks) — Color of the fireball(s) in incident summaries
R-0002 (2 chunks) — Geographic confinement of green-fireball sightings
R-0003 (3 chunks) — Whether the phenomenon was exclusively green or
also red/multicolored
R-0003 connects 3 different declassified documents: the Los Alamos
conference (exclusively-green category), a retrospective document
(red OR green), and Incident 229 (red, blue, yellow — no green).
Real cross-doc contradiction, fully cited.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the third AI detective in the Investigation Bureau runtime: C. Auguste
Dupin, who scans a corpus shortlist for pairs (or small groups) of chunks
that cannot both be true under any ordinary reading.
Runtime:
- prompts/dupin.md — discipline (no contradiction without ≥2 distinct
chunk_ids; reject same-vocabulary near-misses; FEW high-confidence
over MANY weak ones; emit `NO_CONTRADICTIONS` when corpus is silent)
- src/detectives/dupin.ts — hybridSearch with k=18 (more chunks than
Holmes because contradictions emerge from comparing dispersed
claims), strict JSON-array parsing, AT MOST 3 contradictions per call
- src/tools/write_contradiction.ts — validates topic + ≥2 positions
drawn from ≥2 distinct chunks, resolves chunk_pk via DB lookup
(rejects positions citing unknown chunks), INSERTs into
public.contradictions + writes case/contradictions/R-NNNN.md
- orchestrator: new `contradiction_scan` kind dispatching to runDupin;
payload { topic, doc_id?, lang?, context_chunks? }
Chat + UI:
- request_investigation gains kind=contradiction_scan + topic arg;
triggered detective auto-resolves to dupin
- chat-bubble inline card renders dupin in orange (#ff8a4d) to
distinguish from holmes (cyan) and locard (green)
- /jobs/[id] page swaps title + subtitle + tone per detective;
"Question" label becomes "Topic" for contradiction_scan
- /api/jobs/[id] hydrates public.contradictions when outputs[] surfaces
contradiction_ids
- job-status-poller renders ContradictionCard: topic + N positions
(verbatim statements quoted, stance label optional, link to source
chunk) + optional notes panel, with resolution_status badge
(open/resolved/irreconcilable)
R-NNNN shares the contradiction_id_seq slot with relation per
CLAUDE.md naming — same conceptual class (a connection between two
pieces of evidence in tension).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the loop between the chat UI and the Investigation Bureau runtime.
Chat tool (web/lib/chat/tools.ts):
- request_investigation { kind, question, doc_id?, chunks?, claim? }
INSERTs a row in public.investigation_jobs and returns
{ job_id, kind, status, eta_seconds, status_url, detective }.
- kind=hypothesis_tournament → Holmes (1 question → 2-3 rival hypotheses)
- kind=evidence_chain → Locard (1 doc → grade-A/B/C evidence with chain
of custody, default top-5 anomaly chunks)
- Plumbed user.email through ToolHandlerContext so triggered_by audits
the requesting user.
Public job viewer:
- GET /api/jobs/[id] joins investigation_jobs → public.evidence +
public.hypotheses for the IDs surfaced in outputs[]. Returns one
payload the page can render without n+1 round-trips. Strips
triggered_by from the response (it carries the user's email).
- app/jobs/[id]/page.tsx server-renders the case-file shell:
detective lore header (Holmes blue or Locard green), question chip,
scope chip with link back to the document.
- components/job-status-poller.tsx client island that polls every 3 s
while non-terminal, then once on terminal to hydrate evidence +
hypotheses. Renders:
· Phase tracker (queued → running → complete | failed)
· Hypothesis cards w/ prior + posterior bars + Δ delta indicator
+ Tetlock band badge (high/medium/low/speculation)
· Argument-for / argument-against with [[wiki-link]] auto-linking
to /d/<doc>/p<NNN>#<cNNNN>
· Evidence cards w/ Grade A/B/C badge + verbatim blockquote +
bbox crop preview via /api/crop + custody-steps disclosure
· Empty/in-flight panel ("os detetives estão lendo o corpus")
· Failure panel surfacing error + partial outputs
Inline chat-bubble card (components/chat-bubble.tsx):
- ToolTrace.richRender recognises request_investigation results and
renders a detective banner with status + ETA + link to /jobs/[id]
(target=_blank). Error case renders a red strip with the message.
UX flow now: user asks Sherlock a question → request_investigation
queues the job → chat card shows "🔎 Holmes · hypothesis_tournament ·
ETA ~60s" → user clicks → /jobs/<id> live-updates → 60 s later, 2-3
rival hypotheses + their arguments + chunk citations are rendered with
Bayesian update visible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the second AI detective in the Investigation Bureau runtime: Sherlock
Holmes, who builds 2-3 rival hypotheses with calibrated priors + posteriors
against a corpus shortlist.
Pipeline:
1. hybridSearch() grounds Holmes with 8-15 chunks via the same
hybrid_search_chunks RPC the web uses (BM25 + dense + RRF). Default
max_dense_dist=0.55 (runtime favors recall over precision; web's
/api/search/hybrid stays at 0.40 for chat).
2. claude-sonnet-4-6 emits a strict JSON array with position +
argument_for + argument_against + prior + posterior + confidence_band
+ evidence_refs. Citations use [[doc-id/pNNN#cNNNN]] wiki-links.
3. writeHypothesis() validates posterior ∈ [0,1], auto-corrects the
Tetlock band from the posterior (high ≥0.90, medium 0.60-0.89,
low 0.30-0.59, speculation <0.30), checks evidence_refs FK against
public.evidence, INSERTs into public.hypotheses + writes
case/hypotheses/H-NNNN.md.
Discipline guarantees (prompts/holmes.md):
- posteriors across rivals sum to ≈1.0
- no claim without chunk citation
- prefer lower band when ambiguous (anti-inflation)
- declarative one-sentence position, no hedging
- emit `NO_HYPOTHESES` when corpus is silent (refuses to fabricate)
Smoke test (Sandia green fireballs 1948-49):
- H-0001 prior 0.5 → posterior 0.2 (speculation): natural meteoric
- H-0002 prior 0.3 → posterior 0.4 (low): classified weapons / tests
- H-0003 prior 0.2 → posterior 0.4 (low): genuinely unidentified
Bayesian update visible: "natural meteoric" prior dropped 60%; both
rivals climbed. 4 unique chunk citations across the 3 hypotheses.
orchestrator dispatches `hypothesis_tournament` kind via runHolmes;
job marked `failed` if all rivals error, `complete` otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Token consolidation:
- docker-compose web service now reads ${CLAUDE_CODE_OAUTH_TOKEN} directly,
drop the W1-F8 CLAUDE_CODE_OAUTH_TOKEN_FOR_WEB indirection (user feedback:
one var name, no _FOR_WEB suffix).
investigator-runtime claude.ts:
- --system-prompt silently dropped by CLI v2.1.150 for multi-KB prompts;
inline the system content into the user prompt with a separator
(mirrors scripts/reextract/run.py pattern).
- Multi-line prompts via positional -- broke ("Input must be provided …");
pipe via stdin instead.
- --allowedTools "" is rejected; when no tools wanted, omit it and explicitly
--disallowedTools the writer/reader set so the model can't reach for any.
investigator-runtime locard.ts:
- Log the raw response (first 600 chars) to container stderr — saved hours
of debugging when the writer rejected.
- Grade fallback: when Locard omits `grade` but provides custody_steps,
infer the highest grade that fits (≥3 → A, ≥2 → B, ≥1 → C).
investigator-runtime write_evidence.ts:
- Filter related_hypotheses entries with empty/null hypothesis_id silently
(Locard sometimes emits [{}] when it knows no link yet) instead of
failing the whole write.
Migration 0006_investigator_serial_sequences.sql:
- BIGSERIAL on the 7 investigation tables created auto-sequences
(evidence_evidence_pk_seq etc) that 0004 forgot to GRANT to the
investigator role. Without those grants every INSERT failed with
"permission denied for sequence …". Grant USAGE/SELECT/UPDATE on each
auto-seq.
Verified live: Locard wrote E-0002 + E-0003 from real Sandia chunks
(green fireball Feb 1949; cobalt particle analysis). Grade B, confidence
high, custody chain of 3 steps with honest gaps. Cost $0.09 for both,
~70s wall.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scanned docs are messy — duplicate transcriptions (typed + handwritten),
two classification variants of the same narrative, OCR noise, repeated
banners. The doc page showed raw chunks, so everything appeared twice.
40_reading_version.py generates ONE clean, deduplicated, well-structured
bilingual Markdown reading version per doc (Sonnet): merges duplicate versions
without losing unique lines, drops page furniture, formats transcripts as
dialogue. Faithful — invents nothing; redactions kept as markers.
/d/[docId] now defaults to a "📖 leitura" tab rendering this clean version,
with "🔍 trechos · scan original" preserving the faithful per-chunk + per-page
scan view. reading.md lives in raw/<doc>--subagent/ alongside the chunks.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
~414 chunks have ufo_anomaly_detected=true but no type/rationale (extraction
left them null), so the flag rendered "UAP flag: anomaly —" with a dangling
em-dash. Build the label from the parts that exist: fall back to "anomalia"
for a missing type and omit the "—" when there's no rationale. The flag still
shows (the chunk genuinely contains UAP content), just without the noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
29 image chunks have an empty bbox {}, and `fm.bbox ?? default` doesn't catch
an empty object, so the crop URL got w=undefined → /api/crop 400 → broken-image
icon. Now validate bbox (w/h > 0); without it, render the image's textual
description instead of requesting an impossible crop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root-cause fix for "search returns garbage for absent terms". The hybrid RPC's
dense branch always returned its k nearest vectors regardless of distance, so a
query for a term not in the corpus (e.g. "varginha") surfaced unrelated chunks.
The cross-encoder reranker would filter these but costs 18-62s on CPU —
unusable for interactive search.
Add max_dense_dist (default 0.40) to hybrid_search_chunks: dense neighbours
beyond that cosine distance are dropped server-side. Calibrated from measured
distances — strong semantic match ~0.12-0.20, no real match ~0.46-0.53. BM25
full-text still matches literal terms; the reranker becomes opt-in refinement.
Verified live: varginha/abducao → 0, disco voador/roswell → relevant, all <1s.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The hybrid_search RPC always returns up to recall_k dense neighbours, so a
query for a term absent from the corpus (e.g. "varginha") returned its 12
nearest vectors — irrelevant chunks like PAGE_NUMBER "1". Two bugs:
the reranker was skipped whenever results <= top_k, and there was no relevance
floor.
Now always run the cross-encoder reranker (BGE-reranker-v2-m3, normalized
sigmoid) and drop hits below 0.02. Verified: "varginha" → 0 results;
"roswell"/"tic tac"/"disco voador" → relevant hits on top (reranker cleanly
separates 0.0001 garbage from 0.03-0.27 matches).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The reprocess pass added chunks for pages beyond the original total_pages but
never updated the field, so doc-page navigation thought docs ended early
(jumped to next document mid-doc) and the page counter was wrong. Now bump
total_pages to the real max chunk page on each integration.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Doc page (/d/[docId]/[page]) gains prev/next navigation bars (top + bottom):
within a doc it steps page-by-page; at the first/last page it jumps to the
previous/next document. Replaces the disabled-at-boundary links.
Indexer tooling for the VPS repopulation:
- 30-index-chunks-to-db.py: add --no-embed (fast BM25-only index; vectors
backfilled separately) so the app is usable in minutes, not hours of CPU
embedding.
- 57_load_relations_from_json.py: load typed relations into public.relations
from reextract structured fields (deterministic ids, no fuzzy guessing).
- 58_backfill_embeddings.py: async pass to fill chunks.embedding (NULL rows)
via the embed-service.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add reextract pipeline (scripts/reextract/) that rebuilds doc-level entity
JSON from Sonnet-vision chunks via Opus, replacing the noisy per-page
extraction. Add synthesize scripts to regenerate wiki/entities from the 116
_reextract.json (30), aggregate missing page.md from chunks (31), and reprocess
805 pages the doc-rebuilder agent dropped on context overflow (32). Add
maintain scripts 43-56 for chunk-page sync, dedup, generic-entity marking, and
typed relation extraction.
Web: wire relations API + entity-relations component; entity/timeline/doc
pages consume the rebuilt layer.
Note: raw/, processing/, wiki/ remain gitignored (bulk data managed
separately); the 116 reextract JSONs and 7,798 rebuilt entity files live on
disk only. The 27 curated anchor events under wiki/entities/events/ are
preserved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The corpus had two parallel reverse-reference signals: the wiki/pages
entities_extracted blocks (Haiku page-level) and public.entity_mentions
(Sonnet chunk-level, ILIKE-matched). The entity page only consulted the
DB, so it showed "0 menções" for thousands of entities that were anchored
in pages or in cross-entity links the DB never indexed.
Resolved by collapsing all signals into the YAML frontmatter, which is
now the single runtime source for entity metadata.
scripts/maintain/42_sync_entity_stats.py walks every entity and writes:
mentioned_in: [...] # consolidated page refs
total_mentions: max(db, pages)
documents_count: max(db_docs, distinct page docs)
signal_sources:
db_chunks: int
page_refs: int
cross_refs: int
signal_strength: strong | weak | orphan | unverified
referenced_by: [[class/id]] # cross-entity backlinks
Outgoing wikilinks (e.g. OBJ.observed_in_event → EV) count toward the
entity's own cross_refs so anchored-but-not-mentioned entities don't
register as orphan.
OBJ canonical names like "7m long, 1.3m high, two rocket motors,
smooth flow, rotary drive null UAP (OBJ-EV1945-PEYERLSHOTDOWN-01)"
are rewritten to "Peyerl shot down UAP" derived from observed_in_event,
preserving the full description as an alias. --fix-obj-names did this
for every OBJ-* with >80 char canonical_name.
Default behaviour is conservative: --archive-only-junk archives only
single/double-char names and pure-numeric noise. Everything else stays
on disk with signal_strength marked, so the user can review later.
web/lib/retrieval/entity-pages.ts swapped from db-first to yaml-first.
The /e/[cls]/[id] page now reads counts straight from YAML and renders
a "força do sinal" badge with the per-source breakdown. Orphan entities
get a banner explaining they have no cross-references.
DB is still queried for ONE thing: the chunk text for preview cards on
the entity page, so we don't re-parse 21k markdown files on every render.
First-pass result: 9020 strong / 14520 weak / 10814 orphan; OBJ-EV1945-
PEYERLSHOTDOWN-01 now reads "Peyerl shot down UAP · fraca · 1 backlink"
in the live UI.
middleware.ts now checks profiles.role on every /admin/* request and
returns a plain 404 (not a redirect) for anonymous users and any
authenticated user without role='admin'. The 404 wording matches a
non-existent route, so we don't leak the existence of an admin area.
gutomec@gmail.com promoted to admin in the live DB.
openrouter.ts chat now collects artifacts silently during the tool-call
loop instead of streaming them to the SSE. After the model writes its
final prose (and after the forced-synthesis pass if needed), we scan
the assembled text for [[doc-id/p007#cNNNN]] citations and emit ONLY the
artifacts referenced. Duplicates are deduped by chunk_id. The persisted
citations column on messages now stores the filtered set too, so old
sessions reload with the same focused card grid.
Before: every hybrid_search hit (up to 6 per call × 5 calls = 30+ citation
cards plus crop images) flooded the chat regardless of what the model
ended up using. After: only the chunks actually woven into the answer.
Two bugs combined to make the chat reply with only cards and no prose:
1. SQL trigger rollup_session_stats was failing with "column reference
total_cost_usd is ambiguous" because the UPDATE on public.profiles had
a FROM public.chat_sessions clause and both tables expose that column.
Persistence of every user message died at this point — sessions were
created in the DB but had message_count=0 forever. Applied SQL fix
that qualifies columns with p./s. aliases (production DB updated;
ALTER FUNCTION run live, not yet codified in a migration file).
2. The free-tier model (nemotron-3-super:free) spent all 5 tool-loop
turns on hybrid_search calls and never wrote any prose, returning
content_len=0. Added a forced-synthesis pass in openrouter.ts: when
the loop exits with empty assembledText but the model did call tools,
we send ONE final turn with tools omitted from the request payload
and a user message instructing the model to answer in 3-8 sentences
citing chunks. openrouterStreamCall now accepts a `withTools` opt
so the synthesis call can disable tool calling entirely.
Verified end-to-end with the actual user query "O que os astronautas
viram? Quem foi que viu?" on /d/nasa-uap-d6-apollo-17-...:
- content_len: 0 → 947 chars (real synthesis citing Schmitt)
- artifacts: 44 preserved
- assistant message persisted with tool_calls + citations columns
34 of 116 docs were generated with 0-based PNG numbering (p-000.png …
p-008.png) but the Sonnet chunks reference 1-based page numbers in their
YAML frontmatter (page: 9 means the 9th sheet of paper). The /api/crop
handler built p-009.png and got a 500, the browser's Next/Image surfaced
400, and the chunk rendered as a black box on screen.
Fixes:
- web/app/api/crop/route.ts: try p-NNN.png first, fall back to
p-(NNN-1).png if the 1-based file is missing. Cheap insurance for any
doc that comes in with the old convention.
- scripts/01-convert-pdfs.sh: previously printf '%03d' "$num" with $num
starting at 0 (e.g. "008") raised "invalid number" because Bash
parsed it as octal. Wrap with $((10#$num)) to force decimal — this
was silently corrupting page sequences and producing gaps like p-001
... p-008, p-011 (missing p-009/p-010).
- All 34 affected docs re-converted from PDFs with the patched script;
every directory now has continuous 1-based PNGs.
- /processing/png/ rsync'd to VPS, web redeployed.
Smoke: /api/crop?doc=doc-341-…&page=9&… now returns 200 image/webp
instead of 500. Tested in browser: chunk c0026 (diagram, p9) renders
the real engineering diagram.
Fase 3 onda 2 — entity synthesis at scale:
- scripts/synthesize/20_entity_summary.py: queries DB for entities with
total_mentions ≥ threshold + top-K verbatim chunk snippets via
entity_mentions JOIN, prompts Sonnet (Holmes-Watson voice, bilingual),
writes narrative_summary EN+PT-BR + summary_status=synthesized.
Ran on 187 candidates (mentions ≥ 20) → 158 OK · 1 err · 29 skipped (no
snippets). Combined with anchor curation: 20 curated + 158 synthesized
= 178 entities with real narrative (vs 0 a day ago).
Fase 4 — chat with typed artifacts + persistence:
- lib/chat/agui.ts: AG-UI v1 typed Artifact union (citation, crop_image,
entity_card, evidence_card, hypothesis_card, case_card, navigation_offer)
alongside the existing event types.
- lib/chat/tools.ts + openrouter.ts: hybrid_search emits up to 6
citation + crop_image artifacts per query. Provider collects them and
returns in done.artifacts so the route can persist.
- api/sessions/[id]/messages: persist artifacts to messages.citations.
- components/chat-bubble.tsx: ArtifactCard renders inline cards (citation,
crop_image, entity_card, navigation_offer) for streamed and persisted
messages. activeId now persisted in localStorage so navigation between
pages keeps the same conversation. New sessions are lazy (only when user
has zero). loadMessages hydrates tools + artifacts from server. CRUD UI:
rename (✎) + archive (🗑) buttons per session in the list.
Home search:
- doc-list-filters: input now fires hybrid_search (rerank=0 for speed)
in parallel with the local title filter; chunk hits render above the doc
grid with snippet + score + classification.
- api/search/hybrid: accept ?rerank=0 to skip the cross-encoder (1.3s vs 60s).
Auth flow:
- infra: SMTP_HOST=mail.spacemail.com:587 + DMARC published; mail now lands
in inbox. GOTRUE_MAILER_AUTOCONFIRM=false (real email verification).
- kong.yml: proxy /auth/callback on api.disclosure.top → web:3000 so PKCE
email links don't 404 at the gateway.
- web/app/auth/callback: handle both ?code= (OAuth) and ?token=&type=
(PKCE); redirect to the public site host before verifyOtp so the session
cookie lands on the right domain.
Audit deliverables:
- .nirvana/outputs/disclosure-bureau/.../systems-atelier/: 5 docs (code
analysis, tech debt, discovery brief, system arch, 5 ADRs) authored by
sa-principal that produced this roadmap. Kept in-tree for traceability.
User reported a ~ Google Gemini bill against an expected ~ budget.
Permanent ban on Gemini for this project — all LLM inference goes through
claude -p --model sonnet (CLAUDE_CODE_OAUTH_TOKEN) or OPENROUTER_API_KEY as
fallback. The 10 scripts in scripts/ that import google-genai/-generativeai
are moved under scripts/_archived-gemini/ with a DO-NOT-RUN README. Project
memory updated (feedback-no-gemini-ever.md) and the older reference note that
recommended gemini-3.1-pro-preview is revoked.
- scripts/03-dedup-entities.py: stop emitting placeholder narrative ("Stub. Will
be enriched in Phase 7"); write summary_status=none + null fields instead.
- scripts/maintain/41_strip_stubs.py: idempotent migration that cleaned the
22,096 entity .md files (now zero stub strings in wiki/).
- scripts/synthesize/01_anchor_events.py: curated 20 anchor UAP events
(Roswell, Nimitz Tic-Tac, Phoenix Lights, Operação Prato, AATIP, etc.) with
bilingual Holmes-Watson narrative via claude -p --model sonnet
(CLAUDE_CODE_OAUTH_TOKEN). All summary_status=curated, confidence=high.
- web/api/timeline + timeline-view: filter narrative-less events by default,
render "curado" badge for hand-vetted ones, drop the date display alone.
- CLAUDE-schema-full.md: document the summary_status enum and the four states.
- docker-compose.yml: SMTP_HOST=mail.spacemail.com configured;
GOTRUE_MAILER_AUTOCONFIRM flipped to false (real email confirmation working).
- .nirvana/outputs/.../systems-atelier/: 5 deliverables of the architecture
audit that produced this roadmap.
Luiz Gustavo