disclosure-bureau/CHANGELOG.md

Changelog · Disclosure Bureau

All notable changes to this project go here. Newest on top.

[Unreleased]

W3.1–W3.4 — Investigation Bureau foundation

2026-05-23 · systems-atelier engagement trace 794f00ba · cerne do brief

The "8 detectives" branding becomes a real motor. This wave delivers the database schema, the agentic runtime container, the first gated writer, and the first detective end-to-end. Subsequent waves W3.5–W3.10 add the remaining detectives, the chat tool, and the frontend.

• Migration 0004_investigation_bureau.sql — 7 new tables with RLS: investigation_jobs (queue + audit), evidence, hypotheses, contradictions, witnesses, gaps, residual_uncertainties. ID sequences evidence_id_seq etc. for human-readable IDs (E-NNNN / H-NNNN / R-NNNN / W-NNNN / G-NNNN / RU-NNNN). pg_notify trigger on investigation_jobs fires on every INSERT so workers wake up immediately.
• investigator role carved out of the existing Postgres with least-privilege grants: SELECT on the read corpus (chunks/entities/entity_mentions/relations/documents), INSERT/UPDATE on the 7 new tables and their sequences, no service_role, no auth.users / profiles / messages. Per gate #1 of the security audit.
• Migration 0005_investigator_write_policies.sql — fix-up: RLS with only a SELECT policy silently blocked the worker's UPDATE … RETURNING claim query. New INSERT/UPDATE policies on all 7 tables bound to the investigator role (plus service_role + postgres).
• investigator-runtime/ new Bun + TypeScript container: src/main.ts (LISTEN poller + claim-skip-locked + healthcheck file), src/orchestrator.ts (chief-detective dispatch), src/lib/{env,pg, audit,ids,claude}.ts, src/detectives/locard.ts, and src/tools/write_evidence.ts. Dockerfile built on oven/bun:1.1-slim with claude CLI installed for OAuth subprocess calls. Healthcheck touches /tmp/healthy per loop; docker declares unhealthy if stale.
• Locard detective (the simplest of the 8): given a chunk, asks Claude Sonnet 4.6 to extract a verbatim quote + chain of custody. The model emits a strict JSON object; the runtime owns the writer (gate #2 of security audit). System prompt at investigator-runtime/prompts/locard.md.
• write_evidence tool — schema-validated INSERT into public.evidence
  • render case/evidence/E-NNNN.md. Rejects evidence whose verbatim_excerpt isn't found inside the source chunk's content (Sonnet must not paraphrase). Rejects below-grade rows (A ≥ 3 custody steps, B ≥ 2, C ≥ 1). FK to public.chunks so the row can never reference a phantom chunk.
• /api/admin/investigate/test admin endpoint — POST creates a job, GET polls. Gated by middleware (/api/admin/* → 404 for non-admins, per W0-F1). Designed for the chat-based request_investigation tool coming in W3.6.
• End-to-end smoke test on prod:
  1. INSERT a job (evidence_chain, doc dow-uap-d017-…-sandia, chunks [c0030]).
  2. pg_notify investigation_jobs fires.
  3. Worker LISTEN receives the notification.
  4. claimNextJob UPDATE-claims the row (worker_id stamped).
  5. Locard is dispatched.
  6. claude -p subprocess invoked (auth + model lookup successful, version 2.1.150).
  7. Currently Claude OAuth Max 20x weekly quota is exhausted (api_error_status: 429, "You've hit your weekly limit · resets 3pm (UTC)"). The orchestrator catches the typed isQuota error; the job is now marked failed (not complete) with the surfaced reason in error. The plumbing works end-to-end — when the quota resets, the same job replayed succeeds.
• Architecture conforms to the 8 security gates (ADR-002 + section 9 of agentic-layer-spec.md): no service_role in the worker; schema validation before INSERT; created_by stamped on every row; BUDGET_CAP_USD_PER_JOB enforced per call; allowlist tools (only write_evidence for Locard so far, no WebSearch); audit trail at case/audit.jsonl. Gates #6–#8 to land alongside W3.5+.

Verified live (2026-05-23T22:48Z):

• \dt public.{investigation_jobs,evidence,hypotheses,…} → all 7 tables exist.
• psql -U investigator -c 'SELECT COUNT(*) FROM public.chunks' → 28 559 (read works with low-privilege role).
• docker ps disclosure-investigator → Up (healthy).
• Audit log shows runtime_starting → listening → job_claimed → detective_dispatched → job_failed_all_items (quota) chain.
• Job state transitions correctly persisted in public.investigation_jobs.

W3.5+ pending (next session):

• Detective holmes + write_hypothesis tool (hypothesis tournament).
• Detective dupin + write_contradiction tool + daily cron.
• Detectives tetlock, schneier, taleb, poirot, case-writer.
• Chat tool request_investigation + status bar + /jobs/[id] page.
• Frontend tab "Investigation" + /h/[hypothesisId] page.
• Golden hypothesis set (W3.10 quality gate).

W2 — UX latency + retrieval eval + vision tool

2026-05-23 · systems-atelier engagement trace 794f00ba

• TD#8 · Reranker opt-in (hybrid.ts). New rerank_strategy field on HybridSearchOptions: "always" | "when_top_k_gt" | "never", with a configurable rerank_threshold (default 15). Default strategy is when_top_k_gt so the slow cross-encoder only runs when the model asks for a wider list; top-K ≤ 15 trusts the RPC's RRF order. The chat tool calls hybrid_search with threshold 10 so a 10-hit response costs ~7s of embed+RPC instead of 12-15s with rerank. /api/search/hybrid exposes the strategy via ?rerank=always|never|when_top_k_gt plus ?rerank_threshold=N. Back-compat ?rerank=0 still means "never".
• O11 · analyze_image_region chat tool (vision.ts, tools.ts). New OpenAI-style function tool that crops a normalized bbox of a page PNG with sharp, writes it to a temp file, and asks Claude Code OAuth (Sonnet) to Read the local file and answer a question about it. Schema: {doc_id, page, bbox{x,y,w,h}, question, context?}. Emits a crop_image artifact for the UI alongside the textual answer. Cost budget: ~$0.005–0.02 per call, paid against the user's Max 20x quota. Timeout configurable via VISION_TIMEOUT_MS (default 120s).
• TD#12 · react-force-graph-2d removed. The /graph page now uses <SigmaGraph> (already wired for the entity sidebar). One graph library is enough. web/components/force-graph-canvas.tsx deleted; npm uninstall removed 37 transitive deps.
• TD#27 · Context truncation per type configurable (messages/route.ts). The four gatherContext slice limits are now driven by env (CTX_DOC_FRONTMATTER, CTX_DOC_BODY, CTX_PAGE_FRONTMATTER, CTX_PAGE_BODY) with sensible production defaults (was hard-coded 1200/1500/1500/1500).
• TD#22 · Golden RAG eval (tests/rag/). New harness: golden.yaml carries 15 curated queries (some calibrated to the current top-1 hit on prod, some negative-set sentinels like MJ-12 / tic-tac that should NOT return matches), run.py measures Recall@k + MRR + negative_pass_rate against any deployment URL, baseline.json is the gate threshold, last_run.json is the working report. Default behaviour: fail the run when Recall@5 drops > 0.05 from baseline. CI workflow runs against https://disclosure.top on every push.
  • First baseline (rerank=never): Recall@5 = 0.2083, MRR = 0.25, Negative pass = 1.0. Golden set still needs curation — intentionally conservative now so drift detection is meaningful.
• ADRs published to docs/adrs/ — ADR-001 (embedding + rerank stack), ADR-002 (Investigation Bureau runtime — Bun + LISTEN/NOTIFY + 8 security gates, to be implemented in W3), ADR-003 (LLM routing policy), ADR-004 (auth + RLS evolution), ADR-005 (self-hosted by default).

Verified on disclosure.top (2026-05-23T21:55Z):

• /api/search/hybrid?q=Roswell&top_k=5 → HTTP 200 in 6.7s (embed-only, rerank skipped per default strategy)
• /api/search/hybrid?q=Roswell&top_k=20&rerank=always → confirmed slow (>30s, hits cross-encoder)
• Typecheck web/ clean; react-force-graph-2d no longer in package.json
• tests/rag/run.py against prod → 15 queries answered, baseline written
• 5 ADRs committed under docs/adrs/

W1.2 — Glitchtip + Forgejo self-hosted

2026-05-23 · systems-atelier engagement trace 794f00ba

• Glitchtip self-host (Sentry-compatible error monitor). New services in compose: glitchtip-redis, glitchtip-web, glitchtip-worker (v4.2, uWSGI on 8080). Database glitchtip carved out of disclosure-db as a separate role/DB. Bootstrap done via Django manage.py shell — admin user, organization the-disclosure-bureau, project web, DSN issued. SDK wired: @sentry/nextjs + instrumentation.ts
  • sentry.{client,server}.config.ts. /api/admin/throw smoke endpoint is admin-gated. Live at https://glitchtip.disclosure.top (TLS issued by Let's Encrypt via Traefik). Synthetic event verified — POST /api/1/store/ → 200 + event id.
• Forgejo self-host + Actions CI. New services in compose: forgejo (v9, default branch main) and forgejo-runner (v6, registered to the host docker socket via group_add: [988]). Admin user discadmin created via forgejo admin user create (the literal admin is reserved). Runner bootstrap on first start: registers if .runner absent, then forgejo-runner daemon. Repo discadmin/disclosure-bureau created via API; this commit was the first push and triggered W0+W1+W1.2: … workflow at task 1.
• .forgejo/workflows/ci.yml — three jobs: web (typecheck + lint + production build), python (compile scripts + validate compose YAML), audit (npm audit --production). Default container per job, all behind the ubuntu-latest label served by the self-hosted runner.

Verified on the stack (2026-05-23T21:19Z):

• glitchtip.disclosure.top → HTTP 200, real Let's Encrypt cert, Glitchtip CSP headers present.
• POST /api/1/store/ → 200, event_id cb17d723… returned.
• forgejo.disclosure.top → HTTP 200, Forgejo welcome page.
• Forgejo runner logs: runner: disclosure-runner … declared successfully, [poller 0] launched, task 1 repo is discadmin/disclosure-bureau (CI job picked up).
• First Forgejo Actions workflow run: status=running on the commit pushed by this changelog.

W1 — Observability + resilience + Meili autocomplete

2026-05-23 · systems-atelier engagement trace 794f00ba

• Studio container fixed (carry-over from W0) — root cause was Next.js standalone binding to the container hostname only. The docker healthcheck (fetch 127.0.0.1:3000/api/profile) looped on ECONNREFUSED, the service never went healthy, and Traefik returned 404 because the upstream wasn't responding. Fix: HOSTNAME: 0.0.0.0 in the studio env. Studio now healthy, basic auth from W0-F3 enforces correctly (no-auth → 401, valid creds → 307), and Let's Encrypt issued a real cert for studio.disclosure.top once the route started responding.
• TD#10 · PG pool max — PG_POOL_MAX=20 (was hard-coded 5) configurable via .env; default raised for prod. Files: docker-compose.yml, .env.
• W1-F8 · CLAUDE_CODE_OAUTH_TOKEN gated — only injected into the web service when explicitly set in CLAUDE_CODE_OAUTH_TOKEN_FOR_WEB. Default empty since CHAT_PROVIDER=openrouter does not need it. Reduces blast radius if web container is compromised. Files: docker-compose.yml, .env.
• TD#30 · Subprocess timeout configurable — CLAUDE_CODE_TIMEOUT_MS env now controls the claude -p subprocess timeout (default 90s, matches prior hard-coded value). Files: web/lib/chat/claude-code.ts.
• TD#23 · OpenRouter retry + circuit breaker — fetchOpenRouter() wraps every call with: retry up to OPENROUTER_RETRY_MAX (default 2) on 408 / 425 / 429 / 500 / 502 / 503 / 504 and network errors, with exponential backoff and Retry-After honored; in-memory circuit breaker trips when PRIMARY fails CB_THRESHOLD times (default 3) within CB_WINDOW_MS (60s), promoting FALLBACK for CB_COOLDOWN_MS (2 min). Both sendOnce and openrouterStreamCall go through it. Files: web/lib/chat/openrouter.ts.
• TD#6 · Structured logging with pino — web/lib/logger.ts provides a JSON logger (NDJSON in prod, pretty in dev) plus withRequest() helper for correlation-id-bound child loggers. Edge runtime falls back to a console adapter. Middleware now mints a correlation_id for every request, stamps the response header (x-correlation-id), and emits one structured http_request line per /api/* call with method, path, status, and duration. messages/route.ts switched to the new logger. Files: web/lib/logger.ts, web/middleware.ts, web/app/api/sessions/[id]/messages/route.ts, web/package.json.
• Meilisearch indexer + /api/search/autocomplete + UI — the previously idle Meili instance now backs typo-tolerant prefix search. Indexer script scripts/maintain/60_meili_index.py ingests documents (canonical_title + collection) and is-searchable chunks (content_pt + content_en + meta). The new /api/search/autocomplete?q=... route hits both indexes in parallel with a 2s abort and returns a merged payload. SearchAutocomplete React component drops a debounced dropdown under the /search input. Median latency in production: 5–8ms. Files: scripts/maintain/60_meili_index.py, web/app/api/search/autocomplete/route.ts, web/components/search-autocomplete.tsx, web/components/search-panel.tsx.

Verified on disclosure.top (2026-05-23T20:30Z):

• /api/admin/{batch,indexer,stats} → 404 ✓ (W0 still holds)
• studio.disclosure.top no-auth → 401 · admin:<DASHBOARD_PASSWORD> → 307 ✓
• Let's Encrypt cert issued for studio.disclosure.top ✓
• Autocomplete q=Roswell → 8 chunks in 8ms; q=Sandia → 1 doc + 8 chunks in 8ms; q=1947 → 5 docs + 8 chunks in 6ms ✓
• x-correlation-id header present on /api/search/hybrid response (e.g. c48b7cc761dac172) ✓
• 18 513 searchable chunks indexed into Meili ✓
• OpenRouter retry/breaker present (7 references in source) ✓

Deferred to W1.2 / W2 (need user-in-loop steps):

• Glitchtip self-host — needs DNS for glitchtip.disclosure.top, initial signup-as-superuser, project DSN copied to .env. Logger and middleware are already feeding the data; SDK wiring is one PR.
• Forgejo Actions self-host CI — Forgejo server + runner bootstrap, initial admin account, repo migration / mirror. Recommend a separate session because of the depth of setup.

W0 — Hardening (security + reproducibility)

2026-05-23 · systems-atelier engagement trace 794f00ba-7cb6-4b90-a48e-23ebd02d1f44

• F1 · Auth gate em /api/admin/* — middleware now matches /api/admin too; non-admin (including anonymous) gets HTTP 404. Verified: curl on /api/admin/{batch,indexer,stats} returns 404 publicly. Files: web/middleware.ts.
• F2 · Imgproxy filesystem root tightened — IMGPROXY_LOCAL_FILESYSTEM_ROOT moved from / (entire VPS root) to /var/lib/storage (Storage backend mount only). Reduces blast radius of any future imgproxy CVE. Files: infra/disclosure-stack/docker-compose.yml.
• F3 · Studio basic auth label — replaced the dead-end basicauth.usersfile=/dev/null with a real bcrypt-hashed credential (DASHBOARD_USERNAME / DASHBOARD_PASSWORD from .env) and wired the middleware into the router via disclosure-studio.middlewares= disclosure-studio-auth@docker. Caveat: the Studio container itself has a pre-existing instability (restarts in a Next.js loop, status unhealthy) so the front-end currently returns 404 from Traefik. When Studio is stabilized (queue for W1), the basic auth will kick in. Files: infra/disclosure-stack/docker-compose.yml.
• F4 · RLS on public.relations — ENABLE ROW LEVEL SECURITY + public SELECT policy + GRANT SELECT TO anon, authenticated. Aligns with every other public table. Files: infra/supabase/migrations/0003_w0_hardening.sql.
• TD#2 · is_searchable folded into canonical migrations — the column, reclassification rules, partial index, and the updated hybrid_search_chunks RPC (BM25 + dense, both filtered by is_searchable) are now in migration 0003_w0_hardening.sql. A clean bootstrap on a fresh VPS produces a searchable database without any scripts/maintain/47-48 post-hoc patches. Files: infra/supabase/migrations/0003_w0_hardening.sql.

Verified on disclosure.top (2026-05-23T19:30Z):

• /api/admin/batch → HTTP 404 ✓
• /api/admin/indexer → HTTP 404 ✓
• /api/admin/stats → HTTP 404 ✓
• pg_class.relrowsecurity = t for chunks, documents, entities, entity_mentions, relations ✓
• is_searchable distribution: 18 513 searchable / 10 046 not-searchable (35% of corpus deduplicated from results) ✓
• /api/search/hybrid?q=Roswell → HTTP 200, 10 hits, first c0527 ✓
• Studio: Traefik labels in place; container itself unhealthy (separate issue, deferred to W1) ⚠

Notes for clean-install reproducibility:

• 0003_w0_hardening.sql MUST be applied as supabase_admin, not postgres, because public.chunks / .entities / .relations are owned by supabase_admin. The migration file documents this in its header.