discadmin/disclosure-bureau
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(case-writer): doc-scoped retrieval for per-document case files

Browse source 
A per-document case_report grounded itself via hybridSearch keyed on the
document's auto-derived topic ("Fbi Photo B20", "Doc 59 214434 …"). That
garbage topic has no semantic neighbours, so the dense gate rejected every
chunk and the narrator saw zero scenes — skipping 61 of 75 batch documents
that in fact had dozens of embedded chunks.

When doc_id is set, fetch the document's own substantive chunks in reading
order (fetchDocChunks) instead of searching the corpus. Corpus-wide topic
reports still use hybridSearch.

Verified post-deploy: fbi-photo-b19 0→10 scenes, doc-65 →24, dow-uap-d44 →11.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Luiz Gustavo 2026-05-25 04:05:33 -03:00
parent c1260cfa68
commit aaf5618466
2 changed files with 50 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

33
investigator-runtime/src/detectives/case_writer.ts
View file

@ -16,7 +16,7 @@ import { audit } from "../lib/audit";
import { callClaude } from "../lib/claude";
import { env } from "../lib/env";
import { query } from "../lib/pg";
import { hybridSearch, type SearchHit } from "../lib/search";
import { fetchDocChunks, hybridSearch, type SearchHit } from "../lib/search";
import { writeCaseReport } from "../tools/write_case_report";
const HERE = path.dirname(fileURLToPath(import.meta.url));
@ -286,18 +286,27 @@ export async function runCaseWriter(task: CaseWriterTask): Promise<
const filter = `%${topic.toLowerCase()}%`;
// Grounding pass — retrieve top scenes from the corpus via hybrid_search.
// This is what gives the narrator real verbatim material to weave. Without
// this, the case-writer only sees pre-digested artefacts (which is what
// produced the academic prose in v1).
const scenes = await hybridSearch({
query: topic, lang,
doc_id: task.doc_id ?? null,
top_k: 18,
recall_k: 80,
max_dense_dist: 0.55,
}).catch(() => [] as SearchHit[]);
// Grounding pass — assemble the scenes the narrator weaves from.
//
// For a per-document case file (doc_id set) we pull THIS document's own
// chunks in reading order. A hybridSearch keyed on the document's
// auto-derived topic ("Fbi Photo B20", "Doc 59 214434 …") returns zero
// hits even though the doc has dozens of embedded chunks — the dense gate
// rejects them all because the garbage topic has no semantic neighbours.
// That single bug skipped 61 of 75 batch documents on its own.
//
// For a corpus-wide topic report (no doc_id) the semantic search is
// exactly right — we want the strongest chunks across all documents.
const docIdFilter = task.doc_id ?? null;
const scenes = docIdFilter
? await fetchDocChunks(docIdFilter, lang, 24).catch(() => [] as SearchHit[])
: await hybridSearch({
query: topic, lang,
doc_id: null,
top_k: 18,
recall_k: 80,
max_dense_dist: 0.55,
}).catch(() => [] as SearchHit[]);
// Pull artefacts SEQUENTIALLY. The investigator role has rolconnlimit=4 and
// pool.max=4; Promise.all of 5 queries × max_parallel=2 jobs would demand

29
investigator-runtime/src/lib/search.ts
View file

@ -67,6 +67,35 @@ export interface HybridSearchOpts {
max_dense_dist?: number;
}
/**
* Fetch a single document's own chunks in reading order — no semantic
* gating. For a per-document case file the narrator wants THIS document's
* substance, not a corpus search; a hybridSearch keyed on the document's
* (often garbage) auto-derived topic returns zero hits even though the
* doc has dozens of embedded chunks. We pull substantive `is_searchable`
* chunks ordered by `order_global` so the document tells its own story in
* sequence.
*/
export async function fetchDocChunks(
doc_id: string,
_lang: "pt" | "en" = "pt",
limit = 24,
): Promise<SearchHit[]> {
if (!doc_id) return [];
return await query<SearchHit>(
`SELECT chunk_pk, doc_id, chunk_id, page, type, bbox,
content_en, content_pt, classification,
1.0::float8 AS score, NULL::int AS bm25_rank, NULL::int AS dense_rank
FROM public.chunks
WHERE doc_id = $1
AND is_searchable = TRUE
AND length(COALESCE(content_en,'') || COALESCE(content_pt,'')) > 40
ORDER BY order_global ASC NULLS LAST, page ASC, order_in_page ASC
LIMIT $2`,
[doc_id, limit],
);
}
export async function hybridSearch(opts: HybridSearchOpts): Promise<SearchHit[]> {
const {
query: q,