StellarCrowandClaude Fable 5.1
439e83a4cb
Second and third rounds on the dossier pipeline, reviewed locally before commit. Retrieval - wayback_pass.py: retries every BLOCKED/DEAD/THIN/ERROR linked source through the Wayback Machine; blocks carry the snapshot date, archive URL and the live verdict. 143 slots retried: 93 fetched, 37 no snapshot, 12 script shells, 1 PDF. - fetch_sources.py: fetch_wayback() helper. - add_extra_sources.py: files staged extra_* finds (web search, not on the DB record) into the store with a provenance note. - Cases re-sourced by web search: 248 (Oslo: Document.no, Avisa Oslo, NRK; the linked Le Parisien piece is case 242 and is marked OFF-CASE), 258 Kharkiv, 260 Singapore, 358 Bangkok, 365 Las Vegas, 388 Verneuil-sur-Seine, 390 Zoersel, 430 Homestead. 341 of 364 cases now hold a fetched article; 11, 66 and 399 have no public text source (podcast, police video, direct victim report). Trimming - auto_trim.py: case-anchored furniture cut for UNTRIMMED blocks. Finds the body run that mentions the case, merges across subheadings and short furniture gaps, drops teasers, share bars, date/URL/caption lines and subscription pitches; refuses pages with no record term or almost no body. 302 blocks trimmed; 16 left on auto-trim-review.md (7 OFF-CASE suspects). Every cut is labelled AUTO-TRIMMED, UNREVIEWED in the dossier; the full extract stays in staging/. Decisions with anchors in trim-decisions-auto.json. - build_cases.py / verify_all.py: banners for auto-trimmed and archived blocks, staged-file check extended to Wayback blocks, flag legend under the record. Detention rule - README "Definitions": the DB field `kidnappings` is the DETENTION violence type (victim, guard, staff or relative held to force submission or execute the theft), distinct from the Kidnapping scenario (taken away and held). - detention-flag-review.md / detention-flag-corrections.json: 40 records reviewed with evidence; 30 set-to-1 proposals accepted by the owner on 2026-09-06 (listed in corrections-approved.md), 10 still open. - apply_detention_wording.py: "Violence Used" in the 70 reviewed summaries now names detention explicitly (62 of 70 labelled), supported by the summary's own text; idempotent; supersedes the batch scripts' wording. Worklist and docs - sofia-worklist.md: 248 decisions, detention rule item replacing the old "no abduction" item, fresh-search section for the textless cases. - README-START-HERE.md: progress notes, run commands, next steps. - Bug fixed in passing: Wayback blocks stored in-memory text with carriage returns; now stored as read back from disk. verify_all.py PASSES (2321 checks). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GZZENdTLzNGsbNy4DyF1yt
70 lines
3.6 KiB
Python
70 lines
3.6 KiB
Python
"""
|
|
File staged extra_* sources (found by web search, NOT on the DB record) into the article
|
|
store as UNTRIMMED blocks, so build_cases.py renders them and auto_trim.py can cut them.
|
|
|
|
Reads staging/<case>/index.json; every extra_N entry that is not yet in the store becomes
|
|
a block. FETCHED entries carry the staged text, hashed; anything else (THIN, BLOCKED, DEAD,
|
|
NOT AN ARTICLE) is filed as a reference with its verdict. Each block says in its note that
|
|
the URL is not on the record and when it was found, so provenance stays visible.
|
|
|
|
Usage: python add_extra_sources.py 258 260 ... then auto_trim.py <cases>, build, verify.
|
|
"""
|
|
|
|
import hashlib
|
|
import json
|
|
import sys
|
|
from datetime import date
|
|
from pathlib import Path
|
|
|
|
BASE = Path(__file__).resolve().parent
|
|
STAGING = BASE / "staging"
|
|
STORE = BASE / "sourced-articles.json"
|
|
|
|
NAMES = {"document.no": "Document.no", "alt.no": "Avisa Oslo (alt.no)", "nrk.no": "NRK", "ao.no": "Avisa Oslo",
|
|
"mothership.sg": "Mothership", "khaosodenglish.com": "Khaosod English", "8newsnow.com": "8 News Now (KLAS)",
|
|
"thedefiant.io": "The Defiant", "actu17.fr": "Actu17", "lagazette-yvelines.fr": "La Gazette en Yvelines",
|
|
"mantes-actu.net": "Mantes Actu", "vrt.be": "VRT NWS", "nnieuws.be": "NNieuws", "indegazette.be": "In de Gazette",
|
|
"justice.gov": "US Department of Justice", "decrypt.co": "Decrypt", "fortune.com": "Fortune",
|
|
"ukrinform.ua": "Ukrinform", "hromadske.radio": "Hromadske Radio", "sud.ua": "Sud.ua", "atn.ua": "ATN"}
|
|
|
|
|
|
def main():
|
|
cases = [int(a) for a in sys.argv[1:]]
|
|
store = json.loads(STORE.read_text(encoding="utf-8"))
|
|
today = date.today().isoformat()
|
|
added = 0
|
|
for cid in cases:
|
|
idx_path = STAGING / f"{cid:03d}" / "index.json"
|
|
if not idx_path.exists():
|
|
print(f"case {cid}: no staging index"); continue
|
|
blocks = store.setdefault(str(cid), [])
|
|
have = {b.get("field") for b in blocks}
|
|
for e in json.loads(idx_path.read_text(encoding="utf-8")):
|
|
if not e["field"].startswith("extra_") or e["field"] in have:
|
|
continue
|
|
src = NAMES.get(e["host"], e["host"])
|
|
note = f"NOT on the DB record. Found {today} by web search because the linked source(s) gave no usable text."
|
|
if e["verdict"] != "FETCHED":
|
|
blocks.append({"field": e["field"], "verdict": e["verdict"], "url": e["url"], "source": src,
|
|
"note": note + (" " + e["note"] if e.get("note") else "")})
|
|
added += 1
|
|
continue
|
|
path = next(STAGING.joinpath(f"{cid:03d}").glob(f"{e['n']:02d}_*.txt"))
|
|
text = path.read_text(encoding="utf-8")
|
|
blocks.append({"field": e["field"], "verdict": "UNTRIMMED", "url": e["url"], "source": src,
|
|
"language": "unknown", "text": text, "chars": len(text),
|
|
"sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(),
|
|
"staged_file": path.name,
|
|
"paragraphs_kept": f"ALL {len(text.split(chr(10) * 2))} paragraphs, untrimmed",
|
|
"method": "raw HTTP retrieval, deterministic extraction",
|
|
"fetched_at": e.get("fetched_at"), "http": e.get("http"), "note": note,
|
|
"cut_note": "NOT TRIMMED. Full page extract; trim on read-through."})
|
|
added += 1
|
|
print(f"case {cid}: + {e['field']} {src} {len(text):,} chars")
|
|
STORE.write_text(json.dumps(store, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
print(f"added {added} block(s)")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|