feat(sources): Wayback pass, automatic trimming, detention rule, ten cases re-sourced
Second and third rounds on the dossier pipeline, reviewed locally before commit. Retrieval - wayback_pass.py: retries every BLOCKED/DEAD/THIN/ERROR linked source through the Wayback Machine; blocks carry the snapshot date, archive URL and the live verdict. 143 slots retried: 93 fetched, 37 no snapshot, 12 script shells, 1 PDF. - fetch_sources.py: fetch_wayback() helper. - add_extra_sources.py: files staged extra_* finds (web search, not on the DB record) into the store with a provenance note. - Cases re-sourced by web search: 248 (Oslo: Document.no, Avisa Oslo, NRK; the linked Le Parisien piece is case 242 and is marked OFF-CASE), 258 Kharkiv, 260 Singapore, 358 Bangkok, 365 Las Vegas, 388 Verneuil-sur-Seine, 390 Zoersel, 430 Homestead. 341 of 364 cases now hold a fetched article; 11, 66 and 399 have no public text source (podcast, police video, direct victim report). Trimming - auto_trim.py: case-anchored furniture cut for UNTRIMMED blocks. Finds the body run that mentions the case, merges across subheadings and short furniture gaps, drops teasers, share bars, date/URL/caption lines and subscription pitches; refuses pages with no record term or almost no body. 302 blocks trimmed; 16 left on auto-trim-review.md (7 OFF-CASE suspects). Every cut is labelled AUTO-TRIMMED, UNREVIEWED in the dossier; the full extract stays in staging/. Decisions with anchors in trim-decisions-auto.json. - build_cases.py / verify_all.py: banners for auto-trimmed and archived blocks, staged-file check extended to Wayback blocks, flag legend under the record. Detention rule - README "Definitions": the DB field `kidnappings` is the DETENTION violence type (victim, guard, staff or relative held to force submission or execute the theft), distinct from the Kidnapping scenario (taken away and held). - detention-flag-review.md / detention-flag-corrections.json: 40 records reviewed with evidence; 30 set-to-1 proposals accepted by the owner on 2026-09-06 (listed in corrections-approved.md), 10 still open. - apply_detention_wording.py: "Violence Used" in the 70 reviewed summaries now names detention explicitly (62 of 70 labelled), supported by the summary's own text; idempotent; supersedes the batch scripts' wording. Worklist and docs - sofia-worklist.md: 248 decisions, detention rule item replacing the old "no abduction" item, fresh-search section for the textless cases. - README-START-HERE.md: progress notes, run commands, next steps. - Bug fixed in passing: Wayback blocks stored in-memory text with carriage returns; now stored as read back from disk. verify_all.py PASSES (2321 checks). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GZZENdTLzNGsbNy4DyF1yt
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
499fd61f79
commit
439e83a4cb
@@ -0,0 +1,145 @@
|
||||
{
|
||||
"451:url": "THIN",
|
||||
"449:url": "FETCHED",
|
||||
"448:url": "THIN",
|
||||
"446:url": "THIN",
|
||||
"445:url": "THIN",
|
||||
"444:url": "THIN",
|
||||
"442:url_2": "FETCHED",
|
||||
"441:url": "FETCHED",
|
||||
"440:url": "FETCHED",
|
||||
"440:url_2": "FETCHED",
|
||||
"439:url": "FETCHED",
|
||||
"439:url_2": "FETCHED",
|
||||
"438:url_2": "FETCHED",
|
||||
"438:url_3": "FETCHED",
|
||||
"437:url": "FETCHED",
|
||||
"436:url_3": "NO SNAPSHOT",
|
||||
"431:url": "NO SNAPSHOT",
|
||||
"431:url_2": "NO SNAPSHOT",
|
||||
"430:url": "NO SNAPSHOT",
|
||||
"427:url_2": "THIN",
|
||||
"426:url": "FETCHED",
|
||||
"426:url_2": "NO SNAPSHOT",
|
||||
"424:url_2": "FETCHED",
|
||||
"423:url_2": "FETCHED",
|
||||
"422:url_2": "FETCHED",
|
||||
"417:url_2": "FETCHED",
|
||||
"416:url": "THIN",
|
||||
"414:url": "FETCHED",
|
||||
"412:url": "NO SNAPSHOT",
|
||||
"412:url_4": "NO SNAPSHOT",
|
||||
"412:url_5": "NO SNAPSHOT",
|
||||
"411:url_2": "FETCHED",
|
||||
"410:url": "THIN",
|
||||
"408:url": "NO SNAPSHOT",
|
||||
"407:url_3": "FETCHED",
|
||||
"407:url_4": "FETCHED",
|
||||
"405:url_2": "NO SNAPSHOT",
|
||||
"402:url_3": "NO SNAPSHOT",
|
||||
"395:url": "FETCHED",
|
||||
"393:url_2": "FETCHED",
|
||||
"391:url": "FETCHED",
|
||||
"390:url": "NO SNAPSHOT",
|
||||
"390:url_2": "NO SNAPSHOT",
|
||||
"389:url_2": "FETCHED",
|
||||
"388:url": "NO SNAPSHOT",
|
||||
"387:url": "FETCHED",
|
||||
"384:url_2": "NO SNAPSHOT",
|
||||
"383:url_2": "NO SNAPSHOT",
|
||||
"382:url": "FETCHED",
|
||||
"382:url_2": "THIN",
|
||||
"382:url_3": "NO SNAPSHOT",
|
||||
"382:url_4": "NO SNAPSHOT",
|
||||
"381:url": "FETCHED",
|
||||
"379:url_3": "FETCHED",
|
||||
"372:url": "FETCHED",
|
||||
"370:url": "THIN",
|
||||
"369:url": "FETCHED",
|
||||
"367:url": "FETCHED",
|
||||
"365:url": "NO SNAPSHOT",
|
||||
"363:url_4": "NO SNAPSHOT",
|
||||
"361:url": "FETCHED",
|
||||
"360:url_2": "FETCHED",
|
||||
"258:url": "NO SNAPSHOT",
|
||||
"45:url": "FETCHED",
|
||||
"355:url": "FETCHED",
|
||||
"354:url": "FETCHED",
|
||||
"349:url": "FETCHED",
|
||||
"323:url": "FETCHED",
|
||||
"322:url": "FETCHED",
|
||||
"318:url": "NO SNAPSHOT",
|
||||
"307:url": "FETCHED",
|
||||
"293:url": "FETCHED",
|
||||
"290:url": "FETCHED",
|
||||
"288:url": "FETCHED",
|
||||
"285:url": "FETCHED",
|
||||
"271:url": "FETCHED",
|
||||
"263:url": "FETCHED",
|
||||
"256:url": "FETCHED",
|
||||
"255:url_2": "FETCHED",
|
||||
"255:url_3": "FETCHED",
|
||||
"253:url": "NO SNAPSHOT",
|
||||
"248:url": "FETCHED",
|
||||
"247:url": "FETCHED",
|
||||
"237:url": "NO SNAPSHOT",
|
||||
"229:url": "FETCHED",
|
||||
"228:url_2": "FETCHED",
|
||||
"223:url": "FETCHED",
|
||||
"222:url": "FETCHED",
|
||||
"219:url": "FETCHED",
|
||||
"218:url": "FETCHED",
|
||||
"214:url": "NO SNAPSHOT",
|
||||
"213:url": "FETCHED",
|
||||
"212:url": "FETCHED",
|
||||
"210:url": "FETCHED",
|
||||
"209:url_4": "FETCHED",
|
||||
"207:url": "NO SNAPSHOT",
|
||||
"199:url": "FETCHED",
|
||||
"197:url": "NO SNAPSHOT",
|
||||
"195:url": "FETCHED",
|
||||
"194:url": "FETCHED",
|
||||
"188:url_2": "FETCHED",
|
||||
"188:url_3": "FETCHED",
|
||||
"183:url": "FETCHED",
|
||||
"179:url": "NO SNAPSHOT",
|
||||
"176:url": "NO SNAPSHOT",
|
||||
"168:url": "FETCHED",
|
||||
"164:url": "FETCHED",
|
||||
"162:url": "FETCHED",
|
||||
"160:url": "FETCHED",
|
||||
"157:url": "NO SNAPSHOT",
|
||||
"152:url": "FETCHED",
|
||||
"151:url": "FETCHED",
|
||||
"145:url": "FETCHED",
|
||||
"141:url": "FETCHED",
|
||||
"138:url": "NO SNAPSHOT",
|
||||
"116:url": "FETCHED",
|
||||
"113:url": "FETCHED",
|
||||
"111:url": "FETCHED",
|
||||
"110:url": "FETCHED",
|
||||
"105:url": "FETCHED",
|
||||
"101:url": "FETCHED",
|
||||
"91:url": "FETCHED",
|
||||
"80:url": "NO SNAPSHOT",
|
||||
"79:url": "THIN",
|
||||
"71:url": "FETCHED",
|
||||
"69:url": "FETCHED",
|
||||
"67:url": "NO SNAPSHOT",
|
||||
"62:url": "FETCHED",
|
||||
"60:url": "FETCHED",
|
||||
"57:url": "NO SNAPSHOT",
|
||||
"56:url": "FETCHED",
|
||||
"54:url": "FETCHED",
|
||||
"44:url": "FETCHED",
|
||||
"41:url": "FETCHED",
|
||||
"40:url": "FETCHED",
|
||||
"33:url": "FETCHED",
|
||||
"32:url": "NO SNAPSHOT",
|
||||
"30:url": "NO SNAPSHOT",
|
||||
"21:url": "NO SNAPSHOT",
|
||||
"18:url": "PDF",
|
||||
"16:url": "FETCHED",
|
||||
"9:url": "THIN",
|
||||
"6:url": "FETCHED"
|
||||
}
|
||||
Reference in New Issue
Block a user