Sofia/DB-cleanup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity
Files
e36ceae8d6272bb9646068f8b21ab87fe78df722
DB-cleanup/merge-output/wayback-progress.json
T
StellarCrowandClaude Fable 5.1 439e83a4cb feat(sources): Wayback pass, automatic trimming, detention rule, ten cases re-sourced 
Second and third rounds on the dossier pipeline, reviewed locally before commit.

Retrieval
- wayback_pass.py: retries every BLOCKED/DEAD/THIN/ERROR linked source through
  the Wayback Machine; blocks carry the snapshot date, archive URL and the live
  verdict. 143 slots retried: 93 fetched, 37 no snapshot, 12 script shells, 1 PDF.
- fetch_sources.py: fetch_wayback() helper.
- add_extra_sources.py: files staged extra_* finds (web search, not on the DB
  record) into the store with a provenance note.
- Cases re-sourced by web search: 248 (Oslo: Document.no, Avisa Oslo, NRK; the
  linked Le Parisien piece is case 242 and is marked OFF-CASE), 258 Kharkiv,
  260 Singapore, 358 Bangkok, 365 Las Vegas, 388 Verneuil-sur-Seine, 390 Zoersel,
  430 Homestead. 341 of 364 cases now hold a fetched article; 11, 66 and 399
  have no public text source (podcast, police video, direct victim report).

Trimming
- auto_trim.py: case-anchored furniture cut for UNTRIMMED blocks. Finds the body
  run that mentions the case, merges across subheadings and short furniture
  gaps, drops teasers, share bars, date/URL/caption lines and subscription
  pitches; refuses pages with no record term or almost no body. 302 blocks
  trimmed; 16 left on auto-trim-review.md (7 OFF-CASE suspects). Every cut is
  labelled AUTO-TRIMMED, UNREVIEWED in the dossier; the full extract stays in
  staging/. Decisions with anchors in trim-decisions-auto.json.
- build_cases.py / verify_all.py: banners for auto-trimmed and archived blocks,
  staged-file check extended to Wayback blocks, flag legend under the record.

Detention rule
- README "Definitions": the DB field `kidnappings` is the DETENTION violence
  type (victim, guard, staff or relative held to force submission or execute
  the theft), distinct from the Kidnapping scenario (taken away and held).
- detention-flag-review.md / detention-flag-corrections.json: 40 records
  reviewed with evidence; 30 set-to-1 proposals accepted by the owner on
  2026-09-06 (listed in corrections-approved.md), 10 still open.
- apply_detention_wording.py: "Violence Used" in the 70 reviewed summaries now
  names detention explicitly (62 of 70 labelled), supported by the summary's
  own text; idempotent; supersedes the batch scripts' wording.

Worklist and docs
- sofia-worklist.md: 248 decisions, detention rule item replacing the old
  "no abduction" item, fresh-search section for the textless cases.
- README-START-HERE.md: progress notes, run commands, next steps.
- Bug fixed in passing: Wayback blocks stored in-memory text with carriage
  returns; now stored as read back from disk.

verify_all.py PASSES (2321 checks).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GZZENdTLzNGsbNy4DyF1yt
2026-09-06 19:36:45 +02:00

145 lines
3.5 KiB
JSON
Raw Blame History

{
"451:url": "THIN",
"449:url": "FETCHED",
"448:url": "THIN",
"446:url": "THIN",
"445:url": "THIN",
"444:url": "THIN",
"442:url_2": "FETCHED",
"441:url": "FETCHED",
"440:url": "FETCHED",
"440:url_2": "FETCHED",
"439:url": "FETCHED",
"439:url_2": "FETCHED",
"438:url_2": "FETCHED",
"438:url_3": "FETCHED",
"437:url": "FETCHED",
"436:url_3": "NO SNAPSHOT",
"431:url": "NO SNAPSHOT",
"431:url_2": "NO SNAPSHOT",
"430:url": "NO SNAPSHOT",
"427:url_2": "THIN",
"426:url": "FETCHED",
"426:url_2": "NO SNAPSHOT",
"424:url_2": "FETCHED",
"423:url_2": "FETCHED",
"422:url_2": "FETCHED",
"417:url_2": "FETCHED",
"416:url": "THIN",
"414:url": "FETCHED",
"412:url": "NO SNAPSHOT",
"412:url_4": "NO SNAPSHOT",
"412:url_5": "NO SNAPSHOT",
"411:url_2": "FETCHED",
"410:url": "THIN",
"408:url": "NO SNAPSHOT",
"407:url_3": "FETCHED",
"407:url_4": "FETCHED",
"405:url_2": "NO SNAPSHOT",
"402:url_3": "NO SNAPSHOT",
"395:url": "FETCHED",
"393:url_2": "FETCHED",
"391:url": "FETCHED",
"390:url": "NO SNAPSHOT",
"390:url_2": "NO SNAPSHOT",
"389:url_2": "FETCHED",
"388:url": "NO SNAPSHOT",
"387:url": "FETCHED",
"384:url_2": "NO SNAPSHOT",
"383:url_2": "NO SNAPSHOT",
"382:url": "FETCHED",
"382:url_2": "THIN",
"382:url_3": "NO SNAPSHOT",
"382:url_4": "NO SNAPSHOT",
"381:url": "FETCHED",
"379:url_3": "FETCHED",
"372:url": "FETCHED",
"370:url": "THIN",
"369:url": "FETCHED",
"367:url": "FETCHED",
"365:url": "NO SNAPSHOT",
"363:url_4": "NO SNAPSHOT",
"361:url": "FETCHED",
"360:url_2": "FETCHED",
"258:url": "NO SNAPSHOT",
"45:url": "FETCHED",
"355:url": "FETCHED",
"354:url": "FETCHED",
"349:url": "FETCHED",
"323:url": "FETCHED",
"322:url": "FETCHED",
"318:url": "NO SNAPSHOT",
"307:url": "FETCHED",
"293:url": "FETCHED",
"290:url": "FETCHED",
"288:url": "FETCHED",
"285:url": "FETCHED",
"271:url": "FETCHED",
"263:url": "FETCHED",
"256:url": "FETCHED",
"255:url_2": "FETCHED",
"255:url_3": "FETCHED",
"253:url": "NO SNAPSHOT",
"248:url": "FETCHED",
"247:url": "FETCHED",
"237:url": "NO SNAPSHOT",
"229:url": "FETCHED",
"228:url_2": "FETCHED",
"223:url": "FETCHED",
"222:url": "FETCHED",
"219:url": "FETCHED",
"218:url": "FETCHED",
"214:url": "NO SNAPSHOT",
"213:url": "FETCHED",
"212:url": "FETCHED",
"210:url": "FETCHED",
"209:url_4": "FETCHED",
"207:url": "NO SNAPSHOT",
"199:url": "FETCHED",
"197:url": "NO SNAPSHOT",
"195:url": "FETCHED",
"194:url": "FETCHED",
"188:url_2": "FETCHED",
"188:url_3": "FETCHED",
"183:url": "FETCHED",
"179:url": "NO SNAPSHOT",
"176:url": "NO SNAPSHOT",
"168:url": "FETCHED",
"164:url": "FETCHED",
"162:url": "FETCHED",
"160:url": "FETCHED",
"157:url": "NO SNAPSHOT",
"152:url": "FETCHED",
"151:url": "FETCHED",
"145:url": "FETCHED",
"141:url": "FETCHED",
"138:url": "NO SNAPSHOT",
"116:url": "FETCHED",
"113:url": "FETCHED",
"111:url": "FETCHED",
"110:url": "FETCHED",
"105:url": "FETCHED",
"101:url": "FETCHED",
"91:url": "FETCHED",
"80:url": "NO SNAPSHOT",
"79:url": "THIN",
"71:url": "FETCHED",
"69:url": "FETCHED",
"67:url": "NO SNAPSHOT",
"62:url": "FETCHED",
"60:url": "FETCHED",
"57:url": "NO SNAPSHOT",
"56:url": "FETCHED",
"54:url": "FETCHED",
"44:url": "FETCHED",
"41:url": "FETCHED",
"40:url": "FETCHED",
"33:url": "FETCHED",
"32:url": "NO SNAPSHOT",
"30:url": "NO SNAPSHOT",
"21:url": "NO SNAPSHOT",
"18:url": "PDF",
"16:url": "FETCHED",
"9:url": "THIN",
"6:url": "FETCHED"
}
Reference in New Issue View Git Blame Copy Permalink