feat(sources): Wayback pass, automatic trimming, detention rule, ten cases re-sourced

Second and third rounds on the dossier pipeline, reviewed locally before commit.

Retrieval
- wayback_pass.py: retries every BLOCKED/DEAD/THIN/ERROR linked source through
  the Wayback Machine; blocks carry the snapshot date, archive URL and the live
  verdict. 143 slots retried: 93 fetched, 37 no snapshot, 12 script shells, 1 PDF.
- fetch_sources.py: fetch_wayback() helper.
- add_extra_sources.py: files staged extra_* finds (web search, not on the DB
  record) into the store with a provenance note.
- Cases re-sourced by web search: 248 (Oslo: Document.no, Avisa Oslo, NRK; the
  linked Le Parisien piece is case 242 and is marked OFF-CASE), 258 Kharkiv,
  260 Singapore, 358 Bangkok, 365 Las Vegas, 388 Verneuil-sur-Seine, 390 Zoersel,
  430 Homestead. 341 of 364 cases now hold a fetched article; 11, 66 and 399
  have no public text source (podcast, police video, direct victim report).

Trimming
- auto_trim.py: case-anchored furniture cut for UNTRIMMED blocks. Finds the body
  run that mentions the case, merges across subheadings and short furniture
  gaps, drops teasers, share bars, date/URL/caption lines and subscription
  pitches; refuses pages with no record term or almost no body. 302 blocks
  trimmed; 16 left on auto-trim-review.md (7 OFF-CASE suspects). Every cut is
  labelled AUTO-TRIMMED, UNREVIEWED in the dossier; the full extract stays in
  staging/. Decisions with anchors in trim-decisions-auto.json.
- build_cases.py / verify_all.py: banners for auto-trimmed and archived blocks,
  staged-file check extended to Wayback blocks, flag legend under the record.

Detention rule
- README "Definitions": the DB field `kidnappings` is the DETENTION violence
  type (victim, guard, staff or relative held to force submission or execute
  the theft), distinct from the Kidnapping scenario (taken away and held).
- detention-flag-review.md / detention-flag-corrections.json: 40 records
  reviewed with evidence; 30 set-to-1 proposals accepted by the owner on
  2026-09-06 (listed in corrections-approved.md), 10 still open.
- apply_detention_wording.py: "Violence Used" in the 70 reviewed summaries now
  names detention explicitly (62 of 70 labelled), supported by the summary's
  own text; idempotent; supersedes the batch scripts' wording.

Worklist and docs
- sofia-worklist.md: 248 decisions, detention rule item replacing the old
  "no abduction" item, fresh-search section for the textless cases.
- README-START-HERE.md: progress notes, run commands, next steps.
- Bug fixed in passing: Wayback blocks stored in-memory text with carriage
  returns; now stored as read back from disk.

verify_all.py PASSES (2321 checks).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GZZENdTLzNGsbNy4DyF1yt
This commit is contained in:
StellarCrow
2026-09-06 19:36:45 +02:00
co-authored by Claude Fable 5.1
parent 499fd61f79
commit 439e83a4cb
588 changed files with 24800 additions and 13190 deletions
+46 -3
View File
@@ -155,6 +155,27 @@ portion likely contains the Gignac-la-Nerthe detail (458) and any answer on the
<https://www.laprovence.com/article/faits-divers-justice/2193885092828214/quatre-hommes-arretes-en-plein-braquage-sur-fond-de-cryptomonnaies-pres-de-marseille>
- **Case 391** (Chinese national rescue, 2 arrested) — its only linked source (abs-cbn.com) is JS-rendered and returned no article body. Needs a manual/browser fetch or a replacement source. No usable article on file otherwise.
## Cases with no article text — fresh search 2026-09-06
Seven of the ten were found by web search and filed as extra sources (marked "NOT on the DB record"
in the dossier; add the URLs to the record if you agree): 258 Kharkiv (Ukrinform, Hromadske Radio,
Sud.ua, ATN — also gives the victim's age 32, a Mercedes Vito, 14,600 EUR cash taken first, and one
active serviceman among the three), 260 Nate Geier (Mothership), 358 Bangkok (Khaosod English —
victim attacked 1 November on Mahaseth Road, not 10 November; the DB date is the first arrest),
365 Las Vegas (The Defiant; 8 News Now is the original but blocks fetches — it is already the source
on case 323, the duplicate record), 388 Verneuil-sur-Seine (Actu17, Mantes Actu), 390 Zoersel (VRT NWS,
NNieuws, In de Gazette — VRT says two intruders, firearm, facial injuries, Korte Maalstraat, 10 January),
430 Homestead (Decrypt, Fortune; the DOJ press release is the primary source but fetches as an empty
shell — copy it by hand from justice.gov/usao-mdnc, "Twelve Defendants Sentenced ... Cryptocurrency").
Still nothing on file for three cases:
- **case 11** (Josoj, London 2017): the only source is a podcast episode (ivoox, Bitcoin Uncensored).
No text article exists that I could find. Options: transcribe the episode (would be a derived
text, label it so), or link the LocalBitcoins forum thread where the victim posted, if you know it.
- **case 66** (East Lansdowne BTM, May 2020): the only source is the East Lansdowne Police Facebook
video. Search Delaware County local press (Delco Times, 6abc, NBC10) for "bitcoin ATM" "East
Lansdowne" May 2020, or ask the police department for the incident number.
- **case 399** (Dominican Republic, Oct 2025): reported directly to Gart Research by the victim; there
is no public source. File the victim's own account as the source, labelled as a first-person report.
## Round-up articles needing per-case passage isolation (Claude to do, your review)
One article covers several different cases; each case file must keep only its own passage.
- Known set: 68, 315, 422, 445, 450.
@@ -164,8 +185,23 @@ One article covers several different cases; each case file must keep only its ow
## Wrong-source / mismatched cases (the linked article does not describe this record)
- **Case 248** — the linked leparisien.fr article is a DIFFERENT incident (a woman assaulted at home in
Suresnes, France), NOT the Oslo Norwegian-family home invasion this record describes. That Suresnes
event is **case 242**, which already has the correct source — so the misfiled article belongs to 242's
story. Find a correct Oslo source for 248. (248's summary was written from its record fields only.)
event is **case 242**, which already has a BFMTV source — move the Le Parisien URL to 242.
**UPDATE 2026-09-05:** Oslo sources found by web search and filed into 248 (marked "NOT on the DB
record" in the dossier): Document.no 09.08.2025 (the detailed account: Haugerud, Foodora disguise,
parents in their 40s, children 4/8/12 taped up, mother threatened with being shot, crypto demanded,
four men from Sweden aged 18-22), Avisa Oslo 06.08.2025 (police statement, incident reported 23:53
on Tuesday 5 August), NRK 30.08.2025 (nine charged, one man in his late 20s remanded four weeks,
straw men rented a hideout flat), Document.no 04.09.2025 (nine charged for aggravated robbery and
deprivation of liberty, three adults and two minors bound, one threatened with a firearm, one
subjected to serious violence, two arrested abroad on international warrants). Decisions for you:
(a) add these URLs to the record; (b) `date` = August 6 but police log the call at 23:53 on
**5 August** — pick one and say "night of 5-6 August"; (c) the record's "Ledger wallet" search and
"handler / third-party DB leak" notes do not appear in any Oslo source found — the handler detail
is a Suresnes fact and probably leaked in from the wrong article; Document.no instead says police
suspect the job was ordered by people close to the family. Confirm or drop; (d) Document.no says
three adults were bound per police, while the record says a family with three children — the
police count (3 adults + 2 minors) and Document's (2 parents + 3 children) disagree; keep both
with attribution.
- **Case 441** — its fetched sources (url_2, url_3) actually describe case 438 (same Montreal
home-invasion cluster, wrong incident). Decide: drop them, or move to 438.
- **Case 295** — the Forbes source (url_3) predates the event it is filed under, so it cannot be
@@ -210,7 +246,14 @@ row 252 wrong URL / Phuket case). Additional items found during the summary pass
(knives + cleaver seized); 243 (tased); 245 (weapon recovered).
- **`theft` = 0 but property was taken:** 303 (Dh183k + phones/cheques); 297 (Dh1.7m cash); 273 (dog,
MacBook, jewellery); and see 218 above.
- **`kidnappings` = 1 but no abduction (attacked/robbed in place):** 293, 271, 273, 298.
- **`kidnappings` flag — RULE CHANGE 2026-09-06.** The flag is the DETENTION flag (victim held against
their will, any place, any duration), not "abduction". Please rename the DB field `kidnappings` to
`detention` and the sheet column "Kidnappings" to "Detention"; the scenario name "Kidnapping" stays.
Full rule: README-START-HERE.md, "Definitions". Per-case proposals with evidence:
`detention-flag-review.md` / `detention-flag-corrections.json` (40 records; on 2026-09-06 the owner ACCEPTED all 30 set-to-1 proposals (rule clarified: detention of
anyone, guards or staff included, counts); the 10 set-to-0 items incl. the two nulls and 2 checks are still open). This REPLACES the earlier item
"`kidnappings` = 1 but no abduction: 293, 271, 273, 298" — under the detention rule 293 and 271 stay 1
(held at gunpoint / forced back inside), 273 goes to 0 (stabbed, not held), 298 needs a source check.
- **Case 237** — `violence_torture` = 0 but the source says he was beaten; `theft` = 1 though nothing was
taken (released once found "broke").
- **Case 228** (NYC SoHo torture) — the linked ABC source withholds the victim's name; the DB `victim`