feat(sources): Wayback pass, automatic trimming, detention rule, ten cases re-sourced
Second and third rounds on the dossier pipeline, reviewed locally before commit. Retrieval - wayback_pass.py: retries every BLOCKED/DEAD/THIN/ERROR linked source through the Wayback Machine; blocks carry the snapshot date, archive URL and the live verdict. 143 slots retried: 93 fetched, 37 no snapshot, 12 script shells, 1 PDF. - fetch_sources.py: fetch_wayback() helper. - add_extra_sources.py: files staged extra_* finds (web search, not on the DB record) into the store with a provenance note. - Cases re-sourced by web search: 248 (Oslo: Document.no, Avisa Oslo, NRK; the linked Le Parisien piece is case 242 and is marked OFF-CASE), 258 Kharkiv, 260 Singapore, 358 Bangkok, 365 Las Vegas, 388 Verneuil-sur-Seine, 390 Zoersel, 430 Homestead. 341 of 364 cases now hold a fetched article; 11, 66 and 399 have no public text source (podcast, police video, direct victim report). Trimming - auto_trim.py: case-anchored furniture cut for UNTRIMMED blocks. Finds the body run that mentions the case, merges across subheadings and short furniture gaps, drops teasers, share bars, date/URL/caption lines and subscription pitches; refuses pages with no record term or almost no body. 302 blocks trimmed; 16 left on auto-trim-review.md (7 OFF-CASE suspects). Every cut is labelled AUTO-TRIMMED, UNREVIEWED in the dossier; the full extract stays in staging/. Decisions with anchors in trim-decisions-auto.json. - build_cases.py / verify_all.py: banners for auto-trimmed and archived blocks, staged-file check extended to Wayback blocks, flag legend under the record. Detention rule - README "Definitions": the DB field `kidnappings` is the DETENTION violence type (victim, guard, staff or relative held to force submission or execute the theft), distinct from the Kidnapping scenario (taken away and held). - detention-flag-review.md / detention-flag-corrections.json: 40 records reviewed with evidence; 30 set-to-1 proposals accepted by the owner on 2026-09-06 (listed in corrections-approved.md), 10 still open. - apply_detention_wording.py: "Violence Used" in the 70 reviewed summaries now names detention explicitly (62 of 70 labelled), supported by the summary's own text; idempotent; supersedes the batch scripts' wording. Worklist and docs - sofia-worklist.md: 248 decisions, detention rule item replacing the old "no abduction" item, fresh-search section for the textless cases. - README-START-HERE.md: progress notes, run commands, next steps. - Bug fixed in passing: Wayback blocks stored in-memory text with carriage returns; now stored as read back from disk. verify_all.py PASSES (2321 checks). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GZZENdTLzNGsbNy4DyF1yt
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
499fd61f79
commit
439e83a4cb
@@ -155,6 +155,27 @@ portion likely contains the Gignac-la-Nerthe detail (458) and any answer on the
|
||||
<https://www.laprovence.com/article/faits-divers-justice/2193885092828214/quatre-hommes-arretes-en-plein-braquage-sur-fond-de-cryptomonnaies-pres-de-marseille>
|
||||
- **Case 391** (Chinese national rescue, 2 arrested) — its only linked source (abs-cbn.com) is JS-rendered and returned no article body. Needs a manual/browser fetch or a replacement source. No usable article on file otherwise.
|
||||
|
||||
## Cases with no article text — fresh search 2026-09-06
|
||||
Seven of the ten were found by web search and filed as extra sources (marked "NOT on the DB record"
|
||||
in the dossier; add the URLs to the record if you agree): 258 Kharkiv (Ukrinform, Hromadske Radio,
|
||||
Sud.ua, ATN — also gives the victim's age 32, a Mercedes Vito, 14,600 EUR cash taken first, and one
|
||||
active serviceman among the three), 260 Nate Geier (Mothership), 358 Bangkok (Khaosod English —
|
||||
victim attacked 1 November on Mahaseth Road, not 10 November; the DB date is the first arrest),
|
||||
365 Las Vegas (The Defiant; 8 News Now is the original but blocks fetches — it is already the source
|
||||
on case 323, the duplicate record), 388 Verneuil-sur-Seine (Actu17, Mantes Actu), 390 Zoersel (VRT NWS,
|
||||
NNieuws, In de Gazette — VRT says two intruders, firearm, facial injuries, Korte Maalstraat, 10 January),
|
||||
430 Homestead (Decrypt, Fortune; the DOJ press release is the primary source but fetches as an empty
|
||||
shell — copy it by hand from justice.gov/usao-mdnc, "Twelve Defendants Sentenced ... Cryptocurrency").
|
||||
Still nothing on file for three cases:
|
||||
- **case 11** (Josoj, London 2017): the only source is a podcast episode (ivoox, Bitcoin Uncensored).
|
||||
No text article exists that I could find. Options: transcribe the episode (would be a derived
|
||||
text, label it so), or link the LocalBitcoins forum thread where the victim posted, if you know it.
|
||||
- **case 66** (East Lansdowne BTM, May 2020): the only source is the East Lansdowne Police Facebook
|
||||
video. Search Delaware County local press (Delco Times, 6abc, NBC10) for "bitcoin ATM" "East
|
||||
Lansdowne" May 2020, or ask the police department for the incident number.
|
||||
- **case 399** (Dominican Republic, Oct 2025): reported directly to Gart Research by the victim; there
|
||||
is no public source. File the victim's own account as the source, labelled as a first-person report.
|
||||
|
||||
## Round-up articles needing per-case passage isolation (Claude to do, your review)
|
||||
One article covers several different cases; each case file must keep only its own passage.
|
||||
- Known set: 68, 315, 422, 445, 450.
|
||||
@@ -164,8 +185,23 @@ One article covers several different cases; each case file must keep only its ow
|
||||
## Wrong-source / mismatched cases (the linked article does not describe this record)
|
||||
- **Case 248** — the linked leparisien.fr article is a DIFFERENT incident (a woman assaulted at home in
|
||||
Suresnes, France), NOT the Oslo Norwegian-family home invasion this record describes. That Suresnes
|
||||
event is **case 242**, which already has the correct source — so the misfiled article belongs to 242's
|
||||
story. Find a correct Oslo source for 248. (248's summary was written from its record fields only.)
|
||||
event is **case 242**, which already has a BFMTV source — move the Le Parisien URL to 242.
|
||||
**UPDATE 2026-09-05:** Oslo sources found by web search and filed into 248 (marked "NOT on the DB
|
||||
record" in the dossier): Document.no 09.08.2025 (the detailed account: Haugerud, Foodora disguise,
|
||||
parents in their 40s, children 4/8/12 taped up, mother threatened with being shot, crypto demanded,
|
||||
four men from Sweden aged 18-22), Avisa Oslo 06.08.2025 (police statement, incident reported 23:53
|
||||
on Tuesday 5 August), NRK 30.08.2025 (nine charged, one man in his late 20s remanded four weeks,
|
||||
straw men rented a hideout flat), Document.no 04.09.2025 (nine charged for aggravated robbery and
|
||||
deprivation of liberty, three adults and two minors bound, one threatened with a firearm, one
|
||||
subjected to serious violence, two arrested abroad on international warrants). Decisions for you:
|
||||
(a) add these URLs to the record; (b) `date` = August 6 but police log the call at 23:53 on
|
||||
**5 August** — pick one and say "night of 5-6 August"; (c) the record's "Ledger wallet" search and
|
||||
"handler / third-party DB leak" notes do not appear in any Oslo source found — the handler detail
|
||||
is a Suresnes fact and probably leaked in from the wrong article; Document.no instead says police
|
||||
suspect the job was ordered by people close to the family. Confirm or drop; (d) Document.no says
|
||||
three adults were bound per police, while the record says a family with three children — the
|
||||
police count (3 adults + 2 minors) and Document's (2 parents + 3 children) disagree; keep both
|
||||
with attribution.
|
||||
- **Case 441** — its fetched sources (url_2, url_3) actually describe case 438 (same Montreal
|
||||
home-invasion cluster, wrong incident). Decide: drop them, or move to 438.
|
||||
- **Case 295** — the Forbes source (url_3) predates the event it is filed under, so it cannot be
|
||||
@@ -210,7 +246,14 @@ row 252 wrong URL / Phuket case). Additional items found during the summary pass
|
||||
(knives + cleaver seized); 243 (tased); 245 (weapon recovered).
|
||||
- **`theft` = 0 but property was taken:** 303 (Dh183k + phones/cheques); 297 (Dh1.7m cash); 273 (dog,
|
||||
MacBook, jewellery); and see 218 above.
|
||||
- **`kidnappings` = 1 but no abduction (attacked/robbed in place):** 293, 271, 273, 298.
|
||||
- **`kidnappings` flag — RULE CHANGE 2026-09-06.** The flag is the DETENTION flag (victim held against
|
||||
their will, any place, any duration), not "abduction". Please rename the DB field `kidnappings` to
|
||||
`detention` and the sheet column "Kidnappings" to "Detention"; the scenario name "Kidnapping" stays.
|
||||
Full rule: README-START-HERE.md, "Definitions". Per-case proposals with evidence:
|
||||
`detention-flag-review.md` / `detention-flag-corrections.json` (40 records; on 2026-09-06 the owner ACCEPTED all 30 set-to-1 proposals (rule clarified: detention of
|
||||
anyone, guards or staff included, counts); the 10 set-to-0 items incl. the two nulls and 2 checks are still open). This REPLACES the earlier item
|
||||
"`kidnappings` = 1 but no abduction: 293, 271, 273, 298" — under the detention rule 293 and 271 stay 1
|
||||
(held at gunpoint / forced back inside), 273 goes to 0 (stabbed, not held), 298 needs a source check.
|
||||
- **Case 237** — `violence_torture` = 0 but the source says he was beaten; `theft` = 1 though nothing was
|
||||
taken (released once found "broke").
|
||||
- **Case 228** (NYC SoHo torture) — the linked ABC source withholds the victim's name; the DB `victim`
|
||||
|
||||
Reference in New Issue
Block a user