feat(sources): Wayback pass, automatic trimming, detention rule, ten cases re-sourced

Second and third rounds on the dossier pipeline, reviewed locally before commit.

Retrieval
- wayback_pass.py: retries every BLOCKED/DEAD/THIN/ERROR linked source through
  the Wayback Machine; blocks carry the snapshot date, archive URL and the live
  verdict. 143 slots retried: 93 fetched, 37 no snapshot, 12 script shells, 1 PDF.
- fetch_sources.py: fetch_wayback() helper.
- add_extra_sources.py: files staged extra_* finds (web search, not on the DB
  record) into the store with a provenance note.
- Cases re-sourced by web search: 248 (Oslo: Document.no, Avisa Oslo, NRK; the
  linked Le Parisien piece is case 242 and is marked OFF-CASE), 258 Kharkiv,
  260 Singapore, 358 Bangkok, 365 Las Vegas, 388 Verneuil-sur-Seine, 390 Zoersel,
  430 Homestead. 341 of 364 cases now hold a fetched article; 11, 66 and 399
  have no public text source (podcast, police video, direct victim report).

Trimming
- auto_trim.py: case-anchored furniture cut for UNTRIMMED blocks. Finds the body
  run that mentions the case, merges across subheadings and short furniture
  gaps, drops teasers, share bars, date/URL/caption lines and subscription
  pitches; refuses pages with no record term or almost no body. 302 blocks
  trimmed; 16 left on auto-trim-review.md (7 OFF-CASE suspects). Every cut is
  labelled AUTO-TRIMMED, UNREVIEWED in the dossier; the full extract stays in
  staging/. Decisions with anchors in trim-decisions-auto.json.
- build_cases.py / verify_all.py: banners for auto-trimmed and archived blocks,
  staged-file check extended to Wayback blocks, flag legend under the record.

Detention rule
- README "Definitions": the DB field `kidnappings` is the DETENTION violence
  type (victim, guard, staff or relative held to force submission or execute
  the theft), distinct from the Kidnapping scenario (taken away and held).
- detention-flag-review.md / detention-flag-corrections.json: 40 records
  reviewed with evidence; 30 set-to-1 proposals accepted by the owner on
  2026-09-06 (listed in corrections-approved.md), 10 still open.
- apply_detention_wording.py: "Violence Used" in the 70 reviewed summaries now
  names detention explicitly (62 of 70 labelled), supported by the summary's
  own text; idempotent; supersedes the batch scripts' wording.

Worklist and docs
- sofia-worklist.md: 248 decisions, detention rule item replacing the old
  "no abduction" item, fresh-search section for the textless cases.
- README-START-HERE.md: progress notes, run commands, next steps.
- Bug fixed in passing: Wayback blocks stored in-memory text with carriage
  returns; now stored as read back from disk.

verify_all.py PASSES (2321 checks).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GZZENdTLzNGsbNy4DyF1yt
This commit is contained in:
StellarCrow
2026-09-06 19:36:45 +02:00
co-authored by Claude Fable 5.1
parent 499fd61f79
commit 439e83a4cb
588 changed files with 24800 additions and 13190 deletions
+64 -4
View File
@@ -67,6 +67,42 @@ tests - just Python scripts that regenerate the .txt files, plus Markdown notes.
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
Put that inside main() if another script imports the file.
--------------------------------------------------------------------------------
## Definitions: detention (violence type) vs Kidnapping (scenario)
Added 2026-09-06 after the flag turned out to be mislabelled. The rule for every file,
summary and correction from now on:
**Detention** is a VIOLENCE TYPE. The DB flag `kidnappings` (sheet column "Kidnappings") is
the detention flag and should be renamed `detention` on the DB side. It is 1 when the
attackers had physical control of the victim at any point, in any place, for any length of
time: tied, taped, handcuffed or zip-tied; locked in a room or vehicle; held at gunpoint
while forced to unlock or transfer; taken away in a car; held under a fake arrest; kept for
hours or days. It is 0 when control was never gained: a snatch or mugging in passing, a
break-in of empty premises, threats or extortion at a distance, a plot foiled before contact,
an attempt the victim fled or fought off before being restrained. Judgment line: a victim
held at gunpoint for the minutes it takes to force a transfer counts as detained; a victim
punched and left within seconds does not. Who was held does not change the flag: if detention
was used on anyone to force submission or to execute the theft (guards, staff, a concierge, a
relative), the flag is 1 even when the record's named victim was absent (owner's rule, 2026-09-06).
**Kidnapping** is a SCENARIO, one of eleven attack patterns. It means the victim was taken
from where they were to a place under the attackers' control and held there, usually for a
ransom or a forced transfer. Express Kidnapping is the short, ATM-style variant. A Kidnapping
scenario always involves detention, and usually other violence types too: weapon threats,
assault, torture, drugging. Detention also occurs in other scenarios without making them
kidnappings: a family tied up in a Home Invasion, a P2P buyer handcuffed in a hotel room, a
fake arrest under Authority Misuse, a victim zip-tied in a Dodged Attack that was interrupted.
**Wording in summaries.** "Violence Used" names the violence types, and names detention
explicitly whenever it occurred ("Detention — held at gunpoint for about an hour"). The
words kidnapped / abducted are reserved for a victim being taken away; a victim held where
they were is "held", "detained", "tied up", not "kidnapped". Foiled attempts say
"No detention — the attempt was foiled" only when the source says so.
Per-case flag review against this rule: merge-output/detention-flag-review.md (proposals for
Sofia, with the evidence line for each). The DB export stays untouched.
--------------------------------------------------------------------------------
## The deliverable, and what still needs the human
@@ -120,6 +156,8 @@ than 4 categories). These are the targets, worked in descending ID order.
- Round-up articles (one article, several incidents): use ONLY the passage about THIS case;
drop the other incidents and any trend-commentary tail.
- Foiled / "dodged" attacks with no one harmed: "Violence Used: None - the attempt was foiled."
- Name detention explicitly in "Violence Used" whenever the victim was held (see Definitions);
use kidnapped / abducted only for a victim taken away.
### Workflow per batch (about 10 cases)
1. Write build_overrides_batchN.py (copy an existing one; it does existing.update(B) then writes
@@ -189,10 +227,32 @@ Result of the run (all 252 cases processed):
paste, which is kept. Trimming is still to do, same as for cases 357+.
- verify_all.py PASSES (1836 checks).
Next step for the 73 cases with no fetched article: wayback-availability.json lists a
Wayback Machine snapshot for 73 of the 98 failed URL slots (about 70 cases). fetch_sources.py
has no archive path yet; add one that labels the block as an archived snapshot with its
timestamp, so provenance stays honest.
Second pass, same day: Wayback Machine retrieval and automatic trimming.
- wayback_pass.py retries every linked source the live fetch could not read (BLOCKED,
DEAD, THIN, ERROR) through the Wayback Machine. Run from merge-output/:
python wayback_pass.py
Result over 143 failed slots: 93 FETCHED, 37 no snapshot, 12 THIN (script shell), 1 PDF.
Every archived block is labelled with the snapshot date and archive URL in the dossier
and keeps the live verdict in its note. 341 of 364 cases now hold a fetched article;
3 cases still have no article text at all: 11 (podcast only), 66 (police Facebook video only),
399 (direct victim report to Gart Research, no public source). The other seven were sourced by
web search on 2026-09-06; add_extra_sources.py files such finds into the store.
- auto_trim.py cuts site furniture from UNTRIMMED blocks: it finds the paragraph run that
mentions the case (terms from the DB record), widens it across short gaps so ledes and
closing sentences survive, and drops teasers, share bars, date/URL/caption lines and
subscription pitches. Run from merge-output/:
python auto_trim.py (add --reset to redo every automatic cut from staging/)
Result: 288 blocks in 252 cases trimmed, 1.44M -> 0.99M chars. These are MACHINE cuts:
the dossier banner says AUTO-TRIMMED, UNREVIEWED, and the full extract stays in staging/.
Decisions with anchors are in trim-decisions-auto.json.
- auto-trim-review.md lists the 16 blocks left UNTRIMMED and the 15 trimmed on a weak
anchor. The 7 OFF-CASE entries (no record term on the page) are wrong-source candidates;
case 248 is on it, which matches the known Le Parisien mismatch.
- The trimmer removes furniture only. It does NOT separate other incidents inside a
round-up article; that is still a read-through job (see the round-up bucket on the
worklist).
- Case 248: Oslo sources found by web search on 2026-09-05 and filed as extra_1..5 (see the worklist entry). verify_all.py PASSES (2321 checks). Not committed yet; review before pushing.
--------------------------------------------------------------------------------
## Quick-start checklist for the next AI