feat(sources): Wayback pass, automatic trimming, detention rule, ten cases re-sourced
Second and third rounds on the dossier pipeline, reviewed locally before commit. Retrieval - wayback_pass.py: retries every BLOCKED/DEAD/THIN/ERROR linked source through the Wayback Machine; blocks carry the snapshot date, archive URL and the live verdict. 143 slots retried: 93 fetched, 37 no snapshot, 12 script shells, 1 PDF. - fetch_sources.py: fetch_wayback() helper. - add_extra_sources.py: files staged extra_* finds (web search, not on the DB record) into the store with a provenance note. - Cases re-sourced by web search: 248 (Oslo: Document.no, Avisa Oslo, NRK; the linked Le Parisien piece is case 242 and is marked OFF-CASE), 258 Kharkiv, 260 Singapore, 358 Bangkok, 365 Las Vegas, 388 Verneuil-sur-Seine, 390 Zoersel, 430 Homestead. 341 of 364 cases now hold a fetched article; 11, 66 and 399 have no public text source (podcast, police video, direct victim report). Trimming - auto_trim.py: case-anchored furniture cut for UNTRIMMED blocks. Finds the body run that mentions the case, merges across subheadings and short furniture gaps, drops teasers, share bars, date/URL/caption lines and subscription pitches; refuses pages with no record term or almost no body. 302 blocks trimmed; 16 left on auto-trim-review.md (7 OFF-CASE suspects). Every cut is labelled AUTO-TRIMMED, UNREVIEWED in the dossier; the full extract stays in staging/. Decisions with anchors in trim-decisions-auto.json. - build_cases.py / verify_all.py: banners for auto-trimmed and archived blocks, staged-file check extended to Wayback blocks, flag legend under the record. Detention rule - README "Definitions": the DB field `kidnappings` is the DETENTION violence type (victim, guard, staff or relative held to force submission or execute the theft), distinct from the Kidnapping scenario (taken away and held). - detention-flag-review.md / detention-flag-corrections.json: 40 records reviewed with evidence; 30 set-to-1 proposals accepted by the owner on 2026-09-06 (listed in corrections-approved.md), 10 still open. - apply_detention_wording.py: "Violence Used" in the 70 reviewed summaries now names detention explicitly (62 of 70 labelled), supported by the summary's own text; idempotent; supersedes the batch scripts' wording. Worklist and docs - sofia-worklist.md: 248 decisions, detention rule item replacing the old "no abduction" item, fresh-search section for the textless cases. - README-START-HERE.md: progress notes, run commands, next steps. - Bug fixed in passing: Wayback blocks stored in-memory text with carriage returns; now stored as read back from disk. verify_all.py PASSES (2321 checks). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GZZENdTLzNGsbNy4DyF1yt
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
499fd61f79
commit
439e83a4cb
+64
-4
@@ -67,6 +67,42 @@ tests - just Python scripts that regenerate the .txt files, plus Markdown notes.
|
||||
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
|
||||
Put that inside main() if another script imports the file.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
## Definitions: detention (violence type) vs Kidnapping (scenario)
|
||||
|
||||
Added 2026-09-06 after the flag turned out to be mislabelled. The rule for every file,
|
||||
summary and correction from now on:
|
||||
|
||||
**Detention** is a VIOLENCE TYPE. The DB flag `kidnappings` (sheet column "Kidnappings") is
|
||||
the detention flag and should be renamed `detention` on the DB side. It is 1 when the
|
||||
attackers had physical control of the victim at any point, in any place, for any length of
|
||||
time: tied, taped, handcuffed or zip-tied; locked in a room or vehicle; held at gunpoint
|
||||
while forced to unlock or transfer; taken away in a car; held under a fake arrest; kept for
|
||||
hours or days. It is 0 when control was never gained: a snatch or mugging in passing, a
|
||||
break-in of empty premises, threats or extortion at a distance, a plot foiled before contact,
|
||||
an attempt the victim fled or fought off before being restrained. Judgment line: a victim
|
||||
held at gunpoint for the minutes it takes to force a transfer counts as detained; a victim
|
||||
punched and left within seconds does not. Who was held does not change the flag: if detention
|
||||
was used on anyone to force submission or to execute the theft (guards, staff, a concierge, a
|
||||
relative), the flag is 1 even when the record's named victim was absent (owner's rule, 2026-09-06).
|
||||
|
||||
**Kidnapping** is a SCENARIO, one of eleven attack patterns. It means the victim was taken
|
||||
from where they were to a place under the attackers' control and held there, usually for a
|
||||
ransom or a forced transfer. Express Kidnapping is the short, ATM-style variant. A Kidnapping
|
||||
scenario always involves detention, and usually other violence types too: weapon threats,
|
||||
assault, torture, drugging. Detention also occurs in other scenarios without making them
|
||||
kidnappings: a family tied up in a Home Invasion, a P2P buyer handcuffed in a hotel room, a
|
||||
fake arrest under Authority Misuse, a victim zip-tied in a Dodged Attack that was interrupted.
|
||||
|
||||
**Wording in summaries.** "Violence Used" names the violence types, and names detention
|
||||
explicitly whenever it occurred ("Detention — held at gunpoint for about an hour"). The
|
||||
words kidnapped / abducted are reserved for a victim being taken away; a victim held where
|
||||
they were is "held", "detained", "tied up", not "kidnapped". Foiled attempts say
|
||||
"No detention — the attempt was foiled" only when the source says so.
|
||||
|
||||
Per-case flag review against this rule: merge-output/detention-flag-review.md (proposals for
|
||||
Sofia, with the evidence line for each). The DB export stays untouched.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
## The deliverable, and what still needs the human
|
||||
|
||||
@@ -120,6 +156,8 @@ than 4 categories). These are the targets, worked in descending ID order.
|
||||
- Round-up articles (one article, several incidents): use ONLY the passage about THIS case;
|
||||
drop the other incidents and any trend-commentary tail.
|
||||
- Foiled / "dodged" attacks with no one harmed: "Violence Used: None - the attempt was foiled."
|
||||
- Name detention explicitly in "Violence Used" whenever the victim was held (see Definitions);
|
||||
use kidnapped / abducted only for a victim taken away.
|
||||
|
||||
### Workflow per batch (about 10 cases)
|
||||
1. Write build_overrides_batchN.py (copy an existing one; it does existing.update(B) then writes
|
||||
@@ -189,10 +227,32 @@ Result of the run (all 252 cases processed):
|
||||
paste, which is kept. Trimming is still to do, same as for cases 357+.
|
||||
- verify_all.py PASSES (1836 checks).
|
||||
|
||||
Next step for the 73 cases with no fetched article: wayback-availability.json lists a
|
||||
Wayback Machine snapshot for 73 of the 98 failed URL slots (about 70 cases). fetch_sources.py
|
||||
has no archive path yet; add one that labels the block as an archived snapshot with its
|
||||
timestamp, so provenance stays honest.
|
||||
Second pass, same day: Wayback Machine retrieval and automatic trimming.
|
||||
|
||||
- wayback_pass.py retries every linked source the live fetch could not read (BLOCKED,
|
||||
DEAD, THIN, ERROR) through the Wayback Machine. Run from merge-output/:
|
||||
python wayback_pass.py
|
||||
Result over 143 failed slots: 93 FETCHED, 37 no snapshot, 12 THIN (script shell), 1 PDF.
|
||||
Every archived block is labelled with the snapshot date and archive URL in the dossier
|
||||
and keeps the live verdict in its note. 341 of 364 cases now hold a fetched article;
|
||||
3 cases still have no article text at all: 11 (podcast only), 66 (police Facebook video only),
|
||||
399 (direct victim report to Gart Research, no public source). The other seven were sourced by
|
||||
web search on 2026-09-06; add_extra_sources.py files such finds into the store.
|
||||
- auto_trim.py cuts site furniture from UNTRIMMED blocks: it finds the paragraph run that
|
||||
mentions the case (terms from the DB record), widens it across short gaps so ledes and
|
||||
closing sentences survive, and drops teasers, share bars, date/URL/caption lines and
|
||||
subscription pitches. Run from merge-output/:
|
||||
python auto_trim.py (add --reset to redo every automatic cut from staging/)
|
||||
Result: 288 blocks in 252 cases trimmed, 1.44M -> 0.99M chars. These are MACHINE cuts:
|
||||
the dossier banner says AUTO-TRIMMED, UNREVIEWED, and the full extract stays in staging/.
|
||||
Decisions with anchors are in trim-decisions-auto.json.
|
||||
- auto-trim-review.md lists the 16 blocks left UNTRIMMED and the 15 trimmed on a weak
|
||||
anchor. The 7 OFF-CASE entries (no record term on the page) are wrong-source candidates;
|
||||
case 248 is on it, which matches the known Le Parisien mismatch.
|
||||
- The trimmer removes furniture only. It does NOT separate other incidents inside a
|
||||
round-up article; that is still a read-through job (see the round-up bucket on the
|
||||
worklist).
|
||||
- Case 248: Oslo sources found by web search on 2026-09-05 and filed as extra_1..5 (see the worklist entry). verify_all.py PASSES (2321 checks). Not committed yet; review before pushing.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
## Quick-start checklist for the next AI
|
||||
|
||||
Reference in New Issue
Block a user