feat(sources): Wayback pass, automatic trimming, detention rule, ten cases re-sourced
Second and third rounds on the dossier pipeline, reviewed locally before commit. Retrieval - wayback_pass.py: retries every BLOCKED/DEAD/THIN/ERROR linked source through the Wayback Machine; blocks carry the snapshot date, archive URL and the live verdict. 143 slots retried: 93 fetched, 37 no snapshot, 12 script shells, 1 PDF. - fetch_sources.py: fetch_wayback() helper. - add_extra_sources.py: files staged extra_* finds (web search, not on the DB record) into the store with a provenance note. - Cases re-sourced by web search: 248 (Oslo: Document.no, Avisa Oslo, NRK; the linked Le Parisien piece is case 242 and is marked OFF-CASE), 258 Kharkiv, 260 Singapore, 358 Bangkok, 365 Las Vegas, 388 Verneuil-sur-Seine, 390 Zoersel, 430 Homestead. 341 of 364 cases now hold a fetched article; 11, 66 and 399 have no public text source (podcast, police video, direct victim report). Trimming - auto_trim.py: case-anchored furniture cut for UNTRIMMED blocks. Finds the body run that mentions the case, merges across subheadings and short furniture gaps, drops teasers, share bars, date/URL/caption lines and subscription pitches; refuses pages with no record term or almost no body. 302 blocks trimmed; 16 left on auto-trim-review.md (7 OFF-CASE suspects). Every cut is labelled AUTO-TRIMMED, UNREVIEWED in the dossier; the full extract stays in staging/. Decisions with anchors in trim-decisions-auto.json. - build_cases.py / verify_all.py: banners for auto-trimmed and archived blocks, staged-file check extended to Wayback blocks, flag legend under the record. Detention rule - README "Definitions": the DB field `kidnappings` is the DETENTION violence type (victim, guard, staff or relative held to force submission or execute the theft), distinct from the Kidnapping scenario (taken away and held). - detention-flag-review.md / detention-flag-corrections.json: 40 records reviewed with evidence; 30 set-to-1 proposals accepted by the owner on 2026-09-06 (listed in corrections-approved.md), 10 still open. - apply_detention_wording.py: "Violence Used" in the 70 reviewed summaries now names detention explicitly (62 of 70 labelled), supported by the summary's own text; idempotent; supersedes the batch scripts' wording. Worklist and docs - sofia-worklist.md: 248 decisions, detention rule item replacing the old "no abduction" item, fresh-search section for the textless cases. - README-START-HERE.md: progress notes, run commands, next steps. - Bug fixed in passing: Wayback blocks stored in-memory text with carriage returns; now stored as read back from disk. verify_all.py PASSES (2321 checks). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GZZENdTLzNGsbNy4DyF1yt
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
499fd61f79
commit
439e83a4cb
@@ -115,6 +115,30 @@ def fetch_raw(url, timeout=30):
|
||||
return r.status, raw.decode(charset, errors="replace"), r.geturl()
|
||||
|
||||
|
||||
WAYBACK_AVAIL = "https://archive.org/wayback/available?url="
|
||||
|
||||
|
||||
def fetch_wayback(url, timeout=40):
|
||||
"""Fetch the closest Wayback Machine snapshot of a URL.
|
||||
|
||||
Returns (timestamp, snapshot_url, html) or (None, None, None) when nothing is
|
||||
archived. The `id_` flag asks for the original bytes without the Wayback toolbar,
|
||||
so extraction runs on the same markup a live fetch would have seen. A snapshot is
|
||||
a third party's copy taken on a given date, so anything staged from it must say so.
|
||||
"""
|
||||
import urllib.parse
|
||||
q = WAYBACK_AVAIL + urllib.parse.quote(url, safe="")
|
||||
with urllib.request.urlopen(urllib.request.Request(q, headers={"User-Agent": UA}),
|
||||
timeout=timeout) as r:
|
||||
closest = json.loads(r.read().decode("utf-8")).get("archived_snapshots", {}).get("closest")
|
||||
if not closest or not closest.get("available"):
|
||||
return None, None, None
|
||||
ts = closest["timestamp"]
|
||||
raw_url = f"https://web.archive.org/web/{ts}id_/{url}"
|
||||
status, html, final = fetch_raw(raw_url, timeout=timeout)
|
||||
return ts, closest["url"], html
|
||||
|
||||
|
||||
def stage_case(rec, extra_urls=(), new_only=False):
|
||||
cid = rec["id"]
|
||||
out = STAGING / f"{cid:03d}"
|
||||
|
||||
Reference in New Issue
Block a user