diff --git a/project/CHANGELOG.d/SCRIP-CORPUS-086.md b/project/CHANGELOG.d/SCRIP-CORPUS-086.md new file mode 100644 index 00000000..2579c761 --- /dev/null +++ b/project/CHANGELOG.d/SCRIP-CORPUS-086.md @@ -0,0 +1,11 @@ +# SCRIP-CORPUS-086 + +Status: REVIEW_PENDING + +Prepare a source-free revision-identity freeze for the retained Bulgakov corpus candidate while preserving the existing two-source chapter partition. Canonical recovery branch: `scriptorium-corpus-086-source-identities`. Draft PR: #262. + +The branch contains source-free identity capture/replay, strict positive-integer provider identities, identity-only pinned replay, exact 20-title provider inventory validation, closed four-field provider identity schemas, explicit nine-field persisted page rows, a closed top-level manifest envelope, provenance drift checks, explicit composition/capture-scope validation, and deterministic offline regressions. + +The verified semantic/test head for this bounded unit is `ca4b2654318f74d7fdd5042e6713fab86cc123ba`; all 26 pull-request-triggered workflow runs on that head completed successfully, with no submitted reviews or review threads. Canonical bookkeeping is reconciled to that semantic/test head. No literary source prose or live 20-row provider manifest is committed. Literary-body identity, composite identity, single-edition equivalence, FantLab analyzer-input identity, diagnostics, M2 admission and parity remain closed. + +PR #262 remains intentionally Draft. Because this bookkeeping reconciliation advances the branch with documentation-only changes, a later independent wake must inspect fresh checks/reviews/threads on the resulting exact head before Ready/merge. A later bounded production run may perform authoritative 20-page identity-only capture and exact replay only after that judgement is clean. diff --git a/project/STATE_AND_QUEUE.md b/project/STATE_AND_QUEUE.md index fb87dfc0..ce55f5fc 100644 --- a/project/STATE_AND_QUEUE.md +++ b/project/STATE_AND_QUEUE.md @@ -1,22 +1,23 @@ # STATE_AND_QUEUE — Scriptorium -STATE_REVISION: 284 +STATE_REVISION: 286 PHASE: M1 — Deterministic FantLab surface -LAST_COMMITTED_RUN_AT: 2026-09-24T15:54:00Z -LAST_RESULT: SCRIP-CORPUS-085 / Issue #259 / PR #260 completed. A later exact-head judgement independently read all 7 changed files at `7a2a58370bcceb1572fa6064241fa24cd1d9976f` against unchanged authoring base `82e88eb2d9045b199d595fa96d9cab9260cf6328`, found no merge blockers or review threads, and directly re-inspected the exact-head transient facsimile artifact. Artifact `10814683683` ZIP SHA-256 independently recomputed to GitHub metadata `243e83888345fd2b9c7f791de14ebd431fcc623b8ea1825200a5402c4abd14eb`; `rendered-pages.sha256` recomputed to `33457b4afaeed309fa6c604bfb0214f4be85979289033ed272f53f978ce30518`, and all nine mid-body image digests matched the committed source-free JSON. Direct visual judgement confirmed 59→50, 61→52, 109→100, 110→101, 111→102, 159→150, 160→151 and 161→152, while carrier 60 has no visible folio and remains uninterpolated. All 23 PR-triggered workflows at the exact head completed successfully; review `5306813956` records the clean judgement. PR #260 was marked Ready and squash-merged with expected-head protection as `f60957eb2512d8e9ec9ed27e803878dfb1835dec`; Issue #259 closed completed. No global +9 mapping, complete carrier map, body admission or benchmark/parity movement is claimed. +LAST_COMMITTED_RUN_AT: 2026-09-29T01:49:18Z +LAST_RESULT: SCRIP-CORPUS-086 / Issue #261 / Draft PR #262 recovery bookkeeping reconciled to verified semantic/test head `ca4b2654318f74d7fdd5042e6713fab86cc123ba`, whose 26/26 pull-request-triggered workflows completed successfully with no submitted reviews or review threads. The reconciliation is documentation-only and remains REVIEW_PENDING until a later independent exact-head judgement of the resulting PR head. No live 20-row provider manifest, literary-body identity, FantLab analyzer-input identity, benchmark or M2 movement is claimed. LAST_VERIFIED_PROGRESS: M1 deterministic metric families remain executable inferred candidates. Public corpus/showcase evidence includes exact frozen/admitted works and source-free derived outputs, while no retained work has an independently established FantLab analyzer-input identity. Perelman now has independently reviewed exact PDF/DjVu identities, pagination, current Wikisource/FantLab surface boundaries, zero-PDF-PageLabels evidence, front-matter anchors, transition/tail anchors, restored-page interior observations, and canonical broad mid-body exact-PDF anchors that expose an unlabelled carrier page while keeping global mapping closed. Darwin/Rachinsky retains canonical reviewed dependency/replay/provider evidence through SCRIP-CORPUS-084. M2 stays 0/5. ## Current unit ```text -UNIT_ID: SCRIP-CORPUS-085 -ISSUE: #259 (closed completed) -STATUS: COMPLETE -PR: #260 (squash-merged as f60957eb2512d8e9ec9ed27e803878dfb1835dec) -NEXT_ACTION: Resume normal-flow corpus/provenance selection. - A separate bounded Perelman unit may strengthen complete carrier mapping, carrier equivalence or body selection. - Keep carrier page 60 uninterpolated and do not promote a global +9 rule, PDF↔DjVu equivalence, OCR/body identity, - selected-body >=300k, FantLab-input identity or M2/parity until separately evidenced. +UNIT_ID: SCRIP-CORPUS-086 +ISSUE: #261 (open) +STATUS: REVIEW_PENDING +PR: #262 (Draft) +AUTHORED_HEAD: ca4b2654318f74d7fdd5042e6713fab86cc123ba +NEXT_ACTION: Independently review PR #262 on the exact documentation-only head produced by this reconciliation. + Require fresh exact-head checks, no blocking reviews and no unresolved review threads before Ready/merge. + Only after a clean later judgement may a separate bounded unit perform authoritative + 20-page identity-only capture/replay of page ID, revision ID, UTC timestamp and MediaWiki SHA-1. ``` ## Current milestone gate diff --git a/project/run-receipts/SCRIP-CORPUS-086.md b/project/run-receipts/SCRIP-CORPUS-086.md new file mode 100644 index 00000000..ce1eac40 --- /dev/null +++ b/project/run-receipts/SCRIP-CORPUS-086.md @@ -0,0 +1,44 @@ +# Run receipt — SCRIP-CORPUS-086 + +Status: REVIEW_PENDING + +## Exact recovery state + +- Base: `cde1a4e577a2611a0ea2bf7b3930ca4382f5feef`. +- Canonical branch: `scriptorium-corpus-086-source-identities`. +- Issue: #261. +- Draft PR: #262. +- Authoring identity: `andy-zen-dev`. +- Verified semantic/test head for this bounded unit: `ca4b2654318f74d7fdd5042e6713fab86cc123ba`. +- Exact-head verification: 26/26 pull-request-triggered workflow runs completed successfully; no submitted reviews and no review threads were present. +- Unit: source-free exact revision identity freeze for the 20 retained Bulgakov chapter pages while preserving the 1–11 / 12–20 mixed-source partition. + +## Produced + +The branch now contains: + +- source-free current-revision identity capture requesting page ID, revision ID, UTC timestamp and MediaWiki SHA-1 only; +- strict positive-integer identity validation that rejects booleans; +- identity-only pinned replay with no literary content request and fail-closed page ID / revision ID / timestamp / SHA-1 checks; +- exact 20-title provider inventory validation; +- closed four-field provider identity mappings; +- explicit nine-field persisted page rows; +- a closed top-level manifest envelope with provider/source URL/legal-basis validation; +- exact composition and capture-scope checks; +- deterministic offline regressions covering the source partition, identity drift, source-free replay, open-schema rejection and SHA-1 canonicalization. + +## Verification boundary + +The authoring run re-read the post-write implementation and test blobs from GitHub and confirmed all intended schema/replay guards are present. Draft PR #262 was opened successfully. + +Exact-head verification was completed on `ca4b2654318f74d7fdd5042e6713fab86cc123ba`: all 26 pull-request-triggered workflow runs completed successfully, with no submitted reviews and no review threads. This recovery update changes bookkeeping only; the resulting documentation-only PR head still requires a later independent exact-head judgement before Ready/merge. + +No live 20-row provider capture is claimed in this receipt, and no literary prose is committed. + +## Evidence boundary + +Literary-body extraction, composite identity, single-edition equivalence, FantLab analyzer-input identity, diagnostics, benchmark movement, M2 admission and parity remain closed. M2 remains 0/5. + +## Exact reconciliation + +Recovery remains REVIEW_PENDING. Canonical bookkeeping is reconciled to verified semantic/test head `ca4b2654318f74d7fdd5042e6713fab86cc123ba` and its 26/26 successful PR-triggered workflows. Because this reconciliation itself advances the PR with documentation-only commits, a later independent wake must re-read PR #262 on its new exact head and inspect fresh checks/reviews/threads before any Ready/merge decision. Authoritative 20-page identity-only provider capture/replay is a separate bounded production unit and must not be conflated with this recovery/review handoff. diff --git a/scriptorium/white_guard_revisions.py b/scriptorium/white_guard_revisions.py new file mode 100644 index 00000000..cbd787fc --- /dev/null +++ b/scriptorium/white_guard_revisions.py @@ -0,0 +1,461 @@ +"""Freeze source-free revision identities for the retained Bulgakov corpus candidate. + +The retained transcription is explicitly mixed-source: chapters 1-11 and 12-20 +belong to distinct bibliographic source families. This module freezes page/revision +identity only; it does not extract or serialize literary prose. +""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timezone +import json +from pathlib import Path +import re +from typing import Callable, Iterable, Mapping, Sequence + +from .wikisource_freeze import _api_query + +CANDIDATE_ID = "bulgakov-white-guard-ru" +WORK_TITLE = "\u0411\u0435\u043b\u0430\u044f \u0433\u0432\u0430\u0440\u0434\u0438\u044f (\u0411\u0443\u043b\u0433\u0430\u043a\u043e\u0432)" +WORK_URL = "https://ru.wikisource.org/wiki/%D0%91%D0%B5%D0%BB%D0%B0%D1%8F_%D0%B3%D0%B2%D0%B0%D1%80%D0%B4%D0%B8%D1%8F_(%D0%91%D1%83%D0%BB%D0%B3%D0%B0%D0%BA%D0%BE%D0%B2)" +WORK_INDEX_REVISION_ID = 4715350 +MANIFEST_VERSION = "scriptorium-bulgakov-source-revisions-v1" +SOURCE_FAMILY_1927 = "paris-concorde-1927" +SOURCE_FAMILY_1989 = "moscow-pravda-1989" +SOURCE_1927 = "\u041c. \u0411\u0443\u043b\u0433\u0430\u043a\u043e\u0432. \u0414\u043d\u0438 \u0422\u0443\u0440\u0431\u0438\u043d\u044b\u0445 (\u0411\u0435\u043b\u0430\u044f \u0433\u0432\u0430\u0440\u0434\u0438\u044f). \u2014 \u041f\u0430\u0440\u0438\u0436: Concorde, 1927." +SOURCE_1989 = "\u0411\u0443\u043b\u0433\u0430\u043a\u043e\u0432 \u041c. \u0410. \u0411\u0435\u043b\u0430\u044f \u0433\u0432\u0430\u0440\u0434\u0438\u044f. \u0416\u0438\u0437\u043d\u044c \u0433\u043e\u0441\u043f\u043e\u0434\u0438\u043d\u0430 \u0434\u0435 \u041c\u043e\u043b\u044c\u0435\u0440\u0430. \u0420\u0430\u0441\u0441\u043a\u0430\u0437\u044b. \u2014 \u041c.: \u041f\u0440\u0430\u0432\u0434\u0430, 1989." +_HEX40_RE = re.compile(r"^[0-9a-f]{40}$") +_PROVIDER = "Russian Wikisource" +_LEGAL_BASIS = "public_domain_original_russian_work_with_wikisource_reuse_terms" +_PROVIDER_IDENTITY_KEYS = frozenset({"page_id", "revision_id", "revision_timestamp", "mediawiki_sha1"}) +_PAGE_ROW_KEYS = frozenset({ + "ordinal", + "chapter", + "title", + "source_family_id", + "bibliographic_source", + "page_id", + "revision_id", + "revision_timestamp", + "mediawiki_sha1", +}) +_MANIFEST_KEYS = frozenset({ + "manifest_version", + "candidate_id", + "provider", + "source_work_url", + "source_work_index_revision_id", + "legal_basis", + "source_partition", + "composition_contract", + "capture_scope", + "pages", +}) + + +def expected_pages() -> tuple[dict[str, object], ...]: + rows = [] + for chapter in range(1, 21): + rows.append( + { + "ordinal": chapter, + "chapter": chapter, + "title": f"{WORK_TITLE}/\u0413\u043b\u0430\u0432\u0430 {chapter}", + "source_family_id": SOURCE_FAMILY_1927 if chapter <= 11 else SOURCE_FAMILY_1989, + "bibliographic_source": SOURCE_1927 if chapter <= 11 else SOURCE_1989, + } + ) + return tuple(rows) + + +def _validate_timestamp(value: object) -> str: + if not isinstance(value, str) or not value.endswith("Z"): + raise ValueError(f"invalid UTC revision timestamp {value!r}") + parsed = datetime.fromisoformat(value[:-1] + "+00:00") + if parsed.tzinfo != timezone.utc or parsed.microsecond: + raise ValueError(f"invalid UTC revision timestamp {value!r}") + return value + + +def _is_positive_integer(value: object) -> bool: + return type(value) is int and value > 0 + + +def fetch_page_revisions( + pages: Iterable[dict[str, object]], + *, + query: Callable[[dict[str, str]], dict[str, object]] = _api_query, +) -> dict[str, dict[str, object]]: + expected = tuple(pages) + titles = [str(page["title"]) for page in expected] + payload = query({ + "action": "query", + "prop": "info|revisions", + "titles": "|".join(titles), + "rvprop": "ids|timestamp|sha1", + }) + query_obj = payload.get("query") + pages_obj = query_obj.get("pages") if isinstance(query_obj, dict) else None + if not isinstance(pages_obj, list): + raise ValueError("MediaWiki response missing pages") + records: dict[str, dict[str, object]] = {} + for page in pages_obj: + if not isinstance(page, dict): + raise ValueError("unexpected page object") + title = page.get("title") + if not isinstance(title, str) or title not in titles: + raise ValueError(f"unexpected literary title from MediaWiki: {title!r}") + if title in records: + raise ValueError(f"duplicate literary page response: {title}") + if page.get("missing") is True: + raise ValueError(f"expected literary page is missing: {title}") + if page.get("redirect") is True: + raise ValueError(f"expected literary page became a redirect: {title}") + page_id = page.get("pageid") + revisions = page.get("revisions") + if not _is_positive_integer(page_id): + raise ValueError(f"invalid page id for {title!r}") + if not isinstance(revisions, list) or len(revisions) != 1 or not isinstance(revisions[0], dict): + raise ValueError(f"expected one current revision for {title!r}") + revision = revisions[0] + revision_id = revision.get("revid") + timestamp = _validate_timestamp(revision.get("timestamp")) + mediawiki_sha1 = revision.get("sha1") + if not _is_positive_integer(revision_id): + raise ValueError(f"invalid revision id for {title!r}") + if not isinstance(mediawiki_sha1, str) or not _HEX40_RE.fullmatch(mediawiki_sha1.lower()): + raise ValueError(f"invalid MediaWiki SHA-1 for {title!r}") + records[title] = { + "page_id": page_id, + "revision_id": revision_id, + "revision_timestamp": timestamp, + "mediawiki_sha1": mediawiki_sha1.lower(), + } + if set(records) != set(titles): + raise ValueError("literary page inventory mismatch") + return records + + +def fetch_pinned_revision_identities( + identities: Iterable[Mapping[str, object]], + *, + query: Callable[[dict[str, str]], dict[str, object]] = _api_query, +) -> dict[str, dict[str, object]]: + """Replay frozen revision identities without requesting literary source content.""" + + expected = tuple(identities) + by_revision_id: dict[int, Mapping[str, object]] = {} + for row in expected: + revision_id = row.get("revision_id") + page_id = row.get("page_id") + if not _is_positive_integer(revision_id): + raise ValueError("invalid expected revision id") + if not _is_positive_integer(page_id): + raise ValueError(f"invalid expected page id for revision {revision_id}") + if revision_id in by_revision_id: + raise ValueError("revision manifest contains duplicate revision ids") + by_revision_id[revision_id] = row + + records: dict[str, dict[str, object]] = {} + for offset in range(0, len(expected), 40): + batch = expected[offset : offset + 40] + requested_ids = [row["revision_id"] for row in batch] + payload = query( + { + "action": "query", + "prop": "revisions", + "revids": "|".join(str(value) for value in requested_ids), + "rvprop": "ids|timestamp|sha1", + } + ) + query_obj = payload.get("query") + pages_obj = query_obj.get("pages") if isinstance(query_obj, dict) else None + if not isinstance(pages_obj, list): + raise ValueError("MediaWiki response missing pages") + + seen_batch: set[int] = set() + for page in pages_obj: + if not isinstance(page, dict): + raise ValueError("unexpected page object") + title = page.get("title") + page_id = page.get("pageid") + revisions = page.get("revisions") + if not isinstance(title, str): + raise ValueError("MediaWiki revision response missing page title") + if not _is_positive_integer(page_id): + raise ValueError(f"invalid page id from MediaWiki for {title!r}") + if not isinstance(revisions, list) or len(revisions) != 1 or not isinstance(revisions[0], dict): + raise ValueError(f"expected one pinned revision for {title!r}") + revision = revisions[0] + revision_id = revision.get("revid") + if not _is_positive_integer(revision_id) or revision_id not in by_revision_id: + raise ValueError(f"unexpected revision id from MediaWiki: {revision_id!r}") + if revision_id not in requested_ids: + raise ValueError(f"revision {revision_id} returned outside requested batch") + if revision_id in seen_batch: + raise ValueError(f"duplicate revision response: {revision_id}") + seen_batch.add(revision_id) + + identity = by_revision_id[revision_id] + expected_title = str(identity["title"]) + if page_id != identity["page_id"]: + raise ValueError(f"revision {revision_id} page ID drift") + if title != expected_title: + raise ValueError( + f"revision {revision_id} title drift: {title!r}, expected {expected_title!r}" + ) + if revision.get("timestamp") != identity["revision_timestamp"]: + raise ValueError(f"revision {revision_id} timestamp drift") + if revision.get("sha1") != identity["mediawiki_sha1"]: + raise ValueError(f"revision {revision_id} MediaWiki SHA-1 drift") + if expected_title in records: + raise ValueError(f"duplicate literary page response: {expected_title}") + records[expected_title] = { + "page_id": page_id, + "revision_id": revision_id, + "revision_timestamp": revision["timestamp"], + "mediawiki_sha1": revision["sha1"], + } + + if seen_batch != set(requested_ids): + missing = sorted(set(requested_ids) - seen_batch) + raise ValueError(f"MediaWiki omitted pinned revisions: {missing!r}") + + expected_titles = {str(row["title"]) for row in expected} + if set(records) != expected_titles: + missing = sorted(expected_titles - set(records)) + extra = sorted(set(records) - expected_titles) + raise ValueError( + f"pinned literary page identity inventory mismatch; missing={missing!r} extra={extra!r}" + ) + return records + + +def _validated_provider_identity(identity: object, *, title: str) -> dict[str, object]: + if not isinstance(identity, Mapping): + raise ValueError(f"provider identity for {title!r} must be an object") + if set(identity) != _PROVIDER_IDENTITY_KEYS: + missing = sorted(_PROVIDER_IDENTITY_KEYS - set(identity)) + extra = sorted(set(identity) - _PROVIDER_IDENTITY_KEYS) + raise ValueError( + f"provider identity schema drift for {title!r}; missing={missing!r} extra={extra!r}" + ) + page_id = identity["page_id"] + revision_id = identity["revision_id"] + if not _is_positive_integer(page_id): + raise ValueError("invalid page id") + if not _is_positive_integer(revision_id): + raise ValueError("invalid revision id") + timestamp = _validate_timestamp(identity["revision_timestamp"]) + mediawiki_sha1 = identity["mediawiki_sha1"] + if not isinstance(mediawiki_sha1, str) or not _HEX40_RE.fullmatch(mediawiki_sha1.lower()): + raise ValueError("invalid MediaWiki SHA-1") + return { + "page_id": page_id, + "revision_id": revision_id, + "revision_timestamp": timestamp, + "mediawiki_sha1": mediawiki_sha1.lower(), + } + + +def build_manifest( + *, + fetcher: Callable[[Iterable[dict[str, object]]], dict[str, dict[str, object]]] = fetch_page_revisions, +) -> dict[str, object]: + pages = expected_pages() + observed = fetcher(pages) + expected_titles = {str(page["title"]) for page in pages} + if set(observed) != expected_titles: + missing = sorted(expected_titles - set(observed)) + extra = sorted(set(observed) - expected_titles) + raise ValueError(f"provider literary page inventory mismatch; missing={missing!r} extra={extra!r}") + + rows = [] + revision_ids: set[int] = set() + page_ids: set[int] = set() + for page in pages: + title = str(page["title"]) + identity = _validated_provider_identity(observed[title], title=title) + revision_id = identity["revision_id"] + page_id = identity["page_id"] + if revision_id in revision_ids: + raise ValueError(f"duplicate revision id {revision_id}") + if page_id in page_ids: + raise ValueError(f"duplicate page id {page_id}") + revision_ids.add(revision_id) + page_ids.add(page_id) + rows.append( + { + "ordinal": page["ordinal"], + "chapter": page["chapter"], + "title": page["title"], + "source_family_id": page["source_family_id"], + "bibliographic_source": page["bibliographic_source"], + "page_id": identity["page_id"], + "revision_id": identity["revision_id"], + "revision_timestamp": identity["revision_timestamp"], + "mediawiki_sha1": identity["mediawiki_sha1"], + } + ) + return { + "manifest_version": MANIFEST_VERSION, + "candidate_id": CANDIDATE_ID, + "provider": _PROVIDER, + "source_work_url": WORK_URL, + "source_work_index_revision_id": WORK_INDEX_REVISION_ID, + "legal_basis": _LEGAL_BASIS, + "source_partition": [ + {"source_family_id": SOURCE_FAMILY_1927, "chapter_range": "1-11", "bibliographic_source": SOURCE_1927}, + {"source_family_id": SOURCE_FAMILY_1989, "chapter_range": "12-20", "bibliographic_source": SOURCE_1989}, + ], + "composition_contract": { + "literary_page_count": 20, + "order": "chapter-number order 1 through 20", + "single_edition_identity": False, + }, + "capture_scope": { + "literary_page_revisions_frozen": True, + "source_partition_frozen": True, + "literary_body_extraction_frozen": False, + "composite_identity_frozen": False, + "single_edition_identity": False, + "source_text_committed": False, + "fantlab_source_edition_match": "unknown", + }, + "pages": rows, + } + + +def validate_manifest(manifest: Mapping[str, object]) -> tuple[dict[str, object], ...]: + expected_partition = [ + {"source_family_id": SOURCE_FAMILY_1927, "chapter_range": "1-11", "bibliographic_source": SOURCE_1927}, + {"source_family_id": SOURCE_FAMILY_1989, "chapter_range": "12-20", "bibliographic_source": SOURCE_1989}, + ] + expected_composition = { + "literary_page_count": 20, + "order": "chapter-number order 1 through 20", + "single_edition_identity": False, + } + expected_scope = { + "literary_page_revisions_frozen": True, + "source_partition_frozen": True, + "literary_body_extraction_frozen": False, + "composite_identity_frozen": False, + "single_edition_identity": False, + "source_text_committed": False, + "fantlab_source_edition_match": "unknown", + } + if set(manifest) != _MANIFEST_KEYS: + missing = sorted(_MANIFEST_KEYS - set(manifest)) + extra = sorted(set(manifest) - _MANIFEST_KEYS) + raise ValueError(f"manifest schema drift; missing={missing!r} extra={extra!r}") + if manifest.get("manifest_version") != MANIFEST_VERSION: + raise ValueError("unsupported Bulgakov revision-manifest version") + if manifest.get("candidate_id") != CANDIDATE_ID: + raise ValueError("unexpected candidate id") + if manifest.get("provider") != _PROVIDER: + raise ValueError("provider drift") + if manifest.get("source_work_url") != WORK_URL: + raise ValueError("source work URL drift") + if manifest.get("source_work_index_revision_id") != WORK_INDEX_REVISION_ID: + raise ValueError("work-index revision drift") + if manifest.get("legal_basis") != _LEGAL_BASIS: + raise ValueError("legal basis drift") + if manifest.get("source_partition") != expected_partition: + raise ValueError("source partition drift") + if manifest.get("composition_contract") != expected_composition: + raise ValueError("composition contract drift") + if manifest.get("capture_scope") != expected_scope: + raise ValueError("capture scope drift") + rows = manifest.get("pages") + expected = expected_pages() + if not isinstance(rows, list) or len(rows) != 20: + raise ValueError("revision manifest must contain exactly 20 literary pages") + revision_ids: set[int] = set() + page_ids: set[int] = set() + identities = [] + for expected_page, row in zip(expected, rows, strict=True): + if not isinstance(row, dict): + raise ValueError("revision manifest rows must be objects") + if set(row) != _PAGE_ROW_KEYS: + missing = sorted(_PAGE_ROW_KEYS - set(row)) + extra = sorted(set(row) - _PAGE_ROW_KEYS) + raise ValueError( + f"revision manifest row schema drift at chapter {expected_page['chapter']}; " + f"missing={missing!r} extra={extra!r}" + ) + for key in ("ordinal", "chapter", "title", "source_family_id", "bibliographic_source"): + if row.get(key) != expected_page[key]: + raise ValueError(f"literary page/source-partition drift at chapter {expected_page['chapter']}") + page_id = row.get("page_id") + revision_id = row.get("revision_id") + if not _is_positive_integer(page_id) or page_id in page_ids: + raise ValueError("invalid/duplicate page id") + if not _is_positive_integer(revision_id) or revision_id in revision_ids: + raise ValueError("invalid/duplicate revision id") + page_ids.add(page_id) + revision_ids.add(revision_id) + timestamp = _validate_timestamp(row.get("revision_timestamp")) + mediawiki_sha1 = row.get("mediawiki_sha1") + if not isinstance(mediawiki_sha1, str) or not _HEX40_RE.fullmatch(mediawiki_sha1): + raise ValueError("invalid MediaWiki SHA-1") + identities.append({ + "ordinal": row["ordinal"], + "title": row["title"], + "page_id": page_id, + "revision_id": revision_id, + "revision_timestamp": timestamp, + "mediawiki_sha1": mediawiki_sha1, + }) + return tuple(identities) + + +def replay_manifest( + manifest: Mapping[str, object], + *, + fetcher: Callable[[Iterable[Mapping[str, object]]], Mapping[str, object]] = fetch_pinned_revision_identities, +) -> dict[str, object]: + identities = validate_manifest(manifest) + fetched = fetcher(identities) + if set(fetched) != {str(row["title"]) for row in identities}: + raise ValueError("pinned literary page replay inventory mismatch") + return { + "verified": True, + "candidate_id": CANDIDATE_ID, + "verified_literary_page_count": 20, + "source_partition_frozen": True, + "literary_body_extraction_frozen": False, + "composite_identity_frozen": False, + "single_edition_identity": False, + "source_text_committed": False, + "fantlab_source_edition_match": "unknown", + } + + +def canonical_json_text(value: Mapping[str, object]) -> str: + return json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n" + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Capture/replay Bulgakov candidate revision identities.") + subparsers = parser.add_subparsers(dest="command", required=True) + capture = subparsers.add_parser("capture") + capture.add_argument("--output", type=Path, required=True) + replay = subparsers.add_parser("replay") + replay.add_argument("--manifest", type=Path, required=True) + args = parser.parse_args(argv) + if args.command == "capture": + manifest = build_manifest() + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(canonical_json_text(manifest), encoding="utf-8") + return 0 + manifest = json.loads(args.manifest.read_text(encoding="utf-8")) + if not isinstance(manifest, dict): + raise ValueError("revision manifest must be a JSON object") + replay_manifest(manifest) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scriptorium/wikisource_replay.py b/scriptorium/wikisource_replay.py index 13ac9d19..b5a49f57 100644 --- a/scriptorium/wikisource_replay.py +++ b/scriptorium/wikisource_replay.py @@ -67,6 +67,15 @@ def fetch_pinned_chapter_revisions( identity = by_revision_id[revision_id] expected_title = str(identity["title"]) + if "page_id" in identity: + expected_page_id = identity["page_id"] + if type(expected_page_id) is not int or expected_page_id <= 0: + raise ValueError(f"invalid expected page id for revision {revision_id}") + observed_page_id = page.get("pageid") + if type(observed_page_id) is not int or observed_page_id <= 0: + raise ValueError(f"invalid observed page id for revision {revision_id}") + if observed_page_id != expected_page_id: + raise ValueError(f"revision {revision_id} page ID drift") if title != expected_title: raise ValueError( f"revision {revision_id} title drift: {title!r}, expected {expected_title!r}" diff --git a/tests/test_bulgakov_revisions.py b/tests/test_bulgakov_revisions.py new file mode 100644 index 00000000..80e5127f --- /dev/null +++ b/tests/test_bulgakov_revisions.py @@ -0,0 +1,247 @@ +from __future__ import annotations + +import copy +import unittest + +from scriptorium.white_guard_revisions import ( + SOURCE_FAMILY_1927, + SOURCE_FAMILY_1989, + build_manifest, + expected_pages, + fetch_pinned_revision_identities, + replay_manifest, + validate_manifest, +) + + +def _fake_identity_fetcher(pages): + return { + str(page["title"]): { + "page_id": 1000 + int(page["chapter"]), + "revision_id": 2000 + int(page["chapter"]), + "revision_timestamp": "2026-09-24T00:00:00Z", + "mediawiki_sha1": f"{int(page['chapter']):040x}", + } + for page in pages + } + + +class BulgakovRevisionTests(unittest.TestCase): + def test_expected_inventory_and_source_partition(self): + pages = expected_pages() + + self.assertEqual(20, len(pages)) + self.assertEqual(list(range(1, 21)), [page["chapter"] for page in pages]) + self.assertEqual(list(range(1, 21)), [page["ordinal"] for page in pages]) + self.assertEqual( + [SOURCE_FAMILY_1927] * 11 + [SOURCE_FAMILY_1989] * 9, + [page["source_family_id"] for page in pages], + ) + self.assertTrue(all(str(page["title"]).endswith(f"Глава {page['chapter']}") for page in pages)) + + def test_build_manifest_is_source_free_and_fail_closed_by_default(self): + manifest = build_manifest(fetcher=_fake_identity_fetcher) + + self.assertEqual(20, len(manifest["pages"])) + self.assertEqual( + { + "literary_page_count": 20, + "order": "chapter-number order 1 through 20", + "single_edition_identity": False, + }, + manifest["composition_contract"], + ) + self.assertEqual( + { + "literary_page_revisions_frozen": True, + "source_partition_frozen": True, + "literary_body_extraction_frozen": False, + "composite_identity_frozen": False, + "single_edition_identity": False, + "source_text_committed": False, + "fantlab_source_edition_match": "unknown", + }, + manifest["capture_scope"], + ) + self.assertFalse(any("text" in key or "prose" in key for row in manifest["pages"] for key in row)) + + def test_build_manifest_rejects_provider_inventory_and_identity_schema_drift(self): + def extra_title_fetcher(pages): + records = _fake_identity_fetcher(pages) + records["unexpected provider title"] = { + "page_id": 9999, + "revision_id": 9999, + "revision_timestamp": "2026-09-24T00:00:00Z", + "mediawiki_sha1": "f" * 40, + } + return records + + with self.assertRaisesRegex(ValueError, "provider literary page inventory mismatch"): + build_manifest(fetcher=extra_title_fetcher) + + def extra_field_fetcher(pages): + records = _fake_identity_fetcher(pages) + first = next(iter(records.values())) + first["source_text"] = "must never persist" + return records + + with self.assertRaisesRegex(ValueError, "provider identity schema drift"): + build_manifest(fetcher=extra_field_fetcher) + + def test_validate_manifest_rejects_open_schema_and_provenance_drift(self): + baseline = build_manifest(fetcher=_fake_identity_fetcher) + + manifest = copy.deepcopy(baseline) + manifest["source_text"] = "must never persist" + with self.assertRaisesRegex(ValueError, "manifest schema drift"): + validate_manifest(manifest) + + manifest = copy.deepcopy(baseline) + manifest["pages"][0]["source_text"] = "must never persist" + with self.assertRaisesRegex(ValueError, "row schema drift"): + validate_manifest(manifest) + + for field, value, message in ( + ("provider", "Other provider", "provider drift"), + ("source_work_url", "https://example.invalid/work", "source work URL drift"), + ("legal_basis", "unknown", "legal basis drift"), + ): + with self.subTest(field=field): + manifest = copy.deepcopy(baseline) + manifest[field] = value + with self.assertRaisesRegex(ValueError, message): + validate_manifest(manifest) + + def test_build_manifest_canonicalizes_provider_sha1(self): + def uppercase_sha1_fetcher(pages): + records = _fake_identity_fetcher(pages) + for identity in records.values(): + identity["mediawiki_sha1"] = "A" * 40 + return records + + manifest = build_manifest(fetcher=uppercase_sha1_fetcher) + + self.assertTrue(all(row["mediawiki_sha1"] == "a" * 40 for row in manifest["pages"])) + + def test_validate_manifest_preserves_order_and_identity(self): + manifest = build_manifest(fetcher=_fake_identity_fetcher) + identities = validate_manifest(manifest) + + self.assertEqual(20, len(identities)) + self.assertEqual(list(range(1, 21)), [identity["ordinal"] for identity in identities]) + self.assertEqual(2001, identities[0]["revision_id"]) + self.assertEqual(2020, identities[-1]["revision_id"]) + + def test_validate_manifest_rejects_partition_drift(self): + manifest = build_manifest(fetcher=_fake_identity_fetcher) + manifest["pages"][11]["source_family_id"] = SOURCE_FAMILY_1927 + + with self.assertRaisesRegex(ValueError, "source-partition drift"): + validate_manifest(manifest) + + def test_validate_manifest_rejects_boolean_provider_identity(self): + for field in ("page_id", "revision_id"): + with self.subTest(field=field): + manifest = build_manifest(fetcher=_fake_identity_fetcher) + manifest["pages"][0][field] = True + + with self.assertRaises(ValueError): + validate_manifest(manifest) + + def test_validate_manifest_rejects_duplicate_provider_identity(self): + manifest = build_manifest(fetcher=_fake_identity_fetcher) + manifest["pages"][1]["page_id"] = manifest["pages"][0]["page_id"] + + with self.assertRaisesRegex(ValueError, "invalid/duplicate page id"): + validate_manifest(manifest) + + def test_validate_manifest_rejects_composition_contract_drift(self): + manifest = build_manifest(fetcher=_fake_identity_fetcher) + manifest["composition_contract"] = copy.deepcopy(manifest["composition_contract"]) + manifest["composition_contract"]["single_edition_identity"] = True + + with self.assertRaisesRegex(ValueError, "composition contract drift"): + validate_manifest(manifest) + + def test_validate_manifest_rejects_promotion_scope_drift(self): + manifest = build_manifest(fetcher=_fake_identity_fetcher) + manifest["capture_scope"] = copy.deepcopy(manifest["capture_scope"]) + manifest["capture_scope"]["literary_body_extraction_frozen"] = True + + with self.assertRaisesRegex(ValueError, "capture scope drift"): + validate_manifest(manifest) + + def test_identity_replay_does_not_request_source_content(self): + identities = validate_manifest(build_manifest(fetcher=_fake_identity_fetcher)) + calls = [] + + def query(params): + calls.append(params) + requested = {int(value) for value in params["revids"].split("|")} + pages = [] + for identity in identities: + if identity["revision_id"] not in requested: + continue + pages.append( + { + "pageid": identity["page_id"], + "title": identity["title"], + "revisions": [ + { + "revid": identity["revision_id"], + "timestamp": identity["revision_timestamp"], + "sha1": identity["mediawiki_sha1"], + } + ], + } + ) + return {"query": {"pages": pages}} + + replayed = fetch_pinned_revision_identities(identities, query=query) + + self.assertEqual({identity["title"] for identity in identities}, set(replayed)) + self.assertEqual("ids|timestamp|sha1", calls[0]["rvprop"]) + self.assertNotIn("redirects", calls[0]) + self.assertNotIn("rvslots", calls[0]) + self.assertNotIn("content", calls[0]["rvprop"]) + + def bad_query(params): + payload = query(params) + payload["query"]["pages"][0]["pageid"] += 1 + return payload + + with self.assertRaisesRegex(ValueError, "page ID drift"): + fetch_pinned_revision_identities(identities, query=bad_query) + + def test_replay_manifest_keeps_body_and_parity_gates_closed(self): + manifest = build_manifest(fetcher=_fake_identity_fetcher) + + replay_identities = [] + + def replay_fetcher(identities): + rows = tuple(identities) + replay_identities.extend(rows) + return {str(identity["title"]): "transient provider payload" for identity in rows} + + receipt = replay_manifest(manifest, fetcher=replay_fetcher) + + self.assertEqual(1001, replay_identities[0]["page_id"]) + self.assertEqual(1020, replay_identities[-1]["page_id"]) + self.assertEqual( + { + "verified": True, + "candidate_id": "bulgakov-white-guard-ru", + "verified_literary_page_count": 20, + "source_partition_frozen": True, + "literary_body_extraction_frozen": False, + "composite_identity_frozen": False, + "single_edition_identity": False, + "source_text_committed": False, + "fantlab_source_edition_match": "unknown", + }, + receipt, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_wikisource_replay.py b/tests/test_wikisource_replay.py index f9b6dc38..adf70652 100644 --- a/tests/test_wikisource_replay.py +++ b/tests/test_wikisource_replay.py @@ -27,12 +27,14 @@ def test_fetch_pinned_revisions_requires_exact_identity(self): identities = ( { "title": "A", + "page_id": 101, "revision_id": 11, "revision_timestamp": "2024-01-01T00:00:00Z", "mediawiki_sha1": "a" * 40, }, { "title": "B", + "page_id": 102, "revision_id": 12, "revision_timestamp": "2024-01-01T00:00:01Z", "mediawiki_sha1": "b" * 40, @@ -46,6 +48,7 @@ def query(params): "query": { "pages": [ { + "pageid": 102, "title": "B", "revisions": [ { @@ -57,6 +60,7 @@ def query(params): ], }, { + "pageid": 101, "title": "A", "revisions": [ { @@ -84,6 +88,43 @@ def bad_query(params): with self.assertRaisesRegex(ValueError, "SHA-1 drift"): fetch_pinned_chapter_revisions(identities, query=bad_query) + def bad_page_query(params): + payload = query(params) + payload["query"]["pages"][0]["pageid"] = 999 + return payload + + with self.assertRaisesRegex(ValueError, "page ID drift"): + fetch_pinned_chapter_revisions(identities, query=bad_page_query) + + boolean_page_identities = ( + { + **identities[0], + "page_id": True, + }, + identities[1], + ) + with self.assertRaisesRegex(ValueError, "invalid expected page id"): + fetch_pinned_chapter_revisions(boolean_page_identities, query=query) + + provider_boolean_identities = ( + { + **identities[0], + "page_id": 1, + }, + identities[1], + ) + + def boolean_provider_query(params): + payload = query(params) + payload["query"]["pages"][1]["pageid"] = True + return payload + + with self.assertRaisesRegex(ValueError, "invalid observed page id"): + fetch_pinned_chapter_revisions( + provider_boolean_identities, + query=boolean_provider_query, + ) + def test_replay_preserves_manifest_order_and_verifies_composite_identity(self): manifest = json.loads(REVISION_MANIFEST.read_text(encoding="utf-8")) identities = decode_chapter_identities(manifest)