Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions .github/workflows/darwin-render-profile.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Darwin/Rachinsky 1864 render profile

on:
pull_request:
paths:
- ".github/workflows/darwin-render-profile.yml"
- "scriptorium/darwin_render_profile.py"
- "tests/test_darwin_render_profile.py"
- "tests/test_darwin_provenance_trace.py"
- "corpus/candidates/darwin-origin-species-rachinsky-1864-ru.md"
- "corpus/candidates/source-edition-traces/darwin-origin-species-rachinsky-1864-ru.json"
- "corpus/candidates/source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-surface.source-graph.json"
- "corpus/candidates/source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-surface.json"
- "corpus/candidates/source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-profile.json"
workflow_dispatch:

permissions:
contents: read

jobs:
verify:
name: Verify source-free fail-closed profile
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout pull-request head
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
persist-credentials: false

- name: Set up Python 3.13
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.13"

- name: Run profile and provenance tests
run: |
python -m unittest tests.test_darwin_render_profile -v
python -m unittest tests.test_darwin_provenance_trace -v

- name: Deterministically rebuild and compare profile
run: |
set -euo pipefail
mkdir -p build/repro
python -m scriptorium.darwin_render_profile \
--surface corpus/candidates/source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-surface.json \
--output build/repro/darwin-origin-species-rachinsky-1864-ru.render-profile.json
python - <<'PY'
import json
from pathlib import Path
expected = json.loads(Path("corpus/candidates/source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-profile.json").read_text(encoding="utf-8"))
rebuilt = json.loads(Path("build/repro/darwin-origin-species-rachinsky-1864-ru.render-profile.json").read_text(encoding="utf-8"))
assert rebuilt == expected
assert rebuilt["profile_sha256"] == "2ebcfea51282505c28824f54f6a1795bd7e87e2f958ac59f5c843e68c2066f3d"
assert rebuilt["rendering_profile_frozen"] is True
assert rebuilt["renderer_semantics_complete"] is False
assert rebuilt["renderer_implementation_ready"] is False
assert rebuilt["coverage"]["unresolved_template_shape_count"] == 38
assert rebuilt["coverage"]["unresolved_template_invocation_count"] == 4583
assert rebuilt["coverage"]["unresolved_tag_shape_count"] == 5
assert rebuilt["coverage"]["unresolved_tag_token_count"] == 518
assert rebuilt["inter_page_composition_frozen"] is False
assert rebuilt["literary_body_count_and_digests_frozen"] is False
assert rebuilt["minimum_300k_proved"] is False
assert rebuilt["admitted_for_calibration"] is False
assert rebuilt["fantlab_source_edition_match"] == "unknown"
assert rebuilt["diagnostic_ready"] is False
assert rebuilt["m2_parity_admissible"] is False
print("SCRIPTORIUM_DARWIN_RENDER_PROFILE_SHA256=" + rebuilt["profile_sha256"])
PY

- name: Upload source-free profile evidence
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: darwin-rachinsky-1864-render-profile
path: build/repro/darwin-origin-species-rachinsky-1864-ru.render-profile.json
if-no-files-found: error
retention-days: 7
21 changes: 15 additions & 6 deletions corpus/candidates/darwin-origin-species-rachinsky-1864-ru.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ The boundary is evidence-based rather than inferred from nominal page count. The

The implementation fails closed if the 418-Page identity inventory, identity-row shape, topology partition or generated contract drifts. It may later permit **transient exact-revision MediaWiki main-slot fetches only after identity verification**, but it does not serialize Page wikitext or rendered prose.

This is deliberately a composition/input freeze, **not yet a literary-body identity freeze**. The Page-wikitext-to-prose rendering profile and composite separator semantics remain unfrozen, so no body count or body digest is claimed by this unit.
This is deliberately a composition/input freeze, **not yet a literary-body identity freeze**. The candidate-specific rendering decision profile is now frozen separately below, but unresolved provider/template/reference/math semantics and composite separator semantics still prevent a rendered literary-body identity.

## Source-free rendering-surface inventory

Expand All @@ -79,7 +79,13 @@ The full hosted audit covered all **388/388** literary dependencies with `identi

Independent exact-head review re-read all settled CI on `ee97e29af7225131002b6e0922c731536d903d13`, independently downloaded the exact-head artifact, recomputed its audit/receipt/freeze digests, and found no blocker before squash merge `1d9a7795724a68747ea1bb6e1fe1e7ea8b590519`.

This **does not freeze the renderer**. It freezes only the set of markup constructs the future candidate-specific renderer must handle or explicitly reject. `rendering_profile_frozen=false` remains mandatory until a separate implementation proves deterministic semantics for this exact inventory.
## Fail-closed rendering decision profile

**SCRIP-CORPUS-045 / Draft PR #178** adds `scriptorium-darwin-page-render-profile-v1`, bound cryptographically to the reviewed exact-388 source-free surface freeze before any shape classification is accepted. The profile covers all **18 tag shapes / 5,164 tag tokens** and all **38 template shapes / 4,583 template invocations**. Its canonical profile SHA-256 is **`2ebcfea51282505c28824f54f6a1795bd7e87e2f958ac59f5c843e68c2066f3d`**.

The decision table defines only conservative local markup handling: ordinary `big`, `div`, `small`, `sub` and `sup` wrappers preserve inner text; `<noinclude>` is excluded from transcluded content; ProofreadPage `pagequality` metadata is dropped. It intentionally leaves **5 tag shapes / 518 tokens** unresolved for provider semantics: `<math>` open/close, `<ref>` open/close and self-closing `<references/>`. All **38 template shapes / 4,583 invocations** remain unresolved rather than inferring expansion from names; `nop` is additionally marked as inter-page-sensitive.

This is a **frozen decision boundary, not a renderer**. `rendering_profile_frozen=true` coexists deliberately with `renderer_semantics_complete=false`, `renderer_implementation_ready=false`, `rendering_equivalence_claimed=false`, and `inter_page_composition_frozen=false`. Any changed count, new/missing shape, stale surface digest or wrong candidate identity fails closed. No Page wikitext, template argument values, rendered prose, OCR, DjVu/PDF bytes or literary text are committed.

## Independent scan binary identity

Expand All @@ -99,11 +105,12 @@ The retained Wikisource work surface explicitly states that the work is in the p

## Corpus boundary

This candidate is still **not admitted** to the >=300,000-character calibration/profile corpus. The source family, route graph, 418-dependency topology, exact 418 Page revision identities, the 388/30 literary-versus-apparatus composition boundary, the exact 388-Page source-free markup-surface inventory, and the backing DjVu byte identity are now frozen, but the actual literary prose has not yet been deterministically rendered or counted.
This candidate is still **not admitted** to the >=300,000-character calibration/profile corpus. The source family, route graph, 418-dependency topology, exact 418 Page revision identities, the 388/30 literary-versus-apparatus composition boundary, the exact 388-Page source-free markup-surface inventory, fail-closed render decision profile, and backing DjVu byte identity are now frozen, but the actual literary prose has not yet been deterministically rendered or counted.

Still required before corpus admission:

- define and verify a candidate-specific fail-closed Page-wikitext-to-prose rendering profile that covers the frozen construct inventory;
- independently resolve/freeze the provider/template/reference/math semantics explicitly left unresolved by the render profile and separately freeze inter-page composition/separator semantics;
- implement deterministic exact-revision rendering under those frozen semantics without persisting source prose;
- freeze the rendered literary-body character count including spaces plus raw/normalized digests;
- verify that the frozen body itself, not nominal page count, topology, construct counts or scan byte size, clears **300,000 characters**.

Expand All @@ -119,13 +126,15 @@ FantLab work **`work969964`** identifies Darwin's 1859 English monograph. The cu
- [`source-edition-traces/darwin-origin-species-rachinsky-1864-ru.source-graph.json`](source-edition-traces/darwin-origin-species-rachinsky-1864-ru.source-graph.json) — exact route/topology evidence and apparatus roles.
- [`source-edition-traces/darwin-origin-species-rachinsky-1864-ru.page-revisions.index.json`](source-edition-traces/darwin-origin-species-rachinsky-1864-ru.page-revisions.index.json) — authoritative source-free index for all 418 exact Page revision identities and four committed shards.
- [`source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-surface.json`](source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-surface.json) — deterministic source-free 388-Page construct inventory with no source payload.
- [`source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-surface.source-graph.json`](source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-surface.source-graph.json) — provenance edges from frozen Page identities through literary composition and hosted markup audit to the durable inventory.
- [`source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-surface.source-graph.json`](source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-surface.source-graph.json) — provenance edges from frozen Page identities through literary composition and hosted markup audit to the durable inventory and frozen decision profile.
- [`source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-profile.json`](source-edition-traces/darwin-origin-species-rachinsky-1864-ru.render-profile.json) — deterministic fail-closed source-free rendering decision profile with explicit unresolved semantics and closed downstream gates.
- [`source-edition-traces/darwin-origin-species-rachinsky-1864-ru.scan-identity.json`](source-edition-traces/darwin-origin-species-rachinsky-1864-ru.scan-identity.json) — independently verified source-free DjVu byte count, SHA-1 cross-check and SHA-256; scan bytes are not stored.
- [`../../scriptorium/darwin_body_contract.py`](../../scriptorium/darwin_body_contract.py) — deterministic source-free 388/30 composition contract generator and fail-closed validator.
- [`../../scriptorium/darwin_render_surface.py`](../../scriptorium/darwin_render_surface.py) — exact-revision identity-verifying hosted construct audit.
- [`../../scriptorium/darwin_render_surface_freeze.py`](../../scriptorium/darwin_render_surface_freeze.py) — deterministic compact freeze reducer/validator.
- [`../../scriptorium/darwin_render_profile.py`](../../scriptorium/darwin_render_profile.py) — deterministic fail-closed source-free decision-profile builder/validator.
- [`../../scriptorium/darwin_scan_identity.py`](../../scriptorium/darwin_scan_identity.py) — streaming scan-identity capture/replay with provider cross-checks and no binary persistence.

## Next evidence

A later bounded unit should use the now-frozen template/tag surface to define and verify a candidate-specific fail-closed rendering profile for the 388 literary dependencies without persisting source prose. Only then should a separate identity/admission unit compute source-free counts/digests and independently prove the >=300,000-character threshold. FantLab parity remains a separate, stricter source-matching problem.
A later bounded unit should resolve and independently freeze the provider/template/reference/math semantics explicitly left unresolved by `scriptorium-darwin-page-render-profile-v1`; inter-page separator/composition semantics should remain a separately reviewable decision. Only after deterministic rendering exists should a separate identity/admission unit compute source-free counts/digests and independently prove the >=300,000-character threshold. FantLab parity remains a separate, stricter source-matching problem.
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"rendered orthography transformations unless a deterministic transform/identity contract is established"
]
},
"source_identity_status": "rendered_route_graph_418_page_revision_identities_388_30_composition_markup_surface_and_independent_scan_binary_identity_frozen_rendered_body_unfrozen",
"source_identity_status": "rendered_route_graph_418_page_revision_identities_388_30_composition_markup_surface_render_profile_and_independent_scan_binary_identity_frozen_rendered_body_unfrozen",
"source": {
"provider": "Russian Wikisource",
"observed_at": "2026-09-19",
Expand Down Expand Up @@ -77,7 +77,7 @@
],
"topology_frozen": true,
"complete_page_revision_set_frozen": true,
"identity_scope": "all 418 included Page-sequence dependencies have exact captured revision ID, revision timestamp and MediaWiki SHA-1 in a source-free digest-pinned sharded manifest; the 388 literary versus 30 apparatus composition decision is frozen by scriptorium-darwin-literary-body-contract-v1, while Page-wikitext-to-prose rendering and final literary-body identity remain unfrozen"
"identity_scope": "all 418 included Page-sequence dependencies have exact captured revision ID, revision timestamp and MediaWiki SHA-1 in a source-free digest-pinned sharded manifest; the 388 literary versus 30 apparatus composition decision and fail-closed rendering decision profile are frozen, while unresolved provider/template/reference/math/inter-page semantics and final literary-body identity remain unfrozen"
},
"page_revision_identity_manifest": {
"index": "darwin-origin-species-rachinsky-1864-ru.page-revisions.index.json",
Expand Down Expand Up @@ -136,7 +136,12 @@
411
],
"selection_and_partition_frozen": true,
"page_wikitext_to_prose_rendering_profile_frozen": false,
"page_wikitext_to_prose_rendering_profile_frozen": true,
"render_profile_version": "scriptorium-darwin-page-render-profile-v1",
"render_profile_sha256": "2ebcfea51282505c28824f54f6a1795bd7e87e2f958ac59f5c843e68c2066f3d",
"renderer_semantics_complete": false,
"renderer_implementation_ready": false,
"rendering_equivalence_claimed": false,
"composite_separator_semantics_frozen": false,
"literary_body_count_and_digests_frozen": false,
"source_text_committed": false
Expand Down Expand Up @@ -207,6 +212,7 @@
"source-free topology counts: 399 rendered-route Page dependencies and 418 parent+route dependencies",
"exact revision ID, revision timestamp and MediaWiki SHA-1 for all 418 included Page dependencies, committed source-free in four SHA-256-pinned shards",
"scriptorium-darwin-literary-body-contract-v1 selection/partition: 388 exact Page dependencies from numbered routes /1 through /14 are literary inputs and 30 frozen dependencies are apparatus",
"scriptorium-darwin-page-render-profile-v1 fail-closed decision profile over all 18 tag shapes / 5,164 tag tokens and 38 template shapes / 4,583 invocations; profile SHA-256 2ebcfea51282505c28824f54f6a1795bd7e87e2f958ac59f5c843e68c2066f3d; unresolved semantics are not promoted",
"Page-sequence 114 oldid=3364724 source-declared no-text and explicitly excluded from route /4",
"Page-sequence 411 oldid=3364733 source-declared no-text and uniquely aligned with displayed print page 388",
"displayed bibliographic print-page spans 1-387 and 389-399 attached to the rendered routes",
Expand All @@ -218,7 +224,7 @@
],
"unfrozen": [
"full mapping from displayed print page numbers to ProofreadPage sequence positions outside the uniquely resolved page-388 gap",
"candidate-specific Page-wikitext-to-prose rendering behavior over the 388 frozen literary dependencies",
"provider/template/reference/math semantics explicitly marked unresolved by the frozen fail-closed profile",
"composite separator/body-rendering semantics needed to materialize one literary body",
"literary-body character count and raw/normalized digests",
"FantLab analyzer-input bytes/source edition"
Expand All @@ -235,7 +241,7 @@
"page_revision_identity_freeze_is_not_threshold_proof": true,
"composition_contract_is_not_threshold_proof": true,
"admitted_for_calibration": false,
"reason": "The 418 exact Page identities and the 388-literary/30-apparatus composition partition are frozen source-free, and the exact backing DjVu now has an independent byte identity; Page-wikitext rendering, final body composition semantics, literary-body count and digests remain unfrozen, so scan identity does not prove the >=300,000-character rule."
"reason": "The 418 exact Page identities, 388-literary/30-apparatus partition, exact backing DjVu identity and fail-closed render decision profile are frozen source-free; unresolved provider/template/reference/math/inter-page semantics, literary-body count and digests remain unfrozen, so these prerequisites do not prove the >=300,000-character rule."
},
"fantlab": {
"work_id": 969964,
Expand All @@ -254,7 +260,7 @@
"m2_parity_admissible": false
},
"next_evidence": [
"Define and verify a candidate-specific fail-closed Page-wikitext-to-prose rendering profile over the 388 exact literary dependencies selected by scriptorium-darwin-literary-body-contract-v1, without persisting source prose.",
"Resolve and independently freeze the provider/template/reference/math semantics left explicit by scriptorium-darwin-page-render-profile-v1, then implement deterministic exact-revision rendering without persisting source prose; inter-page separator/composition semantics remain a separate decision.",
"Freeze literary-body character count plus raw and scriptorium-text-v1 normalized digests; admit to general calibration/profile corpus only if the rendered body independently clears 300,000 characters including spaces.",
"Keep FantLab matching separate: locate a linguistic result and analyzer-input/source-edition evidence attributable specifically to the Rachinsky translation before assigning diagnostic or M2 weight."
]
Expand Down
Loading
Loading