diff --git a/README.md b/README.md index 4cec5ff6..ab95f4c3 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,24 @@ legally usable and at least 300,000 characters including spaces; copyrighted tex not committed merely because it is accessible online. Shorter texts and excerpts are still valid analyzer inputs, but their results carry representativeness warnings. +## Open a real diagnostic report + +From a checkout with Python 3.13, generate a readable offline report from the existing +full-work Anna Karenina comparison — no novel download is needed: + +```bash +python -m scriptorium.frozen_diagnostic_report \ + benchmarks/fantlab/work270306-wikisource-diagnostic.json \ + --title "Anna Karenina — full-work diagnostic" \ + --output anna-diagnostic.html +``` + +Open `anna-diagnostic.html` in a browser. It shows 28 historical expected/actual/delta +rows, explicit units and missing values, source hashes and diagnostic-only evidence +labels. This is a presentation of existing results, not a new analyzer run or FantLab +parity. Output is create-only; use a new filename on repeat runs. See +[`docs/FROZEN_DIAGNOSTIC_REPORT.md`](docs/FROZEN_DIAGNOSTIC_REPORT.md). + ## What works today Scriptorium now has a standard-library-only deterministic analysis core: diff --git a/docs/FROZEN_DIAGNOSTIC_REPORT.md b/docs/FROZEN_DIAGNOSTIC_REPORT.md new file mode 100644 index 00000000..faead049 --- /dev/null +++ b/docs/FROZEN_DIAGNOSTIC_REPORT.md @@ -0,0 +1,70 @@ +# Read a frozen full-work diagnostic + +From a repository checkout with Python 3.13, turn the existing aggregate Anna Karenina +comparison into a standalone report without downloading the novel: + +```bash +python -m scriptorium.frozen_diagnostic_report \ + benchmarks/fantlab/work270306-wikisource-diagnostic.json \ + --title "Anna Karenina — full-work diagnostic" \ + --output anna-diagnostic.html +``` + +Open `anna-diagnostic.html` in a browser. It needs no network, JavaScript, remote fonts +or external assets. Omit `--output` to write HTML to standard output. The file option is +create-only: an existing output is never overwritten, including after invalid input. +Use a new filename for a second report. Without `--title`, the candidate ID is shown. + +## What the report contains + +The existing historical diagnostic has 28 rows: 23 numeric actuals and five missing +actuals, with 26 `unresolved` and two `not_run` results. Tables group general, dialogue, +vocabulary and punctuation metrics and show units, expected values, actual values and +raw deltas. Captions, row/column headers, a skip link, focus styles and keyboard-focusable +horizontal table regions keep the wide tables inspectable on small screens. + +Replay hashes, chapter count, analyzer revision, Python version and hosted-run identity +are retained. Omitted families and observations are explicitly historical: the report +is not a new replay and does not imply that an old provider limitation is current. + +## Scientific boundary + +This renderer accepts the reviewed `scriptorium-frozen-diagnostic-v1` shape only. Its +source edition is unknown; parity admissibility remains false. The displayed M2 `0/5` +is the value recorded in that historical artifact, not a live project-progress query. +Numeric resemblance, including a zero delta, is never promoted to pass/fail or a +similarity score. No analyzer formula, source body or historical JSON is modified. + +Display uses up to six decimal places and scientific notation for very small nonzero +values. This is readability formatting, **not** FantLab display rounding or a tolerance +change. Percentage deltas are percentage points; missing values are not zero. +Integer delta arithmetic is exact. The `1e-9` absolute consistency check for floating +raw deltas handles serialized arithmetic only; it cannot admit a parity result. + +## Validation and privacy + +The CLI rejects duplicate JSON keys, unexpected fields, invalid hashes/locators, +non-finite or out-of-range numbers, delta inconsistency, `not_run` with an actual value, +pass/fail promotion, source-text publication flags, and source-match/M2 upgrades. +All artifact-controlled strings are HTML-escaped. + +Inputs must already be curated source-free aggregate artifacts. This is not a manuscript +sanitizer: reviewed observations and reason strings are intentionally displayed, so a +producer must not put private source prose in those fields. The renderer never fetches +source text or uploads anything. Generated HTML stays local and is not committed. + +## Verification and publication + +```bash +python -m unittest discover -s tests -p 'test_frozen_diagnostic_report.py' -v +python -m unittest discover -s tests -v +``` + +The focused suite includes the exact canonical diagnostic blob and deterministic output, +large-integer precision, tiny deltas, escaping, invalid-input rejection and preservation +of an existing output. The existing Pages PR workflow runs the full repository suite and +a byte-identical static-site rebuild on the PR head. + +This unit supplies a standalone report, not a new live Pages route. Adding the report to +the publication allow-list and renderer is a subsequent integration; Pages activation +and the existing deployment interlock are unchanged. diff --git a/project/CHANGELOG.d/2026-10-06-scrip-site-007.md b/project/CHANGELOG.d/2026-10-06-scrip-site-007.md new file mode 100644 index 00000000..ac90704e --- /dev/null +++ b/project/CHANGELOG.d/2026-10-06-scrip-site-007.md @@ -0,0 +1,15 @@ +## 2026-10-06 — readable frozen diagnostic report (#267) + +- Adopt SCRIP-SITE-007 into a normal product PR: standalone offline HTML/CLI for the + existing source-free full-work diagnostic, with units, explicit missing values, + historical evidence labels, replay identity and keyboard-accessible tables. +- Reject malformed/overstated input, duplicate keys, non-finite values and delta drift; + preserve integer precision and tiny nonzero deltas; keep output create-only. +- Add 23 focused regressions, including the unchanged canonical Anna diagnostic blob. +- Reconcile the Oct 2 owner selection correction into the compact product backlog: + independent report/CLI/Pages work does not wait on corpus or bookkeeping blockers. +- SCRIP-SITE-008's old local patch-transform self-test is not integration evidence: its + dedented replacement produces an IndentationError on the renderer function shape. + Correct and verify it against actual repository files before adoption. +- No analyzer formulas, historical benchmark JSON, corpus gates, M2 tolerances or Pages + activation/settings are changed. New code still requires later exact-head acceptance. diff --git a/project/STATE_AND_QUEUE.md b/project/STATE_AND_QUEUE.md index 365c980b..867dd37f 100644 --- a/project/STATE_AND_QUEUE.md +++ b/project/STATE_AND_QUEUE.md @@ -1,6 +1,6 @@ # STATE_AND_QUEUE — Scriptorium -STATE_REVISION: 289 +STATE_REVISION: 290 PHASE: M1 — Deterministic FantLab surface LAST_COMMITTED_RUN_AT: 2026-10-02T09:12:28Z LAST_RESULT: SCRIP-CORPUS-087 / Issue #263 / PR #264 completed. Independent exact-head review re-read the six-file recovery patch at `950404bb86c191b7eacdf21eb34c231f1d906fc4`, confirmed 24/24 pull-request-triggered workflow runs completed successfully with no submitted reviews or unresolved review threads, and found no semantic merge blocker. PR #264 was squash-merged with expected-head protection as `a9df79ec04664011f6dddf3593031424b15c1073`; Issue #263 closed completed. Embedded text-layer status remains unverified; literary body, >=300k admission, FantLab-input identity, diagnostics, M2 and parity remain closed. @@ -9,15 +9,21 @@ LAST_VERIFIED_PROGRESS: M1 deterministic metric families remain executable infer ## Current unit ```text -UNIT_ID: SCRIP-CORPUS-087 -ISSUE: #263 (closed completed) -STATUS: COMPLETE -PR: #264 (squash-merged as a9df79ec04664011f6dddf3593031424b15c1073) -REVIEWED_HEAD: 950404bb86c191b7eacdf21eb34c231f1d906fc4 -NEXT_ACTION: Resume normal-flow P1 SCRIP-CORPUS continuation. - Prefer an executable Petersburg/Perelman/Darwin-Rachinsky extraction or provenance unit that strengthens a >=300k candidate while keeping unsupported parity gates closed. +UNIT_ID: SCRIP-SITE-007 +ISSUE: #267 +STATUS: REVIEW_PENDING +PR: Resolve the linked product PR from Issue #267; GitHub owns current head/check/merge facts. +BASE: abe7bb889ef98437ba9faef5a79b023499556991 +NEXT_ACTION: Later exact-head acceptance of the readable frozen diagnostic report. + Inspect code/tests and applicable full-suite/Pages CI; do not merge in the authoring run. + Keep the existing historical diagnostic and all source-match/parity gates unchanged. ``` +PR #266 is merged as `abe7bb889ef98437ba9faef5a79b023499556991`; it is not a pending +recovery blocker. The older LAST_RESULT above describes the completed corpus unit. +The report PR adopts useful local work with 23 focused tests; hosted acceptance and +actual merge/publication are separate native GitHub effects, not implied here. + ## Current milestone gate M0 is closed. M1 remains open. General, dialogue, vocabulary and punctuation metric families are executable inferred candidates. POS retains separate work-page-compatible and diagnostic-only methodology surfaces; runtime `N` noun/cardinal ambiguity is unresolved. @@ -26,11 +32,23 @@ M2 remains open at **0/5 source-matched works**. Public-source freezing, bibliog ## Queue -Recovery/review-ready work and failing required checks preempt normal selection. +The 2026-10-02 owner correction authorizes solo product/backlog ownership and independent +analyzer, CLI and public-experience delivery. Recovery and failing checks preempt only +when executable; unchanged corpus, write or bookkeeping blockers are scoped dependencies, +not a global ordering rule. Preserve scientific/security and later-run acceptance gates. + +Next user-visible milestone: a reader can open a real full-work diagnostic with units, +missing values and provenance, then reach it through the static site; a local-text CLI +must remain usable independently of source-matched corpus admission. This incremental +milestone does not declare M1/M2 complete. | Priority | Unit | Mode | Deliverable | Gate / dependency | | --- | --- | --- | --- | --- | -| P1 | SCRIP-CORPUS continuation | corpus / provenance | Add or strengthen legally usable >=300k candidates, prioritizing diversity beyond 19th-century Russian classics; Petersburg, Perelman and Darwin/Rachinsky have candidate-specific extraction/renderer work available | Preserve translation/edition identity and legal provenance; renderer/OCR work must keep body/admission/FantLab/M2 gates closed until separately verified | +| P1 | SCRIP-SITE-007 / #267 | acceptance / delivery | Accept the offline frozen diagnostic HTML/CLI on its fresh PR head | Full applicable tests/checks; later-run acceptance; no source-match promotion | +| P2 | SCRIP-SITE-008 | public experience | Integrate the reviewed report into the existing Pages allow-list and work route | Reuse report primitives; correct the old local patcher's indentation error; test actual renderer, full build and byte-identical rebuild; deployment rights block live publication only | +| P3 | SCRIP-METRIC-003 | analyzer / CLI | Local UTF-8 text to readable report and JSON using current deterministic metrics | Adopt/revalidate pending local work against actual core modules; private text stays local; dictionary-dependent unknowns stay explicit | +| P4 | SCRIP-SITE-009 | public experience | Reader-first homepage and README with real actions and evidence labels | Revalidate pending local work against accepted Pages items; no invented analyses or capabilities | +| P5 | SCRIP-CORPUS continuation | corpus / provenance | Strengthen a legally usable >=300k diversity candidate with a new executable extraction/provenance result | Preserve translation/edition identity; candidate-specific body/FantLab gates; unchanged scans do not block P1-P4 | ## Retained corpus / provenance orientation diff --git a/scriptorium/frozen_diagnostic_report.py b/scriptorium/frozen_diagnostic_report.py new file mode 100644 index 00000000..ee5d2bb2 --- /dev/null +++ b/scriptorium/frozen_diagnostic_report.py @@ -0,0 +1,467 @@ +"""Readable source-free report for frozen full-work FantLab diagnostics. + +This presentation layer consumes the already source-free +``scriptorium-frozen-diagnostic-v1`` artifact. It does not recompute literary metrics, +recover source text, rank incomparable deltas, or upgrade diagnostic evidence to parity. + +The current full-work Anna Karenina artifact is source-unmatched by design, so every +published metric remains ``unresolved`` or ``not_run`` here. +""" + +from __future__ import annotations + +import argparse +import html +import json +import math +import os +import re +from collections.abc import Mapping, Sequence +from pathlib import Path, PurePosixPath +from typing import Any, Final + +SCHEMA_VERSION: Final = "scriptorium-frozen-diagnostic-v1" +ALLOWED_RESULTS: Final = frozenset({"unresolved", "not_run"}) +HEX64_RE: Final = re.compile(r"^[0-9a-f]{64}$") +GROUPS: Final = ( + ("fantlab.general.", "General"), + ("fantlab.dialogue.", "Dialogue"), + ("fantlab.vocabulary.", "Vocabulary"), + ("fantlab.punctuation.", "Punctuation"), +) + + +class FrozenDiagnosticReportError(ValueError): + """Raised when a frozen diagnostic cannot be presented without overclaim.""" + + +def validate_artifact(artifact: Mapping[str, object]) -> Mapping[str, object]: + value = _mapping(artifact, "artifact") + expected_top = { + "artifact_version", + "benchmark_id", + "candidate_id", + "evidence", + "epistemic_status", + "metrics", + "omitted_metric_families", + "observations", + } + if set(value) != expected_top: + raise FrozenDiagnosticReportError("top-level keys drift from frozen-diagnostic-v1") + if value.get("artifact_version") != SCHEMA_VERSION: + raise FrozenDiagnosticReportError("unsupported frozen diagnostic artifact") + _text(value.get("benchmark_id"), "benchmark_id", 300) + _text(value.get("candidate_id"), "candidate_id", 300) + + evidence = _mapping(value.get("evidence"), "evidence") + required_evidence = { + "hosted_run_id", + "python_version", + "scriptorium_revision", + "reference_path", + "source_manifest_path", + "replay_integrity", + "chapter_count", + "raw_sha256", + "normalized_sha256", + "source_text_committed", + "source_text_uploaded", + } + if set(evidence) != required_evidence: + raise FrozenDiagnosticReportError("evidence keys drift from reviewed contract") + if evidence.get("replay_integrity") != "passed": + raise FrozenDiagnosticReportError("replay_integrity must be passed before presentation") + _positive_int(evidence.get("hosted_run_id"), "evidence.hosted_run_id") + _positive_int(evidence.get("chapter_count"), "evidence.chapter_count") + _text(evidence.get("python_version"), "evidence.python_version", 100) + _text(evidence.get("scriptorium_revision"), "evidence.scriptorium_revision", 100) + _repo_json_path(evidence.get("reference_path"), "evidence.reference_path") + _repo_json_path(evidence.get("source_manifest_path"), "evidence.source_manifest_path") + _sha256(evidence.get("raw_sha256"), "evidence.raw_sha256") + _sha256(evidence.get("normalized_sha256"), "evidence.normalized_sha256") + if evidence.get("source_text_committed") is not False: + raise FrozenDiagnosticReportError("source_text_committed must remain false") + if evidence.get("source_text_uploaded") is not False: + raise FrozenDiagnosticReportError("source_text_uploaded must remain false") + + status = _mapping(value.get("epistemic_status"), "epistemic_status") + required_status = { + "status", + "fantlab_source_edition_match", + "m2_parity_admissible", + "m2_source_matched_progress", + "reason", + } + if set(status) != required_status: + raise FrozenDiagnosticReportError("epistemic_status keys drift from reviewed contract") + if status.get("status") != "diagnostic_only": + raise FrozenDiagnosticReportError("artifact must remain diagnostic_only") + if status.get("fantlab_source_edition_match") != "unknown": + raise FrozenDiagnosticReportError("FantLab source edition must remain unknown") + if status.get("m2_parity_admissible") is not False: + raise FrozenDiagnosticReportError("M2 parity must remain inadmissible") + if status.get("m2_source_matched_progress") != "0/5": + raise FrozenDiagnosticReportError("unexpected M2 source-matched progress") + _text(status.get("reason"), "epistemic_status.reason", 2000) + + metrics = _mapping(value.get("metrics"), "metrics") + if not metrics: + raise FrozenDiagnosticReportError("diagnostic metrics must be non-empty") + for metric_id, raw in metrics.items(): + if not isinstance(metric_id, str) or not metric_id.startswith("fantlab."): + raise FrozenDiagnosticReportError(f"invalid metric id: {metric_id!r}") + row = _mapping(raw, f"metrics.{metric_id}") + if set(row) != {"expected", "actual", "raw_delta", "result"}: + raise FrozenDiagnosticReportError(f"{metric_id} keys drift from reviewed contract") + expected = _number(row.get("expected"), f"{metric_id}.expected") + actual = row.get("actual") + delta = row.get("raw_delta") + result = row.get("result") + if not isinstance(result, str) or result not in ALLOWED_RESULTS: + raise FrozenDiagnosticReportError( + f"{metric_id} result {result!r} would overclaim source-unmatched evidence" + ) + if actual is None: + if delta is not None: + raise FrozenDiagnosticReportError( + f"{metric_id} cannot have raw_delta without actual" + ) + else: + actual_num = _number(actual, f"{metric_id}.actual") + delta_num = _number(delta, f"{metric_id}.raw_delta") + calculated = actual_num - expected + if ( + delta_num != calculated + if all(isinstance(n, int) for n in (expected, actual_num, delta_num)) + else not math.isclose(delta_num, calculated, rel_tol=0.0, abs_tol=1e-9) + ): + raise FrozenDiagnosticReportError( + f"{metric_id}.raw_delta is inconsistent with actual - expected" + ) + if result == "not_run" and actual is not None: + raise FrozenDiagnosticReportError( + f"{metric_id} is not_run but contains an actual value" + ) + + omitted = _mapping(value.get("omitted_metric_families"), "omitted_metric_families") + for family, reason in omitted.items(): + _text(family, "omitted metric family", 100) + _text(reason, f"omitted_metric_families.{family}", 1000) + + observations = value.get("observations") + if not isinstance(observations, list): + raise FrozenDiagnosticReportError("observations must be an array") + for index, observation in enumerate(observations): + _text(observation, f"observations[{index}]", 2000) + + return value + + +def summarize(artifact: Mapping[str, object]) -> dict[str, int]: + value = validate_artifact(artifact) + metrics = _mapping(value["metrics"], "metrics") + actual = 0 + missing = 0 + unresolved = 0 + not_run = 0 + for raw in metrics.values(): + row = _mapping(raw, "metric") + if row["actual"] is None: + missing += 1 + else: + actual += 1 + if row["result"] == "unresolved": + unresolved += 1 + elif row["result"] == "not_run": + not_run += 1 + return { + "metric_count": len(metrics), + "actual_count": actual, + "missing_actual_count": missing, + "unresolved_count": unresolved, + "not_run_count": not_run, + } + + +def render_html(artifact: Mapping[str, object], *, title: str | None = None) -> str: + value = validate_artifact(artifact) + title_text = _text(title if title is not None else value["candidate_id"], "title", 300) + summary = summarize(value) + evidence = _mapping(value["evidence"], "evidence") + status = _mapping(value["epistemic_status"], "epistemic_status") + metrics = _mapping(value["metrics"], "metrics") + omitted = _mapping(value["omitted_metric_families"], "omitted_metric_families") + observations = value["observations"] + + grouped: dict[str, list[tuple[str, Mapping[str, object]]]] = { + label: [] for _, label in GROUPS + } + grouped["Other"] = [] + for metric_id in sorted(metrics): + group = "Other" + for prefix, label in GROUPS: + if metric_id.startswith(prefix): + group = label + break + grouped[group].append((metric_id, _mapping(metrics[metric_id], metric_id))) + + sections = [] + for group, rows in grouped.items(): + if not rows: + continue + rendered = [] + for metric_id, row in rows: + result = str(row["result"]) + rendered.append( + "" + f'{_esc(metric_id)}' + f'{_esc(_unit(metric_id))}' + f'{_esc(_fmt(row["expected"]))}' + f'{_esc(_fmt(row["actual"]))}' + f'{_esc(_signed(row["raw_delta"]))}' + f'{_esc(result)}' + "" + ) + sections.append( + f'

{_esc(group)}

' + f'
' + f'' + '' + '' + f'{"".join(rendered)}
{_esc(group)}: expected / actual / raw delta
MetricUnitFantLab expectedScriptorium actualRaw ΔEvidence result
' + ) + + omitted_rows = "".join( + f"
{_esc(family)}
{_esc(reason)}
" + for family, reason in sorted(omitted.items()) + ) + observation_items = "".join( + f"
  • {_esc(item)}
    • " for item in observations + ) + + css = """ +:root{color-scheme:light dark;font-family:ui-serif,Georgia,Cambria,"Times New Roman",serif;line-height:1.55} +*{box-sizing:border-box}body{margin:0;background:Canvas;color:CanvasText} +main,footer{max-width:96rem;margin:auto;padding:clamp(1rem,4vw,2.5rem)} +.eyebrow{font:800 .78rem/1.2 system-ui,sans-serif;letter-spacing:.1em;text-transform:uppercase} +h1{font-size:clamp(2.5rem,8vw,5.6rem);line-height:.96;max-width:18ch;margin:.35rem 0 1rem} +h2{margin-top:2.2rem}.lede,.notice,li{font:1rem/1.6 system-ui,sans-serif;max-width:78rem} +.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(min(11rem,100%),1fr));gap:.75rem;margin:1.4rem 0 2rem} +.card,.notice{border:1px solid currentColor;border-radius:.9rem;padding:1rem} +.card strong{display:block;font-size:1.7rem}.card span{font:.84rem/1.3 system-ui,sans-serif} +.table-wrap{overflow-x:auto;margin-bottom:2rem}table{width:100%;border-collapse:collapse;font:.84rem/1.4 system-ui,sans-serif} +th,td{text-align:left;padding:.62rem;border-bottom:1px solid currentColor;vertical-align:top} +th[scope=row]{min-width:21rem}.pill{display:inline-block;border:1px solid currentColor;border-radius:999px;padding:.15rem .42rem;font-size:.72rem;font-weight:700} +code{overflow-wrap:anywhere}dd{margin-left:0;overflow-wrap:anywhere}dt{font-weight:700;margin-top:.8rem} +*:focus-visible{outline:3px solid currentColor;outline-offset:3px} a{color:inherit;display:inline-block;min-height:44px;padding:.6rem 0} caption{text-align:left;margin:.5rem 0} .skip:not(:focus){position:absolute;clip-path:inset(50%);width:1px;height:1px;overflow:hidden}footer{opacity:.72;font:.85rem/1.45 system-ui,sans-serif} +@media(max-width:44rem){main,footer{padding:1rem}th[scope=row]{min-width:15rem}th,td{padding:.48rem}} +@media(prefers-reduced-motion:reduce){*{scroll-behavior:auto!important}} +""" + body = ( + '
    ' + '

    Scriptorium · full-work diagnostic

    ' + f'

    {_esc(title_text)}

    ' + '

    A field-by-field, source-free comparison against FantLab’s ' + 'published values. Numeric resemblance is visible, but source identity remains ' + 'unknown, so this page does not promote any row to parity.

    ' + '
    ' + f'{_card(summary["metric_count"], "published metric rows")}' + f'{_card(summary["actual_count"], "numeric actuals")}' + f'{_card(summary["missing_actual_count"], "missing actuals")}' + f'{_card(summary["unresolved_count"], "unresolved")}' + f'{_card(summary["not_run_count"], "not run")}' + '
    ' + '
    Scientific boundary. ' + f'{_esc(status["reason"])} ' + 'M2 source-matched progress in this artifact: 0/5.
    ' + '

    Historical snapshot, not a new analyzer run. Display uses up to six ' + 'decimal places; small nonzero values use scientific notation. This formatting is ' + 'not FantLab display rounding or a parity tolerance. Δ uses the row unit; for ' + 'percentage rows it means percentage points. Missing is not zero.

    ' + '

    Reproducibility identity

    ' + f'
    Benchmark ID
    {_esc(value["benchmark_id"])}
    ' + f'
    Candidate ID
    {_esc(value["candidate_id"])}
    ' + f'
    Scriptorium revision
    {_esc(evidence["scriptorium_revision"])}
    ' + f'
    Python
    {_esc(evidence["python_version"])}
    ' + f'
    Hosted run
    {_esc(evidence["hosted_run_id"])}
    ' + f'
    Frozen chapters
    {_esc(evidence["chapter_count"])}
    ' + f'
    Raw SHA-256
    {_esc(evidence["raw_sha256"])}
    ' + f'
    Normalized SHA-256
    {_esc(evidence["normalized_sha256"])}
    ' + f'
    Source-edition match
    {_esc(status["fantlab_source_edition_match"])}
    ' + '
    ' + + '
    ' + "".join(sections) + "
    " + + '

    Omitted metric families (at capture)

    ' + + omitted_rows + + '

    Reviewed observations (historical)

    Privacy. ' + 'This report contains hashes, aggregate metrics and reviewed observations only. ' + 'No novel prose or replayed source text is embedded.
    ' + ) + return ( + '' + '' + f'{_esc(title_text)} — Scriptorium' + f'{body}\n' + ) + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Render a source-free scriptorium-frozen-diagnostic-v1 artifact as HTML." + ) + parser.add_argument("artifact", type=Path) + parser.add_argument("--title", help="Display title; defaults to the artifact candidate ID") + parser.add_argument("--output", type=Path) + args = parser.parse_args(argv) + try: + artifact = json.loads( + args.artifact.read_text(encoding="utf-8"), object_pairs_hook=_unique_keys + ) + rendered = render_html(_mapping(artifact, str(args.artifact)), title=args.title) + if args.output is None: + print(rendered, end="") + else: + _write_new(args.output, rendered) + except (OSError, UnicodeDecodeError, json.JSONDecodeError, ValueError) as exc: + parser.error(str(exc)) + return 0 + + +def _unique_keys(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + if key in result: + raise FrozenDiagnosticReportError(f"duplicate JSON key: {key}") + result[key] = value + return result + + +def _unit(metric_id: str) -> str: + if metric_id.endswith(".per_1000_words"): + return "events / 1,000 words" + if metric_id.endswith("_percent"): + return "% (delta: percentage points)" + if metric_id == "fantlab.general.mean_word_length_chars": + return "characters / word" + if metric_id in { + "fantlab.general.mean_sentence_length_chars", + "fantlab.dialogue.mean_narration_sentence_length_chars", + "fantlab.dialogue.mean_dialogue_sentence_length_chars", + }: + return "characters / sentence" + if metric_id.endswith("_chars") or metric_id.endswith(".characters"): + return "characters" + if ".uasz_" in metric_id: + return "mean distinct dictionary words / " + metric_id.rsplit("_", 1)[1] + "-word window" + if metric_id.startswith("fantlab.vocabulary."): + return "distinct words (definitions differ)" + if metric_id.endswith(".words"): + return "word tokens" + return "unspecified in artifact" + + +def _mapping(value: object, label: str) -> Mapping[str, Any]: + if not isinstance(value, Mapping): + raise FrozenDiagnosticReportError(f"{label} must be an object") + return value + + +def _text(value: object, label: str, limit: int) -> str: + if not isinstance(value, str) or not value.strip(): + raise FrozenDiagnosticReportError(f"{label} must be non-empty text") + result = value.strip() + if len(result) > limit: + raise FrozenDiagnosticReportError(f"{label} is too long") + return result + + +def _positive_int(value: object, label: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise FrozenDiagnosticReportError(f"{label} must be a positive integer") + return value + + +def _number(value: object, label: str) -> int | float: + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise FrozenDiagnosticReportError(f"{label} must be numeric") + try: + number = float(value) + except OverflowError as exc: + raise FrozenDiagnosticReportError(f"{label} is outside the supported numeric range") from exc + if not math.isfinite(number): + raise FrozenDiagnosticReportError(f"{label} must be finite") + return value + + +def _sha256(value: object, label: str) -> str: + if not isinstance(value, str) or HEX64_RE.fullmatch(value) is None: + raise FrozenDiagnosticReportError(f"{label} must be a lowercase SHA-256") + return value + + +def _repo_json_path(value: object, label: str) -> str: + text = _text(value, label, 500) + if ( + text != value or "\\" in text or ":" in text + or PurePosixPath(text).is_absolute() + or any(part in {"", ".", ".."} for part in text.split("/")) + or not text.endswith(".json") + ): + raise FrozenDiagnosticReportError(f"{label} must be a safe repository JSON path") + return text + + +def _fmt(value: object) -> str: + if value is None: + return "— (missing)" + number = _number(value, "display value") + if isinstance(number, int) or number.is_integer(): + return f"{int(number):,}" + if number and abs(number) < 0.000001: + return f"{number:.6g}" + return f"{number:,.6f}".rstrip("0").rstrip(".") + + +def _signed(value: object) -> str: + if value is None: + return "— (missing)" + number = _number(value, "display delta") + if number == 0: + return "0" + return ("+" if number > 0 else "") + _fmt(number) + + +def _esc(value: object) -> str: + return html.escape(str(value), quote=True) + + +def _card(value: object, label: str) -> str: + return ( + '
    ' + f'{_esc(value)}{_esc(label)}
    ' + ) + + +def _write_new(path: Path, content: str) -> None: + try: + path.parent.mkdir(parents=True, exist_ok=True) + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o666) + except FileExistsError as exc: + raise FrozenDiagnosticReportError(f"output already exists: {path}") from exc + try: + with os.fdopen(fd, "w", encoding="utf-8", newline="\n") as handle: + handle.write(content) + handle.flush() + os.fsync(handle.fileno()) + except Exception: + try: + path.unlink() + except OSError: + pass + raise + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_frozen_diagnostic_report.py b/tests/test_frozen_diagnostic_report.py new file mode 100644 index 00000000..cf688a44 --- /dev/null +++ b/tests/test_frozen_diagnostic_report.py @@ -0,0 +1,271 @@ +import contextlib +import io +import hashlib +import copy +import json +from pathlib import Path +import tempfile +import unittest + +import scriptorium.frozen_diagnostic_report as report + + +def artifact(): + return { + "artifact_version": "scriptorium-frozen-diagnostic-v1", + "benchmark_id": "fantlab-work1-2022-09-19", + "candidate_id": "fixture-work", + "evidence": { + "hosted_run_id": 123, + "python_version": "3.13.15", + "scriptorium_revision": "abc123", + "reference_path": "benchmarks/fantlab/work1.json", + "source_manifest_path": "corpus/candidates/source-edition-traces/work.revisions.json", + "replay_integrity": "passed", + "chapter_count": 2, + "raw_sha256": "a" * 64, + "normalized_sha256": "b" * 64, + "source_text_committed": False, + "source_text_uploaded": False, + }, + "epistemic_status": { + "status": "diagnostic_only", + "fantlab_source_edition_match": "unknown", + "m2_parity_admissible": False, + "m2_source_matched_progress": "0/5", + "reason": "Source edition is not established.", + }, + "metrics": { + "fantlab.general.characters": { + "expected": 10, + "actual": 12, + "raw_delta": 2, + "result": "unresolved", + }, + "fantlab.vocabulary.active_dictionary": { + "expected": 5, + "actual": None, + "raw_delta": None, + "result": "not_run", + }, + "fantlab.punctuation.question.per_1000_words": { + "expected": 1.25, + "actual": 1.0, + "raw_delta": -0.25, + "result": "unresolved", + }, + }, + "omitted_metric_families": { + "pos": "Provider execution not available." + }, + "observations": [ + "A visible difference is diagnostic only.", + ], + } + + +class FrozenDiagnosticReportTests(unittest.TestCase): + def test_summary_keeps_missing_and_results_distinct(self): + self.assertEqual( + report.summarize(artifact()), + { + "metric_count": 3, + "actual_count": 2, + "missing_actual_count": 1, + "unresolved_count": 2, + "not_run_count": 1, + }, + ) + + def test_raw_delta_is_rechecked(self): + bad = artifact() + bad["metrics"]["fantlab.general.characters"]["raw_delta"] = 3 + with self.assertRaisesRegex(report.FrozenDiagnosticReportError, "raw_delta"): + report.validate_artifact(bad) + + def test_missing_actual_cannot_have_delta(self): + bad = artifact() + bad["metrics"]["fantlab.vocabulary.active_dictionary"]["raw_delta"] = 0 + with self.assertRaisesRegex(report.FrozenDiagnosticReportError, "without actual"): + report.validate_artifact(bad) + + def test_not_run_cannot_have_actual(self): + bad = artifact() + row = bad["metrics"]["fantlab.vocabulary.active_dictionary"] + row["actual"] = 5 + row["raw_delta"] = 0 + with self.assertRaisesRegex(report.FrozenDiagnosticReportError, "not_run"): + report.validate_artifact(bad) + + def test_pass_or_fail_is_rejected_for_source_unmatched_artifact(self): + for result in ("pass", "fail"): + bad = artifact() + bad["metrics"]["fantlab.general.characters"]["result"] = result + with self.subTest(result=result): + with self.assertRaisesRegex(report.FrozenDiagnosticReportError, "overclaim"): + report.validate_artifact(bad) + + def test_source_text_flags_fail_closed(self): + for key in ("source_text_committed", "source_text_uploaded"): + bad = artifact() + bad["evidence"][key] = True + with self.subTest(key=key): + with self.assertRaises(report.FrozenDiagnosticReportError): + report.validate_artifact(bad) + + def test_source_edition_and_m2_must_remain_open(self): + bad = artifact() + bad["epistemic_status"]["fantlab_source_edition_match"] = "exact" + with self.assertRaisesRegex(report.FrozenDiagnosticReportError, "source edition"): + report.validate_artifact(bad) + bad = artifact() + bad["epistemic_status"]["m2_parity_admissible"] = True + with self.assertRaisesRegex(report.FrozenDiagnosticReportError, "M2 parity"): + report.validate_artifact(bad) + + def test_html_groups_metric_families_and_preserves_unknown_as_dash(self): + rendered = report.render_html(artifact(), title="Example") + self.assertIn("

    General

    ", rendered) + self.assertIn("

    Vocabulary

    ", rendered) + self.assertIn("

    Punctuation

    ", rendered) + self.assertIn("—", rendered) + self.assertIn("not_run", rendered) + + def test_html_escapes_artifact_controlled_text(self): + value = artifact() + value["observations"] = [""] + rendered = report.render_html(value, title="") + self.assertIn("<script>", rendered) + self.assertIn("<img", rendered) + self.assertNotIn("", output.read_text(encoding="utf-8")) + with self.assertRaises(SystemExit): + report.main([str(source), "--output", str(output)]) + + + def test_canonical_repository_diagnostic_is_unchanged_and_readable(self): + source = Path(__file__).resolve().parents[1] / "benchmarks/fantlab/work270306-wikisource-diagnostic.json" + raw = source.read_bytes() + sha = hashlib.sha1(b"blob " + str(len(raw)).encode() + b"\0" + raw).hexdigest() + self.assertEqual(sha, "0d5e420abb45e2b9e178df75a479ab7cdd5a4150") + value = json.loads(raw) + self.assertEqual(report.summarize(value), { + "metric_count": 28, "actual_count": 23, "missing_actual_count": 5, + "unresolved_count": 26, "not_run_count": 2, + }) + rendered = report.render_html(value) + self.assertEqual(rendered.count('scope="row"'), 28) + self.assertIn("1,705,605", rendered) + self.assertIn("0/5", rendered) + self.assertIn(value["evidence"]["normalized_sha256"], rendered) + + def test_generic_title_comes_from_candidate_not_anna(self): + rendered = report.render_html(artifact()) + self.assertIn("fixture-work", rendered) + self.assertNotIn("Anna Karenina", rendered) + + def test_bad_status_type_has_controlled_error(self): + value = artifact() + value["metrics"]["fantlab.general.characters"]["result"] = [] + with self.assertRaises(report.FrozenDiagnosticReportError): + report.validate_artifact(value) + + def test_huge_numeric_value_has_controlled_error(self): + value = artifact() + value["metrics"]["fantlab.general.characters"]["expected"] = 10 ** 1000 + with self.assertRaises(report.FrozenDiagnosticReportError): + report.validate_artifact(value) + + def test_large_integer_delta_does_not_lose_precision(self): + value = artifact() + row = value["metrics"]["fantlab.general.characters"] + row.update(expected=2 ** 53, actual=2 ** 53 + 1, raw_delta=1) + report.validate_artifact(value) + self.assertIn("9,007,199,254,740,993", report.render_html(value)) + row["raw_delta"] = 0 + with self.assertRaises(report.FrozenDiagnosticReportError): + report.validate_artifact(value) + + def test_tiny_nonzero_delta_does_not_display_as_zero(self): + self.assertEqual(report._signed(1e-10), "+1e-10") + self.assertEqual(report._signed(-1e-10), "-1e-10") + + def test_noncanonical_reference_paths_fail_closed(self): + for path in ("../secret.json", "a/../b.json", "C:/secret.json", + "C:\\secret.json", "a//b.json", "a/./b.json", " a/b.json"): + value = artifact() + value["evidence"]["reference_path"] = path + with self.subTest(path=path): + with self.assertRaises(report.FrozenDiagnosticReportError): + report.validate_artifact(value) + + def test_cli_rejects_duplicate_keys_without_creating_output(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + source, output = root / "input.json", root / "out.html" + source.write_text('{"artifact_version": "bad", "artifact_version": "other"}', encoding="utf-8") + with contextlib.redirect_stderr(io.StringIO()) as errors: + with self.assertRaises(SystemExit) as raised: + report.main([str(source), "--output", str(output)]) + self.assertEqual(raised.exception.code, 2) + self.assertIn("duplicate JSON key", errors.getvalue()) + self.assertFalse(output.exists()) + + def test_invalid_artifact_preserves_existing_output(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + source, output = root / "input.json", root / "out.html" + source.write_text("{}", encoding="utf-8") + output.write_text("last valid report", encoding="utf-8") + with contextlib.redirect_stderr(io.StringIO()): + with self.assertRaises(SystemExit): + report.main([str(source), "--output", str(output)]) + self.assertEqual(output.read_text(encoding="utf-8"), "last valid report") + + def test_table_keyboard_access_units_and_historical_boundary(self): + rendered = report.render_html(artifact()) + self.assertIn('href="#metrics"', rendered) + self.assertIn('id="metrics"', rendered) + self.assertIn('tabindex="0" role="region"', rendered) + self.assertIn('', rendered) + self.assertIn('scope="col"', rendered) + self.assertIn("events / 1,000 words", rendered) + self.assertIn("Historical snapshot, not a new analyzer run", rendered) + self.assertIn("Missing is not zero", rendered) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_frozen_diagnostic_units.py b/tests/test_frozen_diagnostic_units.py new file mode 100644 index 00000000..a1f3d09c --- /dev/null +++ b/tests/test_frozen_diagnostic_units.py @@ -0,0 +1,57 @@ +"""Unit-label regressions; historical numeric values must not change.""" +import copy +import hashlib +import json +from pathlib import Path +import unittest + +from scriptorium import frozen_diagnostic_report as report + +MEAN_UNITS = { + "fantlab.general.mean_word_length_chars": "characters / word", + "fantlab.general.mean_sentence_length_chars": "characters / sentence", + "fantlab.dialogue.mean_narration_sentence_length_chars": "characters / sentence", + "fantlab.dialogue.mean_dialogue_sentence_length_chars": "characters / sentence", +} + + +class FrozenDiagnosticUnitTests(unittest.TestCase): + def test_mean_length_denominators(self): + for metric_id, expected in MEAN_UNITS.items(): + with self.subTest(metric_id=metric_id): + self.assertEqual(report._unit(metric_id), expected) + + def test_other_units_are_not_reclassified(self): + controls = { + "fantlab.general.characters": "characters", + "fantlab.general.words": "word tokens", + "fantlab.dialogue.share_percent": "% (delta: percentage points)", + "fantlab.punctuation.dash.per_1000_words": "events / 1,000 words", + "fantlab.extension.sentence_length_chars": "characters", + "fantlab.unknown": "unspecified in artifact", + } + for metric_id, expected in controls.items(): + with self.subTest(metric_id=metric_id): + self.assertEqual(report._unit(metric_id), expected) + + def test_canonical_html_keeps_four_denominators_and_frozen_bytes(self): + path = Path(__file__).resolve().parents[1] / "benchmarks/fantlab/work270306-wikisource-diagnostic.json" + raw = path.read_bytes() + self.assertEqual( + hashlib.sha1(b"blob " + str(len(raw)).encode() + b"\0" + raw).hexdigest(), + "0d5e420abb45e2b9e178df75a479ab7cdd5a4150", + ) + artifact = json.loads(raw) + before = copy.deepcopy(artifact) + rendered = report.render_html(artifact) + self.assertEqual(rendered.count("characters / word"), 1) + self.assertEqual(rendered.count("characters / sentence"), 3) + self.assertEqual(rendered.count('scope="row"'), 28) + self.assertNotIn("unspecified in artifact", rendered) + self.assertEqual(artifact, before) + self.assertEqual(report.render_html(artifact), rendered) + self.assertFalse(artifact["epistemic_status"]["m2_parity_admissible"]) + + +if __name__ == "__main__": + unittest.main()