Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,21 @@ legally usable and at least 300,000 characters including spaces; copyrighted tex
not committed merely because it is accessible online. Shorter texts and excerpts are
still valid analyzer inputs, but their results carry representativeness warnings.

## Analyze your own text

From a checkout, using Python 3.13 and no third-party dependencies:

```bash
python -m scriptorium.analyze manuscript.txt --output analysis.json
python -m scriptorium.analyze manuscript.txt --format html --output report.html
```

Get 29 deterministic metric rows with explicit units, profile versions and source hashes.
The manuscript stays local; reports contain no source prose. Short texts are supported,
missing values remain explicit, and results are inferred/extension rather than FantLab
parity. Existing output files are never overwritten. See
[local analysis and optional dictionaries](docs/LOCAL_ANALYSIS.md).

## What works today

Scriptorium now has a standard-library-only deterministic analysis core:
Expand Down
78 changes: 78 additions & 0 deletions docs/LOCAL_ANALYSIS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Analyze a local text

From the repository root with Python 3.13, run the existing deterministic engine through
`scriptorium.analyze`. No third-party packages or network access are required.

```bash
python -m scriptorium.analyze manuscript.txt --output analysis.json
python -m scriptorium.analyze manuscript.txt --format html --title "My manuscript" --output report.html
```

JSON is the default; omitting `--output` writes the selected format to standard output.
The HTML file is self-contained and can be opened offline. Its tables include units,
compatibility labels, raw punctuation counts and keyboard-accessible scrolling regions.

## Reproducibility and scientific limits

The `scriptorium-local-analysis-v1` wrapper retains all 29 current engine rows without
changing their values or formulas. It records the metric contract, normalization/metric/
dialogue/vocabulary/punctuation profiles, raw-byte and normalized-text SHA-256 identities,
input size and dictionary identity. `--scriptorium-revision <git-sha>` records an explicit
caller-supplied revision; without it the revision is null, not guessed. The HTML exposes
profiles under **Analysis profiles**.

Inputs must be UTF-8. CRLF/CR and Unicode composition use the existing normalization
profile. Empty inputs are supported: counts may be zero, but undefined means or rates
remain null. A null vocabulary metric can also mean a missing dictionary or an incomplete
UASZ window; HTML therefore says **Unavailable**, not a fabricated zero or pass.

The 300,000-character threshold includes spaces. Shorter inputs are valid local analyses
with limited representativeness; meeting the length floor alone is not corpus admission,
proof of representativeness, an author profile or FantLab parity. FantLab-shaped rows
remain inferred; Scriptorium-only rows remain extensions. No morphology backend is
silently instantiated.

## Optional external dictionary

Supply a UTF-8 file with one lexeme per line and a non-empty profile together:

```bash
python -m scriptorium.analyze manuscript.txt --dictionary dictionary.txt \
--dictionary-profile my-dictionary-v1 --output with-dictionary.json
```

The engine performs its existing NFC/case-folded set normalization. A supplied dictionary
can enable active dictionary/nondictionary counts and complete-window UASZ values; short
inputs still lack complete larger windows. Results retain profile, normalized lexeme-set
SHA-256 and count, not dictionary contents. Reproducibility does not establish equivalence
to FantLab's undisclosed production dictionary.

## Privacy and file safety

Manuscripts and dictionary contents are not embedded in the returned aggregate artifacts,
sent to a service, or added to the corpus. Titles, dictionary profile labels and optional
revision metadata are caller-controlled and intentionally appear in reports. Hashes and
aggregate metrics can themselves identify a known work: treat generated reports as
private unless deliberately shared. There is no telemetry or external asset loading.

Output is create-only, including existing files, hard-link aliases and dangling symlinks.
The writer stages derived output in the destination directory, flushes it, then uses an
atomic hard-link creation without replacement. The filesystem must support hard links;
otherwise the command reports an error rather than falling back to unsafe overwriting.
Handled write failures remove temporary files and leave previous output untouched. A
process/OS crash may leave a hidden `.scriptorium-*.tmp` derived-report file. On POSIX,
new reports have mode 0600. Parent directories are created as needed.

## Library and checks

```python
from pathlib import Path
from scriptorium.analyze import build_local_analysis, render_html

artifact = build_local_analysis(Path("manuscript.txt").read_bytes())
html = render_html(artifact, title="Local analysis")
```

Run focused checks with `python -m unittest discover -s tests -p 'test_analyze*.py' -v`.
Tests exercise the real engine, deterministic output, Unicode normalization, dictionary
limits, privacy, output aliases, failed publication and empty inputs.
13 changes: 13 additions & 0 deletions project/CHANGELOG.d/2026-10-07-scrip-metric-003.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# SCRIP-METRIC-003 — local text analysis

Refs #269. Adopt the pending local-analysis implementation into the real metric engine:
UTF-8 input to deterministic JSON or readable offline HTML, optional explicit dictionary,
29 unchanged engine rows, source hashes/profiles, short-input and unavailable-value labels.

Add 22 focused tests, including direct engine equality, actual CLI file output, dictionary
privacy, normalization identity, empty input, threshold boundaries, non-finite rejection,
create-only aliases and atomic publication failure. README and LOCAL_ANALYSIS document
working commands and report privacy. No source text, corpus admission, morphology
substitution, analyzer formula, parity gate or Pages activation changes.

Full exact-head CI and later-run acceptance remain required before merge.
Loading
Loading