Skip to content

Native OO-LD validator driven by the specification rule catalogue - #114

Open
LukasOro wants to merge 29 commits into
mainfrom
feat/validator-migration
Open

Native OO-LD validator driven by the specification rule catalogue#114
LukasOro wants to merge 29 commits into
mainfrom
feat/validator-migration

Conversation

@LukasOro

@LukasOro LukasOro commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

What

A native Python validation subsystem under src/oold/validation/, replacing use of the reference harness oold-schema/scripts/validate.mjs as a subprocess.

  • 39 checks across 11 families: schema.*, lint.*, context.*, roundtrip.*, generate.*, instance.*, variants, compliance.*, coverage.*, meta.*, and 22 rule.* checks
  • CLI: oold validate, oold validate-instance, oold compliance, oold meta, oold rules, oold checks
  • MCP server exposing the same validation entry points
  • 4 tracked meta-schema versions vendored byte-exact: 0.7.0, 0.8.0, 1.0.0-rc.1, 1.0.0-rc.2

Rationale

  • No Node.js dependency, and no subprocess boundary to marshal results across
  • Resolves @context chains that leave the schema's own directory, which the reference harness cannot do: its loader only maps its own directory
  • One code base validates against several meta-schema versions in a single run

Severity comes from the specification, never from this code

Each rule.* check reports a problem; whether that is a failure or a warning is read from the rule's level in oold-rules.json, the catalogue of numbered normative statements.

  • A rule absent from the selected version's catalogue, or deprecated there, is skipped with a message saying so, rather than checked anyway
  • Versions predating the catalogue skip the whole rule.* family
  • Checks judge the resolved context, not the literal @context, so a schema whose terms arrive through a remote context is not reported as violating them

Meta-schema vendoring

src/oold/validation/meta/<version>/ holds verbatim copies from oold-schema release tags, with a sha256 per file recorded in index.json. Nothing is fetched at runtime for a tracked version, so a released version cannot change meaning between runs. Unreleased upstream state is reachable with --meta remote.

  • Files are stored LF and marked -text in .gitattributes; a CRLF copy hashes differently, passing on Windows and failing on Linux
  • v1.0.0-rc.2 is the first release to ship the catalogue and the first to split the dialect meta-schema into a wrapper plus the base it $refs, so it declares its own file set
  • The fixture slice in tests/data/oold/ is refreshed from the same tag it records in fixtures.tag, so fixtures and meta-schemas never come from different releases

Coupling to oold-schema

The two repositories release on separate schedules, so neither pipeline waits on the other.

  • coverage.rules warns when a catalogued rule has no check, rather than failing, so a specification that has moved ahead cannot break this build
  • Currently 15 of 40 machine-checkable rules have no check: 4 predate OO-LD/oold-schema#124 and 11 arrived with the rules that release added
  • OO-LD/oold-schema#133 reclassifies OOLD-CMP-a05a as not machine-checkable; it reaches this repository at the next release, since tracked catalogues come from tags

Verification

  • 530 tests pass; make check, make validate and make docs-test exit 0
  • Parity tests assert this port reaches the same verdicts as the reference harness on the same fixtures, including check labels and triple counts: 6/6 with OOLD_SCHEMA_DIR set
  • Each deliberately broken fixture in tests/data/oold/broken/ is mapped to the check it must trip, so the checks are proven to fire rather than only to pass on valid input

LukasOro added 29 commits July 31, 2026 12:04
- Ports scripts/validate.mjs, pattern_lint.mjs and schema_to_frame.mjs into src/oold/validation/
- Backs the library API, an `oold validate` CLI (aliased `oold-validate`), and an MCP server
- Covers meta-schema, $ref composition, @context lint, RDF round-trip, remote-context and oneOf/anyOf checks, plus the compliance-suite and vocabulary-coverage cross-checks
- Meta-schemas are versioned under src/oold/validation/meta/, seeded from tag v0.7.0
- Parity is checked in tests/test_validation/test_parity_live.py against 98 pinned ajv-format outcomes
- Vendors v0.8.0 into the meta-schema version history; `latest` now resolves to it, 0.7.0 stays
- Refreshes the fixture slice from the same tag
- Excludes vendored meta-schemas and fixtures from pre-commit's pretty-format-json and whitespace fixers, which broke their recorded sha256 checksums
- Adds .gitattributes marking vendored files -text to stop core.autocrlf rewriting line endings
- Splits test_selection_expands_and_deduplicates into three tests pinning selector order, `all` version order, and explicit order
- MetaBundle loads the optional oold-rules.json, exposing rule(), has_rules and checkable_rules()
- Check gains a `rule` field, surfaced in the CLI (--verbose), the JSON report and MCP payloads
- CHECK_RULES maps checks to rules centrally in _Run.add, covering the four checks that enforce a single identifiable requirement
- New `oold rules list|explain` commands (--area, --unchecked) and a list_oold_rules MCP tool
- New coverage.rules check reports checkable rules with no check; it only ever warns
- MUST-level: rule.id (OOLD-VER-001), rule.id-fragment (OOLD-CMP-005), rule.range-ref (OOLD-EXT-005), rule.instance-type (OOLD-INS-002), rule.free-text-iri (OOLD-INS-009), rule.closed-object (OOLD-INS-005)
- SHOULD-level: rule.version (OOLD-VER-002), rule.id-alias (OOLD-INS-007), rule.dialect (OOLD-EXT-002), rule.processing-mode (OOLD-EXT-001)
- Checks are declared in a registry and export their own check-id-to-rule mapping
- Checks judge the resolved context rather than the schema's literal @context
- Takes coverage.rules from 21 of 25 checkable rules unchecked to 11
- Vendors meta-schema 1.0.0-rc.1, the first version to ship oold-rules.json
- Severity now comes from the catalogue; the FAIL/WARN column is removed from the check registry
- A rule absent from the selected version, or marked deprecated, is skipped with the reason given
- Fixes _version_key so pre-releases sort before their release (1.0.0-rc.1 before 1.0.0)
- Fixtures refreshed from the v1.0.0-rc.1 tag
- Seven hand-authored fixtures were CRLF while every fixture copied from an oold-schema tag is LF
- `.gitattributes` marks the fixture directory -text, so mixed endings looked deliberate
- Whitespace only; each file re-parsed and compared to its previous value after conversion
- tests/data/oold/README.md refreshed by copying from upstream to avoid a line-ending diff
- Linked from oold-schema's `make check` whenever the catalogue gains a rule
- Covers `applies_to`, severity coming from the catalogue rather than the check, and judging the resolved context instead of the literal @context
- Documents that verdicts are pinned by parity with the reference harness
- Documents that severity is read from the specification catalogue, not decided by a check
- Documents that vendored meta-schemas are checksummed bytes no formatter or line-ending conversion may touch
- AGENTS.md stays ignored
- Rule ids answer which requirement was violated and are permanent; check ids answer which check found it and follow the implementation
- Replaces an earlier "frozen inventory" proposal that hand-synced a second file of ids
- Verified against what the validator actually emits, and holds function references rather than path strings
- Kept out of docs/, since an unlisted page there is still built and published
- Deletes RULE_CHECKS, RULE_CHECK_MAP and CHECK_RULES in favor of one registry
- CheckInfo absorbs RuleCheck by gaining an optional `run` predicate
- rule_checks.py folds into check_registry.py
- Drops the emitting site from `oold checks explain`
- Gating is by presence in the vendored catalogue; a new rule's check needs no backward-compatibility code
- Four phase checks carry a rule id but were not gated by anything
- New `requires_rule` flag extends catalogue gating to any check, defaulting to off
- `since` cannot serve this purpose: all 34 rules carry since=1.0.0-rc.1, when the catalogue was minted
- A fourth drift test pins the promise
- `requires_rule` is insufficient for a legacy check whose rule is later superseded
- Replaced with `predates_catalog`, which asks what a pre-catalogue version cannot answer
- A rewording changes nothing, a data-driven change (0.8.0's no-coercion rule) is free, only bespoke detection needs a new predicate
- Marginal cost of a changed rule is one registry line
- Lists the full set of check ids; the previous registry missed six ids the validator actually emits
- Fixes the verification command: grepping `rule_checks` also matched `run_rule_checks()`
- Records the `compliance.<kind>` decision as a single normalised `compliance.*` family entry
- Notes `lint.iri-format` has never been observed to fire; drift test 2 should fail on it
- RuleCheck, RULE_CHECKS, RULE_CHECK_MAP and CHECK_RULES collapse into `CheckInfo`
- rule_checks.py folds into check_registry.py
- Purely structural: verdicts, messages and report shape unchanged, confirmed byte-for-byte and by the parity suite
- cli.py, mcp_server.py and two test modules move to importing from the registry
- Every emittable id now has a registry entry, including rule.checks, meta.self-check, compliance.suite and a single compliance.* family entry
- `predates_catalog` decides check behaviour against versions that ship no rule catalogue
- Adds the missing lint.iri-format fixture so the check is actually exercised
- Fixes drift test 4, which previously counted a gated SKIP as a run
- Verdicts unchanged on all three tracked versions, confirmed by diff and the parity suite
- Leaf.schema.json now requires `name`, defined only by the remote Thing context via ../Thing.schema.json
- Leaf's own inline context defines just `nickname`, so a literal-@context check would misreport `name` as undefined
- No new test needed; test_a_context_chain_leaving_the_directory_resolves already asserts the whole report passes
- Pre-existing drift, not introduced here: the line was exactly 120 characters, which failed `make check` on a clean checkout
- MetaBundle.self_check now validates the catalogue against the new oold-rules.schema.json, vendored alongside it
- A truncated or malformed catalogue previously looked like a version stating fewer rules, with checks skipping silently
- Loading stays lenient; the loss is now reported via meta.self-check instead of failing validation
- Fixture slice provenance moves from README prose into index.json; `fixtures.tag` is compared against the newest tracked version by a test
- Re-vendors rc.1's catalogue, which gained only the $schema line upstream
- meta/README.md now lists all five files to copy, including the catalogue and its schema added in 1.0.0-rc.1
- CONTRIBUTING.md now documents `predates_catalog` and its default
- Clarifies that a new rule.* check needs no broken-fixture test but does need a corpus schema that exercises its predicate
- Fixture README no longer states the slice's tag, now recorded in index.json
- Rule ids move to a minted hex suffix (e.g. OOLD-RT-002 -> OOLD-RT-08f2); all 14 registry-cited ids remapped
- `checkable` renamed to `machine_checkable` throughout, including checkable_rules() and the CLI label
- Four new checks: rule.uuid (OOLD-VER-edb9), rule.multilang-default (OOLD-EXT-dd76), rule.base-alignment (OOLD-CMP-53bf), rule.scoped-context (OOLD-CMP-5266)
- rule.scoped-context only flags an embed arriving by $ref to another document, not a self-reference
- Verdicts unchanged; parity holds at 6/6
- rule.instance-type and rule.closed-object previously passed without ever running, found via line coverage over a full corpus run
- New fixtures make each fail exactly the check it targets; closed_object also trips roundtrip.generated
- base_uri_misaligned is registered in the fixture table alongside them, having been added earlier without being listed
- rule.id-alias is left with a known gap: no fixture yet exercises its non-violating path
- Declaring any markdown_extensions replaces Zensical's default set rather than extending it, silently dropping fifteen defaults
- Admonitions in docs/how-to/backends.md, codegen.md, object-graph-mapping.md and rdf-export.md rendered as literal text
- zensical.toml now restates all 22 defaults of the installed 0.0.45
- scripts/check_markdown_extensions.py compares that restatement against the installed Zensical and fails on drift; wired into `make check`, a pre-commit hook, and docs CI
- pymdownx.smartsymbols stays enabled here, unlike oold-schema, since its trigger sequences only appear inside Mermaid fences
- rule.multilang-shape (OOLD-EXT-ef09), rule.dialect-version (OOLD-EXT-af50), rule.context-array-order (OOLD-CMP-e4a3), rule.versioned-id (OOLD-VER-534a)
- rule.context-array-order and rule.versioned-id read the literal @context and the rule's summary respectively, both documented exceptions
- Three fixtures added so rule.versioned-id is actually reached, not just guarded
- Not implemented: OOLD-CMP-a05a, OOLD-INS-9416, OOLD-RT-d376, OOLD-CMP-f3c7, each with a reason recorded in the code
- Verdicts unchanged; parity holds at 6/6
- Refreshes meta/1.0.0-rc.1/oold-rules.json from oold-schema: 40 rules to 43
- Adds OOLD-VER-befc, OOLD-VER-4261 and OOLD-EXT-1f92, split out of lead-in lists upstream
- `oold rules list --unchecked` goes from 7 to 10
- rules_source now records feat/rule-list-scope (oold-schema PR #124, unmerged)
- rule.versioned-id's docstring corrected: it is now explicitly the umbrella over OOLD-VER-befc and OOLD-VER-4261
- rules_source now records the pull request number alongside the repository, branch and pre-merge commit
- A rebased branch orphans a recorded commit; the entry it replaces pointed at exactly such an orphan on feat/rule-catalog-rc1
- Adds a `merged` slot to fill in once #124 lands
- oold-schema splits the dialect meta-schema into a wrapper plus oold-meta-schema-base.json
- `files` in meta/index.json is now per-source via `meta_files(source)`, so only the remote bundle carries the new base file
- Restores parity, which had been failing 4 of 6 because the wrapper's $ref could not resolve
- declared_keywords() now collects x-oold-* across every document in the bundle, restoring all 26 keywords and picking up x-oold-sssom
- Two meta_store tests updated to serve the four-file remote list
- rule.context-reflects-refs (OOLD-CMP-b926) and rule.branch-context-conflict (OOLD-CMP-1d7e); unenforced count 10 to 8
- Both read the authored @context rather than the resolved view, the same exception rule.context-array-order already takes
- rule.branch-context-conflict is narrowed via `entries`, distinguishing an authored override (dict) from a reflected conflict (string)
- OOLD-INS-1df7 deliberately not implemented; it duplicates what rule.free-text-iri already checks
- Both checks produce zero findings on the corpus and needed a broken fixture each to be reached
- rule.narrow-only enforces OOLD-CMP-f3c7; unenforced count 8 to 7
- CheckInfo gains `run_resolved`, letting a check receive the dereferenced schema instead of the authored one
- Ten comparable keywords are checked per member: numeric bounds, length/item/property bounds, multipleOf, enum, const, type, uniqueItems, additionalProperties
- `pattern` and `required` are deliberately excluded, with reasons recorded in the code
- Zero findings across the corpus; parity holds at 6/6
- Vendor 1.0.0-rc.2 as its own entry; rc.1 keeps its pre-release copy
- Declare its own four-file set: the dialect split into wrapper plus base
- Close rc.1's rules_source.merged; refresh fixture slice and fixtures.tag
- Unchecked rules 7 -> 15 of 40: 4 carried over, 11 newly released
- Fix a UI cross-reference test that asserted on an undefined keyword
@github-actions

github-actions Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Release preview

Merging this PR would release v0.17.0 (current: v0.16.3).

Changelog preview (truncated)
## v0.17.0 (2026-08-15)

### Bug Fixes

- **docs**: Restate and guard Zensical's default Markdown extensions
  ([`ebc3907`](https://github.com/OO-LD/oold-python/commit/ebc3907cae24a685b78edcd856ca514331148a95))

### Chores

- License fix
  ([`daa8d8c`](https://github.com/OO-LD/oold-python/commit/daa8d8cfa325a3745c851c845b9fae7b5e51a0cc))

- **validation**: Vendor the 43-rule catalogue
  ([`cbcf95a`](https://github.com/OO-LD/oold-python/commit/cbcf95a3042c41078c7b670de4ce77b21db36a31))

### Code Style

- Apply ruff-format to the check-registry drift test
  ([`ca11109`](https://github.com/OO-LD/oold-python/commit/ca1110907aa194b32458cfb90599c4b8a51609b5))

- **tests**: Store the hand-written fixtures with LF line endings
  ([`8c965d6`](https://github.com/OO-LD/oold-python/commit/8c965d6cd3aea4963e709fc05e304e789aa73995))

### Continuous Integration

- **release**: Add whats-changed notes with changelog link for zenodo
  ([`147d70f`](https://github.com/OO-LD/oold-python/commit/147d70fbf1c5da070f3cb9055fbe96e831fe8eaa))

- **release**: Update on title and authors
  ([`996c4de`](https://github.com/OO-LD/oold-python/commit/996c4dee6b4f07b6ec994e87cb341155a12e955c))

### Documentation

- Add CLAUDE.md with the conventions agents keep getting wrong
  ([`4fdda5e`](https://github.com/OO-LD/oold-python/commit/4fdda5e796089c4950437918d212c802b4e2e8b4))

- Explain how to turn a specification rule into a check
  ([`988bc38`](https://github.com/OO-LD/oold-python/commit/988bc38de68724a027268721fad8818786bd5184))

- Fix the vendoring procedure and say what a new check owes
  ([`92524af`](https://github.com/OO-LD/oold-python/commit/92524af7fd721819d8d0874757d23dcfbdc6ba92))

- **spec**: Collapse the check mappings into one registry structure
  ([`264d71e`](https://github.com/OO-LD/oold-python/commit/264d71e300095719cdfe52c86726fb5bb64888e1))

- **spec**: Correct the version gate, and cost out a changed rule
  ([`08b9945`](https://github.com/OO-LD/oold-python/commit/08b9945ea338c3a261eb08d28e82b2d2b7a062ea))

- **spec**: Design a check registry and an `oold checks` command
  ([`26ceba9`](https://github.com/OO-LD/oold-python/commit/26ceba9c63a9170baaa376c63b039ebfa666a100))

- **spec**: Pin where compatibility for a new rule's check lives
  ([`3ba00a6`](https://github.com/OO-LD/oold-python/commit/3ba00a6ddff0aa3c739f6739ae7b76b1b11ffda2))

- **spec**: State which ids the registry covers, and fix the grep guard
  ([`3b497b5`](https://github.com/OO-LD/oold-python/commit/3b497b586ed1e8b13699263b492a27e1acdb1141))

- **validation**: Record catalogue's source so a rebase cannot orphan it
  ([`5bcd043`](https://github.com/OO-LD/oold-python/commit/5bcd043721226b48280e7980890264df9cdc623b))

### Features

- **validation**: Add native OO-LD schema and instance validator
  ([`80ee29a`](https://github.com/OO-LD/oold-python/commit/80ee29a1c61cb429f8f39d8db70375508f7a8375))

- **validation**: Check catalogue and fixture slice against facts
  ([`a7a7f3c`](https://github.com/OO-LD/oold-python/commit/a7a7f3cd167e24e38731a117ad9a87494acbd666))

- **validation**: Cite the specification's rule ids in findings
  ([`f39b280`](https://github.com/OO-LD/oold-python/commit/f39b28057f24609789b5718099579fcffb49d560))

- **validation**: Drive rule checks from the specification catalogue
  ([`71c734c`](https://github.com/OO-LD/oold-python/commit/71c734c6fb1864a6d9274efaf017be6032b2b4c6))

- **validation**: Enforce four more catalogued rules
  ([`fb99f5c`](https://github.com/OO-LD/oold-python/commit/fb99f5c3aebdec7227bd2cd68840b4cc2bd8c061))

- **validation**: Enforce narrow-only composition
  ([`13ade19`](https://github.com/OO-LD/oold-python/commit/13ade196c9ba8d6a9a9e8501e20ac213a688aaa6))

Preview via python-semantic-release and conventional commits.

@github-actions

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

Click to see benchmark comparison
📊 Benchmark Comparison (threshold: 1.3x)
============================================================

➖ Unchanged (within threshold):
  ➖ test_simple_dict_document_store: 0.0019s → 0.0017s (-5.8%)
  ➖ test_sqlite_document_store: 0.0020s → 0.0019s (-5.1%)
  ➖ test_local_sparql_store: 0.0432s → 0.0456s (+5.6%)
  ➖ test_oneof_subschema: 0.0636s → 0.0678s (+6.6%)
  ➖ test_enum_docstrings: 0.0545s → 0.0579s (+6.2%)
  ➖ test_subclass_inheritance: 0.0580s → 0.0607s (+4.7%)
  ➖ test_class_hierarchy: 0.0567s → 0.0575s (+1.5%)
  ➖ test_core[v1]: 0.0412s → 0.0424s (+2.8%)
  ➖ test_core[v2]: 0.0469s → 0.0506s (+8.0%)
  ➖ test_schema_generation[v1]: 0.0018s → 0.0018s (+0.9%)
  ➖ test_schema_generation[v2]: 0.0030s → 0.0032s (+7.6%)
  ➖ test_simple_json: 0.0008s → 0.0007s (-4.2%)
  ➖ test_complex_graph: 0.0017s → 0.0017s (+0.6%)

============================================================
Summary: 0 regressions, 0 improvements, 13 unchanged
============================================================

✅ No significant performance regressions

Threshold: 1.3x (30% slower triggers a regression warning)

Note: Benchmarks are informational only and won't fail the build.

💡 Tip: Download the benchmark-results artifact for detailed JSON data

@github-actions

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

Click to see benchmark comparison
📊 Benchmark Comparison (threshold: 1.3x)
============================================================

➖ Unchanged (within threshold):
  ➖ test_simple_dict_document_store: 0.0016s → 0.0016s (+1.3%)
  ➖ test_sqlite_document_store: 0.0017s → 0.0017s (-1.1%)
  ➖ test_local_sparql_store: 0.0370s → 0.0380s (+2.6%)
  ➖ test_oneof_subschema: 0.0543s → 0.0552s (+1.5%)
  ➖ test_enum_docstrings: 0.0458s → 0.0468s (+2.2%)
  ➖ test_subclass_inheritance: 0.0488s → 0.0498s (+2.1%)
  ➖ test_class_hierarchy: 0.0468s → 0.0478s (+2.0%)
  ➖ test_core[v1]: 0.0348s → 0.0355s (+2.0%)
  ➖ test_core[v2]: 0.0398s → 0.0411s (+3.4%)
  ➖ test_schema_generation[v1]: 0.0015s → 0.0015s (+1.1%)
  ➖ test_schema_generation[v2]: 0.0026s → 0.0026s (+0.8%)
  ➖ test_simple_json: 0.0006s → 0.0006s (+0.6%)
  ➖ test_complex_graph: 0.0014s → 0.0015s (+1.9%)

============================================================
Summary: 0 regressions, 0 improvements, 13 unchanged
============================================================

✅ No significant performance regressions

Threshold: 1.3x (30% slower triggers a regression warning)

Note: Benchmarks are informational only and won't fail the build.

💡 Tip: Download the benchmark-results artifact for detailed JSON data

@LukasOro
LukasOro requested a review from simontaurus August 15, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant