Skip to content

Spec conformance test suite #57

@craigsmitham

Description

@craigsmitham

Summary

Create a dedicated test project (GraphZen.SpecConformance.Tests) that maps directly to the GraphQL specification's section structure, with executable test coverage and spec traceability.

Motivation

We have solid test coverage across the codebase, but tests aren't organized as spec conformance tests with traceability back to specific spec sections. There's no way to answer "what percentage of the spec do we cover?" or "which spec rules are we missing?"

Approach

  • One test project (GraphZen.SpecConformance.Tests) with folder-per-chapter structure
  • [SpecSection("X.Y.Z", "Heading")] attribute on each conformance class, with hierarchical filtering
  • Coverage manifest + reflection-based coverage test ensuring every spec subsection has a class
  • The spec is the sole source of truth; graphql-js is a supplement for test cases, not a coverage guide
  • Authoring guidance lives in test/GraphZen.SpecConformance.Tests/CLAUDE.md

Each phase starts with a design stage (consult spec, consult graphql-js, determine infrastructure needs) before implementation — Phase 1 patterns don't necessarily transfer to other sections.

Phases

Phase Scope Status Issue/PR
Phase 1 Section 5 — Validation Complete #59
Phase 2 Section 2 — Language Not started #60
Phase 3 Section 3 — Type System Not started #61
Phase 4 Section 4 — Introspection Not started #62
Phase 5 Section 6 — Execution Not started #63
Phase 6 Section 7 — Response Not started #64

Phase 1 Results

  • 28 conformance classes covering every validation subsection (5.1.1 through 5.8.5)
  • 161 tests: 82 passing, 79 skipped with explicit gap descriptions
  • Rule-specific ExpectValid/ExpectErrors API with ToDeepEqual error assertions
  • Conformance gaps tracked in Finish ignored validation conformance gaps from spec suite #58

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions