diff --git a/.github/agents/developer.agent.md b/.github/agents/developer.agent.md index 35f5dda..a95c562 100644 --- a/.github/agents/developer.agent.md +++ b/.github/agents/developer.agent.md @@ -21,7 +21,7 @@ Perform software development tasks by determining and applying appropriate stand 5. **Formatting**: Run `pwsh ./fix.ps1` to silently apply all available auto-fixers (dotnet format, markdown, YAML) before committing 6. **Build and test** (code changes only): Run `pwsh ./build.ps1` and confirm it - passes — report FAILED if the build or any tests fail + passes - report FAILED if the build or any tests fail 7. **Generate completion report** per the AGENTS.md reporting requirements - save to `.agent-logs/{agent-name}-{subject}-{unique-id}.md` and return the summary to the caller diff --git a/.github/standards/coding-principles.md b/.github/standards/coding-principles.md index 937f39c..9e67fbb 100644 --- a/.github/standards/coding-principles.md +++ b/.github/standards/coding-principles.md @@ -20,9 +20,10 @@ All code MUST follow literate programming principles: matches design intent without reading the full codebase - **Logical Separation**: Complex functions use block comments to separate and describe logical steps within the implementation -- **Full Symbol Documentation**: ALL symbols (public, protected, and private) - have comprehensive documentation because reviewers and auditors must verify - every implementation detail, not just the public interface +- **Full Symbol Documentation**: ALL symbols have comprehensive documentation + because reviewers and auditors must verify every implementation detail, not + just the public interface - access-level specifics (public, protected, + private, internal, etc.) vary by language; see the language-specific standard - **Clarity Over Cleverness**: Code should be immediately understandable by team members ## API Documentation diff --git a/.github/standards/csharp-language.md b/.github/standards/csharp-language.md index 9ada3f1..6df39cd 100644 --- a/.github/standards/csharp-language.md +++ b/.github/standards/csharp-language.md @@ -49,7 +49,7 @@ public double ProcessReading(double reading, CalibrationProfile calibration) Key qualities demonstrated above: - **``** is a brief one-liner explaining *what* the method does -- **``** sits directly after summary and carries the extended intent — +- **``** sits directly after summary and carries the extended intent - *why* it exists, design decisions, thread-safety, and side-effect disclosures - **`` tags** state constraints (finite, non-null) so callers know what is valid without reading the body diff --git a/.github/standards/csharp-testing.md b/.github/standards/csharp-testing.md index b8fd426..181de02 100644 --- a/.github/standards/csharp-testing.md +++ b/.github/standards/csharp-testing.md @@ -18,17 +18,20 @@ Every xUnit v3 test project requires the following package references for | Package | Purpose | | ------- | ------- | -| `xunit.v3` | xUnit v3 framework (monolithic — includes assertions and fixtures) | +| `xunit.v3` | xUnit v3 framework (monolithic - includes assertions and fixtures) | | `Microsoft.NET.Test.Sdk` | Required by the VSTest/`dotnet test` host for test discovery | | `xunit.runner.visualstudio` | VSTest adapter that bridges xUnit v3 to `dotnet test` | -| `NSubstitute` | Mocking library | Omitting `Microsoft.NET.Test.Sdk` or `xunit.runner.visualstudio` causes tests to be silently undiscoverable by `dotnet test`. +If tests require mocking of dependencies, add `NSubstitute` as a package +reference - it is recommended when mocking is needed but is not required for +every test project. + # Test Style -Test names appear in requirements traceability matrices — use the hierarchical +Test names appear in requirements traceability matrices - use the hierarchical naming pattern, and follow AAA with labeled comments: - **System tests**: `{SystemName}_{Functionality}_{Scenario}_{ExpectedBehavior}` @@ -55,11 +58,11 @@ public void UserValidator_ValidateEmail_InvalidFormat_ThrowsArgumentException() These are non-obvious v3 behaviors that differ from v2 or common assumptions: - **`IAsyncLifetime`**: Both `InitializeAsync` and `DisposeAsync` return `ValueTask` - in v3, not `Task` — using `Task` compiles but does not satisfy the v3 interface + in v3, not `Task` - using `Task` compiles but does not satisfy the v3 interface - **`Assert.Multiple`**: Use to collect all assertion failures in a single test rather than stopping at the first - **`[Collection]` without `[CollectionDefinition]`**: Silently disables parallelism - without providing any shared fixture — always pair them or remove `[Collection]` + without providing any shared fixture - always pair them or remove `[Collection]` # Quality Checks @@ -69,6 +72,6 @@ Before submitting C# tests, verify: - [ ] Test names follow hierarchical naming pattern above - [ ] Each test verifies single, specific behavior (no shared state between tests) - [ ] Both success and failure scenarios covered including edge cases -- [ ] External dependencies mocked with NSubstitute +- [ ] External dependencies mocked with NSubstitute (when mocking is needed) - [ ] Tests linked to requirements with source filters where needed - [ ] Test results generated in TRX format for ReqStream compatibility (`dotnet test --logger trx`) diff --git a/.github/standards/design-documentation.md b/.github/standards/design-documentation.md index 846f6dc..3b448f3 100644 --- a/.github/standards/design-documentation.md +++ b/.github/standards/design-documentation.md @@ -112,7 +112,7 @@ src/Project2Name/ If the design references external documents (standards, specifications), include a `## References` section in `introduction.md`. This is the **only** place in the -design document collection where a References section should appear — do not add +design document collection where a References section should appear - do not add one to any other design file. ### Companion Artifact Structure (RECOMMENDED) @@ -177,7 +177,7 @@ implementation specification for formal code review: - **Architectural Clarity**: Clearly define component boundaries and interfaces - **Traceability**: Link to requirements where applicable using ReqStream patterns - **Verbal Cross-References**: Reference other parts of the design by name (e.g., - "See *Parser Design* for more details") — do not use markdown hyperlinks, which + "See *Parser Design* for more details") - do not use markdown hyperlinks, which break in compiled PDFs # Mermaid Diagram Integration diff --git a/.github/standards/requirements-principles.md b/.github/standards/requirements-principles.md index ac89321..b6cf136 100644 --- a/.github/standards/requirements-principles.md +++ b/.github/standards/requirements-principles.md @@ -29,7 +29,7 @@ implementation code. - **Valid**: "The parser shall report the line number of the first syntax error." - **Not a requirement (design decision)**: "The parser shall use a `TokenStream` class." -A unit may use its own name freely — that is identity, not HOW. What is +A unit may use its own name freely - that is identity, not HOW. What is forbidden is describing *internal construction*: class names, method signatures, algorithms, or data structures. diff --git a/.github/standards/software-items.md b/.github/standards/software-items.md index dcc88cb..4e5c90e 100644 --- a/.github/standards/software-items.md +++ b/.github/standards/software-items.md @@ -90,6 +90,6 @@ the item is correct, well-designed, and proven to work: - **Requirements** - WHAT the item must do (drives all other artifacts; applies to all item types) - **Design** - HOW the item satisfies its requirements (in-house items only: system, subsystem, unit) -- **Verification Design** - HOW the requirements will be tested (in-house items only: system, subsystem, unit) +- **Verification Design** - HOW the requirements will be tested (applies to all item types) - **Source code** - The implementation of the design (in-house units only) - **Tests** - PROOF the item does WHAT it is required to do (applies to all item types) diff --git a/.github/standards/technical-documentation.md b/.github/standards/technical-documentation.md index b85573b..7ff5b5a 100644 --- a/.github/standards/technical-documentation.md +++ b/.github/standards/technical-documentation.md @@ -29,18 +29,18 @@ Each document collection under `docs/` follows this layout: ```text docs/{collection}/ - title.txt # MANDATORY — YAML document metadata (title, author, etc.) - definition.yaml # MANDATORY — Pandoc build definition (inputs, template, paths) - introduction.md # MANDATORY — document introduction (Purpose, Scope, References) + title.txt # MANDATORY - YAML document metadata (title, author, etc.) + definition.yaml # MANDATORY - Pandoc build definition (inputs, template, paths) + introduction.md # MANDATORY - document introduction (Purpose, Scope, References) {section}.md # optional checked-in content sections (zero or more) - generated/ # BUILD OUTPUT — never read, edit, or lint these files + generated/ # BUILD OUTPUT - never read, edit, or lint these files {report}.md # generated by CI tools (ReqStream, ReviewMark, SarifMark, etc.) {collection}.html # generated by Pandoc ``` Without `title.txt` and `definition.yaml` the pipeline cannot generate the document. When creating a new document collection, create these three files together and use -the existing collections under `docs/` as templates — they share a consistent +the existing collections under `docs/` as templates - they share a consistent structure across all collections. **`title.txt`** - YAML front matter with document metadata. Use the existing @@ -111,7 +111,7 @@ References in design/technical documents must point to **external specifications Do **not** use markdown hyperlinks to reference other sections or documents. Markdown anchor links (`[text](#heading)`) and relative file links work in a browser but break when compiled to a PDF. -Instead use **verbal references** — plain prose that identifies the target by name: +Instead use **verbal references** - plain prose that identifies the target by name: > See *XYZ Design* for more details. > @@ -140,7 +140,7 @@ for consistency and professional presentation: # Auto-Generated Content (CRITICAL) -**NEVER read, lint, or modify files inside any `generated/` folder** — they are +**NEVER read, lint, or modify files inside any `generated/` folder** - they are build outputs that are overwritten on every CI run: - **Location**: All generated files live in `generated/` subfolders within their diff --git a/.github/standards/verification-documentation.md b/.github/standards/verification-documentation.md index a9ea92d..f6f407f 100644 --- a/.github/standards/verification-documentation.md +++ b/.github/standards/verification-documentation.md @@ -13,7 +13,7 @@ Read these standards first before applying this standard: # Core Principles -Verification design is the bridge between requirements and tests — it documents HOW +Verification design is the bridge between requirements and tests - it documents HOW requirements will be verified, enabling reviewers to confirm test completeness without reading implementation code. @@ -24,38 +24,43 @@ Organize under `docs/verification/` mirroring the software item hierarchy: ```text docs/verification/ ├── introduction.md # Verification overview -└── {system-name}/ # System-level verification folder (one per system) - ├── {system-name}.md # System-level verification design - ├── {subsystem-name}/ # Subsystem (kebab-case); may nest recursively - │ ├── {subsystem-name}.md # Subsystem verification design - │ ├── {child-subsystem}/ # Child subsystem (same structure as parent) - │ └── {unit-name}.md # Unit-level verification design documents - └── {unit-name}.md # Top-level unit verification documents (if not in subsystem) +├── {system-name}/ # System-level verification folder (one per system) +│ ├── {system-name}.md # System-level verification design +│ ├── {subsystem-name}/ # Subsystem (kebab-case); may nest recursively +│ │ ├── {subsystem-name}.md # Subsystem verification design +│ │ ├── {child-subsystem}/ # Child subsystem (same structure as parent) +│ │ └── {unit-name}.md # Unit-level verification design documents +│ └── {unit-name}.md # Top-level unit verification documents (if not in subsystem) +└── ots/ # OTS items (one verification file per OTS item) + └── {ots-name}.md # Verification evidence for each OTS item ``` ## introduction.md (MANDATORY) -Follow the standard `introduction.md` format from `technical-documentation.md`. The Scope -section MUST state that verification documentation covers the same software items as the -design documentation — it MUST NOT include test infrastructure or third-party OTS items. +Follow the standard `introduction.md` format from `technical-documentation.md`. Scope +covers all software items including OTS items (via self-validation if appropriate). Include a Companion Artifact Structure note so agents and reviewers can navigate from any artifact to all related files: ```text -Each software item in the structure above has corresponding artifacts in -parallel directory trees: - +In-house items have parallel artifacts in: - Requirements: `docs/reqstream/{system}/.../{item}.yaml` (kebab-case) -- Design docs: `docs/design/{system}/.../{item}.md` (kebab-case) -- Verification design: `docs/verification/{system}/.../{item}.md` (kebab-case) -- Source code: `src/{System}/.../{Item}.{ext}` (cased per language - see `software-items.md`) -- Tests: `test/{System}.Tests/.../{Item}Tests.{ext}` (cased per language - see `software-items.md`) -- Review-sets: defined in `.reviewmark.yaml` +- Design: `docs/design/{system}/.../{item}.md` (kebab-case) +- Verification: `docs/verification/{system}/.../{item}.md` (kebab-case) +- Source: `src/{System}/.../{Item}.{ext}` (cased per language) +- Tests: `test/{System}.Tests/.../{Item}Tests.{ext}` (cased per language) + +OTS items have parallel artifacts in: +- Requirements: `docs/reqstream/ots/{ots-name}.yaml` (kebab-case) +- Verification: `docs/verification/ots/{ots-name}.md` (kebab-case) +- Tests: `test/{OtsName}.Tests/...` (cased per language, if required) + +Review-sets: defined in `.reviewmark.yaml` ``` If the verification design references external documents (standards, specifications), include -a `## References` section in `introduction.md` only — do not add one to any other verification file. +a `## References` section in `introduction.md` only - do not add one to any other verification file. ## System Verification Design (MANDATORY) @@ -86,15 +91,23 @@ For each unit, create `{unit-name}.md` covering: - Which dependencies are mocked and how they are configured - Coverage mapping of every unit requirement to at least one named test scenario +## OTS Verification Evidence (when OTS items are used) + +For each OTS item, create `docs/verification/ots/{ots-name}.md` covering: + +- The OTS item's required functionality (reference `docs/reqstream/ots/{ots-name}.yaml`) +- Verification of each requirement (using self-validation evidence if appropriate) +- Coverage mapping of OTS requirements to test scenarios + # Writing Guidelines - **Test Coverage**: Map every requirement to at least one named test scenario so reviewers can verify completeness without reading test code -- **Scenario Clarity**: Name each scenario clearly — "Valid input returns parsed result" not "Test 1" +- **Scenario Clarity**: Name each scenario clearly - "Valid input returns parsed result" not "Test 1" - **Boundary Conditions**: Call out boundary values, error inputs, and edge cases explicitly - **Isolation Strategy**: Describe what is mocked or stubbed and why at each level - **Traceability**: Link to requirements where applicable using ReqStream patterns -- **Verbal Cross-References**: Reference other documents by name — do not use markdown +- **Verbal Cross-References**: Reference other documents by name - do not use markdown hyperlinks, which break in compiled PDFs Mermaid diagrams may supplement text descriptions where test flow benefits from visual @@ -111,4 +124,5 @@ Before submitting verification documentation, verify: - [ ] Subsystem documentation folders use kebab-case names mirroring the source subsystem structure - [ ] All documents follow technical documentation formatting standards - [ ] Content is current with requirements and test implementation +- [ ] Every OTS item has `docs/verification/ots/{ots-name}.md` with requirement coverage - [ ] Documents are integrated into ReviewMark review-sets for formal review diff --git a/.reviewmark.yaml b/.reviewmark.yaml index e8b70ae..629bd8b 100644 --- a/.reviewmark.yaml +++ b/.reviewmark.yaml @@ -57,6 +57,7 @@ reviews: - "docs/reqstream/template-dot-net-library/platform-requirements.yaml" - "docs/design/introduction.md" - "docs/design/template-dot-net-library/**/*.md" + - "docs/verification/introduction.md" - id: TemplateDotNetLibrary-AllRequirements title: Review of Template DotNet Library requirements quality and traceability @@ -74,3 +75,64 @@ reviews: - "docs/verification/template-dot-net-library/demo.md" - "src/DemaConsulting.TemplateDotNetLibrary/Demo.cs" - "test/DemaConsulting.TemplateDotNetLibrary.Tests/DemoTests.cs" + + # OTS Items + - id: OTS-BuildMark + title: Review of BuildMark OTS verification evidence + paths: + - "docs/reqstream/ots/buildmark.yaml" + - "docs/verification/ots/buildmark.md" + + - id: OTS-FileAssert + title: Review of FileAssert OTS verification evidence + paths: + - "docs/reqstream/ots/fileassert.yaml" + - "docs/verification/ots/fileassert.md" + + - id: OTS-Pandoc + title: Review of Pandoc OTS verification evidence + paths: + - "docs/reqstream/ots/pandoc.yaml" + - "docs/verification/ots/pandoc.md" + + - id: OTS-ReqStream + title: Review of ReqStream OTS verification evidence + paths: + - "docs/reqstream/ots/reqstream.yaml" + - "docs/verification/ots/reqstream.md" + + - id: OTS-ReviewMark + title: Review of ReviewMark OTS verification evidence + paths: + - "docs/reqstream/ots/reviewmark.yaml" + - "docs/verification/ots/reviewmark.md" + + - id: OTS-SarifMark + title: Review of SarifMark OTS verification evidence + paths: + - "docs/reqstream/ots/sarifmark.yaml" + - "docs/verification/ots/sarifmark.md" + + - id: OTS-SonarMark + title: Review of SonarMark OTS verification evidence + paths: + - "docs/reqstream/ots/sonarmark.yaml" + - "docs/verification/ots/sonarmark.md" + + - id: OTS-VersionMark + title: Review of VersionMark OTS verification evidence + paths: + - "docs/reqstream/ots/versionmark.yaml" + - "docs/verification/ots/versionmark.md" + + - id: OTS-WeasyPrint + title: Review of WeasyPrint OTS verification evidence + paths: + - "docs/reqstream/ots/weasyprint.yaml" + - "docs/verification/ots/weasyprint.md" + + - id: OTS-xUnit + title: Review of xUnit OTS verification evidence + paths: + - "docs/reqstream/ots/xunit.yaml" + - "docs/verification/ots/xunit.md" diff --git a/docs/design/template-dot-net-library/template-dot-net-library.md b/docs/design/template-dot-net-library/template-dot-net-library.md index 3e5e099..aee080b 100644 --- a/docs/design/template-dot-net-library/template-dot-net-library.md +++ b/docs/design/template-dot-net-library/template-dot-net-library.md @@ -23,7 +23,7 @@ best practices. The system consists of: - **ReqStream**: Requirements traceability enforcement - **ReviewMark**: File review management - **BuildMark/VersionMark**: Build and version documentation -- **SonarMark/SARIFMark**: Quality analysis integration +- **SonarMark/SarifMark**: Quality analysis integration ### Public API diff --git a/docs/reqstream/ots/buildmark.yaml b/docs/reqstream/ots/buildmark.yaml index bd2a0ab..9e820c0 100644 --- a/docs/reqstream/ots/buildmark.yaml +++ b/docs/reqstream/ots/buildmark.yaml @@ -18,3 +18,7 @@ sections: tags: [ots] tests: - BuildMark_MarkdownReportGeneration + - BuildMark_GitIntegration + - BuildMark_IssueTracking + - BuildMark_KnownIssuesReporting + - BuildMark_RulesRouting diff --git a/docs/reqstream/ots/fileassert.yaml b/docs/reqstream/ots/fileassert.yaml index 6e11352..5596256 100644 --- a/docs/reqstream/ots/fileassert.yaml +++ b/docs/reqstream/ots/fileassert.yaml @@ -12,11 +12,12 @@ sections: title: FileAssert shall validate generated documents against acceptance criteria. justification: | DemaConsulting.FileAssert validates HTML and PDF documents produced during the - build, asserting that each document exists, has a non-trivial size, is structurally - valid, and contains expected content. It provides OTS evidence for Pandoc and - WeasyPrint and independently confirms file assertion is functioning. Self-validation - proves the tool itself is operational before ReqStream consumes the results. + build, asserting that each document exists and contains expected content. It + provides OTS evidence for Pandoc and WeasyPrint and independently confirms file + assertion is functioning. Self-validation proves the tool itself is operational + before ReqStream consumes the results. tags: [ots] tests: - - FileAssert_VersionDisplay - - FileAssert_HelpDisplay + - FileAssert_Results + - FileAssert_Exists + - FileAssert_Contains diff --git a/docs/reqstream/ots/pandoc.yaml b/docs/reqstream/ots/pandoc.yaml index a87fa2c..dd79478 100644 --- a/docs/reqstream/ots/pandoc.yaml +++ b/docs/reqstream/ots/pandoc.yaml @@ -23,4 +23,5 @@ sections: - Pandoc_ReviewPlanHtml - Pandoc_ReviewReportHtml - Pandoc_DesignHtml + - Pandoc_VerificationHtml - Pandoc_UserGuideHtml diff --git a/docs/reqstream/ots/reqstream.yaml b/docs/reqstream/ots/reqstream.yaml index 5f49d09..8de838b 100644 --- a/docs/reqstream/ots/reqstream.yaml +++ b/docs/reqstream/ots/reqstream.yaml @@ -18,4 +18,18 @@ sections: --enforce proves all requirements are covered and that ReqStream is functioning. tags: [ots] tests: + - ReqStream_RequirementsProcessing + - ReqStream_TraceMatrix + - ReqStream_ReportExport + - ReqStream_TagsFiltering - ReqStream_EnforcementMode + - id: Template-OTS-ReqStream-Lint + title: ReqStream shall identify and report issues in requirements files when run in lint mode. + justification: | + DemaConsulting.ReqStream provides a lint mode that checks requirements YAML files for + structural and semantic issues such as duplicate IDs, missing fields, and broken + references. Running lint in CI provides early detection of requirements authoring errors + before enforcement and report-generation steps execute. + tags: [ots] + tests: + - ReqStream_Lint diff --git a/docs/reqstream/ots/reviewmark.yaml b/docs/reqstream/ots/reviewmark.yaml index dc80162..25d9177 100644 --- a/docs/reqstream/ots/reviewmark.yaml +++ b/docs/reqstream/ots/reviewmark.yaml @@ -8,24 +8,58 @@ sections: sections: - title: ReviewMark Requirements requirements: - - id: Template-OTS-ReviewMark-Plan - title: ReviewMark shall generate a review plan from the review configuration. + - id: Template-OTS-ReviewMark + title: ReviewMark shall produce review documentation from the review configuration. justification: | DemaConsulting.ReviewMark reads the .reviewmark.yaml configuration and the review - evidence store to produce a review plan documenting file review coverage and currency. - It runs in the same CI pipeline that produces the TRX test results, so a successful - pipeline run is evidence that ReviewMark executed without error. + evidence store to produce a review plan and review report documenting file review + coverage and currency. It runs in the same CI pipeline that produces the TRX test + results, so a successful pipeline run is evidence that ReviewMark executed without error. tags: [ots] tests: - ReviewMark_ReviewPlanGeneration - - - id: Template-OTS-ReviewMark-Report - title: ReviewMark shall generate a review report from the review configuration. + - ReviewMark_ReviewReportGeneration + - id: Template-OTS-ReviewMark-IndexScan + title: ReviewMark shall scan PDF evidence files into an index catalogue. justification: | - DemaConsulting.ReviewMark reads the .reviewmark.yaml configuration and the review - evidence store to produce a review report documenting file review coverage and currency. - It runs in the same CI pipeline that produces the TRX test results, so a successful - pipeline run is evidence that ReviewMark executed without error. + ReviewMark's --index command scans review evidence PDFs and writes an index.json + catalogue used by the enforcement pipeline to verify review currency. tags: [ots] tests: - - ReviewMark_ReviewReportGeneration + - ReviewMark_IndexScan + - id: Template-OTS-ReviewMark-Enforce + title: ReviewMark shall enforce review currency by exiting non-zero when files are not current. + justification: | + ReviewMark's --enforce command exits with a non-zero code when any file in a review + set has been modified after the most recent review, making stale reviews a + build-breaking condition. + tags: [ots] + tests: + - ReviewMark_Enforce + - id: Template-OTS-ReviewMark-Elaborate + title: ReviewMark shall elaborate review-set details for agent and pipeline use. + justification: | + ReviewMark's --elaborate command prints the review-set ID, fingerprint, and file + list in a structured Markdown format, enabling automated agents to consume + review-set metadata. + tags: [ots] + tests: + - ReviewMark_Elaborate + - id: Template-OTS-ReviewMark-Lint + title: ReviewMark shall validate review definition files and report configuration issues. + justification: | + ReviewMark's --lint command checks the .reviewmark.yaml configuration for structural + and semantic issues, providing early detection of misconfigurations before pipeline + execution. + tags: [ots] + tests: + - ReviewMark_Lint + - id: Template-OTS-ReviewMark-DirectoryOverride + title: ReviewMark shall support overriding the working directory for file operations. + justification: | + ReviewMark's --dir flag allows callers to specify the working directory used when + resolving file paths, enabling the tool to operate correctly from any invocation + context. + tags: [ots] + tests: + - ReviewMark_WorkingDirectoryOverride diff --git a/docs/reqstream/ots/sarifmark.yaml b/docs/reqstream/ots/sarifmark.yaml index fbc97ec..be92a5a 100644 --- a/docs/reqstream/ots/sarifmark.yaml +++ b/docs/reqstream/ots/sarifmark.yaml @@ -19,3 +19,12 @@ sections: tests: - SarifMark_SarifReading - SarifMark_MarkdownReportGeneration + - id: Template-OTS-SarifMark-Enforcement + title: SarifMark shall return a non-zero exit code when SARIF issues are detected. + justification: | + When invoked in enforcement mode, SarifMark exits non-zero if the SARIF input + contains any reported issues. This enables CI pipelines to fail on detected code + quality violations and confirms the enforcement gate is functioning correctly. + tags: [ots] + tests: + - SarifMark_Enforcement diff --git a/docs/reqstream/ots/versionmark.yaml b/docs/reqstream/ots/versionmark.yaml index f18330e..dfd9bc7 100644 --- a/docs/reqstream/ots/versionmark.yaml +++ b/docs/reqstream/ots/versionmark.yaml @@ -19,3 +19,13 @@ sections: tests: - VersionMark_CapturesVersions - VersionMark_GeneratesMarkdownReport + - id: Template-OTS-VersionMark-Lint + title: VersionMark shall validate its configuration file and report errors for invalid input. + justification: | + DemaConsulting.VersionMark provides a lint mode that checks the .versionmark.yaml + configuration for structural and semantic errors. Running lint mode in CI provides early + detection of misconfigurations before capture-and-publish steps execute. + tags: [ots] + tests: + - VersionMark_LintPassesForValidConfig + - VersionMark_LintReportsErrorsForInvalidConfig diff --git a/docs/reqstream/ots/weasyprint.yaml b/docs/reqstream/ots/weasyprint.yaml index abc6f8b..b8ebe6d 100644 --- a/docs/reqstream/ots/weasyprint.yaml +++ b/docs/reqstream/ots/weasyprint.yaml @@ -23,4 +23,5 @@ sections: - WeasyPrint_ReviewPlanPdf - WeasyPrint_ReviewReportPdf - WeasyPrint_DesignPdf + - WeasyPrint_VerificationPdf - WeasyPrint_UserGuidePdf diff --git a/docs/reqstream/ots/xunit.yaml b/docs/reqstream/ots/xunit.yaml index 5a6003e..e98fbb3 100644 --- a/docs/reqstream/ots/xunit.yaml +++ b/docs/reqstream/ots/xunit.yaml @@ -1,20 +1,38 @@ --- -# xUnit v3 OTS Software Requirements +# xUnit OTS Software Requirements # -# Requirements for the xUnit v3 testing framework functionality. +# Requirements for the xUnit testing framework functionality. sections: - title: OTS Software Requirements sections: - - title: xUnit v3 Requirements + - title: xUnit Requirements requirements: - - id: Template-OTS-XUnit - title: xUnit v3 shall produce passing test-result reports for all discovered test methods. + - id: Template-OTS-xUnit-Execute + title: xUnit shall discover and execute unit tests. justification: | xUnit v3 (xunit.v3 and xunit.runner.visualstudio) is the unit-testing framework used - by the project. It discovers and runs all test methods and writes TRX result files that - feed into coverage reporting and requirements traceability. Passing tests confirm the - framework is functioning correctly. + by the project. It discovers and runs all test methods annotated with [Fact] or + [Theory]. Passing tests confirm the framework correctly identifies and executes each + test method. + tags: [ots] + tests: + - Demo_DemoMethod_DefaultPrefix_ReturnsGreeting + - Demo_DemoMethod_CustomPrefix_ReturnsGreeting + - Demo_DemoMethod_NullInput_ThrowsArgumentNullException + - Demo_DemoMethod_EmptyInput_ThrowsArgumentException + - Demo_Constructor_NullPrefix_ThrowsArgumentNullException + - Demo_Constructor_EmptyPrefix_ThrowsArgumentException + - Demo_DefaultPrefix_Read_IsHello + - Demo_Prefix_WithCustomConstruction_ReturnsCustomPrefix + - Demo_DefaultConstructor_WithNoArgs_SetsDefaultPrefix + - id: Template-OTS-xUnit-Report + title: xUnit shall record test results in TRX output files. + justification: | + xUnit v3 with xunit.runner.visualstudio writes TRX result files that feed into + coverage reporting and requirements traceability via ReqStream. The presence of + named test results in the TRX output confirms the reporting pipeline is functioning + correctly. tags: [ots] tests: - Demo_DemoMethod_DefaultPrefix_ReturnsGreeting diff --git a/docs/verification/definition.yaml b/docs/verification/definition.yaml index 0eb70e6..b8b2bff 100644 --- a/docs/verification/definition.yaml +++ b/docs/verification/definition.yaml @@ -7,6 +7,16 @@ input-files: - docs/verification/introduction.md - docs/verification/template-dot-net-library/template-dot-net-library.md - docs/verification/template-dot-net-library/demo.md + - docs/verification/ots/buildmark.md + - docs/verification/ots/fileassert.md + - docs/verification/ots/pandoc.md + - docs/verification/ots/reqstream.md + - docs/verification/ots/reviewmark.md + - docs/verification/ots/sarifmark.md + - docs/verification/ots/sonarmark.md + - docs/verification/ots/versionmark.md + - docs/verification/ots/weasyprint.md + - docs/verification/ots/xunit.md template: template.html table-of-contents: true number-sections: true diff --git a/docs/verification/introduction.md b/docs/verification/introduction.md index ec1fa3c..4dc8f59 100644 --- a/docs/verification/introduction.md +++ b/docs/verification/introduction.md @@ -26,15 +26,26 @@ constituent software items, specifically: - **TemplateDotNetLibrary (System)** — The complete .NET library template system - **Demo (Unit)** — Demonstration greeting class providing example functionality -This verification documentation covers the same software items as the design documentation. It -does not cover test infrastructure, build pipeline configuration, or third-party OTS components. +The following OTS items are also covered: + +- **BuildMark** — build-notes documentation tool +- **FileAssert** — document assertion tool +- **Pandoc** — Markdown-to-HTML conversion tool +- **ReqStream** — requirements traceability tool +- **ReviewMark** — file review enforcement tool +- **SarifMark** — SARIF report conversion tool +- **SonarMark** — SonarCloud quality report tool +- **VersionMark** — tool-version documentation tool +- **WeasyPrint** — HTML-to-PDF conversion tool +- **xUnit** — unit-testing framework + +This verification documentation covers the same software items as the design documentation. Version applicability: This verification design applies to all versions of the Template DotNet Library. The following topics are explicitly excluded from this verification documentation: -- OTS component verification (xUnit, ReqStream, ReviewMark, and other third-party tools) - Build pipeline and CI/CD process testing - Infrastructure and hosting environment testing @@ -45,16 +56,34 @@ Each software item in the structure below has corresponding artifacts in paralle ```text TemplateDotNetLibrary (System) └── Demo (Unit) + +OTS Items +├── BuildMark +├── FileAssert +├── Pandoc +├── ReqStream +├── ReviewMark +├── SarifMark +├── SonarMark +├── VersionMark +├── WeasyPrint +└── xUnit ``` -Each software item has artifacts in these parallel locations: +In-house items have artifacts in these parallel locations: - Requirements: `docs/reqstream/{system}/.../{item}.yaml` (kebab-case) - Design docs: `docs/design/{system}/.../{item}.md` (kebab-case) - Verification design: `docs/verification/{system}/.../{item}.md` (kebab-case) - Source code: `src/{System}/.../{Item}.cs` (PascalCase for C#) - Tests: `test/{System}.Tests/.../{Item}Tests.cs` (PascalCase for C#) -- Review-sets: defined in `.reviewmark.yaml` + +OTS items have parallel artifacts in: + +- Requirements: `docs/reqstream/ots/{ots-name}.yaml` (kebab-case) +- Verification: `docs/verification/ots/{ots-name}.md` (kebab-case) + +Review-sets: defined in `.reviewmark.yaml` ## References diff --git a/docs/verification/ots/buildmark.md b/docs/verification/ots/buildmark.md new file mode 100644 index 0000000..da5e4bf --- /dev/null +++ b/docs/verification/ots/buildmark.md @@ -0,0 +1,72 @@ +# BuildMark Verification + +This document provides the verification evidence for the `BuildMark` OTS software item. + +## Required Functionality + +DemaConsulting.BuildMark queries the GitHub API to capture workflow run details and renders them as +a markdown build-notes document included in the release artifacts. It runs as part of the same CI +pipeline that produces the TRX test results, so a successful pipeline run is evidence that BuildMark +executed without error. + +## Verification Approach + +BuildMark is verified by two complementary layers of evidence. First, the CI pipeline runs +`buildmark --validate --results artifacts/buildmark-self-validation.trx`, which exercises +BuildMark's built-in self-validation suite against mock data and records results for ReqStream. + +Second, the pipeline invokes BuildMark with live GitHub Actions metadata to generate +`docs/build_notes/generated/build_notes.md`. Pandoc then converts this file to HTML; if the +file were absent or malformed, Pandoc would fail. WeasyPrint renders the HTML to a PDF/A +artifact, and FileAssert asserts the PDF exists, has content, and contains expected text +(`WeasyPrint_BuildNotesPdf`). A CI build failure at any step in this chain is evidence that +BuildMark did not produce the required output. + +## Test Scenarios + +### BuildMark_MarkdownReportGeneration + +**Scenario**: A CI pipeline run triggers BuildMark with live GitHub Actions metadata. + +**Expected**: BuildMark exits without error and produces a non-empty markdown build-notes document +in the release artifacts. + +**Requirement coverage**: `Template-OTS-BuildMark`. + +### BuildMark_GitIntegration + +**Scenario**: BuildMark self-validation reads version tags and commits from a mock Git history. + +**Expected**: Exits 0 and correctly reads version tags and commit history. + +**Requirement coverage**: `Template-OTS-BuildMark`. + +### BuildMark_IssueTracking + +**Scenario**: BuildMark self-validation processes mock GitHub issues and pull requests. + +**Expected**: Exits 0 and correctly fetches and processes mock issues and pull requests. + +**Requirement coverage**: `Template-OTS-BuildMark`. + +### BuildMark_KnownIssuesReporting + +**Scenario**: BuildMark self-validation includes open bugs in the generated report when requested. + +**Expected**: Exits 0 and correctly includes known issues. + +**Requirement coverage**: `Template-OTS-BuildMark`. + +### BuildMark_RulesRouting + +**Scenario**: BuildMark self-validation assigns mock items to report sections based on label and +type rules. + +**Expected**: Exits 0 and correctly routes items to the expected sections. + +**Requirement coverage**: `Template-OTS-BuildMark`. + +## Requirements Coverage + +- **`Template-OTS-BuildMark`**: BuildMark_MarkdownReportGeneration, BuildMark_GitIntegration, + BuildMark_IssueTracking, BuildMark_KnownIssuesReporting, BuildMark_RulesRouting diff --git a/docs/verification/ots/fileassert.md b/docs/verification/ots/fileassert.md new file mode 100644 index 0000000..4eda8ff --- /dev/null +++ b/docs/verification/ots/fileassert.md @@ -0,0 +1,58 @@ +# FileAssert Verification + +This document provides the verification evidence for the FileAssert OTS software item. Requirements +for this OTS item are defined in the FileAssert OTS Software Requirements document. + +## Required Functionality + +DemaConsulting.FileAssert validates HTML and PDF documents produced during the build, asserting +that each document exists and contains expected content. It provides OTS evidence for Pandoc +and WeasyPrint and independently confirms file assertion is functioning. Self-validation proves +the tool itself is operational before ReqStream consumes the results. + +## Verification Approach + +FileAssert is verified by two complementary layers of evidence. First, the CI pipeline runs +`fileassert --validate --results artifacts/fileassert-self-validation.trx` after all documents +have been generated, exercising FileAssert's built-in self-validation suite and recording +functional test results for ReqStream. + +Second, FileAssert is used throughout the pipeline to validate every generated document before +ReqStream runs — Build Notes, Code Quality, Review Plan, Review Report, Design, Verification, +and User Guide. If FileAssert were non-functional, these validation steps would fail or produce +incorrect results, causing `reqstream --enforce` to report missing test coverage and fail the +build. A passing CI build therefore constitutes transitive evidence that FileAssert correctly +asserted document content at each stage of the pipeline. + +## Test Scenarios + +### FileAssert_Results + +**Scenario**: FileAssert self-validation exercises the `--results` option, generating TRX test +results containing both passing and failing outcomes. + +**Expected**: Writes a TRX results file with correctly classified pass and fail entries. + +**Requirement coverage**: `Template-OTS-FileAssert`. + +### FileAssert_Exists + +**Scenario**: FileAssert self-validation exercises a test configuration using a glob pattern to +assert file existence. + +**Expected**: Passes when the specified file is present. + +**Requirement coverage**: `Template-OTS-FileAssert`. + +### FileAssert_Contains + +**Scenario**: FileAssert self-validation exercises a test configuration using a `contains` assertion +to verify file content. + +**Expected**: Passes when the specified content is present. + +**Requirement coverage**: `Template-OTS-FileAssert`. + +## Requirements Coverage + +- **`Template-OTS-FileAssert`**: FileAssert_Results, FileAssert_Exists, FileAssert_Contains diff --git a/docs/verification/ots/pandoc.md b/docs/verification/ots/pandoc.md new file mode 100644 index 0000000..ca29411 --- /dev/null +++ b/docs/verification/ots/pandoc.md @@ -0,0 +1,86 @@ +# Pandoc Verification + +This document provides the verification evidence for the `Pandoc` OTS software item. + +## Required Functionality + +DemaConsulting.PandocTool converts Markdown source documents to HTML as part of the documentation +build pipeline. FileAssert validates that each generated HTML file exists, contains a valid HTML +title element, and includes expected document content. Passing FileAssert assertions for each +document type proves Pandoc executed correctly and produced meaningful output. + +## Verification Approach + +Pandoc is verified by self-validation evidence from the CI pipeline. Each scenario is a FileAssert +assertion that runs after Pandoc converts a specific Markdown document to HTML. A passing pipeline +run for all scenarios constitutes evidence that the requirement is satisfied. + +## Test Scenarios + +### Pandoc_BuildNotesHtml + +**Scenario**: FileAssert asserts the build-notes HTML file exists, contains a valid HTML title +element, and includes expected document content. + +**Expected**: FileAssert exits 0 for the build-notes HTML document. + +**Requirement coverage**: `Template-OTS-Pandoc`. + +### Pandoc_CodeQualityHtml + +**Scenario**: FileAssert asserts the code-quality HTML file exists, contains a valid HTML title +element, and includes expected document content. + +**Expected**: FileAssert exits 0 for the code-quality HTML document. + +**Requirement coverage**: `Template-OTS-Pandoc`. + +### Pandoc_ReviewPlanHtml + +**Scenario**: FileAssert asserts the review plan HTML file exists, contains a valid HTML title +element, and includes expected document content. + +**Expected**: FileAssert exits 0 for the review plan HTML document. + +**Requirement coverage**: `Template-OTS-Pandoc`. + +### Pandoc_ReviewReportHtml + +**Scenario**: FileAssert asserts the review report HTML file exists, contains a valid HTML title +element, and includes expected document content. + +**Expected**: FileAssert exits 0 for the review report HTML document. + +**Requirement coverage**: `Template-OTS-Pandoc`. + +### Pandoc_DesignHtml + +**Scenario**: FileAssert asserts the design document HTML file exists, contains a valid HTML title +element, and includes expected document content. + +**Expected**: FileAssert exits 0 for the design document HTML. + +**Requirement coverage**: `Template-OTS-Pandoc`. + +### Pandoc_VerificationHtml + +**Scenario**: FileAssert asserts the verification HTML file exists, contains a valid HTML title +element, and includes expected verification document content. + +**Expected**: FileAssert exits 0 for the verification document. + +**Requirement coverage**: `Template-OTS-Pandoc`. + +### Pandoc_UserGuideHtml + +**Scenario**: FileAssert asserts the user guide HTML file exists, contains a valid HTML title +element, and includes expected document content. + +**Expected**: FileAssert exits 0 for the user guide HTML document. + +**Requirement coverage**: `Template-OTS-Pandoc`. + +## Requirements Coverage + +- **`Template-OTS-Pandoc`**: Pandoc_BuildNotesHtml, Pandoc_CodeQualityHtml, Pandoc_ReviewPlanHtml, + Pandoc_ReviewReportHtml, Pandoc_DesignHtml, Pandoc_VerificationHtml, Pandoc_UserGuideHtml diff --git a/docs/verification/ots/reqstream.md b/docs/verification/ots/reqstream.md new file mode 100644 index 0000000..b2af701 --- /dev/null +++ b/docs/verification/ots/reqstream.md @@ -0,0 +1,85 @@ +# ReqStream Verification + +This document provides the verification evidence for the `ReqStream` OTS software item. + +## Required Functionality + +DemaConsulting.ReqStream processes requirements.yaml and the TRX test-result files to produce a +requirements report, justifications document, and traceability matrix. When run with `--enforce`, it +exits with a non-zero code if any requirement lacks test evidence, making unproven requirements a +build-breaking condition. A successful pipeline run with `--enforce` proves all requirements are +covered and that ReqStream is functioning. + +## Verification Approach + +ReqStream is verified by two complementary layers of evidence. First, the CI pipeline runs +`reqstream --validate --results artifacts/reqstream-self-validation.trx`, which exercises +ReqStream's built-in self-validation suite against internal test requirements and records +results. + +Second, the pipeline invokes `reqstream --enforce` with `requirements.yaml` and all TRX +test-evidence files accumulated during the build. ReqStream generates `requirements.md`, +`justifications.md`, and `trace_matrix.md`. If ReqStream failed or produced no output, the +subsequent Pandoc step would fail, breaking the CI build. Additionally, `--enforce` exits +non-zero if any requirement lacks test evidence, which would also fail the build. A passing +CI build proves ReqStream correctly processed the project's real requirements and found +complete test coverage. + +## Test Scenarios + +### ReqStream_RequirementsProcessing + +**Scenario**: ReqStream self-validation loads and processes a set of requirements YAML files. + +**Expected**: Correctly loads and parses all requirements. + +**Requirement coverage**: `Template-OTS-ReqStream`. + +### ReqStream_TraceMatrix + +**Scenario**: ReqStream self-validation generates a trace matrix from requirements and TRX test +results. + +**Expected**: Produces a correctly linked trace matrix. + +**Requirement coverage**: `Template-OTS-ReqStream`. + +### ReqStream_ReportExport + +**Scenario**: ReqStream self-validation exports a requirements report to a markdown file. + +**Expected**: Produces a correctly formatted requirements report. + +**Requirement coverage**: `Template-OTS-ReqStream`. + +### ReqStream_TagsFiltering + +**Scenario**: ReqStream self-validation filters requirements by tags. + +**Expected**: Returns only requirements matching the specified tags. + +**Requirement coverage**: `Template-OTS-ReqStream`. + +### ReqStream_EnforcementMode + +**Scenario**: ReqStream self-validation exercises enforcement mode where all requirements have test +coverage. + +**Expected**: Exits 0; would exit non-zero if any requirement lacked coverage. + +**Requirement coverage**: `Template-OTS-ReqStream`. + +### ReqStream_Lint + +**Scenario**: ReqStream self-validation exercises lint mode against a requirements file with +deliberate issues. + +**Expected**: Correctly identifies and reports the issues. + +**Requirement coverage**: `Template-OTS-ReqStream-Lint`. + +## Requirements Coverage + +- **`Template-OTS-ReqStream`**: ReqStream_RequirementsProcessing, ReqStream_TraceMatrix, + ReqStream_ReportExport, ReqStream_TagsFiltering, ReqStream_EnforcementMode +- **`Template-OTS-ReqStream-Lint`**: ReqStream_Lint diff --git a/docs/verification/ots/reviewmark.md b/docs/verification/ots/reviewmark.md new file mode 100644 index 0000000..7a2b9a1 --- /dev/null +++ b/docs/verification/ots/reviewmark.md @@ -0,0 +1,100 @@ +# ReviewMark Verification + +This document provides the verification evidence for the ReviewMark OTS software item. Requirements +for this OTS item are defined in the ReviewMark OTS Software Requirements document. + +## Required Functionality + +DemaConsulting.ReviewMark reads the `.reviewmark.yaml` configuration and the review evidence store +to produce a review plan and review report documenting file review coverage and currency. It runs in +the same CI pipeline that produces the TRX test results, so a successful pipeline run is evidence +that ReviewMark executed without error. + +## Verification Approach + +ReviewMark is verified by two complementary layers of evidence. First, the CI pipeline runs +`reviewmark --validate --results artifacts/reviewmark-self-validation.trx`, which exercises +ReviewMark's built-in self-validation suite against test review configurations and records +results for ReqStream. + +Second, the pipeline invokes ReviewMark to generate +`docs/code_review_plan/generated/plan.md` and `docs/code_review_report/generated/report.md`. +Pandoc converts each to HTML; if either file were absent or malformed, Pandoc would fail. +WeasyPrint renders both to PDF and FileAssert asserts their content +(`WeasyPrint_ReviewPlanPdf`, `WeasyPrint_ReviewReportPdf`). A CI build failure at any step is +evidence that ReviewMark did not produce the required review documents. + +## Test Scenarios + +### ReviewMark_ReviewPlanGeneration + +**Scenario**: ReviewMark self-validation uses `--definition` and `--plan` to generate a review plan +from a test configuration. + +**Expected**: Exits 0 and produces a non-empty review plan markdown file. + +**Requirement coverage**: `Template-OTS-ReviewMark`. + +### ReviewMark_ReviewReportGeneration + +**Scenario**: ReviewMark self-validation uses `--definition` and `--report` to generate a review +report from a test configuration and evidence store. + +**Expected**: Exits 0 and produces a non-empty review report. + +**Requirement coverage**: `Template-OTS-ReviewMark`. + +### ReviewMark_IndexScan + +**Scenario**: ReviewMark self-validation uses `--index` to scan PDF evidence files and write an +`index.json` catalogue. + +**Expected**: Exits 0 and produces a correctly structured `index.json`. + +**Requirement coverage**: `Template-OTS-ReviewMark-IndexScan`. + +### ReviewMark_WorkingDirectoryOverride + +**Scenario**: ReviewMark self-validation uses `--dir` to override the working directory for file +operations. + +**Expected**: Exits 0 and resolves paths relative to the specified directory. + +**Requirement coverage**: `Template-OTS-ReviewMark-DirectoryOverride`. + +### ReviewMark_Enforce + +**Scenario**: ReviewMark self-validation uses `--enforce` against a configuration with review +issues. + +**Expected**: Exits with a non-zero exit code when review issues are present. + +**Requirement coverage**: `Template-OTS-ReviewMark-Enforce`. + +### ReviewMark_Elaborate + +**Scenario**: ReviewMark self-validation uses `--elaborate` to print a Markdown elaboration of a +named review set. + +**Expected**: Exits 0 and prints the review-set ID, fingerprint, and file list. + +**Requirement coverage**: `Template-OTS-ReviewMark-Elaborate`. + +### ReviewMark_Lint + +**Scenario**: ReviewMark self-validation uses `--lint` to validate a definition file and report +issues. + +**Expected**: Exits non-zero and outputs at least one reported issue identifying a structural or +semantic error in the test definition that contains known errors. + +**Requirement coverage**: `Template-OTS-ReviewMark-Lint`. + +## Requirements Coverage + +- **`Template-OTS-ReviewMark`**: ReviewMark_ReviewPlanGeneration, ReviewMark_ReviewReportGeneration +- **`Template-OTS-ReviewMark-IndexScan`**: ReviewMark_IndexScan +- **`Template-OTS-ReviewMark-Enforce`**: ReviewMark_Enforce +- **`Template-OTS-ReviewMark-Elaborate`**: ReviewMark_Elaborate +- **`Template-OTS-ReviewMark-Lint`**: ReviewMark_Lint +- **`Template-OTS-ReviewMark-DirectoryOverride`**: ReviewMark_WorkingDirectoryOverride diff --git a/docs/verification/ots/sarifmark.md b/docs/verification/ots/sarifmark.md new file mode 100644 index 0000000..10d3fcc --- /dev/null +++ b/docs/verification/ots/sarifmark.md @@ -0,0 +1,58 @@ +# SarifMark Verification + +This document provides the verification evidence for the SarifMark OTS software item. Requirements +for this OTS item are defined in the SarifMark OTS Software Requirements document. + +## Required Functionality + +DemaConsulting.SarifMark reads the SARIF output produced by CodeQL code scanning and renders it as +a human-readable markdown document included in the release artifacts. It runs in the same CI +pipeline that produces the TRX test results, so a successful pipeline run is evidence that SarifMark +executed without error. + +## Verification Approach + +SarifMark is verified by two complementary layers of evidence. First, the CI pipeline runs +`sarifmark --validate --results artifacts/sarifmark-self-validation.trx`, which exercises +SarifMark's built-in self-validation suite against mock SARIF data and records results for +ReqStream. + +Second, the pipeline invokes SarifMark with the CodeQL SARIF output to generate +`docs/code_quality/generated/codeql-quality.md`. Pandoc converts this file (along with the +SonarMark output) to HTML; if the file were absent or malformed, Pandoc would fail. WeasyPrint +renders the result to PDF and FileAssert asserts the PDF contains expected content +(`WeasyPrint_CodeQualityPdf`). A CI build failure at any step is evidence that SarifMark did +not produce the required output. + +## Test Scenarios + +### SarifMark_SarifReading + +**Scenario**: SarifMark is invoked with a CodeQL SARIF results file as input. + +**Expected**: Exits 0 and successfully reads the SARIF content. + +**Requirement coverage**: `Template-OTS-SarifMark`. + +### SarifMark_MarkdownReportGeneration + +**Scenario**: SarifMark renders the SARIF input as a markdown report included in the release +artifacts. + +**Expected**: Exits 0 and produces a non-empty markdown report. + +**Requirement coverage**: `Template-OTS-SarifMark`. + +### SarifMark_Enforcement + +**Scenario**: SarifMark self-validation processes a SARIF file containing known issues in +enforcement mode. + +**Expected**: Returns a non-zero exit code when issues are found. + +**Requirement coverage**: `Template-OTS-SarifMark-Enforcement`. + +## Requirements Coverage + +- **`Template-OTS-SarifMark`**: SarifMark_SarifReading, SarifMark_MarkdownReportGeneration +- **`Template-OTS-SarifMark-Enforcement`**: SarifMark_Enforcement diff --git a/docs/verification/ots/sonarmark.md b/docs/verification/ots/sonarmark.md new file mode 100644 index 0000000..57155a6 --- /dev/null +++ b/docs/verification/ots/sonarmark.md @@ -0,0 +1,64 @@ +# SonarMark Verification + +This document provides the verification evidence for the SonarMark OTS software item. Requirements +for this OTS item are defined in the SonarMark OTS Software Requirements document. + +## Required Functionality + +DemaConsulting.SonarMark retrieves quality-gate and metrics data from SonarCloud and renders it as +a markdown document included in the release artifacts. It runs in the same CI pipeline that produces +the TRX test results, so a successful pipeline run is evidence that SonarMark executed without +error. + +## Verification Approach + +SonarMark is verified by two complementary layers of evidence. First, the CI pipeline runs +`sonarmark --validate --results artifacts/sonarmark-self-validation.trx`, which exercises +SonarMark's built-in self-validation suite against mock SonarCloud data and records results +for ReqStream. + +Second, the pipeline invokes SonarMark against the live SonarCloud API to generate +`docs/code_quality/generated/sonar-quality.md`. Pandoc converts this file (along with the +SarifMark output) to HTML; if the file were absent or malformed, Pandoc would fail. WeasyPrint +renders the result to PDF and FileAssert asserts the PDF contains expected content +(`WeasyPrint_CodeQualityPdf`). A CI build failure at any step is evidence that SonarMark did +not retrieve and render quality data correctly. + +## Test Scenarios + +### SonarMark_QualityGateRetrieval + +**Scenario**: SonarMark queries the SonarCloud API for quality-gate status. + +**Expected**: Exits 0 and retrieves quality-gate data. + +**Requirement coverage**: `Template-OTS-SonarMark`. + +### SonarMark_IssuesRetrieval + +**Scenario**: SonarMark queries the SonarCloud API for issues. + +**Expected**: Exits 0 and retrieves issues data. + +**Requirement coverage**: `Template-OTS-SonarMark`. + +### SonarMark_HotSpotsRetrieval + +**Scenario**: SonarMark queries the SonarCloud API for hot spots. + +**Expected**: Exits 0 and retrieves hot-spots data. + +**Requirement coverage**: `Template-OTS-SonarMark`. + +### SonarMark_MarkdownReportGeneration + +**Scenario**: SonarMark renders quality-gate, issues, and hot-spots data as a markdown report. + +**Expected**: Exits 0 and produces a non-empty markdown quality report. + +**Requirement coverage**: `Template-OTS-SonarMark`. + +## Requirements Coverage + +- **`Template-OTS-SonarMark`**: SonarMark_QualityGateRetrieval, SonarMark_IssuesRetrieval, + SonarMark_HotSpotsRetrieval, SonarMark_MarkdownReportGeneration diff --git a/docs/verification/ots/versionmark.md b/docs/verification/ots/versionmark.md new file mode 100644 index 0000000..ad10f9d --- /dev/null +++ b/docs/verification/ots/versionmark.md @@ -0,0 +1,65 @@ +# VersionMark Verification + +This document provides the verification evidence for the `VersionMark` OTS software item. + +## Required Functionality + +DemaConsulting.VersionMark reads version metadata for each dotnet tool used in the pipeline and +writes a versions markdown document included in the release artifacts. It runs in the same CI +pipeline that produces the TRX test results, so a successful pipeline run is evidence that +VersionMark executed without error. + +## Verification Approach + +VersionMark is verified by two complementary layers of evidence. First, the CI pipeline runs +`versionmark --validate --results *.trx` in each build job, exercising VersionMark's built-in +self-validation suite and recording results for ReqStream. + +Second, each CI job runs `versionmark --capture` to collect tool-version JSON files, and the +build-docs job runs `versionmark --publish` to produce +`docs/build_notes/generated/versions.md`. This file is included in the Build Notes document +compiled by Pandoc. If VersionMark failed to produce the versions document, the Build Notes +compilation would be incomplete and the subsequent Pandoc step would fail. A CI build failure +at any step is evidence that VersionMark did not execute correctly. + +## Test Scenarios + +### VersionMark_CapturesVersions + +**Scenario**: VersionMark reads version metadata for each dotnet tool defined in the pipeline. + +**Expected**: Exits 0 and captures version data for every tool. + +**Requirement coverage**: `Template-OTS-VersionMark`. + +### VersionMark_GeneratesMarkdownReport + +**Scenario**: VersionMark writes a versions markdown document to the release artifacts. + +**Expected**: Exits 0 and produces a non-empty versions markdown file. + +**Requirement coverage**: `Template-OTS-VersionMark`. + +### VersionMark_LintPassesForValidConfig + +**Scenario**: VersionMark self-validation exercises lint mode against a valid `.versionmark.yaml` +config. + +**Expected**: Exits 0 and reports no errors. + +**Requirement coverage**: `Template-OTS-VersionMark-Lint`. + +### VersionMark_LintReportsErrorsForInvalidConfig + +**Scenario**: VersionMark self-validation exercises lint mode against a malformed config with +deliberate errors. + +**Expected**: Correctly identifies and reports the configuration errors. + +**Requirement coverage**: `Template-OTS-VersionMark-Lint`. + +## Requirements Coverage + +- **`Template-OTS-VersionMark`**: VersionMark_CapturesVersions, VersionMark_GeneratesMarkdownReport +- **`Template-OTS-VersionMark-Lint`**: VersionMark_LintPassesForValidConfig, + VersionMark_LintReportsErrorsForInvalidConfig diff --git a/docs/verification/ots/weasyprint.md b/docs/verification/ots/weasyprint.md new file mode 100644 index 0000000..a33a892 --- /dev/null +++ b/docs/verification/ots/weasyprint.md @@ -0,0 +1,88 @@ +# WeasyPrint Verification + +This document provides the verification evidence for the WeasyPrint OTS software item. Requirements +for this OTS item are defined in the WeasyPrint OTS Software Requirements document. + +## Required Functionality + +DemaConsulting.WeasyPrintTool converts HTML documents to PDF as part of the documentation build +pipeline. FileAssert validates that each generated PDF file exists, contains at least one page, and +includes expected document content in the rendered text. Passing FileAssert assertions for each +document type proves WeasyPrint executed correctly and produced meaningful output. + +## Verification Approach + +WeasyPrint is verified by self-validation evidence from the CI pipeline. Each scenario is a +FileAssert assertion that runs after WeasyPrint converts a specific HTML document to PDF. A passing +pipeline run for all scenarios constitutes evidence that the requirement is satisfied. + +## Test Scenarios + +### WeasyPrint_BuildNotesPdf + +**Scenario**: FileAssert asserts the build-notes PDF file exists, contains at least one page, and +includes expected document content. + +**Expected**: FileAssert exits 0 for the build-notes PDF document. + +**Requirement coverage**: `Template-OTS-WeasyPrint`. + +### WeasyPrint_CodeQualityPdf + +**Scenario**: FileAssert asserts the code-quality PDF file exists, contains at least one page, and +includes expected document content. + +**Expected**: FileAssert exits 0 for the code-quality PDF document. + +**Requirement coverage**: `Template-OTS-WeasyPrint`. + +### WeasyPrint_ReviewPlanPdf + +**Scenario**: FileAssert asserts the review plan PDF file exists, contains at least one page, and +includes expected document content. + +**Expected**: FileAssert exits 0 for the review plan PDF document. + +**Requirement coverage**: `Template-OTS-WeasyPrint`. + +### WeasyPrint_ReviewReportPdf + +**Scenario**: FileAssert asserts the review report PDF file exists, contains at least one page, and +includes expected document content. + +**Expected**: FileAssert exits 0 for the review report PDF document. + +**Requirement coverage**: `Template-OTS-WeasyPrint`. + +### WeasyPrint_DesignPdf + +**Scenario**: FileAssert asserts the design document PDF file exists, contains at least one page, +and includes expected document content. + +**Expected**: FileAssert exits 0 for the design document PDF. + +**Requirement coverage**: `Template-OTS-WeasyPrint`. + +### WeasyPrint_VerificationPdf + +**Scenario**: FileAssert asserts the verification PDF file exists, contains at least one page, and +includes expected verification document content. + +**Expected**: FileAssert exits 0 for the verification PDF. + +**Requirement coverage**: `Template-OTS-WeasyPrint`. + +### WeasyPrint_UserGuidePdf + +**Scenario**: FileAssert asserts the user guide PDF file exists, contains at least one page, and +includes expected document content. + +**Expected**: FileAssert exits 0 for the user guide PDF document. + +**Requirement coverage**: `Template-OTS-WeasyPrint`. + +## Requirements Coverage + +- **`Template-OTS-WeasyPrint`**: WeasyPrint_BuildNotesPdf, WeasyPrint_CodeQualityPdf, + WeasyPrint_ReviewPlanPdf, WeasyPrint_ReviewReportPdf, WeasyPrint_DesignPdf, + WeasyPrint_VerificationPdf, WeasyPrint_UserGuidePdf diff --git a/docs/verification/ots/xunit.md b/docs/verification/ots/xunit.md new file mode 100644 index 0000000..a78dcbf --- /dev/null +++ b/docs/verification/ots/xunit.md @@ -0,0 +1,119 @@ +# xUnit Verification + +This document provides the verification evidence for the xUnit OTS software item. Requirements +for this OTS item are defined in the xUnit OTS Software Requirements document. + +## Required Functionality + +xUnit v3 (xunit.v3 and xunit.runner.visualstudio) is the unit-testing framework used by the +project. It discovers and runs all test methods and writes TRX result files that feed into coverage +reporting and requirements traceability. Passing tests confirm the framework is functioning +correctly. + +## Verification Approach + +xUnit is verified by self-validation evidence from the CI pipeline. Each scenario names a specific +test method that xUnit must discover, execute, and record in a TRX result file. A passing pipeline +run for all scenarios constitutes evidence that both requirements are satisfied. + +## Test Scenarios + +### Demo_DemoMethod_DefaultPrefix_ReturnsGreeting + +**Scenario**: xUnit discovers and runs this test; the test verifies that DemoMethod returns the +expected greeting using the default prefix. + +**Expected**: xUnit executes the test, the test passes, and the result appears in the TRX output. + +**Requirement coverage**: `Template-OTS-xUnit-Execute`, `Template-OTS-xUnit-Report`. + +### Demo_DemoMethod_CustomPrefix_ReturnsGreeting + +**Scenario**: xUnit discovers and runs this test; the test verifies that DemoMethod returns the +expected greeting using a custom prefix. + +**Expected**: xUnit executes the test, the test passes, and the result appears in the TRX output. + +**Requirement coverage**: `Template-OTS-xUnit-Execute`, `Template-OTS-xUnit-Report`. + +### Demo_DemoMethod_NullInput_ThrowsArgumentNullException + +**Scenario**: xUnit discovers and runs this test; the test verifies that DemoMethod rejects a null +argument with ArgumentNullException. + +**Expected**: xUnit executes the test, the test passes, and the result appears in the TRX output. + +**Requirement coverage**: `Template-OTS-xUnit-Execute`, `Template-OTS-xUnit-Report`. + +### Demo_DemoMethod_EmptyInput_ThrowsArgumentException + +**Scenario**: xUnit discovers and runs this test; the test verifies that DemoMethod rejects an +empty string argument with ArgumentException. + +**Expected**: xUnit executes the test, the test passes, and the result appears in the TRX output. + +**Requirement coverage**: `Template-OTS-xUnit-Execute`, `Template-OTS-xUnit-Report`. + +### Demo_Constructor_NullPrefix_ThrowsArgumentNullException + +**Scenario**: xUnit discovers and runs this test; the test verifies that the constructor rejects a +null prefix with ArgumentNullException. + +**Expected**: xUnit executes the test, the test passes, and the result appears in the TRX output. + +**Requirement coverage**: `Template-OTS-xUnit-Execute`, `Template-OTS-xUnit-Report`. + +### Demo_Constructor_EmptyPrefix_ThrowsArgumentException + +**Scenario**: xUnit discovers and runs this test; the test verifies that the constructor rejects an +empty string prefix with ArgumentException. + +**Expected**: xUnit executes the test, the test passes, and the result appears in the TRX output. + +**Requirement coverage**: `Template-OTS-xUnit-Execute`, `Template-OTS-xUnit-Report`. + +### Demo_DefaultPrefix_Read_IsHello + +**Scenario**: xUnit discovers and runs this test; the test verifies that the DefaultPrefix constant +has the value "Hello". + +**Expected**: xUnit executes the test, the test passes, and the result appears in the TRX output. + +**Requirement coverage**: `Template-OTS-xUnit-Execute`, `Template-OTS-xUnit-Report`. + +### Demo_Prefix_WithCustomConstruction_ReturnsCustomPrefix + +**Scenario**: xUnit discovers and runs this test; the test verifies that the Prefix property +returns the value supplied at construction. + +**Expected**: xUnit executes the test, the test passes, and the result appears in the TRX output. + +**Requirement coverage**: `Template-OTS-xUnit-Execute`, `Template-OTS-xUnit-Report`. + +### Demo_DefaultConstructor_WithNoArgs_SetsDefaultPrefix + +**Scenario**: xUnit discovers and runs this test; the test verifies that the default constructor +sets Prefix to the DefaultPrefix constant. + +**Expected**: xUnit executes the test, the test passes, and the result appears in the TRX output. + +**Requirement coverage**: `Template-OTS-xUnit-Execute`, `Template-OTS-xUnit-Report`. + +## Requirements Coverage + +- **`Template-OTS-xUnit-Execute`**: Demo_DemoMethod_DefaultPrefix_ReturnsGreeting, + Demo_DemoMethod_CustomPrefix_ReturnsGreeting, + Demo_DemoMethod_NullInput_ThrowsArgumentNullException, + Demo_DemoMethod_EmptyInput_ThrowsArgumentException, + Demo_Constructor_NullPrefix_ThrowsArgumentNullException, + Demo_Constructor_EmptyPrefix_ThrowsArgumentException, Demo_DefaultPrefix_Read_IsHello, + Demo_Prefix_WithCustomConstruction_ReturnsCustomPrefix, + Demo_DefaultConstructor_WithNoArgs_SetsDefaultPrefix +- **`Template-OTS-xUnit-Report`**: Demo_DemoMethod_DefaultPrefix_ReturnsGreeting, + Demo_DemoMethod_CustomPrefix_ReturnsGreeting, + Demo_DemoMethod_NullInput_ThrowsArgumentNullException, + Demo_DemoMethod_EmptyInput_ThrowsArgumentException, + Demo_Constructor_NullPrefix_ThrowsArgumentNullException, + Demo_Constructor_EmptyPrefix_ThrowsArgumentException, Demo_DefaultPrefix_Read_IsHello, + Demo_Prefix_WithCustomConstruction_ReturnsCustomPrefix, + Demo_DefaultConstructor_WithNoArgs_SetsDefaultPrefix diff --git a/src/DemaConsulting.TemplateDotNetLibrary/Demo.cs b/src/DemaConsulting.TemplateDotNetLibrary/Demo.cs index 5575027..4ddf13f 100644 --- a/src/DemaConsulting.TemplateDotNetLibrary/Demo.cs +++ b/src/DemaConsulting.TemplateDotNetLibrary/Demo.cs @@ -43,6 +43,12 @@ public Demo() /// /// Thrown when is an empty string. /// + /// + /// Validation is performed at construction time so that a misconfigured instance is + /// refused immediately rather than silently producing malformed output on the first + /// method call. This is the canonical constructor; delegates + /// to this overload with . + /// public Demo(string prefix) { // Validate that the prefix is non-null and non-empty before storing it @@ -53,6 +59,10 @@ public Demo(string prefix) /// /// Gets the greeting prefix used by this instance. /// + /// + /// Exposed so that callers can inspect the configured prefix without maintaining an + /// independent copy of the value passed at construction time. + /// public string Prefix => _prefix; ///