Skip to content

docs: redesign walkthrough pipeline map and clarify execution boundaries - #156

Merged
muratkeremozcan merged 4 commits into
mainfrom
docs/walkthrough-pipeline-redesign
Sep 22, 2026
Merged

muratkeremozcan merged 4 commits into
mainfrom
docs/walkthrough-pipeline-redesign

Conversation

@muratkeremozcan

Copy link
Copy Markdown
Collaborator

Summary

This PR redesigns docs/how-to/author-behavioral-contracts.md around a single, canonical numbered pipeline map that maps directly to the seven numbered walkthrough sections:

  1. Unified Pipeline Map:

    • Replaced multiple competing conceptual diagrams with ONE canonical numbered pipeline map that spans Steps 1 through 7.
    • Visibly distinguishes executed CLI stages (compile, seal, preflight, score) from read-only inspection sections (Steps 2, 3, 7).
  2. Explicit Execution Boundaries & Skipped Gaps:

    • Explicitly documented pipeline ownership: eval-quality CLI owns deterministic offline transformation/reduction, while the caller's evaluation harness owns dynamic environment and evaluator execution.
    • Defined the two execution gaps in the tutorial where prepared fixtures substitute for dynamic execution:
      • Preflight probing gap (before Step 5): The CLI command issues zero network requests and executes preflightFromObservations over prepared observations.json and probes.json. Explained that programmatic use with the TypeScript SDK can drive a live EnvironmentProbePort via runPreflight.
      • Evaluator & SUT execution gap (before Step 6): Explains that neither the defective Notes API nor an LLM evaluator is executed live; score replays the prepared sealed-run-record.json fixture.
  3. Section Headings & Descriptions:

    • Renamed Step 5 from Preflight the environment to Reduce the preflight observations, clearly explaining that this step reduces prepared observations to a PreflightVerdict.
    • Renamed Step 6 to Score the evaluation record and clarified that it evaluates the prepared run record against the probe.
    • Updated the artifact map table to clearly indicate which artifacts are minted in this exercise vs replayed from committed inputs.
    • Updated the final summary flow under Key Takeaways to mirror the unified numbered pipeline.

Verification

  • npm run docs:validate-links: 0 broken links.
  • node scripts/check-docs.mjs: All files pass.
  • node scripts/gates-cli.ts doc-invocations: 0 failures (54 run faithfully, 14 compared).
  • node scripts/gates-cli.ts doc-counts: 42 numerals across 11 files, 0 disagreements.
  • node scripts/gates-cli.ts doc-claims: 0 disagreements.
  • vitest run tests/architecture/doc-gates.test.ts: 47/47 passing.
  • node tools/build-docs.mjs: Clean build (17 pages generated).

@coderabbitai

coderabbitai Bot commented Sep 22, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 58 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: bmad-code-org/bmad-eval-quality/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 20931140-104c-4a0d-93d1-d59a36e4aed7

📥 Commits

Reviewing files that changed from the base of the PR and between fc32a3e and a15b56d.

📒 Files selected for processing (1)
  • docs/how-to/author-behavioral-contracts.md

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@muratkeremozcan
muratkeremozcan force-pushed the docs/walkthrough-pipeline-redesign branch from 18183b7 to f1acc9d Compare September 22, 2026 15:39
@muratkeremozcan
muratkeremozcan changed the base branch from main to docs/retire-what-ships-and-clarify-contract-strength September 22, 2026 15:39
@muratkeremozcan
muratkeremozcan force-pushed the docs/walkthrough-pipeline-redesign branch from f1acc9d to bbfef55 Compare September 22, 2026 15:55
@muratkeremozcan
muratkeremozcan added this pull request to stack #157 September 22, 2026 15:59
Base automatically changed from docs/retire-what-ships-and-clarify-contract-strength to main September 22, 2026 16:01
- Replace competing conceptual diagrams with ONE canonical numbered pipeline map matching sections 1 through 7.
- Visibly distinguish executed CLI stages from read-only inspection sections.
- Add explicit callouts for caller/harness execution boundaries and the two skipped execution gaps (preflight active probing and SUT/evaluator execution).
- Rename Step 5 to 'Reduce the preflight observations' and clarify that the CLI command executes preflightFromObservations over prepared fixtures.
- Rename Step 6 to 'Score the evaluation record' and make explicit that score evaluates prepared evidence from a prior run.
- Update the artifact map to distinguish newly minted artifacts from replayed prepared fixtures.
- Update summary flow in key takeaways to mirror the numbered pipeline.
@muratkeremozcan
muratkeremozcan force-pushed the docs/walkthrough-pipeline-redesign branch from bbfef55 to a15b56d Compare September 22, 2026 16:01
@muratkeremozcan
muratkeremozcan merged commit 467e3a3 into main Sep 22, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant