Summary
Build a static documentation site for Smith, deployed to GitHub Pages from /docs.
Motivation
Smith lacks user-facing documentation. Developers need a clear path from zero to productive — quick start, concept guides, a complete reference, and a few running examples.
Scope
| Page |
Description |
| Overview |
Motivation and architecture of Smith |
| Quick Start |
Installation instructions & configuration. Running example using the HR agent (RagChatbot_MCPServer) from clone to first passing scorecard |
| Policy Creation |
Generate Rego policy from natural-language rules |
| Test Case Generation |
The full test generation pipeline: decompose → variable extraction → case generation → case evaluation → case translation. Red teaming options, configuring promptfooconfig.yaml with contexts and providers. |
| Policy Testing |
Running the OPA server, testing, and reading scorecard results |
| Cross Validation |
Using LLM-verify failed test cases against the policy, reviewing and correcting the generated policy based on test cases. |
| Policy Patching |
The refinement workflow: red_suggestion (clustering of failures), interpreting cluster feedback, and applying narrowly-scoped policy patches |
| Policy Linting |
Regal lint/format fixes, duplication_suggestion for graph + LLM redundancy detection, and applying cleanup |
| Examples |
Walkthroughs for each bundled target agent: HR Agent (RagChatbot_MCPServer), Call-for-Papers (call-for-papers-mcp), Car Price (car-price-mcp-main). Each covers agent setup, guidance authoring, running the full pipeline, and interpreting results |
| Environment Variables |
A complete reference of all configurable .env variables — model settings, path settings, target agent configuration, test generation parameters, clustering/refinement tuning, and promptfoo/ARES integration options |
| CLI Reference |
A complete reference of all smith --flag <stage> commands, their inputs, outputs, required environment variables, and usage examples |
Technical Approach
- Theme: Hugo Book (git submodule)
- Deployment: GitHub Actions → GitHub Pages
- Style: Show-don't-tell, progressive disclosure, realistic examples.
Summary
Build a static documentation site for Smith, deployed to GitHub Pages from
/docs.Motivation
Smith lacks user-facing documentation. Developers need a clear path from zero to productive — quick start, concept guides, a complete reference, and a few running examples.
Scope
RagChatbot_MCPServer) from clone to first passing scorecardpromptfooconfig.yamlwith contexts and providers.red_suggestion(clustering of failures), interpreting cluster feedback, and applying narrowly-scoped policy patchesduplication_suggestionfor graph + LLM redundancy detection, and applying cleanupRagChatbot_MCPServer), Call-for-Papers (call-for-papers-mcp), Car Price (car-price-mcp-main). Each covers agent setup, guidance authoring, running the full pipeline, and interpreting results.envvariables — model settings, path settings, target agent configuration, test generation parameters, clustering/refinement tuning, and promptfoo/ARES integration optionssmith --flag <stage>commands, their inputs, outputs, required environment variables, and usage examplesTechnical Approach