Develop tarun

[Provar-tarunprashar]

fix(mcp): schema-driven validation, provar_step_schema tool, bundled examples, and generation-gap guidance

Improves Provar test-case generation grounding and validation so a Salesforce
UI test case can no longer validate as "perfect" while being unrunnable, and so
the model is steered to the correct step shapes at generation time.

Issues found

1. Local (no-auth) validation only checked required arguments for UiAssert, so a
   test case missing required args on UiConnect/UiWithScreen/UiDoAction/etc.
   still scored 100/100 (validity + quality) — a misleading "verified" signal.
2. The bundled step schema (provar_test_step_schema.json) had wrong UI
   required-args: phantom screenName (UiWithScreen), formLocator (UiFill),
   tableLocator/rowSelector (UiWithRow), action (UiHandleAlert), and a UiAssert
   required set that contradicted the reference doc and the corpus rule.
3. Tool descriptions told the model to fall back to reading the
   provar://docs/step-reference MCP resource, which a tools-only client cannot
   read — a dead-end fallback.
4. examples_retrieve returned count:0 with no usable fallback when no API key was
   configured, leaving first-test-case generation with zero grounding.
5. Second round of gaps (verified against docs/PROVAR_TEST_STEP_REFERENCE.md):
   • Salesforce UI tests generated with UiConnect instead of ApexConnect.
   • Dangling connection references (downstream uiConnectionName did not match
     the connect step's resultName).
   • Invalid interaction name (interaction=click) instead of action/set/file.
   • UiAssert field assertions emitted as flat instead of the
     nested structure.
6. Root cause behind CI Execution using github actions #5: the local structural rules only ran on TOP-LEVEL
   apiCalls; every real UI action/assert lives nested under UiWithScreen
   substeps, so those rules never fired on real UI steps. Net effect: the broken
   sample scored 100 while the correct sample scored lower (inverted signal).

Solution added

• Made local validation schema-driven and recursive: a new STEP-REQUIRED-ARGS-001
  rule derives required args per apiId from the (corrected) schema and applies to
  ALL step types; per-call structural rules now recurse into nested substeps.
• Corrected the schema's UI required-args against the authoritative reference doc.
• Added provar_step_schema tool (the callable form of the schema resource) and
  repointed the "read the resource" fallbacks to it.
• Added an offline bundled-examples fallback for examples_retrieve (no/invalid
  key), guarded by a test that re-validates every bundled example.
• Added rules to catch the 2nd-round gaps: UI-INTERACTION-002 (bad interaction
  name), extended UI-ASSERT-STRUCTURE-001 (namedValues shape), SF-CONNECT-TYPE-001
  (UiConnect w/o ApexConnect on a SF UI test), CONNECT-REF-CONSISTENCY-001
  (dangling connection reference).
• Enriched the schema guidance (single source of truth surfaced by
  provar_step_schema) so generation is steered right: pattern opens with
  ApexConnect, interaction vocabulary (action/set/file, never click), nested
  uiFieldAssertion requirement, connection resultName/reference consistency.

Changes by file (what & why)

• src/mcp/rules/provar_test_step_schema.json — corrected phantom UI required-args
  to match the reference doc; enriched validation_rules with connect-selection,
  interaction-vocabulary and nested-assert guidance; fixed the ui_interaction_pattern
  to open with ApexConnect (was UiConnect). This is the single KB the validator,
  generator guidance, and provar_step_schema all read.
• src/mcp/tools/bestPracticesEngine.ts — added schema-driven STEP-REQUIRED-ARGS-001
  (major) that reads required_arguments from the schema for every apiId (excludes
  UiAssert + NitroX variants, which have dedicated rules).
• src/mcp/rules/provar_best_practices_rules.json — registered STEP-REQUIRED-ARGS-001.
• src/mcp/tools/testCaseValidate.ts — made per-call validation recurse into nested
  substeps; added UI-INTERACTION-002, SF-CONNECT-TYPE-001, CONNECT-REF-CONSISTENCY-001;
  extended UI-ASSERT-STRUCTURE-001 to catch the namedValues shape; repointed the
  step-reference fallback to provar_step_schema.
• src/mcp/rules/provar_layer1_rules.json — cataloged the new Layer-1 rule ids
  (drift-guard source of truth).
• src/mcp/tools/stepSchemaTools.ts (new) — provar_step_schema tool: returns a
  step's required/optional args by api_id, a category listing, or the full index.
• src/mcp/server.ts — import + register provar_step_schema (authoring group).
• src/mcp/examples/bundledExamples.ts (new) — a known-correct, validated
  create+verify Account example + selection helper for the offline fallback.
• src/mcp/tools/qualityHubApiTools.ts — examples_retrieve returns bundled examples
  (source:"bundled") on no-key/invalid-key; repointed the fallback hint.
• src/mcp/tools/testCaseGenerate.ts / testCaseStepTools.ts — repointed the
  "read the resource" fallbacks to the provar_step_schema tool.
• scripts/mcp-smoke.cjs — added a provar_step_schema smoke call.
• docs/mcp.md, docs/VALIDATION_RULE_REGISTRY.md, README.md — documented the new
  tool, rules, bundled-example behavior; regenerated the rule registry; tool
  count 42 -> 43.
• test/fixtures/account-ui-gaps-bad.testcase, account-ui-gaps-good.testcase (new)
  — the broken vs corrected samples, wired in as dogfood fixtures.
• test/unit/mcp/*.test.ts — new/updated tests for every change (see Testing).

Testing done

• Full repo suite: 1535 passing, 0 failing (fresh, no wireit cache).
• tsc --noEmit clean; eslint clean on all changed files.
• Dogfood: account-ui-gaps-bad fires all four gap rules and is_valid=false;
  account-ui-gaps-good is clean of the gap rules and is_valid=true.
• Drift guards: every bundled example re-validates to quality_score 100 / zero
  violations; schema guidance is asserted (pattern opens with ApexConnect,
  interaction vocab lists action/set and forbids click, UiAssert requires
  uiFieldAssertion).
• End-to-end via the local MCP server (provar_step_schema returns correct args;
  examples_retrieve returns bundled examples on no key). Generation manually
  confirmed to now emit ApexConnect + action/set + nested assert.
• Note: scripts/mcp-smoke.cjs targets the globally-installed (published) plugin,
  not the local build, so it validates these changes only after publish/CI.

Codex review summary (working tree vs origin/main)

• [P1] "Imported modules not in the commit" — the new files (stepSchemaTools.ts,
  bundledExamples.ts) were untracked at review time; they ARE included in this
  commit, so a clean checkout builds. Resolved.
• [P2] provar_step_schema is only registered in the authoring tool group, so a
  PROVAR_MCP_TOOLS=validation session (where validate errors now tell the user to
  call provar_step_schema) would not have it available. Known follow-up: also
  register it for validation-only sessions (or keep a resource fallback in that
  guidance). Not addressed in this commit.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com