Bug Report
Summary
The Kane CLI shareable report on Test Manager shows a passed status even though the actual test objective was never completed. The report only reflects that an intermediate step (login) succeeded, not the final goal of the prompt.
Steps to Reproduce
- Run
kane-cli run "<objective with multiple steps>" --agent where the objective requires login + a subsequent action
- The agent successfully logs in but hits a blocker before completing the final objective (e.g., a paywall)
- Open the generated
test_url report in Test Manager
Actual Behavior
- The report shows the test as passed
- The final objective/assertion (e.g., "click Upgrade Now and assert billing URL contains 'product_group_key=kane-ai-group'") is not reflected in the report
- The report appears to mark the run as passed based on partial step completion (login succeeded) without evaluating whether the full prompt goal was achieved
Expected Behavior
- The report should accurately reflect the final result of the complete prompt objective
- If the agent could not complete the main goal (even if intermediate steps passed), the report status should be failed or incomplete
- The full prompt/objective text should be visible in the report so reviewers can understand what was being tested
Run Details
Environment
- kane-cli version: 0.3.4
- Auth: OAuth, prod environment
- OS: macOS Darwin 25.3.0
Impact
This is a trust/correctness issue — developers and reviewers using the shareable report URL to verify a PR (e.g., as a CI gate) will see a misleading "passed" result when the actual test goal was never verified. The report cannot be relied upon as evidence that the change works correctly.
Bug Report
Summary
The Kane CLI shareable report on Test Manager shows a passed status even though the actual test objective was never completed. The report only reflects that an intermediate step (login) succeeded, not the final goal of the prompt.
Steps to Reproduce
kane-cli run "<objective with multiple steps>" --agentwhere the objective requires login + a subsequent actiontest_urlreport in Test ManagerActual Behavior
Expected Behavior
Run Details
run_endstatus:failed(reason:blocker.paywall)product_group_key=kane-ai-groupEnvironment
Impact
This is a trust/correctness issue — developers and reviewers using the shareable report URL to verify a PR (e.g., as a CI gate) will see a misleading "passed" result when the actual test goal was never verified. The report cannot be relied upon as evidence that the change works correctly.