feat(qa): CLI E2E harness tool for terminal applications#415
Conversation
QA agents can E2E-test Electron/web apps (Electron MCP, Puppeteer) but had no way to drive terminal applications. New run_cli_session MCP tool: - Spawns the command in a pseudo-terminal on POSIX (prompts, colors, isatty() behave like a real terminal); plain pipes on Windows - Feeds scripted stdin lines, captures combined output (capped), enforces a timeout, reports found/missing expected substrings - Commands validated against the project security allowlist and always executed inside the project directory - Available to qa_reviewer and qa_fixer Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Oleg Miagkov <mrobenner@gmail.com>
|
Warning Review limit reached
Next review available in: 48 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughChangesCLI session harness
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant QAAgent
participant run_cli_session
participant SecurityAllowlist
participant CLIApplication
QAAgent->>run_cli_session: submit command, inputs, expectations
run_cli_session->>SecurityAllowlist: validate command
run_cli_session->>CLIApplication: run session in project directory
CLIApplication-->>run_cli_session: return exit code and captured output
run_cli_session-->>QAAgent: return verification report
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
shlex.split(posix=False) keeps surrounding quotes, so quoted arguments reached Windows processes with literal quote characters. CreateProcess parses the command line itself; pass the string through unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Oleg Miagkov <mrobenner@gmail.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/backend/core/cli_session.py`:
- Around line 157-161: Update the finally cleanup block in the CLI session flow
so failure closing master_fd cannot prevent process cleanup. Make
os.close(master_fd), the proc.kill() check, and proc.wait() execute
independently, ensuring proc is terminated and reaped even when closing the file
descriptor raises OSError.
- Around line 146-148: Update the write handling around the event loop’s
pending-input logic to check whether master_fd is writable before calling
os.write, using the existing select/poll mechanism or a nonblocking descriptor.
Only pop the pending item and advance next_write after a successful write, while
allowing the loop’s timeout and deadline checks to continue when the pty buffer
is full.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 029004a7-3dd5-449e-814b-df33880d7f43
📒 Files selected for processing (6)
apps/backend/agents/tools_pkg/models.pyapps/backend/agents/tools_pkg/registry.pyapps/backend/agents/tools_pkg/tools/__init__.pyapps/backend/agents/tools_pkg/tools/cli_harness.pyapps/backend/core/cli_session.pytests/test_cli_session.py
Address review comments: - os.write on a full pty buffer could block forever when the child never reads stdin, bypassing the timeout. Master fd is now non-blocking with a write buffer (_flush_input handles partial writes and EAGAIN); BlockingIOError on read treated as no-data, not EOF. - finally block steps are now independent: a failing os.close can no longer skip proc.kill()/wait() and leak a zombie process. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Oleg Miagkov <mrobenner@gmail.com>
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # Independent cleanup steps: a failure in one must not skip the rest | ||
| try: | ||
| os.close(master_fd) | ||
| except OSError: |
2 participants
Summary
Final roadmap phase (5): verification beyond web/Electron, starting with the cheapest high-coverage slice — terminal applications (most C/C++/Go/Rust deliverables are CLIs).
QA agents can already drive Electron apps (Electron MCP) and web apps (Puppeteer), but had no way to run the thing they just built when it's a CLI. New
run_cli_sessionauto-claude MCP tool:isatty(), prompts, and colored output behave as for a real user; plain-pipe fallback on Windows (no stdlib pty)validate_commandallowlist as Bash, always runs inside the project directory, output capped at 100KB, timeout clamped to 300sqa_reviewerandqa_fixertool permissions only (not coder)Split:
core/cli_session.py(pure runner, testable) +agents/tools_pkg/tools/cli_harness.py(MCP wrapper + security gate).Testing
tests/test_cli_session.py: 10 tests — output/exit-code capture, stderr, scripted stdin against a real prompting script, timeout kill, output cap, cwd, empty command, and QA-only registration.tests/test_agent_configs.py+test_agent_architecture.py: 46 passed. Live registration check:create_all_tools()exposesrun_cli_session. Ruff check + format clean.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Tests