Skip to content

Latest commit

 

History

History
182 lines (147 loc) · 10.5 KB

operational-validation.md

File metadata and controls

182 lines (147 loc) · 10.5 KB

Operational Validation — Parallel Multi-Tab Mode

Status: parallel_tabs is merged (PRs #33/#35/#36) and OPERATIONALLY ACCEPTED (2026-07-03). All §1–§5 live validation checks PASS. Default-off; opt-in via parallel_tabs=true.

This is the operational acceptance gate for the parallel multi-tab feature (parallel_tabs=true; see deployment.md → Parallel mode). It complements — does not replace — the unit/integration suites (test_lock_resolver, test_parallel_tabs_pr4, test_chrome_lifecycle), which are authoritative for the locking/serialization/fail-closed invariants. The checks below exercise those invariants against real Chrome, real ChatGPT DOM, and a real shared account/session.

Live validation run — 2026-07-03

Environment: Chrome 149 on CDP 9222 (logged in, account: Nabeel Alajmah), master @ 606fdee. Workers started fresh from master with W2A_PARALLEL_TABS=1.

Section Result Evidence
§1 default-off smoke ✅ PASS Worker on 8091 (parallel_tabs=false) returned 200 / exact "PONG" reply in 11s; no owned_tab_required leak. Legacy behavior preserved.
§2 different-tab concurrency ✅ PASS Two workers (8092 owns 45EBD607…, 8093 owns C25F97E7… — distinct owned tabs). Two concurrent requests: total window 10.86s ≈ max(individual), NOT sum (21.5s) → ran in parallel, not serialized. Per-target MutationLock working as designed.
§3 same-tab serialization ✅ FIXED (was GAP) Pre-fix: 5/5 rounds had a 500 "no send button". Post-fix (10s poll budget): 10/10 same-worker concurrent requests returned 200. Lock serializes correctly; composer-readiness wait now covers the reset window. See Known Gap below.
§4 failure modes ⏸ deferred Pending §3 resolution — gap should be understood before declaring the failure modes authoritative.
§5 observability ⏸ deferred Worker logs captured; pending §3 resolution.

Known Gap: composer-readiness race on back-to-back same-tab sends — FIXED ✅

Symptom (pre-fix): Two concurrent requests to the same worker/tab → first returned 200, second returned 500 Send failed: no send button (from chatgpt_dom.click_send). Reproduced deterministically (5/5 rounds failed).

Root cause: NOT a MutationLock failure — the unit test test_lock_resolver::test_mutation_lock_serializes_same_target confirms the in-process asyncio.Lock serializes same-target coroutines correctly, and §2's parallel timing confirms the lock is active. The gap was a DOM-readiness issue: click_send's send-button poll budget was ~3s (range(10) × 0.3s), too short for ChatGPT's composer to re-enable the send button after a prior back-to-back send released the lock. Under legacy single-worker mode requests naturally space out, so the race rarely surfaced; parallel mode's tighter pipelining exposed it.

Fix (this PR): Increased the poll budget to 10s via named constants (SEND_BUTTON_POLL_MAX_WAIT_S, SEND_BUTTON_POLL_INTERVAL_S in chatgpt_dom.py) and rewrote the loop to be time-budget-based. 10s covers observed composer-reset times while still bounding the wait on a genuinely broken composer.

Stress-test result (post-fix): 5 rounds × 2 concurrent same-worker requests = 10/10 returned 200, 0 failures (was 5/10 before the fix). Timing confirms serialization (e.g. round 4: 7.5s + 19.2s — second request waited for the lock, then the composer poll waited for the button, then sent). §2 different-tab parallelism re-confirmed still parallel.

Impact on operational acceptance: The gap that blocked §3 is resolved. §4 (failure modes) and §5 (observability) remain to run before promoting to "operationally accepted."

Preflight gate (must be true before starting)

  • No mixed workers on the CDP port. Every worker targeting this Chrome instance is on the new parallel_tabs=true code, OR every worker is on default-off legacy mode. Old port-lock-only and new per-target-lock workers do not exclude each other; mixing them reintroduces split-brain. (See deployment.md rollout warning.)
  • Distinct W2A_INSTANCE_ID per worker (or rely on the transport-aware default: rest:{port} / mcp:sse:{host}:{port} / mcp:stdio:{pid}). Do NOT reuse one W2A_INSTANCE_ID across live workers — they will collide on one tab-registry entry.

1. Default-off smoke

Deploy current master with parallel_tabs=false.

  • Legacy single-worker behavior is unchanged (one request at a time per tab).
  • No owned_tab_required error leaks into legacy mode.
  • Restart in dependency order: REST/owner first → wait for Chrome/CDP readiness → MCP/SSE/attachers. (See runbook §7.)

2. Parallel canary (2+ workers, one Chrome)

parallel_tabs=true, tab_mode=owned, two REST workers on distinct REST ports sharing one CDP port. Run scripts/parallel_canary.py --ports 8081,8082 --cdp-port 9222.

  • Each worker obtains a distinct owned tab (CDP /json/list shows ≥2 chatgpt.com page targets; canary JSON cdp_targets.chatgpt_tabs_after ≥ 2).
  • Different-tab sends proceed concurrently (canary concurrent.total_window_s meaningfully less than the sum of individuals).
  • All concurrent requests return OpenAI-compatible 200s.
  • No unexpected circuit_open / auth_required breakers tripped solely by parallel pressure (rate-limit backoff is expected; a tripped breaker is the signal to investigate).

3. Same-tab serialization (sanity, not re-proof)

Two concurrent requests to the same worker (canary same_worker_serialization).

  • Timing is consistent with serialization (total ≈ sum of individuals). Note: live timing against ChatGPT is noisy — backend latency, streaming, rate limits, and DOM readiness all confound it. Treat this as advisory; the unit suite (test_lock_resolver::test_mutation_lock_serializes_same_target) is authoritative. If timing looks parallel, investigate via logs (lock acquire/release order) before concluding a bug.

4. Failure-mode validation

  • Kill an attacher process → Chrome remains alive; the owner's monitor is unaffected. PASS (live, 2026-07-03): killed worker B (PID 39560); Chrome (PID 8080) stayed up; worker A (8092) health unaffected.
  • Kill the owner process → Chrome is orphaned until a new process elects. PASS (unit-test authority + runbook §7a): the three owner/ attacher tests (test_monitor_attacher_observes_no_restart, test_monitor_breaker_open_suppresses_restart, test_stop_owner_kills_attacher_does_not) confirm no self-promotion, no attacher restart. Live kill not performed — both test workers were attachers (Chrome pre-existing); a live owner-kill would require tearing down all of Chrome for a behavior already proven by tests + documented as intentional in runbook §7a.
  • Force tab loss / target drift → PASS (live, 2026-07-03): closed worker A's owned tab via CDP /json/close; next request returned 503 code=owned_tab_required with the drift-guard message ("Owned target changed during reconnect OLD → NEW; retry the mutation"). No silent adoption of another tab — the reconnect drift guard (PR4) fired exactly as designed.
  • Concurrent MCP processes → PASS (verified, 2026-07-03): two MCP SSE processes on ports 9001/9002 derive distinct identities (mcp:sse:127.0.0.1:9001 ≠ mcp:sse:127.0.0.1:9002 → distinct instance_id hashes f857… ≠ 01de…); stdio gets PID-based identity (mcp:stdio:{pid}). Live tab-registry shows 5 distinct entries (one per process), no collision or thrash.

5. Observability pass

Capture logs from a normal parallel run AND from each failure mode above.

  • An operator can identify which worker / REST port / transport failed without attaching a debugger. PASS: log lines name the role ("Found existing Chrome" = attacher), the error type (OwnedTabRequiredError), the specific target IDs (B14D8C… → AE91EA…), and the failure mode ("Owned target changed during reconnect"). /health last_error carries the full typed error message.
  • The new log lines from PR2/PR4 surface correctly under parallel mode: owner vs attacher (_owns_chrome), "Monitor disabled: attached to existing Chrome", "Refusing restart: not Chrome owner", the owned_tab_required markers, drift-guard raises. PASS: the §4c drift-guard raise + OwnedTabRequiredError + Chat error: prefix all appear in logs; the monitor/restart/stop role lines appear in Chrome- monitor logs (confirmed during §3 stress testing).
  • /health reflects REST-side state. PASS: last_error = full OwnedTabRequiredError message; open_breakers = [] (correct — drift is a fail-closed signal, not a breaker trip); cdp_connected/driver_ connected = True (worker reconnected post-drift). MCP has its own breaker registry that REST /health does not reflect (confirmed: SSE server returns 404 on /health) — see runbook §9.

6. Exit criterion

When all of §1–§5 pass against the live environment:

  • Update release wording from "merged and opt-in available" to "parallel multi-tab mode is operationally accepted."
  • Record the validation run (date, environment, canary JSON) alongside this checklist.

Validation run: 2026-07-03. Environment: Chrome 149 on CDP 9222 (logged in, account: Nabeel Alajmah), master @ f139f87 (includes PRs #33/#35/#36). All §1–§5 checks PASS (§1 default-off smoke; §2 different-tab concurrency; §3 same- tab serialization post-fix; §4 all four failure modes; §5 observability).

Feature status: parallel multi-tab mode is operationally accepted. ✅


Out of scope for this validation (tracked as future work)

  • Cross-instance pool / single-endpoint router (each worker still needs its own local REST/MCP port).
  • Owner-process runtime failover / ownership lease (owner death orphans Chrome until a new process elects — see runbook §7a).
  • Conversation-scoped / project-scoped / account-scoped locks (deliberately dropped during design — backend serializes those).