[P1][Plan 2] SF-037 — Prevent long MCP generation from blocking unrelated requests

[Ankit6149]

Confirmed problem

The MCP server processes requests through one serial promise chain. A long campaign generation can block ping, tools/list, provider status, cancellation, and every other request until generation finishes.

Scope

• Remove global serialization while preserving safe per-campaign ordering.
• Run independent read/status/protocol requests concurrently.
• Model long generation as trackable work with status and cancellation.
• Add bounded worker/concurrency controls rather than unlimited parallel execution.
• Ensure one failed request cannot poison the request processor.

Acceptance criteria

• ping, tools/list, status, and cancellation respond while a campaign is generating.
• Requests for the same campaign remain ordered where necessary.
• Independent campaigns respect configured concurrency limits.
• A rejected promise does not block later requests.
• Shutdown/cancellation cleans up active work safely.

Required verification

• Concurrent JSON-RPC test with one long generation and multiple quick requests.
• Cancellation-during-generation test.
• Failure-isolation and ordering tests.
• External MCP client verification.

Dependency

Coordinate with SF-006 and SF-036 so web and MCP use the same generation job model.

[Ankit6149]

Closing as completed against the issue's stated acceptance criteria.

Merged evidence:

• Make MCP campaign execution trackable and non-blocking #324: removed global JSON-RPC serialization; ping/tools/status/cancel remain responsive during long generation; trackable execution IDs; active cancellation; failure isolation
• Bound and order MCP campaign execution #335: registry-level max concurrency 2; independent projects run within the ceiling; same-project execution keys serialize; queued cancellation never starts work; failures release slots/keys
• Drain MCP campaign work safely on shutdown #336: shutdown cancels/drains trackable jobs, aborts blocking HTTP tool work, distinguishes caller cancellation from timeout, and verifies real stdio SIGTERM cleanup

Verification includes:

• concurrent stdio JSON-RPC while generation is blocked
• cancellation during generation
• same-key ordering
• independent campaign concurrency ceiling
• failure-isolation
• queued cancellation
• registry shutdown drain
• real child-process SIGTERM integration
• exact-head CI with MCP, frontend, GP2, production build, trace audit, and Python suites

Durable cross-process/background-worker recovery remains part of the broader durable job architecture and is not required by SF-037's MCP responsiveness contract.