fix(apps): add periodic liveness re-check for app backends

[aniruddhaadak80]

What Problem This Solves

App backends are health-checked exactly once at startup. After _health_check_loop marks a backend healthy, there is no liveness re-check of any kind. A backend that dies later keeps being treated as healthy: the reverse proxy still routes to its port (producing permanent 502/504 errors) and /api/apps reports running: true, healthy: true for a dead process.

Why This Change Was Made

The startup health-check thread (_health_check_loop) already runs on its own daemon thread per backend. After the initial startup poll succeeds, the thread now continues periodic liveness re-checks instead of returning permanently.

Two demotion signals, both cheap-first:

1. Process exit -- proc.poll() catches a dead spawned process instantly (free, no HTTP round-trip). An exited process cannot recover, so one observation is enough.
2. HTTP failure -- a consecutive-failure threshold (_LIVENESS_MAX_CONSECUTIVE_FAILURES = 3) avoids false demotion on a single slow response. Demotion is reversible: if the backend recovers, the next successful probe resets the counter.

Demotion sets healthy=False (stopping proxy routing via get_app_backend_port) and calls _gate_mcp_registration(..., healthy=False) to scrub the MCP entry.

User Impact

• Dead backends are detected within ~90s (3 x 30s interval) instead of never
• Reverse proxy stops routing to dead backends instead of returning 502/504
• Dashboard status accurately reflects backend health
• Recovery is automatic: if a backend comes back, the next successful probe re-registers it

Evidence

• All existing health-check tests pass (test_app_backend.py::TestHealthGatedMcpRegistration, test_apps_backend_coverage.py::TestHealthCheckLoop)
• 3 new tests added in TestLivenessRecheck:
  • test_process_exit_demotes_unhealthy -- killed process triggers immediate demotion
  • test_consecutive_http_failures_demote_unhealthy -- N consecutive failures triggers demotion
  • test_liveness_exits_when_process_removed -- clean exit when backend is stopped

Fixes #5726

[dwu96]

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

[iamwhatever]

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

[iamwhatever]

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

[dwu96]

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

[bolichen97]

Closing this draft as superseded by merged #5929. Main already owns the backend liveness supervisor, failure demotion/recovery, generation fencing, and MCP reconciliation; the remaining safe-probe residual belongs in the narrower #5727 follow-up rather than this duplicate draft. No branch is deleted.

[bolichen97]

Full-diff overlap audit (current head d12e03b0d8e670f736f5c2e50940c31831faa014): merged #5929 (ea4116f6b6323d2ee3e6a8e4d05e91b907f25172) fully covers this smaller app-backend watcher. Its supervisor uses AppProcess identity, polls after startup, counts consecutive HTTP failures, demotes immediately on process exit, fences stale generations, removes stale MCP routing, promotes on recovery/adoption, and retries reconciliation. Its tests include every transition/assertion class in #6079 plus lifecycle races.

Coordination: #6092 literally carries this exact #6079 commit and should drop it while retaining only its still-unique ChatPage indexOf → lastIndexOf fix. #5727 overlaps the same merged liveness core, but still has a residual safe probe (_health_probe_url plus no-proxy/no-redirect loopback_urlopen) that #5929 does not implement; that PR should be reduced to that residual only.

Closing #6079 as fully covered by merged #5929.