[Jobs] Eagerly initialize JobManager on head startup

[zzchun]

Summary

Eagerly initialize JobManager when the head Dashboard Agent starts so
non-terminal submission jobs are recovered without requiring a subsequent
Job API request.

Worker Dashboard Agents retain the existing lazy initialization behavior.

Problem

JobManager.__init__() schedules _recover_running_jobs(), but JobManager
is lazily constructed by Job API handlers. After the head Dashboard Agent
restarts, existing PENDING or RUNNING jobs may therefore remain
unmonitored until another Job API request arrives.

This can leave stale jobs indefinitely in a non-terminal state when their
JobSupervisor no longer exists.

Changes

• Initialize JobManager during head JobAgent.run().
• Keep worker Dashboard Agents unchanged.
• Run ray.init() in an executor so Dashboard Agent startup does not block
  the asyncio event loop.
• Serialize process-level Ray initialization to avoid races with Job API
  handlers.
• Retry initialization with capped exponential backoff and jitter.
• Add bounded

Related issues

Fixes #65037

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request introduces eager initialization of the JobManager on the head node within the JobAgent, along with a robust, budgeted retry mechanism for recovering running jobs. It also refactors the Ray connection initialization into a thread-safe helper (init_ray_connection) run within an executor. Feedback is provided to optimize the @init_ray_and_catch_exceptions decorator by checking if Ray is already initialized in the main thread, thereby avoiding the overhead of thread pool scheduling and lock acquisition on every HTTP request.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit d219241. Configure here.}