[KristiforMilchev]

Third part of milestone M1. Stacked on #2: it targets controller-connect so the diff shows only the scheduler, and GitHub retargets it to main once #2 is merged.

What it does

• controller run create plans a pipeline for an event and stores the run and its jobs. Jobs wait for their needs, are skipped when a need fails, and the run closes as succeeded, failed or cancelled.
• Jobs go only to workers that match the pool (from the run, never the pipeline), OS, architecture, image and labels, up to each worker's capacity.
• Claiming a job is one SQL statement with FOR UPDATE SKIP LOCKED plus a re-check after the lock, so two controllers can't assign the same job.
• Leases: heartbeats renew them. An expired lease requeues the job, up to 3 attempts. A job that was assigned but never started goes back as soon as its worker disconnects, without using an attempt. A late result from a worker that lost the job is ignored, and that worker is told to stop it.
• Cancel works from any process: the serving controller stops the jobs on the workers it holds.

Worker client fixes found by these tests

• Stopping a worker took about 400ms, because Receive ignored context cancellation. It's now about 0.1ms, with a regression test.
• HTTP/2 pings detect a controller that disappeared without closing the connection.

Tests: fake workers built on the real worker client, against a real controller and PostgreSQL. They cover:

• a single job, and a matrix across Linux, Windows and macOS with needs;
• a failure skipping dependent jobs;
• pool isolation, and image and label matching;
• capacity limits;
• lease expiry with a stale late result, and a worker disconnecting before starting a job;
• infrastructure errors retried until attempts run out;
• cancel, both in-process and from another process;
• two controllers with 20 jobs and 4 workers.

To check the tests catch real bugs, I broke the queries on purpose (ignore pool, ignore labels, never skip blocked jobs, no locking). Each change made a test fail; the labels case only did after I added a test for it.

🤖 Generated with Claude Code