Skip to content

runner: Linux job containers run with NO cgroup limits - oom-killer takes the runner listener and the logs #102

Description

@luthermonson

Verified in source: the Linux job container OCI spec (pkg/runtime/runtime.go ~635: WithDefaultSpecForPlatform + WithImageConfig + WithEnv + caps) sets no LinuxResources - no memory.max, no CPU quota. Windows Hyper-V containers DO get limits (4GB/2CPU defaults), Linux gets nothing.

Observed consequence (2026-07-07/08, 4vCPU/16GB VM): >=3 concurrent heavy builds -> kernel oom-killer reaps a whole job container mid-compile -> Runner.Listener dies with it -> GitHub marks the job failed and logs are never uploaded (BlobNotFound), leaving zero diagnostics. Repeated across SDK builds, CI matrix jobs, and E2E lanes.

Proposal:

  1. [runner.linux] memory_bytes / cpu_count config (mirroring the Windows knobs); DEFAULT derived: (vm_memory - 2GB reserve) / max_concurrent, so the sum can never exceed the VM.
  2. Protect the listener: run job step processes in a child cgroup capped slightly below the container limit, or set oom_score_adj so the kernel kills the build process, not Runner.Listener - failed jobs then fail WITH logs.
  3. Surface VM oom events: tail dmesg/kmsg for oom-killer lines in the dispatch agent and mirror them into ephemerd's log with the victim cgroup -> ends the guesswork.
  4. Emit ephemerd_job_memory_limit_bytes so operators can see the derived caps.

Related: fix/claim-retry-resilience (PR #101) handles the claim side of tonight's failures; this handles the execution side.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions