Skip to content

[Bug]: Job log tail lost on dstack stop #4285

Description

@un-def

Steps to reproduce

type: task
name: task
commands:
  - nohup sleep inf &
  - |
    for e in $(seq 100000); do
      printf '%07d\n' $e
      sleep 1
    done

dstack apply, wait until some logs produced, then dstack stop.

Actual behaviour

~/.dstack/runners/task-0-0-a8fbccc0/runner/job.log:

[...]
0000053
0000054
0000055
0000056
^C

dstack logs task (7 records + ^C lost):

[...]
0000045
0000046
0000047
0000048
0000049

~/.dstack/runners/task-0-0-a8fbccc0/runner/runner.log

[...]
time=2026-09-11T04:10:11.541553-04:00 level=info msg=Job state changed new=running
time=2026-09-11T04:10:11.541585-04:00 level=trace msg=Using credentials user=uid=0(root) gid=0 groups=[0] home=/root
time=2026-09-11T04:10:11.544841-04:00 level=info msg=Required capability is missing, cannot set resource limits missing=SYS_RESOURCE
time=2026-09-11T04:10:11.54491-04:00 level=trace msg=Starting exec cmd=[...]
time=2026-09-11T04:11:15.361874-04:00 level=warning msg=Processes still running after the interrupt, sent SIGHUP pids=[78]
time=2026-09-11T04:11:20.38817-04:00 level=error msg=Processes still running after SIGHUP, sent SIGKILL pids=[78]
time=2026-09-11T04:11:20.388462-04:00 level=error msg=Job canceled
time=2026-09-11T04:11:20.388758-04:00 level=info msg=Job state changed new=terminated
time=2026-09-11T04:11:20.388776-04:00 level=error msg=Executor failed err=job canceled: wait for command: exit status 1

dstack logs -d task (termination messages lost):

[...]
time=2026-09-11T04:10:11.541553-04:00 level=info msg=Job state changed new=running
time=2026-09-11T04:10:11.541585-04:00 level=trace msg=Using credentials user=uid=0(root) gid=0 groups=[0] home=/root
time=2026-09-11T04:10:11.544841-04:00 level=info msg=Required capability is missing, cannot set resource limits missing=SYS_RESOURCE
time=2026-09-11T04:10:11.54491-04:00 level=trace msg=Starting exec cmd=[...]

Expected behaviour

No response

dstack version

a15c156

Server logs

Additional information

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions