Feature: stop when last trained epoch time longer than remaining job time

[albertz]

Slurm specific: I think via squeue -h -j $SLURM_JOB_ID -O TimeLeft, we can know how much runtime we have left.

When the remaining runtime is less than the training time of the previous epoch, we should stop now, so that Sisyphus can resubmit, and we don't waste the training of half an epoch.

Btw, how should we stop that Sisyphus will continue? Exit state 0 might incorrectly let Sisyphus think that we have fully finished? Exit status non-zero might let Sisyphus think there is a non-recoverable error? How does Sisyphus determine when a job can be resumed?

[albertz]

Btw, how should we stop that Sisyphus will continue? Exit state 0 might incorrectly let Sisyphus think that we have fully finished? Exit status non-zero might let Sisyphus think there is a non-recoverable error? How does Sisyphus determine when a job can be resumed?

In sisyphus.task.Task.run, a bit simplified:

        try:
            f = getattr(self._job, task)
            f(*args)
        except sp.CalledProcessError as e:
            if e.returncode == 137:
                logging.error("Command got killed by SGE (probably out of memory):")
                logging.error("Cmd: %s" % e.cmd)
                logging.error("Args: %s" % str(e.args))
                logging.error("Return-Code: %s" % e.returncode)
                logging_thread.out_of_memory = True
                logging_thread.stop()
            else:
                self.error(task_id, True)
        except Exception:
            self.error(task_id, True)
        else:
            self.finished(task_id, True)

So the CalledProcessError with returncode == 137 is basically the only code path currently which allows for resume. All other code paths set the job either into error state or finished state. However, setting out_of_memory = True will then also set this out_of_memory flag in the PLOGGING_FILE, and the default update_engine_rqmt will then increase the mem rqmt because of that:

    # Did we run out of time?
    requested_time = last_rqmt.get("time")
    used_time = last_usage.get("used_time", 0)
    if requested_time and requested_time - used_time < 0.1:
        out["time"] = requested_time * 2

    # Did it (nearly) break the memory limits?
    requested_memory = last_rqmt.get("mem")
    used_memory = last_usage.get("max", {}).get("rss", 0)
    if requested_memory and last_usage.get("out_of_memory") or requested_memory - used_memory < 0.25:
        out["mem"] = requested_memory * 2

Btw, similarly, see the check on time limit, which is also relevant here. We maybe should make sure that we also stop in time before that limit?

Anyway, about the question how we should stop, some potential options:

• scancel $SLURM_JOB_ID. That would solve it. No finished or error flag are then set, and thus the job is resumable. The biggest problem here is that this is again Slurm specific.
• Kill the parent Sisyphus worker proc. This is however a bit tricky to determine. E.g. in case of train proc manager, or maybe various other cases, the immediate parent proc is not Sisyphus.
• Raise SIGINT in the whole process group / session? But not sure if this is again always the correct thing.

[albertz]

Raise SIGINT in the whole process group / session? But not sure if this is again always the correct thing.

I remembered where I stumbled upon such a case, where this had very weird side effects:

Nix wants to cleans all procs of the build user, via setuid(uid) + kill(-1, SIGKILL). When nix is run via apptainer/singularity with --fakeroot, it will kill all running procs of your current user even outside of apptainer. --pid is a good idea here...

Reported upstream: NixOS/nix#15489

[albertz]

Raise SIGINT in the whole process group / session? But not sure if this is again always the correct thing.

I remembered where I stumbled upon such a case, where this had very weird side effects:

Nix wants to cleans all procs of the build user, via setuid(uid) + kill(-1, SIGKILL). When nix is run via apptainer/singularity with --fakeroot, it will kill all running procs of your current user even outside of apptainer. --pid is a good idea here...

Reported upstream: NixOS/nix#15489

Note, what I suggested here would be a bit different from that case. Maybe that's more safe. It's actually the standard way how signals are send to spawned processes in your shell. Specifically I'm thinking about:

pid_t pgid = getpgid(0);
kill(-pgid, SIGINT);

[albertz]

pid_t pgid = getpgid(0);
kill(-pgid, SIGINT);

Note, the corresponding Python code:

pgid = os.getpgrp()
os.kill(-pgid, signal.SIGINT)