[simonpintarelli]

If the sqfs file disappears after job submission, slurm is draining the nodes.

Minimum requirement for the fix is slurm 25.05. The github ci container needs to be updated to ubuntu 26.04 LTS (which provides 25.11). On older slurm versions it won't compile.

Reproducer

sbatch.sh

#!/bin/bash
#SBATCH --uenv=<path-to-sqfs>:/user-environment
srun true

1. Submit the job in a held state:

   sbatch --hold sbatch.sh

2. Manually delete the file.

3. Release the job:

   scontrol release <jobid>

ESPANK_ERROR vs ESPANK_JOB_FAILURE (kernel backend)

init_post_opt_remote() (src/slurm/plugin_kernel.cpp) runs inside slurm_spank_init_post_opt in Slurm's remote context — on a batch step, this and slurm_spank_init share one result in Slurm's own spank_init_remote(), and slurmctld's _slurm_rpc_complete_batch_script() drains the compute node for any nonzero result except a short whitelist. A bad or missing squashfs image under the user's scratch tree is not a node-health problem, but returning the codebase's usual -ESPANK_ERROR from here drains the node.
ESPANK_JOB_FAILURE is on that whitelist. Use it instead of -ESPANK_ERROR for failures caused by the job's own data or arguments.

Cf: proc_req.c#2433

• This is not a documented SPANK feature. man spank only describes ESPANK_JOB_FAILURE as a value for the plugin-exported slurm_spank_init_failure_mode variable, and that variable is documented to remap a slurm_spank_init failure only. Returning ESPANK_JOB_FAILURE directly from a different callback (slurm_spank_init_post_opt) works because of how Slurm happens to be implemented, not because the API describeds it.
• Only matters for the kernel backend's remote-context return path. The FUSE backend mounts in slurm_spank_task_init instead (plugin_fuse.cpp), a hook man spank's Errors table documents as never draining the node for any command, so this class of bug doesn't apply there. The -ESPANK_ERROR returns elsewhere in plugin.cpp (option parsing, init_post_opt_local_allocator) run in local/allocator context, which goes through a different Slurm RPC path that never drains a node either.

TODO

Review -ESPANK_ERROR return value in the remaining parts of the plugin.

[simonpintarelli]

Removed ls --garbage from unit tests: depending on the version ls prints a different error message:

root@b3966939dc76:/# ls --version
ls (uutils coreutils) 0.10.0
root@b3966939dc76:/# ls --garbage
error: unexpected argument '--garbage' found

vs

➜  ~ ls --version                                                                          [26/10/16| 1:19PM]
ls (GNU coreutils) 9.12

➜  ~ ls --garbage                                                                          [26/10/16| 1:23PM]
ls: unrecognized option '--garbage'
Try 'ls --help' for more information.