`TimeLeft` utility fails in containerized jobs due to batch system isolation

[aldbr]

Problem
When jobs run inside Apptainer containers via SingularityComputingElement, they cannot access batch system tools (e.g., sacct, qstat, bjobs) needed to compute remaining time. This can prevent elastic jobs from getting a good time left estimate.

The SingularityComputingElement creates an isolated container environment and batch system binaries are not bind-mounted into the container, so tools like sacct (SLURM) or condor_q (HTCondor) are not available.

Current Behavior

• Pilot job: can access batch system, compute time left and store it in /LocalSite/CPUTimeLeft in pilot.cfg
• Job in container:
  • Watchdog calls TimeLeft().getTimeLeft(), which tries to execute batch commands and fails.
  • LHCbDIRAC modules also calls TimeLeft().getTimeLeft() to compute the number of events a given job can process within the allocated time. It fails too.

Result: Jobs cannot determine remaining time and may be killed unexpectedly

Proposed Solution

Jobs should likely read CPUTimeLeft from the pilot.cfg file (through gConfig): the pilot already computes and stores time left information every cycle. We could reuse this value in containerized environments.

Any opinion?

[fstagni]

With "the pilot" here, do you mean the JobAgent, re-evaluating at every cycle (so, before the next match-making attempt)?

[aldbr]

Yes I mean the JobAgent indeed

[fstagni]

Then, your suggestion makes sense. Note that the PoolComputingElement already writes down in the local cfg file this section /Resources/Computing/JobLimits/ for every job that is matched (for later use), so what you are suggesting is an already used pattern.