Skip to content

Conversation

@Loxeris
Copy link
Member

@Loxeris Loxeris commented Jan 14, 2026

Adds slights changes to create_job_wrapper() to use the experimental Job Wrapper for CWL jobs.
If the job executable is set to dirac-cwl-exec in the JDL, the new job wrapper is used, else the defaultWrapperLocation is used like before.
The dirac-cwl project is cloned locally on the pilot to get the latest version of the job wrapper before being run.

See DIRACGrid/dirac-cwl#44

BEGINRELEASENOTES

*WorkloadManagementSystem
NEW: Experimental job wrapper for cwl workflows

ENDRELEASENOTES

Copy link
Contributor

@aldbr aldbr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!
Note for the other reviewer: the new job wrapper is experimental and is not expected to be executed at a large scale. It's only meant to be used for a few jobs in parallel for now.

@Loxeris have you tested it using the SingularityComputingElement?
The JobAgent can use the SingularituComputingElement instead of the InProcessComputingElement. In that case, the job wrapper is generated in a slightly different way:

result = createJobWrapper(
wrapperPath=tmpDir,
rootLocation=self.__innerdir,
jobID=jobDesc.get("jobID", 0),
jobParams=jobDesc.get("jobParams", {}),
resourceParams=jobDesc.get("resourceParams", {}),
optimizerParams=jobDesc.get("optimizerParams", {}),
pythonPath="python",
log=log,
logLevel=logLevel,
extraOptions="" if self.__installDIRACInContainer else "/tmp/pilot.cfg",
)

@Loxeris
Copy link
Member Author

Loxeris commented Jan 14, 2026

@Loxeris have you tested it using the SingularityComputingElement? The JobAgent can use the SingularituComputingElement instead of the InProcessComputingElement. In that case, the job wrapper is generated in a slightly different way

@aldbr I believe the computing element used by the CTAO test instance I did end-to-end tests on is SSHComputingElement. I didn't try using a different type of computing element.

@fstagni
Copy link
Contributor

fstagni commented Jan 14, 2026

Just to say that that PR must first be tested using the DIRAC certification setup (instructions not written down yet, will be there for the hackathon next week)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants