-
Notifications
You must be signed in to change notification settings - Fork 184
feat: Add the possibility to use the dirac-cwl project's job wrapper #8425
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: integration
Are you sure you want to change the base?
Conversation
experimental JobWrapper integration using git cloning
aldbr
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me!
Note for the other reviewer: the new job wrapper is experimental and is not expected to be executed at a large scale. It's only meant to be used for a few jobs in parallel for now.
@Loxeris have you tested it using the SingularityComputingElement?
The JobAgent can use the SingularituComputingElement instead of the InProcessComputingElement. In that case, the job wrapper is generated in a slightly different way:
DIRAC/src/DIRAC/Resources/Computing/SingularityComputingElement.py
Lines 211 to 222 in 11bc43d
| result = createJobWrapper( | |
| wrapperPath=tmpDir, | |
| rootLocation=self.__innerdir, | |
| jobID=jobDesc.get("jobID", 0), | |
| jobParams=jobDesc.get("jobParams", {}), | |
| resourceParams=jobDesc.get("resourceParams", {}), | |
| optimizerParams=jobDesc.get("optimizerParams", {}), | |
| pythonPath="python", | |
| log=log, | |
| logLevel=logLevel, | |
| extraOptions="" if self.__installDIRACInContainer else "/tmp/pilot.cfg", | |
| ) |
@aldbr I believe the computing element used by the CTAO test instance I did end-to-end tests on is |
|
Just to say that that PR must first be tested using the DIRAC certification setup (instructions not written down yet, will be there for the hackathon next week) |
Adds slights changes to
create_job_wrapper()to use the experimental Job Wrapper for CWL jobs.If the job executable is set to
dirac-cwl-execin the JDL, the new job wrapper is used, else thedefaultWrapperLocationis used like before.The dirac-cwl project is cloned locally on the pilot to get the latest version of the job wrapper before being run.
See DIRACGrid/dirac-cwl#44
BEGINRELEASENOTES
*WorkloadManagementSystem
NEW: Experimental job wrapper for cwl workflows
ENDRELEASENOTES