benchcab sub-command for do_meorg

[ccarouge]

I am wondering if it is worth having a subcommand for do_meorg (entry point to the file transfer through the meorg API) for benchcab.

An example that happened to a "friend" (😆 ):

• I set up benchcab config file
• I run benchcab
• I realise the files are not transferred automatically
• :slap: I forgot to set the model_name entry in the config
• Solutions:
  1. transfer by hand
  2. fix the config and re-run benchcab

Both of the solutions are time-consuming as it stands. It would be better to be able to have:

• 3. Fix config and run benchcab meorg-transfer subcommand.

@abhaasgoyal @bschroeter thoughts? Anything that makes it impossible/complicated to implement?

[bschroeter]

@ccarouge I like this. I will investigate feasibility.

[bschroeter]

On the face of it, it looks possible. We will need to create an additional method in the benchcab class object that will load the configuration and then trigger do_meorg somewhere around here:

benchcab/src/benchcab/benchcab.py

Line 448 in b62d5ab

def run(self, config_path: str, skip: list[str]):

Looking at this, we may consider bringing the meorg call up in the hierarchy, as it is currently nested inside the flux site method.

benchcab/src/benchcab/benchcab.py

Line 248 in b62d5ab

# Upload to meorg if meorg_output_name optional key is passed

It makes more sense to have it as a discrete task at the end of the list of commands in the run command.

@ccarouge Shall I proceed with implementation?

[abhaasgoyal]

@bschroeter @ccarouge In the referenced submission to me_org

benchcab/src/benchcab/benchcab.py

Line 249 in b62d5ab

if config.get("meorg_output_name") is not None:

I believe it was written that way due to it being a pre-requisite for the fluxsite-job output (which may be possible only if the fluxsite job was currently running or already run successfully atleast once).
I think it would be nice to have additional check on those two cases if we are to have an independent stage for meorg job submission.

[ccarouge]

additional method in the benchcab class object that will load the configuration and then trigger do_meorg somewhere around here:

benchcab/src/benchcab/benchcab.py

Line 448 in b62d5ab

def run(self, config_path: str, skip: list[str]):

I would actually put it with the build method:

benchcab/src/benchcab/benchcab.py

Line 332 in b62d5ab

def fluxsite_setup_work_directory(self, config_path: str):

Because it will eventually be relevant to both fluxsite and spatial and it is not an integration function (like run or spatial).

With having a separate sub-command for do_meorg, we could call it directly from fluxsite instead of inside submit_job. So it would work the same as the other steps in the process.

Side note: am I the only one to think that the fluxsite_submit_job function:

benchcab/src/benchcab/benchcab.py

Line 201 in b62d5ab

def fluxsite_submit_job(self, config_path: str, skip: list[str]) -> None:

Should be moved to before here:

benchcab/src/benchcab/benchcab.py

Line 393 in b62d5ab

def fluxsite(self, config_path: str, no_submit: bool, skip: list[str]):

[ccarouge]

As @abhaasgoyal said, the meorg command would have to check that necessary conditions have been met.