mnms runs with mbatch

[iabrilcabezas]

need to run noise_gen and noise_sim.py scripts. I am using mbatch.

mbatch requires --output-dir argument (where .sh and output messages of run are store). also need to transform the args.qid argument because it is read in as a string. I believe these changes should not alter current pipelines that run mnms/scripts/

[zatkins2]

Hi Irene, could you add your --output-dir argument to noise_all.py as well?

Can you elaborate more on the change to args.qid? The way it was before I think does the same thing: the options supplied to parser.add_argument causes args.qid to be a list of strings

[iabrilcabezas]

it's related to this problem:

You can reproduce this error with a small script test_mbatch.py like this one:

from mnms import noise_models as nm
import argparse

parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)

parser.add_argument("--output-dir", dest="output_dir", type=str, default=None)

parser.add_argument('--config-name', dest='config_name', type=str, required=True,
                    help='Name of model config file from which to load parameters')

parser.add_argument('--noise-model-name', dest='noise_model_name', type=str, required=True,
                    help='Name of model within config file from which to load parameters')

parser.add_argument('--qid', dest='qid', nargs='+', type=str, required=True,
                    help='list of soapack array "qids"')

args = parser.parse_args()
print(args.qid)
# args.qid = [item for sublist in args.qid for item in sublist.split()]

model = nm.BaseNoiseModel.from_config(args.config_name, args.noise_model_name,
                                      *args.qid)
print('success')

To run it, we use a test mbatch config file looks like this

root_dir: /home/ia404/gitreps/mnms/scripts/

stages:

  stage_test:
    exec: python
    options:
      config-name: 'act_dr6v4_day'
      noise-model-name: 'tile_cmbmask_daywide'
      qid:
       - 'pa5a_dw'
       - 'pa6a_dw'

    script: scripts/test_mbatch.py

This generates
python scripts/test_mbatch.py --config-name=act_dr6v4_day --noise-model-name=tile_cmbmask_daywide --qid="['pa5a_dw', 'pa6a_dw']" --output-dir /home/ia404/gitreps/mnms/scripts/test/stage_test, which doesn't work because qid ends up being ["['pa5a_dw', 'pa6a_dw']"], pa5a and pa6a all together.

Another alternative:

root_dir: /home/ia404/gitreps/mnms/scripts/

stages:

  stage_test:
    exec: python
    options:
      config-name: 'act_dr6v4_day'
      noise-model-name: 'tile_cmbmask_daywide'
      qid: 'pa5a_dw pa6a_dw'

    script: scripts/test_mbatch.py

But in this case we have
python scripts/test_mbatch.py --config-name=act_dr6v4_day --noise-model-name=tile_cmbmask_daywide --qid='pa5a_dw pa6a_dw' --output-dir /home/ia404/gitreps/mnms/scripts/test/stage_test

which also doesn't work, ['pa5a_dw pa6a_dw'] not separated again.

Adding
args.qid = [item for sublist in args.qid for item in sublist.split()] is the best solution I could find for it to be able to run with mbatch, while also being compatible with older scripts.

[zatkins2]

Ok, this is very clear. I still wonder if there is a cleaner solution. Can you test what happens in mbatch if you supply qid more than once in a config file?

qid: 'pa5a_dw'
qid: 'pa6a_dw'

[zatkins2]

Please also target develop :)

[iabrilcabezas]

It's now targeting develop, and I opened an issue on mbatch, hopefully good to merge now --

[zatkins2]

As discussed offline, we will hold off on merging until simonsobs/mbatch#25 is fixed on the mbatch side.