Large control file results in unnecessary disk usage

[lindyblackburn]

The single control file for an experiment is now quite large, especially after scan-by-scan directives for flags, adhoc phasing, global fringe solutions. Since this is now expected to be replicated in every fringe file (scan x baseline x polarization product), the control file size dominates over the actual output visibility data and causes unnecessarily large file output. We may need to break up the control file into some reasonable partitioning balancing the global elements with scan-by-scan elements.

[iniyannatarajan]

Is it possible to tell fourfit not to write all the control file directives into the fringefiles? If we break the control files up any further, we will have to introduce more arbitrary assumptions (probably in both ehthops and eat) about control file naming and organization.

[lindyblackburn]

Not easily -- we rely on the embedded control file in various places now to know how the data were calibrated and e.g. reimplement the same precorrections as in fourfit. We made explicit effort to include this behavior by default so the control codes would be available, and we would have to go back to many places where it is assumed to be present and see if something like a filename might suffice instead. It may make sense to split out some control files by expt_no (particularly some autogenerated ones).

[iniyannatarajan]

Thanks! Sorry, I didn't intend to close this issue. Clicked the wrong button.

Ok, this will probably require filtering for more complex patterns in 0.launch while setting up a stage. And in eat, we should write out scan-specific control commands to individual files. I will give this a try this week.

[lindyblackburn]

Let's come up with a design plan before making broad changes to the code. It's possible that something like expt specific control files for some of the corrections are sufficient.

[lindyblackburn]

Need to document where the pipeline makes use of embedded control files. It's possible to use an external control file instead. Here is one example:
https://github.com/sao-eht/eat/blob/8fbbbd125568812ca76b5bb8d5d6a0a539f06b8f/bin/alma_pcal#L116