Bad amp in only some of the input arc exposures

[jbanovetz]

This pull request is to fix an issue we were running into with matterhorn where some of the input arcs (>2) had bad amps which triggered an the error which triggers when more than 2 arcs have rchi==0, usually corresponding to a fit failure. Because it was not all of the arcs, this skips over a previous check. More information can be found here: #2723

My solution is to mask known exposures with missing bundles from the check for fit failures. I tested this with 20211028 and this successfully ran and produced the expected logs with the warning. See /global/cfs/cdirs/desi/spectro/redux/banovetz/run/scripts/night/20211028/psfnight-20211028-00106397-a0123456789-51950571.log for the log.

[julienguy]

Looks good

[Copilot]

Pull request overview

This PR adjusts specex.mean_psf() to avoid treating bundles that are missing in only some input PSFs (e.g., due to bad amps in some arc exposures) as “fit failures”, preventing psfnight from erroring when multiple inputs have rchi==0 for reasons unrelated to fit quality (ref: desispec#2723).

Changes:

• Exclude exposures where a given bundle is missing from the rchi2==0 “fit failure” counting logic.
• Emit a warning when a bundle is present in only a subset of the input PSFs and therefore fewer PSFs contribute for that bundle.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The missing-bundle warning and the nfailed calculation are inside the per-PARAM loop, so this warning will be emitted once per PARAM entry (potentially many times) even though missing bundles don’t depend on PARAM. Consider moving this missing-bundle filtering / nfailed check outside the for entry loop, or at least guard the warning (and nfailed check) with if entry == 0 to avoid log spam and repeated work.

[sbailey]

Moving to WIP / draft so that we don't accidentally merge this before converging on the right long-term solution.

[jbanovetz]

I added some additional code as before, psfs with failed fibers were being passed. This additional code goes with changes to specex and identifies failed fiber fits (most likely due to fibers crossing) and counts them as failures and causes the pipeline to crash (as intended). These failed fibers will have a STATUS of 4 and this is what the code uses to identify the bad fibers.

[akremin]

Thank you for your work on this! Overall these changes look to do what we want but I do have two important questions I'd like clarification on before I accept the PR.

[akremin]

Can a fiber be both RCHI2=0 and STATUS>0? If so, I think you count that bundle twice and then throw out the bundle in the subsequent lines when really there is only one actual failure.

[jbanovetz]

I don't think so as RCHI2=0 is only true for missing fibers (STATUS<0). When merging the individual bundle files into the primary file, the RCHI2 is set to 0 and is populated with the correct value unless that bundle is missing (

desispec/py/desispec/scripts/specex.py

Lines 449 to 456 in 000242d

	for b in range(20) :
	for key in [ "B{:02d}RCHI2".format(b), "B{:02d}NDATA".format(b),
	"B{:02d}NPAR".format(b) ]:
	if key in psf_hdulist["PSF"].header :
	psf_hdulist["PSF"].header[key] = 0
	# also reset the PSF STATUS
	index = np.where(psf_hdulist["PSF"].data["PARAM"]=="STATUS")[0][0]
	psf_hdulist["PSF"].data["COEFF"][index][:,0]=-1

). So I don't think there is ever a case where this should happen as I also checked and the example of a failed fit does have a RCHI2>0.

[akremin]

Are you confident that the later additions capture all possible corner cases where this could come back to bite us? I worry about including things that failed and then try to exclude bad things later, versus only keeping good things and then explicitly adding back in the failures that we deem acceptable. The former risks corner cases where the latter doesn't. I have a very superficial understanding of this so please let me know if I'm being overly nervous.