Piecewise MAD feature for wideband observations#8
Open
danielreardon wants to merge 6 commits into
Open
Conversation
The surgical cleaner detrends its per-cell statistics piecewise in the frequency direction (subint_scaler) but then normalized them to sigma with a single global MAD across the whole band. On wideband receivers (e.g. Parkes UWL, 704-4032 MHz) the off-pulse RMS varies several-fold with Tsys across the band, so a global MAD inflates the apparent significance of channels in high-Tsys sub-bands and systematically over-flags them (and under-flags real RFI in quiet sub-bands). Compute the MAD per segment, using the same np.linspace segmentation as the piecewise detrend, so each channel is judged against the noise floor of its own sub-band: - clean_utils.py: new _piecewise_mad_scale() helper and a piecewise path in subint_scaler, gated by two new kwargs (piecewise_scale, defaulting to False, and subint_mad_numpieces, defaulting to the finest subint_numpieces value). Sparse segments (<16 unmasked points) and zero-MAD segments fall back to the global median/MAD. channel_scaler is already local in frequency and is unchanged. - surgical.py: register piecewise_scale and subint_mad_numpieces params and pass them through to comprehensive_stats. - default.cfg: add piecewise_scale=False, subint_mad_numpieces=None so the default behaviour is unchanged; wideband configs can opt in. Backward compatible: with the flag off, or when the MAD segmentation reduces to one piece, the output is bitwise-identical to the previous global MAD. Tests: Tsys-ramp unit test (global over-flags high-Tsys, piecewise balances; equal-excess spikes judged locally), exact backward-compat regression, sparse/all-masked/zero-MAD fallback edge cases, and a skipped psrchive integration placeholder. 183 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ChmPorpBcsSQQeJ6Zaoutj
# Conflicts: # coast_guard/configurations/default.cfg
add_param() registers a param's type but never stores its `default`, so a param only has a value if it appears in a parsed config string. The surgical cleaner reads self.configs.piecewise_scale and self.configs.subint_mad_numpieces unconditionally in _clean, but a receiver/custom config loaded via -C (set_override_config) replaces surgical_default_params with a string that does not list those keys -- bypassing the default.cfg copy -- so they were never set and the read raised KeyError. Every receiver config (UHF/UWL/L-band), i.e. the whole point of the wideband feature, hit this. Establish the safe (feature-off) defaults in _set_config_params by parsing 'piecewise_scale=False,subint_mad_numpieces=None' before parsing surgical_default_params. A config that lists the keys still overrides them; a config that omits them now falls back to feature-off instead of crashing. The redundant (and override-bypassed) copies are removed from default.cfg so the defaults have a single source of truth. Adds a regression test loading the surgical cleaner behind a receiver-style override both with and without the piecewise keys. 185 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ChmPorpBcsSQQeJ6Zaoutj
- UWL_3K_Murriyang.cfg: turn piecewise scaling on by default (piecewise_scale=True, subint_mad_numpieces=16 to match the finest subint_numpieces) since the 704-4032 MHz UWL band has a large Tsys gradient that a single global MAD over-flags. - clean_archive.py: add a -pw/--piecewise flag so pipelines that do not use a config file can enable it. apply_surgical_cleaner only appends piecewise_scale=True when the flag is given -- when absent it leaves the (default or -C) config's value untouched, so it never switches off a config that enables it (e.g. -C UWL). Adds a regression test that the shipped UWL config enables piecewise scaling. 186 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ChmPorpBcsSQQeJ6Zaoutj
Brings the 2D-template subtraction and template-estimation safety changes (PR #7) into the piecewise-MAD branch. The merge was clean -- the 2D template loading (surgical._clean) and remove_profile1d changes sit in different regions from the piecewise subint_scaler work. On top of the merge: - clean_utils.check_template_nchan(): warn when a 2D (per-channel) template's channel count does not match the data. remove_profile_inplace indexes the template by data channel, so a mismatch mis-aligns the per-channel subtraction (or silently uses only the first data_nchan channels). Called from surgical._clean after the template shape is resolved. - Tests for the merged safety changes (all pure-Python, no psrchive): * remove_profile1d: all-zero template returns None and warns; the dot-product amplitude guess recovers the true scale; and the zero-median-template case (which broke the old median/median guess) now fits correctly. * check_template_nchan: warns on a 2D channel mismatch, stays quiet when the counts match and for a 1D template. 192 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ChmPorpBcsSQQeJ6Zaoutj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added feature to measure outlier statistics with a piecewise MAD, in addition to the piecewise baseline removal