edges_cal.xrfi.detrend_medfilt

edges_cal.xrfi.detrend_medfilt(data: ndarray, flags: ndarray | None = None, half_size: tuple[int | None] | None = None)[source]

Detrend array using a median filter.

Note

ripped from here: https://github.com/HERA-Team/hera_qm/blob/master/hera_qm/xrfi.py

Parameters:

  • data (array) – Data to detrend. Can be an array of any number of dimensions.

  • flags (boolean array, optional) – Flags specifying data to ignore in the detrend. If not given, don’t ignore anything.

  • half_size (tuple of int/None) – The half-size of the kernel to convolve (kernel size will be 2*half_size+1). Value of zero (for any dimension) omits that axis from the kernel, effectively applying the detrending for each subarray along that axis. Value of None will effectively (but slowly) perform a median along the entire axis before running the kernel over the other axis.

Returns:

out (array) – An array containing the outlier significance metric. Same type and size as data.

Notes

This detrending is very good for data with large RFI compared to the noise, but also reasonably large noise compared to the spectrum steepness. If the noise is small compared to the steepness of the spectrum, individual windows can become almost always monotonic, in which case the randomly non-monotonic bins “stick out” and get wrongly flagged. This can be helped three ways:

  1. Use a smaller bin width. This helps by reducing the probability that a bin will be randomly non-monotonic. However it also loses signal-to-noise on the RFI.

  2. Pre-fit a smooth model that “flattens” the spectrum. This helps by reducing the probability that bins will be monotonic (higher noise level wrt steepness). It has the disadvantage that fitted models can be wrong when there’s RFI there.

  3. Follow the medfilt with a meanfilt: if the medfilt is able to flag most/all of the RFI, then a following meanfilt will tend to “unfilter” the wrongly flagged parts.