Import standard modules:
In [ ]:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from IPython.display import HTML
HTML('../style/course.css') #apply general CSS
Import section specific modules:
In [ ]:
pass
In [ ]:
HTML('../style/code_toggle.html')
Radio Frequency Interference (RFI) are unwanted radio signals present in your data from either natural or man-made sources. For observations of HI the most prominant, and troubling, sources of RFI include the Sun, satellites (particularly GPS and GLONASS), navigation radar for airplanes, broadcast TV, and internal sources from the observatory. RFI can be narrow-band, affecting only a few spectral channels, or broad-band, contaminating an entire spectrum. It can vary on timescales of microseconds to hours or days. Because most externally generated RFI is typically very strong it can still swamp astronomical signals of interest when entering the telescope through the sidelobes. RFI is a problem both for single-dish HI observations and interferometers, particularly on short baselines. In this section, we will discuss the nature of RFI, why it is a problem, and how to avoid, mitigate, and excise it.
Use of the radio frequency spectrum is governed by the International Telecommunications Union and national authorities (in South Africa this is the Independent Communications Authority of South Africa). Every portion of the spectrum is allocated to different users with some level of protection, and the only protections relevant for HI observers are from 1400-1427 MHz (local HI observations) and 606-614 MHz (high redshift HI observations). The rest of the spectrum is allocated to a variety of broadcast services, including TV, radars, satellite navigation, and the like. Unless you are only going to observe HI associated with the Milky Way, you will have to deal with man-made RFI. The plethora of narrowband man-made interference in the L-band (1150-1730 MHz) can be seen in the figure. In addition, as the brightest radio source in the sky the Sun produces broad-band interference that can corrupt any daytime observations.
RFI causes problems for both single-dish and interferometer observations, but it is worst for single-dish telescopes. This is because the cross-correlation of signals from different elements in an interferometer will have the effect of decorrelating RFI signals. Still, RFI will be seen on short baselines and strong RFI can still corrupt longer baselines even when mostly decorrelated.
RFI is most easily identifiable as a time-variable signal that does not vary strongly with the position of the telescope. This is because the sidelobes tend to be much larger on the sky than the primary beam of the telescope. Internally generated RFI tends to be extremely stable over time, but is competely independent of the telescope pointing. External RFI is often strongly polarized, either due to how it was transmitted or due to reflections off of the local terrain or nearby structures. If the RFI is narrowband, then it should remain at a constant frequency in the topocentric frame, but it will drift if you are Doppler tracking your astronomical source. As stated above, interferometers will see most RFI only on their shortest baselines providing yet another way to identify it.
RFI corrupts your data in a variety of ways. For narrowband RFI, if it is relatively weak, then it can mimic an astronomical signal, possibly leading to a spurious detection. Stronger narrowband RFI can produce ringing in the spectrum wiping out large portions of the band. Weak broadband RFI can cause the system temperature to increase, just adding to the noise of your observation, but more often it will cause a quasi-sinusoidal baseline ripple. Such ripples can add incoherently causing a large increase in the system temperature. The strongest broadband RFI will cause the amplifiers in the receiver to respond non-linearly, also producing strong ripples in the data.
There are many ways to remove the effects of RFI from your astronomical observations. They can be broadly classified as avoidance, excision, and mitigation.
Avoiding RFI in the first place is the most effective method for coping with it. If solar interference is a big problem for you, then you should only observe at night. Observatories are often their own worst enemies in terms of generating RFI so it is critical for them to suppress internal RFI. This involves RFI shielding of computers and electronics, and moving possible interferers away from the telescopes. It is particularly important to shield computers in the signal path since this can produce feedback that can rapidly overwelm any astronomical signal.
Keeping sources of interference away from radio telescopes can be difficult, but is quite effective. For example, at the Green Bank Observatory, only diesel cars are allowed near the telescopes, since spark plugs in petrol engines generate radio signals. This is also the main reason that most radio observatories are far from large cities. This is why many radio telescopes are located in valleys so that the landscape provides some shielding of distant sources of RFI. Legal restrictions on radio transmission can also be imposed. The observatory is also located within the National Radio Quiet Zone, which gives them the ability to regulate fixed transmitters within a 34000 square kilometer area. This means that there are few broadcast TV station and radio station signals that can be detected in Green Bank. Within 13 km of the Green Bank Telescope (GBT), the West Virginia Radio Astronomy Zone gives the observatory authority over any transmission, intentional or otherwise. As a result, there are no cellphone towers and no wireless internet in Green Bank, WV.
Nevertheless, a lot of RFI comes from satellites and navigation radars used by airplanes. These sources of RFI can be detected from thousands of miles away and are, therefore, very difficult to avoid. If you have a choice of frequencies to observe (see figure), you could just avoid observing at frequencies with large amounts of RFI, but, if you observing HI, then this implies that there are certain redshifts which you will never be able to observe HI from galaxies.
Since one cannot avoid all RFI, it is necessary to excise it from your data after you collect it. Traditionally this has been done by examining your data and manually flagging (or discarding) RFI-affected data. This may involve flagging integrations at specific times or flagging data at certain frequencies or discarding data from certain baselines in an interferometer or, most likely, a combination of all three. The challenge of manual flagging, however, is one of volume. A 6 hour observation using 5 second integrations yields 4320 separate integrations usually including 2 separate polarizations, each of which may have 32000 channels. To flag these data by hand would be time-consuming, but is still feasible. For an interferometer like MeerKAT, however, you have 2016 individual baselines yielding a total of over 500 billion separate data points! You could smooth your data in time and frequency to reduce the number of points, but this spreads the RFI out and could lead to more data being corrupted by RFI. You could also use the fact that RFI tends to be polarized and flag those signals bright in Stokes Q or V (linearly or circularly polarized). Still manual flagging is a tedious, time-consuming, and imperfect way to remove RFI from modern datasets.
The solution is to use some sort of automated flagging procedure. A basic routine is that developed by [Hunt et al. (2016)] who used a sigma clipping approach to flag RFI in GBT data. This method works by measuring the noise in the RFI-free portions of each spectrum and then flagging channels that deviate by more than 4 times the noise. An additional 2-4 channels were flagged on either side of existing flags to remove weak RFI below the threshold. [Hunt et al.] also used a similar approach to remove broadband RFI by taking the Fourier transform of every integration. Because the Fourier transform of a sinusoidal baseline results in a narrow spike, the same flagging algorithm was used to flag both broadband and narrowband RFI.
More sophisticated versions of this sigma clipping routine are in existing software packages and are optimal for flagging data from interferometers. In the CASA flagdata task there is RFLAG and TFCROP. These calculate the noise in the time-frequency plane and then flag points that deviate from the expected noise by more than a specified amount. TFCROP treats every baseline and polarization separately and can work on uncalibrated data, while RFLAG needs calibrated data and calculates the noise using a sliding window and accounts for variations with baseline distance and polarization. Both can be run iteratively, and the extend option in flagdata can broaden the flags in time and frequency to account for fainter RFI. Another automated flagger is AOFlagger, which was developed for LOFAR but is now used for data from a wide range of telescopes ([Offringa et al. (2010)], [Offringa et al. (2012)]). AOFlagger basically fits a 2D surface to the time-frequency plane and flags points that deviate from that surface. It also works in an iterative fashion.
For any of these routines, it is important to avoid flagging too much good data. This is a particular problem if you have a strong astronomical signal in your data that can be mistaken for RFI. In this case, you need to exclude the relevant times/frequencies that contain real signal from your flagging algorithm. Another challenge is faint RFI that is not visible in a single integration, but that corrupts your data after a few minutes or hours of observing. Both RFLAG and TFCROP can use time and/or frequency averaging to identify and flag fainter RFI at the price of also flagging some good data. It is important to have a good understanding of your science goals and the nature of your astromical signal when using any of these methods.
There are a variety of active RFI mitigation techniques that are being researched, but few are actively being used in a production mode. The goals of these methods are to remove the effects of RFI from your data without losing the obscured astronomical signals.
One technique of active mitigation is radar blanking as is used at Arecibo (e.g. [Ellingson \& Hampson 2003']). When a sudden increase in power is detected (in a certain frequency range), the data is blanked for 100s of microseconds corresponding to the length of the radar pulse plus enough time for the echoes of the signal to fade away. This technique only works for RFI signals of known duration. While the RFI may not corrupt the data, you are still losing data when the blanking is active.
More sophisticated methods have also been tested on various radio telescopes around the world, but none are regularly used. These include using small reference antennas that strongly detect the sources of RFI and then are anti-correlated with the astronomical data to remove the RFI from the data ([Briggs et al. 2000]). Tests of this technique have shown progress but are limited by the need for good signal-to-noise data and a highly linear response from the reference and many antennas. Both focal plane arrays on single-dish telescopes and interferometers use cross-correlation of the individual elements to form beams pointed towards the astronomical source of interest, but one can also form spatial nulls coincident with sources of RFI to suppress its effect (e.g. [Landon et al. 2012]). Particularly for interferometers this requires very short integration times leading to very high, potentially unsustainable, data rates.
RFI has provided a long-standing challenge for radio astronomers, and, with the growing ubiquity of cellular and wireless technology, the situation continues to degrade. There are many techniques for removing the effects of RFI from one's data with their associated benefits and drawbacks. As astronomers continue to push the limits of current telescopes, our challenges in dealing with RFI continue to grow. Effective mitigation and excision of RFI is essential for future progress.