notebook.community

Edit and run

Opening Access to Fast Radio Interferometric Imaging

a.k.a. A Quick Demo of rtpipe

a.k.a. Boutique Radio Astronomy Analysis via docker/conda/pwkit

My science interests lie in fast radio transients, such as "fast radio bursts". But, frankly, a big part of why I enjoy studying these things is that it motivates a lot of fun and challenging work with software, algorithms, and data.

One example of that is our ongoing effort at the Very Large Array to do "fast imaging" --- forming images at millisecond cadence --- for massive transient surveys. Massive in this context is both in time (we've conducted a 200-hour survey and are in the midst of another) and data (1 hours on sky => 1 TB of data!). Doing this right requires a lot of custom software development, including the library I'm demoing here: rtpipe.

rtpipe is a library for radio interferometric data analysis that combines single-dish concepts like dedispersion and filters with interferometric concepts like images, the uv-plane, etc.. But it is probably more generally useful if you are interested in custom radio interferometric data analysis. If you are, then I highly recommend browsing the github contributions and blog of Peter Williams. Peter contributed both code and advice that helped shape rtpipe (the mistakes are mine, though).

So, on with the demo...

Let's say you download one of the many public, TB-scale, VLA fast imaging data sets from the NRAO archive. rtpipe lets you read and visualize that data in python, but of course most importantly, it lets one search it for fast radio transients.





In [58]:



    


import rtpipe.RT as rt
import rtpipe.parsesdm as ps
import rtpipe.parsecal as pc
import rtlib_cython as rtlib
from bokeh.plotting import figure, show
from bokeh.io import output_notebook, gridplot
output_notebook()



    


















    






    


    

    

        
        BokehJS successfully loaded.

Note that we're using bokeh, a library for interactive visualization in python (similar to the popular D3.js stuff you see a lot in the NY Times). Bokeh also easily supports visualization embedded in an IPython notebook, as you can see!

For the purposes of searching for transients, rtpipe is centered on the idea of building a dictionary that defines the state of the pipeline. In other contexts, this state would be encompassed in a class, but that doesn't play so well with Python's multiprocessing library, so I ended up building everything into a dict. One nice thing about this design is that this dictionary can be attached as metadata to the products (Python pickle files) of the search. It makes reproducing the results and understanding the data products straightforward.





In [62]:



    


d = rt.set_pipeline('14A-425_sb29612394_1.56903.3271372338', 22, nsegments=200, sigma_image1=10, 
                    dmarr=[57.], nologfile=True, chans=range(3,125)+range(131,253))



    


















    






INFO:rtpipe.parsesdm:Setting nsegments to 200
INFO:rtpipe.parsesdm:Setting nologfile to True
INFO:rtpipe.parsesdm:Setting dmarr to [57.0]
INFO:rtpipe.parsesdm:Setting sigma_image1 to 10
INFO:rtpipe.parsesdm:Setting chans to [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252]
INFO:rtpipe.parsesdm:

INFO:rtpipe.parsesdm:Metadata summary:
INFO:rtpipe.parsesdm:	 Working directory and data at /ipynb, 14A-425_sb29612394_1.56903.3271372338
INFO:rtpipe.parsesdm:	 Using scan 22, source B0355+54 7mN
INFO:rtpipe.parsesdm:	 nants, nbl: 27, 351
INFO:rtpipe.parsesdm:	 Freq range (1.271 -- 1.520). 2 spw with 244 chans.
INFO:rtpipe.parsesdm:	 Scan has 6780 ints (33.9 s) and inttime 0.005 s
INFO:rtpipe.parsesdm:	 2 polarizations: ['RR', 'LL']
INFO:rtpipe.parsesdm:	 Ideal uvgrid npix=(128,192) and res=53 (oversample 1.0)
INFO:rtpipe.RT:Autodetected telcal file /ipynb/14A-425_sb29612394_1.56903.3271372338.GN
INFO:rtpipe.RT:
INFO:rtpipe.RT:Pipeline summary:
INFO:rtpipe.RT:	 Products saved with 14A-425_sb29612394_1.56903.3271372338. telcal calibration with 14A-425_sb29612394_1.56903.3271372338.GN
INFO:rtpipe.RT:	 Using 200 segments of 43 ints (0.2 s) with overlap of 0.0 s
INFO:rtpipe.RT:	 Downsampling in time/freq by 1/1 and skipping 0 ints from start of scan.
INFO:rtpipe.RT:	 Excluding ants []
INFO:rtpipe.RT:	 Using pols ['RR', 'LL']
INFO:rtpipe.RT:
INFO:rtpipe.RT:	 Search with image1 and threshold 10.0.
INFO:rtpipe.RT:	 Using 1 DMs from 57.0 to 57.0 and dts [1].
INFO:rtpipe.RT:	 Using uvgrid npix=(128,192) and res=53.
INFO:rtpipe.RT:	 Expect 0 thermal false positives per segment.
INFO:rtpipe.RT:
INFO:rtpipe.RT:	 Visibility memory usage is 0.2 GB/segment
INFO:rtpipe.RT:	 Imaging in 1 chunks using max of 0.0 GB/segment
INFO:rtpipe.RT:	 Grand total memory usage: 0 GB/segment

So we've defined a state dictionary d for a single scan (VLA jargon for an observation of a source) of an observation. This observation was pointed at the pulsar B0355+54, which is bright and will produce pulses easy to detect in this simple analysis.

Next, we use rtpipe.parsesdm to read in a bit of data for visualization. We read directly from the SDM format data that is provided by the NRAO archive, thanks to Peter's contribution to sdmreader.





In [66]:



    


segment=18 # chosen not-at-all at random
data = ps.read_bdf_segment(d, segment)
print 'Data shape of (integrations, baselines, channels, polarizations) = %s' % (str(data.shape))



    


















    






INFO:rtpipe.parsesdm:Reading segment 18/199, times 08:34:07.097 to 08:34:07.311
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]










    






Data shape of (integrations, baselines, channels, polarizations) = (43, 351, 244, 2)

Note that rtpipe also has the concept of a "segment", which is a single set of integrations that are read and treated homogeneously. Typically, radio interferometry software will treat each integration independently in terms of calibration and flagging; they are ultimately combined to form sensitive images. However, rtpipe is purely focused on speed and one way to get speed is to vectorize many operations over the time axis. When calculating uv coordinates or subtracting sources, a segment is defined to be short enough to assume that calculations at time mid-point of the segment are valid over the entire segment. (Here, we've actually forced a large number of segments for purposes of controlling memory usage.)

Next, we parse gain calibration from "telcal" files. These are calibration products produced automatically by the VLA system for all calibrator scans. They are typically used for phasing the array (e.g., for pulsar observations), but they are also good for rough calibration. In practice, at L-band (1-2 GHz), the telcal solutions are 90% of the way to a hand-tuned, CASA-based calibration with bandpasses, etc.. Since fast imaging is relatively low sensitivity, telcal is good for our purposes.





In [18]:



    


solutions = pc.telcal_sol('14A-425_sb29612394_1.56903.3271372338.GN')
solutions.set_selection(d['segmenttimes'][segment].mean(), d['freq']*1e9, rtlib.calc_blarr(d))
solutions.apply(data)

A variety of flagging algorithms exist, all ad hoc and based on my experience playing with fast sampled data. The selection of the algorithm and paramaters for each algorithm are defined in the state dictionary. Here we define three flag algorithms, which each iterate over the two polarizations and two spectral windows.





In [19]:



    


rt.dataflag(d, data)



    


















    






INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols [6]/[1], 1.85 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 0), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 1), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 0), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 1), 3.0 sigma: 0.00 % of total flagged

So, now we have calibrated, flagged visibilities in a numpy array. What's it look like? One simple test is to form a Stokes I, phased array by summing over baselines and polarizations. That gives us a 2d data spectrogram to view (and a chance to show off interactive plotting!)





In [20]:



    


dd = data.mean(axis=3).mean(axis=1).real.transpose()
p = figure(plot_width=300, plot_height=800, x_range=(0, len(dd[0])), y_range=(0,len(dd)))
p.image(image=[dd], x=[0], y=[0], dw=[len(dd[0])], dh=[len(dd)], palette='Greys9')
p.xaxis.axis_label='Integration'
p.yaxis.axis_label='Channel'
show(p)

Note that we see no pulse here. However, I chose a segment of time which I know includes one. Where is it? Somewhere other than the phase center! By summing visibility data as I did above, we effectively produce a single synthesized beam on the sky (extending a few tens of arcsec for this D-configuration VLA data).

So how do we find the pulse? Imaging!





In [64]:



    


d['silent'] = True  # suppress some logging
rt.pipeline(d, segment)



    


















    






INFO:rtpipe.RT:Starting search of /ipynb/14A-425_sb29612394_1.56903.3271372338, scan 22, segments [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
WARNING:rtpipe.RT:Data for processing is zeros. Moving on...
INFO:rtpipe.parsesdm:Reading segment 10/199, times 08:34:05.743 to 08:34:05.957
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 10
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 0
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 0
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 10
INFO:rtpipe.parsesdm:Reading segment 11/199, times 08:34:05.912 to 08:34:06.126
INFO:rtpipe.RT:Found 0 cands in scan 22 segment 10 of /ipynb/14A-425_sb29612394_1.56903.3271372338. 
WARNING:rtpipe.RT:Data for processing is zeros. Moving on...
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 11
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 0
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 0
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 11
INFO:rtpipe.parsesdm:Reading segment 12/199, times 08:34:06.081 to 08:34:06.296
INFO:rtpipe.RT:Found 0 cands in scan 22 segment 11 of /ipynb/14A-425_sb29612394_1.56903.3271372338. 
WARNING:rtpipe.RT:Data for processing is zeros. Moving on...
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 12
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 0
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 0
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 12
INFO:rtpipe.parsesdm:Reading segment 13/199, times 08:34:06.251 to 08:34:06.465
INFO:rtpipe.RT:Found 0 cands in scan 22 segment 12 of /ipynb/14A-425_sb29612394_1.56903.3271372338. 
WARNING:rtpipe.RT:Data for processing is zeros. Moving on...
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 13
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 0
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 0
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 13
INFO:rtpipe.parsesdm:Reading segment 14/199, times 08:34:06.420 to 08:34:06.634
INFO:rtpipe.RT:Found 0 cands in scan 22 segment 13 of /ipynb/14A-425_sb29612394_1.56903.3271372338. 
WARNING:rtpipe.RT:Data for processing is zeros. Moving on...
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 14
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 0
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 0
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 14
INFO:rtpipe.parsesdm:Reading segment 15/199, times 08:34:06.589 to 08:34:06.803
INFO:rtpipe.RT:Found 0 cands in scan 22 segment 14 of /ipynb/14A-425_sb29612394_1.56903.3271372338. 
WARNING:rtpipe.RT:Data for processing is zeros. Moving on...
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 15
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 0 chans, 0 ints, 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols [14 14 21 21]/[0 1 0 1], 7.41 % of total flagged
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 0
INFO:rtpipe.RT:Data already flagged for chans 0-121, pol 1
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 0
INFO:rtpipe.RT:Data already flagged for chans 122-243, pol 1
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 15
INFO:rtpipe.parsesdm:Reading segment 16/199, times 08:34:06.758 to 08:34:06.973
INFO:rtpipe.RT:Found 0 cands in scan 22 segment 15 of /ipynb/14A-425_sb29612394_1.56903.3271372338. 
INFO:rtpipe.RT:Searching in 1 chunks with 1 threads
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 16
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 2 chans, 1 ints, 1.57 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 2 chans, 1 ints, 1.57 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 1 chans, 1 ints, 1.37 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 1 chans, 1 ints, 1.37 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 0), 3.0 sigma: 0.20 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 1), 3.0 sigma: 0.14 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 0), 3.0 sigma: 0.24 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 1), 3.0 sigma: 0.21 % of total flagged
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 16
INFO:rtpipe.parsesdm:Reading segment 17/199, times 08:34:06.928 to 08:34:07.142
INFO:rtpipe.RT:Dedispering to max (DM, dt) of (57, 1) ...
INFO:rtpipe.RT:Imaging 34 ints from 0 for (57,1)
INFO:rtpipe.RT:Found 0 cands in scan 22 segment 16 of /ipynb/14A-425_sb29612394_1.56903.3271372338. 
INFO:rtpipe.RT:Searching in 1 chunks with 1 threads
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 17
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 2 chans, 0 ints, 0.41 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 0), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 1), 3.0 sigma: 0.05 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 0), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 1), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 17
INFO:rtpipe.parsesdm:Reading segment 18/199, times 08:34:07.097 to 08:34:07.311
INFO:rtpipe.RT:Dedispering to max (DM, dt) of (57, 1) ...
INFO:rtpipe.RT:Imaging 34 ints from 0 for (57,1)
INFO:rtpipe.RT:Found 0 cands in scan 22 segment 17 of /ipynb/14A-425_sb29612394_1.56903.3271372338. 
INFO:rtpipe.RT:Searching in 1 chunks with 1 threads
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 43 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 18
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 2 chans, 0 ints, 0.41 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 0), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 1), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 0), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 1), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 18
INFO:rtpipe.parsesdm:Reading segment 19/199, times 08:34:07.266 to 08:34:07.481
INFO:rtpipe.RT:Dedispering to max (DM, dt) of (57, 1) ...
INFO:rtpipe.RT:Imaging 34 ints from 0 for (57,1)
INFO:rtpipe.RT:Found 1 cands in scan 22 segment 18 of /ipynb/14A-425_sb29612394_1.56903.3271372338. 
INFO:rtpipe.RT:Got one!  Int=21, DM=57, dt=1: SNR_im=14.6 @ (0.00e+00,-2.06e-03).
INFO:rtpipe.RT:Searching in 1 chunks with 1 threads
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 27 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 19
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 3 chans, 0 ints, 0.61 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 1 chans, 1 ints, 1.37 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 0), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 1), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 0), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 1), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 19
INFO:rtpipe.RT:Dedispering to max (DM, dt) of (57, 1) ...
INFO:rtpipe.RT:Imaging 34 ints from 0 for (57,1)
INFO:rtpipe.RT:Found 0 cands in scan 22 segment 19 of /ipynb/14A-425_sb29612394_1.56903.3271372338.

Pipeline finds a pulse in integration 21. Let's get the data and visualize it.





In [32]:



    


candim,canddata = rt.pipeline_reproduce(d, [segment, 21, 0, 0, 0], product='imdata')



    


















    






INFO:rtpipe.RT:Reproducing candidate...
INFO:rtpipe.parsesdm:Reading segment 40/199, times 08:33:16.161 to 08:33:16.505
INFO:rtpipe.parsesdm:Found online flags for 317 antenna/time ranges.
INFO:rtpipe.parsesdm:Applied online flags to 29 ints.
WARNING:rtpipe.parsesdm:Rolling spw frequencies to increasing order: [1396000000.0, 1268000000.0]
INFO:rtpipe.parsesdm:Calculating uvw for segment 40
INFO:rtpipe.RT:Flagging with flaglist: [('badchtslide', 4.0, 0.0), ('badap', 3.0, 0.2), ('blstd', 3.0, 0.05)]
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 0-121, pol 1), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 0), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad chans/ints flagging for (chans 122-243, pol 1), 4 sigma: 1 chans, 0 ints, 0.20 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 0-121 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Bad basepol flagging for chans 122-243 at 3.0 sigma: ants/pols []/[], 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 0), 3.0 sigma: 0.24 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 0-121, pol 1), 3.0 sigma: 0.68 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 0), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:Blstd flagging for (chans 122-243, pol 1), 3.0 sigma: 0.00 % of total flagged
INFO:rtpipe.RT:No mean time subtraction.
INFO:rtpipe.RT:All data unlocked for segment 40
INFO:rtpipe.RT:Dedispersing with DM=57.0, dt=1...
INFO:rtpipe.RT:Imaging int 13 with 128 192 pixels...
INFO:rtpipe.RT:Made image with SNR min, max: -3.1, 30.4










    






Gridded 1.000 of data. Scaling fft by = 0.2
Pixel sizes (60.8", 40.5"), Field size 3891.8"

















In [61]:



    


pim = figure(plot_width=400, plot_height=400, x_range=(0,len(candim[0])), y_range=(0,len(candim)))
pim.image(image=[candim.transpose()], x=[0], y=[0], dw=[len(candim[0])], dh=[len(candim)], palette='Greys9')
pim.xaxis.axis_label='RA (pixels)'
pim.yaxis.axis_label='Dec (pixels)'

dd = canddata.mean(axis=2).real.transpose()
psp = figure(plot_width=200, plot_height=400, x_range=(0, len(dd[0])), y_range=(0,len(dd)))
psp.image(image=[dd], x=[0], y=[0], dw=[len(dd[0])], dh=[len(dd)], palette='Greys9')
psp.xaxis.axis_label='Integration'
psp.yaxis.axis_label='Channel'

p = gridplot([[pim, psp]])
show(p)



    


















    



























In [ ]:

Content source: caseyjlaw/docker-rtpipe-binder

Similar notebooks:

notebook.community | gallery | about