Radiopadre Tutorial

                                                            O. Smirnov <o.smirnov@ru.ac.za>, January 2018

Radiopadre is a framework, built on the Jupyter notebook, for browsing and visualizing data reduction products. It is particularly useful for visualizing data products on remote servers, where connection latencies and/or lack of software etc. limits the usual visualization options. It includes integration with the JS9 browser-based FITS viewer (with CARTA integration coming soon).

The general use case for Radiopadre is "here I am sitting with a slow ssh connection into a remote cluster node, my pipeline has produced 500 plots/logs/FITS images, how do I make sense of this mess?" More specifically, there are three (somewhat overlapping) scenarios that Radiopadre is designed for:

• Just browsing: interactively exploring the aforementioned 500 files using a notebook.

• Automated reporting: customized Radiopadre notebooks that automatically generate a report composed of a pipeline's outputs and intermediate products. Since your pipeline's output is (hopefully!) structured, i.e. in terms of filename conventions etc., you can write a notebook to exploit that structure and make a corresponding report automatically.

• Sharing notebooks: fiddle with a notebook until everything is visualized just right, insert explanatory text in mardkdown cells in between, voila, you have an instant report you can share with colleagues.

Installing Radiopadre

Refer to README.md on the github repository: https://github.com/ratt-ru/radiopadre

Running this tutorial

Data files for this tutorial are available here: https://www.dropbox.com/sh/be4pc23rsavj67s/AAB2Ejv8cLsVT8wj60DiqS8Ya?dl=0

Download the tutorial and untar itsomewhere. Then run Radiopadre (locally or remotely, if you unpacked the tutorial on a remote node) in the resulting directory. A Jupyter console will pop up in your browser. Click on radiopadre-tutorial.ipynb to open it in a separate window, then click the "Run all" button on the toolbar (or use "Cell|Run all" in the menu, which is the same thing.) Wait for the notebook to run through and render, then carry on reading.

Every Radiopadre notebook starts with this

Most objects knows how to show() themselves

So what can you see from the above? dd is a directory object than can render itself -- you get a directory listing. Clearly, Radiopadre can recognize certain types of files -- you can see an images/ subdirectory above, a measurement set, a couple of FITS files, some PNG images, etc. Clicking on a file will either download it or display it in a new tab (this works well for PNG or text files -- don't click on FITS files unless you mean to download a whole copy!) FITS files have a "JS9" button next to them that invokes the JS9 viewer either below the cell, or in a new browser tab. Try it!

Now let's get some objects from the directory listing and get them to render.

Most things are list-like

What you see above is that different object types know how to show themselves intelligently. You also see that a directory object acts like a Python list -- dd[n] gets the n-th object from the directory. What about a slice?

Since a directory is a list of files, it makes sence that the Python slice syntax [5:10] returns an object that is also a list of files. There are other list-like objects in radiopadre. For example, an MS can be considered a list of rows. So...

And a text file is really just a list of lines, so:

NB: FITS images and PNG images are not lists in any sense, so this syntax doesn't work on them. (In the future I'll consider supporting numpy-like slicing, e.g. [100:200,100:200], to transparently extract subsections of images, but for now this is not implemented.)

And list-like things can be searched with ()

Radiopadre's list-like objects (directories/file lists, text files, CASA tables) also support a "search" function, invoked by calling them like a function. This returns an object that is subset of the original object. Three examples:

Other useful things to do with directories/lists of files

If you have a list of image or FITS files, you can ask for thumbnails by calling .thumbs().

And calling .images on a directory returns a list of images. For which we can, of course, render thumbnails:

Other such "list of files by type" attributes are .fits, .tables, and .dirs:

And the show_all() method will call show() on every file object in the list. This is useful if you want to render a bunch of objects with the same parameters:

Accessing a single file by name

The (pattern) operation applied to a directory always returns a filelist (possibly an empty one), even if the pattern is not relly a pattern and selects only one file:

If you want to get at one specific file, using dd(name_or_pattern)[0] becomes a hassle. Filelists therefore support a direct [name_or_pattern] operation which always returns a single file object. If name_or_pattern matches multiple files, only the first one is returned (but radiopadre will show you a transient warning message).

Working with text files

By default, radiopadre renders the beginning and end of a text file. But you can also explicitly render just the head, or just the tail, or the full file.

"Watching" text files

If you're still running a reduction and want to keep an eye on a log file that's being updated, use the .watch() method. This works exactly like .show() and takes the same arguments, but adds a "refresh" button at the top right corner of the cell, which re-executes the cell every time you click it.

Running shell commands

Use .sh("command") on a directory object to quickly run a shell command in that directory. The result is output as a list-of-lines, so all the usual display tricks work.

Working with FITS files

As you saw above, FITS files can be rendered with show(), or viewed via the JS9 buttons. There's also an explicit .js9() method which invokes the viewer in a cell:

With multiple FITS files, it's possible to load all of them into JS9, and use the "<" and ">" keys to switch between images. Use the "JS9 all" button to do this:

There's a shortcut for doing this directly -- just call .js9() on a list of FITS files (note that "collective" functions such as .thumbs() and .js9() will only work on homogenuous filelists, i.e. lists of FITS files. Don't try calling them on a list contaning a mix of files -- it won't work!)

The .header attribute of a FITS file object returns the FITS header, in the same kind of object (list-of-lines) as a text file. So all the tricks we did on text files above still apply:

If you want to read in data from the FITS file, the .fitsobj attribute returns a PrimaryHDU object, just like astropy.io.fits.open(filename) would:

Working with CASA tables

As you saw above, a CASA table object knows how to render itself as a table. Default is to render rows 0 to 100. With array columns, the default display becomes a little unwieldy:

With optional arguments to .show(), you can render just a subset of rows (given as start_row, nrows), and a subset of columns, taking a slice through an array column. The below tells radiopadre to render the first 10 rows, taking the column TIME in its entirety, and taking a [32:34,:] slice through the DATA column.

If you want to render all columns with a common slice, use the optional _ argument (we saw this above). The given slice will be applied to all columns as much as possible (or at least to those that match the shape):

The .table attribute returns a casacore table object with which you can do all the normal casacore table operations:

But if you want to quickly read data from a table, radiopadre provides some fancier methods. For example, subtables of the table are available as a .SUBTABLE_NAME attribute. This gives another table object, with all the functions above available:

...while columns of the table can be read via a .COLUMN_NAME() function (with optional start_row, nrows, step arguments):

So combining the above, here's how to compute the UVW in wavelengths of all baselines to antenna 1, and make a uv-coverage plot of that subset of baselines:

The ls() function

...is where it all begins. As you saw, ls() gives you the current directory. You can also use ls with filename patterns, and also specify a sort order:

You can also use the "R" switch for a recursive directory listing:

Or give a filename to get an object representing that one file:

Om the same principle, give a subdirectory name to get a directory object:

One thing to note is that ls() (i.e. with no patterns) doesn't necessarily list all files. The files included by default are governed by radiopadre settings. Below we'll see how to change those.

Using and changing settings

The settings object we imported above can be used to set various defaults of Radiopadre. Like most other objects, it knows how to render itself:

Using "with" to change settings temporarily

Python's with statement works with radiopadre settings to change settings temporarily. For example, the default FITS rendering settings look like this:

Here's how we can render FITS images with different settings, without changing the global settings. Whatever we set in with only applies in the body of the with statement. In this case it is particularly useful, as it will also apply to the JS9 displays by default: