Reading data into Astropy Tables

Objectives

Read ASCII files with a defined format
Learn basic operations with astropy.tables
Ingest header information
VOTables

Reading data

Our first task with python was to read a csv file using np.loadtxt(). That function has few properties to define the dlimiter of the columns, skip rows, read commented lines, convert values while reading, etc.

However, the result is an array, without the information of the metadata that file may have included (name, units, ...).

Astropy offers a ascii reader that improves many of these steps while provides templates to read common ascii files in astronomy.



In [1]:

    
from astropy.io import ascii



In [2]:

    
# Read a sample file: sources.dat

Automatically, read has identified the header and the format of each column. The result is a Table object, and that brings some additional properties.



In [3]:

    
# Show the info of the data read



In [4]:

    
# Get the name of the columns



In [5]:

    
# Get just the values of a particular column



In [6]:

    
# get the first element

Astropy can read a variety of formats easily. The following example uses a quite



In [7]:

    
# Read the data from the source
table = ascii.read("ftp://cdsarc.u-strasbg.fr/pub/cats/VII/253/snrs.dat",
                   readme="ftp://cdsarc.u-strasbg.fr/pub/cats/VII/253/ReadMe")



In [8]:

    
# See the stats of the table



In [9]:

    
# If we want to see the first 10 entries



In [10]:

    
# the units are also stored, we can extract them too



In [11]:

    
# Adding values of different columns



In [12]:

    
# adding values of different columns but being aware of column units



In [13]:

    
# Create a new column in the table



In [14]:

    
# Show table's new column



In [15]:

    
# add a description to the new column



In [16]:

    
# Now it does show the values



In [17]:

    
# Using numpy to calculate the sin of the RA



In [18]:

    
# Let's change the units...



In [19]:

    
# does the sin now works?

Properties when reading

the reading of the table has many properties, let's imagine the following easy example:



In [20]:

    
weather_data = """
# Country = Finland
# City = Helsinki
# Longitud = 24.9375
# Latitud = 60.170833
# Week = 32
# Year = 2015
day, precip, type
Mon,1.5,rain
Tues,,
Wed,1.1,snow
Thur,2.3,rain
Fri,0.2,
Sat,1.1,snow
Sun,5.4,snow
"""



In [21]:

    
# Read the table



In [22]:

    
# Blank values are interpreted by default as bad/missing values



In [23]:

    
# Let's define missing values for the columns we want:



In [24]:

    
# Use filled to show the value filled.



In [25]:

    
# We can see the meta as a dictionary, but not as key, value pairs



In [26]:

    
# To get it the header as a table

When the values are not empty, then the keyword fill_values on read has to be used.

Reading VOTables

VOTables are an special type of tables which should be self-consistent and can be tied to a particular scheme. This mean the file will contain where the data comes from (and which query produced it) and the properties for each field, making it easier to ingest by a machine.



In [27]:

    
from astropy.io.votable import parse_single_table



In [28]:

    
# Read the example table from HELIO (hfc_ar.xml)



In [29]:

    
# See the fields of the table



In [30]:

    
# extract one  (NOAA_NUMBER) or all of the columns



In [31]:

    
# Show the data



In [32]:

    
# See the mask



In [33]:

    
# Shee the whole array.



In [34]:

    
# Convert the table to an astropy table



In [35]:

    
# See the table



In [36]:

    
# Different results because quantities are not



In [37]:

    
# And it can also be converted to other units