Last updated: June 29th 2016

Climate data exploration: a journey through Pandas

Welcome to a demo of Python's data analysis package called Pandas. Our goal is to learn about Data Analysis and transformation using Pandas while exploring datasets used to analyze climate change.

The story

The global goal of this demo is to provide the tools to be able to try and reproduce some of the analysis done in the IPCC global climate reports published in the last decade (see for example https://www.ipcc.ch/pdf/assessment-report/ar5/syr/SYR_AR5_FINAL_full.pdf).

We are first going to load a few public datasets containing information about global temperature, global and local sea level infomation, and global concentration of greenhouse gases like CO2, to see if there are correlations and how the trends are to evolve, assuming no fundamental change in the system. For all these datasets, we will download them, visualize them, clean them, search through them, merge them, resample them, transform them and summarize them.

In the process, we will learn about:

Part 1:

1. Loading data
2. Pandas datastructures
3. Cleaning and formatting data
4. Basic visualization

Part 2:

5. Accessing data
6. Working with dates and times
7. Transforming datasets
8. Statistical analysis
9. Data agregation and summarization
10. Correlations and regressions
11. Predictions from auto regression models

Some initial setup

1. Loading data

More details, see http://pandas.pydata.org/pandas-docs/stable/io.html

To find all reading functions in pandas, ask ipython's tab completion:

From a local text file

Let's first load some temperature data which covers all lattitudes. Since read_table is supposed to do the job for a text file, let's just try it:

There is only 1 column! Let's try again stating that values are separated by any number of spaces:

There are columns but the column names are 1880 and -0.1591!

Since we only have 2 columns, one of which would be nicer to access the data (the year of the record), let's try using the index_col option:

Last step: the index is made of dates. Let's make that explicit:

From a chunked file

Since every dataset can contain mistakes, let's load a different file with temperature data. NASA's GISS dataset is written in chunks: look at it in data/temperatures/GLB.Ts+dSST.txt

QUIZ: What happens if you remove the skiprows? skipfooter? engine?

EXERCISE: Load some readings of CO2 concentrations in the atmosphere from the data/greenhouse_gaz/co2_mm_global.txt data file.

From a remote text file

So far, we have only loaded temperature datasets. Climate change also affects the sea levels on the globe. Let's load some datasets with the sea levels. The university of colorado posts updated timeseries for mean sea level globably, per hemisphere, or even per ocean, sea, ... Let's download the global one, and the ones for the northern and southern hemisphere.

That will also illustrate that to load text files that are online, there is no more work than replacing the filepath by a URL n read_table:

There are clearly lots of cleanup to be done on these datasets. See below...

From a local or remote HTML file

To be able to grab more local data about mean sea levels, we can download and extract data about mean sea level stations around the world from the PSMSL (http://www.psmsl.org/). Again to download and parse all tables in a webpage, just give read_html the URL to parse:

That table can be used to search for a station in a region of the world we choose, extract an ID for it and download the corresponding time series with the URL http://www.psmsl.org/data/obtaining/met.monthly.data/< ID >.metdata

2. Pandas DataStructures

For more details, see http://pandas.pydata.org/pandas-docs/stable/dsintro.html

Now that we have used read_** functions to load datasets, we need to understand better what kind of objects we got from them to learn to work with them.

DataFrame, the pandas 2D structure

Descriptors for the vertical axis (axis=0)

Descriptors for the horizontal axis (axis=1)

A lot of information at once including memory usage:

Series, the pandas 1D structure

A series can be constructed with the pd.Series constructor (passing a list or array of values) or from a DataFrame, by extracting one of its columns.

Core attributes/information:

Probably the most important attribute of a Series or DataFrame is its index since we will use that to, well, index into the structures to access te information:

NumPy arrays as backend of Pandas

It is always possible to fall back to a good old NumPy array to pass on to scientific libraries that need them: SciPy, scikit-learn, ...

Creating new DataFrames manually

DataFrames can also be created manually, by grouping several Series together. Let's make a new frame from the 3 sea level datasets we downloaded above. They will be displayed along the same index. Wait, does that makes sense to do that?

So the northern hemisphere and southern hemisphere datasets are aligned. What about the global one?

For now, let's just build a DataFrame with the 2 hemisphere datasets then. We will come back to add the global one later...

Note: there are other ways to create DataFrames manually, for example from a 2D numpy array.

There is still the date in a regular column and a numerical index that is not that meaningful. We can specify the index of a DataFrame at creation. Let's try:

Now the fact that it is failing show that Pandas does auto-alignment of values: for each value of the index, it searches for a value in each Series that maps the same value. Since these series have a dumb numerical index, no values are found.

Since we know that the order of the values match the index we chose, we can replace the Series by their values only at creation of the DataFrame:

3. Cleaning and formatting data

The datasets that we obtain straight from the reading functions are pretty raw. A lot of pre-processing can be done during data read but we haven't used all the power of the reading functions. Let's learn to do a lot of cleaning and formatting of the data.

The GISS temperature dataset has a lot of issues too: useless numerical index, redundant columns, useless rows, placeholder (****) for missing values, and wrong type for the columns. Let's fix all this:

Renaming columns

We can also rename an index by setting its name. For example, the index of the mean_sea_level dataFrame could be called date since it contains more than just the year:

Setting missing values

In the full globe dataset, -999.00 was used to indicate that there was no value for that year. Let's search for all these values and replace them with the missing value that Pandas understand: np.nan

Choosing what is the index

Dropping rows and columns

Let's also set **** to a real missing value (np.nan). We can often do it using a boolean mask, but that may trigger pandas warning. Another way to assign based on a boolean condition is to use the where method:

Adding columns

While building the mean_sea_level dataFrame earlier, we didn't include the values from global_sea_level since the years were not aligned. Adding a column to a dataframe is as easy as adding an entry to a dictionary. So let's try:

The column is full of NaNs again because the auto-alignment feature of Pandas is searching for the index values like 1992.9323 in the index of global_sea_level["msl_ib_ns(mm)"] series and not finding them. Let's set its index to these years so that that auto-alignment can work for us and figure out which values we have and not:

EXERCISE: Create a new series containing the average of the 2 hemispheres minus the global value to see if that is close to 0. Work inside the mean_sea_level dataframe first. Then try with the original Series to see what happens with data alignment while doing computations.

Changing dtype of series

Now that the sea levels are looking pretty good, let's got back to the GISS temperature dataset. Because of the labels (strings) found in the middle of the timeseries, every column only assumed to contain strings (didn't convert them to floating point values):

That can be changed after the fact (and after the cleanup) with the astype method of a Series:

An index has a dtype just like any Series and that can be changed after the fact too.

For now, let's change it to an integer so that values can at least be compared properly. We will learn below to change it to a datetime object.

Removing missing values

Removing missing values - once they have been converted to np.nan - is very easy. Entries that contain missing values can be removed (dropped), or filled with many strategies.

Let's also mention the .interpolate method on a Series:

For now, we will leave the missing values in all our datasets, because it wouldn't be meaningful to fill them.

EXERCISE: Go back to the reading functions, and learn more about other options that could have allowed us to fold some of these pre-processing steps into the data loading.

4. Basic visualization

Now they have been formatted, visualizing your datasets is the next logical step and is trivial with Pandas. The first thing to try is to invoke the .plot to generate a basic visualization (uses matplotlib under the covers).

Line plots

Showing distributions information

QUIZ: How to list the possible kinds of plots that the plot method can allow?

Correlations

There are more plot options inside pandas.tools.plotting:

We will confirm the correlations we think we see further down...

5. Storing our work

For each read_** function to load data, there is a to_** method attached to Series and DataFrames.

EXERCISE: explore how the to_csv method work using ipython's ? and store the giss_temp dataframe. Do the same to store the full_globe_temp series to another file.

Another file format that is commonly used is Excel, and there multiple datasets can be stored in 1 file.

Another, more powerful file format to store binary data, which allows us to store both Series and DataFrames without having to cast anybody is HDF5.

EXERCISE: Add the greenhouse gas dataset in this data store. Store it in a separate folder.