notebook.community

Edit and run

xray + dask

This was modified from a notebook originally written by Stephan Hoyer

Weather data -- especially the results of numerical weather simulations -- is big. Some of the biggest super computers make weather forecasts, and they save their output on increasingly high resolution grids. Even for data analysis purposes, it's easy to need to process 10s or 100s of GB of data.

There are many excellent tools for working with weather data, which is usually stored in the netCDF file format. Many of these have support for out-of-core data, notably including the command line tools NCO and CDO. There are even Python tools, including a netCDF4 library and Iris. However, none of these tools matched the ease of use of pandas. We knew there there was a better way, so we decided to write xray, a library for working with multi-dimensional labeled data.

The latest release of xray includes support for processing datasets that don't fit into memory using dask, a new Python library that extends NumPy to out-of-core datasets by blocking arrays into small chunks and using a simple task scheduling abstraction. Dask allows xray to easily process out of core data and simultaneously make use of all our CPUs resources.

Loading data

First, we'll import dask and setup a ThreadPool for processing tasks. Dask currently doesn't do this automatically.





In [1]:



    


import xray
import dask.array as da
import numpy as np

import dask

We'll use the new xray.open_mfdataset function to open archived weather data from ECMWF. It opens a glob of netCDF files on my local disk and automatically infers how to combine them into a few logical arrays by reading their metadata:





In [1]:



    


!ls /home/mrocklin/data/ecmwf/*.nc3



    


















    






/home/mrocklin/data/ecmwf/2014-01-01.nc3
/home/mrocklin/data/ecmwf/2014-01-02.nc3
/home/mrocklin/data/ecmwf/2014-01-03.nc3
/home/mrocklin/data/ecmwf/2014-01-04.nc3
/home/mrocklin/data/ecmwf/2014-01-05.nc3
/home/mrocklin/data/ecmwf/2014-01-06.nc3
/home/mrocklin/data/ecmwf/2014-01-07.nc3
/home/mrocklin/data/ecmwf/2014-01-08.nc3
/home/mrocklin/data/ecmwf/2014-01-09.nc3
/home/mrocklin/data/ecmwf/2014-01-10.nc3
/home/mrocklin/data/ecmwf/2014-01-11.nc3
/home/mrocklin/data/ecmwf/2014-01-12.nc3
/home/mrocklin/data/ecmwf/2014-01-13.nc3
/home/mrocklin/data/ecmwf/2014-01-14.nc3
/home/mrocklin/data/ecmwf/2014-01-15.nc3
/home/mrocklin/data/ecmwf/2014-01-16.nc3
/home/mrocklin/data/ecmwf/2014-01-17.nc3
/home/mrocklin/data/ecmwf/2014-01-18.nc3
/home/mrocklin/data/ecmwf/2014-01-19.nc3
/home/mrocklin/data/ecmwf/2014-01-20.nc3
/home/mrocklin/data/ecmwf/2014-01-21.nc3
/home/mrocklin/data/ecmwf/2014-01-22.nc3
/home/mrocklin/data/ecmwf/2014-01-23.nc3
/home/mrocklin/data/ecmwf/2014-01-24.nc3
/home/mrocklin/data/ecmwf/2014-01-25.nc3
/home/mrocklin/data/ecmwf/2014-01-26.nc3
/home/mrocklin/data/ecmwf/2014-01-27.nc3
/home/mrocklin/data/ecmwf/2014-01-28.nc3
/home/mrocklin/data/ecmwf/2014-01-29.nc3
/home/mrocklin/data/ecmwf/2014-01-30.nc3
/home/mrocklin/data/ecmwf/2014-01-31.nc3
/home/mrocklin/data/ecmwf/2014-02-01.nc3
/home/mrocklin/data/ecmwf/2014-02-02.nc3
/home/mrocklin/data/ecmwf/2014-02-03.nc3
/home/mrocklin/data/ecmwf/2014-02-04.nc3
/home/mrocklin/data/ecmwf/2014-02-05.nc3
/home/mrocklin/data/ecmwf/2014-02-06.nc3
/home/mrocklin/data/ecmwf/2014-02-07.nc3
/home/mrocklin/data/ecmwf/2014-02-08.nc3
/home/mrocklin/data/ecmwf/2014-02-09.nc3
/home/mrocklin/data/ecmwf/2014-02-10.nc3
/home/mrocklin/data/ecmwf/2014-02-11.nc3
/home/mrocklin/data/ecmwf/2014-02-12.nc3
/home/mrocklin/data/ecmwf/2014-02-13.nc3
/home/mrocklin/data/ecmwf/2014-02-14.nc3
/home/mrocklin/data/ecmwf/2014-02-15.nc3
/home/mrocklin/data/ecmwf/2014-02-16.nc3
/home/mrocklin/data/ecmwf/2014-02-17.nc3
/home/mrocklin/data/ecmwf/2014-02-18.nc3
/home/mrocklin/data/ecmwf/2014-02-19.nc3
/home/mrocklin/data/ecmwf/2014-02-20.nc3
/home/mrocklin/data/ecmwf/2014-02-21.nc3
/home/mrocklin/data/ecmwf/2014-02-22.nc3
/home/mrocklin/data/ecmwf/2014-02-23.nc3
/home/mrocklin/data/ecmwf/2014-02-24.nc3
/home/mrocklin/data/ecmwf/2014-02-25.nc3
/home/mrocklin/data/ecmwf/2014-02-26.nc3
/home/mrocklin/data/ecmwf/2014-02-27.nc3
/home/mrocklin/data/ecmwf/2014-02-28.nc3
/home/mrocklin/data/ecmwf/2014-03-01.nc3
/home/mrocklin/data/ecmwf/2014-03-02.nc3
/home/mrocklin/data/ecmwf/2014-03-03.nc3
/home/mrocklin/data/ecmwf/2014-03-04.nc3
/home/mrocklin/data/ecmwf/2014-03-05.nc3
/home/mrocklin/data/ecmwf/2014-03-06.nc3
/home/mrocklin/data/ecmwf/2014-03-07.nc3
/home/mrocklin/data/ecmwf/2014-03-08.nc3
/home/mrocklin/data/ecmwf/2014-03-09.nc3
/home/mrocklin/data/ecmwf/2014-03-10.nc3
/home/mrocklin/data/ecmwf/2014-03-11.nc3
/home/mrocklin/data/ecmwf/2014-03-12.nc3
/home/mrocklin/data/ecmwf/2014-03-13.nc3
/home/mrocklin/data/ecmwf/2014-03-14.nc3
/home/mrocklin/data/ecmwf/2014-03-15.nc3
/home/mrocklin/data/ecmwf/2014-03-16.nc3
/home/mrocklin/data/ecmwf/2014-03-17.nc3
/home/mrocklin/data/ecmwf/2014-03-18.nc3
/home/mrocklin/data/ecmwf/2014-03-19.nc3
/home/mrocklin/data/ecmwf/2014-03-20.nc3
/home/mrocklin/data/ecmwf/2014-03-21.nc3
/home/mrocklin/data/ecmwf/2014-03-22.nc3
/home/mrocklin/data/ecmwf/2014-03-23.nc3
/home/mrocklin/data/ecmwf/2014-03-24.nc3
/home/mrocklin/data/ecmwf/2014-03-25.nc3
/home/mrocklin/data/ecmwf/2014-03-26.nc3
/home/mrocklin/data/ecmwf/2014-03-27.nc3
/home/mrocklin/data/ecmwf/2014-03-28.nc3
/home/mrocklin/data/ecmwf/2014-03-29.nc3
/home/mrocklin/data/ecmwf/2014-03-30.nc3
/home/mrocklin/data/ecmwf/2014-03-31.nc3
/home/mrocklin/data/ecmwf/2014-04-01.nc3
/home/mrocklin/data/ecmwf/2014-04-02.nc3
/home/mrocklin/data/ecmwf/2014-04-03.nc3
/home/mrocklin/data/ecmwf/2014-04-04.nc3
/home/mrocklin/data/ecmwf/2014-04-05.nc3
/home/mrocklin/data/ecmwf/2014-04-06.nc3
/home/mrocklin/data/ecmwf/2014-04-07.nc3
/home/mrocklin/data/ecmwf/2014-04-08.nc3
/home/mrocklin/data/ecmwf/2014-04-09.nc3
/home/mrocklin/data/ecmwf/2014-04-10.nc3
/home/mrocklin/data/ecmwf/2014-04-11.nc3
/home/mrocklin/data/ecmwf/2014-04-12.nc3
/home/mrocklin/data/ecmwf/2014-04-13.nc3
/home/mrocklin/data/ecmwf/2014-04-14.nc3
/home/mrocklin/data/ecmwf/2014-04-15.nc3
/home/mrocklin/data/ecmwf/2014-04-16.nc3
/home/mrocklin/data/ecmwf/2014-04-17.nc3
/home/mrocklin/data/ecmwf/2014-04-18.nc3
/home/mrocklin/data/ecmwf/2014-04-19.nc3
/home/mrocklin/data/ecmwf/2014-04-20.nc3
/home/mrocklin/data/ecmwf/2014-04-21.nc3
/home/mrocklin/data/ecmwf/2014-04-22.nc3
/home/mrocklin/data/ecmwf/2014-04-23.nc3
/home/mrocklin/data/ecmwf/2014-04-24.nc3
/home/mrocklin/data/ecmwf/2014-04-25.nc3
/home/mrocklin/data/ecmwf/2014-04-26.nc3
/home/mrocklin/data/ecmwf/2014-04-27.nc3
/home/mrocklin/data/ecmwf/2014-04-28.nc3
/home/mrocklin/data/ecmwf/2014-04-29.nc3
/home/mrocklin/data/ecmwf/2014-04-30.nc3
/home/mrocklin/data/ecmwf/2014-05-01.nc3
/home/mrocklin/data/ecmwf/2014-05-02.nc3
/home/mrocklin/data/ecmwf/2014-05-03.nc3
/home/mrocklin/data/ecmwf/2014-05-04.nc3
/home/mrocklin/data/ecmwf/2014-05-05.nc3
/home/mrocklin/data/ecmwf/2014-05-06.nc3
/home/mrocklin/data/ecmwf/2014-05-07.nc3
/home/mrocklin/data/ecmwf/2014-05-08.nc3
/home/mrocklin/data/ecmwf/2014-05-09.nc3
/home/mrocklin/data/ecmwf/2014-05-10.nc3
/home/mrocklin/data/ecmwf/2014-05-11.nc3
/home/mrocklin/data/ecmwf/2014-05-12.nc3
/home/mrocklin/data/ecmwf/2014-05-13.nc3
/home/mrocklin/data/ecmwf/2014-05-14.nc3
/home/mrocklin/data/ecmwf/2014-05-15.nc3
/home/mrocklin/data/ecmwf/2014-05-16.nc3
/home/mrocklin/data/ecmwf/2014-05-17.nc3
/home/mrocklin/data/ecmwf/2014-05-18.nc3
/home/mrocklin/data/ecmwf/2014-05-19.nc3
/home/mrocklin/data/ecmwf/2014-05-20.nc3
/home/mrocklin/data/ecmwf/2014-05-21.nc3
/home/mrocklin/data/ecmwf/2014-05-22.nc3
/home/mrocklin/data/ecmwf/2014-05-23.nc3
/home/mrocklin/data/ecmwf/2014-05-24.nc3
/home/mrocklin/data/ecmwf/2014-05-25.nc3
/home/mrocklin/data/ecmwf/2014-05-26.nc3
/home/mrocklin/data/ecmwf/2014-05-27.nc3
/home/mrocklin/data/ecmwf/2014-05-28.nc3
/home/mrocklin/data/ecmwf/2014-05-29.nc3
/home/mrocklin/data/ecmwf/2014-05-30.nc3
/home/mrocklin/data/ecmwf/2014-05-31.nc3
/home/mrocklin/data/ecmwf/2014-06-01.nc3
/home/mrocklin/data/ecmwf/2014-06-02.nc3
/home/mrocklin/data/ecmwf/2014-06-03.nc3
/home/mrocklin/data/ecmwf/2014-06-04.nc3
/home/mrocklin/data/ecmwf/2014-06-05.nc3
/home/mrocklin/data/ecmwf/2014-06-06.nc3
/home/mrocklin/data/ecmwf/2014-06-07.nc3
/home/mrocklin/data/ecmwf/2014-06-08.nc3
/home/mrocklin/data/ecmwf/2014-06-09.nc3
/home/mrocklin/data/ecmwf/2014-06-10.nc3
/home/mrocklin/data/ecmwf/2014-06-11.nc3
/home/mrocklin/data/ecmwf/2014-06-12.nc3
/home/mrocklin/data/ecmwf/2014-06-13.nc3
/home/mrocklin/data/ecmwf/2014-06-14.nc3
/home/mrocklin/data/ecmwf/2014-06-15.nc3
/home/mrocklin/data/ecmwf/2014-06-16.nc3
/home/mrocklin/data/ecmwf/2014-06-17.nc3
/home/mrocklin/data/ecmwf/2014-06-18.nc3
/home/mrocklin/data/ecmwf/2014-06-19.nc3
/home/mrocklin/data/ecmwf/2014-06-20.nc3
/home/mrocklin/data/ecmwf/2014-06-21.nc3
/home/mrocklin/data/ecmwf/2014-06-22.nc3
/home/mrocklin/data/ecmwf/2014-06-23.nc3
/home/mrocklin/data/ecmwf/2014-06-24.nc3
/home/mrocklin/data/ecmwf/2014-06-25.nc3
/home/mrocklin/data/ecmwf/2014-06-26.nc3
/home/mrocklin/data/ecmwf/2014-06-27.nc3
/home/mrocklin/data/ecmwf/2014-06-28.nc3
/home/mrocklin/data/ecmwf/2014-06-29.nc3
/home/mrocklin/data/ecmwf/2014-06-30.nc3
/home/mrocklin/data/ecmwf/2014-07-01.nc3
/home/mrocklin/data/ecmwf/2014-07-02.nc3
/home/mrocklin/data/ecmwf/2014-07-03.nc3
/home/mrocklin/data/ecmwf/2014-07-04.nc3
/home/mrocklin/data/ecmwf/2014-07-05.nc3
/home/mrocklin/data/ecmwf/2014-07-06.nc3
/home/mrocklin/data/ecmwf/2014-07-07.nc3
/home/mrocklin/data/ecmwf/2014-07-08.nc3
/home/mrocklin/data/ecmwf/2014-07-09.nc3
/home/mrocklin/data/ecmwf/2014-07-10.nc3
/home/mrocklin/data/ecmwf/2014-07-11.nc3
/home/mrocklin/data/ecmwf/2014-07-12.nc3
/home/mrocklin/data/ecmwf/2014-07-13.nc3
/home/mrocklin/data/ecmwf/2014-07-14.nc3
/home/mrocklin/data/ecmwf/2014-07-15.nc3
/home/mrocklin/data/ecmwf/2014-07-16.nc3
/home/mrocklin/data/ecmwf/2014-07-17.nc3
/home/mrocklin/data/ecmwf/2014-07-18.nc3
/home/mrocklin/data/ecmwf/2014-07-19.nc3
/home/mrocklin/data/ecmwf/2014-07-20.nc3
/home/mrocklin/data/ecmwf/2014-07-21.nc3
/home/mrocklin/data/ecmwf/2014-07-22.nc3
/home/mrocklin/data/ecmwf/2014-07-23.nc3
/home/mrocklin/data/ecmwf/2014-07-24.nc3
/home/mrocklin/data/ecmwf/2014-07-25.nc3
/home/mrocklin/data/ecmwf/2014-07-26.nc3
/home/mrocklin/data/ecmwf/2014-07-27.nc3
/home/mrocklin/data/ecmwf/2014-07-28.nc3
/home/mrocklin/data/ecmwf/2014-07-29.nc3
/home/mrocklin/data/ecmwf/2014-07-30.nc3
/home/mrocklin/data/ecmwf/2014-07-31.nc3
/home/mrocklin/data/ecmwf/2014-08-01.nc3
/home/mrocklin/data/ecmwf/2014-08-02.nc3
/home/mrocklin/data/ecmwf/2014-08-03.nc3
/home/mrocklin/data/ecmwf/2014-08-04.nc3
/home/mrocklin/data/ecmwf/2014-08-05.nc3
/home/mrocklin/data/ecmwf/2014-08-06.nc3
/home/mrocklin/data/ecmwf/2014-08-07.nc3
/home/mrocklin/data/ecmwf/2014-08-08.nc3
/home/mrocklin/data/ecmwf/2014-08-09.nc3
/home/mrocklin/data/ecmwf/2014-08-10.nc3
/home/mrocklin/data/ecmwf/2014-08-11.nc3
/home/mrocklin/data/ecmwf/2014-08-12.nc3
/home/mrocklin/data/ecmwf/2014-08-13.nc3
/home/mrocklin/data/ecmwf/2014-08-14.nc3
/home/mrocklin/data/ecmwf/2014-08-15.nc3
/home/mrocklin/data/ecmwf/2014-08-16.nc3
/home/mrocklin/data/ecmwf/2014-08-17.nc3
/home/mrocklin/data/ecmwf/2014-08-18.nc3
/home/mrocklin/data/ecmwf/2014-08-19.nc3
/home/mrocklin/data/ecmwf/2014-08-20.nc3
/home/mrocklin/data/ecmwf/2014-08-21.nc3
/home/mrocklin/data/ecmwf/2014-08-22.nc3
/home/mrocklin/data/ecmwf/2014-08-23.nc3
/home/mrocklin/data/ecmwf/2014-08-24.nc3
/home/mrocklin/data/ecmwf/2014-08-25.nc3
/home/mrocklin/data/ecmwf/2014-08-26.nc3
/home/mrocklin/data/ecmwf/2014-08-27.nc3
/home/mrocklin/data/ecmwf/2014-08-28.nc3
/home/mrocklin/data/ecmwf/2014-08-29.nc3
/home/mrocklin/data/ecmwf/2014-08-30.nc3
/home/mrocklin/data/ecmwf/2014-08-31.nc3
/home/mrocklin/data/ecmwf/2014-09-01.nc3
/home/mrocklin/data/ecmwf/2014-09-02.nc3
/home/mrocklin/data/ecmwf/2014-09-03.nc3
/home/mrocklin/data/ecmwf/2014-09-04.nc3
/home/mrocklin/data/ecmwf/2014-09-05.nc3
/home/mrocklin/data/ecmwf/2014-09-06.nc3
/home/mrocklin/data/ecmwf/2014-09-07.nc3
/home/mrocklin/data/ecmwf/2014-09-08.nc3
/home/mrocklin/data/ecmwf/2014-09-09.nc3
/home/mrocklin/data/ecmwf/2014-09-10.nc3
/home/mrocklin/data/ecmwf/2014-09-11.nc3
/home/mrocklin/data/ecmwf/2014-09-12.nc3
/home/mrocklin/data/ecmwf/2014-09-13.nc3
/home/mrocklin/data/ecmwf/2014-09-14.nc3
/home/mrocklin/data/ecmwf/2014-09-15.nc3
/home/mrocklin/data/ecmwf/2014-09-16.nc3
/home/mrocklin/data/ecmwf/2014-09-17.nc3
/home/mrocklin/data/ecmwf/2014-09-18.nc3
/home/mrocklin/data/ecmwf/2014-09-19.nc3
/home/mrocklin/data/ecmwf/2014-09-20.nc3
/home/mrocklin/data/ecmwf/2014-09-21.nc3
/home/mrocklin/data/ecmwf/2014-09-22.nc3
/home/mrocklin/data/ecmwf/2014-09-23.nc3
/home/mrocklin/data/ecmwf/2014-09-24.nc3
/home/mrocklin/data/ecmwf/2014-09-25.nc3
/home/mrocklin/data/ecmwf/2014-09-26.nc3
/home/mrocklin/data/ecmwf/2014-09-27.nc3
/home/mrocklin/data/ecmwf/2014-09-28.nc3
/home/mrocklin/data/ecmwf/2014-09-29.nc3
/home/mrocklin/data/ecmwf/2014-09-30.nc3
/home/mrocklin/data/ecmwf/2014-10-01.nc3
/home/mrocklin/data/ecmwf/2014-10-02.nc3
/home/mrocklin/data/ecmwf/2014-10-03.nc3
/home/mrocklin/data/ecmwf/2014-10-04.nc3
/home/mrocklin/data/ecmwf/2014-10-05.nc3
/home/mrocklin/data/ecmwf/2014-10-06.nc3
/home/mrocklin/data/ecmwf/2014-10-07.nc3
/home/mrocklin/data/ecmwf/2014-10-08.nc3
/home/mrocklin/data/ecmwf/2014-10-09.nc3
/home/mrocklin/data/ecmwf/2014-10-10.nc3
/home/mrocklin/data/ecmwf/2014-10-11.nc3
/home/mrocklin/data/ecmwf/2014-10-12.nc3
/home/mrocklin/data/ecmwf/2014-10-13.nc3
/home/mrocklin/data/ecmwf/2014-10-14.nc3
/home/mrocklin/data/ecmwf/2014-10-15.nc3
/home/mrocklin/data/ecmwf/2014-10-16.nc3
/home/mrocklin/data/ecmwf/2014-10-17.nc3
/home/mrocklin/data/ecmwf/2014-10-18.nc3
/home/mrocklin/data/ecmwf/2014-10-19.nc3
/home/mrocklin/data/ecmwf/2014-10-20.nc3
/home/mrocklin/data/ecmwf/2014-10-21.nc3
/home/mrocklin/data/ecmwf/2014-10-22.nc3
/home/mrocklin/data/ecmwf/2014-10-23.nc3
/home/mrocklin/data/ecmwf/2014-10-24.nc3
/home/mrocklin/data/ecmwf/2014-10-25.nc3
/home/mrocklin/data/ecmwf/2014-10-26.nc3
/home/mrocklin/data/ecmwf/2014-10-27.nc3
/home/mrocklin/data/ecmwf/2014-10-28.nc3
/home/mrocklin/data/ecmwf/2014-10-29.nc3
/home/mrocklin/data/ecmwf/2014-10-30.nc3
/home/mrocklin/data/ecmwf/2014-10-31.nc3
/home/mrocklin/data/ecmwf/2014-11-01.nc3
/home/mrocklin/data/ecmwf/2014-11-02.nc3
/home/mrocklin/data/ecmwf/2014-11-03.nc3
/home/mrocklin/data/ecmwf/2014-11-04.nc3
/home/mrocklin/data/ecmwf/2014-11-05.nc3
/home/mrocklin/data/ecmwf/2014-11-06.nc3
/home/mrocklin/data/ecmwf/2014-11-07.nc3
/home/mrocklin/data/ecmwf/2014-11-08.nc3
/home/mrocklin/data/ecmwf/2014-11-09.nc3
/home/mrocklin/data/ecmwf/2014-11-10.nc3
/home/mrocklin/data/ecmwf/2014-11-11.nc3
/home/mrocklin/data/ecmwf/2014-11-12.nc3
/home/mrocklin/data/ecmwf/2014-11-13.nc3
/home/mrocklin/data/ecmwf/2014-11-14.nc3
/home/mrocklin/data/ecmwf/2014-11-15.nc3
/home/mrocklin/data/ecmwf/2014-11-16.nc3
/home/mrocklin/data/ecmwf/2014-11-17.nc3
/home/mrocklin/data/ecmwf/2014-11-18.nc3
/home/mrocklin/data/ecmwf/2014-11-19.nc3
/home/mrocklin/data/ecmwf/2014-11-20.nc3
/home/mrocklin/data/ecmwf/2014-11-21.nc3
/home/mrocklin/data/ecmwf/2014-11-22.nc3
/home/mrocklin/data/ecmwf/2014-11-23.nc3
/home/mrocklin/data/ecmwf/2014-11-24.nc3
/home/mrocklin/data/ecmwf/2014-11-25.nc3
/home/mrocklin/data/ecmwf/2014-11-26.nc3
/home/mrocklin/data/ecmwf/2014-11-27.nc3
/home/mrocklin/data/ecmwf/2014-11-28.nc3
/home/mrocklin/data/ecmwf/2014-11-29.nc3
/home/mrocklin/data/ecmwf/2014-11-30.nc3
/home/mrocklin/data/ecmwf/2014-12-01.nc3
/home/mrocklin/data/ecmwf/2014-12-02.nc3
/home/mrocklin/data/ecmwf/2014-12-03.nc3
/home/mrocklin/data/ecmwf/2014-12-04.nc3
/home/mrocklin/data/ecmwf/2014-12-05.nc3
/home/mrocklin/data/ecmwf/2014-12-06.nc3
/home/mrocklin/data/ecmwf/2014-12-07.nc3
/home/mrocklin/data/ecmwf/2014-12-08.nc3
/home/mrocklin/data/ecmwf/2014-12-09.nc3
/home/mrocklin/data/ecmwf/2014-12-10.nc3
/home/mrocklin/data/ecmwf/2014-12-11.nc3
/home/mrocklin/data/ecmwf/2014-12-12.nc3
/home/mrocklin/data/ecmwf/2014-12-13.nc3
/home/mrocklin/data/ecmwf/2014-12-14.nc3
/home/mrocklin/data/ecmwf/2014-12-15.nc3
/home/mrocklin/data/ecmwf/2014-12-16.nc3
/home/mrocklin/data/ecmwf/2014-12-17.nc3
/home/mrocklin/data/ecmwf/2014-12-18.nc3
/home/mrocklin/data/ecmwf/2014-12-19.nc3
/home/mrocklin/data/ecmwf/2014-12-20.nc3
/home/mrocklin/data/ecmwf/2014-12-21.nc3
/home/mrocklin/data/ecmwf/2014-12-22.nc3
/home/mrocklin/data/ecmwf/2014-12-23.nc3
/home/mrocklin/data/ecmwf/2014-12-24.nc3
/home/mrocklin/data/ecmwf/2014-12-25.nc3
/home/mrocklin/data/ecmwf/2014-12-26.nc3
/home/mrocklin/data/ecmwf/2014-12-27.nc3
/home/mrocklin/data/ecmwf/2014-12-28.nc3
/home/mrocklin/data/ecmwf/2014-12-29.nc3
/home/mrocklin/data/ecmwf/2014-12-30.nc3
/home/mrocklin/data/ecmwf/2014-12-31.nc3
/home/mrocklin/data/ecmwf/test1.nc3

















In [2]:



    


ds = xray.open_mfdataset('/home/mrocklin/data/ecmwf/*.nc3', engine='scipy')



    


















    






---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-c6549176712c> in <module>()
----> 1 ds = xray.open_mfdataset('/home/mrocklin/data/ecmwf/*.nc3', engine='scipy')

/home/mrocklin/Software/anaconda/lib/python2.7/site-packages/xray-0.4.1_55_g651d48d-py2.7.egg/xray/backends/api.pyc in open_mfdataset(paths, blockshape, concat_dim, **kwargs)
    182     datasets = [open_dataset(p, **kwargs) for p in paths]
    183     file_objs = [ds._file_obj for ds in datasets]
--> 184     datasets = [ds.reblock(blockshape=blockshape) for ds in datasets]
    185     combined = auto_combine(datasets, concat_dim=concat_dim)
    186     combined._file_obj = _MultiFileCloser(file_objs)

/home/mrocklin/Software/anaconda/lib/python2.7/site-packages/xray-0.4.1_55_g651d48d-py2.7.egg/xray/core/dataset.pyc in reblock(self, blockdims, blockshape)
    923                                                 name=k)
    924                                       if v.ndim > 0 else v))
--> 925                                  for k, v in self.variables.items()])
    926         return self._replace_vars_and_dims(variables)
    927 

/home/mrocklin/Software/anaconda/lib/python2.7/site-packages/xray-0.4.1_55_g651d48d-py2.7.egg/xray/core/variable.pyc in reblock(self, blockdims, blockshape, name)
    466 
    467             data = da.from_array(data, blockdims=blockdims,
--> 468                                  blockshape=blockshape, name=name)
    469 
    470         return type(self)(self.dims, data, self._attrs, self._encoding,

TypeError: from_array() takes at least 2 arguments (2 given)
















In [ ]:



    


ds

11 GB of Data





In [4]:



    


np.prod(ds.dims.values()) * 8 * 2 ** -30



    


















    
Out[4]:








11.324758529663086

4GB of Memory





In [5]:



    


!cat /proc/meminfo | grep MemTotal



    


















    






MemTotal:        3766528 kB

Index with meaningful values, not numbers





In [6]:



    


# x.mean(2)
ds.mean('longitude')



    


















    
Out[6]:








<xray.Dataset>
Dimensions:   (latitude: 721, time: 1464)
Coordinates:
  * latitude  (latitude) >f4 90.0 89.75 89.5 89.25 89.0 88.75 88.5 88.25 88.0 87.75 87.5 87.25 ...
  * time      (time) datetime64[ns] 2014-01-01 2014-01-01T06:00:00 2014-01-01T12:00:00 ...
Data variables:
    u10       (time, latitude) float64 0.8483 0.9731 1.098 1.037 1.258 1.459 1.39 1.322 1.203 ...
    v10       (time, latitude) float64 -0.282 -0.2981 -0.3142 -0.3064 -0.3349 -0.3596 -0.3326 ...
    t2m       (time, latitude) float64 254.0 253.9 253.8 253.9 253.8 253.6 253.4 253.1 252.8 ...

















In [7]:



    


ds.sel(time="2014-04", latitude=(ds.latitude > 10 & (ds.latitude < 40)))



    


















    
Out[7]:








<xray.Dataset>
Dimensions:    (latitude: 360, longitude: 1440, time: 120)
Coordinates:
  * longitude  (longitude) >f4 0.0 0.25 0.5 0.75 1.0 1.25 1.5 1.75 2.0 2.25 2.5 2.75 3.0 3.25 ...
  * latitude   (latitude) >f4 90.0 89.75 89.5 89.25 89.0 88.75 88.5 88.25 88.0 87.75 87.5 87.25 ...
  * time       (time) datetime64[ns] 2014-04-01 2014-04-01T06:00:00 2014-04-01T12:00:00 ...
Data variables:
    u10        (time, latitude, longitude) float64 2.715 2.683 2.65 2.616 2.583 2.551 2.518 ...
    v10        (time, latitude, longitude) float64 -7.506 -7.511 -7.517 -7.523 -7.529 -7.535 ...
    t2m        (time, latitude, longitude) float64 254.1 254.1 254.1 254.1 254.1 254.1 254.1 ...
Attributes:
    Conventions: CF-1.0
    history: 2015-02-11 16:45:42 GMT by grib_to_netcdf-1.13.0: grib_to_netcdf /data/data01/mars-web230-20150211164538-19989-9003.target -o /data/data01/mars-web230-20150211164541-19989-9004.nc

Groupby operations and datetime handling





In [8]:



    


%time ds.groupby('time.month').mean('time').load_data()



    


















    






CPU times: user 5min 3s, sys: 36.4 s, total: 5min 39s
Wall time: 1min 53s










    
Out[8]:








<xray.Dataset>
Dimensions:    (latitude: 721, longitude: 1440, month: 12)
Coordinates:
  * longitude  (longitude) >f4 0.0 0.25 0.5 0.75 1.0 1.25 1.5 1.75 2.0 2.25 2.5 2.75 3.0 3.25 ...
  * latitude   (latitude) >f4 90.0 89.75 89.5 89.25 89.0 88.75 88.5 88.25 88.0 87.75 87.5 87.25 ...
  * month      (month) int64 1 2 3 4 5 6 7 8 9 10 11 12
Data variables:
    u10        (month, latitude, longitude) float64 -2.758 -2.749 -2.74 -2.731 -2.722 -2.713 ...
    v10        (month, latitude, longitude) float64 1.707 1.718 1.729 1.741 1.752 1.764 1.775 ...
    t2m        (month, latitude, longitude) float64 252.4 252.4 252.4 252.4 252.4 252.4 252.4 ...

Bandwidth





In [9]:



    


11e9 / 113 / 1e6 # MB/s



    


















    
Out[9]:








97.34513274336283

Content source: xlhtc007/blaze

Similar notebooks:

notebook.community | gallery | about