Reading MODIS level 1b data in hdf5 format

Here is a walkthrough of how to read metadata and data from a level1b hdf5 file. I've used the the converter I downloaded from http://hdfeos.org/software/h4toh5/bin/ (for linux and osx) and http://www.hdfgroup.org/h4toh5/download.html (Windows and linux64) to convert MYD021KM.A2005188.0405.005.2009232180906.hdf to hdf5 and put it in the folder at http://clouds.eos.ubc.ca/~phil/Downloads/a301/MYD021KM.A2005188.0405.005.2009232180906.h5 (right click to save to your local drive (147 Mbytes))

Convert HDF4 to HDF5 on Windows

Windows user can convert from HDF4 to HDF5 using h4toh5 provided by the HDF group.

Here are the steps:

Download h4h5tools-2.2.2-win32.zip or h4h5tools-2.2.2-win64.zip on your machine.
Uncompress the *.zip file, run installation file *.exe
Launch cmd.exe from the start menu
cd to bin folder (e.g. cd F:\HDF_Group\H4TOH5\2.2.2\bin)

use h4toh5 tool in the following ways:

h4toh5convert input.hdf output.h5
      h4toh5convert -na input.hdf output.h5
      h4toh5convert -nc4 input.hdf output.h5`

type h4toh5convert -h in a command window to see the full help file
You can add h4toh5 on your system's Environmental Variable PATH, or set PATH=F:\HDF_Group\H4TOH5\2.2.2\bin and then you can use h4toh5convert everywhere.

Convert HDF4 to HDF5 on OSX

To download h4toh5 from http://hdfeos.org/software/h4toh5/bin/mac right click on h4toh5 and save as a binary file open a terminal, cd to the directory containing h4toh5 and make it an executable

chmod a+x h4toh5

add this directory to the PATH environmental variable in your ~/.bashrc file to make it generally available, or run it by specifying the full path to h4toh5 like this:

/Users/phil/bin/h4toh5 *hdf

I'm going to use two new modules (lib/modismeta_h5.py and lib/h5dump.py) to read the granule



In [0]:

    
from __future__ import print_function
import os,site
import glob
import h5py
from IPython.display import Image
#
# add the lib folder to the path assuming it is on the same
# level as the notebooks folder
#
libdir=os.path.abspath('../lib')
site.addsitedir(libdir)
from modismeta_h5 import parseMeta
import h5dump
from h5dump import dumph5

the glob function finds a file using a wildcard to save typing (google: python glob wildcard)



In [0]:

    
h5_filename=glob.glob('../lib/MYD02*.h5')
print("found {}".format(h5_filename))



In [0]:

    
h5_file=h5py.File(h5_filename[0])

h5 files have attributes -- stored as a dictionary



In [0]:

    
print(h5_file.attrs.keys())



In [0]:

    
print(h5_file.attrs['Earth-Sun Distance_GLOSDS'])



In [0]:

    
print(h5_file.attrs['HDFEOSVersion_GLOSDS'])

h5 files have variables -- stored in a dictionary The fields are aranged in a hierarchy similar to a set of nested folders



In [0]:

    
reload(h5dump)
h5dump.dumph5(h5_file)

Tools like HDF Explorer or HDF Viewer. give a more complete view of HDF's file structure.



In [0]:

    
Image(url='http://clouds.eos.ubc.ca/~phil/courses/atsc301/text/HDF_file_structure.png')

Read the radiance data from MODIS_SWATH_Type_L1B/Data Fields/EV_1KM_Emissive



In [0]:

    
print(h5_file.keys())



In [0]:

    
print(h5_file['MODIS_SWATH_Type_L1B'].keys())



In [0]:

    
print(h5_file['MODIS_SWATH_Type_L1B']['Data Fields'].keys())



In [0]:

    
print(h5_file['MODIS_SWATH_Type_L1B']['Data Fields']['Band_1KM_Emissive'].shape)
print(h5_file['MODIS_SWATH_Type_L1B']['Data Fields']['Band_1KM_Emissive'][...])
print(h5_file['MODIS_SWATH_Type_L1B']['Data Fields']['EV_1KM_Emissive'].shape)

note that channel 31 occurs at index value 10



In [0]:

    
index31=10

the data is stored as unsigned, 2 byte integers which can hold values from 0 to $2^{16}$ - 1 = 65,535



In [0]:

    
chan31=h5_file['MODIS_SWATH_Type_L1B']['Data Fields']['EV_1KM_Emissive'][index31,:,:]
print(chan31.shape,chan31.dtype)



In [0]:

    
chan31[:3,:3]

we need to apply a scale and offset to convert to radiance (the netcdf module did this for us automatically

$Data = (RawData - offset) \times scale$

this information is included in the attributes of each variable.

(see page 36 of the Modis users guide )

here is the scale for all 16 channels



In [0]:

    
scale=h5_file['MODIS_SWATH_Type_L1B']['Data Fields']['EV_1KM_Emissive'].attrs['radiance_scales'][...]
print(scale)

and here is the offset for 16 channels



In [0]:

    
offset=h5_file['MODIS_SWATH_Type_L1B']['Data Fields']['EV_1KM_Emissive'].attrs['radiance_offsets'][...]
print(offset)



In [0]:

    
chan31=(chan31 - offset[index31])*scale[index31]



In [0]:

    
%matplotlib inline

histogram the calibrated radiances and show that they lie between 0-10 $W\,m^{-2}\,\mu m^{-1}\,sr^{-1}$



In [0]:

    
import matplotlib.pyplot as plt
out=plt.hist(chan31.flat)

Read MODIS_SWATH_Type_L1B/Geolocation Fields/Longitude

note that the longitude and latitude arrays are (406,271) while the actual data are (2030,1354). These lat/lon arrays show only every fifth row and column



In [0]:

    
long_data=h5_file['MODIS_SWATH_Type_L1B']['Geolocation Fields']['Longitude']



In [0]:

    
print(long_data.shape)



In [0]:

    
print(long_data[:3,:3])

Dump a selection of metadata



In [0]:

    
from modismeta_h5 import parseMeta



In [0]:

    
print(parseMeta(h5_file))