Tutorials for Developing Scientific Workflows in Pycroscopy - Part 2: Writing to pycroscopy H5 files

Suhas Somnath

8/8/2017

This set of notebooks will serve as examples for developing end-to-end workflows for and using pycroscopy.

While pycroscopy contains many popular data processing function, it may not have a function you need. Since pycroscopy is data-centric, it is preferable to write processing results back to the same file as well.

In this example, we will write the results of K-Means clustering (on a Scanning Tunnelling Spectroscopy (STS) dataset) back to the file.

K-Means clustering is a quick and simple method to determine the types of spectral responses present in the data and their spatial occurance.

Introduction:

Data structuring and file format:

Before proceeding with this example, we highly recommend you read about the data formatting in pycroscopy as well as reading and writing to HDF5 files. We will summarize some key points below:

• pycroscopy uses the heirarchical data format (HDF5) files to store data
• These HDF5 or H5 files contain datasets and datagroups
• pycroscopy data files have two kinds of datasets:
  • main datasets: These must be of the form: [instance, features].
    • All imaging or measurement data satisfy this category, where positions form the instances and the spectral points form the features. Thus, even standard 2D images or a single spectra also satisfy this condition.
    • A collection of k chosen spectra would still satisfy this condition. Some examples include:
      • the cluster centers obtained from a clustering algorithm like k-Means clustering.
      • The abundance maps obtained from decomposition algorithms like Singular Value Decomposition (SVD) or Non-negetive matrix factorization (NMF)
  • ancillary datasets: All other datasets fall into this category. These include the frequency vector or bias vector as a function of which the main dataset was collected.
• pycroscopy stores all data in two dimensional matrices with all position dimensions collapsed to the first dimension and all other (spectroscopic) dimensions collapsed to the second dimension.
• All these main datasets are always accompanied by four ancillary datasets:
  • Position Indices
  • Position Values
  • Spectroscopic Indices
  • Spectroscopic Values
• These ancillary datasets are always two dimensional.
  • The Position datasets are NxM where N is the total number of positions and M is the number of position dimensions.
  • The Spectroscopic datasets are MxN where M is the number of spectroscopic dimensions and N is the total number os specstroscopic steps.
• All main datasets always have two attributes that describe the measurement itself:
  • quantity: The physical quantity contained in each cell of the dataset - such as voltage, current, force etc.
  • units: The units for the physical quantity such as V for volts, nA for nano amperes, pN for pico newtons etc.
• All main datasets additionally have 4 attributes that provide the references or links to the 4 aforementions ancillary datasets
  • Storing just the references allows us to re-use the same position / spectroscopic datasets without having to remake them
• For more information see the data format documentation

This bookkeeping is necesary for helping the code to understand the dimensionality and structure of the data. While these rules may seem tedious, there are several functions and a few classes that make these tasks much easier

Classes for writing files

In order to deal with the numerous challenges in writing data in a consistent manner, especially during translation, in the pycroscopy format, we developed two main classes: MicroData and ioHDF5.

MicroData

The abstract class MicroData is extended by MicroDataset and MicroDatagroup which are skeletal counterparts for the h5py.Dataset and h5py.Datagroup classes respectively. These classes allow programmers to quickly and simply set up the tree structure that needs to be written to H5 files without having to worry about the low-level HDF5 constructs or defensive programming strategies necessary for writing the H5 files. Besides facilitating the construction of a tree structure, each of the classes have a few features specific to pycroscopy to alleviate file writing.

ioHDF5

While we use h5py to read from pycroscopy files, the ioHDF5 class is used to write data to H5 files. ioHDF5 translates the tree structure described by a MicroDataGroup object and writes the contents to H5 files in a standardized manner. As a wrapper around h5py, tt handles the low-level file I/O calls and includes defensive programming strategies to minimize issues with writing to H5 files.

Why bother with Microdata and ioHDF5?

• These classes simplify the process of writing to H5 files considerably. The programmer only needs to construct the tree structure with simple python objects such as dictionaries for parameters, numpy datasets for storing data, etc.
• It is easy to corrupt H5 files. ioHDF5 uses defensive programming strategies to solve these problems.

Translation can be challenging in many cases:

• It may not be possible to read the entire data from the raw data file to memory as we did in the tutorial on Translation
  • ioHDF5 allows the general tree structure and the attributes to be written before the data is populated.
• Sometimes, the raw data files do not come with sufficient parameters that describe the size and shape of the data. This makes it challenging to prepare the H5 file.
  • ioHDF5 allows dataets to be dataFile I/O is expensive and we don't want to read the same raw data files multiple times

Loading the dataset

We wil start by downloading the raw data file as generated by the microscope and then translate the file to a pycroscopy H5 file.

Reading the H5 dataset

This data is a Scanning Tunnelling Spectroscopy (STS) dataset wherein current was measured as a function of voltage on a two dimensional grid of points. Thus, the data has three dimensions (X, Y, Bias). Note, that in pycroscopy, all position dimensions are collapsed to the first dimension and all spectroscopic (only bias in this case) dimensions are collapsed to the second axis of a two dimensional matrix. So, the data is represented as (position, bias) instead.

Performing k-Means Clustering:

Now that the data is loaded to memory, we can perform k-Means clustering on data_mat. As a reminder, K-Means clustering is a quick and simple method to determine the types of spectral responses present in the data and their spatial occurance.

Let us assume that we have a P x S dataset with P positions each with spectra that are S long. When K-Means is asked to identify k clusters, it will produce two results:

• cluster_centers: This contains the different kinds of spectral responses and is represented as a two dimensional array of the form [cluster number, representative spectra for this cluster]. Thus this dataset will have a shape of k x S
• labels: This provides the information about which spatial pixel belongs to which group. It will be a 1 dimensional array of size P wherein the value for each element in the array (cluster id for each pixel) will be within [0, k)

Our goal is to write back these two datasets to the H5 file

Visualize the results:

We will visualize both the raw results from k-Means as well as the distance-sorted results from pycroscopy. You will notice that the sorted results are easier to understand and interpret. This is an example of the kind of additional value that can be packed into pycroscopy wrappers on existing data analysis / processing functions.

A second example of value addition - The pycroscopy wrapper for Clustering handles real, complex, and compound valued datasets seamlessly in the background.

Preparing to write results

The two datasets we need to write back to the H5 file are the centroids and labels matrices. Both the centroids and labels matrices satisfy the condition to be elevated to the status of main datasets. However, in order to be recognized as main datasets, they need the four ancillary datasets to go along with them. Recall that the main datasets only need to store references to the ancillary datasets and that we do not need to store copies of the same ancillary datasets if multiple main datasets use them.

Here, we will refer to the dataset on which K-means was performed as the source dataset.

Identifying the ancillary datasets:

• centroids:
  • Spectroscopic Indices and Values: Since the source dataset and the centroids datasets both contain the same spectral information, the centroids dataset can simply reuse the ancillary spectroscopic datasets used by the source dataset.
  • Position Indices and Values: The centroids dataset has k instances while the source dataset has P, so we need to create a new position indicies and a new position values dataset for centroids
• labels:
  • Spectroscopic Indices and Values: Unlike the source dataset that has spectra of length S, this dataset only has a single value (cluster index) at each location. Consequently, labels needs two new ancilary datasets
  • Position Indices and Values: Since both source and labels have the same number of positions and the positions mean the same quantities for both datasets, we can simply reuse the ancillary dataset from source for labels

Preparing the missing ancillary arrays

Reshape the matricies to the correct dimensions

1. Since labels is a main dataset, it needs to be two dimensional matrix of size P x 1
2. The Spectroscopic ancillary datasets for labels need to be of the form dimension x points. Since the spectroscopic axis of labels is only one deep, labels has only one spectroscopic dimension which itself has just one point. Thus the Spectroscopic matrix should be of size 1 x 1
3. The centroids matrix is already of the form: position x spectra, so it does not need any reshaping
4. The Position ancillary datasets for centroids need to be of the form points x dimensions as well. In this case, centroids has k positions all in one dimension. Thus the matrix needs to be reshaped to k x 1

Create the Main MicroDataset objects

Remember that it is important to either inherit or add the quantity and units attributes to each main dataset

Create the ancillary MicroDataset objects

Create the group that will contain these datasets

We will be appending data to the existing h5 file and since HDF5 uses a tree structure to store information, we would need to specify where to add the sub-tree that we are building.

Recall that the name of the DataGroup provides information of the operation that has been performed on the source dataset. Therefore, we need to be careful about naming the group.

It is also important to add relevant information about the operation. For example, the name of our operation is Cluster analogous to the SkLearn package organization. Thus, the name of the algorithm - k-Means needs to be written as an attribute of the group as well.

Occasionaly, the same operation may be performed multiple times on the same dataset with different parameters. In the case of K-means it may be the number of clusters. pycroscopy allows all these results to be stored instead of being overwritten by appending an index number to the end of the group name. Thus, one could have a tree that contains the following groups:

• Raw_Data-Cluster_000 <--- K-means with 9 clusters
• Raw_Data-Cluster_001 <--- Agglomerative clustering
• Raw_Data-Cluster_002 <--- K-means again with 4 clusters

Leaving a '_' at the end of the group name will instruct ioHDF5 to look for the last instance of the same operation being performed on the same dataset. The index will then be updated accordingly

Write to H5 and access the written objects

Once the tree is prepared (previous cell), ioHDF5 will handle all the file writing.

Look at the H5 file contents now

Compare this tree with the one printed earlier. The new group and datasets should be apparent

Make centroids and labels -> main datasets

We elevate the status of these datasets by linking them to the four ancillary datasets. This part is also made rather easy by a few pycroscopy functions.

Why bother with all this?

• Though long, this simple file writing procedure needs to be written once for a given data analysis / processing tool
• The general nature of this Clustering algorithm facilitates the application to several other datasets regardless of their origin
• Once the data is written in the pycroscopy format, it is possible to apply other data analytics operations to the datasets with a single line
• Generalized versions of visualization algorithms can be written to visualize clustering results quickly.

Here is an example of very quick visualization with effectively just a single parameter - the group containing clustering results. The ancillary datasets linked to labels and centroids instructed the code about the spatial and spectroscopic dimensionality and enabled it to automatically render the plots below

Cleanup

Deletes the temporary files created in the example