The Center for Nanophase Materials Science and The Institute for Functional Imaging for Materials
Oak Ridge National Laboratory
4/6/2020
This Jupyter notebook uses pycroscopy to analyze Band Excitation data. We request you to reference the Arxiv paper titled "USID and Pycroscopy - Open frameworks for storing and analyzing spectroscopic and imaging data" in your publications.
This is a Jupyter Notebook - it contains text and executable code cells
. To learn more about how to use it, please see this video. Please see the image below for some basic tips on using this notebook.
If you have any questions or need help running this notebook, please get in touch with your host if you are a users at the Center for Nanophase Materials Science (CNMS) or our google group.
Image courtesy of Jean Bilheux from the neutron imaging GitHub repository.
In [ ]:
# Make sure needed packages are installed and up-to-date
import sys
!conda install --yes --prefix {sys.prefix} numpy scipy matplotlib scikit-learn Ipython ipywidgets h5py
!{sys.executable} -m pip install -U --no-deps pycroscopy # this will automatically install pyUSID as well
In [ ]:
# Ensure python 3 compatibility
from __future__ import division, print_function, absolute_import
# Import necessary libraries:
# General utilities:
import os
# Computation:
import numpy as np
import h5py
# Visualization:
import matplotlib.pyplot as plt
from IPython.display import display, HTML
# The engineering components supporting pycroscopy:
import pyUSID as usid
# Finally, pycroscopy itself
import pycroscopy as px
# Make Notebook take up most of page width
display(HTML(data="""
<style>
div#notebook-container { width: 95%; }
div#menubar-container { width: 65%; }
div#maintoolbar-container { width: 99%; }
</style>
"""))
In [ ]:
# set up notebook to show plots within the notebook
%matplotlib notebook
This notebook performs some functional fitting whose duration can be substantially decreased by using more memory and CPU cores. We have provided default values below but you may choose to change them if necessary. Setting max_cores
to None
will allow usage of all but one CPU core for the computations.
By default, results of the functional fitting will be written back to the same HDF5 file. However, if you prefer to write results into different HDF5 files, please set the results_to_new_file
parameter to True
instead. Users of the DataFed Scientific Data Management System may want to set this parameter to True
.
In [ ]:
max_mem = 1024*8 # Maximum memory to use, in Mbs. Default = 1024
max_cores = None # Number of logical cores to use in fitting. None uses all but 2 available cores.
results_to_new_file = True
Converting the raw data into a pycroscopy compatible hierarchical data format (HDF or .h5) file gives you access to the fast fitting algorithms and powerful analysis functions within pycroscopy
You can select desired file type by choosing the second option in the pull down menu on the bottom right of the file window
In [ ]:
input_file_path = usid.io_utils.file_dialog(caption='Select translated .h5 file or raw experiment data',
file_filter='Parameters for raw BE data (*.txt *.mat *xls *.xlsx);; \
Translated file (*.h5)')
(data_dir, filename) = os.path.split(input_file_path)
if input_file_path.endswith('.h5'):
# No translation here
h5_path = input_file_path
force = True # Set this to true to force patching of the datafile.
tl = px.io.translators.LabViewH5Patcher()
tl.translate(h5_path, force_patch=force)
else:
# Set the data to be translated
data_path = input_file_path
(junk, base_name) = os.path.split(data_dir)
# Check if the data is in the new or old format. Initialize the correct translator for the format.
if base_name == 'newdataformat':
(junk, base_name) = os.path.split(junk)
translator = px.io.translators.BEPSndfTranslator(max_mem_mb=max_mem)
else:
translator = px.io.translators.BEodfTranslator(max_mem_mb=max_mem)
if base_name.endswith('_d'):
base_name = base_name[:-2]
# Translate the data
h5_path = translator.translate(data_path, show_plots=True, save_plots=False)
folder_path, h5_raw_file_name = os.path.split(h5_path)
h5_file = h5py.File(h5_path, 'r+')
print('Working on:\n' + h5_path)
h5_main = usid.hdf_utils.find_dataset(h5_file, 'Raw_Data')[0]
The file contents are stored in a tree structure, just like files on a conventional computer. The data is stored as a 2D matrix (position, spectroscopic value) regardless of the dimensionality of the data. Thus, the positions will be arranged as row0-col0, row0-col1.... row0-colN, row1-col0.... and the data for each position is stored as it was chronologically collected
The main dataset is always accompanied by four ancillary datasets that explain the position and spectroscopic value of any given element in the dataset.
In [ ]:
print('Datasets and datagroups within the file:\n------------------------------------')
usid.hdf_utils.print_tree(h5_file)
print('\nThe main dataset:\n------------------------------------')
print(h5_main)
print('\nMetadata or attributes in the measurement datagroup\n------------------------------------')
for key, val in usid.hdf_utils.get_attributes(h5_main.parent.parent).items():
print('{} : {}'.format(key, val))
In [ ]:
h5_pos_inds = h5_main.h5_pos_inds
pos_dims = h5_main.pos_dim_sizes
pos_labels = h5_main.pos_dim_labels
print(pos_labels, pos_dims)
h5_meas_grp = h5_main.parent.parent
parm_dict = usid.hdf_utils.get_attributes(h5_meas_grp)
expt_type = usid.hdf_utils.get_attr(h5_file, 'data_type')
is_ckpfm = expt_type == 'cKPFMData'
if is_ckpfm:
num_write_steps = parm_dict['VS_num_DC_write_steps']
num_read_steps = parm_dict['VS_num_read_steps']
num_fields = 2
if expt_type != 'BELineData':
vs_mode = usid.hdf_utils.get_attr(h5_meas_grp, 'VS_mode')
try:
field_mode = usid.hdf_utils.get_attr(h5_meas_grp, 'VS_measure_in_field_loops')
except KeyError:
print('field mode could not be found. Setting to default value')
field_mode = 'out-of-field'
try:
vs_cycle_frac = usid.hdf_utils.get_attr(h5_meas_grp, 'VS_cycle_fraction')
except KeyError:
print('VS cycle fraction could not be found. Setting to default value')
vs_cycle_frac = 'full'
In [ ]:
fig = px.viz.be_viz_utils.jupyter_visualize_be_spectrograms(h5_main)
Fit each of the acquired spectra to a simple harmonic oscillator (SHO) model to extract the following information regarding the response:
By default, the cell below will take any previous result instead of re-computing the SHO fit
In [ ]:
sho_fit_points = 5 # The number of data points at each step to use when fitting
sho_override = False # Force recompute if True
h5_sho_targ_grp = None
if results_to_new_file:
h5_sho_file_path = os.path.join(folder_path,
h5_raw_file_name.replace('.h5', '_sho_fit.h5'))
print('\n\nSHO Fits will be written to:\n' + h5_sho_file_path + '\n\n')
f_open_mode = 'w'
if os.path.exists(h5_sho_file_path):
f_open_mode = 'r+'
h5_sho_file = h5py.File(h5_sho_file_path, mode=f_open_mode)
h5_sho_targ_grp = h5_sho_file
sho_fitter = px.analysis.BESHOfitter(h5_main, cores=max_cores, verbose=False, h5_target_group=h5_sho_targ_grp)
sho_fitter.set_up_guess(guess_func=px.analysis.be_sho_fitter.SHOGuessFunc.complex_gaussian,
num_points=sho_fit_points)
h5_sho_guess = sho_fitter.do_guess(override=sho_override)
sho_fitter.set_up_fit()
h5_sho_fit = sho_fitter.do_fit(override=sho_override)
h5_sho_grp = h5_sho_fit.parent
Here, we visualize the parameters for the SHO fits. BE-line (3D) data is visualized via simple spatial maps of the SHO parameters while more complex BEPS datasets (4+ dimensions) can be visualized using a simple interactive visualizer below.
You can choose to visualize the guesses for SHO function or the final fit values from the first line of the cell below.
Use the sliders below to inspect the BE response at any given location.
In [ ]:
h5_sho_spec_inds = h5_sho_fit.h5_spec_inds
sho_spec_labels = h5_sho_fit.spec_dim_labels
if is_ckpfm:
# It turns out that the read voltage index starts from 1 instead of 0
# Also the VDC indices are NOT repeating. They are just rising monotonically
write_volt_index = np.argwhere(sho_spec_labels == 'write_bias')[0][0]
read_volt_index = np.argwhere(sho_spec_labels == 'read_bias')[0][0]
h5_sho_spec_inds[read_volt_index, :] -= 1
h5_sho_spec_inds[write_volt_index, :] = np.tile(np.repeat(np.arange(num_write_steps), num_fields), num_read_steps)
(Nd_mat, success, nd_labels) = usid.hdf_utils.reshape_to_n_dims(h5_sho_fit, get_labels=True)
print('Reshape Success: ' + str(success))
print(nd_labels)
print(Nd_mat.shape)
In [ ]:
use_sho_guess = False
use_static_viz_func = True
if use_sho_guess:
sho_dset = h5_sho_guess
else:
sho_dset = h5_sho_fit
if expt_type == 'BELineData' or len(pos_dims) != 2:
use_static_viz_func = True
step_chan = None
else:
if vs_mode not in ['AC modulation mode with time reversal',
'DC modulation mode']:
use_static_viz_func = True
else:
if vs_mode == 'DC modulation mode':
step_chan = 'DC_Offset'
else:
step_chan = 'AC_Amplitude'
if not use_static_viz_func:
try:
# use interactive visualization
px.viz.be_viz_utils.jupyter_visualize_beps_sho(sho_dset, step_chan)
except:
raise
print('There was a problem with the interactive visualizer')
use_static_viz_func = True
else:
chan_grp = h5_main.parent
meas_grp = chan_grp.parent
# show plots of SHO results vs. applied bias
figs = px.viz.be_viz_utils.visualize_sho_results(sho_dset, show_plots=True, save_plots=False,
expt_type=expt_type, meas_type=vs_mode,
field_mode=field_mode)
In [ ]:
# Do the Loop Fitting on the SHO Fit dataset
loop_success = False
h5_loop_group = None
if results_to_new_file:
h5_loop_file_path = os.path.join(folder_path,
h5_raw_file_name.replace('.h5', '_loop_fit.h5'))
print('\n\nLoop Fits will be written to:\n' + h5_loop_file_path + '\n\n')
f_open_mode = 'w'
if os.path.exists(h5_loop_file_path):
f_open_mode = 'r+'
h5_loop_file = h5py.File(h5_loop_file_path, mode=f_open_mode)
h5_loop_group = h5_loop_file
loop_fitter = px.analysis.BELoopFitter(h5_sho_fit, expt_type, vs_mode, vs_cycle_frac,
cores=max_cores, h5_target_group=h5_loop_group,
verbose=False)
loop_fitter.set_up_guess()
h5_loop_guess = loop_fitter.do_guess(override=False)
# Calling explicitely here since Fitter won't do it automatically
h5_guess_loop_parms = loop_fitter.extract_loop_parameters(h5_loop_guess)
loop_fitter.set_up_fit()
h5_loop_fit = loop_fitter.do_fit(override=False)
h5_loop_group = h5_loop_fit.parent
loop_success = True
In [ ]:
# Prepare some variables for plotting loops fits and guesses
# Plot the Loop Guess and Fit Results
if loop_success:
h5_projected_loops = usid.USIDataset(h5_loop_guess.parent['Projected_Loops'])
h5_proj_spec_inds = h5_projected_loops.h5_spec_inds
h5_proj_spec_vals = h5_projected_loops.h5_spec_vals
# reshape the vdc_vec into DC_step by Loop
sort_order = usid.hdf_utils.get_sort_order(h5_proj_spec_inds)
dims = usid.hdf_utils.get_dimensionality(h5_proj_spec_inds[()],
sort_order[::-1])
vdc_vec = np.reshape(h5_proj_spec_vals[h5_proj_spec_vals.attrs['DC_Offset']], dims).T
#Also reshape the projected loops to Positions-DC_Step-Loop
# Also reshape the projected loops to Positions-DC_Step-Loop
proj_nd = h5_projected_loops.get_n_dim_form()
proj_3d = np.reshape(proj_nd, [h5_projected_loops.shape[0],
proj_nd.shape[2], -1])
In [ ]:
use_static_plots = True
if loop_success:
if not use_static_plots:
try:
fig = px.viz.be_viz_utils.jupyter_visualize_beps_loops(h5_projected_loops, h5_loop_guess, h5_loop_fit)
except:
print('There was a problem with the interactive visualizer')
use_static_plots = True
if use_static_plots:
for iloop in range(h5_loop_guess.shape[1]):
fig, ax = px.viz.be_viz_utils.plot_loop_guess_fit(vdc_vec[:, iloop], proj_3d[:, :, iloop],
h5_loop_guess[:, iloop], h5_loop_fit[:, iloop],
title='Loop {} - All Positions'.format(iloop))
In [ ]:
h5_loop_parameters = h5_loop_group['Fit_Loop_Parameters']
fig = px.viz.be_viz_utils.jupyter_visualize_parameter_maps(h5_loop_parameters)
In [ ]:
map_parm = 'Work of Switching'
plot_cycle = 0
plot_position = (int(pos_dims[0]/2), int(pos_dims[1]/2))
plot_bias_step = 0
fig = px.viz.be_viz_utils.plot_loop_sho_raw_comparison(h5_loop_parameters, h5_sho_grp, h5_main,
selected_loop_parm=map_parm,
selected_loop_cycle=plot_cycle,
selected_loop_pos=plot_position,
selected_step=plot_bias_step)
In [ ]:
h5_file.close()
if results_to_new_file:
h5_sho_fit.file.close()
h5_loop_fit.file.close()