Using Stream Serialization (Pickling)

In python pickling|unpickling is a process of serializing a python object into a file. Basically it takes the hunk of memory that a file is sitting in and writes it out to disk.



In [2]:

    
from IPython.display import YouTubeVideo
YouTubeVideo('yYey8ntlK_E', width=800, height=500)









    Out[2]:

There are two implementations of pickle in python:

pickle
cPickle

The cPickle implementation is faster.

I like using pickles in two situations

When passing complex objects between python programs
For debugging programs that take a long time to run.

Say we have a program that runs for 15 min, then produces an error. We could fire up the python debugger and dig into the problem, but each time we change something we have to re-run the entire program. Instead lets use pickling to help us out.

First lets make some helper functions.



In [3]:

    
import cPickle as pk

This first function takes a dictionary and pickles it.



In [4]:

    
def pickleDict(objDict, fname):
    """ Pickle a dictionary of "var name": values for debugging.

    Arguments:
        :param dict objDict: A dictionary where each key is the name of a
            variable and the value is the object itself.

        :param str fname: File name of the pickle.
    """
    with open(fname, 'wb') as FH:
        pk.dump(objDict, FH)

This second function unpickles a file and retunrs the dictionary.



In [5]:

    
def unPickleDict(fname):
    """ Pickle a dictionary of "var name": values for debugging.

    Arguments:
        :param str fname: File name of the pickle.

    Returns:
        :rtype: dict
        :returns: A dictionary where each key is the name of a
            variable and the value is the object itself.

    """
    with open(fname, 'rb') as FH:
        objDict = pk.load(FH)

    return objDict

Ok, now lets say our program is erroring out at a specific function. We know that the function takes three objects.

my_cool_flags
my_big_data
my_list_o_genes

I can create a dictionary where the key is my object name and the value is my object.



In [6]:

    
# Make up some data
my_cool_flags = [0, 0, 0, 0, 0, 1]
my_big_data = ['one', 'two', 'three']
my_list_o_genes = ['Sxl', 'dsx', 'fru']



In [7]:

    
# Put all of these in a dictionary
objDict = {'my_cool_flags': my_cool_flags,
           'my_big_data': my_big_data,
           'my_list_o_genes': my_list_o_genes}

# We can pickle that
pickleDict(objDict, '/tmp/mypickle.pkl')

Now in a separate ipython environment you can unpickle your objects, import your broken function and go to town.



In [9]:

    
objDict = unPickleDict('/tmp/mypickle.pkl')
print objDict









    



{'my_big_data': ['one', 'two', 'three'], 'my_cool_flags': [0, 0, 0, 0, 0, 1], 'my_list_o_genes': ['Sxl', 'dsx', 'fru']}

# In ipython, you can tell it to reload a function everything you call 
# it. That way you can make changes and have them work.
>>> %load_ext autoreload
>>> %autoreload 2

# Then import you broken function and start tweaking
>>> from longProgram import brokenFunction
>>> brokenFunction(objDict['my_cool_flags'], objDict['my_big_data'], objDict['my_list_o_genes'])

# You can also activate the IPython debugger if you want to use it.
>>> %pdb

You can also add multiple objects to a pickle using pk.dump multiple times, and unpickle them by running pk.load multiple times, but the dictionary approach is easier to tell what is what.

NOTE: These two helper functions are already in mclib_Python.



In [ ]: