

In [1]:

    
from tempfile import mkdtemp
from joblib import Memory



In [2]:

    
cachedir = mkdtemp()
memory = Memory(cachedir=cachedir, verbose=0)


@memory.cache
def f(x):
    print('Running f(%s)' % x)
    return x









    



/home/roo/.virtualenvs/hhw/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: The 'cachedir' parameter has been deprecated in version 0.12 and will be removed in version 0.14.
You provided "cachedir='/tmp/tmpcpsb9tn9'", use "location=None" instead.



In [3]:

    
print(f(1))









    



Running f(1)
1



In [4]:

    
print(f(1))

When we call this function twice with the same argument, it does not get executed the second time, and the output gets loaded from the pickle file:

Using with numpy



In [5]:

    
import numpy as np

@memory.cache
def g(x):
     print('A long-running calculation, with parameter %s' % x)
     return np.hamming(x)

@memory.cache
def h(x):
    print('A second long-running calculation, using g(x)')
    return np.vander(x)



In [9]:

    
a = g(3)



In [10]:

    
a









    Out[10]:





array([0.08, 1.  , 0.08])



In [11]:

    
g(3)









    Out[11]:





array([0.08, 1.  , 0.08])



In [12]:

    
h(a)









    



A second long-running calculation, using g(x)






    Out[12]:





array([[0.0064, 0.08  , 1.    ],
       [1.    , 1.    , 1.    ],
       [0.0064, 0.08  , 1.    ]])



In [15]:

    
h(a)









    Out[15]:





array([[0.0064, 0.08  , 1.    ],
       [1.    , 1.    , 1.    ],
       [0.0064, 0.08  , 1.    ]])

using memmapping

To speed up cache looking of large numpy arrays, you can load them using memmapping (memory mapping):



In [17]:

    
cachedir2 = mkdtemp()
memory2 = Memory(cachedir=cachedir2, mmap_mode='r')
square = memory2.cache(np.square)
a = np.vander(np.arange(3)).astype(np.float)
square(a)









    



________________________________________________________________________________
[Memory] Calling square...
square(array([[0., 0., 1.],
       [1., 1., 1.],
       [4., 2., 1.]]))
___________________________________________________________square - 0.0s, 0.0min






    



/home/roo/.virtualenvs/hhw/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: The 'cachedir' parameter has been deprecated in version 0.12 and will be removed in version 0.14.
You provided "cachedir='/tmp/tmps3wbltl3'", use "location=None" instead.
  






    Out[17]:





memmap([[ 0.,  0.,  1.],
        [ 1.,  1.,  1.],
        [16.,  4.,  1.]])



In [20]:

    
# call it again
res = square(a)
res









    Out[20]:





memmap([[ 0.,  0.,  1.],
        [ 1.,  1.,  1.],
        [16.,  4.,  1.]])

Note: If the memory mapping mode used was ‘r’, as in the above example, the array will be read only, and will be impossible to modified in place. On the other hand, using ‘r+’ or ‘w+’ will enable modification of the array, but will propagate these modification to the disk, which will corrupt the cache. If you want modification of the array in memory, we suggest you use the ‘c’ mode: copy on write.



In [ ]: