Example form this site: https://ipython.org/ipython-doc/2/parallel/parallel_demos.html

In this section we describe two more involved examples of using an IPython cluster to perform a parallel computation. We will be doing some plotting, so we start IPython with matplotlib integration by typing:

download a needed file from: https://github.com/ipython/ipython-py3k/blob/master/docs/examples/newparallel/pidigits.py



In [5]:

    
from IPython.parallel import Client



In [6]:

    
c = Client(profile='slurm')



In [9]:

    
v = c[:]
c.ids









    Out[9]:





[0, 1, 2, 3, 4]

Download a helper file



In [18]:

    
%%bash 
wget https://raw.githubusercontent.com/ipython/ipython-py3k/master/docs/examples/newparallel/pidigits.py









    



--2016-03-22 15:45:43--  https://raw.githubusercontent.com/ipython/ipython-py3k/master/docs/examples/newparallel/pidigits.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.31.17.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.31.17.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4135 (4,0K) [text/plain]
Saving to: `pidigits.py'

     0K ....                                                  100%  647M=0s

2016-03-22 15:45:43 (647 MB/s) - `pidigits.py' saved [4135/4135]



In [19]:

    
# init content of the downloaded file
run pidigits.py



In [20]:

    
filestring = 'pi200m.ascii.%(i)02dof20'



In [24]:

    
files = [filestring % {'i':i} for i in range(len(c.ids))]
files









    Out[24]:





['pi200m.ascii.00of20',
 'pi200m.ascii.01of20',
 'pi200m.ascii.02of20',
 'pi200m.ascii.03of20',
 'pi200m.ascii.04of20']



In [25]:

    
v.map(fetch_pi_file, files)









    Out[25]:





<AsyncMapResult: fetch_pi_file>



In [ ]:

    
# This is the parallel calculation using the Client.map method
# which applies compute_two_digit_freqs to each file in files in parallel