MPI in Jupyter

We will use:

However, we will limit usage of ipyparallel features to minimum. Most of parallel logic will be in MPI standard.

The classical way

Write a file and use mpirun directly.



In [4]:

    
%%writefile mpi001.py

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
print "OK, rank= ",rank,size









    



Writing mpi001.py



In [12]:

    
!mpirun -n 4 /opt/conda/envs/py27/bin/python mpi001.py









    



OK, rank=  0 OK, rank=  1 4
OK, rank=  2 4
OK, rank=  3 4
4

Problems:

wrong python
where is my file ?



In [20]:

    
!mpirun -n 4 python2 mpi001.py









    



Traceback (most recent call last):
  File "mpi001.py", line 2, in <module>
    from mpi4py import MPI
ImportError: No module named mpi4py
Traceback (most recent call last):
  File "mpi001.py", line 2, in <module>
    from mpi4py import MPI
ImportError: No module named mpi4py
Traceback (most recent call last):
  File "mpi001.py", line 2, in <module>
    from mpi4py import MPI
ImportError: No module named mpi4py
Traceback (most recent call last):
  File "mpi001.py", line 2, in <module>
    from mpi4py import MPI
ImportError: No module named mpi4py



In [21]:

    
!mpirun -n 4 python mpi001.py









    



  File "mpi001.py", line 7
    print "OK, rank= ",rank,size
                     ^
SyntaxError: Missing parentheses in call to 'print'
  File "mpi001.py", line 7
    print "OK, rank= ",rank,size
                     ^
SyntaxError: Missing parentheses in call to 'print'
  File "mpi001.py", line 7
    print "OK, rank= ",rank,size
                     ^
SyntaxError: Missing parentheses in call to 'print'
  File "mpi001.py", line 7
    print "OK, rank= ",rank,size
                     ^
SyntaxError: Missing parentheses in call to 'print'

Use ipyparallel cluster:

First - setup an MPI profile!



In [14]:

    
import ipyparallel as ipp
c = ipp.Client(profile='mpi')
print(c.ids)
view = c[:]
view.activate()









    



[0, 1, 2, 3]

First Way:

write a file and do view.run("myfile.py")



In [24]:

    
%%writefile mpi001.py

from mpi4py import MPI
import time
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
time.sleep(5)
import os
a = 1.23 + rank     
print ("OK, rank= ",rank,size,os.getpid())









    



Overwriting mpi001.py



In [25]:

    
ar = view.run('mpi001.py')



In [35]:

    
ar









    Out[35]:





<AsyncResult: execute:finished>



In [30]:

    
ar.display_outputs()









    



[stdout:0] OK, rank=  3 4 38536
[stdout:1] OK, rank=  0 4 38533
[stdout:2] OK, rank=  1 4 38534
[stdout:3] OK, rank=  2 4 38535



In [32]:

    
ar = view.execute('mpi001.py')



In [33]:

    
view['a']









    Out[33]:





[4.23, 1.23, 2.23, 3.23]

this is equivalent to:



In [36]:

    
view.pull('a', block=True)









    Out[36]:





[4.23, 1.23, 2.23, 3.23]



In [ ]:

Second Way:

use magics: %%px ....

important!



In [55]:

    
view.activate()



In [37]:

    
%%px 

from mpi4py import MPI
import time
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
time.sleep(5)
import os
a = 1.23 + rank
print ("OK, rank= ",rank,size,os.getpid(),a)









    Out[37]:





<AsyncResult: execute>



In [38]:

    
%pxresult









    



[stdout:0] OK, rank=  3 4 38536 4.23
[stdout:1] OK, rank=  0 4 38533 1.23
[stdout:2] OK, rank=  1 4 38534 2.23
[stdout:3] OK, rank=  2 4 38535 3.23

Worth trying:

%%px --block
%%px --targets ::2
%autopx



In [40]:

    
%%px --block 

from mpi4py import MPI
import numpy as np

comm = MPI.COMM_WORLD
rank = comm.rank
print(comm.size,comm.rank)









    



[stdout:0] 4 3
[stdout:1] 4 0
[stdout:2] 4 1
[stdout:3] 4 2

Working directory in ipyparallel

If one wants to work in specyfic directory, then it is possible to set it using view.apply_sync commands:



In [12]:

    
import ipyparallel as ipp
c = ipp.Client(profile='mpi')
print(c.ids)
view = c[:]
view.activate()









    



[0, 1, 2, 3, 4, 5]



In [13]:

    
import os

print(view.apply_sync(os.getcwd))









    



['/home/users/marcin.kostur', '/home/users/marcin.kostur', '/home/users/marcin.kostur', '/home/users/marcin.kostur', '/home/users/marcin.kostur', '/home/users/marcin.kostur']



In [14]:

    
view.map(os.chdir, ['ProgramowanieRownolegle/MPI']*len(c.ids))









    Out[14]:





<AsyncMapResult: chdir>



In [15]:

    
print(view.apply_sync(os.getcwd))









    



['/home/users/marcin.kostur/ProgramowanieRownolegle/MPI', '/home/users/marcin.kostur/ProgramowanieRownolegle/MPI', '/home/users/marcin.kostur/ProgramowanieRownolegle/MPI', '/home/users/marcin.kostur/ProgramowanieRownolegle/MPI', '/home/users/marcin.kostur/ProgramowanieRownolegle/MPI', '/home/users/marcin.kostur/ProgramowanieRownolegle/MPI']



In [ ]: