Using third-party Native Libraries

Sometimes, the functionnality you need are onmy available in third-party native libraries. There's still an opportunity to use them from within Pythran, using Pythran support for capsules.

Pythran Code

The pythran code requires function pointers to the third-party functions, passed as parameters to your pythran routine, as in the following:



In [1]:

    
import pythran
%load_ext pythran.magic



In [2]:

    
%%pythran 
#pythran export pythran_cbrt(float64(float64), float64)

def pythran_cbrt(libm_cbrt, val):
    return libm_cbrt(val)

In that case libm_cbrt is expected to be a capsule containing the function pointer to libm's cbrt (cube root) function.

This capsule can be created using ctypes:



In [3]:

    
import ctypes

# capsulefactory
PyCapsule_New = ctypes.pythonapi.PyCapsule_New
PyCapsule_New.restype = ctypes.py_object
PyCapsule_New.argtypes = ctypes.c_void_p, ctypes.c_char_p, ctypes.c_void_p

# load libm
libm = ctypes.CDLL('/lib/x86_64-linux-gnu/libm.so.6')

# extract the proper symbol
cbrt = libm.cbrt

# wrap it
cbrt_capsule = PyCapsule_New(cbrt, "double(double)", None)

The capsule is not usable from Python context (it's some kind of opaque box) but Pythran knows how to use it. beware, it does not try to do any kind of type verification. It trusts your #pythran export line.



In [4]:

    
pythran_cbrt(cbrt_capsule, 8.)









    Out[4]:





2.0

With Pointers

Now, let's try to use the sincos function. It's C signature is void sincos(double, double*, double*). How do we pass that to Pythran?



In [5]:

    
%%pythran

#pythran export pythran_sincos(None(float64, float64*, float64*), float64)
def pythran_sincos(libm_sincos, val):
    import numpy as np
    val_sin, val_cos = np.empty(1), np.empty(1)
    libm_sincos(val, val_sin, val_cos)
    return val_sin[0], val_cos[0]

There is some magic happening here:

None is used to state the function pointer does not return anything.
In order to create pointers, we actually create empty one-dimensional array and let pythran handle them as pointer. Beware that you're in charge of all the memory checking stuff!

Apart from that, we can now call our function with the proper capsule parameter.



In [6]:

    
sincos_capsule = PyCapsule_New(libm.sincos, "uncheck any way", None)



In [7]:

    
pythran_sincos(sincos_capsule, 0.)









    Out[7]:





(0.0, 1.0)

With Cython

The capsule pythran uses may come from Cython-generated code. This uses a little-known feature from cython: api and __pyx_capi__. nogil is of importance here: Pythran releases the GIL, so better not call a cythonized function that uses it.



In [8]:

    
!rm -f cube.c cube.pyx cube.so



In [9]:

    
%%file cube.pyx

cdef api double cube(double x) nogil:
    return x * x * x









    



Writing cube.pyx

Sorry about that, old-school compilation of a Python module, as a one-liner :-) In a real project, you would use distutils and stuff, of course.



In [10]:

    
!cythonize cube.pyx
!gcc cube.c -shared -fPIC -O2 -o cube.so `python-config --cflags --libs`









    



Compiling /home/serge/sources/pythran/docs/examples/cube.pyx because it changed.
[1/1] Cythonizing /home/serge/sources/pythran/docs/examples/cube.pyx

The cythonized module has a special dictionary that holds teh capsule we're looking for.



In [11]:

    
import cube
print(type(cube.__pyx_capi__['cube']))









    



<type 'PyCapsule'>



In [12]:

    
cython_cube = cube.__pyx_capi__['cube']
pythran_cbrt(cython_cube, 2.)









    Out[12]:





8.0

With Pythran

It is naturally also possible to use capsule generated by Pythran. In that case, no type shenanigans is required, we're in our small world.

One just need to use the capsule keyword to indicate we want to generate a capsule.



In [13]:

    
%%pythran

## This is the capsule.
#pythran export capsule corp((int, str), str set)
def corp(param, lookup):
    res, key = param
    return res if key in lookup else -1

## This is some dummy callsite
#pythran export brief(int, int((int, str), str set)):
def brief(val, capsule):
    return capsule((val, "doctor"), {"some"})

It's not possible to call the capsule: it's... an opaque capsule.



In [14]:

    
try:
    corp((1,"some"),set())
except TypeError as e:
    print(e)









    



'PyCapsule' object is not callable

It's possible to pass it to the according pythran function though.



In [15]:

    
brief(1, corp)









    Out[15]:





-1