wrap c libraries with cython
one way to use c libraries in python is to write an extension module by hand; it takes time and effort; fortunately, there are tools we can leverage to make this job easier: cython is one of them;
cython overview
an introduction of cython is given in its overview; to make it short, cython is a programming language aiming to become a superset of python; cython compiles to optimized c (we ignore c++) code which is then compiled as extension modules; cython supports optional static type declarations that can often accelerate code execution; from user perspective, we write cython file using python-like grammar and compiles it into c code, thereby saving us great time than craft this c code by hand;
there are 3 kinds of cython files:
-
the implementation files, carrying a
.py
or.pyx
suffix; -
the definition files, carrying a
.pxd
suffix; -
the include files, carrying a
.pxi
suffix;
we will cover .pyx
and .pxd
files in this article;
job description
we have a c file lib.c
that contains a function add
:
int add(int a, int b)
{
return a + b;
}
we want to use this function in python programming;
solution 1: when c library does not have a header file
when lib.c
does not come with a header file, we need to write one by hand; to
be specific, we can use this wrap.pyx
to wrap c function add
:
cdef extern int add(int a, int b);
def wrap_add(a, b):
return add(a, b)
the first line does two things: declare the function name add
in cython, and
generate a declaration in compiled c file; now wrap_add
can call add
as
shown; there are some auto type conversions: what get passed to wrap_add
are
python objects, cython auto converts them to c int
type and passes them to c
function add
; the return value of add
is c int
type, which cython converts
back to python object and returns it; wow, cython is smart;
to build wrap.pyx
with lib.c
, we use this setup.py
:
from Cython.Build import cythonize
from setuptools import Extension
from setuptools import setup
setup(
name='wrap',
ext_modules=cythonize([
Extension('wrap', [ 'wrap.pyx', 'lib.c' ])
]),
)
there are basically 2 more steps than if we craft this extension module by hand:
add wrap.pyx
in ext_modules
, then feed ext_modules
through cythonize
;
now we can build, install and test:
# python3 setup.py build
# python3 setup.py install
# python3 -c "import wrap; print(wrap.wrap_add(3, 4))"
7
solution 2: when c library has a header file
if lib.c
has an accompanying header file lib.h
that contains declaration of
the function add
, we can utilize this header file:
int add(int a, int b);
in this case our wrap.pyx
becomes this:
cdef extern from "lib.h":
int add(int a, int b);
def wrap_add(a, b):
return add(a, b)
not a big change; we still have to declare the function in cython; the
difference is, cython will not put our declaration in compiled c file; instead,
it emits #include "lib.h"
in the c file, so that the c file is using the
authentic declaration provided by the library itself; this reduces the
possibility of a mistake; in fact, solution 1 still works, but solution 2 is
better;
build, install and test is the same as solution 1; result is also the same;
solution 3: using a .pxd
file
the .pxd
file is analagous to a c header file, and used like a python module
in cython; its main purpose is to share declarations between .pyx
files; now
we work from solution 2 and move declarations from wrap.pyx
into wrap.pxd
:
cdef extern from "lib.h":
int add(int a, int b);
and modify wrap.pyx
into this:
from wrap cimport add
def wrap_add(a, b):
return add(a, b)
it is worth mentioning that .pyx
and .pxd
files do not need to have the same
name; but if there is a .pxd
with the same name as .pyx
, then that .pxd
is
searched before processing the .pyx
and if found processed before the .pyx
;
the difference in wrap.pyx
is this line: from wrap cimport add
; note that it
is cimport
not import
; the syntax of cimport
parallels import
in python,
but they are not the same thing and cannot be interchanged;
build, install and test, and the result is the same;
next we show why using .pxd
file helps: it helps when python and c names clash
with each other; in our example, we have been providing wrap_add
as the python
name for the c function add
; but it would feel natural to simply provide add
as the python name; we were not able to do this, because cython puts python and
c names in the same module-level namespace; but we can do this now with the
.pxd
file; we can modify wrap.pyx
into this:
from wrap cimport add as cadd
def add(a, b):
return cadd(a, b)
use from ... cimport ... as
in cython the same way as from ... import ... as
in python; this binds c function add
to name cadd
, leaving add
available
for use as python name; now the test becomes:
# python3 -c "import wrap; print(wrap.add(3, 4))"
7
this new python name add
looks nicer than wrap_add
;
finally, we have based solution 3 on solution 2, but it can also be based on
solution 1; namely, the .pxd
file can contain a handcrafted declaration, too;
however, .pxd
is often used to wrap a c header file in cython; so using the c
header file when it is available is recommended;