one way to use c libraries in python is to write an extension module by hand; it takes time and effort; fortunately, there are tools we can leverage to make this job easier: cython is one of them;

cython overview

an introduction of cython is given in its overview; to make it short, cython is a programming language aiming to become a superset of python; cython compiles to optimized c (we ignore c++) code which is then compiled as extension modules; cython supports optional static type declarations that can often accelerate code execution; from user perspective, we write cython file using python-like grammar and compiles it into c code, thereby saving us great time than craft this c code by hand;

there are 3 kinds of cython files:

  • the implementation files, carrying a .py or .pyx suffix;

  • the definition files, carrying a .pxd suffix;

  • the include files, carrying a .pxi suffix;

we will cover .pyx and .pxd files in this article;

job description

we have a c file lib.c that contains a function add:

int add(int a, int b)
{
    return a + b;
}

we want to use this function in python programming;

solution 1: when c library does not have a header file

when lib.c does not come with a header file, we need to write one by hand; to be specific, we can use this wrap.pyx to wrap c function add:

cdef extern int add(int a, int b);

def wrap_add(a, b):
    return add(a, b)

the first line does two things: declare the function name add in cython, and generate a declaration in compiled c file; now wrap_add can call add as shown; there are some auto type conversions: what get passed to wrap_add are python objects, cython auto converts them to c int type and passes them to c function add; the return value of add is c int type, which cython converts back to python object and returns it; wow, cython is smart;

to build wrap.pyx with lib.c, we use this setup.py:

from Cython.Build import cythonize
from setuptools import Extension
from setuptools import setup

setup(
    name='wrap',
    ext_modules=cythonize([
        Extension('wrap', [ 'wrap.pyx', 'lib.c' ])
    ]),
)

there are basically 2 more steps than if we craft this extension module by hand: add wrap.pyx in ext_modules, then feed ext_modules through cythonize;

now we can build, install and test:

# python3 setup.py build
# python3 setup.py install
# python3 -c "import wrap; print(wrap.wrap_add(3, 4))"
7

solution 2: when c library has a header file

if lib.c has an accompanying header file lib.h that contains declaration of the function add, we can utilize this header file:

int add(int a, int b);

in this case our wrap.pyx becomes this:

cdef extern from "lib.h":
    int add(int a, int b);

def wrap_add(a, b):
    return add(a, b)

not a big change; we still have to declare the function in cython; the difference is, cython will not put our declaration in compiled c file; instead, it emits #include "lib.h" in the c file, so that the c file is using the authentic declaration provided by the library itself; this reduces the possibility of a mistake; in fact, solution 1 still works, but solution 2 is better;

build, install and test is the same as solution 1; result is also the same;

solution 3: using a .pxd file

the .pxd file is analagous to a c header file, and used like a python module in cython; its main purpose is to share declarations between .pyx files; now we work from solution 2 and move declarations from wrap.pyx into wrap.pxd:

cdef extern from "lib.h":
    int add(int a, int b);

and modify wrap.pyx into this:

from wrap cimport add

def wrap_add(a, b):
    return add(a, b)

it is worth mentioning that .pyx and .pxd files do not need to have the same name; but if there is a .pxd with the same name as .pyx, then that .pxd is searched before processing the .pyx and if found processed before the .pyx;

the difference in wrap.pyx is this line: from wrap cimport add; note that it is cimport not import; the syntax of cimport parallels import in python, but they are not the same thing and cannot be interchanged;

build, install and test, and the result is the same;

next we show why using .pxd file helps: it helps when python and c names clash with each other; in our example, we have been providing wrap_add as the python name for the c function add; but it would feel natural to simply provide add as the python name; we were not able to do this, because cython puts python and c names in the same module-level namespace; but we can do this now with the .pxd file; we can modify wrap.pyx into this:

from wrap cimport add as cadd

def add(a, b):
    return cadd(a, b)

use from ... cimport ... as in cython the same way as from ... import ... as in python; this binds c function add to name cadd, leaving add available for use as python name; now the test becomes:

# python3 -c "import wrap; print(wrap.add(3, 4))"
7

this new python name add looks nicer than wrap_add;

finally, we have based solution 3 on solution 2, but it can also be based on solution 1; namely, the .pxd file can contain a handcrafted declaration, too; however, .pxd is often used to wrap a c header file in cython; so using the c header file when it is available is recommended;