Boost and Python 3.x

Question

How does boost.python deal with Python 3? Is it Python 2 only?

Russon

Seems that they are working on it. boost.org/doc/libs/1_46_0/libs/python/doc/news.html – Aria

Answer 1

Newer versions of Boost should work fine with Python V3.x. This support has been added quite some time ago, I believe after a successful Google Summer of Code project back in 2009.

The way to use Python V3 with Boost is to properly configure the build system by adding for instance:

using python : 3.1 : /your_python31_root ;

to your user-config.jam file.

Azzieb

You can simply do ./bootstrap.sh --with-python-version=X.Y if it is installed on the system and detectable (in the usual place). If not, you can specify these options (instead of editing a file) on the command line. See ./bootstrap.sh --help – Guberniya

Turns out, the space between the python path and the semicolon... is important! – Derosier

Answer 2

libboostpython needs to be built with python3 in order to do this. This doesn't work with boost 1.58 (which comes with Ubuntu 16.04), so make sure you download the latest boost distribution. I just did this with boost_1_64_0.

As mentioned above, find the file "user-config.jam" in you boost code distribution, and copy it to $HOME.

cp /path/to/boost_1_64_0/tools/build/example/user-config.jam $HOME

Then edit the python line (the last line) so that is says:

using python : 3.5 : /usr/bin/python3 : /usr/include/python3.5m : /usr/lib ;

This is correct for Ubuntu 16.04. You can use pkg-config to find the correct include directory.

user@computer > pkg-config --cflags python3
-I/usr/include/python3.5m -I/usr/include/x86_64-linux-gnu/python3.5m

And you only need the first include directory.

Then build boost from scratch. (Sorry.) I install it to /usr/local

cd /path/to/boost_1_64_0
./bootstrap.sh --prefix=/usr/local
./b2 
sudo ./b2 install

Now jump into the python example directory, and build the tutorial

cd /path/to/boost_1_64_0/libs/python/example/tutorial
bjam

This will not build correctly if you have a system install of boost, because, under the hood, bjam is linking to libboostpython using the g++ parameter "-lboost". But, on Ubuntu 16.04, this will just go and find "/usr/lib/x86_64-linux-gnu/libboost_python-py27.so.1.58.0", and then the python bindings will fail to load. In fact, you'll get his error:

ImportError: /usr/lib/x86_64-linux-gnu/libboost_python-py27.so.1.58.0: undefined symbol: PyClass_Type

If you want to see the g++ commands that bjam is using, do this:

user@computer > bjam -d2 -a | grep g++
g++  -ftemplate-depth-128 -O0 -fno-inline -Wall -g -fPIC -I/usr/include/python3.5m -c -o "hello.o" "hello.cpp"
g++ -o hello_ext.so -Wl,-h -Wl,hello_ext.so -shared -Wl,--start-group hello.o  -Wl,-Bstatic  -Wl,-Bdynamic -lboost_python -ldl -lpthread -lutil -Wl,--end-group

Here we see the problem, you need "-L/usr/includ/lib" just before "-lboost_python". So execute this to link the shared library correctly:

g++ -o hello_ext.so -Wl,-h -Wl,hello_ext.so -shared -Wl,--start-group hello.o  -Wl,-Bstatic  -Wl,-Bdynamic -L/usr/local/lib -lboost_python -ldl -lpthread -lutil -Wl,--end-group

You may need to rerun ldconfig (or reboot)

sudo ldconfig

And you are finally ready to go:

user@computer > python3
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hello_ext
>>> hello_ext.greet()
'hello, world'
>>> exit()

Rb

this was totally helpful to me. the key to getting my project to build with a custom-installed everything (gcc, python3, boost) was to edit my project-config.jam file, as you indicated. namely, to adjust the using python :... line to list each of binary executable for the python3 interpreter, the include folder for it, and the lib directory. after re-building boost and installing, my project correctly builds using python 3. thanks! – Aurita

Nicely explained. very helpful! – Builtup

Answer 3

Yes this question is super old, but I had to do something that wasn't specified in any of the answers here (though it was built off some of the suggestions), so I'll quickly jot down my entire process:

1. Download boost_X_Y_Z.tar.bz2 (I used boost 1.68.0)
2. tar --bzip2 -xf boost_1_68_0.tar.bz2 (where you want folder to be temporarily)
3. cd boost_1_68_0
4. ./bootstrap.sh --with-python-version=3.6 --prefix=/usr/local
5. ./b2
6. sudo ./bjam install
7. cp tools/build/example/user-config.jam $HOME, then modify the contents of this file to say using python : 3.6 : /usr/bin/python3 : /usr/include/python3.6m : /usr/lib ; (or whatever folders are appropriate for your environment)

Given this C++ source file BoostPythonHelloWorld.cpp:

#include <boost/python.hpp>

char const* say_hi()
{
    return "Hi!";
}

BOOST_PYTHON_MODULE(BoostPythonHelloWorld)
{
    boost::python::def("say_hi", say_hi);
}

And this Python script BoostPythonHelloWorld.py:

import BoostPythonHelloWorld
print(BoostPythonHelloWorld.say_hi())

It can be compiled and ran as such:

gcc -c -fPIC -I/path/to/boost_1_68_0 -I/usr/include/python3.6 /other_path/to/BoostPythonHelloWorld.cpp
gcc -shared -Wall -Werror -Wl,--export-dynamic BoostPythonHelloWorld.o -L/path/to/boost_1_68_0/stage/lib -lboost_python36 -o BoostPythonHelloWorld.so
python3 BoostPythonHelloWorld.py

The part that was different for me was -Wl,--export-dynamic BoostPythonHelloWorld.o, I had not seen that anywhere else, and I was getting a Python error concerning an undefined symbol until I added that.

Hope this helps someone down the line!

Pentad

This particular answer worked for me: https://mcmap.net/q/63784/-how-to-install-boost-on-ubuntu I had the latest version boost_1_71_0 – Commensurate

To get the options available ./bootstrap.sh --help. There are 3 options related to python: --with-python=PYTHON --with-python-root=DIR --with-python-version=X.Y. If command option works, then it is more preferable to modifying user-config.jam. The later has potential problem if your python3 changes from one source to another. – Taxation

Answer 4

If you get "error: No best alternative for /python_for_extension" be sure to have

using python : 3.4 : C:\\Python34 : C:\\Python34\\include : C:\\Python34\\libs ;

only in user-config.jam in your home path and nowhere else. Use double backslashes when compiling under windows with mingw (toolset=gcc) or MSVC (toolset=msvc). Compile with cmd, not msys, and if you also have python 2.7 installed remove that from PATH in that shell. First do

bootstrap.bat gcc/msvc

assuming you have the gcc/msvc tools available via PATH (/ for the alternatives, but use only one, or leave away)

Afterward you can also do

booststrap.sh --with-bjam=b2

in msys to generate a project-config.jam, but need to edit it to remove the "using python" and "/usr",..

Then the following

b2 variant=debug/shared link=static/shared toolset=gcc/msvc > b2.log

With static the python quickstart examples did not work for me, although I would prefer to do without the boost_python dll.

I did not try on linux, but it should be more straightforward there.

Beamends

Answer 5

You can even specify the python distribution via

./bootstrap.sh --with-python=<path to your python binary>

e.g.

./bootstrap.sh --with-python=python3

for your system's python3 or

./bootstrap.sh --with-python=$VIRTUAL_ENV/bin/python

for the python version of your currently active virtual env python.

Regretful

Answer 6

Refer this to know how to build boost with python. It shows the way to build with python2 with Visual Studio 10.0 (2010). But I go through the same procedure for a project that I am currently working on and it works fine with python 3.5 and Visual Studio 14.1 (2017).

If you get this error when building your python boost project, just add BOOST_ALL_NO_LIB value to Preprocessor Definitions (inside C\C++ > preprocessor tab) in your project properties.
And also, do not forget to add boost .dll files location to your system path.

Mchale

Answer 7

When the path to Python contains spaces, you will be in for quite a ride. After a whole lot of trial and error, I finally managed to get something that works. Behold my user-config.jam (which has to be in my home directory for bjam to find it):

import toolset : using ;

using python : 3.6
         : \"C:\\Program\ Files\ (x86)\\Microsoft\ Visual\ Studio\\Shared\\Python36_64\\python.exe\"
         : C:\\Program\ Files\ (x86)\\Microsoft\ Visual\ Studio\\Shared\\Python36_64\\include
         : C:\\Program\ Files\ (x86)\\Microsoft\ Visual\ Studio\\Shared\\Python36_64\\libs
         ;

The inconsistent quoting is intended and seems to be required. With this I can build boost-python and use it as Boost::python36 in my CMakeLists.txt. Still, one issue remains: I have to link to python manually viz

target_link_libraries(MyTarget
    Boost::boost Boost::python36
"C:/Program Files (x86)/Microsoft Visual Studio/Shared/Python36_64/libs/python36.lib")
target_include_directories(MyTarget PRIVATE
    "C:/Program Files (x86)/Microsoft Visual Studio/Shared/Python36_64/include")

Tiebold

Answer 8

In my case adding "Using Python : 3 etc." into user-config.jam in my home directory didn't work. I had to add the line into project-config.jam instead, which resides in the root directory of unpacked boost.

Specifically the line was:

using python : 3.9 : /usr/bin/python3 : /usr/include/python3.9 : /usr/lib ;

and the version of boost was 1_78_0

Scrivenor