I have a high-performant function written in Julia, how can I use it from Python?

Question

I have a found a Julia function that nicely does the job I need. How can I quickly integrate it to be able to call it from Python?

Suppose the function is

f(x,y) = 2x.+y

What is the best and most elegent way to use it from Python?

Micrococcus

Answer 1

Assuming your Python and Julia are installed you need to take the following steps.

1. Run Julia and install PyCall

   using Pkg
   pkg"add PyCall"
   

2. Put your code into a Julia package

   using Pkg
   Pkg.generate("MyPackage")
   

   In the folder src you will find MyPackage.jl, edit it to look like this:

   module MyPackage
   f(x,y) = 2x.+y
   export f
   end
   

3. Install pyjulia

   python -m pip install julia
   

   (On Linux systems you might want to use python3 instead of python command)

   For this step note that while an external Python can be used with Julia. However, for a convenience it might be worth to consider using a Python that got installed together with Julia as PyCall. In that case for installation use a command such this one:

   %HOMEPATH%\.julia\conda\3\python -m pip install julia
   

   or on Linux

   ~/.julia/conda/3/python -m pip install julia
   

   Note that if you have JULIA_DEPOT_PATH variable defined you can replace %HOMEPATH%\.julia or ~/.julia/ with its value.

4. Run the appropiate Python and tell it to configure the Python-Julia integration:

   import julia
   julia.install()
   

5. Now you are ready to call your Julia code:

   >>> from julia import Pkg
   >>> Pkg.activate(".\\MyPackage") #use the correct path
       Activating environment at `MyPackage\Project.toml`
   >>> from julia import MyPackage
   >>> MyPackage.f([1,2],5)
       [7,9]
   

It is worth noting that the proposed approach in this answer has several advantages over a standalone Julia file which would be possible, although is less recommended. The advantages include:

1. Packages get precompiled (so they are faster in subsequent runs) and can be loaded as a package in Python.
2. Packages come with their own virtual environment via 1Project.toml` which makes production deployments much comfortable.
3. A Julia package can be statically compiled into Julia's system image which can slash itsloading time --- see https://github.com/JuliaLang/PackageCompiler.jl .

EDIT

In February 2022 juliacall was announced, as of December 2022 juliacall might be an easier option for some users - have a look at: How to load a custom Julia package in Python using Python's juliacall

Micrococcus

Do you actually need to install PyCall.jl at all? – Adventure

Yes! However, I could just let julia.install() do it. When you run julia.install() from Python it checks for Julia PyCall and installs or rebuilds it as needed. Since I like installing PyCall myself rather than letting others do it I added a step. As a side result in this way you also have option of using Python bundled together with PyCall which is basically, at least for me, more comfortable to use. Like any tutorial this is subjective but I wrote it because I could not find a Python-to-Julia tutorial that I liked. – Micrococcus

@Przemyslaw, have you found or written a tutorial that you like, since October ? (I too find tutorials important, and subjective -- concise and precise is rare.) – Ennius

@Ennius I assume that this question is my version of tutorial. If you feel that anything is missing here let me know - I will be glad to update it. – Micrococcus

I found this answer very helpful for a problem I had...additionally I wanted to call my fast julia function in parallel using Python's pool, and to make that work (I guess pyJulia is not thread safe?) I had to do: multiprocessing.set_start_method("spawn",force=True) Additionally I didn't have my PyCall configured correctly for pyJulia, so make sure you have this set up right (it will give you an error telling you what's wrong) or you can work around this by doing: jl = julia.Julia(compiled_modules=False) Which I inserted before the from julia import Pkg block. – Chaste

Consider making the Julia function parallel instead. Julia has much more robust parallelism than Python. An equivalent of import multiprocessing in Julia is using Distributed. Julia has multi-threading too. – Micrococcus

Answer 2

I found that the python package PyCall did not work for me (segaults in unexplicable ways) however the more recent library JuliaCall worked. The differences are listed in the above link to the official pip-package. There is also an example on how to get started but I will describe the process here as well.

Advantages of using juliacall

The advantages of using juliacall over pycall are

• it works more reliably for me (python 3.9/3.10, julia 1.6)
• I can use my existing Julia environment (in production one might want to create a seperate julia-env which is also possible, see the default way in the docs)
• just need a single julia script file (dont need to create a package)
• Furthermore the documentation for juliacall is better: https://cjdoris.github.io/PythonCall.jl/stable/juliacall/

---

Instructions

1. Install PythonCall into Julia either by entering package mode via ] and writing add PythonCall or execute in normal julia mode:

julia> using Pkg
julia> Pkg.add("PythonCall")

2. Install the python package juliacall into you python environment:

$ python -m pip install juliacall

3. note the path of your julia executable. Enter the shell mode in julia with the key ; and execute:

shell> which julia
/home/user/.juliaup/bin/julia

4. note the path to your julia environment by first importing Pkg and then entering shell mode again to execute:

julia> using Pkg
shell> pwd $(Pkg.envdir())
/home/user/.julia/environments
shell> ls -lah $(Pkg.envdir())
v1.6

So my julia-env path is: /home/user/.julia/environments/v1.6/

5. Execute the following python script:

import os
# setting env variables so that juliacall finds the correct julia env
# see the github page for docs about env variables https://github.com/cjdoris/PyJuliaPkg
# juliacall makes use of the package pyjuliapkg to find the correct julia version, so you can read up on that here: https://github.com/cjdoris/pyjuliapkg/blob/main/src/juliapkg/find_julia.py
os.environ['PYTHON_JULIAPKG_EXE'] = "/home/user/.juliaup/bin/julia"
os.environ['PYTHON_JULIAPKG_OFFLINE']= 'yes' # such that nothing new is downloaded
os.environ['PYTHON_JULIAPKG_PROJECT'] = '/home/user/.julia/environments/v1.6/'

from juliacall import Main as jl


def main():
    print('in main...')
    
    # show installed julia packages to make sure you are in the right env 
    jl.seval('using Pkg')
    jl.seval('Pkg.status()') 

    # import your script and execute
    jl.seval('include("test_julia_simple.jl")')
    result = jl.seval('f([2,3], [4,5])')
    print(result)

    print('...done')
    return


if __name__ == "__main__":
    main()

The accompanying julia script test_julia_simple.jl contains your function and in this case looks as follows:

function f(x,y)
    return 2x.+y
end

Alcoholic

great! need to definitely try that. Stability of Python - to - Julia integration is an important issue. Have you used that in production and in particular dockerized environments with CPU affinity? – Micrococcus

I haven't used that in combination with docker. Also using that in in parallel a way was not successful sofar. (see my SO questioni on that #75451697) – Alcoholic

One more question. Are you able to replicate the environment where the pair Python juliacall or Python julia combined with PyCall segfaults. I have observed it few times but did not find the steps needed to reproduce this behavior. Some weeks ago I spoke with JuliaHub people and they would be very eager to repair that - but of course the first step is to identify the environment the replicates the behavior. Occasional Python-Julia segfaulting is and issue that actually negatively affects Julia adoption. – Micrococcus

I think it's better to move the discussion into chat: chat.stackoverflow.com/rooms/252108/parallel-juliacall – Alcoholic