Python and .Net Integration options

Question

I want to integrate Python with C#. I found two approaches using Interprocess communication and IronPython

Interprocess communication requires Python.exe to be installed on all client machines so not a viable solution.

We started using IronPython, but it only supports 2.7 python version for now. We are using 3.7 version.

Following code we tried using IronPython:

private void BtnJsonPy_Click(object sender, EventArgs e)
    {
        // 1. Create Engine
        var engine = Python.CreateEngine();

        //2. Provide script and arguments
        var script = @"C:\Users\user\source\path\repos\SamplePy\SamplePy2\SamplePy2.py"; // provide full path
        var source = engine.CreateScriptSourceFromFile(script);

        // dummy parameters to send Python script 
        int x = 3;
        int y = 4;

        var argv = new List<string>();
        argv.Add("");
        argv.Add(x.ToString());
        argv.Add(y.ToString());

        engine.GetSysModule().SetVariable("argv", argv);

        //3. redirect output
        var eIO = engine.Runtime.IO;

        var errors = new MemoryStream();
        eIO.SetErrorOutput(errors, Encoding.Default);

        var results = new MemoryStream();
        eIO.SetOutput(results, Encoding.Default);

        //4. Execute script
        var scope = engine.CreateScope();

        var lib = new[]
        {
            "C:\\Users\\user\\source\\repos\\SamplePy\\CallingWinForms\\Lib",
            "C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.2.7.9\\lib",
            "C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.2.7.9",
            "C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.StdLib.2.7.9"
            //"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37 - 32\\Lib",
            //"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37-32\\python.exe",
            //"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37 - 32",
            //"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37-32\\DLLs"
        };

        engine.SetSearchPaths(lib);
        engine.ExecuteFile(script, scope);
        //source.Execute(scope);

        //5. Display output
        string str(byte[] x1) => Encoding.Default.GetString(x1);

        Console.WriteLine("Errrors");
        Console.WriteLine(str(errors.ToArray()));
        Console.WriteLine();
        Console.WriteLine("Results");
        Console.WriteLine(str(results.ToArray()));

        lblAns.Text = str(results.ToArray());
    }

The problem sometimes is, to do heavy Machine Learning programming we need to add "Modules". These Modules would be dependent on other modules. This increases point 4, Execute Scripts part of code, as more data path of that module has to be given here var lib = new[] and also some modules are not supported with Iron Python as well (for e.g. modules concerning OCR operations etc.)

Due to these limitations I found Pythonnet which also helps in integrating .net applications with Python. But I am new to it, so want some ideas on implementing the same, and code samples available, and is it feasible or recommended to use with Python 3.7

I checked that setting up Pythonnet is cumbersome initially, so want help or steps on how to set up the same. Also would like to know if in future Iron Python would support Python 3.X as well or not.

Petrous

Answer 1

I am not familiar with IronPython, but I use pythonnet quite a lot for the same purpose - integrate Python with C#, so I can elaborate on that.

The advantage of using pythonnet for your purposes is having all the CPython packages available for you to use (numpy, scipy, pandas, Theano, Keras, scikit-learn etc), but avoiding the overhead of calling python.exe as separate process (pythonnet works by loading pythonXY.dll into your process).

Pay attention that pythonnet also requires to have stand-alone Python availiable, but you can use Embeddable Python package which is very light-weight and can be distributed with your application.

pythonnnet supports Python 3.7, but the published NuGet packages are only for Python 3.5. You have several choices to obtain pythonnet for Python 3.7:

1. Download pythonnet wheel package from PyPi and extract Python.Runtime.dll from it
2. Download NuGet package from pythonnet appveyor build artifacts, as advised on pythonnet installation wiki
3. Build from sources

Important note: pythonnet version has to match your Python version and bitness. For example, if you are using Python 3.7 32-bit, download pythonnet-2.4.0-cp37-cp37m-win32.whl. If your Python is 64-bit, download pythonnet-2.4.0-cp37-cp37m-win_amd64.whl. Your C# project platform target also has to match (x86 for 32-bit or x64 for 64-bit).

Code sample with similar functionality to what you have posted, using pythonnet (tested with Python 3.7.4 on Windows 7 and pythonnet NuGet from latest build artifacts):

        private void Test()
        {
            // Setup all paths before initializing Python engine
            string pathToPython = @"C:\Users\user\AppData\Local\Programs\Python\Python37-32";
            string path = pathToPython + ";" +
                          Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine);
            Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
            Environment.SetEnvironmentVariable("PYTHONHOME", pathToPython, EnvironmentVariableTarget.Process);

            var lib = new[]
            {
                @"C:\Users\user\source\path\repos\SamplePy\SamplePy2",
                Path.Combine(pathToPython, "Lib"),
                Path.Combine(pathToPython, "DLLs")
            };

            string paths = string.Join(";", lib);
            Environment.SetEnvironmentVariable("PYTHONPATH", paths, EnvironmentVariableTarget.Process);

            using (Py.GIL()) //Initialize the Python engine and acquire the interpreter lock
            {
                try
                {
                    // import your script into the process
                    dynamic sampleModule = Py.Import("SamplePy");

                    // It is more maintainable to communicate with the script with
                    // function parameters and return values, than using argv 
                    // and input/output streams.

                    int x = 3;
                    int y = 4;
                    dynamic results = sampleModule.sample_func(x, y);
                    Console.WriteLine("Results: " + results);
                }
                catch (PythonException error)
                {
                    // Communicate errors with exceptions from within python script -
                    // this works very nice with pythonnet.
                    Console.WriteLine("Error occured: ", error.Message);
                }
            }
        }

SamplePy.py:

def sample_func(x, y):
    return x*y

Burkes

This is exactly what I was looking for. Works like a charm.Thanks. – Knar

So, the difference between IronPython and Python.NET is that w. IronPython you don't need to have Python installed on the client computer correct? If I wanted to use Python in a .NET Core website, then Python.NET wouldn't work would it unless Python was installed on the server that's hosting the site, correct? – Heliogabalus

Python .NET requires separate Python installation, that has to match in version and bitness. I don't know about IronPython. – Burkes

Instead sending the file to engine how can we pass the python script – Denverdeny

@Denverdeny you mean instead of importing the module? There is also an option to pass the script as string to the python engine, for example: PythonEngine.RunSimpleString("import sys"); – Burkes

@Burkes how can we read output from that engine. Do you have any sample please share with me. – Denverdeny

@Denverdeny No, I don't have such sample, you will have to figure out something. Maybe write to a file? – Burkes

Be careful with using Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine): if other components already modified the PATH of the process (using EnvironmentVariableTarget.Process) and relies on that, overwriting that with a different path might be unwanted. – Oviposit