Simplest way to communicate between Python and C# using IPC?

Question

I have some C# code that needs to call a Python script several thousand times, each time passing a string, and then expecting a float back. The python script can be run using ANY version of Python, so I cannot use Iron python. It's been recommended that I use IPC named pipes. I have no experience with this, and am having trouble figuring out how to do this between C# and Python. Is this a simple process, or am I looking at a decent amount of work? Is this the best way to solve my problem?

Babette

Is STDIN/STDOUT fast enough? I'd expect that to be a good approach. – Outrelief

That's what i'm using now. I need to improve the speed. – Babette

Answer 1

Use zeromq.

• allows very light messaging without a need for broker
• is available for many platforms
• can communicate over TCP socket, unix socket or inside a process

Here is my answer using zeromq https://mcmap.net/q/726497/-distributed-lock-manager-for-python

It serves for messaging between different Python programs, but exactly the same method of communication can be used between other platforms. Just keep the messages passed around interopable - both sides must understand the content properly. You can play with binary data, but very often json representation work quick and easy too. But there are many serialization frameworks like result buffers and others, which might help you to encode result.

Kierkegaardian

This could be a great solution, but may have security implications because you are binding to a tcp port. As a minimum, restrict to localhost unless you really do want to go live over the net. – Thirzi

@Babette regarding the speed, be prepared for really fast one. Tens of thousands of messages a second, with smaller messages it often outperforms TCP as it manages multiple messages into one TCP packet (depends on real usage scenario). – Kierkegaardian

Complete newbie to this area of coding. Can you explain very briefly what would be needed on the c# side for this to work? – Babette

@Babette for C# binding see: github.com/zeromq/clrzmq Also the page zeromq.org is very good starting point for zeromq in general - it is very well written - you will laugh quite often. – Kierkegaardian

@Babette If you have some Python experience, I would suggest starting experimenting in Python, it shall be more interactive and easy. Apart from general ZMQ guide you can find great (functional and short) examples at github.com/zeromq/pyzmq/tree/master/examples – Kierkegaardian

Answer 2

Environment initiation

In order to achieve this, firstly, you will need to set up the python environment inside the directory of the C# executable in order to have a modular application and also to enable an effective inter-process communication. The modularity aspect offered by this approach is given by the fact that the application is now independent as a whole from the operating system, because it has a self contained Python virtual environment in which Python applications can be run. This will make the application not depend on the host machine's probability of having a Python runtime already installed. To create a virtual environment on either Linux or Windows enter: python -m venv [ PATH WHERE THE ENVIRONMENT SHOULD BE CREATED ]. On Linux, in order to create Python virtual environments, you need to enter the command sudo apt-get install python[python version]-venv, in order to download the package that has the functionality of creating Python virtual environments, for example: sudo apt-get install python3.11-venv

After the environment is initialized, go to the directory where the python.exe is located on Windows, or python binary is located on Linux, and create or upload your Python script files that need to be run.

Inter-Process communication methods

Script initialization with parameters

An elegant and stable method of inter-process communication will be to call the script by passing parameters on the script initialization. This method is perfect if you only have to pass some information to the script and no data exchange between the C# application and the Python application is performed.

[ C# Code ]

        static void Main(string[] args)
        {
            System.Diagnostics.Process proc = new System.Diagnostics.Process();  // <-----   Process object
            proc.StartInfo.WorkingDirectory = "C:\\Users\\Teodor Mihail\\source\\repos\\Inter_Process_Communication_Example_Python\\Inter_Process_Communication_Example_Python\\bin\\Debug\\Scripts";   // <----- Path where python executable is located
            proc.StartInfo.FileName = "python.exe";   // <-----  Executable name ( Windows ) or binary (Linux/MacOS)
            proc.StartInfo.Arguments = "main.py Param1 Param2";  // // <----- Python file to be executed by the Python executable and the command line arguments passed to the process
            proc.Start();  // <---- Start the process



            Console.ReadLine();
        }

[ Python Code ]

import sys


def main(param1, param2):
    print("Param1 =", str(param1))
    print("Param2=", str(param2))


# Get the first command line argument passed
# to the application and store it in a variable.
received_param1 = sys.argv[1]


# Get the second command line argument passed
# to the application and store it in a variable.
received_param2 = sys.argv[2]


# Call the "main" function and pass the two command 
# line arguments to the method as parameters
main(received_param1, received_param2)
input()

Real-time data transfer using stdin, stdout, and stderr

Stdin, Stdout, and Stderr, are the primary I/O streams for applications used by the OS kernel to receive input, send output, and send error messages related to the functions. These primary I/O streams can be used for inter-process communication and this is done by redirecting the flow of these I/O streams from the child process to the OS, to the parent-process.

[ C# Code ]

        static void Main(string[] args)
        {
            System.Diagnostics.Process proc = new System.Diagnostics.Process(); // <-----   Process object
            proc.StartInfo.WorkingDirectory = "C:\\Users\\Teodor Mihail\\source\\repos\\Inter_Process_Communication_Example_Python\\Inter_Process_Communication_Example_Python\\bin\\Debug\\Scripts"; // <----- Path where python executable is located
            proc.StartInfo.FileName = "python.exe";  // <-----  Executable name ( Windows ) or binary (Linux/MacOS)
            proc.StartInfo.Arguments = "main.py";  // <----- Python file to be executed by the Python executable
            proc.StartInfo.RedirectStandardInput = true;  // <----- Redirect the Stdin stream of the Python application through the C# application
            proc.StartInfo.RedirectStandardOutput = true;  // <----- Redirect the Stdout stream of the Python application through the C# application
            proc.StartInfo.RedirectStandardError = true;  // <----- Redirect the Stderr stream of the Python application through the C# application
            proc.StartInfo.UseShellExecute = false;  // <----- Do not use the OS shell to execute the application and use the C# application as the shell
            proc.Start();  // <---- Start the process


            // Read the output of the Python application on the Stdout stream
            char[] buffer = new char[1000];
            proc.StandardOutput.Read(buffer, 0, buffer.Length);
            Console.WriteLine(buffer);


            // Send a message to the Python application through the Stdin stream
            proc.StandardInput.WriteLine("Hello from C# application over STDIN");
            proc.StandardInput.FlushAsync();


            // Read the output of the Python application on the Stdout stream
            buffer = new char[1000];
            proc.StandardOutput.Read(buffer, 0, buffer.Length);
            Console.WriteLine(buffer);


            // Read the error message thrown by the Python application on the Stderr stream
            buffer = new char[1000];
            proc.StandardError.Read(buffer, 0, buffer.Length);
            Console.WriteLine(buffer);

            Console.ReadLine();
        }

[ Python Code ]

import sys

def main():

    # Send a message to the C# application on the Stdout stream
    sys.stdout.write("Hello from Python application over STDOUT")


    # Receive a message from the C# application on the
    # Stdin stream and store it inside a variable.
    received = input()


    # Send the message received from the C# application
    # back to the C# application through the Stdout stream
    sys.stdout.write(received)


    # Send an error message through the Stderr stream to the C# application
    raise Exception("\n\n\nHello from Python application over STDERR")


main()

The data from the stdout and stderr must be read char-wise, otherwise the streams could lock in a wait for response state. This is because the stdout and stderr are not asychronous I/O streams and these could cause the streams to lock waiting for the buffered data: https://devblogs.microsoft.com/oldnewthing/20110707-00/?p=10223

char[] buffer = new char[1000];
proc.StandardOutput.Read(buffer, 0, buffer.Length);

Real-time data transfer using named pipes

Pipes are a type of socket that use the OS file system to send and receive information on a connection. This type of IPC is ideal for fast data transfer and has the advantage over stdin, stdout, and stderr of having the capacity of having multiple running connections at once: https://www.baeldung.com/cs/pipes-vs-sockets.

[ C# Code ]

        static void Main(string[] args)
        {
            System.Diagnostics.Process proc = new System.Diagnostics.Process(); // <-----   Process object
            proc.StartInfo.WorkingDirectory = "C:\\Users\\Teodor Mihail\\source\\repos\\Inter_Process_Communication_Example_Python\\Inter_Process_Communication_Example_Python\\bin\\Debug\\Scripts"; // <----- Path where python executable is located
            proc.StartInfo.FileName = "python.exe";  // <-----  Executable name ( Windows ) or binary (Linux/MacOS)
            proc.StartInfo.Arguments = "main.py";  // <----- Python file to be executed by the Python executable
            proc.Start();  // <---- Start the process




            // Named pipe server object with an "Out" direction. This means that this pipe can only send messages
            System.IO.Pipes.NamedPipeServerStream connection1 = new System.IO.Pipes.NamedPipeServerStream("main_write_pipe", System.IO.Pipes.PipeDirection.Out);


            try
            {
                // Wait for a conection to e established on the pipe. This is a blocking method call, meaning that the thread will wait for this method to finish the execution
                connection1.WaitForConnection();



                // Byte buffer that stores the UTF8 encoded binary value of the string "Message from C# application over FIFO Pipe"
                byte[] buffer = Encoding.UTF8.GetBytes("Message from C# application over FIFO Pipe");



                // Write the binary buffer's contents on the pipe's I/O stream
                connection1.Write(buffer, 0, buffer.Length);



                // Flush the binary buffer's contents on the pipe's I/O stream
                connection1.Flush();
            }
            catch
            {

            }
            finally
            {
                if (connection1 != null)
                {
                    connection1.Dispose();
                }
            }




            // Named pipe server object with an "In" direction. This means that this pipe can only read messages
            System.IO.Pipes.NamedPipeServerStream connection2 = new System.IO.Pipes.NamedPipeServerStream("main_read_pipe", System.IO.Pipes.PipeDirection.In);


            try
            {
                // Wait for a conection to e established on the pipe. This is a blocking method call, meaning that the thread will wait for this method to finish the execution
                connection2.WaitForConnection();



                // Byte buffer that stores the UTF8 encoded binary value of the string "Message from Python application over FIFO Pipe"
                byte[] buffer = new byte[1024];
                connection2.Read(buffer, 0, buffer.Length);



                // Print the message
                Console.WriteLine(Encoding.UTF8.GetString(buffer));
            }
            catch
            {

            }
            finally
            {
                if (connection1 != null)
                {
                    connection1.Dispose();
                }
            }




            Console.ReadLine();

        }

[ Python Code ]

def main():
    # On Linux and MacOs the pipes created by C# are located in "/tmp" so you have to enter "/tmp/pipe_name"


    # Open the OS's file system pipe FIFO and specify that it has only "read" permissions.
    # This must be done because the "main_write_pipe" pipe server in C# is created with
    # write permissions and the receiver can only read from the stream.
    pipe_fifo1 = open(r"\\.\pipe\main_write_pipe", "r")


    # Read the content from the stream and store it in a variable.
    # Because the stream is buffered, the data will be received
    # after the pipe server instance in the C# application is
    # closed.
    received = pipe_fifo1.readline()


    # Open the OS's file system pipe FIFO and specify that it has only "write" permissions.
    # This must be done because the "main_read_pipe" pipe server in C# is created with
    # read permissions and the receiver can only write to the stream.
    pipe_fifo1 = open(r"\\.\pipe\main_read_pipe", "w")


    # Write the content to the pipe stream
    pipe_fifo1.write("Message from Python over FIFO Pipe")


    # Flush the stream to ensure that all the data within the
    # stream is flushed to the receiver.
    pipe_fifo1.flush()
    input()


main()

Real-Time data transfer using TCP/UDP Sockets

Sockets are a type of binary receiver/transmitters that receive or send binary data over an IP address. For inter-process data communication the IP address used is the loopback address (127.0.0.1), which is the address used by the computer for self-calls.

[ C# Code ]

        static void Main(string[] args)
        {
            System.Diagnostics.Process proc = new System.Diagnostics.Process();
            proc.StartInfo.WorkingDirectory = "C:\\Users\\Teodor Mihail\\source\\repos\\Inter_Process_Communication_Example_Python\\Inter_Process_Communication_Example_Python\\bin\\Debug\\Scripts";
            proc.StartInfo.FileName = "python.exe";
            proc.StartInfo.Arguments = "main.py";
            proc.Start();

            Socket server = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            server.Bind(new System.Net.IPEndPoint(System.Net.IPAddress.Loopback, 80));
            server.Listen(1);

            try
            {

                Socket client = server.Accept();
                try
                {
                    byte[] buffer = Encoding.UTF8.GetBytes("Message from Python application over TCP");

                    client.Receive(buffer, 0);

                    Console.WriteLine(Encoding.UTF8.GetString(buffer));

                    buffer = Encoding.UTF8.GetBytes("Message from C# application over TCP");

                    client.Send(buffer, 0);
                }
                catch
                {

                }
                finally
                {
                    if(client != null)
                    {
                        client.Dispose();
                    }
                }
            }
            catch
            {

            }
            finally
            {
                if(server != null)
                {
                    server.Dispose();
                }
            }


            Console.ReadLine();
        }

[ Python Code ]

import socket


def main():
    client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    client.connect(("127.0.0.1", 80))

    client.send("Message from Python application over TCP".encode("utf-8"))
    info = client.recv(len("Message from C# application over TCP".encode("utf-8")))
    print(info.decode("utf-8"))

    input()


main()

Pashalik

can you please crop your screenshots to get rid of all the huge wasted space that isn't relevant? – Krahmer

brilliant answer, [Real-time data transfer using named pipes] worked for me with the following tweaks: I had to declare both new System.IO.Pipes.NamedPipeServerStream() pipes at front (else, the python script would not find main_read_pipe); I had to change connection1.Write() to connection1.WriteAsync() and connection2.Read() to connection2.ReadAsync() – Excited

Answer 3

Based on the what you have said, you can connect to the python process and catch standard output text. Easy, fast and reliable!

Boutis