Python process forked by NodeJS - Alternative to process.send() for Python?

Question

I'm forking a Python script with NodeJS and when forked, by default, NodeJS create an IPC between this new process and the parent.

With NodeJS, to send message from a child to the parent I do process.send({msg : 'toto'})

How can I do that with Python ?

http://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options

Melamine

Have you checked semanchuk.com/philip/posix_ipc – Giacometti

I just checked it, so it's a separated module ? Is there a way to use what Python provide, builtin, to try to interact with NodeJS IPC ? – Melamine

Unfortunately I do not know. If knew I would post an answer, not a comment as a hint :) – Giacometti

For future lurkers, it's possible to have a Python parent and a Node.js child too: https://mcmap.net/q/1325065/-node-js-ipc-between-a-python-parent-and-a-node-js-child/242684 – Juju

Answer 1

Ok I found it, finally is quite easy. It's only about writing on the right file descriptor.

On the NodeJS side parameter, spawn your script like that :

var child = child_process.spawn('python', ['hello.py'], {
  stdio:[null, null, null, 'ipc']
});

child.on('message', function(message) {
  console.log('Received message...');
  console.log(message);
});

As the 'ipc' channel is the 4rd parameter, you will have to write on the filedescriptor 3. On the Python side :

import os

os.write(3, '{"dt" : "This is a test"}\n', "utf8")

Done. You will receive the message on the child.on('message' callback.

Cheers !

Melamine

how about python read from node.js? I tried ipc option on stdin, it keeps reading and blocks execution – Tinaret

github.com/extrabacon/python-shell - it has a send() that sends data to stdin of the python program – Ultraviolet

In Python 3, one has to use the bytes function: os.write(3, bytes('{"dt" : "This is a test"}\n', 'utf8')) – Vanitavanity

Answer 2

After following @Unitech's answer and running into issues, I discovered that there's a bit more to what's happening. I'm unsure if this has to do with my Node version, or something else. (I'm running v16.13.1 and python 3.10.8) However it appears to be that the way messages are traded has changed. The message now starts with 8 bytes of something, and 8 more bytes which is the message length in little Endian format.

Anyway, I've written the following python code which I've had success with sending and receiving messages with.

import os
import json

# get the FD from ENV
NODEIPCFD = int(os.environ["NODE_CHANNEL_FD"])

def sendMessage(text):
  'sends a Node IPC message to parent proccess'
  # encode message as json string + newline in bytes
  bytesMessage = (json.dumps(text) + "\n").encode()
  # I'm not actually sure what this number is for,
  # but not including it causes problems.
  # probably encodes something to do with the 'advanced' serialization
  os.write(NODEIPCFD, int.to_bytes(1, 8, "little"))
  # send the length as an 8-byte number in little Endian format
  os.write(NODEIPCFD, len(bytesMessage).to_bytes(8, "little"))
  # send message
  os.write(NODEIPCFD, bytesMessage)


def readMessage():
  'read in next message from parent Node process via built-in node IPC'
  # read and discard 8 bytes. Again, don't know why...
  os.read(NODEIPCFD, 8)
  # read and parse 8 bytes as length in little Endian format
  length = int.from_bytes(os.read(NODEIPCFD, 8), "little")
  # read 'length' bytes and pass to json parser
  return json.loads(os.read(NODEIPCFD, length))

I started the python code using this on the node side:

const child_process = require('node:child_process');

var cp = child_process.spawn('python', ['child_process.py'], {
  stdio:[null, null, null, 'ipc']
});

I just wish the Node documentation included the underlying process for how messages are traded.

Radiothorium