Use an OS process like a bash pipe: Send it STDIN and get its STDOUT

Question

I'm trying to use an external process which reads the STDIN, and writes to STDOUT.

I want to write the equivalent of this in Elixir, without using an external library or wrapper script:

$ echo foo | nkf
foo

i.e. send data to nkf on stdin, and get the converted result back from nkf's stdout, knowing that it has finished processing the stream.

I was trying to do this with ports, but the problem is a single sent message can be returned in multiple received messages, so there's no way to tell when the end of the message has been reached (simplified example, "foo" is a whole file in reality):

iex(1)> port = Port.open({:spawn, "nkf -u"}, [:binary])
#Port<0.7>
iex(2)> Port.command(port, "foo")
true
iex(3)> flush
{#Port<0.7>, {:data, "fo"}}
{#Port<0.7>, {:data, "o"}}
:ok

How can I get the same bash pipe behaviour with Ports in Elixir?

Luciferase

is there a problem with using System.cmd/3? – Tirol

What would be the issue with concatenating the data? – Striated

The Port documentation seems to cover this: you send the port a :close message, and it sends you a :closed message back when the process is exited. – Lengthwise

@Tirol did you try it? – Luciferase

@AlekseiMatiushkin the problem is that I don't know when the end of the input is. – Luciferase

@DavidMaze I'll have another look, but in my previous attempts closing closed the stream, which caused a broken pipe. – Luciferase

It looks like this bash behaviour is just not possible in the beam right now without extra help, although I'd like to believe otherwise! Here's an 11-year-old related Erlang question: #6571266 – Luciferase

If what you want to do can be accomplished with C, C++, or Rust, another option is to write it in one of those languages and wrap it with a NIF. Rust makes this easy using a library called Rustler: github.com/rusterlium/rustler – Arium

@Arium yes that is also one of the suggestions on the linked question - however this question is for doing it using only the standard library. – Luciferase

A related question with some possible solutions in the comments: #76398978 – Tabling

Answer 1

The beam does not currently provide a way to close the stream to the process and wait for the stream from the process to finish sending. Using ports, closing the port will also close the stream from the external process, even if the process has not finished sending data. Due to this, it is not currently possible to do using only built-in features - it's necessary to enlist the help of external tools like porcelain or erlexec.

Luciferase

Answer 2

You can use :os.cmd("echo foo | nkf")

Mentioned here:

Shell commands If you desire to execute a trusted command inside a shell, with pipes, redirecting and so on, please check :os.cmd/1.

Bimonthly

Also System.shell/2. This offloads the communication to the shell, which has different behaviour and is vulnerable to injection attacks as documented. What I want to do is to talk to the running process directly. – Luciferase