How to read to and write from a pipe in Perl?

Question

I need to modify an existing Perl program. I want to pipe a string (which can contain multiple lines) through an external program and read the output from this program. This external program is used to modify the string. Let's simply use cat as a filter program. I tried it like this, but it doesn't work. (Output of cat goes to STDOUT instead of being read by perl.)

#!/usr/bin/perl

open(MESSAGE, "| cat |") or die("cat failed\n");
print MESSAGE "Line 1\nLine 2\n";
my $message = "";
while (<MESSAGE>)
{
    $message .= $_;
}
close(MESSAGE);
print "This is the message: $message\n";

I've read that this isn't supported by Perl because it may end up in a deadlock, and I can understand it. But how do I do it then?

Housekeeper

The perlipc manual page has a discussion and plenty of examples of different approaches. – Diamagnetic

The simplest solution is: $pid = open(README, "program arguments |"); $pid = open(WRITEME, "| program arguments") – Superordinate

Answer 1

You can use IPC::Open3 to achieve bi-directional communication with child.

use strict;
use IPC::Open3;

my $pid = open3(\*CHLD_IN, \*CHLD_OUT, \*CHLD_ERR, 'cat')
    or die "open3() failed $!";

my $r;

for(my $i=1;$i<10;$i++) {
    print CHLD_IN "$i\n";
    $r = <CHLD_OUT>;
    print "Got $r from child\n";
}

Centering

Works perfectly. Thanks. Used open2 instead of open3 because I don't care about stderr. – Housekeeper

And it's worth noting that the first two arguments to open2 are the opposite way around to open3! It's open2(\*CHLD_OUT, \*CHLD_IN, 'cat') for example. – Cowl

Answer 2

This involves system programming, so it’s more than a basic question. As written, your main program doesn’t require full-duplex interaction with the external program. Dataflow travels in one direction, namely

string → external program → main program

Creating this pipeline is straightforward. Perl’s open has a useful mode explained in the “Safe pipe opens” section of the perlipc documentation.

Another interesting approach to interprocess communication is making your single program go multiprocess and communicate between—or even amongst—yourselves. The open function will accept a file argument of either "-|" or "|-" to do a very interesting thing: it forks a child connected to the filehandle you’ve opened. The child is running the same program as the parent. This is useful for safely opening a file when running under an assumed UID or GID, for example. If you open a pipe to minus, you can write to the filehandle you opened and your kid will find it in his STDIN. If you open a pipe from minus, you can read from the filehandle you opened whatever your kid writes to his STDOUT.

This is an open that involves a pipe, which gives nuance to the return value. The perlfunc documentation on open explains.

If you open a pipe on the command - (that is, specify either |- or -| with the one- or two-argument forms of open), an implicit fork is done, so open returns twice: in the parent process it returns the pid of the child process, and in the child process it returns (a defined) 0. Use defined($pid) or // to determine whether the open was successful.

To create the scaffolding, we work in right-to-left order using open to fork a new process at each step.

1. Your main program is already running.
2. Next, fork a process that will eventually become the external program.
3. Inside the process from step 2
   1. First fork the string-printing process so as to make its output arrive on our STDIN.
   2. Then exec the external program to perform its transformation.
4. Have the string-printer do its work and then exit, which kicks up to the next level.
5. Back in the main program, read the transformed result.

With all of that set up, all you have to do is implant your suggestion at the bottom, Mr. Cobb.

#! /usr/bin/env perl

use 5.10.0;  # for defined-or and given/when
use strict;
use warnings;

my @transform = qw( tr [A-Za-z] [N-ZA-Mn-za-m] );  # rot13
my @inception = (
  "V xabj, Qnq. Lbh jrer qvfnccbvagrq gung V pbhyqa'g or lbh.",
  "V jnf qvfnccbvagrq gung lbh gevrq.",
);

sub snow_fortress { print map "$_\n", @inception }

sub hotel {
  given (open(STDIN, "-|") // die "$0: fork: $!") {  # / StackOverflow hiliter
    snow_fortress when 0;
    exec @transform or die "$0: exec: $!";
  }
}

given (open(my $fh, "-|") // die "$0: fork: $!") {
  hotel when 0;

  print while <$fh>;
  close $fh or warn "$0: close: $!";
}

Thanks for the opportunity to write such a fun program!

Brigidbrigida

Answer 3

I want to expand on @Greg Bacon's answer without changing it.

I had to execute something similar, but wanted to code without the given/when commands, and also found there was explicit exit() calls missing because in the sample code it fell through and exited.

I also had to make it also work on a version running ActiveState perl, but that version of perl does not work. See this question How to read to and write from a pipe in perl with ActiveState Perl?

#! /usr/bin/env perl

use strict;
use warnings;

my $isActiveStatePerl = defined(&Win32::BuildNumber);

sub pipeFromFork
{
    return open($_[0], "-|") if (!$isActiveStatePerl);
    die "active state perl cannot cope with dup file handles after fork";

    pipe $_[0], my $child or die "cannot create pipe";
    my $pid = fork();
    die "fork failed: $!" unless defined $pid;
    if ($pid) {         # parent
        close $child; 
    } else {            # child 
        open(STDOUT, ">&=", $child) or die "cannot clone child to STDOUT";
        close $_[0];
    }
    return $pid;
}


my @transform = qw( tr [A-Za-z] [N-ZA-Mn-za-m] );  # rot13
my @inception = (
  "V xabj, Qnq. Lbh jrer qvfnccbvagrq gung V pbhyqa'g or lbh.",
  "V jnf qvfnccbvagrq gung lbh gevrq.",
);

sub snow_fortress { print map "$_\n", @inception }

sub hotel 
{
    my $fh;
    my $pid = pipeFromFork($fh);     # my $pid = open STDIN, "-|";
    defined($pid)  or die "$0: fork: $!";
    if (0 == $pid) {
        snow_fortress;
        exit(0);
    }
    open(STDIN, "<&", $fh)  or  die "cannot clone to STDIN";
    exec @transform or die "$0: exec: $!";
}

my $fh;
my $pid = pipeFromFork($fh);            # my $pid = open my $fh, "-|";
defined($pid) or die "$0: fork: $!";
if (0 == $pid) {
    hotel;
    exit(0);
}

print while <$fh>;
close $fh or warn "$0: close: $!";

Flask

Answer 4

You can use the -n commandline switch to effectively wrap your existing program code in a while-loop... look at the man page for -n:

 LINE:
            while (<>) {
                ...             # your program goes here
            }

Then you can use the operating system's pipe mechanism directly

cat file | your_perl_prog.pl

(Edit) I'll try to explain this more carefully...

The question is not clear about what part the perl program plays: filter or final stage. This works in either case, so I will assume it is the latter.

'your_perl_prog.pl' is your existing code. I'll call your filter program 'filter'.

Modify your_perl_prog.pl so that the shebang line has an added '-n' switch: #!/usr/bin/perl -n or #!/bin/env "perl -n"

This effectively puts a while(<>){} loop around the code in your_perl_prog.pl

add a BEGIN block to print the header:

BEGIN {print "HEADER LINE\n");}

You can read each line with '$line = <>;' and process/print

Then invoke the lot with

cat sourcefile |filter|your_perl_prog.pl

Tonsillotomy

Answer 5

the simplest -- not involving all these cool internals -- way to do what the OP needs, is to use a temporary file to hold the output until the external processor is done, like so:

open ToTemp, "|/usr/bin/tac>/tmp/MyTmp$$.whee" or die "open the tool: $!";
print ToTemp $TheMessageWhateverItIs;
close ToTemp;
my $Result = `cat /tmp/MyTmp$$.whee`;  # or open and read it, or use File::Slurp, etc
unlink "/tmp/MyTmp$$.whee"; 

Of course, this isn't going to work for something interactive, but co-routines appear to be out of the scope of the original question.

Playmate

docstore.mik.ua/orelly/perl/cookbook/ch16_05.htm. $pid = open(README, "program arguments |"); $pid = open(WRITEME, "| program arguments") – Superordinate

yes that would work to, as would lang-perl my $Result = <<`END_SCRIPT`; /usr/bin/tac <<MARKER \Q$TheMessageWhateverItIs\E MARKER END_SCRIPT you have to add the newlines, can't do a code block in a comment – Playmate