I want to get output from two processes and merge them into one file, like:
proc1 >> output &
proc2 >> output &
The problem is that output may be mixed up in the final file. For example if first process writes:
hellow
and the second process writes:
bye
the result may be something like:
hebylloe
but I expect them to be in seperate lines like (order is not important):
bye
hello
So I used flock to synchronize writing to the file with the following script:
exec 200>>output
while read line;
flock -w 2 200
do echo $line>>output
flock -u 200
done
And run the processes like:
proc1 | script &
proc2 | script &
Now the problem is that the performance is decreased significantly. without synchronization each process could write with the speed of 4MB/sec but using the synchronization script the write speed is 1MB/sec.
Can anyone help me how to merge the output from two processes and prevent mixing outputs up?
edit: I realized that there is a relation between line length and std buffer size, if size of each line is less than std buffer size, then every thing works well, nothing is mixed (at least in my tests). so I ran each script with bufsize command:
bufsize -o10KB proc1 | script &
bufsize -o10KB proc2 | script &
Now I want to make sure that this solution is bulletproof. I can not find any relation between buffer size and what happens now!!!
echo "$line" >> output
(with quotes). – Reconditeoutput
of 4 Mb/sec ? – Recondite