StandardOutput.ReadToEnd() hangs [duplicate]

Question

I have a program that frequently uses an external program and reads its outputs. It works pretty well using your usual process redirect output, but one specific argument for some reason hangs when I try to read it, no error message - no exception, it just 'stops' when it reaches that line. I of course use a centralized function to call and read output from the program, which is this:

public string ADBShell(string adbInput)
{
    try
    {
        //Create Empty values
        string result = string.Empty;
        string error = string.Empty;
        string output = string.Empty;
        System.Diagnostics.ProcessStartInfo procStartInfo 
            = new System.Diagnostics.ProcessStartInfo(toolPath + "adb.exe");

        procStartInfo.Arguments = adbInput;
        procStartInfo.RedirectStandardOutput = true;
        procStartInfo.RedirectStandardError = true;
        procStartInfo.UseShellExecute = false;
        procStartInfo.CreateNoWindow = true;
        procStartInfo.WorkingDirectory = toolPath;
        System.Diagnostics.Process proc = new System.Diagnostics.Process();
        proc.StartInfo = procStartInfo;
        proc.Start();
        // Get the output into a string
        proc.WaitForExit();
        result = proc.StandardOutput.ReadToEnd();
        error = proc.StandardError.ReadToEnd();  //Some ADB outputs use this
        if (result.Length > 1)
        {
            output += result;
        }
        if (error.Length > 1)
        {
            output += error;
        }
        Return output;
    }
    catch (Exception objException)
    {
        throw objException;
    }
}

The line that hangs is result = proc.StandardOutput.ReadToEnd();, but again, not every time, only when sent a specific argument ("start-server"). All other arguments work just fine - it reads the value and returns it. It's also strange the way it hangs. It doesn't freeze or give an error or anything, it just stops processing. As if it was a 'return' command, except it doesn't even return to the calling function, it just stops everything with the interface still up and running. Anyone experienced this before? Anyone have any idea what I should try? I'm assuming it's something unexpected within the stream itself, but is there a way I can handle/ignore this so that it reads it anyway?

Hades

Here's where I got my [same] problem answered: #140093 – Gracious

You might be interested in this post, which explains how to handle deadlocking with .NET process streams. MedallionShell library, which simplifies dealing with process io streams – Limpet

Run in console the same program with the same parameters. It it is prompting for user interaction like entering password or congirmation after warnings, it might be the cause. For example pgAdmin (postgress database admiinistration) hangs asking for password for databases that are not in his configuration file. But this can only be seen running from console – Pattison

If you have used input stream, this answer is for you: https://mcmap.net/q/744177/-standardoutput-readtoend-hangs-duplicate – Impose

Answer 1

Proposed solutions with BeginOutputReadLine() are a good way but in situations such as that, it is not applicable, because process (certainly with using WaitForExit()) exits earlier than async output finished completely.

So, I tried to implement it synchronously and found that the solution is in using Peek() method from StreamReader class. I added check for Peek() > -1 to sure that it is not the end of the stream as in MSDN article described and finally it works and stop hanging!

Here is the code:

var process = new Process();
process.StartInfo.CreateNoWindow = true;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.StartInfo.WorkingDirectory = @"C:\test\";
process.StartInfo.FileName = "test.exe";
process.StartInfo.Arguments = "your arguments here";

process.Start();
var output = new List<string>();

while (process.StandardOutput.Peek() > -1)
{
    output.Add(process.StandardOutput.ReadLine());
}

while (process.StandardError.Peek() > -1)
{
    output.Add(process.StandardError.ReadLine());
}
process.WaitForExit();

Crownwork

I just implemented this change and my process is still hanging on the process.StandardError.ReadLine()... – Gracious

@Gracious may be your process.StandardError.ReadLine() returns null? – Crownwork

I'll try that check and let you know. – Gracious

Inside my while loop for the standarderror output I added an "if (errorString == Environment.NewLine) break;" statement and that worked, now it's hanging on the "process.WaitForExit();" line... – Gracious

Hmmm, even if I set process.RedirectStandardError = false, my process still hangs. I verified that my zip file is not corrupt (I'm executing a WinZip command-line process) so now I'm really stumped. – Gracious

@Gracious Sorry, haven't noticed your comment before. Have you solved your problem? Need to say, that in some (rare?) cases solution which I proposed works best, but in other cases it is more appropriate to use async output. – Crownwork

I got it fixed, but I implemented something that someone else answered from a different question, here's the link to the answer that I used: #140093 – Gracious

It work! A really strange problem. When I only have a few files, I never got the hang, but when I increase the number... Do you know what is the reason? – Gally

After rechecking, actually it "works" but cannot get any output from the process. I find a working solution here: #140093 – Gally

how about checking process.StandardOutput.EndOfStream instead? using your process.StandardOutput.Peek() > -1, only the first of my multi-line output is displayed – Cyrano

Mine was still hanging so I just added this inside the while loop: if (process.StandardOutput.EndOfStream) return; And also added a timeout as:process.WaitForExit(60000*10); to be safe. That will make sure the processes terminates in at most 10 minutes. I know that the processes that I am running should never take more than 10 minutes. – Sarsenet

@Gracious there is a known bug in streamwriter implementation, when using Peek() on an empty output it will hang forever – Paprika

Is there any reason you repeat that loop? – Behka

@Behka this is to ensure that all the output is read completely before disconnecting – Crownwork

Answer 2

The problem is that you are using the synchronous ReadToEnd methods on both the StandardOutput and the StandardError streams. This can lead to a potential deadlock you are experiencing. This is even described in the MSDN. The solution is described there. Basically, it is: Use the asynchronous version BeginOutputReadLine to read the data of the StandardOutput stream:

p.BeginOutputReadLine();
string error = p.StandardError.ReadToEnd();
p.WaitForExit();

Implementation of Async reading using BeginOutputReadLine see in ProcessStartInfo hanging on "WaitForExit"? Why?

Regalia

Thanks for replying. I'm afraid this didn't work, it still hung as soon as it got to 'StandardError.ReadToEnd();'. I even tried using 'BeginErrorReadLine();' but that also hung. The only thing that DID work was adding a timeout to 'WaitForExit'. Since this specific argument that hangs always gives an output almost immediately, I timed it out at about 3 seconds and everything works fine. It's not very elegant, but it works. Thanks again for helping. – Hades

This solution works and it is the solution depicted in the MSDN docs – Nuncle

Answer 3

What about something like:

process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();

process.OutputDataReceived += (sender, args) =>
{
    var outputData = args.Data;
    // ...
};
process.ErrorDataReceived += (sender, args) =>
{
    var errorData = args.Data;
    // ...
};
process.WaitForExit();

Gangster

So close. Set the event handler before turning on the pump. Then remove the event handler at the end. – Ga

Answer 4

I had the same deadlock problem. This code snippet worked for me.

ProcessStartInfo startInfo = new ProcessStartInfo("cmd")
{
    WindowStyle = ProcessWindowStyle.Hidden,
    UseShellExecute = false,
    RedirectStandardInput = true,
    RedirectStandardOutput = true,
    CreateNoWindow = true
};

Process process = new Process();
process.StartInfo = startInfo;
process.Start();
process.StandardInput.WriteLine("echo hi");
process.StandardInput.WriteLine("exit");
var output = process.StandardOutput.ReadToEnd();
process.Dispose();

Contrasty

Answer 5

I had the same kind of problem that error was just hanging.

Based on your response to Daniel Hilgarth I didn't even try using those codes though i think they would have worked for me.

Since I want to be able do some fancier output still eventually i decided that I would do it with both of the outputs being done in a background thread.

public static class RunCommands
{
    #region Outputs Property

    private static object _outputsLockObject;
    private static object OutputsLockObject
    { 
        get
        {
            if (_outputsLockObject == null)
                Interlocked.CompareExchange(ref _outputsLockObject, new object(), null);
            return _outputsLockObject;
        }
    }

    private static Dictionary<object, CommandOutput> _outputs;
    private static Dictionary<object, CommandOutput> Outputs
    {
        get
        {
            if (_outputs != null)
                return _outputs;

            lock (OutputsLockObject)
            {
                _outputs = new Dictionary<object, CommandOutput>();
            }
            return _outputs;
        }
    }

    #endregion

    public static string GetCommandOutputSimple(ProcessStartInfo info, bool returnErrorIfPopulated = true)
    {
        // Redirect the output stream of the child process.
        info.UseShellExecute = false;
        info.CreateNoWindow = true;
        info.RedirectStandardOutput = true;
        info.RedirectStandardError = true;
        var process = new Process();
        process.StartInfo = info;
        process.ErrorDataReceived += ErrorDataHandler;
        process.OutputDataReceived += OutputDataHandler;

        var output = new CommandOutput();
        Outputs.Add(process, output);

        process.Start();

        process.BeginErrorReadLine();
        process.BeginOutputReadLine();

        // Wait for the process to finish reading from error and output before it is finished
        process.WaitForExit();

        process.ErrorDataReceived -= ErrorDataHandler;
        process.OutputDataReceived -= OutputDataHandler;

        Outputs.Remove(process);

        if (returnErrorIfPopulated && (!String.IsNullOrWhiteSpace(output.Error)))
        {
            return output.Error.TrimEnd('\n');
        }

        return output.Output.TrimEnd('\n');
    }

    private static void ErrorDataHandler(object sendingProcess, DataReceivedEventArgs errLine)
    {
        if (errLine.Data == null)
            return;

        if (!Outputs.ContainsKey(sendingProcess))
            return;

        var commandOutput = Outputs[sendingProcess];

        commandOutput.Error = commandOutput.Error + errLine.Data + "\n";
    }

    private static void OutputDataHandler(object sendingProcess, DataReceivedEventArgs outputLine)
    {
        if (outputLine.Data == null)
            return;

        if (!Outputs.ContainsKey(sendingProcess))
            return;

        var commandOutput = Outputs[sendingProcess];

        commandOutput.Output = commandOutput.Output + outputLine.Data + "\n";
    }
}
public class CommandOutput
{
    public string Error { get; set; }
    public string Output { get; set; }

    public CommandOutput()
    {
        Error = "";
        Output = "";
    }
}

This worked for me and allowed me to not have to use a timeout for the read.

Phenacetin

Thank you for implementing the reference from MSDN! The only thing you are missing is that you aren't removing the event handlers at the end: process.ErrorDataReceived -= ErrorDataHandler; and process.OutputDataReceived -= OutputDataHandler; – Ga

I think the garbage collection would already remove them so it wouldn't matter. However I've edited the code to include your suggestion just in case. – Phenacetin

Thanks! From: learn.microsoft.com/en-us/dotnet/csharp/programming-guide/… "In order to prevent resource leaks, you should unsubscribe from events before you dispose of a subscriber object. Until you unsubscribe from an event, the multicast delegate that underlies the event in the publishing object has a reference to the delegate that encapsulates the subscriber's event handler. As long as the publishing object holds that reference, garbage collection will not delete your subscriber object." – Ga

Answer 6

Something that is elegant and worked for me is:

Process nslookup = new Process()
{
   StartInfo = new ProcessStartInfo("nslookup")
   {
      RedirectStandardInput = true,
      RedirectStandardOutput = true,
      UseShellExecute = false,
      CreateNoWindow = true,
      WindowStyle = ProcessWindowStyle.Hidden
   }
};

nslookup.Start();
nslookup.StandardInput.WriteLine("set type=srv");
nslookup.StandardInput.WriteLine("_ldap._tcp.domain.local"); 

nslookup.StandardInput.Flush();
nslookup.StandardInput.Close();

string output = nslookup.StandardOutput.ReadToEnd();

nslookup.WaitForExit();
nslookup.Close();

This answer I found here and the trick is using Flush() and Close() on standard input.

Forbid

If you got stucked when have used input stream, this answer is the right way. – Impose

Answer 7

The accepted answer's solution didn't work for me. I had to use tasks in order to avoid the deadlock:

//Code to start process here

String outputResult = GetStreamOutput(process.StandardOutput);
String errorResult = GetStreamOutput(process.StandardError);

process.WaitForExit();

With a GetStreamOutput function as follows:

private string GetStreamOutput(StreamReader stream)
{
   //Read output in separate task to avoid deadlocks
   var outputReadTask = Task.Run(() => stream.ReadToEnd());

   return outputReadTask.Result;
}

Bowles

I like this answer the best,... separate threads/tasks are even recommended by MSDN, but it still deadlocks for me. – Martinelli

Even this variant deadlocks string cvout = (Task<string>.Run(async () => { return await p.StandardOutput.ReadToEndAsync(); })).Result; – Martinelli

Even this variant deadlocks string cvout = (Task<string>.Run(async () => { return await p.StandardOutput.ReadToEndAsync().ConfigureAwait(false); })).Result; – Martinelli

Actually, the above deadlocks for a couple of minutes until some of the underlying threads timeout, and then it continues. The delay is unacceptable, but provides me with a ray of hope. – Martinelli

Here is what I ended up with: https://mcmap.net/q/87636/-processstartinfo-hanging-on-quot-waitforexit-quot-why – Martinelli

Answer 8

Just in case someone stumbles upon this question while wiling to use Windows Forms and TextBox (or RichTextBox) to show the errors and outputs the process returns in real time (as they are written to process.StandardOutput / process.StandardError).

You need to use OutputDataReceived() / ErrorDataReceived() in order to read both streams without deadlocks, there is no way (as far as I know) to avoid deadlocks otherwise, even Fedor's answer, which now holds the "Answer" tag and the most likes up to date, does not do the trick for me.

However, when you use the RichTextBox (or TextBox) to output the data, another problem you encounter is how to actually write the data into the textbox in real time (once it arrives). You receive the access to the data inside one of the background threads OutputDataReceived() / ErrorDataReceived() and you can only AppendText() from the main thread.

What I first tried doing was calling process.Start() from a background thread and then calling BeginInvoke() => AppendText() in OutputDataReceived() / ErrorDataReceived() threads while the main thread was process.WaitForExit().

However, this led to my form freezing and ultimately hanging for eternity. After a few days of trying I ended up with the solution below, that seems to work pretty well.

Shortly speaking, you need to add the messages into a concurrent collection inside OutputDataReceived() / ErrorDataReceived() threads while the main thread should constantly try to extract messages from that collection and append them into the textbox:

            ProcessStartInfo startInfo
                = new ProcessStartInfo(File, mysqldumpCommand);

            process.StartInfo.FileName = File;
            process.StartInfo.Arguments = mysqldumpCommand;
            process.StartInfo.CreateNoWindow = true;
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
            process.StartInfo.RedirectStandardInput = false;
            process.StartInfo.RedirectStandardError = true;
            process.StartInfo.RedirectStandardOutput = true;
            process.StartInfo.StandardErrorEncoding = Encoding.UTF8;
            process.StartInfo.StandardOutputEncoding = Encoding.UTF8;
            process.EnableRaisingEvents = true;

            ConcurrentQueue<string> messages = new ConcurrentQueue<string>();

            process.ErrorDataReceived += (object se, DataReceivedEventArgs ar) =>
            {
                string data = ar.Data;
                if (!string.IsNullOrWhiteSpace(data))
                    messages.Enqueue(data);
            };
            process.OutputDataReceived += (object se, DataReceivedEventArgs ar) =>
            {
                string data = ar.Data;
                if (!string.IsNullOrWhiteSpace(data))
                    messages.Enqueue(data);
            };

            process.Start();
            process.BeginErrorReadLine();
            process.BeginOutputReadLine();
            while (!process.HasExited)
            {
                string data = null;
                if (messages.TryDequeue(out data))
                    UpdateOutputText(data, tbOutput);
                Thread.Sleep(5);
            }

            process.WaitForExit();

The only downside of this approach is the fact that you can loose messages in a quite rare case, when process starts writing them between process.Start() and process.BeginErrorReadLine() / process.BeginOutputReadLine(), just keep that in mind. The only way to avoid that is to read the full streams and (or) gain access to them only when the process finishes.

Rothwell

Answer 9

first

     // Start the child process.
     Process p = new Process();
     // Redirect the output stream of the child process.
     p.StartInfo.UseShellExecute = false;
     p.StartInfo.RedirectStandardOutput = true;
     p.StartInfo.FileName = "Write500Lines.exe";
     p.Start();
     // Do not wait for the child process to exit before
     // reading to the end of its redirected stream.
     // p.WaitForExit();
     // Read the output stream first and then wait.
     string output = p.StandardOutput.ReadToEnd();
     p.WaitForExit();

second

 // Do not perform a synchronous read to the end of both 
 // redirected streams.
 // string output = p.StandardOutput.ReadToEnd();
 // string error = p.StandardError.ReadToEnd();
 // p.WaitForExit();
 // Use asynchronous read operations on at least one of the streams.
 p.BeginOutputReadLine();
 string error = p.StandardError.ReadToEnd();
 p.WaitForExit();

This is from MSDN

Savoyard