How to redirect binary gbak output to a Delphi stream?

Question

I want the Firebird backup tool gbak to write its output to a Delphi stream (with no intermediate file). There is a command line parameter to write to stdout rather than a file. I then use the Execute method in JEDI's JclSysUtils to launch gbak and process that output.

It looks like this:

procedure DoBackup;
var
  LBackupAbortFlag: Boolean;
  LBackupStream: TStringStream;
begin
  LBackupAbortFlag := False;
  LBackupStream := TStringStream.Create;
  try
    Execute('"C:\path to\gbak.exe" -b -t -v -user SYSDBA -pas "pw" <db> stdout',
      LBackupStream.WriteString, // Should process stdout (backup)
      SomeMemo.Lines.Append, // Should process stderr (log)
      True, // Backup is "raw"
      False, // Log is not
      @LBackupAbortFlag);
    LBackupStream.SaveToFile('C:\path to\output.fbk');
  finally
    LBackupStream.Free;
  end;
end;

The problem is that the output file is way too small to contain that actual backup. Still I see elements of the file's content. I tried different stream types, but that doesn't seem to make a difference. What could be going wrong here?

Update

To be clear: other solutions are welcome as well. Most of all, I need something reliable. That's why I went with JEDI in the first place, not to reinvent such a thing. Then, it would be nice, if it would be not too complicated.

Baud

Could it simply be that you're writing the output to LBackupStream, but the stream you actually save to disk is BackupStream? Those are two different variables, and thus probably two different stream objects. If you want to save the stream to disk, then why not just use a TFileStream? Or tell gbak to write directly to disk and leave your program out of it? – Extension

@RobKennedy No, that was just a mistake in the example, since I had to simplify it before putting it on the internet. I want to process the data as in compress, encrypt, et cetera on the fly before sending it elsewhere. – Baud

What does the stream contain, and how does it compare to what you get when you skip the stream and have gbak write the backup directly to disk? If you put a breakpoint in WriteString, do you see all the data you expect to see? – Extension

@RobKennedy Not sure how to describe this. It's for example only 136566 out of 10127872 bytes. The first part of the file (looks like column definitions) has gaps, then the actual data is missing and then again gaps (more definitions). – Baud

I can answer the question in the body, but I cannot easily answer the one in the title. Which one did you intend to get an answer for? – Extension

@RobKennedy Well, I'm curious why the above doesn't work, but other solutions are acceptable if they achieve the same thing in an equally straightforward way. I just found PJConsoleApp myself, which seems to work, but still needs to gain my trust - it's really sensitive for the TimeSlice value and once I got the impression it skipped the very last bytes. – Baud

@ThijsvanDien: Are you sure the stream method to write a string is the proper one to write binary data (as you say in the question title)? – Scalariform

Answer 1

My first answer is effective when you wish to merge stdout and stderr. However, if you need to keep these separate, that approach is no use. And I can now see, from a closer reading of your question, and your comments, that you do wish to keep the two output streams separate.

Now, it is not completely straightforward to extend my first answer to cover this. The problem is that the code there uses blocking I/O. And if you need to service two pipes, there is an obvious conflict. A commonly used solution in Windows is asynchronous I/O, known in the Windows world as overlapped I/O. However, asynchronous I/O is much more complex to implement than blocking I/O.

So, I'm going to propose an alternative approach that still uses blocking I/O. If we want to service multiple pipes, and we want to use blocking I/O then the obvious conclusion is that we need one thread for each pipe. This is easy to implement – much easier than the asynchronous option. We can use almost identical code but move the blocking read loops into threads. My example, re-worked in this way, now looks like this:

{$APPTYPE CONSOLE}

uses
  SysUtils, Classes, Windows;

type
  TProcessOutputPipe = class
  private
    Frd: THandle;
    Fwr: THandle;
  public
    constructor Create;
    destructor Destroy; override;
    property rd: THandle read Frd;
    property wr: THandle read Fwr;
    procedure CloseWritePipe;
  end;

constructor TProcessOutputPipe.Create;
const
  PipeSecurityAttributes: TSecurityAttributes = (
    nLength: SizeOf(TSecurityAttributes);
    bInheritHandle: True
  );
begin
  inherited;
  Win32Check(CreatePipe(Frd, Fwr, @PipeSecurityAttributes, 0));
  Win32Check(SetHandleInformation(Frd, HANDLE_FLAG_INHERIT, 0));//don't inherit read handle of pipe
end;

destructor TProcessOutputPipe.Destroy;
begin
  CloseHandle(Frd);
  if Fwr<>0 then
    CloseHandle(Fwr);
  inherited;
end;

procedure TProcessOutputPipe.CloseWritePipe;
begin
  CloseHandle(Fwr);
  Fwr := 0;
end;

type
  TReadPipeThread = class(TThread)
  private
    FPipeHandle: THandle;
    FStream: TStream;
  protected
    procedure Execute; override;
  public
    constructor Create(PipeHandle: THandle; Stream: TStream);
  end;

constructor TReadPipeThread.Create(PipeHandle: THandle; Stream: TStream);
begin
  inherited Create(False);
  FPipeHandle := PipeHandle;
  FStream := Stream;
end;

procedure TReadPipeThread.Execute;
var
  Buffer: array [0..4096-1] of Byte;
  BytesRead: DWORD;
begin
  while ReadFile(FPipeHandle, Buffer, SizeOf(Buffer), BytesRead, nil) and (BytesRead<>0) do begin
    FStream.WriteBuffer(Buffer, BytesRead);
  end;
end;

function ReadOutputFromExternalProcess(const ApplicationName, CommandLine: string; stdout, stderr: TStream): DWORD;
var
  stdoutPipe, stderrPipe: TProcessOutputPipe;
  stdoutThread, stderrThread: TReadPipeThread;
  StartupInfo: TStartupInfo;
  ProcessInfo: TProcessInformation;
  lpApplicationName: PChar;
  ModfiableCommandLine: string;
begin
  if ApplicationName='' then
    lpApplicationName := nil
  else
    lpApplicationName := PChar(ApplicationName);
  ModfiableCommandLine := CommandLine;
  UniqueString(ModfiableCommandLine);

  stdoutPipe := nil;
  stderrPipe := nil;
  stdoutThread := nil;
  stderrThread := nil;
  try
    stdoutPipe := TProcessOutputPipe.Create;
    stderrPipe := TProcessOutputPipe.Create;

    ZeroMemory(@StartupInfo, SizeOf(StartupInfo));
    StartupInfo.cb := SizeOf(StartupInfo);
    StartupInfo.dwFlags := STARTF_USESHOWWINDOW or STARTF_USESTDHANDLES;
    StartupInfo.wShowWindow := SW_HIDE;
    StartupInfo.hStdOutput := stdoutPipe.wr;
    StartupInfo.hStdError := stderrPipe.wr;
    Win32Check(CreateProcess(lpApplicationName, PChar(ModfiableCommandLine), nil, nil, True,
      CREATE_NO_WINDOW or NORMAL_PRIORITY_CLASS, nil, nil, StartupInfo, ProcessInfo));

    stdoutPipe.CloseWritePipe;//so that the process is able to terminate
    stderrPipe.CloseWritePipe;//so that the process is able to terminate

    stdoutThread := TReadPipeThread.Create(stdoutPipe.rd, stdout);
    stderrThread := TReadPipeThread.Create(stderrPipe.rd, stderr);
    stdoutThread.WaitFor;
    stderrThread.WaitFor;

    Win32Check(WaitForSingleObject(ProcessInfo.hProcess, INFINITE)=WAIT_OBJECT_0);
    Win32Check(GetExitCodeProcess(ProcessInfo.hProcess, Result));
  finally
    stderrThread.Free;
    stdoutThread.Free;
    stderrPipe.Free;
    stdoutPipe.Free;
  end;
end;

procedure Test;
var
  stdout, stderr: TFileStream;
  ExitCode: DWORD;
begin
  stdout := TFileStream.Create('C:\Desktop\stdout.txt', fmCreate);
  try
    stderr := TFileStream.Create('C:\Desktop\stderr.txt', fmCreate);
    try
      ExitCode := ReadOutputFromExternalProcess('', 'cmd /c dir /s C:\Windows\system32', stdout, stderr);
    finally
      stderr.Free;
    end;
  finally
    stdout.Free;
  end;
end;

begin
  Test;
end.

If you wish to add support for cancelling, then you would simply add in a call to TerminateProcess when the user cancelled. This would bring everything to a halt, and the function would return the exit code that you supplied to TerminateProcess. I'm hesitant right now to suggest a cancellation framework for you, but I think that the code in this answer is now pretty close to meeting your requirements.

Peers

Excellent! It works very well. Two minor issues with the code: inherited Create in the constructor of TReadPipeThread is missing a value for CreateSuspended - I bet that should be False - and Code Insight complains about the somewhat recursive definition of PipeSecurityAttributes - should it be fine if I turn it into SizeOf(TSecurityAttributes)? I'll accept the answer once I implemented the termination thing. – Baud

Code fine in my version of Delphi. I guess yours is older. The constructor of overload I call sets CreateSuspended to False. Obvious really otherwise the thread would never start. Find to change security attr as you describe. – Peers

Currently I'm reworking your example to allow for input (to restore the backup), so I'm trying to develop a better understanding of what's going on. Few more questions about your solution: in many comparable snippets, PeekNamedPipe is used. Was that a conscious decision? May things break if I add it? And what is this HANDLE_FLAG_INHERIT about? MSDN isn't very helpful by saying it defines whether the child process may inherit the handle if I don't know what that means/results in. – Baud

If the external process inherits the pipe read handle, then the pipe never closes because the external process has a handle to it which it never closes. PeekNamedPipe would be used in an ansync solution. Because we used blocking I/O, we don't have any need for it. – Peers

Coming back to this (yup, it's been a while)... It is my intuition that the easiest yet acceptable way to force this function to return (canceling) would be to kill the process it launched from another thread. Otherwise I'd have to change all the Waits into WaitForMultipleObject and use SetEvent or something? If you dislike this proposal, I'm going to open another question just about that... – Baud

Your only option is to kill the process. No need for another question! – Peers

Well, it works like this. :) Just a little pain to make really sure the external process is terminated under all circumstances. I was wondering why we need the stdoutThread.WaitFor; stderrThread.WaitFor; at all if they're followed by a WaitForSingleObject on the process. If I can leave the WaitFor out, that would reduce termination logic quite a bit. – Baud

If you leave them out then you deadlock IIRC – Peers

Answer 2

I expect that your code is failing because it tries to put binary data through a text oriented stream. In any case, it's simple enough to solve your problem with a couple of Win32 API calls. I don't see any compelling reason to use third party components for just this task.

Here's what you need to do:

1. Create a pipe that you will use as a communication channel between the two processes.
2. Create the gbak process and arrange for its stdout to be the write end of the pipe.
3. Read from the read end of the pipe.

Here's a simple demonstration program:

{$APPTYPE CONSOLE}

uses
  SysUtils, Classes, Windows;

procedure ReadOutputFromExternalProcess(const ApplicationName, CommandLine: string; Stream: TStream);
const
  PipeSecurityAttributes: TSecurityAttributes = (
    nLength: SizeOf(PipeSecurityAttributes);
    bInheritHandle: True
  );
var
  hstdoutr, hstdoutw: THandle;
  StartupInfo: TStartupInfo;
  ProcessInfo: TProcessInformation;
  lpApplicationName: PChar;
  ModfiableCommandLine: string;
  Buffer: array [0..4096-1] of Byte;
  BytesRead: DWORD;
begin
  if ApplicationName='' then begin
    lpApplicationName := nil;
  end else begin
    lpApplicationName := PChar(ApplicationName);
  end;

  ModfiableCommandLine := CommandLine;
  UniqueString(ModfiableCommandLine);

  Win32Check(CreatePipe(hstdoutr, hstdoutw, @PipeSecurityAttributes, 0));
  Try
    Win32Check(SetHandleInformation(hstdoutr, HANDLE_FLAG_INHERIT, 0));//don't inherit read handle of pipe
    ZeroMemory(@StartupInfo, SizeOf(StartupInfo));
    StartupInfo.cb := SizeOf(StartupInfo);
    StartupInfo.dwFlags := STARTF_USESHOWWINDOW or STARTF_USESTDHANDLES;
    StartupInfo.wShowWindow := SW_HIDE;
    StartupInfo.hStdOutput := hstdoutw;
    StartupInfo.hStdError := hstdoutw;
    if not CreateProcess(
      lpApplicationName,
      PChar(ModfiableCommandLine),
      nil,
      nil,
      True,
      CREATE_NO_WINDOW or NORMAL_PRIORITY_CLASS,
      nil,
      nil,
      StartupInfo,
      ProcessInfo
    ) then begin
      RaiseLastOSError;
    end;
    CloseHandle(ProcessInfo.hProcess);
    CloseHandle(ProcessInfo.hThread);
    CloseHandle(hstdoutw);//close the write end of the pipe so that the process is able to terminate
    hstdoutw := 0;
    while ReadFile(hstdoutr, Buffer, SizeOf(Buffer), BytesRead, nil) and (BytesRead<>0) do begin
      Stream.WriteBuffer(Buffer, BytesRead);
    end;
  Finally
    CloseHandle(hstdoutr);
    if hstdoutw<>0 then begin
      CloseHandle(hstdoutw);
    end;
  End;
end;

procedure Test;
var
  Stream: TFileStream;
begin
  Stream := TFileStream.Create('C:\Desktop\out.txt', fmCreate);
  Try
    ReadOutputFromExternalProcess('', 'cmd /c dir /s C:\Windows\system32', Stream);
  Finally
    Stream.Free;
  End;
end;

begin
  Test;
end.

Peers

What I like about your solution is that it's fast compared to PJConsoleApp. Unfortunately I'm missing some things like the ability to timeout or abort manually (making sure the process is terminated), separate output for stderr and the error code. I'm reluctant to try fit this in myself - I looked for different libraries because I'm not very familiar with WinAPI and it seems easy to get a detail wrong. Is there some limited set of resources you can recommend to help understand what's going on above? – Baud

Error code you obtain by calling GetExitCodeProcess. Separate stdout/stderr just involves creating two pipes instead of one. For timeout/abort, you could pop this code in a thread and abort with TerminateProcess. If you don't want to move to a different thread, and want to do it by checking a flag, as you do in the code in the Q, perform that check in the while loop. – Peers

I tried to modify it was you suggested. Wasn't sure what the loop should look like. Code above. Something is wrong, given that it hangs. – Baud

Sorry, I haven't got the time to debug this now. – Peers

What happens if you add in a call to GetFileSize(hstderrr) before you attempt to read from the stderr pipe? Otherwise what I guess happens is that the ReadFile for stderr blocks because nothing is written to that pipe. And then you can never get back to empty the stdout pipe which is needed in order for everything to close down. – Peers

Doesn't change anything; tried it for hstdoutr too. Should I rewrite this thing to something like while ProcessRunning do begin if PeekOut then WriteOut; if PeekErr then WriteErr; end? – Baud

What I suggested works here. But it's a rather lame approach. The right way to do this is to use async I/O. But it's fearsomely complex. Have a look at the JEDI code. The JEDI code assumes you want text which is obviously the problem. Would you be interested in a simple solution based on threads? One thread per pipe. – Peers

The means don't really matter as long as it gives me everything mentioned, is reliable and doesn't require reinvention. JPConsoleApp comes rather close to that. I'm just uncomfortable with the fact that it's so sensitive for the TimeSlice value and I once got the impression it missed the last chunk(s) of data. I don't want to feel lucky that it works. Your first solution seems rather solid, except for that I can't check if the output is all fine, because that's written to stderr. I could then redirect that to a file (there's a parameter for that) but I was trying to avoid files altogether. – Baud