Downloading a file from the Internet while being able to abort the download any time

Question

I'd like to download a file from my Delphi program in a separate thread dedicated to the download.

The problem is that the main program can be closed any time (thus the download thread can be terminated any time as well). So even if there is no connection or the server lags for precious seconds I need a way to terminate the download thread within a second or two.

What function do you guys recommend to use?

I've experimented with InterOpenURL/InternetReadFile but it doesn't have a time-out parameter. It does have an asynchronous version but I could't find any working Delphi examples and I'm not sure if the asynchronous version will protect me from hangs...

Using sockets have been recommended to me before, but TClientSocket doesn't seem to have a time-out function either.

I need to be fully-prepared so if the user has Internet-connection problems on his/her computer or the webserver keeps lagging my application doesn't hang before closing.

Please keep in mind when answering that I don't want to use neither any third party components nor Indy. Any example code is greatly appreciated.

Thank you!

Tion

The InternetReadFile can be called with an arbitrary amount of bytes to be fetched in a loop. If you combine this with a Terminated check in your thread, flag the thread for termination when closing the application and wait for the thread to terminate, you shouldn't have any "hangs". – Cozart

Answer 1

TWinSocketStream has a timeout. You basically create a blocking socket, and then create a TWinSocketStream using that socket for reading/writing. (sort of like a using a StreamWriter). Then you loop until the connection is closed, the thread is terminated, or you reach the number of expected bytes. There's a WaitForData() which allows you to check for incoming data w/ a timeout, so you're never 'locked' past that timeout value.

You would, though, have to handle the http process yourself. ie, send the 'GET', and then read the response header to determine how many bytes to expect, but thats not too difficult.

Junejuneau

Alas, not personally. I generally use the synapse library for this sort of thing. I did find this online - it looks right, but YMMV. delphi.xcjc.net/viewthread.php?tid=29580 This example uses a socketthread, but the principle would be the same - wait for data, read data, exit if it times out. – Junejuneau

Answer 2

This works.

var
  DownloadAbort: boolean;

procedure Download(const URL, FileName: string);
var
  hInet: HINTERNET;
  hURL: HINTERNET;
  Buffer: array[0..1023] of AnsiChar;
  BufferLen, amt: cardinal;
  f: file;
const
  UserAgent = 'Delphi Application'; // Change to whatever you wish
begin
  DownloadAbort := false;
  hInet := InternetOpen(PChar(UserAgent), INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
  try
    hURL := InternetOpenUrl(hInet, PChar(URL), nil, 0, 0, 0);
    try
      FileMode := fmOpenWrite;
      AssignFile(f, FileName);
      try
        Rewrite(f, 1);
        repeat
          InternetReadFile(hURL, @Buffer, SizeOf(Buffer), BufferLen);
          BlockWrite(f, Buffer[0], BufferLen, amt);
          if DownloadAbort then
            Exit;
        until BufferLen = 0;
      finally
        CloseFile(f);
      end;
    finally
      InternetCloseHandle(hURL);
    end;
  finally
    InternetCloseHandle(hInet);
  end;
end;

In practice, the code above will be part of your thread's Execute method, and you do not need DownloadAbort; intead, you check

if Terminated then
  Exit;

Tamra

Hello! The code above will not work if InternetReadFile hangs. Before posting my question I've browsed for hours and read reports of this happening. – Tion

@Steve: Have you ever observed InternetReadFile hang? Usually, you can trust the Windows API. After all, it is used every day by millions of developers. – Tamra

It's not hard to find info about it, check this one out for example: codeguru.com/forum/archive/index.php/t-355462.html But if you think about it, there are numerous parts where it can go wrong: problem with the user's Internet connection, a non-working proxy, problem with the wifi router, slow response from the Internet server... I think even InternetOpenURL could keep hanging. So I need functions that will not wait for a response, but give back control in an instant and allow cancellation if needed. Any ideas? – Tion

Put the code above in a thread and kill the thread if it does not respond to a normal abort – Recha

Lars, killing the thread is not a very elegant solution and could lead to many problems, including memory leaks... – Tion

@Steve: Memory leaks don't really matter if you are closing the application anyway. The OS will reclaim the memory. – Renal

@Gerry, to be hones I'm a little bit confused on this matter, as some people say memory leaks on program exit are disastrous, while others say exactly what you do. Where can I read more about this? – Tion

@Tion - refer to MS docs, e.g. msdn.microsoft.com/en-us/library/aa366803(v=VS.85).aspx The ExitProcess function in the __except block automatically releases virtual memory allocations – Renal

@Steve: It is at the very heart of a modern operating system not to let a single process damage the system. Memory leaks will only be an issue during runtine of your program; they cannot hurt the operating system when your program is closed. – Tamra

Answer 3

Thanks to GrandmasterB, here is the code I've managed to put together: (it's in my thread's Execute block)

var
  Buffer: array [0..1024 - 1] of Char;
  SocketStream: TWinSocketStream;
  RequestString: String;
  BytesRead: Integer;
begin
  try
    ClientSocket.Active := true;
    DownloadedData := '';
    FillChar(Buffer, SizeOf(Buffer), #0);

    if not Terminated then
    begin
      // Wait 10 seconds
      SocketStream := TWinSocketStream.Create(ClientSocket.Socket, 10000);
      try
        // Send the request to the webserver
        RequestString := 'GET /remotedir/remotefile.html HTTP/1.0'#13#10 +
          'Host: www.someexamplehost.com'#13#10#13#10;
        SocketStream.Write(RequestString[1], Length(RequestString));

        // We keep waiting for the data for 5 seconds
        if (not Terminated) and SocketStream.WaitForData(5000) then
          repeat
            // We store the data in our buffer
            BytesRead := SocketStream.Read(Buffer, SizeOf(Buffer));

            if BytesRead = 0 then
              break
            else
              DownloadedData := DownloadedData + Buffer;
          until Terminated or (not SocketStream.WaitForData(2000));
      finally
        // Close the connection
        SocketStream.Free;
        if ClientSocket.Active then
          ClientSocket.Active := false;
      end;
    end;
  except
  end;

You have to put this to the constructor of the thread:

  ClientSocket := TClientSocket.Create(nil);
  ClientSocket.Host := 'www.someexamplehost.com';
  ClientSocket.Port := 80;
  ClientSocket.ClientType := ctBlocking;

  inherited Create(false);  // CreateSuspended := false;

And this to the desctructor:

  ClientSocket.Free;
  inherited;

Tion

That looks good! The response will have http response headers in them, so the file you receive will actually start after two consecutive #D#A's, which separate the header from the data. You can close the socket as soon as you receive the number of bytes specified in the response header - that will prevent the last 5 seconds from hanging while it waits to see if any more data is coming. – Junejuneau

Answer 4

From delphi.about.com

uses ExtActns, ...

type
   TfrMain = class(TForm)
   ...
   private
     procedure URL_OnDownloadProgress
        (Sender: TDownLoadURL;
         Progress, ProgressMax: Cardinal;
         StatusCode: TURLDownloadStatus;
         StatusText: String; var Cancel: Boolean) ;
   ...

implementation
...

procedure TfrMain.URL_OnDownloadProgress;
begin
   ProgressBar1.Max:= ProgressMax;
   ProgressBar1.Position:= Progress;
end;

function DoDownload;
begin
   with TDownloadURL.Create(self) do
   try
     URL:='http://0.tqn.com/6/g/delphi/b/index.xml';
     FileName := 'c:\ADPHealines.xml';
     OnDownloadProgress := URL_OnDownloadProgress;

     ExecuteTarget(nil) ;
   finally
     Free;
   end;
end;

{ Note: URL property points to Internet FileName is the local file }

Sidra