Using WinInet to identify total file size before downloading it

Question

I got the source below from a third-party site explaining how to download a file from the internet using WinInet. I'm not too familiar with API, and I took at look at the WinInet unit but did not see any API calls like what I need.

What I'm doing is adding the ability to report back progress of downloading a file. This procedure I've already wrapped inside of a TThread and everything works fine. However, just one missing piece: Finding the total size of the source file before downloading.

See below where I have a comment //HOW TO GET TOTAL SIZE? This is where I need to find out what is the total size of the file BEFORE I begin downloading it. How do I go about doing this? Because this code seems to not know the size of the file until it's done being downloaded - and that makes this addition irrelevant.

procedure TInetThread.Execute;
const
  BufferSize = 1024;
var
  hSession, hURL: HInternet;
  Buffer: array[1..BufferSize] of Byte;
  BufferLen: DWORD;
  f: File;
  S: Bool;
  D: Integer;
  T: Integer;
  procedure DoWork(const Amt: Integer);
  begin
    if assigned(FOnWork) then
      FOnWork(Self, FSource, FDest, Amt, T);
  end;
begin
  S:= False;
  try
    try
      if not DirectoryExists(ExtractFilePath(FDest)) then begin
        ForceDirectories(ExtractFilePath(FDest));
      end;
      hSession:= InternetOpen(PChar(FAppName), INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
      try
        hURL:= InternetOpenURL(hSession, PChar(FSource), nil, 0, 0, 0);
        try
          AssignFile(f, FDest);
          Rewrite(f, 1);
          T:= 0; //HOW TO GET TOTAL SIZE?
          D:= 0;
          DoWork(D);
          repeat
            InternetReadFile(hURL, @Buffer, SizeOf(Buffer), BufferLen);
            BlockWrite(f, Buffer, BufferLen);
            D:= D + BufferLen;
            DoWork(D);
          until BufferLen = 0;
          CloseFile(f);
          S:= True;
        finally
          InternetCloseHandle(hURL);
        end
      finally
        InternetCloseHandle(hSession);
      end;
    except
      on e: exception do begin
        S:= False;
      end;
    end;
  finally
    if assigned(FOnComplete) then
      FOnComplete(Self, FSource, FDest, S);
  end;
end;

Shagreen

I implemented just such a feature and found that using WinInet causes a dread "timeout bug" to happen in my app. Http-Head requests that normally take 100 mSec, were taking up to 15 seconds to return. It is a known problem in calling WinInet from Delphi on some versions of Windows/WinInet. I'm mentioning this in case you later experience such weird glitches. If you can go Indy here or something non-WinInet (such as WinHttp), please consider it! :-) – Knock

..It is a known problem in calling WinInet from Delphi on some versions of Windows/WinInet @WarrenP I never had this issue using WinInet from delphi. Can you point some documentation or links about this topic? – Eyed

Here's a link: jgobserve.blogspot.com/2009/03/… -- My observations are that the problems are not restricted to the fact that when the underlying network fails, you get long waits. Sometimes everything appears fine EXCEPT winInet which has timeouts that I cannot otherwise explain. Code I wrote in Python, or in Delphi using INDY or ICS does NOT exhibit the same failure pattern. – Knock

I posted this a year and a half ago, and as I read over my code I realize that I did not thread-protect the events with synchronize. All my latest threads for the past year I carefully design the critical sections, but this was back when I didn't know how to make anything thread-safe. – Shagreen

Answer 1

You can use the HEAD method and check the Content-Length to retrieve the file size of a remote file

Check these two Methods

WinInet

If you want execute a HEAD method you must use the HttpOpenRequest, HttpSendRequest and HttpQueryInfo WinInet functions .

uses
 SysUtils,
 Windows,
 WinInet;

function GetWinInetError(ErrorCode:Cardinal): string;
const
   winetdll = 'wininet.dll';
var
  Len: Integer;
  Buffer: PChar;
begin
  Len := FormatMessage(
  FORMAT_MESSAGE_FROM_HMODULE or FORMAT_MESSAGE_FROM_SYSTEM or
  FORMAT_MESSAGE_ALLOCATE_BUFFER or FORMAT_MESSAGE_IGNORE_INSERTS or  FORMAT_MESSAGE_ARGUMENT_ARRAY,
  Pointer(GetModuleHandle(winetdll)), ErrorCode, 0, @Buffer, SizeOf(Buffer), nil);
  try
    while (Len > 0) and {$IFDEF UNICODE}(CharInSet(Buffer[Len - 1], [#0..#32, '.'])) {$ELSE}(Buffer[Len - 1] in [#0..#32, '.']) {$ENDIF} do Dec(Len);
    SetString(Result, Buffer, Len);
  finally
    LocalFree(HLOCAL(Buffer));
  end;
end;


procedure ParseURL(const lpszUrl: string; var Host, Resource: string);
var
  lpszScheme      : array[0..INTERNET_MAX_SCHEME_LENGTH - 1] of Char;
  lpszHostName    : array[0..INTERNET_MAX_HOST_NAME_LENGTH - 1] of Char;
  lpszUserName    : array[0..INTERNET_MAX_USER_NAME_LENGTH - 1] of Char;
  lpszPassword    : array[0..INTERNET_MAX_PASSWORD_LENGTH - 1] of Char;
  lpszUrlPath     : array[0..INTERNET_MAX_PATH_LENGTH - 1] of Char;
  lpszExtraInfo   : array[0..1024 - 1] of Char;
  lpUrlComponents : TURLComponents;
begin
  ZeroMemory(@lpszScheme, SizeOf(lpszScheme));
  ZeroMemory(@lpszHostName, SizeOf(lpszHostName));
  ZeroMemory(@lpszUserName, SizeOf(lpszUserName));
  ZeroMemory(@lpszPassword, SizeOf(lpszPassword));
  ZeroMemory(@lpszUrlPath, SizeOf(lpszUrlPath));
  ZeroMemory(@lpszExtraInfo, SizeOf(lpszExtraInfo));
  ZeroMemory(@lpUrlComponents, SizeOf(TURLComponents));

  lpUrlComponents.dwStructSize      := SizeOf(TURLComponents);
  lpUrlComponents.lpszScheme        := lpszScheme;
  lpUrlComponents.dwSchemeLength    := SizeOf(lpszScheme);
  lpUrlComponents.lpszHostName      := lpszHostName;
  lpUrlComponents.dwHostNameLength  := SizeOf(lpszHostName);
  lpUrlComponents.lpszUserName      := lpszUserName;
  lpUrlComponents.dwUserNameLength  := SizeOf(lpszUserName);
  lpUrlComponents.lpszPassword      := lpszPassword;
  lpUrlComponents.dwPasswordLength  := SizeOf(lpszPassword);
  lpUrlComponents.lpszUrlPath       := lpszUrlPath;
  lpUrlComponents.dwUrlPathLength   := SizeOf(lpszUrlPath);
  lpUrlComponents.lpszExtraInfo     := lpszExtraInfo;
  lpUrlComponents.dwExtraInfoLength := SizeOf(lpszExtraInfo);

  InternetCrackUrl(PChar(lpszUrl), Length(lpszUrl), ICU_DECODE or ICU_ESCAPE, lpUrlComponents);

  Host := lpszHostName;
  Resource := lpszUrlPath;
end;

function GetRemoteFileSize(const Url : string): Integer;
const
  sUserAgent = 'Mozilla/5.001 (windows; U; NT4.0; en-US; rv:1.0) Gecko/25250101';

var
  hInet    : HINTERNET;
  hConnect : HINTERNET;
  hRequest : HINTERNET;
  lpdwBufferLength: DWORD;
  lpdwReserved    : DWORD;
  ServerName: string;
  Resource: string;
  ErrorCode : Cardinal;
begin
  ParseURL(Url,ServerName,Resource);
  Result:=0;

  hInet := InternetOpen(PChar(sUserAgent), INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
  if hInet=nil then
  begin
    ErrorCode:=GetLastError;
    raise Exception.Create(Format('InternetOpen Error %d Description %s',[ErrorCode,GetWinInetError(ErrorCode)]));
  end;

  try
    hConnect := InternetConnect(hInet, PChar(ServerName), INTERNET_DEFAULT_HTTP_PORT, nil, nil, INTERNET_SERVICE_HTTP, 0, 0);
    if hConnect=nil then
    begin
      ErrorCode:=GetLastError;
      raise Exception.Create(Format('InternetConnect Error %d Description %s',[ErrorCode,GetWinInetError(ErrorCode)]));
    end;

    try
      hRequest := HttpOpenRequest(hConnect, PChar('HEAD'), PChar(Resource), nil, nil, nil, 0, 0);
        if hRequest<>nil then
        begin
          try
            lpdwBufferLength:=SizeOf(Result);
            lpdwReserved    :=0;
            if not HttpSendRequest(hRequest, nil, 0, nil, 0) then
            begin
              ErrorCode:=GetLastError;
              raise Exception.Create(Format('HttpOpenRequest Error %d Description %s',[ErrorCode,GetWinInetError(ErrorCode)]));
            end;

             if not HttpQueryInfo(hRequest, HTTP_QUERY_CONTENT_LENGTH or HTTP_QUERY_FLAG_NUMBER, @Result, lpdwBufferLength, lpdwReserved) then
             begin
              Result:=0;
              ErrorCode:=GetLastError;
              raise Exception.Create(Format('HttpQueryInfo Error %d Description %s',[ErrorCode,GetWinInetError(ErrorCode)]));
             end;
          finally
            InternetCloseHandle(hRequest);
          end;
        end
        else
        begin
          ErrorCode:=GetLastError;
          raise Exception.Create(Format('HttpOpenRequest Error %d Description %s',[ErrorCode,GetWinInetError(ErrorCode)]));
        end;
    finally
      InternetCloseHandle(hConnect);
    end;
  finally
    InternetCloseHandle(hInet);
  end;

end;

Indy

Also check this code using indy.

function GetRemoteFilesize(const Url :string) : Integer;
var
  Http: TIdHTTP;
begin
  Http := TIdHTTP.Create(nil);
  try
    Http.Head(Url);
    result:= Http.Response.ContentLength;
  finally
    Http.Free;
  end;
end;

Eyed

+1 Point taken, I should use Indy instead :D Simply for clean code – Shagreen

If you know you're going to download the resource anyway, couldn't you just send a GET request and read the Content-Length header from that instead? It would save you an extra HTTP connection. – Artless

@RobKennedy - yes, you can, as long as the data is not being sent in chunks using the Transfer-Encoding: chunked header, in which case the Content-Length header is not used, and there is no way to know the total size until the last chunk has been received. – Cesura

Answer 2

Answering the question of how to get a download size with WinInet. This is out of one of my file downloaders that is based on WinInet.

This is the method I use to get the download size:

function TWebDownloader.GetContentLength(URLHandle: HINTERNET): Int64;
// returns the expected download size.  Returns -1 if one not provided
   var
     SBuffer: Array[1..20] of char;
     SBufferSize: Integer;
     srv: integer;
   begin
     srv := 0;
    SBufferSize := 20;
    if HttpQueryInfo(URLHandle, HTTP_QUERY_CONTENT_LENGTH, @SBuffer, SBufferSize, srv) then
       Result := StrToFloat(String(SBuffer))
    else
       Result := -1;
   end;

Use of this method requires an open request handle, and does NOT require reading any of the data:

 URLHandle := HttpOpenRequest(ConnectHandle, 'GET', Pchar(sitepath), nil,
                  nil, nil, INTERNET_FLAG_NO_CACHE_WRITE, 0);
 ...
 DownloadSize := GetContentLength(URLHandle);

HTH

Nuriel

+1 Great stuff, and 1/6 the code as the other answer :) btw how is that huge downloader project of yours coming along? – Shagreen

I'm really curious how it works, because conditions are mutually exclusive a) method is GET b) no transfer of requested resource. My guess it closes the connection when headers has been recv'd. – Clydeclydebank

@JerryDodge it's finished enough for my needs and I moved on to other things. It still needs much clean-up work though. – Nuriel

Answer 3

after fixing the types it looks better like this:

function GetContentLength(URLHandle:HINTERNET):Int64;
// returns the expected download size.  Returns -1 if one not provided
var
 SBufferSize, srv:Cardinal;
begin
 srv:=0;
 SBufferSize:=20;
 if Not HttpQueryInfo(URLHandle, HTTP_QUERY_CONTENT_LENGTH or HTTP_QUERY_FLAG_NUMBER, {@SBuffer} @Result, SBufferSize, srv) then Result:=-1;
end;

to call it:

{get the file handle}
hURL:=InternetOpenURL(hSession, PChar(URL), nil, 0, 0, 0);
if hURL=Nil then
begin
 InternetCloseHandle(hSession);
 ShowMessage('The link is incorrect!');
 exit;
end;
{get the file size}
filesize:=GetContentLength(hURL);

Callum