Is TIdHTTPServer Compatible with Microsoft BITS

Question

We are trying to write an update server for our software using the TIdHTTPServer component. Currently we are serving an XML file that lists the available updates and their file versions etc.., when the client program finds a updated version it should start to download it using BITS.

Now this is where we have a problem, our programs are requesting the XML file and seeing there is an update available. It then creates a BITS job to download it, however BITS keeps reporting that the download failed. We can download the file using the same URL and IE/Firefox/Chrome.

so my question:

Is TIdHTTPServer compatible with BITS?

I ask this as I have discovered that there are these download requirements for bits to work.
HTTP Requirements for BITS Downloads

BITS supports HTTP and HTTPS downloads and uploads and requires that the server supports the HTTP/1.1 protocol. For downloads, the HTTP server's Head method must return the file size and its Get method must support the Content-Range and Content-Length headers. As a result, BITS only transfers static file content and generates an error if you try to transfer dynamic content, unless the ASP, ISAPI, or CGI script supports the Content-Range and Content-Length headers.

BITS can use an HTTP/1.0 server as long as it meets the Head and Get method requirements.

To support downloading ranges of a file, the server must support the following requirements:

Allow MIME headers to include the standard Content-Range and Content-Type headers, plus a maximum of 180 bytes of other headers. Allow a maximum of two CR/LFs between the HTTP headers and the first boundary string.

Dmitri

OK having looked at the wireshark traces and Indy source it looks indeed as if the TIdHTTPServer is not supporting the Range / Content-Range headers for Range Retrieval Requests. – Dmitri

So is it possible to read the Range header from the GET request, then read the requested bytes into a memory stream from the source file, assign the memory stream to the content stream for sending and finalyy set the content-range header. – Dmitri

TIdHTTPServer supports the Content-Type, Content-Range, and Range headers, but it is the responsibility of the application's OnCommandGet event handler to actually look at them and send the correct data accordingly. The TIdHTTPHeaderEntityInfo class has ContentType, ContentRangeStart, ContentRangeEnd, ContentRangeInstanceLength, and Range properties available. TIdHTTPServer does not handle the actual data ranging natively, the user's code has to manage it. – Creationism

Answer 1

So the answer to this question is:

Yes TIdHTTPServer is Bits Compatible.

But only if you are prepared to do the work yourself.

As suggested by @Rob Kennedy and Myself it is possible to read the headers and send the data back using the requested ranges, one chunk at a time.

Here is an example of what I am doing in the OnCommandGet event

procedure TForm3.IdHTTPServer1CommandGet(AContext: TIdContext;
  ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
var
  Ranges : TIdEntityRanges;
  DataChunk: TMemoryStream;
  ReqFile: TFileStream;
  ChunkLength: Int64;
  Directory, FileName: string;
begin
  Directory := 'H:';

  case ARequestInfo.Ranges.Count of
  0:
    begin
      //serve file normally
    end;
  1:
    begin
      //serve range of bytes specified for file

      filename := Directory + ARequestInfo.Document;

      if FileExists(FileName) then
      begin
        ReqFile := TFileStream.Create(FileName, fmOpenRead);
        try
          ChunkLength := Succ(ARequestInfo.Ranges.Ranges[0].EndPos - ARequestInfo.Ranges.Ranges[0].StartPos);

          if ChunkLength > ReqFile.Size then
            ChunkLength := ReqFile.Size;

          DataChunk := TMemoryStream.Create;
          DataChunk.Posistion := ARequestInfo.Ranges.Ranges[0].StartPos;  
          DataChunk.CopyFrom(ReqFile, ChunkLength);

          AResponseInfo.ContentStream := DataChunk;
          AResponseInfo.ContentType := IdHTTPServer1.MIMETable.GetFileMIMEType(FileName);
          AResponseInfo.ContentRangeUnits := ARequestInfo.Ranges.Units;
          AResponseInfo.ContentRangeStart := ARequestInfo.Ranges.Ranges[0].StartPos;
          AResponseInfo.ContentRangeEnd := ARequestInfo.Ranges.Ranges[0].StartPos + Pred(ChunkLength);
          AResponseInfo.ContentRangeInstanceLength := ReqFile.Size;
          AResponseInfo.ResponseNo := 206;
        finally
          ReqFile.Free;
        end;
      end
      else
        AResponseInfo.ResponseNo := 404;

    end
  else
    begin
      //serve the file as multipart/byteranges
    end;
  end;

end;

This is by no means finished but it shows the basics of responding to the range requests from BITS. Most importantly it works.

Any comments on the code would be appreciated, constructive criticism always welcome.

Dmitri

Answer 2

Just found a bug in indy that prevents transfer of files over 2.1GB when using range requests.

here it is

IdHTTPHeaderInfo.pas aprox line 770

procedure TIdEntityRange.SetText(const AValue: String);
var
  LValue, S: String;
begin
  LValue := Trim(AValue);
  if LValue <> '' then
  begin
    S := Fetch(LValue, '-'); {do not localize}
    if S <> '' then begin
      FStartPos := StrToIntDef(S, -1);
      FEndPos := StrToIntDef(Fetch(LValue), -1);
      FSuffixLength := -1;
    end else begin
      FStartPos := -1;
      FEndPos := -1;
      FSuffixLength := StrToIntDef(Fetch(LValue), -1);
    end;
  end else begin
    FStartPos := -1;
    FEndPos := -1;
    FSuffixLength := -1;
  end;
end;

This should be

procedure TIdEntityRange.SetText(const AValue: String);
var
  LValue, S: String;
begin
  LValue := Trim(AValue);
  if LValue <> '' then
  begin
    S := Fetch(LValue, '-'); {do not localize}
    if S <> '' then begin
      FStartPos := StrToInt64Def(S, -1);
      FEndPos := StrToInt64Def(Fetch(LValue), -1);
      FSuffixLength := -1;
    end else begin
      FStartPos := -1;
      FEndPos := -1;
      FSuffixLength := StrToInt64Def(Fetch(LValue), -1);
    end;
  end else begin
    FStartPos := -1;
    FEndPos := -1;
    FSuffixLength := -1;
  end;
end;

One for Remy to fix

Dmitri

Answer 3

When you handle the OnCommandGet event, you are given a TIdRequestHeaderInfo, which descends from TIdEntityHeaderInfo; that contains all the headers the request contained, and it even parses out some header values to read as properties, including ContentRangeStart, ContentRangeEnd, and ContentLength.

You can use those properties to populate the stream that you assign to the TIdHTTPResponseInfo.ContentStream property. The entire stream will get sent.

It's your job to differentiate between GET and HEAD requests; OnCommandGet will get triggered either way. Check the IdHTTPRequestInfo.CommandType property.

So, although Indy may not support BITS, it provides all the tools you need to write a program that does support BITS.

Vast

Thanks for the info we are currently looking into this and I will post our results. any help from Remy Lebeau would be appreciated – Dmitri

You where half way there it looks like the ContentRangeStart/End return their defaults of -1. as BITS sends range = bytes=0-1999. To get the Range that BITS sends you need to use the Ranges property on TIdRequestHeaderInfo. This contains a list of the ranges specified in the request header. so in the case of BITS ARequestInfo.Ranges[0].StartPos; and ARequestInfo.Ranges[0].EndPos; will get you the required info you may also want to check the unit to see if it is in bytes in which case you need to check ARequestInfo.Ranges.Units – Dmitri

@MikeT: the ContentRange... properties are RESPONSE properties, not REQUEST properties. You need to use the Range property when receiving a request, and then use the ContentRange... properties when sending a response. – Creationism

@Remy Lebeau - TeamB I understand this now having done my research and read the Indy source code. I was trying to explain to Rob Kennnedy that the ContentRange was for a Response and NOT Requests, therefore the properties mentioned default to -1 in the request header. So you should use Ranges for the Request, and ContentRangeStart/End for Response. – Dmitri