IdHTTP.get returning HTTP1.1/ 403 Forbidden

Question

I'm trying to access an update.txt file on my website, using a DelphiXE compiled program and the IdHTTP component.

The code I'm using is as follows:

procedure TFormAbout.SpeedButtonUpdateClick(Sender: TObject);

function CheckUpdates: String;
var lIdHttp: TIdHTTP;
begin
  lIdHttp := TIdHTTP.Create(nil);
  result := lIdHttp.Get('http://www.test.com/test_down_load/update.txt');
end;

var
sWebVersion: String;
sVersionList: TStringList;

begin
try
  sWebVersion := Checkupdates;
except
  on E: Exception do
  begin 
    ShowMEssage(E.ErrorMessage);
    MessageDlg('An Error occured checking for an update.',mtError,[mbOK],0);
  end;
end;
if sWebVersion <> '' then
  begin
    sVersionList.CommaText := sWebVersion;
    ShowMessage('Version: ' + sVersionList[0] + ' - ' + 'Date: ' + sVersionList[1]);
  end;
end;

This however results in error: HTTP1.1/ 403 Forbidden

The IdHTTP component has been set up with the following properties.

HandleRedirects := true;
HTTPOptions [hoForceEncodeParams];
ProtocolVersion := pv1_1;
Request.UserAgent := Mozilla/5.0 (compatible; Test)

If I enter the URL in an IE browser, it returns the file without errors, but when accessing from my program, I get the error. Any pointers will be appreciated. .htaccess is correct for the site. Permissions for the file are correct on the website: 0644.

Do I have to set any of the other properties for the IdHTTP component. I only have this component on the about form. Do I need anything else.

The updateinfo.txt file simply contains the text in quotes: "18.3.5,2011/12/17"

I have simply used "test" here in place of my actual program name and URL.

Regards Adrian

Allspice

Answer 1

403 means you do not have permission to access the requested URL. The server likely requires you to provide a username/password, especially since you are using an .htaccess file. Use the Request.UserName and Request.Password properties for that. As for why the browser does not ask for username/password, my guess is the browser has them cached from an earlier access.

BTW, your SpeedButtonUpdateClick() has a memory leak. You are creating a new TIdHTTP object but you are not freeing it.

Oldham

Hello Remy, I have no username or password set up for the site. I can access the url from a webbrowser on another PC, that doesn't belong to me. Thx for the memory leak .. just sample code, but you are correct, it needs to be freed. – Allspice

Then use a packet sniffer, such as Wireshark, to see what the HTTP requests look like from your browser and TIdHTTP, and then compare them for differences. Offhand, maybe your server does not like the value of the Request.UserAgent that you are using. Some servers are UserAgent-sensitive. – Oldham

I was worried about the Request.UserAgent too, and tried many variations, without any success. – Allspice

Hello Remy, Actually the problem lies with me. As you can see in the code, I'm creating TIdHTTP, but was not changing its Request.UserAgent, but the IdHttp1 component I had dropped on the form, thus the default Mozilla/3.0 (compatible; Indy Library) was being sent, and not what I thought I was sending all along. So all the changes I was making to the component.properties and testing were never going through. Adding this line of code did it after the create of the lIdHttp: lIdHttp.request.useragent := 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; MAAU)'; Thx Adrian – Allspice

@Remy Lebeau - TeamB, 403 is forbidden, 401 is for credential challenge. – Sola

401 is used to ask for credentials in a particular format specified by the "WWW-Authentication" header(s). 403 is used when the request is rejected, such as if suitable credentials were specified but not valid for the requested resource. – Oldham

When using Indy's Get() function - if you set the UserAgent property, as shown in follow up answer, the request will have the correct credentials and not get a 403 error. – Currin

Changing the Request.UserAgent does not affect how Indy sends credentials in any way, it only affects how some HTTP servers process requests. – Oldham

FYI, I just a few minutes ago checked in an update where the default TIdHTTP.Request.UserAgent is now customizable. There is a new GIdDefaultUserAgent global variable in the IdHTTPHeaderInfo.pas unit. Instead of setting TIdHTTP.Request.UserAgent separately for every TIdHTTP object used, you can now set GIdDefaultUserAgent once at program startup and every TIdHTTP request will use it (you can still use TIdHTTP.Request.UserAgent to override the UserAgent on a per-request basis, if needed). – Oldham

Answer 2

I run into the exact same problem when using Indy's Get() function.

It more than likely that you are getting this error because you haven't set the UserAgent property and the website knowing that you are not accessing the file as a Browser is kicking up a fuss.

function CheckUpdates: String;
var lIdHttp: TIdHTTP;
begin
  lIdHttp := TIdHTTP.Create(nil);
  //avoid getting '403 Forbidden' by setting UserAgent
  lIdHttp.Request.UserAgent :=
      'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0';
  result := lIdHttp.Get('http://www.test.com/test_down_load/update.txt');
end;

Similar question, but correct answer was logged here: https://mcmap.net/q/1633196/-why-do-i-get-quot-403-forbidden-quot-when-i-connect-to-whatismyip-com

Currin

Answer 3

The answers given did not work for me until i put all of them together.

//add Request.Username and supply the correct mysql username
tidHttpObject.Request.Username := 'username';

//do the same for the password
tidHttpObject.Request.Password := 'password';

//then add a UserAgent property with the string below
tidHttpObject.Request.UserAgent :=  'Mozilla/5.0 (Windows NT 6.1; WOW64;
rv:12.0) Gecko/20100101 Firefox/12.0';

//finally call the get() url method of the tidHttp object
Result :=  tidHttpObject.Get(url);

Lubra