idhttp indy post, do request (parameters) with utf-8

Question

I'm trying to post a request, in utf-8 however the server is getting it in Ascii.

Tried the TstringList format of post.

Tried the stream format

Tried forcing the TStringStream to have UTF8 encoding.

Tried updating indy to the xe5 indy

Here is an example code:

var
  server:TIdHttp;
  Parameters,response:TStringStream;
begin
  response := TStringStream.Create;
  Parameters := TStringSTream.create(UTF8String('param1=Value1&param2=عربي/عرب&param3=Value3'),TEncoding.UTF8);
  Server.Post(TIdURI.URLEncode('http://www.example.com/page.php'),Parameters,response);
end;

Now the Arab coding is passed as Ascii in network sniffer.

0060 d8 b9 d8 b1 d8 a8 d9 8a 2f d8 b9 d8 b1 d8 a8 26 ........ /......&

How can I force Indy Http id to pass the request parameters in Utf-8 and not in Ascii?

Agnella

On what did you base the conclusion that it's ascii? The number of bytes and the pattern seems to suggest that it contains UTF8 indeed. It's just the sniffer that ignores the encoding and displays raw bytes (the hex dump) and tries to display them as ascii. – Garbe

@Garbe beacuse unicode-table.com/en/#D8B9 d8b9 is not a arab letter, niehter is little endian b9d8 – Agnella

Take a look at this table: utf8-chartable.de/… There you see that D8B9 in UTF-8 is ARABIC LETTER AIN or ع with unicode code point U+0639. UTF-8 is made to be more compact and better compatible with ASCII and Latin Ansi encoding, but a byte range in UTF-8 doesn't respresent the actual unicode code point. – Garbe

Answer 1

TStringStream in D2009+ uses UnicodeString and is TEncoding-aware so DO NOT create a UTF8String manually:

var
  server: TIdHttp;
  Parameters,response: TStringStream;
begin
  response := TStringStream.Create;
  Parameters := TStringStream.Create('param1=Value1&param2=عربي/عرب&param3=Value3', TEncoding.UTF8);
  Server.Post('http://www.example.com/page.php',Parameters,response);
end;

Alternatively, the TStrings version also encodes to UTF-8 by default:

var
  server: TIdHttp;
  Parameters: TStringList;
  Response: TStringStream;
begin
  response := TStringStream.Create;
  Parameters := TStringList.Create;
  Parameters.Add('param1=Value1');
  Parameters.Add('param2=عربي/عرب');
  Parameters.Add('param3=Value3');
  Server.Post('http://www.example.com/page.php',Parameters,response);
end;

Either way, you should set the request charset before calling Post() so the server knows you are sending UTF-8 encoded data:

Server.Request.ContentType := 'application/x-www-form-urlencoded';
Server.Request.Charset := 'utf-8';

Ishmaelite

TstringList didnt cut it , so i still use TStringStream. and followed the code above it worked. one chagne, TStringStream needs & delim between parameter and value as the question show(no use of new line) – Agnella

I assure you that the TStrings version of Post() encodes UTF-8 just fine (I know because I wrote it), so either you are using an outdated version of Indy, or you are not using it correctly. And my TStringStream example does have & delims between the name=value pairs where they belong. – Ishmaelite

Interestingly, I had same problem with TStringList on Delphi Berlin 10.1 when posting some Baltic or Latin characters, TStringStream worked tho. – Imitative

@Imitative my earlier comment still stands. The TStrings version of Post() encodes the strings to UTF-8 and then url-encodes them. It is specifically designed for application/x-www-form-urlencoded requests, and has worked fine for handling Unicode strings for many years now. I've tested it many times – Ishmaelite