How do you UrlEncode without using System.Web?

Question

I am trying to write a windows client application that calls a web site for data. To keep the install to a minimum I am trying only use dlls in the .NET Framework Client Profile. Trouble is that I need to UrlEncode some parameters, is there an easy way to do this without importing System.Web.dll which is not part of the Client Pofile?

Subtitle

Could you show how are you performing the call to the web site? Maybe there's something that can be done there. – Willette

Out of curiosity, how do you call a website for data without using System.Web? – Dehorn

@Patrick, he is probably using WebRequest or WebClient. That's the reason I asked about this particular code because there are things that can be done about properly url encoding data. – Willette

I'm using a System.Net.WebRequest object. Then I call GetRequestStream and write my Post parameters out to the stream. I also set the ContentType to "application/x-www-form-urlencoded". – Subtitle

Of course this would equally well apply if I was performing a GET request and appending the parameters to the URL. – Subtitle

Answer 1

System.Uri.EscapeUriString() can be problematic with certain characters, for me it was a number / pound '#' sign in the string.

If that is an issue for you, try:

System.Uri.EscapeDataString() //Works excellent with individual values

Here is a SO question answer that explains the difference:

What's the difference between EscapeUriString and EscapeDataString?

and recommends to use Uri.EscapeDataString() in any aspect.

Deciliter

False: blogs.msdn.com/b/yangxind/archive/2006/11/09/… You'll have problems with plus signs as they won't be unencoded. – Foursome

That blog post is a bit old and I just have "Uri Escaped" a full url and all spaces has become %20, so I think they fixed it. I am using .Net 4.5. – Punkah

EscapeDataString also does not support very long strings if you are preparing data for a POST operation. #6695708 – Eada

Uri.EscapeUriString is indeed very problematic and should not be used, as it tries to do something (escaping full URIs) that is actually impossible to do consistently See this answer for a detailed explanation. – Ellen

Also a ' ' Space character. – Ceciliacecilio

Answer 2

In .Net 4.5+ use WebUtility

Just for formatting I'm submitting this as an answer.

Couldn't find any good examples comparing them so:

string testString = "http://test# space 123/text?var=val&another=two";
Console.WriteLine("UrlEncode:         " + System.Web.HttpUtility.UrlEncode(testString));
Console.WriteLine("EscapeUriString:   " + Uri.EscapeUriString(testString));
Console.WriteLine("EscapeDataString:  " + Uri.EscapeDataString(testString));
Console.WriteLine("EscapeDataReplace: " + Uri.EscapeDataString(testString).Replace("%20", "+"));

Console.WriteLine("HtmlEncode:        " + System.Web.HttpUtility.HtmlEncode(testString));
Console.WriteLine("UrlPathEncode:     " + System.Web.HttpUtility.UrlPathEncode(testString));

//.Net 4.0+
Console.WriteLine("WebUtility.HtmlEncode: " + WebUtility.HtmlEncode(testString));
//.Net 4.5+
Console.WriteLine("WebUtility.UrlEncode:  " + WebUtility.UrlEncode(testString));

Outputs:

UrlEncode:             http%3a%2f%2ftest%23+space+123%2ftext%3fvar%3dval%26another%3dtwo
EscapeUriString:       http://test#%20space%20123/text?var=val&another=two
EscapeDataString:      http%3A%2F%2Ftest%23%20space%20123%2Ftext%3Fvar%3Dval%26another%3Dtwo
EscapeDataReplace:     http%3A%2F%2Ftest%23+space+123%2Ftext%3Fvar%3Dval%26another%3Dtwo

HtmlEncode:            http://test# space 123/text?var=val&another=two
UrlPathEncode:         http://test#%20space%20123/text?var=val&another=two

//.Net 4.0+
WebUtility.HtmlEncode: http://test# space 123/text?var=val&another=two
//.Net 4.5+
WebUtility.UrlEncode:  http%3A%2F%2Ftest%23+space+123%2Ftext%3Fvar%3Dval%26another%3Dtwo

In .Net 4.5+ use WebUtility.UrlEncode

This appears to replicate HttpUtility.UrlEncode (pre-v4.0) for the more common characters:
Uri.EscapeDataString(testString).Replace("%20", "+").Replace("'", "%27").Replace("~", "%7E")
Note: EscapeUriString will keep a valid uri string, which causes it to use as many plaintext characters as possible.

See this answer for a Table Comparing the various Encodings:
https://mcmap.net/q/13942/-url-encoding-using-c

Line Breaks All of them listed here (other than HttpUtility.HtmlEncode) will convert "\n\r" into %0a%0d or %0A%0D

Please feel free to edit this and add new characters to my test string, or leave them in the comments and I'll edit it.

Xenophobia

In my case I had to use EscapeDataString rather than EscapeUriString as we were encoding carriage returns and line feeds and these required the more aggressive escaping performed by EscapeDataString – Abhorrence

more examples, you can provide your own test cases if you want. Here's a sample of running it and the other encoding methods that shows differences dotnetfiddle.net/12IFw1 – Stupendous

WebUtility.UrlEncode() and WebUtility.UrlDecode() are 4.5+. They don't exist in 4.0. – Pincenez

The msdn says: "Universal Windows Platform: Available since 4.5, .NET Framework: Available since 4.0"... – Xenophobia

Answer 3

You can use

Uri.EscapeUriString (see http://msdn.microsoft.com/en-us/library/system.uri.escapeuristring.aspx)

Seaworthy

Is there a difference between this and EscapeDataString? – Subtitle

You want to use EscapeUriString. The EscapeUriString will try to encode the whole url (include http:// part) while EscapeUriString understands what parts actually should be encoded – Seaworthy

I see, so in this instance I would probably want EscapeDataString as I may want to pass a URL as a get parameter. I am appending to a URL in this instance. – Subtitle

@MatthewManela I'm pretty sure your Oct1 comment should read The EscapeDataString will try to encode ... – Stupendous

Don't use Uri.EscapeUriString. It does not "understand" what parts should be encoded, it's just a misguided attempt at doing something (escaping full URIs) which is actually impossible to do consistently. See this answer for a detailed explanation. – Ellen

Answer 4

The answers here are very good, but still insufficient for me.

I wrote a small loop that compares Uri.EscapeUriString with Uri.EscapeDataString for all characters from 0 to 255.

NOTE: Both functions have the built-in intelligence that characters above 0x80 are first UTF-8 encoded and then percent encoded.

Here is the result:

******* Different *******

'#' -> Uri "#" Data "%23"
'$' -> Uri "$" Data "%24"
'&' -> Uri "&" Data "%26"
'+' -> Uri "+" Data "%2B"
',' -> Uri "," Data "%2C"
'/' -> Uri "/" Data "%2F"
':' -> Uri ":" Data "%3A"
';' -> Uri ";" Data "%3B"
'=' -> Uri "=" Data "%3D"
'?' -> Uri "?" Data "%3F"
'@' -> Uri "@" Data "%40"


******* Not escaped *******

'!' -> Uri "!" Data "!"
''' -> Uri "'" Data "'"
'(' -> Uri "(" Data "("
')' -> Uri ")" Data ")"
'*' -> Uri "*" Data "*"
'-' -> Uri "-" Data "-"
'.' -> Uri "." Data "."
'_' -> Uri "_" Data "_"
'~' -> Uri "~" Data "~"

'0' -> Uri "0" Data "0"
.....
'9' -> Uri "9" Data "9"

'A' -> Uri "A" Data "A"
......
'Z' -> Uri "Z" Data "Z"

'a' -> Uri "a" Data "a"
.....
'z' -> Uri "z" Data "z"

******* UTF 8 *******

.....
'Ò' -> Uri "%C3%92" Data "%C3%92"
'Ó' -> Uri "%C3%93" Data "%C3%93"
'Ô' -> Uri "%C3%94" Data "%C3%94"
'Õ' -> Uri "%C3%95" Data "%C3%95"
'Ö' -> Uri "%C3%96" Data "%C3%96"
.....

EscapeUriString is to be used to encode URLs, while EscapeDataString is to be used to encode for example the content of a Cookie, because Cookie data must not contain the reserved characters '=' and ';'.

Inman

nice analysis and breakdown here, very helpful. if anyone has or knows of performance benchmarks (comparing all three methods) that would also be nice to see – Veroniqueverras

This is a good analysis and the takeaway is that you shouldn't use Uri.EscapeUriString, because escaping full URIs is impossible to do consistently. See this answer for a detailed explanation. – Ellen

Answer 5

There's a client profile usable version, System.Net.WebUtility class, present in client profile System.dll. Here's the MSDN Link:

WebUtility

Selfimprovement

I'd note that the help page for that class specifically says "Provides methods for encoding and decoding URLs when processing Web requests." so it could just be that they didn't name the methods well. – Constitutionalism

Good point, say, why don't you vote a brother up ;) this down vote has been haunting me for 2 years! JK... but honestly that's probably why I posted the link, unfortunate that I take a reputation hit for errors in Microsoft's docs... – Selfimprovement

It looks like UrlEncode and UrlDecode were only added to WebUtility in in the 4.5 version of .Net. – Subtitle

Answer 6

Here's an example of sending a POST request that properly encodes parameters using application/x-www-form-urlencoded content type:

using (var client = new WebClient())
{
    var values = new NameValueCollection
    {
        { "param1", "value1" },
        { "param2", "value2" },
    };
    var result = client.UploadValues("http://foo.com", values);
}

Willette

Answer 7

To UrlEncode without using System.Web:

String s = System.Net.WebUtility.UrlEncode(str);
//fix some different between WebUtility.UrlEncode and HttpUtility.UrlEncode
s = Regex.Replace(s, "(%[0-9A-F]{2})", c => c.Value.ToLowerInvariant());

more details: https://www.samnoble.co.uk/2014/05/21/beware-webutility-urlencode-vs-httputility-urlencode/

Prent

Answer 8

I've been forced to use .NET 4.0 for some of the projects I've built, and due to this, neither WebUtility nor HttpUtility will contain these. I used the Uri.EscapeDataString() method which works really well, but I did not like the fact it didn't encode all standard special characters (meaning ! "#$%&'()*+,-./:;<=>?@[\]^_`{|}~) in one go. I also deal more with Visual Basic than C#, so I do not know for sure what it would take to convert the following, but it works fantastic for my basic needs.

I won't be dealing with any UTF-8 formatted strings with this as it's only used for some very basic text manipulation and has served me well so far. It will not parse out line breaks in any fashion (the text I'm manipulating won't have them), and you have to process the % sign first to prevent it from ruining the encoding of the rest of the symbols. A bit garish, but it works.

Function EncodeURL(ByVal DecodedString As String) As String

  DecodedString = Replace(DecodedString, "%", "%25", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, " ", "%20", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "!", "%21", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, """", "%22", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "#", "%23", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "$", "%24", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "&", "%26", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "'", "%27", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "(", "%28", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, ")", "%29", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "*", "%2A", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "+", "%2B", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, ",", "%2C", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "-", "%2D", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, ".", "%2E", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "/", "%2F", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, ":", "%3A", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, ";", "%3B", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "<", "%3C", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "=", "%3D", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, ">", "%3E", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "?", "%3F", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "@", "%40", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "[", "%5B", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "\", "%5C", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "]", "%5D", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "^", "%5E", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "_", "%5F", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "`", "%60", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "{", "%7B", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "|", "%7C", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "}", "%7D", 1, vbTextCompare)
  DecodedString = Replace(DecodedString, "~", "%7E", 1, vbTextCompare)

  EncodeURL = DecodedString

End Function

Input:

! "#$%&'()*+,-./:;<=>?@[]^_`{|}~

Output

%21%20%22%23%24%25%26%27%28%29%2A%2B%2C%2D%2E%2F%3A%3B%3C%3D%3E%3F%40%5B%5C%5D%5E%5F%60%7B%7C%7D%7E

Maureenmaureene

You might make this code more manageable and performant by using a Dictionary<char, string> to define the mappings from each character-to-be-replaced to the replacement text (e.g. '!' → "21"), then enumerate DecodedString and Append() to a StringBuilder the replacement text (if the input char is found in the Dictionary<,>), otherwise the input char. Alternatively, knowing that characters are encoded as their hexadecimal ASCII value, you could use a HashSet<char>, a switch, or simple ranges (e.g. 0x23 ≤ c ≤ 0x2C, etc.) to define chars to replace, then emit (byte) c. – Windbreak

OH! I love this Idea! This is very similar to processing huge ranges of cells in Excel - putting it into an Array allows you to read data MUCH faster. – Maureenmaureene

Also - I had to check out how you edited the post to allow all the symbols in the code style text and the proper vb.net language formatting. Thanks a heap for this too - my posts in the future will be better formatted! 😁 – Maureenmaureene

Very good. If you mean how I was able to include a backtick within inline code, the sections at the end of this answer to "How do I format my code blocks?" document the ways to preserve backticks. I used the lang-vbnet language hint to specify VB.NET, but there are aliases you can use (e.g. lang-vb) as well as just the name of a tag (e.g. vb.net) if it has an associated language code. See this answer to "What is syntax highlighting and how does it work?" – Windbreak

Answer 9

System.Net.WebUtility.HtmlDecode

Galiot

WebUtility class provides methods for encoding and decoding URLs when processing Web requests. It does the same thing as HttpUtility but it's out of System.Web namespace – Fertile

It's wrong because it HtmlDecodes and does not UrlEncode like the question asked. Even HtmlEncode would be wrong as HTML encoding is different to URL encoding. – Subtitle