Uri.EscapeDataString() - Invalid URI: The Uri string is too long

Question

I'm using compact framework/C# on windows mobile.

In my application I am uploading data to the server by serializing objects and using a HttpWebRequest/POST request to send the information up. On the server the post data is de-serialised and saved to the db.

The other day I realised that I had a problem with special characters in the post data (ampersands etc..). So I introduced Uri.EscapeDataString() into the method and all was well.

However, today I have discovered that there is a problem when the application attempts to upload a large amount of data (I'm unsure of what exactly denotes "large" at the moment!)

Existing code (Kind of)

var uploadData = new List<Things>();

uploadData.Add(new Thing() { Name = "Test 01" });
uploadData.Add(new Thing() { Name = "Test 02" });
uploadData.Add(new Thing() { Name = "Test with an & Ampersand " }); // Do this a lot!!

var postData = "uploadData=" + Uri.EscapeDataString(JsonConvert.SerializeObject(uploadData, new IsoDateTimeConverter()));

Problem

The call to Uri.EscapeDataString() is causing the following exception:

System.UriFormatException: Invalid URI: The Uri string is too long.

Question

Are there any other ways to prepare the data for upload?

As far as I can see HttpUtility (which has its own Encode/Decode methods) is not available for the compact framework.

Doro

You could write you're own implementation? EscapeDataString() seems mostly convinience... do a normal String.Replace based on a library of characters that need to be escaped? – Corneous

Msdn states : UriFormatException - The length of stringToEscape exceeds 32766 characters. – Peccavi

As Smudge202 suggested, i simply wrote my own implementation. – Doro

How about posting this implementation? – Gesso

I would have posted the implementation but it was a bit scabby!! I've recently changed to use the accepted answer. – Doro

Since .NET Framework 4.5 and .NET Standard 1.0 you should use WebUtility.UrlEncode. See this answer for why. – Differentiation

@CharlieKilian - This was asked a long, long, long time ago, but the main problem was because I was having to use the Compact Framework. If memory serves me well WebUtility wouldn't be available on CF. – Doro

@Doro That's fair. I wasn't leaving this comment to tell you you'd been wrong back then. I was leaving it because this had confused me as I was doing my own research, and once I'd found a better answer these eight years later, I thought I'd help out anyone who comes across it so they could get to the current answer faster. Definitely not a criticism of this question or its answers! In fact, you'd already got my +1. – Differentiation

Answer 1

Or you could simply split your string and call Uri.EscapeDataString(string) for each block, in order to avoid reimplementing the function.

Sample Code:

        String value = "large string to encode";
        int limit = 2000;

        StringBuilder sb = new StringBuilder();
        int loops = value.Length / limit;

        for (int i = 0; i <= loops; i++)
        {
            if (i < loops)
            {
                sb.Append(Uri.EscapeDataString(value.Substring(limit * i, limit)));
            }
            else
            {
                sb.Append(Uri.EscapeDataString(value.Substring(limit * i)));
            }
        }

Comnenus

the limit in .net 4.5 for EscapeDataString is 65520 characters - so that could be used to reduce the iterations needed. – Austere

Cool. Is there this kind of problem with Uri.Unescape? It seems that not, but I wonder just in case – Casefy

@Knagis I am not sure why you mention number of iterations here since that can hardly be a big part of execution time. Initializing StringBuilder with size of value.Length definately sounds like a better performance boost. – Casefy

Just an update: correct current limit of EscapeDataString in .NET 4.5 is 32766 characters (not 65520 as mentioned by @Knagi above): msdn.microsoft.com/en-us/library/… – Prosthodontics

@Prosthodontics if you actually try it, you may find that 65520 is the actual limit (exclusive, so 65519 is the most that will work) despite what the documentation says. – Vue

Answer 2

The answer of "Alberto de Paola" is good.

Nonetheless, to unescape the escaped data is little bit trickier, because you have to avoid cutting the encoded string at the middle of an encoded char (or you will break the integrity of the original string).

Here's my way of fixing this issue :

public static string EncodeString(string str)
{
    //maxLengthAllowed .NET < 4.5 = 32765;
    //maxLengthAllowed .NET >= 4.5 = 65519;
    int maxLengthAllowed = 65519;
    StringBuilder sb = new StringBuilder();
    int loops = str.Length / maxLengthAllowed;

    for (int i = 0; i <= loops; i++)
    {
        sb.Append(Uri.EscapeDataString(i < loops
            ? str.Substring(maxLengthAllowed * i, maxLengthAllowed)
            : str.Substring(maxLengthAllowed * i)));
    }

    return sb.ToString();
}

public static string DecodeString(string encodedString)
{
    //maxLengthAllowed .NET < 4.5 = 32765;
    //maxLengthAllowed .NET >= 4.5 = 65519;
    int maxLengthAllowed = 65519;

    int charsProcessed = 0;
    StringBuilder sb = new StringBuilder();

    while (encodedString.Length > charsProcessed)
    {
        var stringToUnescape = encodedString.Substring(charsProcessed).Length > maxLengthAllowed
            ? encodedString.Substring(charsProcessed, maxLengthAllowed)
            : encodedString.Substring(charsProcessed);

        // If the loop cut an encoded tag (%xx), we cut before the encoded char to not loose the entire char for decoding
        var incorrectStrPos = stringToUnescape.Length == maxLengthAllowed ? stringToUnescape.IndexOf("%", stringToUnescape.Length - 4, StringComparison.InvariantCulture) : -1;
        if (incorrectStrPos > -1)
        {
            stringToUnescape = encodedString.Substring(charsProcessed).Length > incorrectStrPos
                ? encodedString.Substring(charsProcessed, incorrectStrPos)
                : encodedString.Substring(charsProcessed);
        }

        sb.Append(Uri.UnescapeDataString(stringToUnescape));
        charsProcessed += stringToUnescape.Length;
    }

    var decodedString = sb.ToString();

    // ensure the string is sanitized here or throw exception if XSS / SQL Injection is found
    SQLHelper.SecureString(decodedString);
    return decodedString;
}

To test these functions :

var testString = "long string to encode";
var encodedString = EncodeString(testString);
var decodedString = DecodeString(encodedString);

Console.WriteLine(decodedString == testString ? "integrity respected" : "integrity broken");

Hope this can help avoiding some headaches ;)

Wednesday

This builds a better overall solution. I was getting bit by the split in the middle of a character to be translated. – Rab

Answer 3

StringBuilder stringBuilder = new StringBuilder();
for (int i = 0; i < originalString.Length; i++)
{
    if ((originalString[i] >= 'a' && originalString[i] <= 'z') || 
        (originalString[i] >= 'A' && originalString[i] <= 'Z') || 
        (originalString[i] >= '0' && originalString[i] <= '9'))
    {
        stringBuilder.Append(originalString[i]);
    }
    else
    {
        stringBuilder.AppendFormat("%{0:X2}", (int)originalString[i]);
    }
}

string result = stringBuilder.ToString();

Launder

Answer 4

I have been using System.Web.HttpUtility.UrlEncode and seems to handle the longer strings much better.

Nonstop

Answer 5

Use System.Web.HttpUtility.UrlEncode (based on this answer):

        value = HttpUtility.UrlEncode(value)
            .Replace("!", "%21")
            .Replace("(", "%28")
            .Replace(")", "%29")
            .Replace("*", "%2A")
            .Replace("%7E", "~"); // undo escape

Gerah

What about % percent ? – Neiman

The linked answer includes: WebUtility.UrlEncode encodes space as +; Uri.EscapeDataString encodes it as %20. Shouldn't we therefore add .Replace("+", "%20")? – Aciculate

Answer 6

I needed another solution because the solution from Pouki does not work when Cyrillic is processed and symbol is cut.

The alternative solution is as follow:

    protected const int MaxLengthAllowed = 32765;
    private static string UnescapeString(string encodedString)
    {
        var charsProccessed = 0;

        var sb = new StringBuilder();

        while (encodedString.Length > charsProccessed)
        {
            var isLastIteration = encodedString.Substring(charsProccessed).Length < MaxLengthAllowed;

            var stringToUnescape = isLastIteration
                ? encodedString.Substring(charsProccessed)
                : encodedString.Substring(charsProccessed, MaxLengthAllowed);

            while (!Uri.IsWellFormedUriString(stringToUnescape, UriKind.RelativeOrAbsolute) || stringToUnescape.Length == 0)
            {
                stringToUnescape = stringToUnescape.Substring(0, stringToUnescape.Length - 1);
            }

            sb.Append(Uri.UnescapeDataString(stringToUnescape));
            charsProccessed += stringToUnescape.Length;
        }

        return sb.ToString();
    }

Lusatia