Java URL encoding of query string parameters

Question

Say I have a URL

http://example.com/query?q=

and I have a query entered by the user such as:

random word £500 bank $

I want the result to be a properly encoded URL:

http://example.com/query?q=random%20word%20%A3500%20bank%20%24

What's the best way to achieve this? I tried URLEncoder and creating URI/URL objects but none of them come out quite right.

Tafilelt

What do you mean by "none of them come out quite right"? – Eirene

I have used URI.create and replaced spaces with + in querystring. At the client site it converted + back to spaces when I selected the query strings. That has worked for me. – Squamosal

Possible duplicate of Is there a Java method that encodes a collection of parameters as a URL query component? – Armbrecht

Why do you expect $ to be percent-encoded? – Apparently

Answer 1

URLEncoder is the way to go. You only need to keep in mind to encode only the individual query string parameter name and/or value, not the entire URL, for sure not the query string parameter separator character & nor the parameter name-value separator character =.

String q = "random word £500 bank $";
String url = "https://example.com?q=" + URLEncoder.encode(q, StandardCharsets.UTF_8);

When you're still not on Java 10 or newer, then use StandardCharsets.UTF_8.name() as charset argument, or when you're still not on Java 7 or newer, then use "UTF-8".

---

Note that spaces in query parameters are represented by +, not %20, which is legitimately valid. The %20 is usually to be used to represent spaces in URI itself (the part before the URI-query string separator character ?), not in query string (the part after ?).

Also note that there are three encode() methods. One without Charset as second argument and another with String as second argument which throws a checked exception. The one without Charset argument is deprecated. Never use it and always specify the Charset argument. The javadoc even explicitly recommends to use the UTF-8 encoding, as mandated by RFC3986 and W3C.

All other characters are unsafe and are first converted into one or more bytes using some encoding scheme. Then each byte is represented by the 3-character string "%xy", where xy is the two-digit hexadecimal representation of the byte. The recommended encoding scheme to use is UTF-8. However, for compatibility reasons, if an encoding is not specified, then the default encoding of the platform is used.

See also:

• What every web developer must know about URL encoding

Insole

There can be 2 types of parameters in URL. Query string ( followed by ?) and path parameter (Typically part of URL itself). So, what about path parameters. URLEncoder produces + for space even for path parameters. In-fact it just does not handles anything other than query string. Also, this behavior is not in sync with node js servers. So for me this class is a waste and cannot be used other than for very specific / special scenarios. – Overstuff

@sharadendusinha: as documented and answered, URLEncoder is for URL-encoded query parameters conform application/x-www-form-urlencoded rules. Path parameters don't fit in this category. You need an URI encoder instead. – Insole

As I predicted would happen ... users getting confused because obviously the problem is people need to encode more than just the parameter value. Its a very rare case that you only need to encode a parameter value. Its why I provided my "confused" wiki answer to help folks like @sharadendusinha. – Sexpartite

how to fix this using JSP tags ? – Meaty

Why should I not encode the entire url (instead of just query parameters) ? – Handwriting

@WijaySharma: Because URL-specific characters would get encoded as well. You should only do that when you want to pass the entire URL as a query parameter of another URL. – Insole

+1 for providing the link to talisman.org/~erlkonig/misc/… – Fluxmeter

This solution does not work (anymore?), as documented special chars such as - or . are NOT encoded. Thus this is a pretty useless function. – Konstanz

... for the specific task you had in mind, which is clearly not URL-encoding of query string parameters. – Insole

For at least Android 6, "StandardCharsets.UTF_8.toString()" is not the way to go, use "StandardCharsets.UTF_8.name()". Otherwise you get an exception java.nio.charset.IllegalCharsetNameException: java.nio.charset.CharsetICU[UTF-8] – Wesla

Answer 2

I would not use URLEncoder. Besides being incorrectly named (URLEncoder has nothing to do with URLs), inefficient (it uses a StringBuffer instead of Builder and does a couple of other things that are slow) Its also way too easy to screw it up.

Instead I would use URIBuilder or Spring's org.springframework.web.util.UriUtils.encodeQuery or Commons Apache HttpClient. The reason being you have to escape the query parameters name (ie BalusC's answer q) differently than the parameter value.

The only downside to the above (that I found out painfully) is that URL's are not a true subset of URI's.

Sample code:

import org.apache.http.client.utils.URIBuilder;

URIBuilder ub = new URIBuilder("http://example.com/query");
ub.addParameter("q", "random word £500 bank \$");
String url = ub.toString();

// Result: http://example.com/query?q=random+word+%C2%A3500+bank+%24

Sexpartite

Why does it have nothing to do with URLs? – Transubstantiation

@Luis: URLEncoder is as its javadoc says intented to encode query string parameters conform application/x-www-form-urlencoded as described in HTML spec: w3.org/TR/html4/interact/…. Some users indeed confuse/abuse it for encoding whole URIs, like the current answerer apparently did. – Insole

@LuisSep in short URLEncoder is for encoding for form submission. It is not for escaping. Its not the exact same escaping that you would use to create URLs to be put in your web page but happens to be similar enough that people abuse it. The only time you should be using URLEncoder is if your writing a HTTP client (and even then there are far superior options for encoding). – Sexpartite

@Insole "Some users indeed confuse/abuse it for encoding whole URIs, like the current answerer apparently did.". You assumed wrong. I never said I screwed up with it. I have just seen others that have done it, who's bugs I have to fix. The part that I screwed up is that the Java URL class will accept unescaped brackets but not the URI class. There are a lot of way to screw up constructing URLs and not everyone is brilliant like you. I would say that most users that are looking on SO for URLEncoding probably are "users indeed confuse/abuse" URI escaping. – Sexpartite

Question wasn't about that yet your answer implies that. – Insole

Yeah it is. He is concatenating strings to make a URL. URLEncoder is for encoding to the mime type not for making URLs. I have no reputation interest (hence I marked this a wiki post and only want to steer people correctly). – Sexpartite

Not sure why, but I just stumbled on this. URLEncoder is correct (see W3C HTML 4.01 Specification). The URL passed query parameters are to be encoded using the "default content type" of application/x-www-form-urlencoded. Regardless of what the mime type was intended for, the rules of encoding for the mime type, is exactly what is expected for the query parameters. in the OP's question "....?q="+value, the value must be encoded to this mime type. The use or URLEncoder may not be the most efficient, but it used in this case for exactly it's intended purpose. – Scrip

The URL specification is separate from the HTML spec. It's also has changed and just like HTML there is a newer spec. – Sexpartite

Answer 3

You need to first create a URI like:

String urlStr = "http://www.example.com/CEREC® Materials & Accessories/IPS Empress® CAD.pdf"
URL url = new URL(urlStr);
URI uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(), url.getPort(), url.getPath(), url.getQuery(), url.getRef());

Then convert that URI to an ASCII string:

urlStr = uri.toASCIIString();

Now your URL string is completely encoded. First we did simple URL encoding and then we converted it to an ASCII string to make sure no character outside US-ASCII remained in the string. This is exactly how browsers do it.

Bestir

Thanks! It's stupid that your solution works, but built-in URL.toURI() doesn't. – Bijection

Unfortunately this doesn't seem to work with "file:///" (e.g.: "file:///some/directory/a file containing spaces.html"); it bombs with MalformedURLException in "new URL()"; any idea how to fix this? – Floriaflorian

You need to do something like this: String urlStr = "some/directory/a file containing spaces.html"; URL url= new URL(urlStr); URI uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(), url.getPort(), url.getPath(), url.getQuery(), url.getRef()); urlStr=uri.toASCIIString(); urlStr.replace("http://","file:///"); I have not tested it, but I think it will work.... :) – Bestir

I have used this since I started in Android, but today I noticed that the + and the = in the query strings don't get encoded, any solution? – Fasano

where do you want to have + and = sign in url ? can you give an example of such url ? – Bestir

is it also possible to decode it back to the original? – Floozy

@Floozy you can simply use uri.toString() method to convert it to string instead of Ascii string. – Bestir

The API I was working with didn't accept the + replacement for spaces, but accepted the %20 so this solution worked better than BalusC, thanks! – Alderney

This is a correct way to encode the path component of the URL. It is not a correct way to encode query parameter names or values, which is what the question is about. – Deterrent

@user207421it encodes both path components and querry params. Have a look: URI uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(), url.getPort(), url.getPath(), url.getQuery(), url.getRef()); – Bestir

This solution does not work! The query parameter are not encoded, only spaces are. – Konstanz

@Konstanz on the contrary, it double encodes the query containing escaped ampersands and delimiter ampersands – Fixation

Answer 4

Guava 15 has now added a set of straightforward URL escapers.

Intellectualism

These suffer from the same goofy escaping rules as URLEncoder. – Abwatt

not sure they have the problem. they differentiate for instance "+" or "%20" to escape " " (form param or path param) which URLEncoder doesn't. – Intellectualism

This worked for me I just replaced call to URLEncoder() to call to UrlEscapers.urlFragmentEscaper() and it worked, not clear if I should be using UrlEscapers.urlPathSegmentEscaper() instead. – Romney

Actually it didnt work for me because unlike URLEncoder it doesnt encode '+' it leaves it alone, server decodes '+' as space whereas if I use URLEncoder '+'s are converted to %2B and correctly decoded back to + – Romney

Link update: UrlEscapers – Unsling

Answer 5

The code

URL url = new URL("http://example.com/query?q=random word £500 bank $");
URI uri = new URI(url.getProtocol(), url.getUserInfo(), IDN.toASCII(url.getHost()), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
String correctEncodedURL = uri.toASCIIString();
System.out.println(correctEncodedURL);

Prints

http://example.com/query?q=random%20word%20%C2%A3500%20bank%20$

What is happening here?

1. Split URL into structural parts. Use java.net.URL for it.

2. Encode each structural part properly!

3. Use IDN.toASCII(putDomainNameHere) to Punycode encode the hostname!

4. Use java.net.URI.toASCIIString() to percent-encode, NFC encoded Unicode - (better would be NFKC!). For more information, see: How to encode properly this URL

In some cases it is advisable to check if the URL is already encoded. Also replace '+' encoded spaces with '%20' encoded spaces.

Here are some examples that will also work properly

{
      "in" : "http://نامه‌ای.com/",
     "out" : "http://xn--mgba3gch31f.com/"
},{
     "in" : "http://www.example.com/‥/foo",
     "out" : "http://www.example.com/%E2%80%A5/foo"
},{
     "in" : "http://search.barnesandnoble.com/booksearch/first book.pdf",
     "out" : "http://search.barnesandnoble.com/booksearch/first%20book.pdf"
}, {
     "in" : "http://example.com/query?q=random word £500 bank $",
     "out" : "http://example.com/query?q=random%20word%20%C2%A3500%20bank%20$"
}

The solution passes around 100 of the test cases provided by Web Platform Tests.

Apparently

Answer 6

Using Spring's UriComponentsBuilder:

UriComponentsBuilder
        .fromUriString(url)
        .build()
        .encode()
        .toUri()

Passivism

For the spring users, confirming this solution work well !!! – Sammysamoan

Answer 7

The Apache HttpComponents library provides a neat option for building and encoding query parameters.

With HttpComponents 4.x use:

URLEncodedUtils

For HttpClient 3.x use:

EncodingUtil

Fumigate

The link for "URLEncodedUtils" is broken (404). – Antipater

Answer 8

Here's a method you can use in your code to convert a URL string and map of parameters to a valid encoded URL string containing the query parameters.

String addQueryStringToUrlString(String url, final Map<Object, Object> parameters) throws UnsupportedEncodingException {
    if (parameters == null) {
        return url;
    }

    for (Map.Entry<Object, Object> parameter : parameters.entrySet()) {

        final String encodedKey = URLEncoder.encode(parameter.getKey().toString(), "UTF-8");
        final String encodedValue = URLEncoder.encode(parameter.getValue().toString(), "UTF-8");

        if (!url.contains("?")) {
            url += "?" + encodedKey + "=" + encodedValue;
        } else {
            url += "&" + encodedKey + "=" + encodedValue;
        }
    }

    return url;
}

Diner

Answer 9

In Android, I would use this code:

Uri myUI = Uri.parse("http://example.com/query").buildUpon().appendQueryParameter("q", "random word A3500 bank 24").build();

Where Uri is a android.net.Uri

Narcolepsy

This is not using the standard Java API. So please specify library used. – Cyclist

This is the simplest solution to be used in an Android project. – Damalas

Answer 10

In my case I just needed to pass the whole URL and encode only the value of each parameters. I didn't find common code to do that, so (!!) so I created this small method to do the job:

public static String encodeUrl(String url) throws Exception {
    if (url == null || !url.contains("?")) {
        return url;
    }

    List<String> list = new ArrayList<>();
    String rootUrl = url.split("\\?")[0] + "?";
    String paramsUrl = url.replace(rootUrl, "");
    List<String> paramsUrlList = Arrays.asList(paramsUrl.split("&"));
    for (String param : paramsUrlList) {
        if (param.contains("=")) {
            String key = param.split("=")[0];
            String value = param.replace(key + "=", "");
            list.add(key + "=" +  URLEncoder.encode(value, "UTF-8"));
        }
        else {
            list.add(param);
        }
    }

    return rootUrl + StringUtils.join(list, "&");
}

public static String decodeUrl(String url) throws Exception {
    return URLDecoder.decode(url, "UTF-8");
}

It uses Apache Commons' org.apache.commons.lang3.StringUtils.

Polyptych

Answer 11

1. Use this：

    URLEncoder.encode(query, StandardCharsets.UTF_8.displayName());
   

   or this:

    URLEncoder.encode(query, "UTF-8");
   

2. You can use the following code.

    String encodedUrl1 = UriUtils.encodeQuery(query, "UTF-8"); // No change
    String encodedUrl2 = URLEncoder.encode(query, "UTF-8"); // Changed
    String encodedUrl3 = URLEncoder.encode(query, StandardCharsets.UTF_8.displayName()); // Changed
   
    System.out.println("url1 " + encodedUrl1 + "\n" + "url2=" + encodedUrl2 + "\n" + "url3=" + encodedUrl3);
   

Checker

Not correct. You have to encode the parameter names and values separately. Encoding the entire query string will also encode the = and & separators, which is not correct. – Deterrent

The UriUtils.encodeQuery() is missing the management of java.io.UnsupportedEncodingException – Jammiejammin