'+' (plus sign) not encoded with RestTemplate using String url, but interpreted as ' ' (space)

Question

We are moving from Java 8 to Java 11, and thus, from Spring Boot 1.5.6 to 2.1.2. We noticed, that when using RestTemplate, the '+' sign is not encoded to '%2B' anymore (changes by SPR-14828). This would be okay, because RFC3986 doesn't list '+' as a reserved character, but it is still interpreted as a ' ' (space) when received in a Spring Boot endpoint.

We have a search query which can take optional timestamps as query parameters. The query looks something like http://example.com/search?beforeTimestamp=2019-01-21T14:56:50%2B00:00.

We can't figure out how to send an encoded plus sign, without it being double-encoded. Query parameter 2019-01-21T14:56:50+00:00 would be interpreted as 2019-01-21T14:56:50 00:00. If we were to encode the parameter ourselves (2019-01-21T14:56:50%2B00:00), then it would be received and interpreted as 2019-01-21T14:56:50%252B00:00.

An additional constraint is, that we want to set the base url elsewhere, when setting up the restTemplate, not where the query is being executed.

Alternatively, is there a way to force '+' not to be interpreted as ' ' by the endpoint?

I have written a short example demonstrating some ways of achieving stricter encoding with their drawbacks explained as comments:

package com.example.clientandserver;

import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.util.DefaultUriBuilderFactory;
import org.springframework.web.util.UriComponentsBuilder;
import org.springframework.web.util.UriUtils;

import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;

@SpringBootApplication
@RestController
public class ClientAndServerApp implements CommandLineRunner {

    public static void main(String[] args) {
        SpringApplication.run(ClientAndServerApp.class, args);
    }

    @Override
    public void run(String... args) {
        String beforeTimestamp = "2019-01-21T14:56:50+00:00";

        // Previously - base url and raw params (encoded automatically). 
        // This worked in the earlier version of Spring Boot
        {
            RestTemplate restTemplate = new RestTemplateBuilder()
               .rootUri("http://localhost:8080").build();
            UriComponentsBuilder b = UriComponentsBuilder.fromPath("/search");
            if (beforeTimestamp != null) {
                b.queryParam("beforeTimestamp", beforeTimestamp);
            }
            restTemplate.getForEntity(b.toUriString(), Object.class);
            // Received: 2019-01-21T14:56:50 00:00
            //       Plus sign missing here ^
        }

        // Option 1 - no base url and encoding the param ourselves.
        {
            RestTemplate restTemplate = new RestTemplate();
            UriComponentsBuilder b = UriComponentsBuilder
                .fromHttpUrl("http://localhost:8080/search");
            if (beforeTimestamp != null) {
                b.queryParam(
                    "beforeTimestamp",
                    UriUtils.encode(beforeTimestamp, StandardCharsets.UTF_8)
                );
            }
            restTemplate.getForEntity(
                b.build(true).toUri(), Object.class
            ).getBody();
            // Received: 2019-01-21T14:56:50+00:00
        }

        // Option 2 - with templated base url, query parameter is not optional.
        {
            RestTemplate restTemplate = new RestTemplateBuilder()
                .rootUri("http://localhost:8080")
                .uriTemplateHandler(new DefaultUriBuilderFactory())
                .build();
            Map<String, String> params = new HashMap<>();
            params.put("beforeTimestamp", beforeTimestamp);
            restTemplate.getForEntity(
                "/search?beforeTimestamp={beforeTimestamp}",
                Object.class,
                params);
            // Received: 2019-01-21T14:56:50+00:00
        }
    }

    @GetMapping("/search")
    public void search(@RequestParam String beforeTimestamp) {
        System.out.println("Received: " + beforeTimestamp);
    }
}

Liegeman

I assume this was an expected change? – Dermatitis

If leaving the + (plus) sign as is were expected, then a receiving Spring Boot endpoint should not attempt to decode the + (plus) as a ` ` (space). Unfortunately, that is not the case due to seemingly conflicting standards. – Liegeman

I was having the same problem but I had started with a templated URL. The success of Option 2 seems to hinge on setting the uriTemplateHandler as you do... but why? I'm not grasping the difference between the DefaultUriBuilderFactor that you use and the DefaultUriTemplateHandler that would otherwise be used. – Morbilli

@PatrickM DefaultUriTemplateHandler seems to be deprecated, but the docs indicate the difference might be, that "DefaultUriBuilderFactory has a different default for the parsePath property (from false to true)." – Liegeman

Answer 1

We realized the URL can be modified in an interceptor after the encoding is done. So a solution would be to use an interceptor, that encodes the plus sign in the query params.

RestTemplate restTemplate = new RestTemplateBuilder()
        .rootUri("http://localhost:8080")
        .interceptors(new PlusEncoderInterceptor())
        .build();

A shortened example:

public class PlusEncoderInterceptor implements ClientHttpRequestInterceptor {

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution) throws IOException {
        return execution.execute(new HttpRequestWrapper(request) {
            @Override
            public URI getURI() {
                URI u = super.getURI();
                String strictlyEscapedQuery = StringUtils.replace(u.getRawQuery(), "+", "%2B");
                return UriComponentsBuilder.fromUri(u)
                        .replaceQuery(strictlyEscapedQuery)
                        .build(true).toUri();
            }
        }, body);
    }
}

Liegeman

Here are the needed includes: import java.io.IOException; import java.net.URI; import org.springframework.http.HttpRequest; import org.springframework.http.client.ClientHttpRequestExecution; import org.springframework.http.client.ClientHttpRequestInterceptor; import org.springframework.http.client.ClientHttpResponse; import org.springframework.http.client.support.HttpRequestWrapper; import org.springframework.util.StringUtils; import org.springframework.web.util.UriComponentsBuilder; – Piece

Answer 2

The issue has been discussed here as well.

Encoding of URI Variables on RestTemplate [SPR-16202]

A simpler solution is to set the encoding mode on the URI builder to VALUES_ONLY.

    DefaultUriBuilderFactory builderFactory = new DefaultUriBuilderFactory();
    builderFactory.setEncodingMode(DefaultUriBuilderFactory.EncodingMode.VALUES_ONLY);
    RestTemplate restTemplate = new RestTemplateBuilder()
            .rootUri("http://localhost:8080")
            .uriTemplateHandler(builderFactory)
            .build();

This achieved the same result as using the PlusEncodingInterceptor when using query parameters.

Aircrewman

It didn't work along with UriComponentsBuilder, as the query parameters aren't necessarily required, but optional. – Liegeman

Answer 3

Thanks https://stackoverflow.com/users/4466695/gregor-eesmaa, it solved my issue. Just wanted to add that in case if you can format URL before calling RestTemplate, you can fix the URL at once (instead of replacing it in PlusEncoderInterceptor):

UriComponentsBuilder uriBuilder = UriComponentsBuilder.fromUriString("/search");
uriBuilder.queryParam("beforeTimestamp", "2019-01-21T14:56:50+00:00");
URI uriPlus = uriBuilder.encode().build(false).toUri();

// import org.springframework.util.StringUtils;
String strictlyEscapedQuery = StringUtils.replace(uriPlus.getRawQuery(), "+", "%2B");
URI uri = UriComponentsBuilder.fromUri(uriPlus)
        .replaceQuery(strictlyEscapedQuery)
        .build(true).toUri();

// prints "/search?beforeTimestamp=2019-01-21T14:56:50%2B00:00"
System.out.println(uri);

Then you can use in RestTemplate call:

RequestEntity<?> requestEntity = RequestEntity.get(uri).build();
ResponseEntity<String> responseEntity = restTemplate.exchange(requestEntity, String.class);

Spaceless

Answer 4

To get around this kind of issue, I found it easier to build the URI by hand.

URI uri = new URI(siteProperties.getBaseUrl()
  + "v3/elements/"
  + URLEncoder.encode("user/" + user + "/type/" + type, UTF_8)
  + "/"
  + URLEncoder.encode(id, UTF_8)
);

restTemplate.exchange(uri, DELETE, new HttpEntity<>(httpHeaders), Void.class);

Thurlow

Answer 5

Just want to add on!, You can try with this without configure every where u used RestTemplate in your project, config also support UTF-8 encoding respons:

public class PlusEncoderInterceptor implements ClientHttpRequestInterceptor {

@Override
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution)
        throws IOException {
    return execution.execute(new HttpRequestWrapper(request) {
        @NotNull
        @Override
        public URI getURI() {
            URI u = super.getURI();
            String strictlyEscapedQuery = StringUtils.replace(u.getRawQuery(), "+", "%2B");
            return UriComponentsBuilder.fromUri(u).replaceQuery(strictlyEscapedQuery).build(true).toUri();
        }
    }, body);
}
}

@Configuration
public class RestTemplateConfig {
@Bean
public RestTemplate restTemplate() {
    RestTemplate restTemplate = new RestTemplateBuilder().interceptors(new PlusEncoderInterceptor()).build();
    restTemplate.getMessageConverters().add(0, new StringHttpMessageConverter(StandardCharsets.UTF_8));
    return restTemplate;
}
}

Disgraceful