UTF-8 encoded characters from REST-query not rendered properly
Asked Answered
L

5

3

I'm consuming an external REST service that provides all content as UTF-8 encoded.

For some reason my application cannot properly handle the response. If I dump the response I will se things like LuleÃ¥ (should be Luleå).

EDIT: The same behavior happens if i forward (without altering) the string to the UI, ex.:

flash.message = "Test" + integrationService.testEncoding()

What I did was to create a _Events.groovy file in the /script folder and specifying there that

eventConfigureTomcat = { tomcat ->
    tomcat.connector.URIEncoding = "UTF-8"
    tomcat.connector.useBodyEncodingForURI = true
}

I also have the following in my Config.groovy:

grails.views.gsp.encoding = "UTF-8"
grails.converters.encoding = "UTF-8"

But that changed nothing. The response is still wrongly shown. I'm not sure if this is a configuration issue with Grails, with the embedded tomcat or with something else. I'm currently running my test setup on windows 7, but the same issue happens on my server running on Centos. Please advice.

EDIT2: If i consume the REST service using curl, everything is rendered correctly in the output.

EDIT3: I'm using org.springframework.web.client.RestTemplate and HttpComponents to consume the service:

private static final HttpHeaders requestHeaders
static{
    requestHeaders = new HttpHeaders()
    requestHeaders.set(HttpHeaders.CONTENT_TYPE, "application/json")
    requestHeaders.set(HttpHeaders.ACCEPT, "application/json")
    requestHeaders.set("Accept-Encoding", "gzip")
}

private final static RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory(
    HttpClientBuilder.create().build()))
...
...
public def testEncoding(){
    ResponseEntity<String> response = restTemplate.exchange(
            "https://www.url.com", HttpMethod.GET, new HttpEntity<Object>(requestHeaders),
            String.class)
    def gamesJson = JSON.parse(response.getBody())
    //...
    //parse value from gamesJson
    //...
    return testValue
}
Lello answered 12/2, 2015 at 15:16 Comment(11)
Are you sure that the place you "dump" the response is properly able to handle UTF-8? If you're just doing a println on the Windows console, for example, then you might be printing UTF-8 bytes but having the console interpret them as if they were a single byte encoding like windows-1252Relic
@ian roberts Probably not. If I query the api with a browser (IE) I get a response that has the same character encoding problem. I I try to output the queried "text" in a gsp page, I have the same encoding problem. I'm a bit at a loss here about what is causing this.Lello
Can you create a sample app replicating the issue? or at least show the code where the service is consumed? Are you using rest client builder. Try setting the content-type in headers as "application/json;charset=utf-8"Keon
@Keon the http response has those in the header. Added example code. I can make an example app if neccessary.Lello
What do you have in requestHeaders?Keon
@Keon Edited into questionLello
Now as I said earlier to use the content type as application/json;charset=utf-8. Modify as requestHeaders.set(HttpHeaders.CONTENT_TYPE, "application/json;charset=utf-8")Keon
@Keon No effect. Still get Jääkiekko.Lello
@dmahapatro, so I receive UTF-8 bytes, but they are shown as ANSI by the browser. Yet the page contains <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">. Now I'm really lost, why is the browser not rendering utf-8 bytes as utf-8?Lello
Refer #7810431Keon
@Keon Thank you for your help, but I finally managed to find a solution here, mainly thanks to you leading me in the right direction.Lello
A
8

Per my previous answer:

You just need to add the StringHttpMessageConverter to the template's message converters:

RestTemplate template = new RestTemplate();
template.getMessageConverters()
        .add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));
ResponseEntity<Object> response = template.exchange(endpoint, method, entity, 
                                                    Object.class);
Arlenarlena answered 22/5, 2015 at 6:33 Comment(0)
C
3

The encoding type can be enforced in the environment itself.

JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8 -Dclient.encoding.override=UTF-8

Just try setting the above encoding settings in windows/linux. I hope this should resolve the issue.

In this case, JVM will pickup the default encoding type from environment variable.

Cook answered 20/5, 2015 at 7:35 Comment(2)
This does not change the behavior. Problem persists.Lello
i'm using groovy RestConnector on cygwin+windows and this solve a problem connecting to a service (elasticsearch), thanks!Hornsby
R
2

Our team experienced a similar issue before, we have a 3rd party service and they said their output is encoded in UTF-8. But the strings returned are still garbled. After a bit of testing, it turns out that they were returning ISO-8859-1 encoded strings. What we did was to decode/encode their input into UTF-8 encoded characters so we can use those properly.

For your case, I think this is a similar issue:

UTF-8: Luleå

ISO-8859-1: Luleå

In Java, we did something like this:

Charset initialEncoding = Charsets.ISO_8859_1;
Charset outputEncoding = Charsets.UTF_8; 
byte[] byteArray = input.getBytes(initialEncoding);
String output = new String(new String(byteArray, outputEncoding));

In Groovy, I think you could do something like

import groovy.json.JsonSlurper;

def main = {
    def response = '{"name":"Luleå"}' 

    def slurper = new JsonSlurper()
    def result = slurper.parse(response.getBytes(), 'UTF-8')

    println result.name // prints Luleå
}
Relegate answered 21/5, 2015 at 9:55 Comment(1)
Thank you for the effort! However, the solution was much easier.Lello
L
2

The answer to my problem is already found on Stack Exchange.

You just need to add the StringHttpMessageConverter to the template's message converters:

restTemplate.getMessageConverters()
    .add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));
Lello answered 22/5, 2015 at 6:12 Comment(0)
D
0

In my case that i had the same problem with contents received from my REST web service from the server and not my local enviroment. I had search a lot and finally i found a solution that resolved my issue. In Windows I added a new environment variable with key: JAVA_TOOL_OPTIONS and set its value to: -Dfile.encoding=UTF8. The (Java) System property will be set automatically every time a JVM is started. You will know that the parameter has been picked up because the following message will be posted to System.err:

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Discreet answered 16/3, 2020 at 8:47 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.