How to parse gzip encoded response with RestTemplate in Spring-Web

Question

After I modified Consuming a RESTful Web Service example to call get users by id from api.stackexchange.com I get JsonParseException:

com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 31)): only regular white space (\r, \n, \t) is allowed between tokens

Response from api.stackexchange.com is gzip compressed.

How to add support for gzip compressed response into Spring-Web RestTemplate?

I am using Spring boot parent ver. 1.3.1.RELEASE hence Spring-Web 4.2.4-RELEASE

Here’s my adjusted example:

User.java

package stackexchange.dto;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.PropertyNamingStrategy.LowerCaseWithUnderscoresStrategy;
import com.fasterxml.jackson.databind.annotation.JsonNaming;

@JsonIgnoreProperties(ignoreUnknown = true)
@JsonNaming(LowerCaseWithUnderscoresStrategy.class)
public class User {

    // Properties made public in order to shorten the example
    public int userId;
    public String displayName;
    public int reputation;

    @Override
    public String toString() {
        return "user{"
                + "display_name='" + displayName + '\''
                + "reputation='" + reputation + '\''
                + "user_id='" + userId + '\''
                + '}';
    }
}

CommonWrapper.java

package stackexchange.dto;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.PropertyNamingStrategy.LowerCaseWithUnderscoresStrategy;
import com.fasterxml.jackson.databind.annotation.JsonNaming;

@JsonIgnoreProperties(ignoreUnknown = true)
@JsonNaming(LowerCaseWithUnderscoresStrategy.class)
public class CommonWrapper {

    // Properties made public in order to shorten the example
    public boolean hasMore;
    // an array of the type found in type
    public User[] items;
    public int page;
    public int pageSize;
    public int quotaMax;
    public int quotaRemaining;

    @Override
    public String toString() {
        StringBuilder sb = new StringBuilder();
        for (User user : items) {
            sb.append("{" + user.toString() + "}\n");
        }

        return "common_wrapper{"
        + "\"items\"=[\n"
        + sb
        + "]"
        + "has_more='" + hasMore + '\''
        + "page='" + page + '\''
        + "page_size='" + pageSize + '\''
        + "quota_max='" + quotaMax + '\''
        + "quota_remaining='" + quotaRemaining + '\''
        + '}';
    }
}

StackExchange.java

package stackexchange;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.web.client.RestTemplate;

import stackexchange.dto.CommonWrapper;

import com.fasterxml.jackson.databind.PropertyNamingStrategy.LowerCaseWithUnderscoresStrategy;
import com.fasterxml.jackson.databind.annotation.JsonNaming;

@JsonNaming(LowerCaseWithUnderscoresStrategy.class)
public class StackExchange implements CommandLineRunner{

    private static final Logger log = LoggerFactory.getLogger(StackExchange.class);

    public static void main(String args[]) {
        SpringApplication.run(StackExchange.class);
    }

    @Override
    public void run(String... strings) throws Exception {

        RestTemplate restTemplate = new RestTemplate();
        CommonWrapper response = restTemplate
                .getForObject(
                        "https://api.stackexchange.com/2.2/users/4607349?site=stackoverflow",
                        CommonWrapper.class);

        log.info(response.toString());
    }

}

pom.xml - same as in example

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>stackexchangetest</groupId>
  <artifactId>stackexchangetest</artifactId>
  <version>0.0.1</version>
  <name>stackexchangetest</name>
  <description>api.stackexchange.com Test</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.1.RELEASE</version>
    </parent>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-web</artifactId>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </dependency>   
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Grandiloquent

did you read the doc : docs.spring.io/autorepo/docs/spring-android/1.0.x/reference/… ??? – Probabilism

Yes, I read it. It’s solution for Spring Android, while I use Spring Web. HttpHeaders does not contain setAcceptEncoding method in Spring Web. I will state it in the question to avoid confusion. Thx – Grandiloquent

I love how meta this question is. – Tedie

Answer 1

Replace the default requestFactory with one from Apache HttpClient (which decodes GZIP on the fly):

HttpComponentsClientHttpRequestFactory clientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory(
            HttpClientBuilder.create().build());
RestTemplate restTemplate = new RestTemplate(clientHttpRequestFactory);

Add Apache Http Client into pom.xml

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <!--Version is not needed when used with Spring Boot parent pom file -->
    <version>4.5.1</version>
</dependency>

Grandiloquent

Thanks Michal it worked. Usual documentation describe android not spring web as you said. Why don't you also add reference if there is any blog or link for it? – Collide

Great, I am happy I could help. I did not find any article or a blog post at the time. Feel free to document it and let Google to properly index it, so that nobody will struggle again :) – Grandiloquent

Hi Michal are you aware of any disadvantage of using HttpComponentsClientHttpRequestFactory in multithreaded environment. I mean when hits from users are very high. – Collide

No I am not aware of any issues. I find HttpComponentsClientHttpRequestFactory much better than the default factory. It better handles cookies, redirects, proxy servers, etc. – Grandiloquent

Linking this article because it actually explains how/where httpclient decompresses the response if necessary. See @Garry answer on this post – Untraveled

This solution is not perfect. HttpClient has its own thread pool. When server side timesout , in this approach HttpClient could be waiting forever. This will eventually exhausts all the threads in the pool. ``` at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:393) .. at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776) ··· – Ramadan

See additional info in the answer provided by @kerbermeister (a different library is needed) – Lodgings

Answer 2

I wanted to solve the same issue without using additional libraries. What helped me was to

1. use the responseType of byte[].class
2. unzipping response myself
3. mapping the entity myself by ObjectMapper mapper.

It is not the most elegant solution but it works.

    ResponseEntity<byte[]> response = restTemplate.exchange(url, HttpMethod.GET, new HttpEntity<byte[]>(createHeaders()), byte[].class);
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    try (GZIPInputStream gzipInputStream = new GZIPInputStream(new ByteArrayInputStream(response.getBody()))) {
        gzipInputStream.transferTo(byteArrayOutputStream);
    }
    byte[] content = byteArrayOutputStream.toByteArray();
    GenericResponseDto question = mapper.readValue(content, GenericResponseDto .class);  
    log.info("Response :" + question.toString());

Nucleolated

Answer 3

If you want to use the approach above (with Apache Http Client), then you should now that the artifact was moved to

org.apache.httpcomponents.client5 » httpclient5

So, your dependency would look like this:

<dependency>
    <groupId>org.apache.httpcomponents.client5</groupId>
    <artifactId>httpclient5</artifactId>
</dependency>

Don't forget to specify the version unless you're using Spring Boot POM

Preacher

Answer 4

private String callViaRest(String requestString, Steps step) {
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.TEXT_XML);
    headers.add("Accept-Encoding", "application/gzip");
    HttpEntity<String> entity = new HttpEntity<String>(requestString, headers);

    byte[] responseBytes = jsonRestTemplate
            .exchange("yourUrl", HttpMethod.POST, entity, byte[].class).getBody();
    String decompressed = null;
    try {
        decompressed= new String(CompressionUtil.decompressGzipByteArray(responseBytes),Charsets.UTF_8);
    } catch (IOException e) {
        LOGGER.error("network call failed.", e);
    }
    return decompressed;
}

Collide

Add dependency for org.apache.commons.jcs to pom. import org.apache.commons.jcs.utils.zip.CompressionUtil; – Collide

by adding headers.add("Accept-Encoding", "application/gzip"); working for me thanks for solution. – Madera

I know it's been a long time since this answer was posted but I have a question. What about responseBody when a HttpStatusCodeException is thrown? I mean, with the factory posted in the accepted answer you can get the response body if an HttpStatusCodeException is thrown by exchange method, but with this answer, I wasn't able to recover response in that error case. – Tuff