Jackson error "Illegal character... only regular white space allowed" when parsing JSON

Question

I am trying to retrieve JSON data from a URL but get the following error:

Illegal character ((CTRL-CHAR, code 31)):
only regular white space (\r, \n,\t) is allowed between tokens

My code:

final URI uri = new URIBuilder(UrlConstants.SEARCH_URL)
      .addParameter("keywords", searchTerm)
      .addParameter("count", "50")
      .build();
  node = new ObjectMapper().readTree(new URL(uri.toString())); <<<<< THROWS THE ERROR

The url constructed is i.e https://www.example.org/api/search.json?keywords=iphone&count=50

What is going wrong here? And how can I parse this data successfully?

---

Imports:

import com.google.appengine.repackaged.org.codehaus.jackson.JsonNode;
import com.google.appengine.repackaged.org.codehaus.jackson.map.ObjectMapper;
import com.google.appengine.repackaged.org.codehaus.jackson.node.ArrayNode;
import org.apache.http.client.utils.URIBuilder;

example response

{
    meta: {
        indexAllowed: false
    },
    products: {
        products: [ 
            {
                id: 1,
                name: "Apple iPhone 6 16GB 4G LTE GSM Factory Unlocked"
            },
            {
                id: 2,
                name: "Apple iPhone 7 8GB 4G LTE GSM Factory Unlocked"
            }
        ]
    }
}

Disappearance

could you add the JSON returned by URL to be parsed? – Mooneye

Just a note: Please always use example.org or example.com for example domain names. If you make up a domain, you may create trouble for whoever happens to own it. See example.com on Wikipedia. – Sheol

emaple response added – Disappearance

@rogger2016: The problem is probably an invisible character in the response. Could you add a hexdump of the response? For example on Linux or using cygwin on Windows, run: curl http://example.org/my-rest-url |hexdump -C . – Sheol

Hi @sleske, I have curled the URL and get a response...then I run the JSON through JSLint and get 'Valid JSON' – Disappearance

@rogger2016: Well, how does that matter? JSLint is for checking JavsScript not for JSON. The two are different (though they may look similar). At any rate, some checking tools may be more tolerant than Jackson, so even if a checker finds the JSON ok that may not matter. So, please post the hexdump as I wrote, otherwise we're getting nowhere... – Sheol

Answer 1

The message should be pretty self-explanatory:

There is an illegal character (in this case character code 31, i.e. the control code "Unit Separator") in the JSON you are processing.

In other words, the data you are receiving is not proper JSON.

---

Background:

The JSON spec (RFC 7159) says:

2. JSON Grammar

A JSON text is a sequence of tokens. The set of tokens includes six tructural characters, strings, numbers, and three literal names.

[...]

Insignificant whitespace is allowed before or after any of the six structural characters.

ws = *(

%x20 / ; Space

%x09 / ; Horizontal tab

%x0A / ; Line feed or New line

%x0D ) ; Carriage return

In other words: JSON may contain whitespace between the tokens ("tokens" meaning the part of the JSON, i.e. lists, strings etc.), but "whitespace" is defined to only mean the characters Space, Tab, Line feed and Carriage return.

Your document contains something else (code 31) where only whitespace is allowed, hence is not valid JSON.

---

To parse this:

Unfortunately, the Jackson library you are using does not offer a way to parse this malformed data. To parse this successfully, you will have to filter the JSON before it is handled by Jackson.

You will probably have to retrieve the (pseudo-)JSON yourself from the REST service, using standard HTTP using, e.g. java.net.HttpUrlConnection. Then suitably filter out "bad" characters, and pass the resulting string to Jackson. How to do this exactly depends on how you use Jackson.

Feel free to ask a separate questions if you are having trouble :-).

Sheol

Cheers for the response...Im not in control of the JSON, Is there a way around his...When I hit the url in the browser I get a response so is there a way to make ObjectMapper less strict??? – Disappearance

@rogger2016: That's a different question :-). I'll try to expand my answer. – Sheol

#42658981 @Sheol – Disappearance

@rogger2016: I have already added some help to my answer above. If you are still having trouble, please ask a more specific question describing what you tried and where it failed. – Sheol

Answer 2

I got this same issue, and I found that it was caused by the Content-Encoding: gzip header. The client application (where the exception was being thrown) was not able to handle this content-encoding. FWIW the client application was using io.github.openfeign:feign-core:9.5.0, and this library appears to have some issues around compression (link).

You might try adding the header Accept-Encoding: identity to your request, however, not all web servers/web applications are configured properly, and some seem to disregard this header. See this question for more details about how to prevent gzipped content.

Appalachian

Thanks a lot. You are really a life saver. – Radish

Thank you for you answer, you saved me from hours of debugging! – Schorl

Those who face this issue after the version update of spring-cloud-openfeign-core to 2.2.5.RELEASE, please refer #63609235 – Petiolate

Thank you! This fixed the issue for me! – Boabdil

Check the below link. I added the above header but had no luck. Spring is not able to Decode the response on the fly, so you need to define a custom GZip Decoder. #23561240 – Leatherman

I tried it and it worked 🙏 – Xanthine

Answer 3

I had a similar issue. After some research, I found out that restTemplate uses the SimpleClientHttpRequestFactory which does not support gzip encoding. To enable gzip encoding for your response, you will need to set a new request factory for the rest template object - HttpComponentsClientHttpRequestFactory.

restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory());

Wheresoever

This saved my day! – Messere

Answer 4

I had the same problem. After setting Gzip it was fixed. Please refer my code

public String sendPostRequest(String req) throws Exception {

    // Create connection
    URL urlObject = new URL(mURL);
    HttpURLConnection connection = (HttpURLConnection) urlObject.openConnection();
    connection.setRequestMethod("POST");
    connection.setRequestProperty("Content-Type", "application/json");
    connection.setRequestProperty("Content-Length", Integer.toString(req.getBytes().length));
    connection.setRequestProperty("Content-Language", "en-US");
    connection.setUseCaches(false);
    connection.setDoOutput(true);

    // Send request
    DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
    wr.writeBytes(req);
    wr.close();

    //Response handling
    InputStream responseBody                = null;
    if (isGzipResponse(connection)) {
        responseBody                = new GZIPInputStream(connection.getInputStream());         
    }else{
        responseBody = connection.getInputStream();
    }
    convertStreamToString(responseBody);

    return response.toString();

}

protected boolean isGzipResponse(HttpURLConnection con) {
    String encodingHeader = con.getHeaderField("Content-Encoding");
    return (encodingHeader != null && encodingHeader.toLowerCase().indexOf("gzip") != -1);
}

public void convertStreamToString(InputStream in) throws Exception {
    if (in != null) {

        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        byte[] buffer = new byte[4096];
        int length = 0;
        while ((length = in.read(buffer)) != -1) {
            baos.write(buffer, 0, length);
        }

        response = new String(baos.toByteArray());

        baos.close();

    } else {
        response = null;
    }

}

Hagiology

Answer 5

I had the same issue with zalando logbook in my spring boot application, and after reading the answers in here carefully, I realized, that the response interceptor must be applied after whatever takes care for decompression:

@Configuration
public class RestTemplateConfig {

   [....]

   @Bean
   public RestTemplate restTemplate() {
       return new RestTemplateBuilder()
               .requestFactory(new MyRequestFactorySupplier())
               .build();
   }

   class MyRequestFactorySupplier implements Supplier<ClientHttpRequestFactory> {
       @Override
       public ClientHttpRequestFactory get() {
           CloseableHttpClient client = HttpClientBuilder.create()
                   .addInterceptorFirst(logbookHttpRequestInterceptor)
        // wrong:  .addInterceptorFirst(logbookHttpResponseInterceptor)
                   .addInterceptorLast(logbookHttpResponseInterceptor)
                   .build();
           HttpComponentsClientHttpRequestFactory clientHttpRequestFactory = 
                  new HttpComponentsClientHttpRequestFactory(client);
           return clientHttpRequestFactory;
       }
   }
}

Greaser

Answer 6

Those who use FeignClient, please refer to this answer spring-feign-not-compressing-response

Spring is not able to Decode the response on the fly, so you need to define a custom GZip Decoder.

Solved for me.

Leatherman

Answer 7

We had the same issue in out integration tests recently. We have a spring boot application and we use wiremock to mock a integrated microservice server. For one of the test get requests that we had implemented we started getting this error. We had to downgrade wiremock from 2.18.0 to 2.17.0 and it worked fine. Due to some bug the jackson parser and the that particular version of wiremock didn't work together. We didnt have time to figure out what actually the bug was in those libraries.

Garry

Answer 8

We were using Compression

feign.compression.request.enabled=true
feign.compression.request.mime-types=text/xml,application/xml,application/json
feign.compression.request.min-request-size=2048
feign.compression.response.enabled=true

which caused this. This happened only after migrating from OkHttp to Apache Http and there may have been other factors involved as well which were not reproducible anymore, as it was a lot of guesswork.

Value

Answer 9

Migration issue:

We faced this issue while migrating from Spring boot version 2.7 to version 3.0. Some of our test cases were failing after upgrading to the latest version.

Observations:

• We were using Test Containers for integration testing AWS END to END flow.

• I upgraded my AWS dependency version to the latest available version, but again had to downgrade my aws.version from 2.26.7 to 2.17.163.

  <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>bom</artifactId>
      <version>2.17.163</version>
      <type>pom</type>
      <scope>runtime</scope>
  </dependency>
  

  It was alone sufficient to resolved my issue.

---

• We have 2 types of files, viz. compressed(GZIP) and normal files.

• The issue was caused when we upload compressed files to the AWS and later try to download the same file.

• When we tried to download the files using s3Client.getObject(..);

• We receive ContentEncoding header in the response as aws-chunked and not gzip.

• Later, when checking the content encoding:

  if (GZIP.equals(contentEncoding)) {
       return decompress.apply(bytes);
  else {
       return bytes;
  }
  

  This is where the problem originated for us. We used to give the compressed file to the Jackson for parsing and that used to cause the issue.

• GitHub issue for the same: https://github.com/aws/aws-sdk-java-v2/issues/4746

Ezechiel