Why does volley's response string use an encoding different from that in the response headers?

Question

When doing a volley request (either StringRequest or JsonObjectRequest), using the OkHttp stack, the response string's encoding is set to ISO-8995-1, which is the default encoding. The response has a header: content-type=text/html; charset=utf-8, which should be detected. Why isn't it?

Loring

Answer 1

Both of those request types call HttpHeaderParser.parseCharset, which is able to determine the charset from the headers. However, it requires that the header be Content-Type, not content-type: it is case sensitive. (I'm not sure of the behavior if using the default HurlStack, it's possible this is an implementation detail difference with the OkHttp stack.)

Solution 1: copy the original request type, but manually override the charset

Solution 2: copy the original request type, but force the expected header to exist

import com.android.volley.NetworkResponse;
import com.android.volley.ParseError;
import com.android.volley.Response;
import com.android.volley.Response.ErrorListener;
import com.android.volley.Response.Listener;
import com.android.volley.toolbox.HttpHeaderParser;
import com.android.volley.toolbox.JsonRequest;

import org.json.JSONException;
import org.json.JSONObject;

import java.io.UnsupportedEncodingException;

public class JsonUTF8Request extends JsonRequest<JSONObject> {
    public JsonUTF8Request(int method, String url, JSONObject jsonRequest,
                           Listener<JSONObject> listener, ErrorListener errorListener) {
        super(method, url, (jsonRequest == null) ? null : jsonRequest.toString(), listener,
                errorListener);
    }

    @Override
    protected Response<JSONObject> parseNetworkResponse(NetworkResponse response) {
        try {
            // solution 1:
            String jsonString = new String(response.data, "UTF-8");
            // solution 2:
            response.headers.put(HTTP.CONTENT_TYPE,
                response.headers.get("content-type"));
            String jsonString = new String(response.data, HttpHeaderParser.parseCharset(response.headers));
            //
            return Response.success(new JSONObject(jsonString),
                    HttpHeaderParser.parseCacheHeaders(response));
        } catch (UnsupportedEncodingException e) {
            return Response.error(new ParseError(e));
        } catch (JSONException je) {
            return Response.error(new ParseError(je));
        }
    }
}

Loring

this is an old question, but if it helps today, you can kind of mix both solutions into one. String jsonString = new String(response.data, HttpHeaderParser.parseCharset(response.headers, "UTF-8")); That way you can set the default charset if none exists. – Whitehurst

Answer 2

First thanks alot @mjibson for the 2 solutions you posted here, I had a similar problem, in my case the content type was always missing so did the following:

    protected static final String TYPE_UTF8_CHARSET = "charset=UTF-8";

    @Override
    protected Response<String> parseNetworkResponse(
            NetworkResponse response) {
        try {
            String type = response.headers.get(HTTP.CONTENT_TYPE);
            if (type == null) {
                Log.d(LOG_TAG, "content type was null");
                type = TYPE_UTF8_CHARSET;
                response.headers.put(HTTP.CONTENT_TYPE, type);
            } else if (!type.contains("UTF-8")) {
                Log.d(LOG_TAG, "content type had UTF-8 missing");
                type += ";" + TYPE_UTF8_CHARSET;
                response.headers.put(HTTP.CONTENT_TYPE, type);
            }
        } catch (Exception e) {
            //print stacktrace e.g.
        }
        return super.parseNetworkResponse(response);
    }

i just wanted to share this for others to run into a similar problem. its also important to read the parseCharset method in https://android.googlesource.com/platform/frameworks/volley/+/master/src/com/android/volley/toolbox/HttpHeaderParser.java to get why this works

Rodrigues

Answer 3

Override the Method parseNetworkResponse of the Request<T> Class.
You can do like this:

/**
 * A canned request for retrieving the response body at a given URL as a String.
 */
public class StringRequest extends Request<String> {
    private final Listener<String> mListener;


    /**
     * the parse charset.
     */
    private String charset = null;

    /**
     * Creates a new request with the given method.
     *
     * @param method the request {@link Method} to use
     * @param url URL to fetch the string at
     * @param listener Listener to receive the String response
     * @param errorListener Error listener, or null to ignore errors
     */
    public StringRequest(int method, String url, Listener<String> listener,
            ErrorListener errorListener) {
        super(method, url, errorListener);
        mListener = listener;
    }

    /**
     * Creates a new GET request.
     *
     * @param url URL to fetch the string at
     * @param listener Listener to receive the String response
     * @param errorListener Error listener, or null to ignore errors
     */
    public StringRequest(String url, Listener<String> listener, ErrorListener errorListener) {
        this(Method.GET, url, listener, errorListener);
    }

    /**
     * Creates a new GET request with the given Charset.
     *
     * @param url URL to fetch the string at
     * @param listener Listener to receive the String response
     * @param errorListener Error listener, or null to ignore errors
     */
    public StringRequest(String url, String charset, Listener<String> listener, ErrorListener errorListener) {
        this(Method.GET, url, listener, errorListener);
        this.charset = charset;
    }

    @Override
    protected void deliverResponse(String response) {
        mListener.onResponse(response);
    }

    @Override
    protected Response<String> parseNetworkResponse(NetworkResponse response) {
        String parsed;
        try {
            if(charset != null) {
                parsed = new String(response.data, charset);
            } else {
                parsed = new String(response.data, HttpHeaderParser.parseCharset(response.headers));
            }
        } catch (UnsupportedEncodingException e) {
            parsed = new String(response.data);
        }
        return Response.success(parsed, HttpHeaderParser.parseCacheHeaders(response));
    }

    /**
     * @return the Parse Charset Encoding
     */
    public String getCharset() {
        return charset;
    }

    /**
     * set the Parse Charset Encoding
     * @param charset
     */
    public void setCharset(String charset) {
        this.charset = charset;
    }

}

Ofeliaofella

Answer 4

Change the Method from GET to POST for UTF-8 suppport

JsonObjectRequest jsonReq = new JsonObjectRequest(Method.POST,
            URL_FEED, null, new Response.Listener<JSONObject>() {
                @Override
                public void onResponse(JSONObject response) {
                    VolleyLog.d(TAG, "Response: " + response.toString());
                    Log.d("SHY", "Response: " + response.toString());
                    if (response != null) {
                        parseJsonFeed(response);
                    }
                }
            }, new Response.ErrorListener() {
                @Override
                public void onErrorResponse(VolleyError error) {
                    VolleyLog.d(TAG, "Error: " + error.getMessage());
                }
            });

. . . .

Losel

Answer 5

Thank you @Simon Heinen. Based on your reply, I wrote a function.

private void addEncodeing2Request(NetworkResponse response) {
    try {
        String type = response.headers.get(HTTP.CONTENT_TYPE);
        if (type == null) {
            //Content-Type:
            Log.d("RVA", "content type was null");
            type = TYPE_UTF8_CHARSET;
            response.headers.put(HTTP.CONTENT_TYPE, type);
        } else if (!type.contains("charset")) {
            //Content-Type: text/plain;
            Log.d("RVA", "charset was null, added encode utf-8");
            type += ";" + TYPE_UTF8_CHARSET;
            response.headers.put(HTTP.CONTENT_TYPE, type);
        } else {
            //nice! Content-Type: text/plain; charset=utf-8'
            Log.d("RVA", "charset is " + type);
        }

    } catch (Exception e) {
        e.printStackTrace();
    }
}

Usage:

protected Response<String> parseNetworkResponse(NetworkResponse response) {
          addEncodeing2Request(response);
          return super.parseNetworkResponse(response);
      }

In addition, override getParamsEncoding() may also work.

protected String getParamsEncoding() {
            return "utf-8";
        }

Sclerenchyma