Retrofit: how to parse GZIP'd response without Content-Encoding: gzip header

Question

I'm trying to process a server response which is GZIP'd. The response comes with a header

Content-Type: application/x-gzip

but does not have header

Content-Encoding: gzip

If I add that header using a proxy, the response gets parsed just fine. I don't have any control over the server, so I can't add the header.

Can I force Retrofit to treat it as GZIP content? Is there a better way? The URL for the server is: http://crowdtorch.cms.s3.amazonaws.com/4474/Updates/update-1.xml

Baccarat

Answer 1

There is a better way than reinventing the wheel. Just add the Content-Encoding header yourself.

.addNetworkInterceptor((Interceptor.Chain chain) -> {
    Request req = chain.request();
    Headers.Builder headersBuilder = req.headers().newBuilder();

    String credential = Credentials.basic(...);
    headersBuilder.set("Authorization", credential);

    Response res = chain.proceed(req.newBuilder().headers(headersBuilder.build()).build());

    return res.newBuilder()
        .header("Content-Encoding", "gzip")
        .header("Content-Type", ""application/json")
        .build();
})

In fact, your code is a classic example of the evils of using internal code (like com.sun packages from the JDK). RealResponseBody doesn't have that constructor anymore.

Marcello

Interesting. Use an interceptor. Good idea. Sounds like it ought to work. – Baccarat

@Baccarat it does, I use it – Marcello

Answer 2

I figured it out. The idea is to add a custom interceptor which will take the not-yet-unzipped response, and unzip it 'manually' - do the same thing that OkHttp would do automatically based on Content-Encoding header, but without requiring that header.

is like dis:

    OkHttpClient.Builder clientBuilder = new OkHttpClient.Builder()
            .addInterceptor(new UnzippingInterceptor());
    OkHttpClient client = clientBuilder.build();

And the Interceptor is like dis:

private class UnzippingInterceptor implements Interceptor {
    @Override
    public Response intercept(Chain chain) throws IOException {
        Response response = chain.proceed(chain.request());
        return unzip(response);
    }
}

And the unzip function is like dis:

    // copied from okhttp3.internal.http.HttpEngine (because is private)
private Response unzip(final Response response) throws IOException {

    if (response.body() == null) {
        return response;
    }

    GzipSource responseBody = new GzipSource(response.body().source());
    Headers strippedHeaders = response.headers().newBuilder()
            .removeAll("Content-Encoding")
            .removeAll("Content-Length")
            .build();
    return response.newBuilder()
            .headers(strippedHeaders)
            .body(new RealResponseBody(strippedHeaders, Okio.buffer(responseBody)))
            .build();
}

Baccarat

Answer 3

This worked for me with gzip files:

private fun decompressGzip(responseBody: ResponseBody): String {
    try {
        // Create a GZIPInputStream to decompress the data
        val gzipInputStream = GZIPInputStream(responseBody.byteStream())

        // Use a ByteArrayOutputStream to hold the decompressed data
        val byteArrayOutputStream = ByteArrayOutputStream()

        // Buffer for reading data
        val buffer = ByteArray(1024)
        var bytesRead: Int

        // Read from the GZIPInputStream and write to the ByteArrayOutputStream
        while (gzipInputStream.read(buffer).also { bytesRead = it } != -1) {
            byteArrayOutputStream.write(buffer, 0, bytesRead)
        }

        // Convert the ByteArrayOutputStream to a byte array
        val decompressedData = byteArrayOutputStream.toByteArray()

        // Optionally convert the byte array to a string or other format as needed
        return String(decompressedData, StandardCharsets.UTF_8)

    } catch (e: Exception) {
        e.printStackTrace()
        // Handle exceptions
    }

    return ""
}

fun getGzip(
    country: String,
    onRadioMutableListGetCallback: OnRadioListGetCallback
) {
    val call = apiServiceBackup.MutableListByCountryBackup(
        country.toLowerCase(),
    )
    call.enqueue(object : Callback<ResponseBody> {
        override fun onResponse(call: Call<ResponseBody>, response: Response<ResponseBody>) {
            val res = response.body() ?: return
            val json = decompressGzip(res)
            val gson = Gson()
            val myObjectListType = object : TypeToken<List<RadioData>>() {}.type
            val stations = gson.fromJson<MutableList<RadioData>>(json, myObjectListType)
            onRadioMutableListGetCallback.onRadioListGet(stations)
        }

        override fun onFailure(call: Call<ResponseBody>, t: Throwable) {
            onRadioMutableListGetCallback.onRadioListGet(ArrayList())
        }
    })
}

    @GET("{country}.json.gz")
    fun MutableListByCountryBackup(
        @Path("country") user: String?
    ): Call<ResponseBody>

Squamous