Volley out of memory error, weird allocation attempt

Question

Sometimes randomly Volley crashes my app upon startup, it crashes in the application class and a user would not be able to open the app again until they go into settings and clear app data

java.lang.OutOfMemoryError
at com.android.volley.toolbox.DiskBasedCache.streamToBytes(DiskBasedCache.java:316)
at com.android.volley.toolbox.DiskBasedCache.readString(DiskBasedCache.java:526)
at com.android.volley.toolbox.DiskBasedCache.readStringStringMap(DiskBasedCache.java:549)
at com.android.volley.toolbox.DiskBasedCache$CacheHeader.readHeader(DiskBasedCache.java:392)
at com.android.volley.toolbox.DiskBasedCache.initialize(DiskBasedCache.java:155)
at com.android.volley.CacheDispatcher.run(CacheDispatcher.java:84)

The "diskbasedbache" tries to allocate over 1 gigabyte of memory, for no obvious reason

how would I make this not happen? It seems to be an issue with Volley, or maybe an issue with a custom disk based cache but I don't immediately see (from the stack trace) how to 'clear' this cache or do a conditional check or handle this exception

Insight appreciated

Amortize

Answer 1

In the streamToBytes(), first it will new bytes by the cache file length, does your cache file was too large than application maximum heap size ?

private static byte[] streamToBytes(InputStream in, int length) throws IOException {
    byte[] bytes = new byte[length];
    ...
}

public synchronized Entry get(String key) {
    CacheHeader entry = mEntries.get(key);

    File file = getFileForKey(key);
    byte[] data = streamToBytes(..., file.length());
}

If you want to clear the cache, you could keep the DiskBasedCache reference, after clear time's came, use ClearCacheRequest and pass that cache instance in :

File cacheDir = new File(context.getCacheDir(), DEFAULT_CACHE_DIR);
DiskBasedCache cache = new DiskBasedCache(cacheDir);
RequestQueue queue = new RequestQueue(cache, network);
queue.start();

// clear all volley caches.
queue.add(new ClearCacheRequest(cache, null));

this way will clear all caches, so I suggest you use it carefully. of course, you can doing conditional check, just iterating the cacheDir files, estimate which was too large then remove it.

for (File cacheFile : cacheDir.listFiles()) {
    if (cacheFile.isFile() && cacheFile.length() > 10000000) cacheFile.delete();
}

Volley wasn't design as a big data cache solution, it's common request cache, don't storing large data anytime.

------------- Update at 2014-07-17 -------------

In fact, clear all caches is final way, also isn't wise way, we should suppressing these large request use cache when we sure it would be, and if not sure? we still can determine the response data size whether large or not, then call setShouldCache(false) to disable it.

public class TheRequest extends Request {
    @Override
    protected Response<String> parseNetworkResponse(NetworkResponse response) {
        // if response data was too large, disable caching is still time.
        if (response.data.length > 10000) setShouldCache(false);
        ...
    }
}

Brophy

okay, so I could clear this cache when the application class first opens, or in my main activity.. i'll have to inspect the disk based cache file – Amortize

this was helpful but it turned out one of my third party libraries was also initializing a new Volley request, I got them to change their library – Amortize

If you sure some request will take large data, you can suppressing these request don't cache it via Request.setShouldCache(false) method. – Brophy

Should i clear the cache in application class or my activity ? Where should i put this code? – Bellringer

@user3753273 of course, you can clear cache anywhere you wanted, but volley is not much convenient for clearing caches. i've been described all the ways. – Brophy

i tried putting if (response.data.length > 10000) setShouldCache(false); inside all the parseNetworkResponse() methods in volley but still the error appears – Bellringer

as I ever said before, it seems DiskBasedCache.streamToBytes() trying to allocating a tremendous memory so causing this error. i'd recommend you to debug(i.e. blocking) that method, checking what length is, also figure out why that length was large. – Brophy

@Amortize can you share which third party library was causing this for you? I am now having an identical issue, but adding these fixes to my own code didn't work (plus I am not loading any huge files). I'd love to know if maybe I'm using the same library that you were which caused this problem. – Java

@Java it was a library from a client that is not publicly distributed, and I told them to fix their constructor to let me pass my own volley object into their library, so it isn't the same one – Amortize

I may be misunderstanding something, but the issue stacktrace reports an overflow when reading a header, not cache content. A header should never be huge, and seems to pointe to a different issue than just the cache being full. – Ralph

Answer 2

I experienced the same issue.

We knew we didn't have files that were GBs in size on initialization of the cache. It also occurred when reading header strings, which should never be GBs in length.

So it looked like the length was being read incorrectly by readLong.

We had two apps with roughly identical setups, except that one app had two independent processes created on start up. The main application process and a 'SyncAdapter' process following the sync adapter pattern. Only the app with two processes would crash. These two processes would independently initialize the cache.

However, the DiskBasedCache uses the same physical location for both processes. We eventually concluded that concurrent initializations were resulting in concurrent reads and writes of the same files, leading to bad reads of the size parameter.

I don't have a full proof that this is the issue, but I'm planning to work on a test app to verify.

In the short term, we've just caught the overly large byte allocation in streamToBytes, and throw an IOException so that Volley catches the exception and just deletes the file. However, it would probably be better to use a separate disk cache for each process.

 private static byte[] streamToBytes(InputStream in, int length) throws IOException {
    byte[] bytes;

    // this try-catch is a change added by us to handle a possible multi-process issue when reading cache files
    try {
        bytes = new byte[length];
    } catch (OutOfMemoryError e) {
        throw new IOException("Couldn't allocate " + length + " bytes to stream. May have parsed the stream length incorrectly");
    }

    int count;
    int pos = 0;
    while (pos < length && ((count = in.read(bytes, pos, length - pos)) != -1)) {
        pos += count;
    }
    if (pos != length) {
        throw new IOException("Expected " + length + " bytes, read " + pos + " bytes");
    }
    return bytes;
}

Ralph

Rather than try to recover from OOME, you could reject lengths that are larger than the cache size (or negative). The default DiskBasedCache size is only 5MB. Note that others have reported NegativeArraySizeException thrown from the same location: code.google.com/p/android/issues/detail?id=209471 – Greathearted

Here are related issues in the android-volley mirror: github.com/mcxiaoke/android-volley/issues/37 github.com/mcxiaoke/android-volley/issues/61 – Greathearted

Answer 3

Once the problem occurs, it seems to recur on every subsequent initialization, pointing to an invalid cached header.

Fortunately, this issue has been fixed in the official Volley repo:

• https://github.com/google/volley/issues/12

See related issues in the android-volley mirror:

• https://github.com/mcxiaoke/android-volley/issues/141
• https://github.com/mcxiaoke/android-volley/issues/61
• https://github.com/mcxiaoke/android-volley/issues/37

Greathearted