FileWriter out of memory

Asked 27/5, 2015 at 19:15 Answered 23/10, 2018 at 3:28

I want to format and write the contents of a large Map (1.785.530 entries) to a text file. After about 85% of the entries are processed, it gets very slow and then I get an OutOfMemoryException.

This same error occurs, even if I:

periodically call flush() or close() on my FileWriter
use BufferedWriter
write each line to a StringBuffer before writing to the file

This is my code:

private static final TreeMap<Date, Integer> accessesPerSecondMap = new    
    TreeMap<>();

...

private static void writeOutputFile() throws IOException {
    FileWriter writer = new FileWriter(FILENAME_OUTPUT);

    writer.write("Date");
    writer.write(',');
    writer.write("Request Count");
    writer.write('\n');

    for (Date date : accessesPerSecondMap.keySet()) {

        // first and last date are not precise so do not write it in the
        // file
        if (date == accessesPerSecondMap.firstKey()
                || date == accessesPerSecondMap.lastKey()) {
            continue;
        }

        writer.write(String.valueOf(date));
        System.out.println("FileMerger wrote: " + String.valueOf(date));
        writer.write(',');
        writer.write(String.valueOf(accessesPerSecondMap.get(date)));
        writer.write('\n');

    }

    writer.flush();
    writer.close();
}

And here is the exception that is thrown:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at sun.util.resources.TimeZoneNames.getContents(Unknown Source)
    at sun.util.resources.OpenListResourceBundle.loadLookup(Unknown Source)
    at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary(Unknown Source)
    at sun.util.resources.OpenListResourceBundle.handleKeySet(Unknown Source)
    at java.util.ResourceBundle.containsKey(Unknown Source)
    at sun.util.locale.provider.LocaleResources.getTimeZoneNames(Unknown Source)
    at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayNameArray(Unknown Source)
    at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayName(Unknown Source)
    at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getName(Unknown Source)
    at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(Unknown Source)
    at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(Unknown Source)
    at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(Unknown Source)
    at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(Unknown Source)
    at sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayName(Unknown Source)
    at java.util.TimeZone.getDisplayName(Unknown Source)
    at java.util.Date.toString(Unknown Source)
    at java.lang.String.valueOf(Unknown Source)
    at FileMerger.writeOutputFile(FileMerger.java:95)
    at FileMerger.main(FileMerger.java:26)

Increasing the heap space works for me, but is not a really satisfying solution in my opinion. In a few days I will have to write files that are twice as big; hope the space is enough for them too.

I'm running Java 1.8.0_45 on Windows. The code above is the real code, and the only code running. For testing purposes I use the following code for filling the TreeMap:

int accesses = 43267;

for (int i = 0; i < 1785530; i++) {
    Date date = new Date(i);
    accessesPerSecondMap.put(date, accesses);
}

I've added the following to the for-loop to track the memory usage:

...
    int count = 100000;
    for (Date date : accessesPerSecondMap.keySet()) {

        if (count == 100000) {
            count = 0;
            printOutMemoryUsage();
        }
        count++;
...

private void printOutMemoryUsage() {
    Runtime runtime = Runtime.getRuntime();

    NumberFormat format = NumberFormat.getInstance();

    StringBuilder sb = new StringBuilder();
    long maxMemory = runtime.maxMemory();
    long allocatedMemory = runtime.totalMemory();
    long freeMemory = runtime.freeMemory();

    sb.append("free memory: " + format.format(freeMemory / 1024) + " ");
    sb.append("allocated memory: " + format.format(allocatedMemory / 1024)
            + " ");
    sb.append("max memory: " + format.format(maxMemory / 1024) + " ");
    sb.append("total free memory: "
            + format.format((freeMemory + (maxMemory - allocatedMemory)) / 1024)
            + "<br/>");
    System.out.println(sb);
}

I am getting this output when I don't increase the heap space:

free memory: 28.868 allocated memory: 156.792 max memory: 253.440 total free memory: 125.516
free memory: 89.847 allocated memory: 253.440 max memory: 253.440 total free memory: 89.847
free memory: 87.796 allocated memory: 253.440 max memory: 253.440 total free memory: 87.796
free memory: 89.758 allocated memory: 253.440 max memory: 253.440 total free memory: 89.758
free memory: 32.478 allocated memory: 253.440 max memory: 253.440 total free memory: 32.478
free memory: 35.182 allocated memory: 253.440 max memory: 253.440 total free memory: 35.182
free memory: 37.269 allocated memory: 253.440 max memory: 253.440 total free memory: 37.269
free memory: 45.165 allocated memory: 253.440 max memory: 253.440 total free memory: 45.165
free memory: 42.943 allocated memory: 253.440 max memory: 253.440 total free memory: 42.943
free memory: 32.055 allocated memory: 253.440 max memory: 253.440 total free memory: 32.055
free memory: 13.053 allocated memory: 253.440 max memory: 253.440 total free memory: 13.053
free memory: 14.281 allocated memory: 253.440 max memory: 253.440 total free memory: 14.281
free memory: 12.797 allocated memory: 253.440 max memory: 253.440 total free memory: 12.797
free memory: 1.973 allocated memory: 253.440 max memory: 253.440 total free memory: 1.973
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

With increased heapspace -Xms512m -Xmx1230m i get:

free memory: 372.630 allocated memory: 506.816 max memory: 1.217.536 total free memory: 1.083.350
free memory: 329.031 allocated memory: 506.816 max memory: 1.217.536 total free memory: 1.039.751
free memory: 273.121 allocated memory: 506.816 max memory: 1.217.536 total free memory: 983.841
free memory: 333.700 allocated memory: 506.816 max memory: 1.217.536 total free memory: 1.044.420
free memory: 276.392 allocated memory: 506.816 max memory: 1.217.536 total free memory: 987.112
free memory: 220.482 allocated memory: 506.816 max memory: 1.217.536 total free memory: 931.202
free memory: 279.896 allocated memory: 506.816 max memory: 1.217.536 total free memory: 990.616
free memory: 223.986 allocated memory: 506.816 max memory: 1.217.536 total free memory: 934.706
free memory: 284.565 allocated memory: 506.816 max memory: 1.217.536 total free memory: 995.285
free memory: 228.654 allocated memory: 506.816 max memory: 1.217.536 total free memory: 939.374
free memory: 169.949 allocated memory: 506.816 max memory: 1.217.536 total free memory: 880.669
free memory: 230.528 allocated memory: 506.816 max memory: 1.217.536 total free memory: 941.248
free memory: 174.618 allocated memory: 506.816 max memory: 1.217.536 total free memory: 885.338
free memory: 235.197 allocated memory: 506.816 max memory: 1.217.536 total free memory: 945.917
free memory: 179.287 allocated memory: 506.816 max memory: 1.217.536 total free memory: 890.007
free memory: 123.376 allocated memory: 506.816 max memory: 1.217.536 total free memory: 834.096
free memory: 183.956 allocated memory: 506.816 max memory: 1.217.536 total free memory: 894.676
free memory: 128.046 allocated memory: 506.816 max memory: 1.217.536 total free memory: 838.766
FileMerger main method finished

Sulphathiazole answered 27/5, 2015 at 19:15 Comment(9)

You can increase Heap space from Eclipse very easily – Architectural 27/5, 2015 at 19:18

Have you tried flushing periodically? – Muddle 27/5, 2015 at 19:24

You can try to fix this by increase Java default heap size, but you may still get this error.So you should try to implement your program such a way so that it is capable to prevent memory leak and does not throw OutOfMemoryError. – Lehr 27/5, 2015 at 19:31

@Arin: I agree with you, but I do not know how to achieve this. – Sulphathiazole 28/5, 2015 at 7:21

I see absolutely nothing in the "duplicate" question or any of its answers related to the problem in this question. If any of the duplicate voters know of an answer that's the same for both of these questions, please post it instead of keeping it to yourselves... – Stoltz 28/5, 2015 at 13:47

(To be more specific, that other question's answers talk about time efficiency, reducing system calls and writes to disk. In this question, there is a problem which causes heap memory to run out, as seen in the OP's question title, their words in the question, the posted stack trace, and the question tags. So if the answers are the same, this cannot be seen from reading the answers on the linked question.) – Stoltz 28/5, 2015 at 13:53

I agree. I believe this should not be marked as a duplicate as this specifically asks about the use of heap space versus a different between two writers. – Tuneberg 29/5, 2015 at 15:49

@Sulphathiazole just to make sure, this is the real code, and the only code running when the error occurs, right? When I tried running your code, it didn't cause increasing memory usage for me. I wonder if your TreeMap of dates just happens to take up almost all of your JVM's memory. You should try, before your output loop, running garbage collection and checking how much free memory your JVM has. See https://mcmap.net/q/194610/-how-do-i-check-cpu-and-memory-usage-in-java/3004881 – Stoltz 2/6, 2015 at 15:22

Also, what version of the JVM are you running? The stack trace shows resource bundle processing occurring (loadLookup()) that I'd think wouldn't need to happen in the middle of the program. I wonder if you could fix or get around this problem by (a) using a newer JVM, (b) storing strings or millisecond Longs in your TreeMap instead of Dates, and/or (c) printing time stamps using a custom output format (SimpleDateFormat) instead of using the default format. (And even if BufferedWriter has nothing to do with your problem, it's a good idea for your program, if you didn't realize yet.) – Stoltz 2/6, 2015 at 15:32

In eclipse...on the toolbar click Run->Run Configuration find the Name of the class you have been running, select it, click the Arguments tab then add:

-Xms512M -Xmx1524M // This may be XS512M or Xx1524M.

to the Arguments section. This will allow your larger files to run more smoothly with the larger heap space as @AliGajani mentioned in the comments.

Tuneberg answered 27/5, 2015 at 19:22 Comment(5)

then I get Error occurred during initialization of VM Could not reserve enough space for 1560576KB object heap – Sulphathiazole 27/5, 2015 at 19:31

I suggest doing a combination of my answer and @Sumption 's answer. Write for the maximum allowance with the new heap size, and then close the writer. Then reopen and do it again. – Tuneberg 27/5, 2015 at 19:32

If your problem really is located in the area of having not enough memory for the JVM try a 64bit JVM, it will allow you to use more than the 1,5GB of memory you have here (if your computer and os architecture support this) – Beaver 27/5, 2015 at 19:34

If the OP or anyone who looks at this post for future reference doesn't have a 64bit JVM (Unlikely I know), what should they do? I know the answer but I suggest you post that as an answer. – Tuneberg 27/5, 2015 at 19:36

probably u r using 32 bit OS system, try with little bit less (around 1.4 GB). It should work. – Lehr 27/5, 2015 at 19:38

There are generally two ways to get rid of this error

java.lang.OutOfMemoryError: Java heap space

Allow the JVM to use more memory

With the -Xmx JVM argument, you can set the heap size. Use java -Xms -Xmx on the command line.

Use Eclipse Memory Analyzer Eclipse memory analyzer is a tool from eclipse foundation to analyze java heap dump. It helps to find classloader leaks and memory leaks and helps to minimize memory consumption.you can use MAT to analyze heap dump carrying millions of object and it also helps you to extract suspect of memory leak.

Felicio answered 27/5, 2015 at 19:24 Comment(2)

This only patches over the problem - BufferedWriter is needed to keep memory usage under control. – Breechcloth 28/5, 2015 at 0:34

@KenY-N could you explain what the relationship between BufferedWriter and controlling memory usage is? I can't find anything in the linked question or anywhere else. – Stoltz 29/5, 2015 at 20:57

Try to write for example first 10000 records. Close writer, then create it again and try to append another 10000 records and so on.
Try to use StringBuilder to build one single string per loop iteration and invoke write() one time in a loop iteration.

Sumption answered 27/5, 2015 at 19:23 Comment(0)

In eclipse...on the toolbar click Run->Run Configuration find the Name of the class you have been running, select it, click the Arguments tab then add:

-Xms512M -Xmx1524M // This may be XS512M or Xx1524M.

to the Arguments section. This will allow your larger files to run more smoothly with the larger heap space as @AliGajani mentioned in the comments.

Tuneberg answered 27/5, 2015 at 19:22 Comment(5)

then I get Error occurred during initialization of VM Could not reserve enough space for 1560576KB object heap – Sulphathiazole 27/5, 2015 at 19:31

probably u r using 32 bit OS system, try with little bit less (around 1.4 GB). It should work. – Lehr 27/5, 2015 at 19:38

Try by putting writer.flush(); inside the for loop. By doing this I believe your buffer will be free instead of getting heavy.

Egyptology answered 27/5, 2015 at 19:33 Comment(0)

when you write the data to file, the writer will hold the reference of the data, so the GC cannot recycle the memory. so you should writer.reset() to release

Bjorn answered 23/10, 2018 at 3:28 Comment(0)

Recommended topics

Hot tags