scbl exception Heap exhausted during garbage collection

Question

When our application run for some time, for example , run for hours, the sbcl will throw heap exhausted exception.

Heap exhausted during garbage collection: 1968 bytes available, 2128 requested.
 Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB   LUB  !move  Alloc  Waste   Trig    WP  GCs Mem-age
   0:     0     0     0     0     0     0     0     0     0        0     0  5368709    0   0  0.0000
   1:     0     0     0     0     0     0     0     0     0        0     0  5368709    0   0  0.0000
   2:     0     0     0     0     0     0     0     0     0        0     0  5368709    0   0  0.0000
   3: 101912 101913     0     0 19362 20536     0     0     0 162867456 554752 102714709    0   1  1.4405
   4: 130984 131071     0     0 29240 18868     0     0    25 191196152 5854216 128537781 14785   1  0.6442
   5: 75511 81013     0     0 16567 17127    92    99    36 132974568 5818392  2000000 16565   0  0.0000
   6:     0     0     0     0  7949  1232     0     0     0 37605376     0  2000000 7766   0  0.0000
   Total bytes allocated    = 524643552
   Dynamic-space-size bytes = 536870912
GC control variables:
   *GC-INHIBIT* = true
   *GC-PENDING* = true
   *STOP-FOR-GC-PENDING* = false
fatal error encountered in SBCL pid 3281(tid 3067845440):
Heap exhausted, game over.

Welcome to LDB, a low-level debugger for the Lisp runtime environment.
ldb> 

Any suggestion?

Hiro

Suggestion: do not exhaust your heap. It seems that you have some memory leak, i. e. are holding on to things so that they cannot be garbage collected. – Fishgig

I occasionally ran into the same problem and it was not deterministic thus I was (yet) unable to file a bugreport or find the error on my part. But the common pattern I encountered was that I allocated a LOT of memory for short-term usage. As SBCL uses [generational garbage collection]{sbcl.org/manual/#History-and-Implementation-of-SBCL} this might be due to bad clearing of higher generations. Thus you might want to force shortliving functions with high mem usage into seperate threads as this solved the problem for me as after a thread dies the mem will be freed. – Firstling

another solution might be sbcl --dynamic-space-size <putahighnumberhere> – Firstling

Take a look at this article, this will help you blo.udoidio.info/2008/10/out-of-memory-sad-case.html – Chromatophore

Answer 1

SBCL does not allow you to allocate more than (sb-ext:dynamic-space-size) bytes on the heap. Here you have a 512MB default size (536870912 bytes) and the Lisp program already was using nearly that amount when it attempted to make another allocation.

You could double the amount of heap space available to 1024MB by starting SBCL with --dynamic-space-size 1024. However, as several comments point out, there may be a memory leak, where objects are referenced somehow proportional to the time that the system has been running, so this will offer only a temporary respite.

The (room t) standard Common Lisp function call might help debug this, if you call it periodically.

More advanced code like this http://dwim.hu/darcsweb/darcsweb.cgi?r=HEAD%20hu.dwim.debug;a=headblob;f=/source/path-to-root.lisp#l42 which delves into the SB-VM internal map of allocations could shed more light, and SBCL has a statistical profiler, http://www.sbcl.org/manual/#Statistical-Profiler that supports reporting on allocations too.

Isometropia

Can you point to working examples of using SBCL profiler? I don't seem to be able to get any significant output out of it. I am using Fiveam and CLOS and run into similar problem as the asker. – Lavalley

Is it possible to set the dynamic-space-size in the .sbclrc? – Ankara