SBCL error: "binding stack exhausted" when running Maxima on Linux machine

Question

I realize there are many places I can ask this question at but I thought I'd try here. I've already seemingly attained as much help as I can from the good people at Maxima.

I run Maxima with SBCL and consistently get errors;

INFO: Binding stack guard page unprotected
Binding stack guard page temporarily disabled: proceed with caution

Maxima encountered a Lisp error:

 Binding stack exhausted.

PROCEED WITH CAUTION.

Automatically continuing.
To enable the Lisp debugger set *debugger-hook* to nil.
INFO: Binding stack guard page reprotected

I've modified the call to Maxima (its executable) by adding larger values for the dynamic space size and control stack size, and I have looked at ./.../sbcl -help for any ideas of arguments to add to the $MAXIMA_LISP_OPTIONS in the maxima executable.

Additionally, I usually do these just before I run it (though I imagine they are unnecessary as the OS is smart, well maybe the last one needs to be fiddled with);

sudo fstrim -v /
echo 3 | sudo tee /proc/sys/vm/drop_caches
echo 262144 | sudo tee /proc/sys/vm/max_map_count

and after a couple calculations while doing my Maxima work, I throw in a couple of

:lisp (sb-ext:gc :full t)

in the hopes of avoiding this error. I don't know lisp that well and certainly don't understand all there is about garbage collection.

My calculations are somewhat intensive and recursive, though I am utilizing memoization in the Maxima work. My computer is described by inxi -b as,

System:    Host: XXX-MacBookPro Kernel: 4.10.0-33-generic x86_64 (64 bit) Desktop: Cinnamon 3.4.6
           Distro: Linux Mint 18.2 Sonya
Machine:   System: Apple (portable) product: MacBookPro11 3 v: 1.0
           Mobo: Apple model: Mac-2BD1B313 v: MacBookPro11 3
           Bios: Apple v: MBP112.88Z.0138.B25.1702171721 date: 02/17/2017
CPU:       Quad core Intel Core i7-4980HQ (-HT-MCP-) speed/max: 1402/4000 MHz
Graphics:  Card: NVIDIA GK107M [GeForce GT 750M Mac Edition]
           Display Server: X.Org 1.18.4 drivers: nvidia (unloaded: fbdev,vesa,nouveau)
           Resolution: [email protected]
           GLX Renderer: GeForce GT 750M/PCIe/SSE2 GLX Version: 4.5.0 NVIDIA 375.66
Network:   Card-1: Broadcom BCM4360 802.11ac Wireless Network Adapter driver: wl
           Card-2: Broadcom NetXtreme BCM57762 Gigabit Ethernet PCIe driver: tg3
Drives:    HDD Total Size: 1000.6GB (17.5% used)
Info:      Processes: 291 Uptime: 43 min Memory: 3366.6/15953.7MB Client: Shell (bash) inxi: 2.2.35 

and my Maxima and SBCL are built from GIT and are quite new ~ about 2 weeks, and from head and have passed all their make tests. Additionally my swap looks like;

XXX@XXX-MacBookPro ~/ResearchWC $ cat /proc/swaps 
Filename                Type        Size    Used    Priority
/70GiB.swap                             file        73400316    0   -2
/dev/sda7                               partition   25564776    0   -1

and I often am basically out of memory and around 20-30G into swap.

Usually it seems to hang eventually (after say 100 hours, as I notice htop ceases showing certain activity and the fan doesn't go up and down) and I think the exhaustion errors are sometimes buried in the embedded recursive calls. I got this error message above because I avoided calling the function at the recursion level I want and instead built them up "by hand" at the terminal. E.G., instead of just calling something like fib(10), I instead called successively fib(1), fib(2), fib(3), where each previous value was memoized.

I have the time but seemingly don't know how to maximize my swap - watching htop I have never seen it use more than say ~25%.

1.) Does anyone know what else I can do with SBCL to avoid these errors?

2.) Would another lisp be a better one to run in these situations, e.g. ecl, cml, etc.?

Thanks in advance for any advice, and I can provide more details if needed.

UPDATE

After upping the dynamic space size , stack size, and the binding stack size, I was up against the heap limit instead of the binding stack this time when it crashed. Attached is the output of backtrace (Not sure what the two registers pc and fp are... - program counter and frame pointer?). I ran this with trace(residue,taylor) as well but never saw anything fishy...

ldb> backtrace
Backtrace:
   0: SB-BIGNUM::MULTIPLY-BIGNUM-AND-FIXNUM, pc = 0x21cb1336, fp = 0x7ffff3943f18
   1: SB-KERNEL::TWO-ARG-*, pc = 0x21cb00a7, fp = 0x7ffff3943f98
   2: MAXIMA::CTIMES, pc = 0x21e076b4, fp = 0x7ffff3943fc0
   3: MAXIMA::PCTIMES, pc = 0x21de5f4c, fp = 0x7ffff3943ff0
   4: MAXIMA::PCTIMES1, pc = 0x21e78f1e, fp = 0x7ffff3944048
   5: MAXIMA::PCTIMES, pc = 0x21de6033, fp = 0x7ffff3944078
   6: MAXIMA::PCETIMES1, pc = 0x21fe0560, fp = 0x7ffff39440d8
   7: MAXIMA::PTIMES1, pc = 0x21f457e5, fp = 0x7ffff3944148
   8: MAXIMA::PTIMES, pc = 0x21db6561, fp = 0x7ffff3944180
   9: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39441b8
  10: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39441f0
  11: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944228
  12: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944260
  13: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944298
  14: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39442d0
  15: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944308
  16: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944340
  17: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944378
  18: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39443b0
  19: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39443e8
  20: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944420
  21: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944458
  22: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944490
  23: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39444c8
  24: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944500
  25: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944538
  26: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944570
  27: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39445a8
  28: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39445e0
  29: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944618
  30: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944650
  31: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944688
  32: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39446c0
  33: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39446f8
  34: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944730
  35: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944768
  36: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39447a0
  37: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39447d8
  38: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944810
  39: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944848
  40: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944880
  41: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39448b8
  42: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39448f0
  43: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944928
  44: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944960
  45: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944998
  46: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39449d0
  47: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944a08
  48: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944a40
  49: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944a78
  50: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944ab0
  51: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944ae8
  52: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944b20
  53: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944b58
  54: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944b90
  55: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944bc8
  56: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944c00
  57: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944c38
  58: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944c70
  59: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944ca8
  60: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944ce0
  61: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944d18
  62: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944d50
  63: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944d88
  64: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944dc0
  65: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944df8
  66: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944e30
  67: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944e68
  68: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944ea0
  69: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944ed8
  70: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944f10
  71: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944f48
  72: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944f80
  73: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3944fb8
  74: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3944ff0
  75: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3945028
  76: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3945060
  77: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3945098
  78: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39450d0
  79: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3945108
  80: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3945140
  81: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3945178
  82: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39451b0
  83: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39451e8
  84: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3945220
  85: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3945258
  86: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3945290
  87: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39452c8
  88: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3945300
  89: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3945338
  90: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3945370
  91: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39453a8
  92: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39453e0
  93: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3945418
  94: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3945450
  95: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff3945488
  96: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff39454c0
  97: MAXIMA::PSIMP, pc = 0x21dd0617, fp = 0x7ffff39454f8
  98: MAXIMA::PALGSIMP, pc = 0x22166db7, fp = 0x7ffff3945530

Caprine

Stack and heap are two different things. 'Binding stack exhausted' would usually point to high recursion depth. A large heap/swap would be caused by memory leaking or a GC which can't free all memory. The best place to ask is the SBCL mailing list. An alternative implementation would be Clozure CL. – Touchback

@RainerJoswig Is there a way to determine the maximum call stack depth in Common Lisp? or if not then specifically for SBCL? I seem to recall that there is a constant which tells the maximum stack depth but a web search doesn't seem to find any such thing. Perhaps it's a bit of wishful thinking on my part. – Extract

The only place I see mention of "maximum call stack depth" is with use in profiling, section 15.2.4 & 15.2.5 at sbcl.org/manual/index.html and a description of binding & unbinding in the internals manual in section 8.2 at sbcl.org/sbcl-internals/… . Quick two question (maybe what you @RobertDodier asked @RainerJoswig) is the binding stack depth controlled by some modifiable variable, and does "control stack size" have any affect on this? I did see a #define in validate.h setting BINDING_STACK_SIZE to 1024*1024... Can I increase it? – Caprine

I'm not sure whether it's possible to increase it. Be that as it may, unless you can determine that there isn't a bug in your program, I don't think there is any purpose to increasing the stack size. My advice is to look first for bugs and then to look for ways to avoid deep recursions. – Extract

Answer 1

The most common cause of bind stack overflow is a recursive function which calls itself indefinitely (i.e. a bug in the recursion). The second most common cause is a recursive function which is programmed correctly, but calls itself too many times for the Lisp implementation to handle it.

I forget what the stack limit is for SBCL -- it might be 8192 but that's just a guess. You could probably determine it by experiment (if not by reading the SBCL documentation).

In either case you can try to figure out which function or functions are causing trouble via

trace (mymaximafun1, mymaximafun2, ...);

for Maxima functions and/or

:lisp (trace mylispfun1 mylispfun2 ...)

for Lisp functions.

About the second problem, you can try to avoid deep recursions by reworking recursive functions as iterations.

You mentioned that you have long-running computations. A strategy to lessen the effect of crashes is to call the save function to save the program state every now and then, e.g.:

save ("mycheckpointfile.lisp", all);

Note that save takes a lot of options, so maybe take a look at the documentation via ? save.

You can automatically generate file names via some recipe such as file_name : printf(false, "mycheckpointfile~d.lisp", 1000 + random(9000)) which generates a random 4-digit number and pastes it into the file name. Of course there are many such recipes.

Extract

Thanks, your trace suggestion hasn't lead to any obvious errors, but still hunting... – Caprine

You might be able to get a stack trace -- if the program goes to an SBCL debugger prompt, try :backtrace. I don't know if that will work. – Extract