How to set the size of the C stack in R?

Question

I'm trying to use the spread() function from the tidyr package in R on a dataframe that has about three million observations. It's returning the following error message:

Error : C stack usage  26498106 is too close to the limit

When I run Cstack_info(), it tells me

> Cstack_info()
      size    current  direction eval_depth 
   7969177      15272          1          2 

Following the advice in the answer to this question, I've tried increasing the stack size by running ulimit -s 32768 in a terminal window and opening Rstudio from the terminal. When I try this, however, the output of Cstack_info() is unchanged, and when I run my code, I get the same error message. Following another answer to the same earlier question I've tried updating R and Rstudio, also to no avail. What am I doing wrong here?

I am running R 3.3 on mac os x 10.12.2 with 16 GB of memory.

Fluoresce

You should consider carefully if you can't avoid the spread. You probably can and should. – Torosian

Please don't use the rstudio tag unless you have a specific question about rstudio. R and rstudio are independent pieces of software. – Serotherapy

Thanks @Roland, I've writen my code differently to avoid the spread, though it reads clunkily. Is there any reason to avoid spread in general? coming from Stata I find it convenient to reshape my data like that as I create new variables. – Fluoresce

R data.frames are optimized for many rows with few columns. Most R functions expect "long format" data.frames. – Torosian

Answer 1

I encountered a similar situation, and in my case, I could increase the size of C Stack by setting the launchctl limit.

Before setting the launchctl limit:

> Cstack_info()
      size    current  direction eval_depth 
   7969177      17072          1         2

$ ulimit -s
stack size              (kbytes, -s) 8192

$ sudo launchctl limit
stack       8388608        67104768 

Setting the launchctl limit:

$ cd /Library/LaunchDaemons/
$ sudo vi limit.stack.plist # create a new plist file to set new stack value

<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>limit.maxfiles</string>
    <key>ProgramArguments</key>
    <array>
      <string>launchctl</string>
      <string>limit</string>
      <string>stack</string>
      <string>67104768</string>
      <string>67104768</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>ServiceIPC</key>
    <false/>
  </dict>
</plist>

After rebooting:

> Cstack_info()
      size    current  direction eval_depth 
   63749529      17072          1          2

$ ulimit -s
stack size              (kbytes, -s) 65532

$ sudo launchctl limit
stack       67104768       67104768  

---

R version 3.5.0
macOS High Sierra 10.13.5

Cyte