Change Erlang file handle limit?

Question

I'm running into trouble with an Erlang OTP + Cowboy app that does not allow me to open enough files simultaneously.

How do I change the number of open file handles allowed in the BEAM?

Potentially, I'll need about 500 small text files open at the same time, but it appears that the file limit is 224. I've got the value of 224 from this little test program:

-module(test_fds).
-export([count/0]).

count() -> count(1, []).

count(N, Fds) ->
  case file:open(integer_to_list(N), [write]) of
    {ok, F} ->
      count(N+1, [F| Fds]);

    {error, Err} ->
      [ file:close(F) || F <- Fds ],
      delete(N-1),
      {Err, N}
  end.

delete(0) -> ok;
delete(N) -> 
  case file:delete(integer_to_list(N)) of
    ok         -> ok;
    {error, _} -> meh
  end,

delete(N-1).

This gives

$ erl
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [kernel-poll:false]

Eshell V9.2  (abort with ^G)
1> c(test_fds).     
{ok,test_fds}
2> test_fds:count().
{emfile,224}
3> 

This seems to be an Erlang problem rather than a Mac OSX problem since from the command line, I get:

$ sysctl -h kern.maxfiles
kern.maxfiles: 49,152
$ sysctl -h kern.maxfilesperproc
kern.maxfilesperproc: 24,576

Dangle

What's the output of ulimit -n in your shell? – Phuongphycology

in my shell ulimit returns unlimited, but this is clearly not true! – Dangle

Answer 1

The number of open file descriptors is most likely being limited by your shell. You can increase it by running ulimit -n 1000 (or more) in your shell before invoking erl. On my system, the default value was 7168 and your script could open 7135 files before returning emfile. Here's the output of me running your script with different ulimit values:

$ ulimit -n 64; erl -noshell -eval 'io:format("~p~n", [test_fds:count()]), erlang:halt()'
{emfile,32}
$ ulimit -n 128; erl -noshell -eval 'io:format("~p~n", [test_fds:count()]), erlang:halt()'
{emfile,96}
$ ulimit -n 256; erl -noshell -eval 'io:format("~p~n", [test_fds:count()]), erlang:halt()'
{emfile,224}
$ ulimit -n 512; erl -noshell -eval 'io:format("~p~n", [test_fds:count()]), erlang:halt()'
{emfile,480}

erl is most likely opening 32 file descriptors before it starts evaluating our code which explains the constant difference of 32 in the ulimit and the output.

Phuongphycology

Thanks. That's fixed it! However, on my machine ulimit returns unlimited - which clearly is not true! – Dangle

What does ulimit -n print? My plain ulimit's output is also unlimited. -n is how you print the open file descriptors limit. – Phuongphycology

ulimit -n was set to 256. Thanks – Dangle