Elixir/Erlang: How to find the source of high CPU usage?

Question

My Elixir app is using about 50% of the CPU, but it really should only be using <1%. I'm trying to figure out what is causing the high CPU usage and I'm having some trouble.

In a remote console, I tried

1. Listing all processes with Process.list
2. Looking at the process info with Process.info
3. Sorting the processes by reduction count
4. Sorting the processes by message queue length

The message queues are all close to 0, but the reduction counts are very high for some processes. The processes with high reduction counts are named

1. :file_server_2
2. ReactPhoenix.ReactIo.Pool
3. :code_server

(1) and (3) are both present in my other apps, so I feel like it must be (2). This is where I'm stuck. How can I go further and figure out why (2) is using so much CPU?

I know that ReactPhoenix uses react-stdio. Looking at top, react-sdtio doesn't use any resources, but the beam does.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        87 53.2  1.2 2822012 99212 ?       Sl   Nov20 580:03 /app/erts-9.1/bin/beam.smp -Bd -- -root /app -progname app/releases/0.0.1/hello.sh -- -home /root -- -noshell -noshell -noinput -boot /app/
root     13873  0.0  0.0   4460   792 ?        Rs   13:54   0:00 /bin/sh -c deps/react_phoenix/node_modules/.bin/react-stdio

I saw in this StackOverflow post that stdin can cause resource issues, but I'm unsure if that applies here. Anyway, any help would be greatly appreciated!

Negligee

Answer 1

Did you try etop?

iex(2)> :etop.start

========================================================================================
 nonode@nohost                                                             14:57:45
 Load:  cpu         0               Memory:  total       26754    binary        143
        procs      51                        processes    8462    code         7201
        runq        0                        atom          292    ets           392

Pid            Name or Initial Func    Time    Reds  Memory    MsgQ Current Function
----------------------------------------------------------------------------------------
<0.6.0>        erl_prim_loader          '-'  458002  109280       0 erl_prim_loader:loop
<0.38.0>       code_server              '-'  130576  196984       0 code_server:loop/1  
<0.33.0>       application_controll     '-'   58731  831632       0 gen_server:loop/7   
<0.88.0>       etop_server              '-'   58723  109472       0 etop:data_handler/2 
<0.53.0>       group:server/3           '-'   19364 2917928       0 group:server_loop/3 
<0.61.0>       disk_log:init/2          '-'   16246  318352       0 disk_log:loop/1     
<0.46.0>       file_server_2            '-'    3838   18752       0 gen_server:loop/7   
<0.51.0>       user_drv                 '-'    3720   13832       0 user_drv:server_loop
<0.0.0>        init                     '-'    2559   34440       0 init:loop/1         
<0.37.0>       kernel_sup               '-'    2093   58600       0 gen_server:loop/7   
========================================================================================

http://erlang.org/doc/man/etop.html

Kendricks

What would I do to make it available in prod? Attaching to iex and running :etop.start results in “** (UndefinedFunctionError) function :etop.start/0 is undefined (module :etop is not available)”. – Lexicology

According to the doc linked, you can specify the node you wish to connect etop to : etop -node testnode@myhost -setcookie MyCookie – Harbot

@Harbot if this was an answer to my question, the problem is, as specified by the error message, “module etop in not available.” I doubt specifying the node would magically bring this module to avail. – Lexicology

Surely you should have the script on your machine ? erl -s etop – Harbot

@mudasobwa does including runtime_tools fix this? See tkowal.wordpress.com/2016/04/23/…. – Profundity

@Profundity I have runtime_tools included as extra_applications. – Lexicology

@mudasobwa Not only runtime_tools, also observer. – Insobriety

why is cpu 0 under the load section and time empty? I get that locally as well, and what I really want is a sense of cpu usage per process like would get with top in linux – Repellent

Answer 2

I had the chance to use WombatOAM. For me it was useful in finding bottlenecks. There is a free trial option available. Hope you get lucky.

https://www.erlang-solutions.com/capabilities/wombatoam/

Taxeme