Graceful shutdown of GenServer

Question

I writing an Elixir app with GenServer that starts an external application on boot and shuts it down and does other clean-up on exit. I've added bootup functionality in the init/1 callback and cleanup code in the terminate/2 callback.

The init code works fine when the GenServer is started, and the terminate method is also called when the :stop signal is manually sent, but in the cases of unexpected shutdowns and interrupts (as in the case of hitting Ctrl+C) in IEx, the terminate code is not called.

---

Currently, I've gone over tons of forum threads, blog posts and documentation, including:

• Getting Started: GenServers
• Elixir-Lang-Talk: Graceful shutdown of GenServer(s) on exiting iex -S mix
• Elixir-Lang-Talk: Stopping Genserver vs Process.exit

From Elixir Docs - GenServers:

If the GenServer receives an exit signal (that is not :normal) from any process when it is not trapping exits it will exit abruptly with the same reason and so not call terminate/2. Note that a process does NOT trap exits by default and an exit signal is sent when a linked process exits or its node is disconnected.

Therefore it is not guaranteed that terminate/2 is called when a GenServer exits. For such reasons, we usually recommend important clean-up rules to happen in separated processes either by use of monitoring or by links themselves.

but I have absolutely no idea how to get :init.stop, linked processes or anything else to work with this (since this is my first time with GenServers).

---

This is my code:

defmodule MyAwesomeApp do
  use GenServer

  def start do
    GenServer.start_link(__MODULE__, nil)
  end

  def init(state) do
    # Do Bootup stuff

    IO.puts "Starting: #{inspect(state)}"
    {:ok, state}
  end

  def terminate(reason, state) do
    # Do Shutdown Stuff

    IO.puts "Going Down: #{inspect(state)}"
    :normal
  end
end

MyAwesomeApp.start

Machismo

Answer 1

To increase chances of the terminate callback being invoked, the server process should trap exits. However, even with that, the callback might not be invoked in some situations (e.g. when the process is brutally killed, or when it crashes itself). For more details see here.

As mentioned, if you want to politely shutdown your system, you should invoke :init.stop, which will recursively shutdown the supervision tree causing terminate callbacks to be invoked.

As you noticed, there is no way of catching abrupt BEAM OS process exits from within. It's a self-defining property: the BEAM process terminates suddenly, so it can't run any code (since it terminated) 🙂. Hence, if BEAM is brutally terminated, the callback will not be invoked.

If you unconditionally want to do something when BEAM dies, you need to detect this from another OS process. I'm not sure what's your exact use case, but assuming you have some strong needs for this, then running another BEAM node, on the same (or another) machine, could work here. Then you could have one process on one node monitoring another process on another node, so you can react even if BEAM is brutally killed.

However, your life will be simpler if you don't need to unconditionally run some cleanup logic, so consider whether the code in terminate is a must, or rather a nice-to-have.

Airlee

> As you noticed, there is no way of catching abrupt BEAM OS process exits from within. This is no longer true since OTP 20+: github.com/erlang/otp/pull/1315 – Epigone

Answer 2

I can suggest you two solutions.

The first one is mentioned in docs.

Note that a process does NOT trap exits.

You have to make your gen server process trap exits. To do this:

Process.flag(:trap_exit, true)

This makes your process call terminate/2 upon exit.

But another solution, is to hand over this initialization to the upper supervisor. Then have supervisor pass the external application reference to gen server. But here, you don't have a terminate-like callback to exit external application if necessary. The external application will just be killed, when supervisor stops.

Gazelle

Process.flag(:trap_exit, true) does not work for me. Can you please tell where I'm supposed to call this in a genserver? – Machismo

You should set it in the process, meaning init/1 – Gazelle

Seems like the problem is not with gen server itself. Starting an iex shell, Process.flag(:trap_exit, true) and writing a simple receive to match all, doesn't seem to work either. If your problem is just to shutdown external application, then it's done when gen server stops automatically, but about other clean-ups you mention, maybe tell what your are going to do in more detail. – Gazelle

Answer 3

If your trying to get it to work in iex and Process.flag(:trap_exit, true) is not working, make sure you are using GenServer.start instead of GenServer.start_link otherwise the shell process will crash and the trapping won't matter.

Here's an example:

defmodule Server do

    use GenServer
    require Logger

    def start() do
      GenServer.start(__MODULE__, [], [])
    end


    def init(_) do
      Logger.info "starting"
      Process.flag(:trap_exit, true) # your trap_exit call should be here
      {:ok, :some_state}
    end

    # handle the trapped exit call
    def handle_info({:EXIT, _from, reason}, state) do
      Logger.info "exiting"
      cleanup(reason, state)
      {:stop, reason, state} # see GenServer docs for other return types
    end

    # handle termination
    def terminate(reason, state) do
      Logger.info "terminating"
      cleanup(reason, state)
      state
    end

    defp cleanup(_reason, _state) do
      # Cleanup whatever you need cleaned up
    end

end

In iex you should now see a trapped exit call

iex> {:ok, pid} = Server.start()
iex> Process.exit(pid, :something_bad)

Open

What is the difference handle_info({:EXIT, ... vs terminate/2 ? – Bicorn

According to groups.google.com/g/elixir-lang-talk/c/vxOtIXdqiWw/m/… the handle_info({:EXIT, ... happens first. terminate/2 is a GenServer callback (for you to handle graceful termination) that gets invoked only when a {:stop, ... is returned, as is done in handle_info({:EXIT, ... above. – Neilson