How to create many GenServer processes from List and map data stored in them?

Question

In both approaches I stuck on How to map processes by given set of ids or groups and then map stored struct to filter data.

%{group => [users]} implementation.

I realized that groups will be limited in opposite to users, so I've created one process module that uses groups names as keys.

I'm afraid that in future there will be to many users in few groups, so my question is how can I split current UserGroupServer module to keep many separated processes identified by group names? I would like to keep functionality of current module, within init processes by groups list, additionally I don't know how to map each process to get groups by user_id?

Currently I start only one process in Phoenix lib/myapp.ex by including module in children tree list, so I can call UserGroupServer in Channels directly.

defmodule UserGroupServer do
  use GenServer

  ## Client API
  def start_link(opts \\ []) do
   GenServer.start_link(__MODULE__, :ok, opts)
  end

  def update_user_groups_state(server, data) do
    {groups, user_id} = data
    GenServer.call(server, {:clean_groups, user_id}, :infinity)
    users = Enum.map(groups, fn(group) ->
      GenServer.call(server, {:add_user_group, group, user_id}, :infinity)
    end)
    Enum.count(Enum.uniq(List.flatten(users)))
  end

  def get_user_groups(server, user_id) do
    GenServer.call(server, {:get_user_groups, user_id})
  end

  def users_count_in_gorup(server, group) do
    GenServer.call(server, {:users_count_in_gorup, group})
  end

  ## Callbacks (Server API)

  def init(_) do
    {:ok, Map.new}
  end

  def handle_call({:clean_groups, user_id}, _from, user_group_dict) do
    user_group_dict = user_group_dict
    |> Enum.map(fn({gr, users}) -> {gr, List.delete(users, user_id)} end)
    |> Enum.into(%{})
    {:reply, user_group_dict, user_group_dict}
  end

  def handle_call({:add_user_group, group, user_id}, _from, user_group_dict) do
    user_group_dict = if Map.has_key?(user_group_dict, group) do
      Map.update!(user_group_dict, group, fn(users) -> [user_id | users] end)
    else
      Map.put(user_group_dict, group, [user_id])
    end
    {:reply, Map.fetch(user_group_dict, group), user_group_dict}
  end
end

test:

defmodule MyappUserGroupServerTest do
  use ExUnit.Case, async: false

  setup do
    {:ok, server_pid} = UserGroupServer.start_link
    {:ok, server_pid: server_pid}
  end

  test "add users", context do
    c1 = UserGroupServer.update_user_groups_state(context[:server_pid], {[:a, :b, :c], 1})
    assert(1 == c1)
    c2 = UserGroupServer.update_user_groups_state(context[:server_pid], {[:c, :d], 2})
    assert(2 == c2)
    c3 = UserGroupServer.update_user_groups_state(context[:server_pid], {[:x], 2})
    assert(1 == c3)
    c4 = UserGroupServer.update_user_groups_state(context[:server_pid], {[:d], 1})
    assert(1 == c4)
    c5 = UserGroupServer.update_user_groups_state(context[:server_pid], {[:d, :c], 2})
    assert(2 == c5)
  end
end

Old approach %{user => [groups]}

Monitor stores groups list assigned to user_id. How to find users that are in given group? Have I to create separate processes that will handle m..n relation between groups and user ids? What should I change to get each user groups and then Map them?

Server implementation:

defmodule Myapp.Monitor do
  use GenServer

  def create(user_id) do
    case GenServer.whereis(ref(user_id)) do
      nil -> Myapp.Supervisor.start_child(user_id)
    end
  end

  def start_link(user_id) do
    GenServer.start_link(__MODULE__, [], name: ref(user_id))
  end

  def set_groups(user_pid, groups) do
    try_call user_pid, {:set_groups, groups}
  end

  def handle_call({:set_groups, groups}, _from, state) do
    { :reply, groups, groups } # reset user groups on each set_group call.
  end

  defp ref(user_id) do
    {:global, {:user, user_id}}
  end

  defp try_call(user_id, call_function) do
    case GenServer.whereis(ref(user_id)) do
      nil -> {:error, :invalid_user}
      user_pid -> GenServer.call(user_pid, call_function)
    end
  end
end

Supervisor:

defmodule Myapp.Supervisor do
  use Supervisor

  def start_link do
    Supervisor.start_link(__MODULE__, :ok, name: __MODULE__)
  end

  def start_child(user_id) do
    Supervisor.start_child(__MODULE__, [user_id])
  end

  def init(:ok) do
    supervise([worker(Myapp.Monitor, [], restart: :temporary)], strategy: :simple_one_for_one)
  end
end

Example:

Monitor.create(5)
Monitor.set_groups(5, ['a', 'b', 'c'])
Monitor.create(6)
Monitor.set_groups(6, ['a', 'b'])
Monitor.set_groups(6, ['a', 'c'])

# Monitor.users_in_gorup('a') # -> 2
# Monitor.users_in_gorup('b') # -> 1
# Monitor.users_in_gorup('c') # -> 2

# or eventually more desired:
# Monitor.unique_users_in_groups(['a', 'b', 'c']) # -> 2
# or return in set_groups unique_users_in_groups result

Badgett

Answer 1

Before jumping to processes and gen_servers you always need to think about data structure.

How are you going to add the data? How often? How are you going to query it? How often?

In your example you mention three operations:

• set groups for user (resetting all previously set groups)
• return all users in group
• return unique users in set of groups

Using the most basic types in Elixir (lists and maps) you can arrange your data in two ways:

• map where key is user and value is list of groups (%{user => [groups]})
• or the other way round (%{group => [users]})

For those two implementations you can asses how fast are the operations. For %{user => [groups]}:

• set groups for user is O(1) (just update the key in map)
• return all users in group O(n*m) where n is the number of users and m is number of groups (for all n users you need to check if it is in the group by scanning potentially m group names)
• unique users in group is the same as above + sorting and deduplicating

For the implementation with %{group => [users]}:

• set groups for user is O(n*m) (you need to scan all groups if the user is there, delete it and then set it only for the new ones) If set groups only added user to new groups without deleting it first, it would just add the user in time proportional to number of groups in the input (not all groups)
• return all users in group O(1) - just query the map
• proportional to number of groups in query + sorting and deduplicating

This shows that first implementation is much better if your monitor is updated rapidly and queried less often. Second one is much better if you are updating it less frequently, but query it all the time.

After you implement one of those solutions without any actors or gen_server and can tell that it works you might want to treat pids as map keys and rewrite the algorithms. You may also consider using only one process to store all the data. It also depends on your exact problem. Good luck!

Nightingale

Thanks for answer, it seems that for my use case %{user => [groups]} would fit better, groups will be changed almost on each server request. Initially I would like to focus only on users_in_gorup feature. The only problem is that I can't find out how to manage this issue in processes or how to map groups outside user reference in my Monitor, it's my first GenServer. – Badgett

Instead of iterating over map keys you would have to iterate over PIDs. The hard part is keeping track of them. There is an entire talk about discovering processes in Elixir: elixirconf.eu/elixirconf2016/sasa-juric I would simply store all this state in one gen_server for now and only split it if it proves to be a bottleneck. – Nightingale

As I said, you are going to iterate over processes which will involve potentially a lot of message passing. It is good idea to make computation local to the data and keep entire data structure in one process. This also eliminates the problem of storing PIDs. – Nightingale