Mnesia: unexpectedly getting aborted, cyclic transactions

Question

I have a 5 processes that insert/update the same 3 records in a mnesia table. Each of these processes does its insert/updates within a single transaction.

I have 5 other process that read these very same 3 records, also within a single transaction.

Unless I lock the entire table as part of the multi-record transaction, I get an {aborted, {cyclic, node....}} error. My intuition is that my use-case is ordinary and should not, in of itself, result in in an aborted transaction. Can someone help me with my bone-headed thinking? All I am doing is inserting (or reading) multiple rows in a cache (mnesia table) in one transaction.

Inserting the 3 records looks like this

insert_keylist(Keys) ->  
    F = fun() -> insert_iter(Keys) end,  
    transactional_execute(F).

insert_iter([]) ->
    ok;
insert_iter([{Key, Value} | KVs]) ->
   insert(Key, Value),
   insert_iter(Kvs).

insert(Key, Value) ->
  F = 
      fun() ->
        case sc_store:lookup(Key) of
            {ok, _Value}       -> sc_store:replace(Key, Value);
            {error, not_found} -> sc_store:insert(Key,Value)
        end
      end,
  transactional_execute(F).    


transactional_execute(F) ->
    case mnesia:is_transaction() of
       true -> F();
       false ->
           try mnesia:sync_transaction(F) of
               {atomic, Result}   -> Result;
               {aborted, Reason}  -> {aborted, Reason}
           catch
                ErrorCl:Error     -> {error, {ErrorCl, Error}}
        end
     end.

sc_store:replace, insert and lookup are is as follows:

replace(Key, Value) ->
    try
       case mnesia:wread({key_to_value, Key}) of
          [#key_to_value{} = Rec] -> 
            mnesia:write(Rec#key_to_value{value = Value});
          []                       -> 
        {error, not_found}
        end
    catch
       _Err:Reason ->
         {error, Reason}
    end.

insert(Key, Value, Type, Scope, TTL, TTLMessage, Ref) ->
   try
      NowDT = calendar:now_to_datetime(erlang:now()),
      ok = mnesia:write(#key_to_value{key = Key, 
                type = Type,
                scope = Scope,
                value = Value,
                create_time_dt = NowDT,
                ttl_secs = TTL,
                ttl_message = TTLMessage,
                ref = Ref})
   catch
      _Error:Reason ->
        {error, Reason}
   end.

lookup(Key) ->
   try 
      case mnesia:read(key_to_value, Key) of
         [#key_to_value{type = Type, scope = Scope, value = Value}] -> 
            {ok, {Value, Type, Scope}};
         []                       -> 
            {error, not_found}
      end
   catch
      _Err:Reason -> {error, Reason}
   end.

Caritacaritas

Answer 1

Actually, turns out the problem was using try/catch around mnesia operations within a transaction. See here for more.

Caritacaritas

Answer 2

Mnesia is right !!
According to your code, the function insert_keylist/1 calls the function transactional_execute/1 with a fun. Inside the function: transactional_execute/1, you ask mnesia wether its already in a transaction, and if its true, you execute the fun F. However, looking closely in fun F, you see that it calls the function: transactional_execute/1 all over again through the iterative function: insert_iter/1. So, this creates a loop of transactions. Hence your problem is arising because a process is executing the top most function: insert_keylist/1 in a transaction which is creating a nested loop of transactions which never get executed, only, they keep asking mnesia wether they are in a transaction and it accepts yet again they keep asking and asking and looping without ever getting executed !!

Before we modify your code, we have not seen whats in the functions: sc_store:replace/2 and sc_store:insert/1. I will assume that these functions are again NOT creating transactions within themselves!! I will assume that they are directly using mnesia functions (That must be executed within a transaction), such as : mnesia:write/1, mnesia:read/1,mnesia:write/3 e.t.c. Otherwise if you were creating another transaction in these functions, surely this would be a mess of mnesia transactions!! Now lets correct the code as shown below:

%% This function will always be sure to be in a 
%% mnesia transaction. So no need to call
%% transactional execute

insert_iter([])-> ok;
insert_iter([{Key, Value} | KVs]) ->
   case sc_store:lookup(Key) of
        {ok, _Value} -> sc_store:replace(Key, Value);
        {error, not_found} -> sc_store:insert(Key,Value)
    end,
    insert_iter(Kvs).

That's the only change required, assuming that the functions sc_store:insert/1 and sc_store:replace/2 access mnesia write and read functions directly without creating other transactions also.

Theiss

So, thanks for the help, but I don't think this is correct. (I tried your suggestion and it didn't work). Transactional_execute/1 should not create nested transactions - it's designed specifically to stop that - and when I trace the code it appears to be calling mnesia:sync_transaction only once. Also, I have no problem when I repeatedly run multiple insert process (code above) by themselves, it's only when I combine it with multiple reading processes (which run nearly identical code except they just return the value of lookup/1 as opposed to replacing/inserting the value. Any other ideas? – Caritacaritas

Actually, I can break it just by running the insert process on its own. I just refactored the code guaranteeing that transactional_execute/1 is only being called once (and verified it via debug session) and it still breaks – Caritacaritas

HOLY CRAP! I think it might be my try ... catch surrounding the sc:store functions. When I remove them everything works.... could it be that I am intercepting mnesia's retries / deadlock avoidance by surrounding those functions in a try/catch? – Caritacaritas

I hope its now good. It must have been a small logical problem some where. actually, i always try to avoid the functions: mnesia:wread which i can see in your sc_store:replace/2. These functions mess up the logic and normally lead to side effects. The functions are okay by themselves but their usage normally is wrong. try changing that part as well. Also, for real-time access, the reading processes can get immediate written values through mnesia events. In this way they do not block the writing processes. I have used this method to track real-time changes to some parameters – Theiss