How to fix problem with max connections exceed?

Question

I executing Lua-function in async mode and I use NIOEventLoops. When I try getting the next event loop and execute Lua-function I have some exception:

com.aerospike.client.AerospikeException$Connection: Error -7 from BB967EF43270008 127.0.0.1 3000: Node BB967EF43270008 127.0.0.1 3000 event loop 5 max connections 100 would be exceeded.
   at com.aerospike.client.cluster.Node.getAsyncConnection(Node.java:657) ~[aerospike-client-4.2.2.jar:?]
   at com.aerospike.client.async.NioCommand.executeCommand(NioCommand.java:184) [aerospike-client-4.2.2.jar:?]
   at com.aerospike.client.async.NioCommand.run(NioCommand.java:146) [aerospike-client-4.2.2.jar:?]
   at com.aerospike.client.async.NioEventLoop.registerCommands(NioEventLoop.java:211) [aerospike-client-4.2.2.jar:?]
   at com.aerospike.client.async.NioEventLoop.runCommands(NioEventLoop.java:173) [aerospike-client-4.2.2.jar:?]
   at com.aerospike.client.async.NioEventLoop.run(NioEventLoop.java:156) [aerospike-client-4.2.2.jar:?]
   at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

That is my method for getting the next event loop:

 private synchronized EventLoop getNextEventLoop() {
        EventLoop next;
        do {
            next = eventLoops.next();
        } while (next.getProcessSize() >= maxConnectionsPerEventLoop);
        return next;
    }

That is how I executing my Lua-function:

as.execute(getNextEventLoop()
                , new ExecuteListener() {
                    @Override
                    public void onSuccess(Key key, Object obj) {

                       ...
                    }

                    @Override
                    public void onFailure(AerospikeException exception) {
                       ...
                    }
                }
                , writePolicy
                , key
                , "lua-pack"
                , "funcName"
                , Value.get(binName), Value.get(value)
        );

How can I avoid this exception?

Cotillion

Answer 1

You are hitting the ClientPolicy.maxConnsPerNode limit. For 500 maxConnsPerNode and 5 event loops, each event loop can use max 100 (500/5) connections for each node. I assume your maxConnectionsPerEventLoop value is >= 100. Also, it's not guaranteed that the slot will still be available when the next command is sent to the event loop after getProcessSize() is called.

The solution is to limit maxCommandsInProcess on each event loop. This is the thread-safe way to limit inflight commands (and thus connections).

EventPolicy eventPolicy = new EventPolicy();
eventPolicy.maxCommandsInProcess = 100;  // max 100 commands per event loop.
eventPolicy.maxCommandsInQueue = 0;      // unbounded queue

int eventLoopSize = 5;
EventLoops eventLoops = new NioEventLoops(eventPolicy, eventLoopSize);

ClientPolicy clientPolicy = new ClientPolicy();
clientPolicy.eventLoops = eventLoops;
clientPolicy.maxConnsPerNode = eventPolicy.maxCommandsInProcess * eventLoopSize;

The async command queue is used for commands when maxCommandsInProcess limit is reached. If the rate of incoming commands consistently exceeds the rate at which commands are processed, then your application can potentially run out of memory. To avoid this case, you may need to limit your queue size.

eventPolicy.maxCommandsInQueue = 10000;

If this queue size is reached, the AerospikeException.AsyncQueueFull exception is thrown. Your application should respond by delaying commands in a non-event loop thread until eventLoop.getQueueSize() is sufficiently low.

Preconize