Lua sockets - Asynchronous Events

Question

In current lua sockets implementation, I see that we have to install a timer that calls back periodically so that we check in a non blocking API to see if we have received anything.

This is all good and well however in UDP case, if the sender has a lot of info being sent, do we risk loosing the data. Say another device sends a 2MB photo via UDP and we check socket receive every 100msec. At 2MBps, the underlying system must store 200Kbits before our call queries the underlying TCP stack.

Is there a way to get an event fired when we receive the data on the particular socket instead of the polling we have to do now?

Cali

Answer 1

There are a various ways of handling this issue; which one you will select depends on how much work you want to do.^*

But first, you should clarify (to yourself) whether you are dealing with UDP or TCP; there is no "underlying TCP stack" for UDP sockets. Also, UDP is the wrong protocol to use for sending whole data such as a text, or a photo; it is an unreliable protocol so you aren't guaranteed to receive every packet, unless you're using a managed socket library (such as ENet).

Lua51/LuaJIT + LuaSocket

Polling is the only method.

• Blocking: call socket.select with no time argument and wait for the socket to be readable.
• Non-blocking: call socket.select with a timeout argument of 0, and use sock:settimeout(0) on the socket you're reading from.

Then simply call these repeatedly. I would suggest using a coroutine scheduler for the non-blocking version, to allow other parts of the program to continue executing without causing too much delay.

Lua51/LuaJIT + LuaSocket + Lua Lanes (Recommended)

Same as the above method, but the socket exists in another lane (a lightweight Lua state in another thread) made using Lua Lanes (latest source). This allows you to instantly read the data from the socket and into a buffer. Then, you use a linda to send the data to the main thread for processing.

This is probably the best solution to your problem.

I've made a simple example of this, available here. It relies on Lua Lanes 3.4.0 (GitHub repo) and a patched LuaSocket 2.0.2 (source, patch, blog post re' patch)

The results are promising, though you should definitely refactor my example code if you derive from it.

LuaJIT + OS-specific sockets

If you're a little masochistic, you can try implementing a socket library from scratch. LuaJIT's FFI library makes this possible from pure Lua. Lua Lanes would be useful for this as well.

For Windows, I suggest taking a look at William Adam's blog. He's had some very interesting adventures with LuaJIT and Windows development. As for Linux and the rest, look at tutorials for C or the source of LuaSocket and translate them to LuaJIT FFI operations.

(LuaJIT supports callbacks if the API requires it; however, there is a signficant performance cost compared to polling from Lua to C.)

LuaJIT + ENet

ENet is a great library. It provides the perfect mix between TCP and UDP: reliable when desired, unreliable otherwise. It also abstracts operating system specific details, much like LuaSocket does. You can use the Lua API to bind it, or directly access it via LuaJIT's FFI (recommended).

_{^* Pun unintentional.}

Guttapercha

Answer 2

I use lua-ev https://github.com/brimworks/lua-ev for all IO-multiplexing stuff. It is very easy to use fits into Lua (and its function) like a charm. It is either select/poll/epoll or kqueue based and performs very good too.

 local ev = require'ev'
 local loop = ev.Loop.default
 local udp_sock -- your udp socket instance
 udp_sock:settimeout(0) -- make non blocking
 local udp_receive_io = ev.IO.new(function(io,loop)
       local chunk,err = udp_sock:receive(4096)
       if chunk and not err then
           -- process data
       end
    end,udp_sock:getfd(),ev.READ)

 udp_receive_io:start(loop) 
 loop:loop() -- blocks forever

In my opinion Lua+luasocket+lua-ev is just a dream team for building efficient and robust networking applications (for embedded devices/environments). There are more powerful tools out there! But if your resources are limited, Lua is a good choice!

Bassorilievo

Could this be used on iPhone/Android with Gideros? (www.giderosmobile.com) I haven't seen an iPhone/Android port. – Cali

Sorry, I don't know if it is possible to extend Lua modules by "native" C bindings in the world of gideros / corona. lua-ev (which is based on libev) compiles for "any" Unix. – Bassorilievo

Answer 3

Lua is inherently single-threaded; there is no such thing as an "event". There is no way to interrupt executing Lua code. So while you could rig something up that looked like an event, you'd only ever get one if you called a function that polled which events were available.

Generally, if you're trying to use Lua for this kind of low-level work, you're using the wrong tool. You should be using C or something to access this sort of data, then pass it along to Lua when it's ready.

Parkins

I realize Lua is indeed single thread system with system events using callbacks. I was just hoping we would get a call back when there is data available instead of keep looking at it. – Cali

You would still have to transfer control to whoever it was that was doing the callback in order to receive the callback. So whenever you would have done that, just check to see if the data's there. – Parkins

I disagree heavily. Using lua-ev or another event does NOT impose polling! Using Lua + "Some Event Loop" is used highly productively in embedded devices (where node.js is to big) for serious tasks with very good performance and low mem footprint. For some situations this is a perfect match. There are modules around which makes it very easy to work with the OS on a very low-level (github.com/justincormack/ljsyscall). – Bassorilievo

Answer 4

You are probably using a non-blocking select() to "poll" sockets for any new data available. Luasocket doesn't provide any other interface to see if there is new data available (as far as I know), but if you are concerned that it's taking too much time when you are doing this 10 times per second, consider writing a simplified version that only checks one socket you need and avoids creating and throwing away Lua tables. If that's not an option, consider passing nil to select() instead of {} for those lists you don't need to read and pass static tables instead of temporary ones:

local rset = {socket}
... later
...select(rset, nil, 0)

instead of

...select({socket}, {}, 0)

Selfsuggestion