LuaLanes and LuaSockets

Question

I'm working on a little Lua app (under Lua for Windows, if that matters) that uses sockets to communicate with the outside world. (LuaSocket)

And I'm trying to make several requests in parallel. So I thought LuaLanes was the way to go. (I'm open to alternatives, of course, if there's a better solution, but would prefer not to deal with coroutines for this.)

Something like this:

server = assert (socket.bind ('*', 1234))
client = server : accept ()
-- set id to some unique value
allClients [id] = client
theLane = lanes.gen ("", laneTest) ( id )
print (theLane [1])

Where the laneTest function is defined as follows:

function laneTest (id)
    local client = allClients [id]
    print ('peer: ', client:getpeername())
end

My problem is that inside the laneTest function, when run as a lane, I get this lovely error message:

attempt to index local 'client' (a userdata value)

(from the line client:getpeername())

So.. I'm not sure what's going on here? Is lanes incompatible with sockets, or am I doing something very wrong?

I guess it could be that the version of lanes that ships with Lua for Windows is ancient (luaforwindows) and doesn't work with sockets, but the latest version might? (Lanes 2.0.4 vs the more recent 3.xx)

I don't really know how to go about updating the version of Lanes I've got, else I would have tried that by now, so. I'd appreciate any advice if that's where I could be heading or there's something more obvious that I've done wrong.

Edit: I went ahead and installed lanes via luarocks, and have the same problem using lanes 3.1.6-1 that's installed as a rock.

Edit 2: Tried this (and still failed):

require ('socket')
require ('lanes')
local allClients = {}

function theLane (id)
    print ('the id:', id) -- correctly prints out the id passed to the function
    local SOCKET = require ('socket')
    local client = allClients [id]
    print ('peer:', client:getpeername())
    client : close ()
end

local server = assert (SOCKET.bind ('*', 1234))
local ip, port = server:getsockname ()
local laneFunc = lanes.gen('', theLane)
local client = server:accept ()
allClients [1] = client
local x = laneFunc (1)
print (x[1])

1. This fails claiming: attempt to call global 'require' (a nil value)
2. Removing the require ('socket') line inside the function and retrying also fails saying: attempt to index local 'client' (a userdata value)

I apologize in advance for missing the obvious, but... how do you get sockets to work with lanes?

Edit 3:

Well, I'm editing this in for future reference :)

As far as I can tell, there's no way of using Lanes with Sockets without patching luasockets. See discussion here for more about that; but in short (and as explained in Deco's answer): lanes does not work with userdata. luasocket does not provide any other way of accessing socket/socket info.

I have no desire to patch luasocket so as much as I would have rather used lanes, I'm going go ahead and stick with copas or couroutines.

Thanks all!

Goring

Answer 1

Programming Lua has example on non-preemptive multithreading (using coroutines), which you can probably use almost directly. In my opinion coroutines will be a better solution for your use case.

There is also copas library that describes itself as "a dispatcher based on coroutines that can be used by TCP/IP servers", but you can actually use it for sending requests asynchronously too (using a combination of addthread and step calls).

Brilliancy

Thanks for the links, I think I'll probably end up working with coroutines (and will be studying the dispatcher model from your example) for that, but if I can get things working with lanes, it seems more.. elegant ? "If" being the important part :P – Goring

Answer 2

Lua Lanes creates an entirely new (but minimal) Lua state for each lane. Any upvalues or arguments passed are copied, not referenced; this means your allClients table is being copied, along with the sockets it contains.

The error is occurring because the sockets are userdata, which Lua Lanes does not know how to copy without some advice from a C module. Unfortunately, LuaSocket offers no such advice. (There are ways around this, but be cautious: LuaSocket is not thread-safe and synchronisation bugs are very difficult to track down.)

Although it will not solve your problem, you should note that you need to require LuaSocket in your spawned lane; it is not copied by default.

Solutions!

These are ordered from easy to hard (and mostly transcribed from my other answer here).

Single-threaded Polling

Repeatedly call polling functions in LuaSocket:

• Blocking: call socket.select with no time argument and wait for the socket to be readable.
• Non-blocking: call socket.select with a timeout argument of 0, and use sock:settimeout(0) on the socket you're reading from.

Simply call these repeatedly. I would suggest using a coroutine scheduler for the non-blocking version, to allow other parts of the program to continue executing without causing too much delay.

(If you go for this solution, I suggest reviewing Paul's answer.)

Dual-threaded Polling

Your main thread does not deal with sockets at all. Instead, it spawns another lane which requires LuaSocket, handles all clients and communicates with the main thread via a linda.

This is probably the most viable option for you.

Multi-threaded Polling

Same as above, except each thread handles a subset of all the clients (1 to 1 is possible, but diminishing return will set in with large quantities of clients).

I've made a simple example of this, available here. It relies on Lua Lanes 3.4.0 (GitHub repo) and a patched LuaSocket 2.0.2 (source, patch, blog post re' patch)

The results are promising, though you should definitely refactor my example code if you derive from it.

LuaJIT + ENet

ENet is a great library. It provides the perfect mix between TCP and UDP: reliable when desired, unreliable otherwise. It also abstracts operating system specific details, much like LuaSocket does. You can use the Lua API to bind it, or directly access it via LuaJIT's FFI (recommended).

Brenna

Deco: FYI, your link in the Solutions paragraph includes /edit in it. – Brilliancy

Thanks for the comprehensive solution :) I'm trying to go for a Dual-threaded or Multi-threaded Polling solution, as you described, but I'm still having trouble getting lanes to work for me. (Also thanks for pointing out that I'll need to require socket inside the lane, I was not aware of that). I'll edit my initial question to post an example of what I'm doing and how it's failing, if you get the time to take a look, maybe I'm missing something? Thanks – Goring