How do I deal with Lua libraries that don't coroutine.yield()?

Question

I want to download a large file and concurrently handle other things.

However, luasocket.http never calls coroutine.yield(). Everything else freezes while the file downloads.

Here's an illustrative example, in which I try to simultaneously download a file and print some numbers:

local http = require'socket.http'

local downloadRoutine = coroutine.create(function ()
    print 'Downloading large file'
    -- Download an example file
    local url = 'http://ipv4.download.thinkbroadband.com/5MB.zip'
    local result, status = http.request(url)
    print('FINISHED download ('..status..', '..#result..'bytes)')
end)

local printRoutine = coroutine.create(function ()
    -- Print some numbers
    for i=1,10 do
        print(i)
        coroutine.yield()
    end
    print 'FINISHED printing numbers'
end)

repeat
    local printActive = coroutine.resume(printRoutine)
    local downloadActive = coroutine.resume(downloadRoutine)
until not downloadActive and not printActive
print 'Both done!'

Running it produces this:

1
Downloading large file
FINISHED download (200, 5242880bytes)
2
3
4
5
6
7
8
9
10
FINISHED printing numbers
Both done!

As you can see, printRoutine is resumed first. It prints the number 1 and yields. The downloadRoutine is then resumed, which downloads the entire file, without yielding. Only then are the rest of the numbers printed.

I don't want to write my own socket library! What can I do?

Edit (later the same day): Some MUSH users have also noticed. They provide helpful ideas.

Gregoor

Coroutines are not threads. You should not treat them like threads. If a process doesn't want to yield, then it's not going to, nor can you force it to yield. LuaSocket has some facilities for non-blocking IO, but I'm not very familiar with LuaSocket, so you'll have to investigate them. – Skywriting

LuaSocket supports asynchronous (i.e. non-blocking) operations. RTFM before rewriting the library. – Derril

Mud: Yep, the raw socket does. But socket.http does not. ( See mail-archive.com/[email protected]/msg04969.html .) – Gregoor

Answer 1

I don't see why you can't use PiL advice or copas library (this is almost the same answer as is given here).

Copas wraps the socket interface (not socket.http), but you can use low level interface to get what you need with something like this (not tested):

require("socket")
local conn = socket.tcp()
conn:connect("ipv4.download.thinkbroadband.com", 80)
conn:send("GET /5MB.zip HTTP/1.1\n\n")
local file, err = conn:receive()
print(err or file)
conn:close()

You can then use addthread from copas to give you a non-blocking socket and use step/loop functions to do receive while there is something to receive.

Using copas is less work, while using settimeout(0) directly gives you more control.

Geiger

Thanks you for the helpful alternatives! That solves my specific problem. My question is more general though: Many other libraries, such as LuaSec's ssl.https (non-trivial to re-implement) are still synchronous... – Gregoor

Good point, although you may still be able to do the same thing with ssl.https. In fact, the example I gave is a simplified version of my example with ssl.https (notebook.kulchenko.com/programming/…). Having said that, I'm not sure if ssl.https works the same way with settimeout(0) as I haven't had a chance to test it. – Geiger

Answer 2

Coroutines are not threads; they are cooperative, not simultaneous. When one coroutine yields to/from another, it gets blocked. You can't have two simultaneous execution pointers in vanilla Lua.

However, you can use external libraries to that effect. One of the most popular ones is Lua Lanes.

Stumpf

Rewriting socket.http to yield more often seems easy in comparison. But for working around a larger piece of blocking library code, that would be a great hack! – Gregoor