io_submit() blocks until a previous operation will be completed

Question

I'm using the Linux kernel AIO through libaio, and I have to submit the next reading operation before the previous one was completed. The problem is that io_submit() blocks for some time and, as I can deduce from the interval, it waits the previous operation to be completed.

I know that I can enqueue a several operations with a single io_submit(), but it is not an option for me, because I don't know how exactly the next read operation would like when it's already a time to submit the first one.

Is it working like that only for me, or for everyone? In the second case, may I ask if I'm looking for something feasible, or I have to fallback to a threaded model?

Emeric

io_submit will block under weird, unpredictable conditions (which is why I'm not using it). I once asked around and got an answer like "of course, it has to work that way, there is a limited-size request queue". Happens that e.g. doing one large request may be broken into several smaller ones, so... queue full and blocks. The threaded glibc async I/O implementation works better but be aware that it will spawn threads on demand (if that is not acceptable, use your own pool of workers). – Torrez

Also note that if you don't turn off buffering, kernel aio will run synchronously anyway. A patch (by some Indian guy) which would allow for buffered async I/O has been around for a decade or so, but was turned down on the base "nobody needs that". – Torrez

In my use of it, I've seen it blocking for tens of microseconds, occasional blip in the 100-200us range, but nothing much higher. Still, it was enough to put it in it's own thread. If yours is blocking longer than that, make sure you opened with O_DIRECT and that you are using 512-byte aligned memory buffers (and lengths). Also, if you're doing it to a file on a filesystem, make sure the filesystem supports it. It's tricky! – Insincere

I'm using ext4, which is supporting it. And did you tried to do a second io_submit() while the first operation is not completed? – Emeric

@Minoru, yes, I am submitting I/O's concurrently with io_submit(), though not on ext4. – Insincere

@gubblebozer, may I ask you to share some of your code with me? Just to make sure that I'm doing everything right. – Emeric

@Minoru I can't share this piece of code. Plus, it's actually quite a lot of code surrounding how the structures are built up. I can, however, take a look at yours -- that is, if you can share it. Unfortunately, the io_submit interface tends to fail in exactly the way you're experiencing. It "works", but the I/O's block. – Insincere

I just stumbled across this again after 2 years and noticed mention of "ext4". Note that ext4 does not really support aio at all (or the other way around, aio doesn't support ext4). Even if you do everything "correctly" (O_DIRECT, which is a major anti-optimization, and aligned reads), you may still have io_submit block. If, for whatever reason, filesystem metadata needs be read, this will block inside io_submit, and you're out of luck. Sadly, blocking and waiting for one seek is about the same as waiting for several megabytes transferred anyway, so this renders aio kinda pointless. – Torrez

I believe if the underlying (block) device driver (and/or hardware) queues are full - the submission can/will block until there are resources available. – Asthenopia

Answer 1

Frustratingly there are lots of reasons why io_submit may block including:

• You're doing buffered I/O
• You're doing I/O into a filesystem and your submission is queued behind a synchronous operation.

It's known that ext4 and AIO may not be the best mix:

Blocking during io_submit on ext4, on buffered operations, network access, pipes, etc. [...] AIO access to a file on a filesystem like ext4 is partially supported: if a metadata read is required to look up the data block (ie if the metadata is not already in memory), then the io_submit call will block on the metadata read. Certain types of file-enlarging writes are completely unsupported and block for the entire duration of the operation.

(extract is from a document that was called AIOUserGuide)

See the asynchronous IO io_submit latency in Ubuntu Linux question for other detailed answers.

Holliman