Can I await an enumerable I create with a generator?

Question

Let's say I have a sequence of integers I obtain asynchronously.

async Task<int> GetI(int i){
    return await Task.Delay(1000).ContinueWith(x => i);
}

I want to create a generator over that sequence, if the sequence was synchronous I'd do:

IEnumerable<int> Method()
{
    for (var i = 0; i < 100; i++)
    {
        yield return GetI(i); // won't work, since getI returns a task
    }
}

So, I figured the analogy is making the generator async and yielding from it:

async Task<IEnumerable<int>> Method()    
{
    for (var i = 0; i < 100; i++)
    {
        yield return await Task.Delay(1000).ContinueWith(x => i);
    }
}

This won't work, since a method with yield must return an IEnumerable of something, the alternative, which makes more sense is IEnumerable<Task<int>> but that won't compile since async methods must return Tasks or void.

Now, I realize I can simply remove the await and return an IEnumerable<Task<int>> but that won't help me since the iteration will keep asking for data before any of it is ready, so it doesn't solve my issue.

• Is there any way to nicely mix enumerables and tasks with the nice sugar the language gives me with await and yield?
• Is there any way to nicely consume it?

^{(From searching online, I suspect the answer to the first question is false and the second one is an observer/observable, but I couldn't find any canonical reference and I'm interested in the best way to implement this pattern in C#)}

Isoclinal

Are you actually trying to get a Task<IEnumerable<int>>, or an IEnumerable<Task<int>>? The latter is significantly easier to achieve... but it's not clear what you're actually trying to get out of this. – Howland

(I do understand that expressing what you're trying to achieve in this sort of situation can be very difficult, by the way.) – Howland

@JonSkeet On the off chance of sounding stupid, I think I'm trying to get a Task<IEnumerable<Task<int>>>, I want to get an iterator that I'll wait for initially and then wait for at each iteration. However, an IEnumerable<Task<int>> is also acceptable, the problem is as I mention in the question is that the IEnumerable will get consumed synchronously on the consumer where I want to wait for it at every iteration before consuming the next element. I'm wondering if there is a clever way to do this that does not involve a regular for loop pausing the iteration by await there. – Isoclinal

@JonSkeet ( Honest full disclosure, I'm trying to formulate a better opinion on this after watching this and I'm trying to figure how C# addresses that problem, I'd probably use an event emitter or something similar in C# normally)) – Isoclinal

@BenjaminGruenbaum, this question is closely related. AFAICT, what you're trying to do would be a great C# language + LINQ runtime feature, but it's just not available yet. Check also Stephen Toub's "Tasks, Monads, and LINQ". – Milieu

What you're looking for is something like IAsyncEnumerable which exposes per-element asynchrony. – Suborn

In 2016 perspective - I was thinking about async iterators - they now exist natively in Python and are being considered for JavaScript. – Isoclinal

Answer 1

Asynchronous sequences are interesting. There's a number of different approaches, depending on exactly what you want to do. I'm not entirely clear on your desired semantics, so these are some of the options.

Task<IEnumerable<T>> is an asynchronously-retrieved collection. There is only one task - one asynchronous operation - that retrieves the entire collection. This does not sound like it's what you want.

IEnumerable<Task<T>> is a (synchronous) sequence of (asynchronous) data. There are multiple tasks, which may or may not all be processing simultaneously. There are a couple of options for implementing this. One is using an enumerator block and yielding tasks; this approach will start a new asynchronous operation each time the next item is retrieved from the enumerable. Alternatively, you can create and return a collection of tasks with all tasks running concurrently (this can be done elegantly over a source sequence via LINQ's Select followed by ToList/ToArray). However, this has a couple of drawbacks: there is no way to asynchronously determine if the sequence is already ended, and it's not easy to immediately start the next item processing after returning the current item (which is commonly desired behavior).

The core problem is that IEnumerable<T> is inherently synchronous. There are a couple of workarounds. One is IAsyncEnumerable<T>, which is an asynchronous equivalent of IEnumerable<T> and available in the Ix-Async NuGet package. This approach has its own drawbacks, though. Of course, you lose the nice language support for IEnumerable<T> (namely, enumerator blocks and foreach). Also, the very notion of an "asynchronous enumerable" is not exactly performant; ideally, asynchronous APIs should be chunky rather than chatty, and enumerables are very chatty. More discussion on the original design here, and on the chunky/chatty considerations here.

So, these days a much more common solution is to use observables or dataflows (both also available via NuGet). In these cases, you have to think of the "sequence" as something with a life of its own. Observables are push-based, so the consuming code is (ideally) reactive. Dataflows have an actor feel, so they act more independent, again pushing results to the consuming code.

Hispanicize

Answer 2

You could return an IEnumerable<Task<int>>:

IEnumerable<Task<int>> Method()
{
    for (var i = 0; i < 100; i++)
    {
        yield return Task.Delay(1000).ContinueWith(x => i);
    }
}

And call it like this:

foreach(var i in Method())
    Console.WriteLine(await i);

Sarmiento

Note that a consumer doesn't need to wait for the task to be completed before asking for the next task, so if the generator isn't build to work properly when multiple items are requested in parallel it could be problematic. – Mimi

what about using await in here? await Task.Delay(1000) above the yield return, would it cause current and all future iterations to be resolved only after an addtiaional 1000 milliseconds? – Comitative