How to handle an "infinite" IEnumerable?

Question

A trivial example of an "infinite" IEnumerable would be

IEnumerable<int> Numbers() {
  int i=0;
  while(true) {
    yield return unchecked(i++);
  }
}

I know, that

foreach(int i in Numbers().Take(10)) {
  Console.WriteLine(i);
}

and

var q = Numbers();
foreach(int i in q.Take(10)) {
  Console.WriteLine(i);
}

both work fine (and print out the number 0-9).

But are there any pitfalls when copying or handling expressions like q? Can I rely on the fact, that they are always evaluated "lazy"? Is there any danger to produce an infinite loop?

Blanco

Just to be picky, wouldn't your "infinite" example throw an exception when i = Int32.MaxValue and you do i++? Or does it cycle through to Int32.MinValue? Hmm..... – Mckim

You are right. It will probably throw an overflow exception ... I'll edit it. – Blanco

Just being picky here, your point still came across. :) Also, I tried it and it does loop to Int32.MinValue. No OverflowException, so it was in fact an infinite loop. – Mckim

C# compiles without overflow checking by default. the safe way is always to write unchecked(i++) because you could compile with /checked, and then it would puke. Using unchecked() tells the compiler it is ok if an overflow occurs even if compiled with /checked. – Uphemia

Than back to the more beautiful original :D – Blanco

Aha, I learn something new every day. I know in assembly you don't have any overflow checking, I thought C# would have it, being a "higher level" language, but I guess it's optional. I think using unchecked() also makes your intentions clear, which is always a good thing. And I think it's unchecked { }, not unchecked( ). :) – Mckim

@Nelson: no, both unchecked {} and unchecked() are allowed in c#. see msdn.microsoft.com/en-us/library/a569z7k8%28VS.71%29.aspx – Impudence

I sincerly hope that code like this NEVER makes it into a production app that you're working on. If it does it belongs in the Daily WTF. – Nubile

@Josiah: Well it is a nice feature supporting a stream-like view on data. For example receiving network data or parsing a log file or random number generation or ... – Blanco

@dalo, the link Philip posted says: "If neither checked nor unchecked is used, a constant expression uses the default overflow checking at compile time, which is checked." This contradicts with what you said, but on a default WPF project template, it did not give me an OverflowException. I don't know who's finally right. – Mckim

@Nelson: It doesn't "contradict with what I said" because the expression isn't a constant expression. It cannot be evaluated at compile time. Hence, the compiler options influence the overflow checking, and for any version of the C# compiler available until today, the checking is turned off by default, hence NO overflow exception (in VB the behavior is different). – Uphemia

Good point. short x = 32767 + 10; says "Constant value '32777' cannot be converted to a 'short'. I'm guessing that's an example of "overflow checking" in a constant expression? – Mckim

Answer 1

Yes, you are guaranteed that the code above will be executed lazily. While it looks (in your code) like you'd loop forever, your code actually produces something like this:

IEnumerable<int> Numbers()
{
    return new PrivateNumbersEnumerable();
}

private class PrivateNumbersEnumerable : IEnumerable<int>
{
    public IEnumerator<int> GetEnumerator() 
    { 
        return new PrivateNumbersEnumerator(); 
    }
}

private class PrivateNumbersEnumerator : IEnumerator<int>
{
    private int i;

    public bool MoveNext() { i++; return true; }   

    public int Current
    {
        get { return i; }
    }
}

(This obviously isn't exactly what will be generated, since this is pretty specific to your code, but it's nonetheless similar and should show you why it's going to be lazily evaluated).

Involuntary

Answer 2

As long as you only call lazy, un-buffered methods you should be fine. So Skip, Take, Select, etc are fine. However, Min, Count, OrderBy etc would go crazy.

It can work, but you need to be cautious. Or inject a Take(somethingFinite) as a safety measure (or some other custom extension method that throws an exception after too much data).

For example:

public static IEnumerable<T> SanityCheck<T>(this IEnumerable<T> data, int max) {
    int i = 0;
    foreach(T item in data) {
        if(++i >= max) throw new InvalidOperationException();
        yield return item;
    }
}

Metz

+1 I have a competing answer, but I just have to give an upvote for an extension method for IEnumerable called SanityCheck. Best function name ever. I'm gonna go implement that . . . – Desma

+1 for pointing this out. Also if the OP actually call foreach(int i in Numbers()) that would also cause problems. – Variolous

Answer 3

You would have to avoid any greedy functions that attempt to read to end. This would include Enumerable extensions like: Count, ToArray/ToList, and aggregates Avg/Min/Max, etc.

There's nothing wrong with infinite lazy lists, but you must make conscious decisions about how to handle them.

Use Take to limit the impact of an endless loop by setting an upper bound even if you don't need them all.

Patagium

Answer 4

Yes, your code will always work without infinite looping. Someone might come along though later and mess things up. Suppose they want to do:

var q = Numbers().ToList();

Then, you're hosed! Many "aggregate" functions will kill you, like Max().

Desma

Answer 5

If it wasn't lazy evaluation, your first example won't work as expected in the first place.

Discounter