Why can't iterator methods take either 'ref' or 'out' parameters?
Asked Answered
I

6

53

I tried this earlier today:

public interface IFoo
{
    IEnumerable<int> GetItems_A( ref int somethingElse );
    IEnumerable<int> GetItems_B( ref int somethingElse );
}


public class Bar : IFoo
{
    public IEnumerable<int> GetItems_A( ref int somethingElse )
    {
        // Ok...
    }

    public IEnumerable<int> GetItems_B( ref int somethingElse )
    {
        yield return 7; // CS1623: Iterators cannot have ref or out parameters            

    }
}

What's the rationale behind this?

Intrados answered 15/6, 2009 at 23:48 Comment(2)
I discuss some of these design considerations here: blogs.msdn.com/ericlippert/archive/2009/05/04/…Ablate
Eric's article can now be found hereSpear
R
62

C# iterators are state machines internally. Every time you yield return something, the place where you left off should be saved along with the state of local variables so that you could get back and continue from there.

To hold this state, C# compiler creates a class to hold local variables and the place it should continue from. It's not possible to have a ref or out value as a field in a class. Consequently, if you were allowed to declare a parameter as ref or out, there would be no way to keep the complete snapshot of the function at the time we had left off.

EDIT: Technically, not all methods that return IEnumerable<T> are considered iterators. Just those that use yield to produce a sequence directly are considered iterators. Therefore, while the splitting the iterator into two methods is a nice and common workaround, it doesn't contradict with what I just said. The outer method (that doesn't use yield directly) is not considered an iterator.

Reggy answered 15/6, 2009 at 23:50 Comment(5)
"It's not possible to have a ref or out value as a field in a class." --The compiler could easily implement ref parameters to iterators by allocating a single element array in the caller, putting the argument into that, and passing the array to the iterator, and having the iterator operate on array[0]. This would be a very small amount of work on the part of the compiler compared to turning the iterator into a state machine.Gotama
@JimBalter That would be true if the compiler controlled every piece of the code that ran. Unfortunately, that plan would require a different API signature to be generated in the binary--i.e. callers from the outside world passing in "ref" variables would not be able to see them changing.Reggy
Um, callers from the outside world can't currently pass in ref parameters at all ... no one can. So there would be no harm in specifying that ref T parameters to iterators are actually represented as T[]. In any case, using an array or a class instance as a box is a simple way to get around the limitation ... simpler than the other solutions offered here.Gotama
@JimBalter they will be if you allow ref T, which is the whole point.Reggy
Um, if C# generates a signature with T[], that doesn't outside callers to invoke it with ref T, just as they can't now. It always amuses me when people talk about "the whole point" when that totally ignores what the other person said. I have no interest in further discussion when I encounter that ... over and out.Gotama
D
26

If you want to return both an iterator and an int from your method, a workaround is this:

public class Bar : IFoo
{
    public IEnumerable<int> GetItems( ref int somethingElse )
    {
        somethingElse = 42;
        return GetItemsCore();
    }

    private IEnumerable<int> GetItemsCore();
    {
        yield return 7;
    }
}

You should note that none of the code inside an iterator method (i.e. basically a method that contains yield return or yield break) is executed until the MoveNext() method in the Enumerator is called. So if you were able to use out or ref in your iterator method, you would get surprising behavior like this:

// This will not compile:
public IEnumerable<int> GetItems( ref int somethingElse )
{
    somethingElse = 42;
    yield return 7;
}

// ...
int somethingElse = 0;
IEnumerable<int> items = GetItems( ref somethingElse );
// at this point somethingElse would still be 0
items.GetEnumerator().MoveNext();
// but now the assignment would be executed and somethingElse would be 42

This is a common pitfall, a related issue is this:

public IEnumerable<int> GetItems( object mayNotBeNull ){
  if( mayNotBeNull == null )
    throw new NullPointerException();
  yield return 7;
}

// ...
IEnumerable<int> items = GetItems( null ); // <- This does not throw
items.GetEnumerators().MoveNext();                    // <- But this does

So a good pattern is to separate iterator methods into two parts: one to execute immediately and one that contains the code that should be lazily executed.

public IEnumerable<int> GetItems( object mayNotBeNull ){
  if( mayNotBeNull == null )
    throw new NullPointerException();
  // other quick checks
  return GetItemsCore( mayNotBeNull );
}

private IEnumerable<int> GetItemsCore( object mayNotBeNull ){
  SlowRunningMethod();
  CallToDatabase();
  // etc
  yield return 7;
}    
// ...
IEnumerable<int> items = GetItems( null ); // <- Now this will throw

EDIT: If you really want the behavior where moving the iterator would modify the ref-parameter, you could do something like this:

public static IEnumerable<int> GetItems( Action<int> setter, Func<int> getter )
{
    setter(42);
    yield return 7;
}

//...

int local = 0;
IEnumerable<int> items = GetItems((x)=>{local = x;}, ()=>local);
Console.WriteLine(local); // 0
items.GetEnumerator().MoveNext();
Console.WriteLine(local); // 42
Danaides answered 16/6, 2009 at 8:2 Comment(2)
Re: the edit with getter/setter lambdas, this as a way of simulating pointers to value types (although without address manipulation, of course), more here: incrediblejourneysintotheknown.blogspot.com/2008/05/…Allenaallenby
Simpler than passing a getter and setter is to put the "ref" parameter in a box -- a class instance if you have multiple "ref" parameters, or simply a single element array if you just have one.Gotama
O
6

At a highish level, A ref variable can point to many locations including to value types that are on the stack. The time at which the iterator is initially created by calling the iterator method and when the ref variable would be assigned are two very different times. It is not possible to guarantee that the variable which originally was passed by reference is still around when the iterator actually executes. Hence it is not allowed (or verifiable)

Occultism answered 16/6, 2009 at 0:3 Comment(0)
G
6

Others have explained why your iterator can't have a ref parameter. Here's a simple alternative:

public interface IFoo
{
    IEnumerable<int> GetItems( int[] box );
    ...
}

public class Bar : IFoo
{
    public IEnumerable<int> GetItems( int[] box )
    {
        int value = box[0];
        // use and change value and yield to your heart's content
        box[0] = value;
    }
}

If you have several items to pass in and out, define a class to hold them.

Gotama answered 12/6, 2014 at 3:57 Comment(0)
B
2

I've gotten around this problem using functions, when the value that I need to return is derived from the iterated items:

// One of the problems with Enumerable.Count() is
// that it is a 'terminator', meaning that it will
// execute the expression it is given, and discard
// the resulting sequence. To count the number of
// items in a sequence without discarding it, we 
// can use this variant that takes an Action<int>
// (or Action<long>), invokes it and passes it the
// number of items that were yielded.
//
// Example: This example allows us to find out
//          how many items were in the original
//          source sequence 'items', as well as
//          the number of items consumed by the
//          call to Sum(), without causing any 
//          LINQ expressions involved to execute
//          multiple times.
// 
//   int start = 0;    // the number of items from the original source
//   int finished = 0; // the number of items in the resulting sequence
//
//   IEnumerable<KeyValuePair<string, double>> items = // assumed to be an iterator
//
//   var result = items.Count( i => start = i )
//                   .Where( p => p.Key = "Banana" )
//                      .Select( p => p.Value )
//                         .Count( i => finished = i )
//                            .Sum();
//
//   // by getting the count of items operated 
//   // on by Sum(), we can calculate an average:
// 
//   double average = result / (double) finished; 
//
//   Console.WriteLine( "started with {0} items", start );
//   Console.WriteLine( "finished with {0} items", finished );
//

public static IEnumerable<T> Count<T>( 
    this IEnumerable<T> source, 
    Action<int> receiver )
{
  int i = 0;
  foreach( T item in source )
  {
    yield return item;
    ++i ;
  }
  receiver( i );
}

public static IEnumerable<T> Count<T>( 
    this IEnumerable<T> source, 
    Action<long> receiver )
{
  long i = 0;
  foreach( T item in source )
  {
    yield return item;
    ++i ;
  }
  receiver( i );
}
Blasius answered 15/6, 2009 at 23:48 Comment(0)
C
0

You can easily replace the ref param with something like

public class ItRef<T> where T:struct
{
    public T Value;
}

Which is something I've done in environments where we don't care so much about performance and instead are optimizing mostly for maintainability (provided you can't just change the param to a reference type).

Calder answered 15/4, 2024 at 18:16 Comment(0)

© 2022 - 2025 — McMap. All rights reserved.