Multiple generators in a single loop within PHP

Question

I need to write a simple script that loads data from multiple files and merges it somehow. However, given the fact that the files might be quite huge I'd like to load data partially. To do so I decided to use yield. And according to examples I found I could use following construction for single generator:

$generator = $someClass->load(); //load method uses yield so it returns generator object
foreach($generator as $i) {
  // do something
}

But what if I want to use two generators at once?

$generatorA = $someClass1->load(); //load method uses yield so it returns generator object
$generatorB = $someClass2->load(); //load method uses yield so it returns generator object
foreach($generatorA as $i) {
  // how can I access to resultSet from generatorB here?
}

Roach

Answer 1

Generators in PHP implement the Iterator interface, so you can merge / combine multiple Generators like you can combine multiple Iterators.

If you want to iterate over both generators one after the other (merge A + B), then you can make use of the AppendIterator.

$aAndB = new AppendIterator();
$aAndB->append($generatorA);
$aAndB->append($generatorB);

foreach ($aAndB as $i) {
    ...

If you want to iterate over both generator at once, you can make use of MultipleIterator.

$both = new MultipleIterator();
$both->attachIterator($generatorA);
$both->attachIterator($generatorB);
 
foreach ($both as list($valueA, $valueB)) {
    ...

Example for those two incl. examples and caveats are in this blog-post of mine as well:

• Iterating over Multiple Iterators at Once

Generators can not Rewind

This is a caveat useful to understand when passing Iterators along that are Generators, and as it may happen when composing them.

As calling a generator function already executes to the first yield (or return), it is an iterator that can not rewind and throw if they would be rewound:

PHP Fatal error: Uncaught Exception: Cannot rewind a generator that was already run (PHP 8.2)

PHP Fatal error: Uncaught exception 'Exception' with message 'Cannot rewind a generator that was already run' (PHP 5.6)

PHP Fatal error: Uncaught Exception: Cannot traverse an already closed generator (PHP 8.2)

PHP Fatal error: Uncaught exception 'Exception' with message 'Cannot traverse an already closed generator' (PHP 5.6)

In Nikic's iter library you can find an implementation of a rewindable generator that works by invoking the generator function with its arguments again.

When decorating or composing Generators, you may want to handle this alternatively by rendering the rewind() method of the Iterator protocol void.

PHP has a standard implementation for that with the NoRewindIterator. Wrapping the Generator within then allows to re-iterate over the generator without throwing.

This can have the benefit to hide the throwing behaviour and make a Generator behave more expected with whole the Iterator protocol.

$genFunc = static function () {
    yield 'k' => 'v';
};
$iter = new NoRewindIterator($genFunc());
foreach (new LimitIterator($iter, 0, 1) as $k => $v) {
    var_dump("[ $k => $v ]");
}
foreach (new LimitIterator($iter, 0, 1) as $k => $v) {
    var_dump("[ $k => $v ]");
}

At very rare places if the abstraction still leaks, there is also CachingIterator but I don't have a practical example at hand, only remembering a scenario where getting the count of an overall collection in advance, but then having segments to pull and then yield, so a chain of generators from a generator that could also be empty, by optimistically lazy fetching and the collection then could be smaller or larger as by the initial count.

Hullo

I think you hit the nail on the head with MultipleIterator maybe an example? – Percaline

@DavidBarker: See edit, also the linked blog post with even more examples. – Hullo

I actually used a mix from your edits but works fine $both = new MultipleIterator(MultipleIterator::MIT_KEYS_ASSOC); $both->attachIterator($firstFileParser->loadData(), 'a'); $both->attachIterator($secondFileParser->loadData(), 'b'); – Churchwoman

@Moby04: Refresh the answer, I made the MultipleIterator example a bit less verbose with new PHP 5.5 features available for foreach. – Hullo

Seems nice. Anyway, I realized I cannot use generators for some reason in this specific case but still, it might be useful sometimes so thank :) – Churchwoman

Answer 2

From https://www.php.net/manual/en/language.generators.syntax.php#control-structures.yield.from

Generator delegation via yield from

In PHP 7, generator delegation allows you to yield values from another generator, Traversable object, or array by using the yield from keyword. The outer generator will then yield all values from the inner generator, object, or array until that is no longer valid, after which execution will continue in the outer generator.

So it's possible to combine two (or more) generators using yield from.

/**
  * Yield all values from $generator1, then all values from $generator2
  * Keys are preserved
  */
function combine_sequentially(Generator $generator1, Generator $generator2): Generator
{
    yield from $generator1;
    yield from $generator2;
};

Or something more fancy (here, it's not possible to use yield from):

/**
  * Yield a value from $generator1, then a value from $generator2, and so on
  * Keys are preserved
  */
function combine_alternatively(Generator $generator1, Generator $generator2): Generator
{
    while ($generator1->valid() || $generator2->valid()) {
        if ($generator1->valid()) {
            yield $generator1->key() => $generator1->current();
            $generator1->next();
        }
        if ($generator2->valid()) {
            yield $generator2->key() => $generator2->current();
            $generator2->next();
        }
    }
};

Disinfection

Answer 3

You can use yield from

  
function one()
{ 

   yield 1;
   yield 2;

}

function two()
{
   yield 3;
   yield 4;
}

function merge()
{
   yield from one();
   yield from two();

}
foreach(merge() as $i)
{
   echo $i;
}

An example Reusable function

function iterable_merge( iterable ...$iterables ): Generator {
   

    foreach ( $iterables as $iterable ) {
        
        yield from $iterable;
    }
}

$merge=iterable_merge(one(),two());

Bicuspid

Answer 4

While AppendIterator works for Iterators, it has some issues.

Firstly it is not so nice to need to construct a new object rather than just calling a function. What is even less nice is that you need to mutate the AppendIterator, since you cannot provide the inner iterators in its constructor.

Secondly AppendIterator only takes Iterator instances, so if you have a Traversable, such as IteratorAggregate, you are out of luck. Same story for other iterable that are not Iterator, such as array.

This PHP 7.1+ function combines two iterable:

/**
 * array_merge clone for iterables using lazy evaluation
 *
 * As with array_merge, numeric elements with keys are assigned a fresh key,
 * starting with key 0. Unlike array_merge, elements with duplicate non-numeric
 * keys are kept in the Generator. Beware that when converting the Generator
 * to an array with a function such as iterator_to_array, these duplicates will
 * be dropped, resulting in identical behaviour as array_merge.
 *
 *
 * @param iterable ...$iterables
 * @return Generator
 */
function iterable_merge( iterable ...$iterables ): Generator {
    $numericIndex = 0;

    foreach ( $iterables as $iterable ) {
        foreach ( $iterable as $key => $value ) {
            yield is_int( $key ) ? $numericIndex++ : $key => $value;
        }
    }
}

Usage example:

foreach ( iterable_merge( $iterator1, $iterator2, $someArray ) as $k => $v ) {}

This function is part of a small library for working with iterable, where it is also extensively tested.

Christ

Answer 5

If you want to use Generators with AppendIterator you'll need to use NoRewindIterator with it:

https://3v4l.org/pgiXB

<?php
function foo() {
        foreach ([] as $foo) {
                yield $foo;
        }
}
$append = new AppendIterator();
$append->append(new NoRewindIterator(foo()));

var_dump(iterator_to_array($append));

Trying to traverse a bare Generator with AppendIterator will cause a fatal error if the Generator never actually calls yield:

https://3v4l.org/B4Qnh

<?php
function foo() {
        foreach ([] as $foo) {
                yield $foo;
        }
}
$append = new AppendIterator();
$append->append(foo());

var_dump(iterator_to_array($append));

Output:

Fatal error: Uncaught Exception: Cannot traverse an already closed generator in /in/B4Qnh:10
Stack trace:
#0 [internal function]: AppendIterator->rewind()
#1 /in/B4Qnh(10): iterator_to_array(Object(AppendIterator))
#2 {main}
  thrown in /in/B4Qnh on line 10

Process exited with code 255.

Atli

Answer 6

Try this:

<?php
foreach($generatorA as $key=>$i) {
    $A=$i;//value from $generatorA
    $B=$generatorB[$key];//value from $generatorB
}

Rataplan

Answer 7

Something like:

$generatorA = $someClass1->load(); //load method uses yield so it returns generator object
$generatorB = $someClass2->load(); //load method uses yield so it returns generator object

$flag = true;
$i = 0;
while($flag === false) {

 if ($i >= count($generatorA) || $i >= count($generatorB)) {
      $flag = true;
 }

 // Access both generators
 $genA = $generatorA[$i];
 $genB = $generatorB[$i];

$i++;
}

Rodenhouse