How can I make a ruby enumerator that does lazy iteration through two other enumerators?

Question

Let's say I have two enumerators, enum1 and enum2 that must be lazily iterated through (because they have side effects). How do I construct a third enumerator enum3 where enum3.each{|x| x} would lazily return the equivalent of enum1 + enum2?

In my real world use case, I'm streaming in two files, and need to stream out the concatenation.

Lapotin

Answer 1

This seems to work just how I want;

enums.lazy.flat_map{|enum| enum.lazy }

Here's the demonstration. Define these yielding methods with side-effects;

def test_enum
  return enum_for __method__ unless block_given?
  puts 'hi'
  yield 1
  puts 'hi again'
  yield 2
end  

def test_enum2
  return enum_for __method__ unless block_given?
  puts :a
  yield :a
  puts :b
  yield :b
end  

concated_enum = [test_enum, test_enum2].lazy.flat_map{|en| en.lazy }

Then call next on the result, showing that the side effects happen lazily;

[5] pry(main)> concated_enum.next
hi
=> 1
[6] pry(main)> concated_enum.next
hi again
=> 2

Lapotin

Thanks so much for the flat_map solution. Even though we'd already verified the laziness of flat_map, this didn't occur to us, to use it as a lazy append! :) – Sottish

The drawback of this solution is that concated_enum.size always returns nil. – Cryptonymous

@Cryptonymous I think that's a drawback of lazy enums; you can never know how big they are until you've iterated through them. Their size might be a function of how long you take to iterate. They can even be infinite. – Lapotin

enums.lazy.flat_map(&:lazy) is a bit cleaner. might also add an explanation of how it works. – Bebe

More recently, since Ruby 2.6 you can use Enumerable#chain/Enumerator::Chain. See my answer below. – Deca

Answer 2

Since Ruby 2.6 you can use Enumerable#chain/Enumerator::Chain:

a = [1, 2, 3].lazy
b = [4, 5, 6].lazy

a.chain(b).to_a
# => [1, 2, 3, 4, 5, 6]

Enumerator::Chain.new(a, b).to_a
# => [1, 2, 3, 4, 5, 6]

Deca

Answer 3

Here's some code I wrote for fun awhile back with lazy enumeration thrown in:

def cat(*args)
  args = args.to_enum

  Enumerator.new do |yielder|
    enum = args.next.lazy

    loop do
      begin
        yielder << enum.next
      rescue StopIteration
        enum = args.next.lazy
      end
    end
  end
end

You would use it like this:

enum1 = [1,2,3]
enum2 = [4,5,6]
enum3 = cat(enum1, enum2)

enum3.each do |n|
  puts n
end
# => 1
#    2
#    3
#    4
#    5
#    6

...or just:

cat([1,2,3],[4,5,6]).each {|n| puts n }

Type