How to create a "clone"-able enumerator for external iteration?

Question

I want to create an enumerator for external iteration via next that is clone-able, so that the clone retains the current enumeration state.

As an example, let's say I have a method that returns an enumerator which yields square numbers:

def square_numbers
  return enum_for(__method__) unless block_given?

  n = d = 1
  loop do
     yield n
     d += 2
     n += d
   end
end

square_numbers.take(10)
#=> [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

And I want to enumerate the first 5 square numbers, and for each value, print the subsequent 3 square numbers. Something that's trivial with each_cons:

square_numbers.take(8).each_cons(4) do |a, *rest|
  printf("%2d: %2d %2d %2d\n", a, *rest)
end

Output:

 1:  4  9 16
 4:  9 16 25
 9: 16 25 36
16: 25 36 49
25: 36 49 64

But unlike the above, I want to use external iteration using two nested loops along with next and clone:

outer_enum = square_numbers
5.times do
  i = outer_enum.next
  printf('%2d:', i)

  inner_enum = outer_enum.clone
  3.times do
    j = inner_enum.next
    printf(' %2d', j)
  end
  print("\n")
end

Unfortunately, the above attempt to clone raises a:

`initialize_copy': can't copy execution context (TypeError)

I understand that Ruby doesn't provide this out-of-the-box. But how can I implement it myself? How can I create an Enumerator that supports clone?

I assume that it's a matter of implementing initialize_copy and copying the two variable values for n and d, but I don't know how or where to do it.

Sisterly

BTW, I managed to get this working using a proc and an explicit binding that holds the local variables. It's a pretty dirty hack ... but it is possible. – Sisterly

Why don't you post what you did? I expect it's just a roundabout way of creating a class, like I suggested (wiki.c2.com/?ClosuresAndObjectsAreEquivalent). – Puto

Answer 1

Ruby fibers cannot be copied, and the C implementation of Enumerator stores a pointer to a fiber which does not appear to be exposed to Ruby code in any way.

https://github.com/ruby/ruby/blob/752041ca11c7e08dd14b8efe063df06114a9660f/enumerator.c#L505

if (ptr0->fib) {
    /* Fibers cannot be copied */
    rb_raise(rb_eTypeError, "can't copy execution context");
}

Looking through the C source, it's apparent that Enumerators and Fibers are connected in a pretty profound way. So I doubt that there is any way to change the behavior of initialize_copy to permit clone.

Lawlor

I understand "Fiber cannot be copied" in terms of "not automatically", so you can't clone arbitrary enumerators. But I have full control over my enumerator. Maybe that's a different story? – Sisterly

Well, the comment Fibers cannot be copied is in the C source code for Enumerator, so it sounds pretty final. – Lawlor

Answer 2

Perhaps you could just write a class of your own that does what you ask:

class NumberSquarer
  def initialize
    @n = @d = 1
  end

  def next
    ret = @n
    @d += 2
    @n += @d
    ret
  end
end

ns1 = NumberSquarer.new
Array.new(5) { ns1.next }
# => [1, 4, 9, 16, 25]

ns2 = ns1.clone
Array.new(5) { ns2.next }
# => [36, 49, 64, 81, 100]

Puto

Sure, re-implemening Enumerator is an alternative. But not really what I want :-) – Sisterly

I don't think it can be done with the built-in. Enumerator, like Proc, are not serializable objects. I believe it's because they internally hold references to live runtime memory state. By defining your own class, you're designing something with clearly defined variables and stateless functions, i.e something that can be copied. – Puto

Answer 3

Disclaimer: I'm answering my own question

---

One way to achieve this is by sub-classing Enumerator. In particular, the now-deprecated variant that takes an object and a method:

class ObjectEnumerator < Enumerator
  attr_reader :object, :method

  def initialize(object, method = :each)
    @object = object
    @method = method
    super
  end

  def initialize_copy(orig)
    initialize(orig.object.clone, orig.method)
  end
end

That orig.object.clone above is where the magic happens: it clones the object we are traversing.

In addition, we need such clone-able object. A simple way is to have a class which holds the state as instance variables: (shamelessly copied from Kache's answer)

class NumberSquarer
  def initialize
    @d = -1
    @n = 0
  end

  def each
    return ObjectEnumerator.new(self, __method__) unless block_given?

    loop do
      @d += 2
      @n += @d  #    had to be reordered b/c
      yield @n  # <- yield has to come last
    end
  end
end

This gives us a basic, clone-able enumerator:

e = NumberSquarer.new.each
#=> #<ObjectEnumerator: #<NumberSquarer:0x00007fde60915e10 @d=-1, @n=0>:each>

e.next #=> 1
e.next #=> 4

other = enum.clone
#=> #<ObjectEnumerator: #<NumberSquarer:0x00007fcf23842520 @d=3, @n=4>:each>

enum.next #=> 9
enum.next #=> 16

other.next #=> 9

Sisterly

Answer 4

I'm providing a different solution that is not a straight answer to the question:

How can I create an Enumerator that supports clone?

But if I'm not wrong the only purpose of cloning the not clonable Ruby's Enumerator is to get a reference to the next object in the enumerator.

In this case, we need both values stored in odd_sum and square in the example below.

We can store those values in an Array and return the array instead of a single value, then we can use Enumerator.peek in order to have the array that is used to initialize a new Enumerator.

def square_numbers(starters = {})
  return enum_for(__method__, starters) unless block_given?

  last_odd = starters.fetch(:square_odd, [1,1])[1]
  square = starters.fetch(:square_odd, [1,1])[0]

  loop do
     yield [square, last_odd]
     last_odd += 2
     square += last_odd
   end
end

outer_enum = square_numbers
5.times do
  i = outer_enum.next[0]
  printf('%2d:', i)

  inner_enum = square_numbers(square_odd: outer_enum.peek)
  3.times do
    j = inner_enum.next[0]
    printf(' %2d', j)
  end
  print("\n")
end

Tease