Sharing an enumerator across threads

Question

I want to call a common enumerator from different threads. When I do the following,

enum = (0..1000).to_enum
t1 = Thread.new do
  p enum.next
  sleep(1)
end
t2 = Thread.new do
  p enum.next
  sleep(1)
end
t1.join
t2.join

it raises an error:

Fiber called across threads.

when enum is called from t2 after once being called from t1.

• Why is Ruby designed to not allow an enumerator (or fiber) to be called across threads, and
• Is there an alternative way to provide a similar function?

I am guessing that atomicity of an operation on an enumerator/fiber is relevant here, but am not fully sure. If that is the issue, then exclusively-locking the enumerator/fiber while in use shall solve the problem, and I don't know why calling an enumerator/fiber across threads is prohibited in general. If an alternative can be provided by using locking, that would satisfy my need.

Streeter

will a Queue satisfy your needs? – Firer

@UriAgassi If you can do it with it, it would. – Streeter

@sawa, what did you end up doing? – Precontract

Answer 1

You can use Queue

queue = Queue.new
(0..1000).map(&queue.method(:push))

t1 = Thread.new do
  while !queue.empty?
    p queue.pop(true)
    sleep(0.1)
  end
end
t2 = Thread.new do
  while !queue.empty?
    p queue.pop(true)
    sleep(0.1)
  end
end
t1.join
t2.join

Firer

I want to be able to reuse it. By popping, it will change. – Streeter

This replaces the enumerator, not the array/range/whatever... instead of rewind - rebuild it – Firer

I am trying to do this same thing in a ROR application, and I need to be able to use an ActiveRecord query (with a large number of results) as an Enumerator. If I map the whole thing to a Queue, I end up using too much ram. (I'm using the find_each method, so that it only queries in batches of 1000. If I map it to a Queue, I will have the whole table sitting in a variable, which is not practical). See this question. – Allopathy

@Allopathy - the trick here is to spin up the threads before filling up the queue. This way the threads will consume the results as you are requesting for them - actually working when you are waiting for the DB to bring the next batch. – Firer