ndb aysnc "yield next" to consume get_multi_async

Question

Is there a builtin or canonical way to consume the first and all successive results for a ndb.get_multi_async call in the order of their completion ?

I would expect, and this illustrates, it along the lines of:

def yield_next(futures):
   while futures:
     yield ndb.Future.wait_any(futures)
     # We have to remove the yielded future from the futures.
     # How do we know which future was removed?
     # ... futures.remove(???)

for model in yield_next(ndb.get_multi_async(keys)):
   ...

One way to remove the future consumed is to check futures to see whether it is done. There is a race condition inherent: If any futures complete simultaneously or in any case before the remove call, multiple elements of futures may be done. I am otherwise not aware of a reliable way to determine which future was consumed.

One would expect that this is quite a common pattern, but I have not seen it resolved. Looking through ndb/tasklets.py there seem to be some exotic (read: undocumented) possibilities like ReducingFuture or MultiFuture, but I have never used them. Perhaps the answer lies with tasklets themselves.

Pipage

Answer 1

It's easy -- just use a set:

futures = set(futures)
while futures:
    f = ndb.Future.wait_any(futures)
    futures.remove(f)
    yield f

Doughty