What are "iterable", "iterator", and "iteration" in Python? How are they defined?
See also: How to build a basic iterator?
What are "iterable", "iterator", and "iteration" in Python? How are they defined?
See also: How to build a basic iterator?
Iteration is a general term for taking each item of something, one after another. Any time you use a loop, explicit or implicit, to go over a group of items, that is iteration.
In Python, iterable and iterator have specific meanings.
An iterable is an object that has an __iter__
method which returns an iterator, or which defines a __getitem__
method that can take sequential indexes starting from zero (and raises an IndexError
when the indexes are no longer valid). So an iterable is an object that you can get an iterator from.
An iterator is an object with a next
(Python 2) or __next__
(Python 3) method.
Whenever you use a for
loop, or map
, or a list comprehension, etc. in Python, the next
method is called automatically to get each item from the iterator, thus going through the process of iteration.
A good place to start learning would be the iterators section of the tutorial and the iterator types section of the standard types page. After you understand the basics, try the iterators section of the Functional Programming HOWTO.
collections.abc.AsyncIterator
tests for __aiter__
and __anext__
methods. This is a new addition in 3.6. –
Ceramist __len__
be necessarily tied to iteration? How would knowing the length of something help you iterate over it? –
Riflery __getitem__
. –
Grevera {'a': 'hi', 'b': 'bye'}
has length of 2, but cannot be indexed by 0, 1, or 2. –
Riflery __iter__
method. I think jlh is referring to objects that are iterable specifically because they define: "a __getitem__
method that can take sequential indexes starting from zero". –
Celia __getitem__
without supporting the length protocol. len
simply has nothing to do with iteration. –
Auditor __getitem__
could check the len
to see which numerical indices are valid, and jlh assumed that was how it was implemented ... till they found out it wasn't. –
Celia __getitem__
until it raises an exception, then raise StopIteration
in its place. __len__
is ignored because it's not needed with __getitem__
and not useful without it. –
Auditor Here's the explanation I use in teaching Python classes:
An ITERABLE is:
for x in iterable: ...
oriter()
that will return an ITERATOR: iter(obj)
or__iter__
that returns a fresh ITERATOR,
or it may have a __getitem__
method suitable for indexed lookup.An ITERATOR is an object:
__next__
method that:
StopIteration
__iter__
method that returns self
).Notes:
__next__
method in Python 3 is spelt next
in Python 2, andnext()
calls that method on the object passed to it.For example:
>>> s = 'cat' # s is an ITERABLE
# s is a str object that is immutable
# s has no state
# s has a __getitem__() method
>>> t = iter(s) # t is an ITERATOR
# t has state (it starts by pointing at the "c"
# t has a next() method and an __iter__() method
>>> next(t) # the next() function returns the next value and advances the state
'c'
>>> next(t) # the next() function returns the next value and advances
'a'
>>> next(t) # the next() function returns the next value and advances
't'
>>> next(t) # next() raises StopIteration to signal that iteration is complete
Traceback (most recent call last):
...
StopIteration
>>> iter(t) is t # the iterator is self-iterable
for
loops, and the 1st bullet is about "looping over". Could you pls address these? –
Baro iter()
" as "anything you can pass to iter()
" –
Baro __iter__()
method? (with just a __getitem__()
?) –
Admissive The above answers are great, but as most of what I've seen, don't stress the distinction enough for people like me.
Also, people tend to get "too Pythonic" by putting definitions like "X is an object that has __foo__()
method" before. Such definitions are correct--they are based on duck-typing philosophy, but the focus on methods tends to get between when trying to understand the concept in its simplicity.
So I add my version.
In natural language,
In Python,
iterable is an object that is, well, iterable, which simply put, means that
it can be used in iteration, e.g. with a for
loop. How? By using iterator.
I'll explain below.
... while iterator is an object that defines how to actually do the
iteration--specifically what is the next element. That's why it must have
next()
method.
Iterators are themselves also iterable, with the distinction that their __iter__()
method returns the same object (self
), regardless of whether or not its items have been consumed by previous calls to next()
.
So what does Python interpreter think when it sees for x in obj:
statement?
Look, a
for
loop. Looks like a job for an iterator... Let's get one. ... There's thisobj
guy, so let's ask him."Mr.
obj
, do you have your iterator?" (... callsiter(obj)
, which callsobj.__iter__()
, which happily hands out a shiny new iterator_i
.)OK, that was easy... Let's start iterating then. (
x = _i.next()
...x = _i.next()
...)
Since Mr. obj
succeeded in this test (by having certain method returning a valid iterator), we reward him with adjective: you can now call him "iterable Mr. obj
".
However, in simple cases, you don't normally benefit from having iterator and iterable separately. So you define only one object, which is also its own iterator. (Python does not really care that _i
handed out by obj
wasn't all that shiny, but just the obj
itself.)
This is why in most examples I've seen (and what had been confusing me over and over), you can see:
class IterableExample(object):
def __iter__(self):
return self
def next(self):
pass
instead of
class Iterator(object):
def next(self):
pass
class Iterable(object):
def __iter__(self):
return Iterator()
There are cases, though, when you can benefit from having iterator separated from the iterable, such as when you want to have one row of items, but more "cursors". For example when you want to work with "current" and "forthcoming" elements, you can have separate iterators for both. Or multiple threads pulling from a huge list: each can have its own iterator to traverse over all items. See @Raymond's and @glglgl's answers above.
Imagine what you could do:
class SmartIterableExample(object):
def create_iterator(self):
# An amazingly powerful yet simple way to create arbitrary
# iterator, utilizing object state (or not, if you are fan
# of functional), magic and nuclear waste--no kittens hurt.
pass # don't forget to add the next() method
def __iter__(self):
return self.create_iterator()
Notes:
I'll repeat again: iterator is not iterable. Iterator cannot be used as
a "source" in for
loop. What for
loop primarily needs is __iter__()
(that returns something with next()
).
Of course, for
is not the only iteration loop, so above applies to some other
constructs as well (while
...).
Iterator's next()
can throw StopIteration to stop iteration. Does not have to,
though, it can iterate forever or use other means.
In the above "thought process", _i
does not really exist. I've made up that name.
There's a small change in Python 3.x: next()
method (not the built-in) now
must be called __next__()
. Yes, it should have been like that all along.
You can also think of it like this: iterable has the data, iterator pulls the next item
Disclaimer: I'm not a developer of any Python interpreter, so I don't really know what the interpreter "thinks". The musings above are solely demonstration of how I understand the topic from other explanations, experiments and real-life experience of a Python newbie.
for
loop needs an iterator ("Look, a for loop. Looks like a job for an iterator... Let's get one."). But then you say in the notes at the end that "Iterator cannot be used as a source in a for
loop"...? –
Celestecelestia pass
in the code for those next
definitions? I'll assume you just mean that someone has to implement a way to get the next one, since next has to return something. –
Domino pass
is for, after all.) –
Wellbeing pass
, I think it is there for syntactic reasons. I just ran across the answers at ellipsis object which are quite interesting: you can use ...
to indicate a "todo later" block. NotImplemented
is also available. –
Domino for
loop'. I get the point of your answer, and like it otherwise, but I think it would benefit from fixing this. –
Celia An iterable is a object which has a __iter__()
method. It can possibly iterated over several times, such as list()
s and tuple()
s.
An iterator is the object which iterates. It is returned by an __iter__()
method, returns itself via its own __iter__()
method and has a next()
method (__next__()
in 3.x).
Iteration is the process of calling this next()
resp. __next__()
until it raises StopIteration
.
Example:
>>> a = [1, 2, 3] # iterable
>>> b1 = iter(a) # iterator 1
>>> b2 = iter(a) # iterator 2, independent of b1
>>> next(b1)
1
>>> next(b1)
2
>>> next(b2) # start over, as it is the first call to b2
1
>>> next(b1)
3
>>> next(b1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>> b1 = iter(a) # new one, start over
>>> next(b1)
1
Iterable
. –
Choong Iterator
is always Iterable
and is its own Iterator
, two calls of iter()
do not necessarily give two independent Iterator
s. –
Poleyn for i in [1,3,4,6]: print(i)
/ for i in {1,3,4,6}: print(i)
/ for i in (1,3,4,6): print(i)
. Also, have a look at the documentation resp. language specification. –
Poleyn I don’t know if it helps anybody but I always like to visualize concepts in my head to better understand them. So as I have a little son I visualize iterable/iterator concept with bricks and white paper.
Suppose we are in the dark room and on the floor we have bricks for my son. Bricks of different size, color, does not matter now. Suppose we have 5 bricks like those. Those 5 bricks can be described as an object – let’s say bricks kit. We can do many things with this bricks kit – can take one and then take second and then third, can change places of bricks, put first brick above the second. We can do many sorts of things with those. Therefore this bricks kit is an iterable object or sequence as we can go through each brick and do something with it. We can only do it like my little son – we can play with one brick at a time. So again I imagine myself this bricks kit to be an iterable.
Now remember that we are in the dark room. Or almost dark. The thing is that we don’t clearly see those bricks, what color they are, what shape etc. So even if we want to do something with them – aka iterate through them – we don’t really know what and how because it is too dark.
What we can do is near to first brick – as element of a bricks kit – we can put a piece of white fluorescent paper in order for us to see where the first brick-element is. And each time we take a brick from a kit, we replace the white piece of paper to a next brick in order to be able to see that in the dark room. This white piece of paper is nothing more than an iterator. It is an object as well. But an object with what we can work and play with elements of our iterable object – bricks kit.
That by the way explains my early mistake when I tried the following in an IDLE and got a TypeError:
>>> X = [1,2,3,4,5]
>>> next(X)
Traceback (most recent call last):
File "<pyshell#19>", line 1, in <module>
next(X)
TypeError: 'list' object is not an iterator
List X here was our bricks kit but NOT a white piece of paper. I needed to find an iterator first:
>>> X = [1,2,3,4,5]
>>> bricks_kit = [1,2,3,4,5]
>>> white_piece_of_paper = iter(bricks_kit)
>>> next(white_piece_of_paper)
1
>>> next(white_piece_of_paper)
2
>>>
Don’t know if it helps, but it helped me. If someone could confirm/correct visualization of the concept, I would be grateful. It would help me to learn more.
I don't think that you can get it much simpler than the documentation, however I'll try:
You can think Iterator as a helper pseudo-method (or pseudo-attribute) that gives (or holds) the next (or first) item in the iterable. (In practice it is just an object that defines the method next()
)
Iteration is probably best explained by the Merriam-Webster definition of the word :
b : the repetition of a sequence of computer instructions a specified number of times or until a condition is met — compare recursion
Iterables have a
__iter__
method that instantiates a new iterator every time.Iterators implement a
__next__
method that returns individual items, and a__iter__
method that returnsself
.Therefore, iterators are also iterable, but iterables are not iterators.
Luciano Ramalho, Fluent Python.
Iterable:- something that is iterable is iterable; like sequences like lists ,strings etc.
Also it has either the __getitem__
method or an __iter__
method. Now if we use iter()
function on that object, we'll get an iterator.
Iterator:- When we get the iterator object from the iter()
function; we call __next__()
method (in python3) or simply next()
(in python2) to get elements one by one. This class or instance of this class is called an iterator.
From docs:-
The use of iterators pervades and unifies Python. Behind the scenes, the for statement calls iter()
on the container object. The function returns an iterator object that defines the method __next__()
which accesses elements in the container one at a time. When there are no more elements, __next__()
raises a StopIteration exception which tells the for loop to terminate. You can call the __next__()
method using the next()
built-in function; this example shows how it all works:
>>> s = 'abc'
>>> it = iter(s)
>>> it
<iterator object at 0x00A1DB50>
>>> next(it)
'a'
>>> next(it)
'b'
>>> next(it)
'c'
>>> next(it)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
next(it)
StopIteration
Ex of a class:-
class Reverse:
"""Iterator for looping over a sequence backwards."""
def __init__(self, data):
self.data = data
self.index = len(data)
def __iter__(self):
return self
def __next__(self):
if self.index == 0:
raise StopIteration
self.index = self.index - 1
return self.data[self.index]
>>> rev = Reverse('spam')
>>> iter(rev)
<__main__.Reverse object at 0x00A1DB50>
>>> for char in rev:
... print(char)
...
m
a
p
s
Iterators are objects that implement the iter and next methods. If those methods are defined, we can use for loop or comprehensions.
class Squares:
def __init__(self, length):
self.length = length
self.i = 0
def __iter__(self):
print('calling __iter__') # this will be called first and only once
return self
def __next__(self):
print('calling __next__') # this will be called for each iteration
if self.i >= self.length:
raise StopIteration
else:
result = self.i ** 2
self.i += 1
return result
Iterators get exhausted. It means after you iterate over items, you cannot reiterate, you have to create a new object. Let's say you have a class, which holds the cities properties and you want to iterate over.
class Cities:
def __init__(self):
self._cities = ['Brooklyn', 'Manhattan', 'Prag', 'Madrid', 'London']
self._index = 0
def __iter__(self):
return self
def __next__(self):
if self._index >= len(self._cities):
raise StopIteration
else:
item = self._cities[self._index]
self._index += 1
return item
Instance of class Cities is an iterator. However if you want to reiterate over cities, you have to create a new object which is an expensive operation. You can separate the class into 2 classes: one returns cities and second returns an iterator which gets the cities as init param.
class Cities:
def __init__(self):
self._cities = ['New York', 'Newark', 'Istanbul', 'London']
def __len__(self):
return len(self._cities)
class CityIterator:
def __init__(self, city_obj):
# cities is an instance of Cities
self._city_obj = city_obj
self._index = 0
def __iter__(self):
return self
def __next__(self):
if self._index >= len(self._city_obj):
raise StopIteration
else:
item = self._city_obj._cities[self._index]
self._index += 1
return item
Now if we need to create a new iterator, we do not have to create the data again, which is cities. We creates cities object and pass it to the iterator. But we are still doing extra work. We could implement this by creating only one class.
Iterable is a Python object that implements the iterable protocol. It requires only __iter__()
that returns a new instance of iterator object.
class Cities:
def __init__(self):
self._cities = ['New York', 'Newark', 'Istanbul', 'Paris']
def __len__(self):
return len(self._cities)
def __iter__(self):
return self.CityIterator(self)
class CityIterator:
def __init__(self, city_obj):
self._city_obj = city_obj
self._index = 0
def __iter__(self):
return self
def __next__(self):
if self._index >= len(self._city_obj):
raise StopIteration
else:
item = self._city_obj._cities[self._index]
self._index += 1
return item
Iterators has __iter__
and __next__
, iterables have __iter__
, so we can say Iterators are also iterables but they are iterables that get exhausted. Iterables on the other hand never become exhausted
because they always return a new iterator that is then used to iterate
You notice that the main part of the iterable code is in the iterator, and the iterable itself is nothing more than an extra layer that allows us to create and access the iterator.
Python has a built function iter() which calls the __iter__()
. When we iterate over an iterable, Python calls the iter() which returns an iterator, then it starts using __next__()
of iterator to iterate over the data.
NOte that in the above example, Cities creates an iterable but it is not a sequence type, it means we cannot get a city by an index. To fix this we should just add __get_item__
to the Cities class.
class Cities:
def __init__(self):
self._cities = ['New York', 'Newark', 'Budapest', 'Newcastle']
def __len__(self):
return len(self._cities)
def __getitem__(self, s): # now a sequence type
return self._cities[s]
def __iter__(self):
return self.CityIterator(self)
class CityIterator:
def __init__(self, city_obj):
self._city_obj = city_obj
self._index = 0
def __iter__(self):
return self
def __next__(self):
if self._index >= len(self._city_obj):
raise StopIteration
else:
item = self._city_obj._cities[self._index]
self._index += 1
return item
iterable = [1, 2]
iterator = iter(iterable)
print(iterator.__next__())
print(iterator.__next__())
so,
iterable
is an object that can be looped over. e.g. list , string , tuple etc.
using the iter
function on our iterable
object will return an iterator object.
now this iterator object has method named __next__
(in Python 3, or just next
in Python 2) by which you can access each element of iterable.
so, OUTPUT OF ABOVE CODE WILL BE:
1
2
To see if the object has this method iter() we can use the below function.
ls = ['hello','bye']
print(dir(ls))
Output
['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']
As you can see has the iter() that's mean that is a iterable object, but doesn't contain the next() method which is a feature of the iterator object
Whenever you use a for loop or map or a list comprehension in Python the next method is called automatically to get each item from the iteration
Before dealing with the iterables and iterator the major factor that decide the iterable and iterator is sequence
Sequence: Sequence is the collection of data
Iterable: Iterable are the sequence type object that support __iter__
method.
Iter method: Iter method take sequence as an input and create an object which is known as iterator
Iterator: Iterator are the object which call next method and transverse through the sequence. On calling the next method it returns the object that it traversed currently.
example:
x=[1,2,3,4]
x is a sequence which consists of collection of data
y=iter(x)
On calling iter(x)
it returns a iterator only when the x object has iter method otherwise it raise an exception.If it returns iterator then y is assign like this:
y=[1,2,3,4]
As y is a iterator hence it support next()
method
On calling next method it returns the individual elements of the list one by one.
After returning the last element of the sequence if we again call the next method it raise an StopIteration error
example:
>>> y.next()
1
>>> y.next()
2
>>> y.next()
3
>>> y.next()
4
>>> y.next()
StopIteration
Here's another view using collections.abc
. This view may be useful the second time around or later.
From collections.abc
we can see the following hierarchy:
builtins.object
Iterable
Iterator
Generator
i.e. Generator is derived from Iterator is derived from Iterable is derived from the base object.
Hence,
[1, 2, 3]
and range(10)
are iterables, but not iterators. x = iter([1, 2, 3])
is an iterator and an iterable.iter()
on an iterator or a generator returns itself. Thus, if it
is an iterator, then iter(it) is it
is True.[2 * x for x in nums]
or a for loop like for x in nums:
, acts as though iter()
is called on the iterable (nums
) and then iterates over nums
using that iterator. Hence, all of the following are functionally equivalent (with, say, nums=[1, 2, 3]
):
for x in nums:
for x in iter(nums):
for x in iter(iter(nums))
:for x in iter(iter(iter(iter(iter(nums))))):
Other people already explained comprehensively, what is iterable and iterator, so I will try to do the same thing with generators.
IMHO the main problem for understanding generators is a confusing use of the word “generator”, because this word is used in 2 different meanings:
yield
statement(s) in its body),Generator as a tool of the 1st type:
In[2]: def my_generator():
...: yield 100
...: yield 200
In[3]: my_generator
Out[3]: <function __main__.my_generator()>
In[4]: type(my_generator)
Out[4]: function
Generator as a result (i.e. an iterator) of the use of this tool:
In[5]: my_iterator = my_generator()
In[6]: my_iterator
Out[6]: <generator object my_generator at 0x00000000053EAE48>
In[7]: type(my_iterator)
Out[7]: generator
Generator as a tool of the 2nd type — indistinguishable from the resulting iterator of this tool:
In[8]: my_gen_expression = (2 * i for i in (10, 20))
In[9]: my_gen_expression
Out[9]: <generator object <genexpr> at 0x000000000542C048>
In[10]: type(my_gen_expression)
Out[10]: generator
For me, Python's glossary was most helpful for these questions, e.g. for iterable it says:
An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as list, str, and tuple) and some non-sequence types like dict, file objects, and objects of any classes you define with an iter() method or with a getitem() method that implements Sequence semantics.
Iterables can be used in a for loop and in many other places where a sequence is needed (zip(), map(), …). When an iterable object is passed as an argument to the built-in function iter(), it returns an iterator for the object. This iterator is good for one pass over the set of values. When using iterables, it is usually not necessary to call iter() or deal with iterator objects yourself. The for statement does that automatically for you, creating a temporary unnamed variable to hold the iterator for the duration of the loop. See also iterator, sequence, and generator.
© 2022 - 2024 — McMap. All rights reserved.
__getitem__
method is considered iterable, but any existing__len__
method is completely ignored. Principle of least astonishment failing hard here... – Grevera