Generators & Iterators

What is a generator (StackOverflow)

  1. A generator is simply a function which returns an object on which you can call next() on, such that for every call it returns some value, until it raises a StopIteration exception, signaling that all values have been generated. Such an object is called an iterator.
  2. Using yield anywhere in a function makes it a generator.

Summary:

  • Generator return an iterator
  • Generators use yield instead of return
  • Iterators can be used with next()

In [1]:
def gencubes(n):
    for num in range(n):
        yield num**3
        
for x in gencubes(10):
    print x


0
1
8
27
64
125
216
343
512
729

In [5]:
# the above is actually *functionally* the same as
def gencubes_2(n):
    out = []
    for num in range(n):
        out.append(num**3)
    return out

for x in gencubes_2(10):
    print x
    
# however, in gencubes_2, a list is created and stored in memory, whereas
# gencubes is maintaining its state and returning one result at a time
# without holding it all in memory


0
1
8
27
64
125
216
343
512
729

In [9]:
def genfibon(n):
    a = 1
    b = 1
    for i in range(n):
        yield a
        temp = a
        a = b
        b = temp + b
    
for i in genfibon(10):
    print i


1
1
2
3
5
8
13
21
34
55

In [13]:
def fibon(n):
    a = 1
    b = 1 
    output = []
    for i in range(n):
        output.append(a)
        a,b = b, a+b
        
    return output
fibon(10)


Out[13]:
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

In [17]:
def simple_gen():
    for x in range(3):
        yield x
g = simple_gen()

print next(g)
print next(g)
print next(g)

# calling this next() a 4th time throws an error for StopIteration
# because there's nothing left to yield. A for-in loop prevents this 
# by stopping when there is nothing left to iterate over
print next(g)


0
1
2
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-17-24c72087f212> in <module>()
      7 print next(g)
      8 print next(g)
----> 9 print next(g)

StopIteration: 

In [18]:
# being an iterator is not the same as being iterable
# for example, a string is iterable, but not an iterator:
s = 'hello'
for let in s:
    print let


h
e
l
l
o

In [19]:
# Running this line prompts the error:
# TypeError: str object is not an iterator
next(s)


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-19-bc0566bea448> in <module>()
----> 1 next(s)

TypeError: str object is not an iterator

In [22]:
s_iter = iter(s)
print next(s_iter)
print next(s_iter)
print next(s_iter)
print next(s_iter)
print next(s_iter)
# one more call will throw a StopIteration error


h
e
l
l
o

In [25]:
def mygen(n):
    yield n
    yield n+1

g = mygen(6)
print next(g)
print next(g)
print next(g) # error


6
7
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-25-30e6d5efeda0> in <module>()
      6 print next(g)
      7 print next(g)
----> 8 print next(g) # error

StopIteration: 

In [39]:
# generator expressions: used to have shorthand for the most common gens
g = (n for n in range(3,6))
print next(g)
print next(g)

# note their similarity to list comprehensions
h = [n for n in range(3, 6)]
print h


3
4
[3, 4, 5]

Observe that a generator object is generated once, but its code is not run all at once. Only calls to next actually execute (part of) the code. Execution of the code in a generator stops once a yield statement has been reached, upon which it returns a value. The next call to next then causes execution to continue in the state in which the generator was left after the last yield. This is a fundamental difference with regular functions: those always start execution at the "top" and discard their state upon returning a value.


In [ ]: