Lesson_06_iterators_generators


Python Training - Lesson 6 - Iterables, Iteration, Generators, Generator Expressions

Iteration

Examples of what iteration can look like


In [ ]:
a = 10
while a > 0:
    print(a)
    a -= 1

In [ ]:
for a in range(0,10):
    print(a)

In [ ]:
[print(a) for a in range(0,10)]

Silly example

Let's say you need to stop your loop when you have a specific condition.


In [35]:
for i in range(0,500):
    if i%376==0:
        print("Got it!")
        break


Got it!

.... and sometimes the condition changes?


In [39]:
from random import randint

divisor = randint(1,500)

for i in range(1,500):
    if i%divisor==0:
        print("Got it: " + str(i))
        break


Got it: 266

It seems we don't need to generate the big list at all. Imagine it took some time, like 0.5 seconds for each element to check. Do we need to have them all before we check?

We need a lazy evaluation, not eager.

Am I done yet? No? Then give me the next one. Just the one.

Iteration protocol


In [53]:
# Normal method of using collections - just generate all at once
a = [1,2,3,4,5]

In [54]:
# Let's change the list into an ITERATOR
my_iterator = iter(a)
print(type(my_iterator))
print(my_iterator)


<class 'list_iterator'>
<list_iterator object at 0x047A05F0>

In [55]:
# Let's iterate the iterator
next(my_iterator)


Out[55]:
1

In [56]:
# And some more:
next(my_iterator)
next(my_iterator)
next(my_iterator)
next(my_iterator)


Out[56]:
5

In [57]:
# ... and more:
next(my_iterator)


---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-57-1780c22948c2> in <module>()
      1 # ... and more:
----> 2 next(my_iterator)

StopIteration: 

Woops! When iterator is done, it always throws a STOP sign.

A better approach is to just iterate over the iterator.


In [58]:
some_list = [1,2,3,4,5,5]
some_iter = iter(some_list)

for x in some_iter:
    print(x)


1
2
3
4
5
5

What's going on under the hood?

Let's try and implement something like that. A lazy range.

To mimic the behavior of "next" and "iter", we implement such methods ourselves, but use the built-in Python functions with two underscores.


In [66]:
class lazy_range:
    def __init__(self):
        self.counter = 0
    
    def __iter__(self):
        return self
    
    def __next__(self):
        self.counter += 1
        return self.counter

In [67]:
# Something like iter(some_list)

my_lazy_range = lazy_range()
next(my_lazy_range)


Out[67]:
1

In [68]:
for x in lazy_range():
    print(x)
    if x >10:
        break


1
2
3
4
5
6
7
8
9
10
11

That looks terrible. Is there an easy way?

Generators


In [82]:
def laziest_range(N):
    i = 0
    while i<N:
        yield i
        i += 1

In [83]:
a = laziest_range(5)

In [84]:
next(a)


Out[84]:
0

In [85]:
next(a)


Out[85]:
1

In [86]:
for x in laziest_range(10):
    print(x)


0
1
2
3
4
5
6
7
8
9

Generator expressions


In [87]:
laziest_shortest_range = (x for x in range(10))

In [88]:
next(laziest_shortest_range)


Out[88]:
0

In [89]:
next(laziest_shortest_range)


Out[89]:
1

Comparison of sizes


In [91]:
a = [x for x in range(0,100000)]
b = (x for x in range(0,100000))

In [94]:
import sys
print(sys.getsizeof(a))
print(sys.getsizeof(b))


412236
48

Meaning of 'yield'


In [97]:
def semaphore():
    print("You go")
    yield
    print("No, you go")
    yield
    print("No, really, you go!")
    yield

In [100]:
a = semaphore()

In [104]:
next(a)


You go

In [105]:
next(a)


No, you go

In [106]:
next(a)


No, really, you go!

Examples with time


In [114]:
from time import sleep
from random import randint

def get_latest_customers():
    rv = []
    for i in range(10):  
        sleep(.5)
        rv.append(randint(1,20))
    return rv


customer_id = 18

In [115]:
# We gather lot's of customer ID's and then check if our customer ID=19 is there.

print(customer_id in get_latest_customers())


True

In [117]:
def get_latest_customers_lazy(n):
    for i in range(n):  
        sleep(.5)
        yield(randint(1,20))

In [118]:
iterator = get_latest_customers_lazy(2000000000)

In [120]:
for x in iterator:
    if customer_id is x:
        print("Found him!")
        break
    else:
        print("Still looking..." + str(x))


Still looking...20
Still looking...19
Still looking...17
Still looking...4
Still looking...2
Still looking...20
Still looking...15
Still looking...1
Still looking...7
Still looking...1
Still looking...1
Still looking...7
Still looking...10
Still looking...5
Still looking...11
Still looking...1
Still looking...20
Still looking...8
Still looking...14
Still looking...5
Still looking...10
Still looking...7
Still looking...2
Still looking...19
Still looking...2
Still looking...10
Still looking...11
Still looking...19
Still looking...5
Still looking...20
Still looking...14
Still looking...17
Still looking...8
Still looking...5
Still looking...5
Still looking...17
Still looking...3
Still looking...17
Still looking...12
Still looking...4
Found him!

In [ ]: