Generators_and_Coroutines


Silicon Forest Math Series
Oregon Curriculum Network

Generators and Coroutines

Generator functions relate to generator expressions, objects which delay exectution until pressed into service as iterators, a type with a __next__ method.

The for loop counts on an iterator for its target and implicitly applies iter(obj) to whatever it gets. Iterators save on memory as they know how to share and underlying iterable.

Some iterators also have the ability to function as objects able to pause and resume operations, without forgetting their internal state.


In [1]:
powers = (lambda x: pow(x, n) for n in range(-4,5))
phi = (1 + pow(5,0.5)) * 0.5 # golden proportion
for n, f in enumerate(powers, start=-4):  # iterates through lambda expressions
    print("phi ** {:2} == {:10.8f}".format(n, f(phi)))


phi ** -4 == 0.14589803
phi ** -3 == 0.23606798
phi ** -2 == 0.38196601
phi ** -1 == 0.61803399
phi **  0 == 1.00000000
phi **  1 == 1.61803399
phi **  2 == 2.61803399
phi **  3 == 4.23606798
phi **  4 == 6.85410197

Any object expecting to be the target of a for loop, if not already an iterator, needs to either:

  • dedicate an __iter__ method to showing what to return when iter() gets applied, or
  • have a __getitem__ ready to go, as iter( ) is smart enough to make up an object wherein consecutive integers, starting from 0, go to any __getitem__ method.

In [2]:
class Any:
    
    def __init__(self):
        self.__dict__ = {0:'scissors', 1:'paper', 2:'rock'}
    
    def __getitem__(self, n):  # enough for iter() to go on
        if n == len(self.__dict__):
            raise StopIteration  # tells for loop when to stop
        return self.__dict__[n]
    
for thing in Any():
    print(thing)


scissors
paper
rock

The generator function below shows what most Pythonistas consider the base case for the keyword yield: using it much like return, to provide an object to the generator function's caller, in this case a next prime number, going in sequence.

The trial-by-division algorithm requires keeping a growing sequence of successive primes, and using them to test new candidates. After taking care of the only even prime, 2, it makes sense to jump forward through the odds, screening out all the composites.


In [1]:
import pprint

def primes():
    """generate successive prime numbers (trial by division)"""
    candidate = 1
    _primes_so_far = [2]     # first prime, only even prime
    yield _primes_so_far[0]  # share it!
    while True:
        candidate += 2    # check odds only from now on
        for prev in _primes_so_far:
            if prev**2 > candidate:
                yield candidate  # new prime!
                _primes_so_far.append(candidate)
                break
            if not divmod(candidate, prev)[1]: # no remainder!
                break                          # done looping
                
p = primes()  # generator function based iterator
pp = pprint.PrettyPrinter(width=40, compact=True)
pp.pprint([next(p) for _ in range(50)])  # next 30 primes please!


[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31,
 37, 41, 43, 47, 53, 59, 61, 67, 71, 73,
 79, 83, 89, 97, 101, 103, 107, 109,
 113, 127, 131, 137, 139, 149, 151, 157,
 163, 167, 173, 179, 181, 191, 193, 197,
 199, 211, 223, 227, 229]

Now lets take a look at pretty much the same iterator written a different way: with an explicit __iter__ and __next__.


In [4]:
class Primes:

    def __init__(self):
        self.candidate = 1
        self._primes_so_far = [2]  # first prime, only even prime

    def __iter__(self):
        return self
        
    def __next__(self):
        while True:
            self.candidate += 2    # check odds only from now on
            for prev in self._primes_so_far:
                if prev**2 > self.candidate:
                    self._primes_so_far.append(self.candidate)
                    return self._primes_so_far[-2]    
                if not divmod(self.candidate, prev)[1]: # no remainder!
                    break

pp = pprint.PrettyPrinter(width=40, compact=True)
p = Primes()  # class based iterator
pp.pprint([next(p) for _ in range(30)])  # n


[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31,
 37, 41, 43, 47, 53, 59, 61, 67, 71, 73,
 79, 83, 89, 97, 101, 103, 107, 109,
 113]

Feeding an open-ended iterator to a for loop runs the risk of a "forever loop". The itertools module is filled with tools designed to rein in the infinite loopers. Just use islice(obj, start, stop) to keep the for loop finite:


In [5]:
from itertools import islice
p = Primes()
for n in islice(p, 0, 20):
    print(n, end=", ")


2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 

In the code below, we see (yield) turned around and used not to return objects, but to receive them. The parentheses are by convention and suggest a "mouth" bringing something in from outside.

A generator function's .send() method resumes its execution inside its body with an intake of some passed-in object from the caller, the argument to send. In the above example, two callers are nested, the inner one writing a prime to a file, the outer one feeding it next primes for said catalog.

The coroutine decorator may seem a bit mysterious at first. A generator function does not run any of its code upon being instanced. No yield statement has yet been encountered, so use of .send(obj) would raise an exception were obj any object but None.

The decorator has already fed a .send(None) to the generator in question, equivalent to feeding it to next() a first time. The decorator applies a first action somewhat like cocking a pistol, putting the generator or coroutine in the firing position, positioned at a first yield.


In [6]:
# -*- coding: utf-8 -*-
"""
Created on Thu Oct 13 13:48:52 2016

@author: Kirby Urner

David Beazley:
https://youtu.be/Z_OAlIhXziw?t=23m42s

Trial by division, but this time the primes coroutine acts 
more as a filter, passing qualified candidates through to
print_me, which writes to a file.
"""
import pprint

def coroutine(func):
    """
    Advances decorated generator function to the first yield
    """
    def start(*args, **kwargs):
        cr = func(*args, **kwargs)
        cr.send(None)  # or next(cr) or cr.__next__()
        return cr
    return start
        
@coroutine
def print_me(file_name):
    with open(file_name, 'w') as file_obj:
        while True:
            to_print = (yield)
            file_obj.write(str(to_print)+"\n")
    
@coroutine
def primes(target):
    _primes_so_far = [2]
    target.send(2)
    while True:
        candidate = (yield)
        for prev in _primes_so_far:
            if not divmod(candidate, prev)[1]:
                break
            if prev**2 > candidate:
                _primes_so_far.append(candidate)
                target.send(candidate)
                break

output = print_me("primes.txt")
p = primes(output)

for x in range(3, 200, 2):  # test odds 3-199
    p.send(x)

with open("primes.txt", 'r') as file_obj:
    print(", ".join(file_obj.read().split("\n"))[:-2])



In the top example, the generator provides a mechanism for advancing to a next odd number for testing, whereas the coroutine version consumes the odds we feed it, passing through those which pass the test for printing, as well as accumulating them internally.

For Further Reading

Linked Jupyter Notebooks: