[Part I] Introduction to Generators and Coroutines

generator


In [17]:
def countdown(n):
    print '> counting down from {}'.format(n)
    while n > 0:
        yield n
        n -= 1
    print ''
    print '< countdown'

In [18]:
for n in countdown(10):
    print n,


> counting down from 10
10 9 8 7 6 5 4 3 2 1 
< countdown
  • generator 함수를 호출하는것은 generator 객체를 생성하는것이지 함수를 실행하는 것이 아님
  • generator.next() 를 호출하면 함수가 실행되고,
  • yield 를 통해서 값을 생성하고, 함수의 실행을 잠시 중단하고,
  • .next() 호출을 통해 실행을 재개한다.
  • generator 가 리턴하면, iteration 은 멈춘다.

In [26]:
# calling generator fucntion creates the generator object not start the function
x = countdown(3)
print x

# call `.next()` starts generator object.
print x.next()
print x.next()
print x.next()
print x.next()


<generator object countdown at 0x035BD8F0>
> counting down from 3
3
2
1

< countdown
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-26-b93f85de60e0> in <module>()
      7 print x.next()
      8 print x.next()
----> 9 print x.next()
     10 

StopIteration: 

tail -f (python version)


In [32]:
import os
print os.getcwd()


d:\work.python\ipython_notebook\python_async_stuffs

In [37]:
"""
generator 강의자료에 run/foo/xxx 등이 있는데 거기에 있는걸 복사해서 실행환경을 만들면 됨
"""
# follow.py
#
# Follow a file like tail -f.

import time

def follow(thefile):
    thefile.seek(0,2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line

# Example use
# Note : This example requires the use of an apache log simulator.
# 
# Go to the directory run/foo and run the program 'logsim.py' from
# that directory.   Run this program as a background process and
# leave it running in a separate window.  We'll write program
# that read the output file being generated
# 

#logfile = open("run/foo/access-log","r")
logfile = open(r'd:\work.python\python_async_stuffs\coroutine_www.dabeaz.com\run\foo\access-log')
for line in follow(logfile):
    print line,


---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-37-857ca5c48920> in <module>()
     28 #logfile = open("run/foo/access-log","r")
     29 logfile = open(r'd:\work.python\python_async_stuffs\coroutine_www.dabeaz.com\run\foo\access-log')
---> 30 for line in follow(logfile):
     31     print line,

<ipython-input-37-857ca5c48920> in follow(thefile)
     13         line = thefile.readline()
     14         if not line:
---> 15             time.sleep(0.1)
     16             continue
     17         yield line

KeyboardInterrupt: 

Coroutine

  • yield 를 표현식으로,
  • next() 또는 send(None) 메소드로 함수를 시작하고
  • yield 를 통해 값을 대기하고, send() 메소드로 값을 전달

In [40]:
def grep(pattern):
    print 'looking for %s' % pattern
    while True:
        line = (yield)
        if pattern in line:
            print line,

In [47]:
g = grep("python")
g.next()   # prime it!
g.send("hey!!")
g.send("welcome to koread")
g.send("wow python rocks!")
g.send("really?")


looking for python
wow python rocks!
  • next() 는 까먹기 쉬우니까 decorator 를 이용하자.
  • 종료시에는 close() 를 호출하자 (가비지 컬렉터가 알아서 close() 를 호출하지만)
  • close() 를 호출하면 GeneratorExit 예외가 발생하니까 coroutine 에서 잡아준다.

In [55]:
def coroutine(func):
    """
    
    """
    def start_coroutine(*args, **kwargs):
        crtn = func(*args, **kwargs);
        crtn.next()
        return crtn
    return start_coroutine

In [77]:
@coroutine
def grep(pattern):
    print '[*] looking for %s' % pattern
    try:
        while True:
            line = (yield)
            if pattern in line:
                print line,
    except GeneratorExit as e:
        print '\n[*] Going away, bye (gc or U called close())'

In [79]:
g = grep('python')
#g.next()   # no need to call next()
g.send("hey!!")
g.send("welcome to koread")
g.send("wow python rocks!")
g.close()


[*] looking for python
wow python rocks! 
[*] Going away, bye (gc or U called close())

예외 던지기


In [83]:
g = grep('python')
g.send('wow! python is rock!')
g.send('wow! python is rock!')
g.send('wow! python is rock!')
g.throw(RuntimeError, "exception thrown")


[*] looking for python

[*] Going away, bye (gc or U called close())
wow! python is rock! wow! python is rock! wow! python is rock!
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-83-573c8937e6d6> in <module>()
      3 g.send('wow! python is rock!')
      4 g.send('wow! python is rock!')
----> 5 g.throw(RuntimeError, "exception thrown")

<ipython-input-77-849e2bda9321> in grep(pattern)
      4     try:
      5         while True:
----> 6             line = (yield)
      7             if pattern in line:
      8                 print line,

RuntimeError: exception thrown

bogus sample


In [85]:
# bogus.py
#
# Bogus example of a generator that produces and receives values

def countdown(n):
    print "Counting down from", n
    while n >= 0:
        newvalue = (yield n)
        # If a new value got sent in, reset n with it
        if newvalue is not None:
            n = newvalue
        else:
            n -= 1

# The holy grail countdown
c = countdown(5)
for x in c:
    print x
    if x == 5:
        c.send(3)


Counting down from 5
5
2
1
0

정리

  • generator 는 iteration 을 위해 데이터를 생성한다. (producer)
  • coroutine 은 데이터의 사용자이다. (consumer)
  • coroutine 은 iteration 과 아무 상관없다.

[Part II] Coroutines, Pipelines, and Dataflow

  • coroutine 은 파이프를 설치하는데 사용할 수 있음
  • send() -> [coroutine] -> send() -> [coroutine] ->

example


In [13]:
import time

def coroutine(func):
    """A decorator function that takes care of starting a coroutine
    automatically on call.

    """
    def start(*args,**kwargs):
        cr = func(*args,**kwargs)
        cr.next()
        #print 'coroutine started...'
        return cr
    
    return start

# data source
def follow(thefile, target):
    thefile.seek(0, 2)      # goto end of the file
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        
        target.send(line)    # 최초로 호출되는 시점에 객체(target)가 생성됨 
        
# sink - a coroutine that receives data
@coroutine
def printer():
    while True:
        line = (yield)
        print line
        
# useage
f = open(r'd:\work.python\python_async_stuffs\coroutine_www.dabeaz.com\run\foo\access-log')
#follow(f, printer)   # 요거는 오류 남, 'AttributeError: 'function' object has no attribute 'send'
follow( f, printer() )


67.195.44.107 - - [24/Feb/2008:06:00:26 -0600] "GET /robots.txt HTTP/1.0" 200 71

86.157.119.197 - - [24/Feb/2008:06:00:44 -0600] "GET /favicon.ico HTTP/1.1" 404 133

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-13-03233fc77ce1> in <module>()
     35 f = open(r'd:\work.python\python_async_stuffs\coroutine_www.dabeaz.com\run\foo\access-log')
     36 #follow(f, printer)   # 요거는 오류 남, 'AttributeError: 'function' object has no attribute 'send'
---> 37 follow( f, printer() )
     38 

<ipython-input-13-03233fc77ce1> in follow(thefile, target)
     20         line = thefile.readline()
     21         if not line:
---> 22             time.sleep(0.1)
     23             continue
     24 

KeyboardInterrupt: 

Filter example


In [16]:
# data source
def follow(thefile, target):
    thefile.seek(0, 2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue            
        target.send(line)
        
# filter        
@coroutine
def grep(pattern, target):
    while True:
        line = (yield)              # receive a line
        if pattern in line:
            target.send(line)       # send to next stage
        
# sink - a coroutine that receives data
@coroutine
def printer():
    while True:
        line = (yield)
        print line,

# useage
f = open(r'd:\work.python\python_async_stuffs\coroutine_www.dabeaz.com\run\foo\access-log')
follow( f, grep('python', printer()) )


no  128.143.38.123 - - [24/Feb/2008:10:31:14 -0600] "GET /favicon.ico HTTP/1.1" 404 133

no  71.206.180.32 - - [24/Feb/2008:10:31:38 -0600] "GET /ply/ply.html HTTP/1.1" 200 97238

no  71.206.180.32 - - [24/Feb/2008:10:31:40 -0600] "GET /favicon.ico HTTP/1.1" 404 133

no  71.206.180.32 - - [24/Feb/2008:10:31:40 -0600] "GET /favicon.ico HTTP/1.1" 404 133

no  74.6.8.73 - - [24/Feb/2008:10:34:02 -0600] "GET /ply/ply-1.3.1.tar.gz HTTP/1.0" 304 -

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-16-012153cf2e24> in <module>()
     28 # useage
     29 f = open(r'd:\work.python\python_async_stuffs\coroutine_www.dabeaz.com\run\foo\access-log')
---> 30 follow( f, grep('python', printer()) )

<ipython-input-16-012153cf2e24> in follow(thefile, target)
      5         line = thefile.readline()
      6         if not line:
----> 7             time.sleep(0.1)
      8             continue
      9         target.send(line)

KeyboardInterrupt: 

Interlude


In [ ]: