To my knowledge, Python 3.3 was the first in the Python 3.x to introduce brand new functionality / language syntax extensions for generators. This was the ability to delegate to subgenerators, via the new yield from construct. This was proposed in PEP 380.
At its most basic, yield from can be used to make it easier to correctly iterate through, and yield values from another iterable.
Any time the iterable in the yield from expression yields a value, the generator immediately yields that same value. Execution continues inside the iterable, until it is exhausted. After that point, further execution may continue inside the outer generator.
In [1]:
def chain_with_yield(*iterables):
for iterable in iterables:
for item in iterable:
yield item
list(chain_with_yield(range(4), range(4, 8)))
Out[1]:
In [2]:
def chain_with_yield_from(*iterables):
for iterable in iterables:
yield from iterable
list(chain_with_yield_from(range(4), range(4, 8)))
Out[2]:
The naive substitute for yield from
for item in iterable:
yield item
is only sufficient if the outer generator is being treated as a normal iterator. If you want to use send() or throw(), they will only have an effect on the outer generator. Correctly delegating to the in-progress subgenerator requires very tricky and ugly code.
But when using yield from, the Python interpretter handles that delegation for you. Calling send() or throw() delegate through all yield from expressions in the current stack, all the way to the deepest yield expression, and then perform the send() or throw() at that location.
An interesting consequence is that the outer generator loses all control of execution until the subgenerator is exhausted. Calls to __next__(), send(), and throw() will always be proxied directly to the subgenerator. The outer generator has no say in this behavior, and has no way to resume control until the subgenerator raises StopIteration or some other exception.
In [3]:
import traceback
class IgnorableError(Exception): pass
class StopSubGenerator(Exception): pass
def yield_sent_values():
value = None
while True:
print('yielding', value)
try:
value = (yield value)
except (StopSubGenerator, GeneratorExit) as exc:
print(f'Received {exc.__class__.__name__}, returning')
return
except Exception as exc:
print('Received', repr(exc), 'and will re-raise')
raise
print('received', value)
def delegator_function():
while True:
try:
yield from yield_sent_values()
except IgnorableError:
print('Caught IgnorableError from subgenerator, restarting subgenerator')
except GeneratorExit:
print('Received GeneratorExit from subgenerator, returning')
return
except Exception as exc:
print('Received', repr(exc), 'from subgenerator, re-raising')
raise
else:
print('Subgenerator stopped cleanly, restarting subgenerator')
delegator = delegator_function()
next(delegator)
print(delegator.send(3))
print(delegator.send(4))
print(delegator.throw(IgnorableError))
print(delegator.throw(StopSubGenerator))
delegator.close()
Just like yield, the yield from construct can be used as an expression. It can be assigned to a variable, passed to another function, etc.
The value is the object that gets returned as the subgenerator exits, or the .value attribute of the raised StopIteration.
Again ignoring the semantics of send() and throw(), this means that RESULT = yield from EXPR is similar to the following code.
iter = (EXPR).__iter__()
running = True
while running:
try:
yield iter.__next__()
except StopIteration as exc:
RESULT = exc.value
running = False
In [4]:
def subgenerator():
yield 'yielded_value'
yield '1'
yield '2'
return 'returned_value'
def generator():
value = yield from subgenerator()
print('generator received final return value', value, 'from subgenerator')
for item in generator():
print('generator yielded', item)
In [5]:
import collections
class SubGenerator(collections.Generator):
def send(self, value):
raise StopIteration('returned_value')
def throw(self, *exc_info):
next(self)
def generator():
value = yield from SubGenerator()
print('generator received final return value', value, 'from SubGenerator')
for item in generator():
pass
Before PEP 380 and Python 3.3, there was no StopIteration.value attribute, and generators could not return values. Thus, the main purposes of generators were to serve as a special syntax for defining an iterator, and in advanced cases to serve as a general mechanism for suspending and resuming execution of a block of code.
Now that generators can return values, they are able to take on a new meaning. A generator which returns a value can be thought of as a computation that produces a result, but which suspends itself in the middle of that computation. It might do this in order to
yield an intermediate value;send(), throw(), and close();We already showed an example of 1., with the generator-based adders that we defined in the previous chapter. The concept of 2. is a bit more complex, so we will save that for an upcoming chapter. The generator-based adder also shows off 3., though for another example see <http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Parser/parser.txt>. And we'll show off an example 4. in the very next chapter.
In the case of 3. and 4., the outer executor might not even care about the values that get yielded from the generator, and the generator may just be able to yield without a value.
The language of PEP 380 refers to these behaviors in an interesting way. The proposal document has a section titled "Generators as Threads". Here is a quote of the most interesting part of that section:
A motivation for generators being able to return values concerns the use of generators to implement lightweight threads. When using generators in that way, it is reasonable to want to spread the computation performed by the lightweight thread over many functions. One would like to be able to call a subgenerator as though it were an ordinary function, passing it parameters and receiving a returned value.
Using the proposed syntax, a statement such as
y = f(x)where f is an ordinary function, can be transformed into a delegation call
y = yield from g(x)where g is a generator. One can reason about the behaviour of the resulting code by thinking of g as an ordinary function that can be suspended using a yield statement.
When they say "thread", they don't mean a threading.Thread or an operating system thread. Rather, they are referring to a computation, procedure, or program that can be run concurrently with other such threads. This concept will be explored when we talk about event loops and async tasks.
A related line of thinking is to view these generators as futures. Suppose again that g is a generator function which returns a value. Then y = yield from g(x) assigns that return value to y. However, since the computation of g(x) involves yield and possibly yield from statements, it will suspend itself one or more times, and the final result will not be available right away. But when the final result is available, it is immediately assigned to y, and computation of that function resumes. This isn't quite the same as a future, since futures are objects that can be passed around, mapped, assigned callbacks, etc.
When a function returns a future, the caller's execution is not suspended until/unless the caller decides to explicitly wait on the result. When thinking of generators as futures, y = yield from g(x) is analagous to future = g(x); y = future.block_and_get_result().
When we explore event loops, we'll see an actual future that takes advantage of yield from.
Unfortunately, not everything is sunshine and rainbows, even with the introduction of yield from. Say you are writing a function f, and you want to call a function g that produces some return value. However, g is a generator function, whereas f is currently a normal function. Here, you have three options:
y = yield from g() to get the return value. However, this will turn f into a generator function, which means that its caller(s) need to run f or must themselves yield from f(), and so on and so forth.Run g() through to completion, perhaps manually, or perhaps using a helper function like
def run_until_complete(generator):
try:
while True:
next(generator)
except StopIteration as exc:
return exc.value
This has the downside that your program loses out on the ability to possibly interleave other computations and do concurrent programming.
Reimplement g as a non-generator function, and call that function instead.
Going with 1. isn't necessarily a bad thing. But it does show how using generators to represent suspendable computations can be infectious. Later we'll see how this is necessary in order to gain the ability to do async programming.
License: Apache License, Version 2.0
Jordan Moldow, 2017
Copyright 2017 Jordan Moldow Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.