[ Python 常見問題 ] In practice, what are the main uses for the new “yield from” syntax in Python 3.3?

Source From Here
Question
I'm having a hard time wrapping my brain around PEP 380.

* What are the situations where "yield from" is useful?
* What is the classic use case?
* Why is it compared to micro-threads?

How-To
Let's get one thing out of the way first. The explanation that yield from g is equivalent to for v in g: yield v does not even begin to do justice to what yield from is all about. Because, let's face it, if all yield from does is expand the for loop, then it does not warrant adding yield from to the language and preclude a whole bunch of new features from being implemented in Python 2.x.

What yield from does is it establishes a transparent bidirectional connection between the caller and the sub-generator:

* The connection is "transparent" in the sense that it will propagate everything correctly too, not just the elements being generated (e.g. exceptions are propagated).
* The connection is "bidirectional" in the sense that data can be both sent from and to a generator.

BTW, if you are not sure what sending data to a generator even means, you need to drop everything and read about coroutines first—they're very useful (contrast them with subroutines), but unfortunately lesser-known in Python. Dave Beazley's Curious Course on Coroutines is an excellent start. Read slides 24-33 for a quick primer.

Reading data from a generator using yield from

view plaincopy to clipboardprint?

1. def reader():  
2.     """A generator that fakes a read from a file, socket, etc."""  
3.     for i in range(4):  
4.         yield '<< %s' % i  
5.   
6. def reader_wrapper(g):  
7.     # Manually iterate over data produced by reader  
8.     for v in g:  
9.         yield v  
10.   
11. wrap = reader_wrapper(reader())  
12. for i in wrap:  
13.     print(i)  

Output:

<< 0
<< 1
<< 2
<< 3

Instead of manually iterating over reader(), we can just yield from it:

view plaincopy to clipboardprint?

1. def reader_wrapper(g):  
2.     yield from g  

That works, and we eliminated one line of code. And probably the intent is a little bit clearer (or not). But nothing life changing.

Sending data to a generator (coroutine) using yield from - Part 1
Now let's do something more interesting. Let's create a coroutine called writer that accepts data sent to it and writes to a socket, fd, etc.

view plaincopy to clipboardprint?

1. def writer():  
2.     """A coroutine that writes data *sent* to it to fd, socket, etc."""  
3.     while True:  
4.         w = (yield)  
5.         print('>> ', w)  

Now the question is, how should the wrapper function handle sending data to the writer, so that any data that is sent to the wrapper is transparently sent to the writer()?

view plaincopy to clipboardprint?

1. def writer_wrapper(coro):  
2.     # TBD  
3.     pass  
4.   
5. w = writer()  
6. wrap = writer_wrapper(w)  
7. wrap.send(None)  # "prime" the coroutine  
8. for i in range(4):  
9.     wrap.send(i)  

Expected output:

>> 0
>> 1
>> 2
>> 3

The wrapper needs to accept the data that is sent to it (obviously) and should also handle the StopIteration when the for loop is exhausted. Evidently just doing for x in coro: yield x won't do. Here is a version that works:

view plaincopy to clipboardprint?

1. def writer_wrapper(coro):  
2.     coro.send(None)  # prime the coro  
3.     while True:  
4.         try:  
5.             x = (yield)  # Capture the value that's sent  
6.             coro.send(x)  # and pass it to the writer  
7.         except StopIteration:  
8.             pass  

Or, we could do this.

view plaincopy to clipboardprint?

1. def writer_wrapper(coro):  
2.     yield from coro  

That saves 6 lines of code, make it much much more readable and it just works. Magic!

Sending data to a generator yield from - Part 2 - Exception handling
Let's make it more complicated. What if our writer needs to handle exceptions? Let's say the writer handles a SpamException and it prints *** if it encounters one:

view plaincopy to clipboardprint?

1. class SpamException(Exception):  
2.     pass  
3.   
4. def writer():  
5.     while True:  
6.         try:  
7.             w = (yield)  
8.         except SpamException:  
9.             print('***')  
10.         else:  
11.             print('>> ', w)  

What if we don't change writer_wrapper? Does it work? Let's try:

view plaincopy to clipboardprint?

1. # writer_wrapper same as above  
2.   
3. w = writer()  
4. wrap = writer_wrapper(w)  
5. wrap.send(None)  # "prime" the coroutine  
6. for i in [0, 1, 2, 'spam', 4]:  
7.     if i == 'spam':  
8.         wrap.throw(SpamException)  
9.     else:  
10.         wrap.send(i)  

Expected Result:

>> 0
>> 1
>> 2
***
>> 4

Actual Result

view plaincopy to clipboardprint?

1. >>  0  
2. >>  1  
3. >>  2  
4. Traceback (most recent call last):  
5.   ... redacted ...  
6.   File ... in writer_wrapper  
7.     x = (yield)  
8. __main__.SpamException  

Um, it's not working because x = (yield) just raises the exception and everything comes to a crashing halt. Let's make it work, but manually handling exceptions and sending them or throwing them into the sub-generator (writer):

view plaincopy to clipboardprint?

1. def writer_wrapper(coro):  
2.     """Works. Manually catches exceptions and throws them"""  
3.     coro.send(None)  # prime the coro  
4.     while True:  
5.         try:  
6.             try:  
7.                 x = (yield)  
8.             except Exception as e:   # This catches the SpamException  
9.                 coro.throw(e)  
10.             else:  
11.                 coro.send(x)  
12.         except StopIteration:  
13.             pass  

This works.

>> 0
>> 1
>> 2
***
>> 4

But so does this!

view plaincopy to clipboardprint?

1. def writer_wrapper(coro):  
2.     yield from coro  

The yield from transparently handles sending the values or throwing values into the sub-generator.

This still does not cover all the corner cases though. What happens if the outer generator is closed? What about the case when the sub-generator returns a value (yes, in Python 3.3+, generators can return values), how should the return value be propagated? That yield from transparently handles all the corner cases is really impressive. yield from just magically works and handles all those cases.

In summary, it's best to think of yield from as a transparent two way channel between the caller and the sub-generator. References:

* PEP 380 - Syntax for delegating to a sub-generator (Ewing) [v3.3, 2009-02-13]
* PEP 342 - Coroutines via Enhanced Generators (GvR, Eby) [v2.5, 2005-05-10]

Supplement
* Python3: 淺談 Python 3.3 的 Yield From 表達式