Python generators

A generator is most easily defined with a function that uses "yield" instead of "return". When the function is called, it is not immediately run. Instead, an iterator is automatically created and returned.

Calling next() on that iterator enters the function and runs it until a yield statement is encountered. The value that is yielded (or None in the case of a bare "yield" statement) becomes the return value for next(), and execution of the function pauses after the call to yield. The next time that next() is called, execution resumes inside the function where it left off during the previous call to next(). If you've worked with continuations, you will likely notice parallels between continuations and generators.

If a call to next() ever falls off the end of the function, StopIteration is raised. The required __iter__ method is also defined automatically.

Generators offer us a simple technique for implementing an iterator since they handle the details of the iterator protocol for us.

Definining our counter using a generator

Definition

def counter (start=0, stop=None, by=1):
    if stop is None:
        while True:
            yield start
            start += by
    else:
        while start < stop:
            yield start
            start += by

Usage is the same

cntr = counter(0, 10)
cntr.next() # 0
cntr.next() # 1
for n in cntr:
    print n,
# prints 2 3 4 5 6 7 8 9

Advantages

The implementation is much shorter, with all iterator boilerplate effectively removed. The only code we have now is the critical stuff that describes how we iterate.
Since execution continues where it left off on subsequent calls to next(), we can easily test for "stop is None" just one time instead of at each iteration, improving efficiency slightly. To accomplish this with our iterator counter, we would have been forced to introduce even more complexity.

In order to easily compare all counter versions (including the aforementioned more-efficient-but-complex iterator version), see counters.py.

Now let's improve our Queue implementation