In-depth understanding of Python generators (Generator)

We can create a list simply and directly through list generation, but due to memory constraints, the list capacity is definitely limited. Moreover, creating a list containing 1 million elements not only takes up a lot of storage space, but if we only need to access the first few elements, the space occupied by most of the subsequent elements is wasted.

So, if the list elements can be calculated according to a certain algorithm, can we continuously calculate subsequent elements during the loop? This eliminates the need to create a complete list, saving a lot of space. In Python, this mechanism of looping and calculating at the same time is called a generator.

To create a generator, there are many ways. The first method is very simple. Just change the [] of a list generation expression to () to create a generator:

>>> mylist = [ x for x in range(1, 10)]
>>> mylist
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> gen = (x for x in range(1,10))
>>> gen
<generator object <genexpr> at 0x7f1d7fd0f5a0>

The difference between creating mylist and gen is only the outermost [] and ( ), mylist is a list, and gen is a generator.

We can directly print out each element of the list, but how do we print out each element of the generator?

If you want to print them out one by one, you can use the next() method of the generator:

>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
3
...
>>> gen.next()
9
>>> gen.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

We have said that the generator saves the algorithm. Each time next() is called, the value of the next element is calculated. , until the last element is calculated and there are no more elements, a StopIteration error is thrown.

In fact, we can use for loop instead of next() method, which is more in line with efficient programming ideas:

>>> gen = ( x for x in range(1, 10))
>>> for num in gen:
...     print num
... 
1
2
3
4
5
6
7
8
9

generator is very powerful. If the calculation algorithm is relatively complex and cannot be implemented using a for loop similar to list generation, you can also use a function to implement it.

For example, in the famous Fibonacci sequence, except for the first and second numbers, any number can be obtained by adding the first two numbers:

1, 1, 2, 3, 5, 8, 13, 21, 34, ...

Fi The Boracchi sequence cannot be written using list generation, but it is easy to print it out using a function:

def fib(max):
    n = 0 
    a, b = 0, 1
    while n < max:
        print b
        a, b = b, a + b
        n = n + 1

The above function can output the first N numbers of the Fibonacci sequence:

>>> fib(6)
1
1
2
3
5
8

If you look closely, you can see that the fib function actually defines the calculation rules of the Fibonacci sequence. It can start from the first element and calculate any subsequent elements. This logic is actually very similar to the generator.

In other words, the above function is only one step away from the generator. To turn the fib function into a generator, just change print b to yield b:

def fib(max):
    n = 0 
    a, b = 0, 1
    while n < max:
        yield b
        a, b = b, a + b
        n = n + 1

This is another way to define a generator. If a function definition contains the yield keyword, then the function is no longer an ordinary function, but a generator:

>>> fib(6)
<generator object fib at 0x104feaaa0>

Here, the most difficult thing to understand is that the execution flow of generator and function is different. Functions are executed sequentially and return when encountering a return statement or the last line of function statements. The function that becomes a generator is executed every time next() is called, returns when encountering a yield statement, and continues execution from the yield statement returned last time when executed again.

As a simple example, define a generator that returns the numbers 1, 3, and 5 in sequence:

>>> def odd():
...     print 'step 1'
...     yield 1
...     print 'step 2'
...     yield 3
...     print 'step 3'
...     yield 5
...
>>> o = odd()
>>> o.next()
step 1
1
>>> o.next()
step 2
3
>>> o.next()
step 3
5
>>> o.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

You can see that odd is not an ordinary function, but a generator. During execution, it encounters It will be interrupted when yield is reached and execution will continue next time. After executing yield three times, there is no more yield to execute, so an error is reported when next() is called for the fourth time.

Back to the fib example, if we keep calling yield during the loop, it will continue to be interrupted. Of course, you need to set a condition for the loop to exit the loop, otherwise an infinite number will be listed.

Similarly, after changing the function to a generator, we basically never use next() to call it, but directly use a for loop to iterate:

>>> for n in fib(6):
...     print n
...
1
1
2
3
5
8

generator is a very powerful tool, In Python, you can simply change the list generation into a generator, or you can use functions to implement complex logic generators.

To understand the working principle of the generator, it continuously calculates the next element during the for loop and ends the for loop under appropriate conditions. For a generator changed from a function, when the return statement is encountered or the last line of the function body is executed, it is the instruction to end the generator, and the for loop ends accordingly.