How do generators in Python work?

What is a python generator

The generator is a special iterator. It also has the __iter__ method and the __next__ method inside it. When terminating the generator Sometimes, the StopIteration exception will still be thrown to exit the loop, but compared to the iterator, the generator also has the feature of saving the "intermediate value". The next time it runs, it will also use this " Intermediate value" to operate. The keyword of the generator is yield. Let’s write the simplest generator below.

#!/usr/bin/env python

def printNums():
    i = 0
    while i<10:
        yield i
        i = i + 1


def main():
    for i in printNums():
        print(i)

if __name__ == &#39;__main__&#39;:
    main()

If you look at the code at a glance, you may wonder what this is. Why don’t you just use range to generate it instead of using yield? Oh, don’t worry. , let’s go on to see why a generator is needed, or what problem the generator solves.

Why python generator is needed

Before explaining this problem, let’s first write a requirement to output data within 0-10000000, and then run to view the screenshot of the exported memory operation.

Auxiliary instructions for calling python program memory information

Here you can use the memory_profiler module of python to detect the occupancy of program memory.

Installationmemory_profilerLibrary:

pip3 install memory_profiler

The method of use is very simple. Just add the @profile decorator before the function or code that needs to be detected. , for example:

@profile
def main():
    pass

Generate .dat file

mprof run

Export icon, you can use

mprof plot --output=filename

python case code

The following two programs both output data between 0-9999999. The difference is that the first program uses range and then append into list, while the second one uses an iterator to generate the data.

main.pyProgram

@profile
def main():
    data = list(range(10000000))
    for i in data:
        pass

if __name__ == '__main__':
    main()

main_2.pyProgram

def printNum():
    i = 0 
    while i < 10000000:
        yield i
        i = i + 1

@profile
def main():
    for i in printNum():
        pass

if __name__ == &#39;__main__&#39;:
    main()

Running program

The code is also there Now, you can run the program as above and export the memory information

View the memory information after running

main.py Running memory graph

main_2.py Running memory graph

Comparison of the above 2 pictures , when we superimpose the data into the list and then output it, it takes up nearly 400M of memory, while using an iterator to calculate the next value only uses 16M of memory.

Through the above cases, we should know why we should use generators.

Python generator principle

Since the generator expression yield statement involves the internal mechanism of python interpretation rights, it is difficult to view its source code. It is difficult to obtain its principle, but we can use the pause mechanism of yield to explore the generator.

You can write the following code:

def testGenerator():
    print("进入生成器")
    yield "pdudo"
    print("第一次输出")
    yield "juejin"
    print("第二次输出")

def main():
    xx = testGenerator()
    print(next(xx))
    print(next(xx))

if __name__ == '__main__':
    main()

The effect after operation is as follows

Through the above example, combined with the operation of the following generator The process will deepen the feeling of the generator.

When python encounters the yield statement, the running status of the current function will be recorded, execution will be suspended, and the result will be thrown. It will continue to wait for the next call to the __next__ method. After this method is called, the function will resume running until the next yield statement or the end of the function. There will be no yield# at the end of the execution. ##When the function is executable, StopIteration will be thrown to mark the end of the generator.

Generator expression

In

python, in addition to being written in a function and returned using yield, the generator can also be used directly Generator expressions, eh. . . It may be abstract, but if you look at the code below, you'll understand.

def printNums():
    for i in [1,2,3,4,5]:
        yield i

def main():
    for i in printNums():
        print(i)

    gener = (i for i in [1,2,3,4,5])
    for i in gener:
        print(i)

if __name__ == '__main__':
    main()

Among them, the code

(i for i in [1,2,3,4,5]) is equivalent to the printNums function, and its types are generated Container, we can use type to print it out and take a look.

Change the code and the output will be as follows:

The above is the detailed content of How do generators in Python work?. For more information, please follow other related articles on the PHP Chinese website!