How to implement Python's lazy import-lazy import

If your Python program has a large number of imports and starts very slowly, then you should try lazy import. This article shares a method to achieve lazy import. Although PEP0690[1] has proposed to allow the Python compiler (-L) or the standard library to add this feature, the current Python version has not yet implemented it.

As we all know, Python applications will perform import operations before performing the actual operations of the user. Different modules may come from different locations, some modules may be very time-consuming to run, and some modules may not work at all. It is called by the user, so the import of many modules is a pure waste of time.

So we need lazy import. When applying lazy import, running import foo will only add the name foo to the global full name space (globals()) as a lazy reference. The compiler The actual import operation is not performed until any code accessing foo is encountered. Similarly, from foo import bar will add bar to the namespace, and when it encounters code that calls bar, it will import foo.

Write code to implement

So how to write code to implement it? In fact, there is no need to write code to implement it. There is already a project that implements the lazy import function, that is TensorFlow. Its code does not depend on any third-party libraries. I put it here. In the future, when you need lazy import, you can directly put the LazyLoader[2] class Just copy it to your own project.

The source code is as follows:

# Code copied from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/util/lazy_loader.py
"""A LazyLoader class."""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import importlib
import types


class LazyLoader(types.ModuleType):
"""Lazily import a module, mainly to avoid pulling in large dependencies.

`contrib`, and `ffmpeg` are examples of modules that are large and not always
needed, and this allows them to only be loaded when they are used.
"""

# The lint error here is incorrect.
def __init__(self, local_name, parent_module_globals, name):# pylint: disable=super-on-old-class
self._local_name = local_name
self._parent_module_globals = parent_module_globals

super(LazyLoader, self).__init__(name)

def _load(self):
# Import the target module and insert it into the parent's namespace
module = importlib.import_module(self.__name__)
self._parent_module_globals[self._local_name] = module

# Update this object's dict so that if someone keeps a reference to the
# LazyLoader, lookups are efficient (__getattr__ is only called on lookups
# that fail).
self.__dict__.update(module.__dict__)

return module

def __getattr__(self, item):
module = self._load()
return getattr(module, item)

def __dir__(self):
module = self._load()
return dir(module)

Code description:

Class LazyLoader inherits from types.ModuleType, the initialization function ensures that the lazy module will behave like a real module It is also correctly added to the global variable. Only when the module is actually used, that is, when __getattr__ or __dir__ is executed, the actual module will be imported, the global variable will be updated to point to the actual module, and all its status (__dict__) Updated to the state of the actual module so that the reference to lazy loading, loading the module does not need to go through the loading process every time it is accessed.

Code usage:

Normally we import the module like this:

import tensorflow.contrib as contrib

The corresponding lazy import version is as follows:

contrib = LazyLoader('contrib', globals(), 'tensorflow.contrib')

PEP0690 Suggested approach

PEP0690's proposal is to implement it at the compiler (C code) level, so that the performance will be better. There are two ways to use it.

One

One way is to add the -L parameter when executing the Python script. For example, there are two files spam.py with the following contents:

The content of

import time
time.sleep(10)
print("spam loaded")

egg.py is as follows:

import spam
print("imports done")

Under normal import conditions, "spam loaded" will be printed first after 10 seconds, and then "imports done" will be printed. When executing python -L eggs.py , the spam module is never imported, and the application spam module is not used at all. If the content of egg.py is as follows:

import spam
print("imports done")
spam

When python -L eggs.py is executed, "imports done" will be printed first, and "spam loaded" will be printed 10 seconds later).

Second

Another way is to call the standard library importlib method:

import importlib
importlib.set_lazy_imports(True)

If some modules cannot Lazy loading needs to be excluded, it can be like this

import importlib
importlib.set_lazy_imports(True,excluding=["one.mod", "another"])

It can also be like this:

from importlib import eager_imports

with eager_imports():
import foo
import bar

Final words

After professionals After testing on a real Python command line program, applying lazy import can increase startup time by 70% and reduce memory usage by 40%, which is very impressive.

References

[1]PEP0690: https://github.com/python/peps/blob/main/pep-0690.rst

[2]LazyLoader : https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/util/lazy_loader.py

The above is the detailed content of How to implement Python's lazy import-lazy import. For more information, please follow other related articles on the PHP Chinese website!