How to use Python3 Loguru output log tool

1. Preface

The Python logging module defines functions and classes that implement flexible event logging for applications and libraries.

During the program development process, many programs have the need to record logs, and the information contained in the logs includes normal program access logs and may also include error, warning and other information output. Python's logging module provides standard logs An interface through which logs in various formats can be stored. Logging provides a set of convenience functions for simple logging usage.

The main benefit of using the Python Logging module is that all Python modules can participate in logging. The Logging module provides a large number of flexible functions.

Why use loguru?

It is simple and convenient to help us output the required log information:

If you use Python to write programs or scripts, the problems you often encounter are: The log needs to be deleted. On the one hand, it can help us troubleshoot problems when there is a problem with the program, and on the other hand, it can help us record the information that needs attention.
However, if we use the built-in logging module, we need to perform different initialization and other related work. For students who are not familiar with this module, it is still a bit difficult, such as the need to configure Handler/Formatter, etc. As business complexity increases, there are higher requirements for log collection, such as: log classification, file storage, asynchronous writing, custom types, etc.

loguru is a simple and powerful third party for Python Logging library that aims to make Python logging less painful by adding a set of useful features that address the caveats of the standard logger.

2. Use loguru elegantly

1. Install loguru

pip install loguru

2. Function and feature introduction

has many advantages, the more important ones are listed below A few points:

• Ready to use out of the box, no preparation required

• No need to initialize, just import the function to use

• Easier file logging and dumping/retention/compression methods

• More elegant string formatted output

• Exceptions can be caught in threads or main threads

• Different levels of logging styles can be set

• Supports asynchronous, threading and multi-process Security

• Supports lazy evaluation

• Works with scripts and libraries

• Fully compatible with standard logging

• Better date and time handling

3. Works out of the box, no preparation required

from loguru import logger  
logger.debug("That's it, beautiful and simple logging!")

No need After initialization, the imported function can be used, so you must ask, how to solve the problem?

• How to add a handler?

• How to set the log format (logs formatting)?

• How to filter messages?

• How to set the level (log level)?

# add  
logger.add(sys.stderr, \  
    format="{time} {level} {message}",\  
    filter="my_module",\  
    level="INFO")

Isn’t it very easy~

4. Easier file logging and transfer/retention/compression methods

# 日志文件记录  
logger.add("file_{time}.log")  
# 日志文件转存  
logger.add("file_{time}.log", rotation="500 MB")  
logger.add("file_{time}.log", rotation="12:00")  
logger.add("file_{time}.log", rotation="1 week")  
# 多次时间之后清理  
logger.add("file_X.log", retention="10 days")  
# 使用zip文件格式保存  
logger.add("file_Y.log", compression="zip")

5. More Elegant string formatted output

logger.info(  
    "If you're using Python {}, prefer {feature} of course!",  
    3.10, feature="f-strings")

6. Catch exceptions in child threads or main threads

@logger.catch  
def my_function(x, y, z):  
    # An error? It's caught anyway!  
    return 1 / (x + y + z)  
my_function(0, 0, 0)

7. Different levels of logging styles can be set

Loguru will Automatically add different colors to distinguish different log levels, and also support custom colors~

logger.add(sys.stdout,  
    colorize=True,  
    format="<green>{time}</green> <level>{message}</level>")  
logger.add(&#39;logs/z_{time}.log&#39;,  
           level=&#39;DEBUG&#39;,  
           format=&#39;{time:YYYY-MM-DD :mm:ss} - {level} - {file} - {line} - {message}&#39;,  
           rotation="10 MB")

8. Support asynchronous and thread and multi-process safety

• By default, the log information added to the logger is thread-safe. But this is not multi-process safe, we can ensure log integrity by adding the enqueue parameter.

• If we want to use logging in asynchronous tasks, we can also use the same parameters to ensure it. And wait for execution to complete through complete().

# 异步写入  
logger.add("some_file.log", enqueue=True)

You read that right, you only need enqueue=True to execute asynchronously

9. Complete description of the exception

For bug tracing that logs exceptions that occur in your code, Loguru helps you identify problems by allowing the entire stack trace to be displayed (including variable values)

logger.add("out.log", backtrace=True, diagnose=True)  
def func(a, b):  
    return a / b  
def nested(c):  
    try:  
        func(5, c)  
    except ZeroDivisionError:  
        logger.exception("What?!")  
nested(0)

10. Structured logging

• Serialize logs to make it easier to parse or pass data structures, using the serialization parameter, to convert each log message to a JSON string before sending it to the configured receiver.

• Also, using the bind() method, logger messages can be put into context by modifying additional record properties. You can also have more fine-grained control over logging by combining bind() and filter.

• Finally the patch() method allows appending dynamic values ​​to the records dict for each new message.

# 序列化为json格式  
logger.add(custom_sink_function, serialize=True)  
# bind方法的用处  
logger.add("file.log", format="{extra[ip]} {extra[user]} {message}")  
context_logger = logger.bind(ip="192.168.2.174", user="someone")  
context_logger.info("Contextualize your logger easily")  
context_logger.bind(user="someone_else").info("Inline binding of extra attribute")  
context_logger.info("Use kwargs to add context during formatting: {user}", user="anybody")  
# 粒度控制  
logger.add("special.log", filter=lambda record: "special" in record["extra"])  
logger.debug("This message is not logged to the file")  
logger.bind(special=True).info("This message, though, is logged to the file!")  
# patch()方法的用处  
logger.add(sys.stderr, format="{extra[utc]} {message}")  
loggerlogger = logger.patch(lambda record: record["extra"].update(utc=datetime.utcnow()))

11. Lazy calculation

Sometimes you want to log details in a production environment without affecting performance. You can use the opt() method to achieve this.

logger.opt(lazy=True).debug("If sink level <= DEBUG: {x}", x=lambda: expensive_function(2**64))  
# By the way, "opt()" serves many usages  
logger.opt(exception=True).info("Error stacktrace added to the log message (tuple accepted too)")  
logger.opt(colors=True).info("Per message <blue>colors</blue>")  
logger.opt(record=True).info("Display values from the record (eg. {record[thread]})")  
logger.opt(raw=True).info("Bypass sink formatting\n")  
logger.opt(depth=1).info("Use parent stack context (useful within wrapped functions)")  
logger.opt(capture=False).info("Keyword arguments not added to {dest} dict", dest="extra")

12. Customizable levels

new_level = logger.level("SNAKY", no=38, color="<yellow>", icon="????")  
logger.log("SNAKY", "Here we go!")

13. Suitable for scripts and libraries

# For scripts  
config = {  
    "handlers": [  
        {"sink": sys.stdout, "format": "{time} - {message}"},  
        {"sink": "file.log", "serialize": True},  
    ],  
    "extra": {"user": "someone"}  
}  
logger.configure(**config)  
# For libraries  
logger.disable("my_library")  
logger.info("No matter added sinks, this message is not displayed")  
logger.enable("my_library")  
logger.info("This message however is propagated to the sinks")

14. Fully compatible with standard logging

• Want to use Loguru as the built-in log handler?

• Need to send Loguru messages to the standard log?

• Want to intercept standard log messages and summarize them in Loguru?

handler = logging.handlers.SysLogHandler(address=('localhost', 514)) 
logger.add(handler)  
class PropagateHandler(logging.Handler):  
    def emit(self, record):  
        logging.getLogger(record.name).handle(record)  
logger.add(PropagateHandler(), format="{message}")  
class InterceptHandler(logging.Handler):  
    def emit(self, record):  
        # Get corresponding Loguru level if it exists  
        try:  
            level = logger.level(record.levelname).name  
        except ValueError:  
            level = record.levelno  
        # Find caller from where originated the logged message  
        frame, depth = logging.currentframe(), 2  
        while frame.f_code.co_filename == logging.__file__:  
            frameframe = frame.f_back  
            depth += 1  
        logger.opt(depthdepth=depth, exception=record.exc_info).log(level, record.getMessage())  
logging.basicConfig(handlers=[InterceptHandler()], level=0)

15. 非常方便的解析器

从生成的日志中提取特定的信息通常很有用，这就是为什么 Loguru 提供了一个 parse() 方法来帮助处理日志和正则表达式。

pattern = r"(?P<time>.*) - (?P<level>[0-9]+) - (?P<message>.*)"  # Regex with named groups  
caster_dict = dict(time=dateutil.parser.parse, level=int)        # Transform matching groups  
for groups in logger.parse("file.log", pattern, cast=caster_dict):  
    print("Parsed:", groups) 
    # {"level": 30, "message": "Log example", "time": datetime(2018, 12, 09, 11, 23, 55)}

16. 通知机制 (邮件告警)

import notifiers  
params = {  
    "username": "you@gmail.com",  
    "password": "abc123",  
    "to": "dest@gmail.com"  
}  
# Send a single notification  
notifier = notifiers.get_notifier("gmail")  
notifier.notify(message="The application is running!", **params)  
# Be alerted on each error message  
from notifiers.logging import NotificationHandler  
handler = NotificationHandler("gmail", defaults=params)  
logger.add(handler, level="ERROR")

17. Flask 框架集成

• 现在最关键的一个问题是如何兼容别的 logger，比如说 tornado 或者 django 有一些默认的 logger。

• 经过研究，最好的解决方案是参考官方文档的，完全整合 logging 的工作方式。比如下面将所有的 logging都用 loguru 的 logger 再发送一遍消息。

import logging  
import sys  
from pathlib import Path  
from flask import Flask  
from loguru import logger  
app = Flask(__name__)  
class InterceptHandler(logging.Handler):  
    def emit(self, record):  
        loggerlogger_opt = logger.opt(depth=6, exception=record.exc_info)  
        logger_opt.log(record.levelname, record.getMessage())  
def configure_logging(flask_app: Flask):  
    """配置日志"""  
    path = Path(flask_app.config['LOG_PATH'])  
    if not path.exists():  
        path.mkdir(parents=True)  
    log_name = Path(path, 'sips.log')  
    logging.basicConfig(handlers=[InterceptHandler(level='INFO')], level='INFO')  
    # 配置日志到标准输出流  
    logger.configure(handlers=[{"sink": sys.stderr, "level": 'INFO'}])  
    # 配置日志到输出到文件  
    logger.add(log_name, rotation="500 MB", encoding='utf-8', colorize=False, level='INFO')

18. 要点解析

介绍，主要函数的使用方法和细节 - add()的创建和删除

• add() 非常重要的参数 sink 参数

• 具体的实现规范可以参见官方文档

• 可以实现自定义 Handler 的配置，比如 FileHandler、StreamHandler 等等

• 可以自行定义输出实现

• 代表文件路径，会自动创建对应路径的日志文件并将日志输出进去

• 例如 sys.stderr 或者 open(‘file.log’, ‘w’) 都可以

• 可以传入一个 file 对象

• 可以直接传入一个 str 字符串或者 pathlib.Path 对象

• 可以是一个方法

• 可以是一个 logging 模块的 Handler

• 可以是一个自定义的类

def add(self, sink, *,  
    level=_defaults.LOGURU_LEVEL, format=_defaults.LOGURU_FORMAT,  
    filter=_defaults.LOGURU_FILTER, colorize=_defaults.LOGURU_COLORIZE,  
    serialize=_defaults.LOGURU_SERIALIZE, backtrace=_defaults.LOGURU_BACKTRACE,  
    diagnose=_defaults.LOGURU_DIAGNOSE, enqueue=_defaults.LOGURU_ENQUEUE,  
    catch=_defaults.LOGURU_CATCH, **kwargs  
):

另外添加 sink 之后我们也可以对其进行删除，相当于重新刷新并写入新的内容。删除的时候根据刚刚 add 方法返回的 id 进行删除即可。可以发现，在调用 remove 方法之后，确实将历史 log 删除了。但实际上这并不是删除，只不过是将 sink 对象移除之后，在这之前的内容不会再输出到日志中，这样我们就可以实现日志的刷新重新写入操作

from loguru import logger  
trace = logger.add('runtime.log')  
logger.debug('this is a debug message')  
logger.remove(trace)  
logger.debug('this is another debug message')

三、总结

我们在开发流程中, 通过日志快速定位问题, 高效率解决问题, 我认为 loguru 能帮你解决不少麻烦, 赶快试试吧~

当然, 使用各种也有不少麻烦, 例如:

1. 常见错误1:

--- Logging error in Loguru Handler #3 ---
Record was: None
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/loguru/_handler.py", line 272, in _queued_writer
    message = queue.get()
  File "/usr/local/lib/python3.9/multiprocessing/queues.py", line 366, in get
    res = self._reader.recv_bytes()
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
    chunk = read(handle, remaining)
OSError: [Errno 9] Bad file descriptor
--- End of logging error ---

解决办法:
尝试将logs文件夹忽略git提交, 避免和服务器文件冲突即可;
当然也不止这个原因引起这个问题, 也可能是三方库(ciscoconfparse)冲突所致.解决办法: https://github.com/Delgan/loguru/issues/534

2.常见错误2:

File "/home/ronaldinho/xxx/xxx/venv/lib/python3.9/site-packages/loguru/_logger.py", line 939, in add
    handler = Handler(
  File "/home/ronaldinho/xxx/xxx/venv/lib/python3.9/site-packages/loguru/_handler.py", line 86, in __init__
    self._queue = multiprocessing.SimpleQueue()
  File "/home/ronaldinho/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/home/ronaldinho/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/queues.py", line 342, in __init__
    self._rlock = ctx.Lock()
  File "/home/ronaldinho/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/home/ronaldinho/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/synchronize.py", line 162, in __init__
  File "/home/ronaldinho/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/synchronize.py", line 57, in __init__
OSError: [Errno 24] Too many open files

你可以 remove()添加的处理程序，它应该释放文件句柄。 

The above is the detailed content of How to use Python3 Loguru output log tool. For more information, please follow other related articles on the PHP Chinese website!