Example of analyzing nginx logs with python+pandas

The following is an example of analyzing nginx logs using python pandas. It has good reference value and I hope it will be helpful to everyone. Let’s take a look together

Requirements

By analyzing the nginx access log, obtain the maximum, minimum, and average response time of each interface and visits.

Implementation principle

Store the nginx log uriuriupstream_response_time field in the pandas dataframe, and then implement it through grouping and data statistics functions.

Implementation

1. Preparation

#创建日志目录，用于存放日志
mkdir /home/test/python/log/log
#创建文件，用于存放从nginx日志中提取的$uri $upstream_response_time字段
touch /home/test/python/log/log.txt
#安装相关模块
conda create -n science numpy scipy matplotlib pandas
#安装生成execl表格的相关模块
pip install xlwt

2. Code implementation

#!/usr/local/miniconda2/envs/science/bin/python
#-*- coding: utf-8 -*-
#统计每个接口的响应时间
#请提前创建log.txt并设置logdir
import sys
import os
import pandas as pd
mulu=os.path.dirname(__file__)
#日志文件存放路径
logdir="/home/test/python/log/log"
#存放统计所需的日志相关字段
logfile_format=os.path.join(mulu,"log.txt")
print "read from logfile \n"
for eachfile in os.listdir(logdir):
 logfile=os.path.join(logdir,eachfile)
 with open(logfile, 'r') as fo:
  for line in fo:
   spline=line.split()
   #过滤字段中异常部分
   if spline[6]=="-":
    pass
   elif spline[6]=="GET":
    pass
   elif spline[-1]=="-":
    pass
   else:
    with open(logfile_format, 'a') as fw:
     fw.write(spline[6])
     fw.write('\t')
     fw.write(spline[-1])
     fw.write('\n')
print "output panda"
#将统计的字段读入到dataframe中
reader=pd.read_table(logfile_format,sep='\t',engine='python',names=["interface","reponse_time"] ,header=None,iterator=True)
loop=True
chunksize=10000000
chunks=[]
while loop:
 try:
  chunk=reader.get_chunk(chunksize)
  chunks.append(chunk)
 except StopIteration:
  loop=False
  print "Iteration is stopped."
df=pd.concat(chunks)
#df=df.set_index("interface")
#df=df.drop(["GET","-"])
df_groupd=df.groupby('interface')
df_groupd_max=df_groupd.max()
df_groupd_min= df_groupd.min()
df_groupd_mean= df_groupd.mean()
df_groupd_size= df_groupd.size()
#print df_groupd_max
#print df_groupd_min
#print df_groupd_mean
df_ana=pd.concat([df_groupd_max,df_groupd_min,df_groupd_mean,df_groupd_size],axis=1,keys=["max","min","average","count"])
print "output excel"
df_ana.to_excel("test.xls")

##3. The printed form is as follows:

Key Points

1. When the log file is relatively large Do not use readlines() or readline() when reading, as this will read all the logs into the memory, causing the memory to become full. Therefore, the for line in fo iteration method is used here, which basically does not occupy memory.

2. To read nginx logs, you can use pd.read_table(log_file, sep=' ', iterator=True), but the sep we set here cannot match the split normally, so first split nginx with split , and then save it to pandas.

3. Pandas provides IO tools to read large files in chunks, use different chunk sizes to read and then call pandas.concat to connect to the DataFrame

Related recommendations:

python3 pandas reads MySQL data and inserts

##

The above is the detailed content of Example of analyzing nginx logs with python+pandas. For more information, please follow other related articles on the PHP Chinese website!