Backend Development
Python Tutorial
Detailed explanation of real IP request Pandas for Python data analysis
Detailed explanation of real IP request Pandas for Python data analysis
Foreword
pandas is a data analysis package built based on Numpy that contains more advanced data structures and tools. Similar to Numpy, whose core is ndarray, pandas also revolves around the two core data structures of Series and DataFrame. Series and DataFrame correspond to one-dimensional sequence and two-dimensional table structure respectively. The conventional import method of pandas is as follows:
from pandas import Series,DataFrame import pandas as pd
1.1. Pandas analysis steps
1. Load log data
2. Load area_ip data
3. Count the number of real_ip requests. SQL similar to the following:
SELECT inet_aton(l.real_ip), count(*), a.addr FROM log AS l INNER JOIN area_ip AS a ON a.start_ip_num <= inet_aton(l.real_ip) AND a.end_ip_num >= inet_aton(l.real_ip) GROUP BY real_ip ORDER BY count(*) LIMIT 0, 100;
1.2. Code
cat pd_ng_log_stat.py
#!/usr/bin/env python
#-*- coding: utf-8 -*-
from ng_line_parser import NgLineParser
import pandas as pd
import socket
import struct
class PDNgLogStat(object):
def __init__(self):
self.ng_line_parser = NgLineParser()
def _log_line_iter(self, pathes):
"""解析文件中的每一行并生成一个迭代器"""
for path in pathes:
with open(path, 'r') as f:
for index, line in enumerate(f):
self.ng_line_parser.parse(line)
yield self.ng_line_parser.to_dict()
def _ip2num(self, ip):
"""用于IP转化为数字"""
ip_num = -1
try:
# 将IP转化成INT/LONG 数字
ip_num = socket.ntohl(struct.unpack("I",socket.inet_aton(str(ip)))[0])
except:
pass
finally:
return ip_num
def _get_addr_by_ip(self, ip):
"""通过给的IP获得地址"""
ip_num = self._ip2num(ip)
try:
addr_df = self.ip_addr_df[(self.ip_addr_df.ip_start_num <= ip_num) &
(ip_num <= self.ip_addr_df.ip_end_num)]
addr = addr_df.at[addr_df.index.tolist()[0], 'addr']
return addr
except:
return None
def load_data(self, path):
"""通过给的文件路径加载数据生成 DataFrame"""
self.df = pd.DataFrame(self._log_line_iter(path))
def uv_real_ip(self, top = 100):
"""统计cdn ip量"""
group_by_cols = ['real_ip'] # 需要分组的列,只计算和显示该列
# 直接统计次数
url_req_grp = self.df[group_by_cols].groupby(
self.df['real_ip'])
return url_req_grp.agg(['count'])['real_ip'].nlargest(top, 'count')
def uv_real_ip_addr(self, top = 100):
"""统计real ip 地址量"""
cnt_df = self.uv_real_ip(top)
# 添加 ip 地址 列
cnt_df.insert(len(cnt_df.columns),
'addr',
cnt_df.index.map(self._get_addr_by_ip))
return cnt_df
def load_ip_addr(self, path):
"""加载IP"""
cols = ['id', 'ip_start_num', 'ip_end_num',
'ip_start', 'ip_end', 'addr', 'operator']
self.ip_addr_df = pd.read_csv(path, sep='\t', names=cols, index_col='id')
return self.ip_addr_df
def main():
file_pathes = ['www.ttmark.com.access.log']
pd_ng_log_stat = PDNgLogStat()
pd_ng_log_stat.load_data(file_pathes)
# 加载 ip 地址
area_ip_path = 'area_ip.csv'
pd_ng_log_stat.load_ip_addr(area_ip_path)
# 统计 用户真实 IP 访问量 和 地址
print pd_ng_log_stat.uv_real_ip_addr()
if __name__ == '__main__':
main()Running statistics and output results
python pd_ng_log_stat.py
count addr
real_ip
60.191.123.80 101013 浙江省杭州市
- 32691 None
218.30.118.79 22523 北京市
......
136.243.152.18 889 德国
157.55.39.219 889 美国
66.249.65.170 888 美国
[100 rows x 2 columns]Summary
The above is the entire content of this article. I hope the content of this article will be helpful to everyone’s study or work. If you have any questions, you can leave a message to communicate.
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1377
52
Do mysql need to pay
Apr 08, 2025 pm 05:36 PM
MySQL has a free community version and a paid enterprise version. The community version can be used and modified for free, but the support is limited and is suitable for applications with low stability requirements and strong technical capabilities. The Enterprise Edition provides comprehensive commercial support for applications that require a stable, reliable, high-performance database and willing to pay for support. Factors considered when choosing a version include application criticality, budgeting, and technical skills. There is no perfect option, only the most suitable option, and you need to choose carefully according to the specific situation.
HadiDB: A lightweight, horizontally scalable database in Python
Apr 08, 2025 pm 06:12 PM
HadiDB: A lightweight, high-level scalable Python database HadiDB (hadidb) is a lightweight database written in Python, with a high level of scalability. Install HadiDB using pip installation: pipinstallhadidb User Management Create user: createuser() method to create a new user. The authentication() method authenticates the user's identity. fromhadidb.operationimportuseruser_obj=user("admin","admin")user_obj.
Can mysql workbench connect to mariadb
Apr 08, 2025 pm 02:33 PM
MySQL Workbench can connect to MariaDB, provided that the configuration is correct. First select "MariaDB" as the connector type. In the connection configuration, set HOST, PORT, USER, PASSWORD, and DATABASE correctly. When testing the connection, check that the MariaDB service is started, whether the username and password are correct, whether the port number is correct, whether the firewall allows connections, and whether the database exists. In advanced usage, use connection pooling technology to optimize performance. Common errors include insufficient permissions, network connection problems, etc. When debugging errors, carefully analyze error information and use debugging tools. Optimizing network configuration can improve performance
Navicat's method to view MongoDB database password
Apr 08, 2025 pm 09:39 PM
It is impossible to view MongoDB password directly through Navicat because it is stored as hash values. How to retrieve lost passwords: 1. Reset passwords; 2. Check configuration files (may contain hash values); 3. Check codes (may hardcode passwords).
How to solve mysql cannot connect to local host
Apr 08, 2025 pm 02:24 PM
The MySQL connection may be due to the following reasons: MySQL service is not started, the firewall intercepts the connection, the port number is incorrect, the user name or password is incorrect, the listening address in my.cnf is improperly configured, etc. The troubleshooting steps include: 1. Check whether the MySQL service is running; 2. Adjust the firewall settings to allow MySQL to listen to port 3306; 3. Confirm that the port number is consistent with the actual port number; 4. Check whether the user name and password are correct; 5. Make sure the bind-address settings in my.cnf are correct.
Does mysql need the internet
Apr 08, 2025 pm 02:18 PM
MySQL can run without network connections for basic data storage and management. However, network connection is required for interaction with other systems, remote access, or using advanced features such as replication and clustering. Additionally, security measures (such as firewalls), performance optimization (choose the right network connection), and data backup are critical to connecting to the Internet.
How to optimize MySQL performance for high-load applications?
Apr 08, 2025 pm 06:03 PM
MySQL database performance optimization guide In resource-intensive applications, MySQL database plays a crucial role and is responsible for managing massive transactions. However, as the scale of application expands, database performance bottlenecks often become a constraint. This article will explore a series of effective MySQL performance optimization strategies to ensure that your application remains efficient and responsive under high loads. We will combine actual cases to explain in-depth key technologies such as indexing, query optimization, database design and caching. 1. Database architecture design and optimized database architecture is the cornerstone of MySQL performance optimization. Here are some core principles: Selecting the right data type and selecting the smallest data type that meets the needs can not only save storage space, but also improve data processing speed.
How to use AWS Glue crawler with Amazon Athena
Apr 09, 2025 pm 03:09 PM
As a data professional, you need to process large amounts of data from various sources. This can pose challenges to data management and analysis. Fortunately, two AWS services can help: AWS Glue and Amazon Athena.
