Python(Stackless) + MongoDB Apache 日志(2G)分析
为何选择Stackless? http://www.stackless.com Stackless可以简单的认为是Python一个增强版,最吸引眼球的非“微线程”莫属。微线程是轻量级的线程,与线程相比切换消耗的资源更小,线程内共享数据更加便捷。相比多线程代码更加简洁和可读。此项目是由EVE O
为何选择Stackless? http://www.stackless.com
Stackless可以简单的认为是Python一个增强版,最吸引眼球的非“微线程”莫属。微线程是轻量级的线程,与线程相比切换消耗的资源更小,线程内共享数据更加便捷。相比多线程代码更加简洁和可读。此项目是由EVE Online推出,在并发和性能上确实很强劲。安装和Python一样,可以考虑替换原系统Python。:)
为何选择MongoDB? http://www.mongodb.org
可以在官网看到很多流行的应用采用MongoDB,比如sourceforge,github等。相比RDBMS有啥优势?首先在速度和性能上优势最为明显,不仅可以当作类似KeyValue数据库来使,还包含了一些数据库查询(Distinct、Group、随机、索引等特性)。再有一点特性就是:简单。不论是应用还是文档,还是第三方API,几乎略过一下就可以使用。不过有点遗憾的就是,存储的数据文件很大,超过正常数据的2-4倍之间。本文测试的Apache日志大小是2G,生产的数据文件有6G。寒...希望在新版里能有所缩身,当然这个也是明显的以空间换速度的后果。
本文除去上面提及到的两个软件,还需要安装pymongo模块。http://api.mongodb.org/python/
模块安装方式有源码编译和easy_install,这里就不再累赘。
- 从Apache日志中分析出需要保存的资料,比如IP,时间,GET/POST,返回状态码等。
fmt_str = '(?P[.\d]+) - - \[(?P.*?)\] "(?P.*?) (?P.*?) HTTP/1.\d" (?P\d+) (?P.*?) "(?P.*?)" "(?P.*?)"' fmt_name = re.findall('\?P', fmt_str) fmt_re = re.compile(fmt_str)
定义了一个正则用于提取每行日志的内容。fmt_name就是提取尖括号中间的变量名。
- 定义MongoDB相关变量,包括需要存到collection名称。Connection采取的是默认Host和端口。
conn = Connection() apache = conn.apache logs = apache.logs
- 保存日志行
def make_line(line): m = fmt_re.search(line) if m: logs.insert(dict(zip(fmt_name, m.groups())))
- 读取Apache日志文件
def make_log(log_path): with open(log_path) as fp: for line in fp: make_line(line.strip())
- 运行把。
if __name__ == '__main__': make_log('d:/apachelog.txt')
脚本大致情况如此,这里没有放上stackless部分代码,可以参考下面代码:
import stackless def print_x(x): print x stackless.tasklet(print_x)('one') stackless.tasklet(print_x)('two') stackless.run()
tasklet操作只是把类似操作放入队列中,run才是真正的运行。这里主要用于替换原有多线程threading并行分析多个日志的行为。
补充:
Apache日志大小是2G,671万行左右。生成的数据库有6G。
硬件:Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz 台式机
系统:RHEL 5.2 文件系统ext3
其他:Stackless 2.6.4 MongoDB 1.2
在保存300万左右时候,一切正常。不管是CPU还是内存,以及插入速度都很不错,大概有8-9000条/秒。和以前笔记本上测试结果基本一致。再往以后,内存消耗有点飙升,插入速度也降低。500万左右记录时候CPU达到40%,内存消耗2.1G。在生成第二个2G数据文件时候似乎速度和效率又提升上去了。最终保存的结果不是太满意。
后加用笔记本重新测试了一下1000万数据,速度比上面的671万明显提升很多。初步怀疑有两个地方可能会影响性能和速度:
文件系统的差异。笔记本是Ubuntu 9.10,ext4系统。搜了下ext3和ext4在大文件读写上会有所差距。
正则匹配上。单行操作都是匹配提取。大文件上应该还有优化的空间。
原文地址:Python(Stackless) + MongoDB Apache 日志(2G)分析, 感谢原作者分享。

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics



VS Code can be used to write Python and provides many features that make it an ideal tool for developing Python applications. It allows users to: install Python extensions to get functions such as code completion, syntax highlighting, and debugging. Use the debugger to track code step by step, find and fix errors. Integrate Git for version control. Use code formatting tools to maintain code consistency. Use the Linting tool to spot potential problems ahead of time.

In VS Code, you can run the program in the terminal through the following steps: Prepare the code and open the integrated terminal to ensure that the code directory is consistent with the terminal working directory. Select the run command according to the programming language (such as Python's python your_file_name.py) to check whether it runs successfully and resolve errors. Use the debugger to improve debugging efficiency.

VS Code can run on Windows 8, but the experience may not be great. First make sure the system has been updated to the latest patch, then download the VS Code installation package that matches the system architecture and install it as prompted. After installation, be aware that some extensions may be incompatible with Windows 8 and need to look for alternative extensions or use newer Windows systems in a virtual machine. Install the necessary extensions to check whether they work properly. Although VS Code is feasible on Windows 8, it is recommended to upgrade to a newer Windows system for a better development experience and security.

VS Code extensions pose malicious risks, such as hiding malicious code, exploiting vulnerabilities, and masturbating as legitimate extensions. Methods to identify malicious extensions include: checking publishers, reading comments, checking code, and installing with caution. Security measures also include: security awareness, good habits, regular updates and antivirus software.

Python excels in automation, scripting, and task management. 1) Automation: File backup is realized through standard libraries such as os and shutil. 2) Script writing: Use the psutil library to monitor system resources. 3) Task management: Use the schedule library to schedule tasks. Python's ease of use and rich library support makes it the preferred tool in these areas.

VS Code is the full name Visual Studio Code, which is a free and open source cross-platform code editor and development environment developed by Microsoft. It supports a wide range of programming languages and provides syntax highlighting, code automatic completion, code snippets and smart prompts to improve development efficiency. Through a rich extension ecosystem, users can add extensions to specific needs and languages, such as debuggers, code formatting tools, and Git integrations. VS Code also includes an intuitive debugger that helps quickly find and resolve bugs in your code.

MongoDB is suitable for handling large-scale unstructured data, and Oracle is suitable for enterprise-level applications that require transaction consistency. 1.MongoDB provides flexibility and high performance, suitable for processing user behavior data. 2. Oracle is known for its stability and powerful functions and is suitable for financial systems. 3.MongoDB uses document models, and Oracle uses relational models. 4.MongoDB is suitable for social media applications, while Oracle is suitable for enterprise-level applications.

Yes, VS Code can run Python code. To run Python efficiently in VS Code, complete the following steps: Install the Python interpreter and configure environment variables. Install the Python extension in VS Code. Run Python code in VS Code's terminal via the command line. Use VS Code's debugging capabilities and code formatting to improve development efficiency. Adopt good programming habits and use performance analysis tools to optimize code performance.
