Python 多线程抓取图片效率对比
目的:
是学习python 多线程的工作原理,及通过抓取400张图片这种IO密集型应用来查看多线程效率对比
import requests import urlparse import os import time import threading import Queue path = '/home/lidongwei/scrapy/owan_img_urls.txt' #path = '/home/lidongwei/scrapy/cc.txt' fetch_img_save_path = '/home/lidongwei/scrapy/owan_imgs/' # 读取保存再文件里面400个urls with open(path) as f : urls = f.readlines() urls = urls[:400] # 使用Queue来线程通信,因为队列是线程安全的(就是默认这个队列已经有锁) q = Queue.Queue() for url in urls: q.put(url) start = time.time() def fetch_img_func(q): while True: try: # 不阻塞的读取队列数据 url = q.get_nowait() i = q.qsize() except Exception, e: print e break; print 'Current Thread Name Runing %s ... 11' % threading.currentThread().name url = url.strip() img_path = urlparse.urlparse(url).path ext = os.path.splitext(img_path)[1] print 'handle %s pic... pic url %s ' % (i, url) res = requests.get(url, stream=True) if res.status_code == 200: save_img_path = '%s%s%s' % (fetch_img_save_path, i, ext) # 保存下载的图片 with open(save_img_path, 'wb') as fs: for chunk in res.iter_content(1024): fs.write(chunk) print 'save %s pic ' % i # 可以开多个线程测试不同效果 t1 = threading.Thread(target=fetch_img_func, args=(q, ), name="child_thread_1") #t2 = threading.Thread(target=fetch_img_func, args=(q, ), name="child_thread_2") #t3 = threading.Thread(target=fetch_img_func, args=(q, ), name="child_thread_3") #t4 = threading.Thread(target=fetch_img_func, args=(q, ), name="child_thread_4") t1.start() #t2.start() #t3.start() #t4.start() t1.join() #t2.join() #t3.join() #t4.join() end = time.time() print 'Done %s ' % (end-start)
实验结果
400图片
4线程 Done 12.443133831 3线程 Done 12.9201757908 2线程 Done 32.8628299236 1线程 Done 54.6115460396
总结
Python 自带GIL 大锁, 没有真正意义上的多线程并行执行。GIL 大锁会在线程阻塞的时候释放,此时等待的线程就可以激活工作,这样如此类推,大大提高IO阻塞型应用的效率。

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Google AI has started to provide developers with access to extended context windows and cost-saving features, starting with the Gemini 1.5 Pro large language model (LLM). Previously available through a waitlist, the full 2 million token context windo

How to download DeepSeek Xiaomi? Search for "DeepSeek" in the Xiaomi App Store. If it is not found, continue to step 2. Identify your needs (search files, data analysis), and find the corresponding tools (such as file managers, data analysis software) that include DeepSeek functions.

The key to using DeepSeek effectively is to ask questions clearly: express the questions directly and specifically. Provide specific details and background information. For complex inquiries, multiple angles and refute opinions are included. Focus on specific aspects, such as performance bottlenecks in code. Keep a critical thinking about the answers you get and make judgments based on your expertise.

Just use the search function that comes with DeepSeek. Its powerful semantic analysis algorithm can accurately understand the search intention and provide relevant information. However, for searches that are unpopular, latest information or problems that need to be considered, it is necessary to adjust keywords or use more specific descriptions, combine them with other real-time information sources, and understand that DeepSeek is just a tool that requires active, clear and refined search strategies.

DeepSeek is not a programming language, but a deep search concept. Implementing DeepSeek requires selection based on existing languages. For different application scenarios, it is necessary to choose the appropriate language and algorithms, and combine machine learning technology. Code quality, maintainability, and testing are crucial. Only by choosing the right programming language, algorithms and tools according to your needs and writing high-quality code can DeepSeek be successfully implemented.

Question: Is DeepSeek available for accounting? Answer: No, it is a data mining and analysis tool that can be used to analyze financial data, but it does not have the accounting record and report generation functions of accounting software. Using DeepSeek to analyze financial data requires writing code to process data with knowledge of data structures, algorithms, and DeepSeek APIs to consider potential problems (e.g. programming knowledge, learning curves, data quality)

Python is an ideal programming introduction language for beginners through its ease of learning and powerful features. Its basics include: Variables: used to store data (numbers, strings, lists, etc.). Data type: Defines the type of data in the variable (integer, floating point, etc.). Operators: used for mathematical operations and comparisons. Control flow: Control the flow of code execution (conditional statements, loops).

Pythonempowersbeginnersinproblem-solving.Itsuser-friendlysyntax,extensivelibrary,andfeaturessuchasvariables,conditionalstatements,andloopsenableefficientcodedevelopment.Frommanagingdatatocontrollingprogramflowandperformingrepetitivetasks,Pythonprovid
