node.js - Python有了asyncio和aiohttp在爬虫这类型IO任务中多线程/多进程还有存在的必要吗？

Question

最近正在学习Python中的异步编程，看了一些博客后做了一些小测验：对比asyncio+aiohttp的爬虫和asyncio+aiohttp+concurrent.futures(线程池/进程池)在效率中的差异，注释：在爬虫中我几乎没有使用任何计算性任务，为了探测异步的性能，全部都只是做了网络IO请求，就是说aiohttp把网页get完就程序就done了。

结果发现前者的效率比后者还要高。我询问了另外一位博主，(提供代码的博主没回我信息)，他说使用concurrent.futures的话因为我全部都是IO任务，如果把这些IO任务分散到线程池/进程池，反而多线程/多进程之间的切换开销还会降低爬虫的效率。我想了想的确如此。

那么我的问题是：仅仅在爬取网页的过程中，就是request.get部分，多线程肯定是没有存在的必要了，因为GIL这个大坑，进程池可能好点，但是性能还是不如异步爬虫，而且更加浪费资源。既然这样，是不是以后在爬虫的爬取网页阶段我们完全都可以用兴起的asyncio+aiohttp代替。(以及其他IO任务比如数据库/文件读写)

当然在数据处理阶段还是要采用多进程，但是我觉得多线程是彻底没用了，原本它相比多进程的优势在于IO型任务，现看来在它的优势完全被异步取代了。(当然问题建立在不考虑兼容2.x)

注：还有一个额外的问题就是，看到一些博客说requests库不支持异步编程是什么意思，为了充分发回异步的优势应该使用aiohttp，我没有看过requests的源代码，但是一些结果显示aiohttp的性能确实更好，各位网友能解释一下吗？

代码

asyncio+aiohttp

import aiohttp


async def fetch_async(a):
    async with aiohttp.request('GET', URL.format(a)) as r:
        data = await r.json()
    return data['args']['a']
    
start = time.time()
event_loop = asyncio.get_event_loop()
tasks = [fetch_async(num) for num in NUMBERS]
results = event_loop.run_until_complete(asyncio.gather(*tasks))

for num, result in zip(NUMBERS, results):
    print('fetch({}) = {}'.format(num, result))

asyncio+aiohttp+线程池比上面要慢1秒

async def fetch_async(a):
    async with aiohttp.request('GET', URL.format(a)) as r:
        data = await r.json()
    return a, data['args']['a']


def sub_loop(numbers):
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    tasks = [fetch_async(num) for num in numbers]
    results = loop.run_until_complete(asyncio.gather(*tasks))
    for num, result in results:
        print('fetch({}) = {}'.format(num, result))


async def run(executor, numbers):
    await asyncio.get_event_loop().run_in_executor(executor, sub_loop, numbers)


def chunks(l, size):
    n = math.ceil(len(l) / size)
    for i in range(0, len(l), n):
        yield l[i:i + n]                                                     

event_loop = asyncio.get_event_loop()
tasks = [run(executor, chunked) for chunked in chunks(NUMBERS, 3)]
results = event_loop.run_until_complete(asyncio.gather(*tasks))

print('Use asyncio+aiohttp+ThreadPoolExecutor cost: {}'.format(time.time() - start))

传统的requests + ThreadPoolExecutor比上面慢了3倍

import time
import requests
from concurrent.futures import ThreadPoolExecutor

NUMBERS = range(12)
URL = 'http://httpbin.org/get?a={}'

def fetch(a):
    r = requests.get(URL.format(a))
    return r.json()['args']['a']

start = time.time()
with ThreadPoolExecutor(max_workers=3) as executor:
    for num, result in zip(NUMBERS, executor.map(fetch, NUMBERS)):
        print('fetch({}) = {}'.format(num, result))

print('Use requests+ThreadPoolExecutor cost: {}'.format(time.time() - start))

补充

以上问题建立在CPython，至于我喜欢用多线程，不喜欢协程风格这类型的回答显然不属于本题讨论范畴。我主要想请教的是：
如果Python拿不下GIL，我认为未来理想的模型应该是多进程 + 协程(asyncio+aiohttp)。uvloop和sanic以及500lines一个爬虫项目已经开始这么干了。不讨论兼容型问题，上面的看法是否正确，有一些什么场景协程无法取代多线程。

异步有很多方案，twisted, tornado等都有自己的解决方案，问题建立在asyncio+aiohttp的协程异步。

还有一个问题也想向各位网友请教一下

Answer 1

I don’t know much about Python crawlers, but generally Scrapy is used to make crawlers. It is based on the twisted asynchronous framework.

Multiple processes can make full use of multiple cores. Currently, the ideal one is multi-process + coroutine.

Because the synchronous method is still used in requests, it will block the thread. In this case, it is meaningless to use asynchronous. You can understand it as using the time.sleep method instead of the asyncio.sleep method in asyncio.

Answer 2

Check out this article: http://aosabook.org/en/500L/a...

Answer 3

asyncio adopts the idea of ​​coroutine, which is to process multiple asynchronous tasks in one thread. What are the asynchronous tasks, such as timing, asynchronous IO, etc.

But what if the task does not support asynchronous?

For example, reading and writing a blocking IO, or doing time-consuming a lot of calculations. Coroutines will solve the problem of task blocking, and the advantages of multi-process and multi-thread will be reflected.

The usage scenarios of the two are different. Different scenarios, different plans.

Answer 4

asyncio requires related third-party library support, so basically all the original third-party libraries need to be written separately, such as serial ports, network protocols, including requests and http. In bad cases, after these two As of version time, many of the libraries used are already asynchronous. Includes requests.

Answer 5

asyncio needs an asynchronous API to support it (synchronous non-blocking API is also available, but Python does not have such a thing, you may need to hack it). setInterval

If it is a synchronous blocking API, if one callback is stuck, other callbacks cannot be executed. You can take a look. The IO APIs you have seen so far are basically blocking.

Answer 6

Python multi-threading is not practical due to the existence of GIL, but multi-process is still very useful