Efficiency comparison experiment of single thread, multi-thread and multi-process in Python

Comparative experiment

The data shows that if the multi-threaded process is CPU-intensive, multi-threading will not improve the efficiency much. On the contrary, the efficiency may decrease due to frequent switching of threads. It is recommended to use multi-threading; If it is IO-intensive, the multi-threaded process can use the idle time while waiting for IO blocking to execute other threads to improve efficiency. So we compare the efficiency of different scenarios based on experiments

| Operating system | CPU | Memory | Hard disk |
|-----------|-------|------| --------|
| Windows 10 | Dual core|8GB|Mechanical hard drive|

(1)Introduce the required modules

import requests
import time
from threading import Thread
from multiprocessing import Process

(2)Define CPU-intensive calculation functions

def count(x, y):
    # 使程序完成150万计算
    c = 0
    while c < 500000:
        c += 1
        x += x
        y += y

(3)Definition IO-intensive file read and write function

def write():
    f = open("test.txt", "w")
    for x in range(5000000):
        f.write("testwrite\n")
    f.close()

def read():
    f = open("test.txt", "r")
    lines = f.readlines()
    f.close()

(4) Define network request function

_head = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36'}
url = "http://www.tieba.com"
def http_request():
    try:
        webPage = requests.get(url, headers=_head)
        html = webPage.text
        return {"context": html}
    except Exception as e:
        return {"error": e}

(5) Test the time required for linear execution of IO-intensive operations, CPU-intensive operations, and network request-intensive operations

# CPU密集操作
t = time.time()
for x in range(10):
    count(1, 1)
print("Line cpu", time.time() - t)

# IO密集操作
t = time.time()
for x in range(10):
    write()
    read()
print("Line IO", time.time() - t)

# 网络请求密集型操作
t = time.time()
for x in range(10):
    http_request()
print("Line Http Request", time.time() - t)

Output

CPU intensive: 95.6059999466, 91.57099986076355 92.52800011634827, 99.96799993515015

IO intensive: 24.25, 21.76699995994568, 21.76999998 0926514, 22.060999870300293

Network request intensive: 4.519999980926514, 8.563999891281128, 4.371000051498413, 4.522000074386597, 14.671000003 814697

(6) Test multi-threaded concurrent execution CPU intensive Time required for operation

counts = []
t = time.time()
for x in range(10):
    thread = Thread(target=count, args=(1,1))
    counts.append(thread)
    thread.start()

e = counts.__len__()
while True:
    for th in counts:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print(time.time() - t)

Output: 99.9240000248, 101.26400017738342, 102.32200002670288

(7) Test the time required for multi-threaded concurrent execution of IO-intensive operations

def io():
    write()
    read()

t = time.time()
ios = []
t = time.time()
for x in range(10):
    thread = Thread(target=count, args=(1,1))
    ios.append(thread)
    thread.start()

e = ios.__len__()
while True:
    for th in ios:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print(time.time() - t)

Output: 25.697000026702 88. 24.02400016784668

(8) Test multi-threaded concurrent execution network intensive Time required for operation

t = time.time()
ios = []
t = time.time()
for x in range(10):
    thread = Thread(target=http_request)
    ios.append(thread)
    thread.start()

e = ios.__len__()
while True:
    for th in ios:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Thread Http Request", time.time() - t)

Output: 0.7419998645782471, 0.3839998245239258, 0.3900001049041748

(9) Test the time required for multiple processes to concurrently perform CPU-intensive operations

counts = []
t = time.time()
for x in range(10):
    process = Process(target=count, args=(1,1))
    counts.append(process)
    process.start()
e = counts.__len__()
while True:
    for th in counts:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Multiprocess cpu", time.time() - t)

Output: 54.3420000 07629395, 53.437999963760376

(10) Testing multi-process concurrent execution of IO intensive Type operation

t = time.time()
ios = []
t = time.time()
for x in range(10):
    process = Process(target=io)
    ios.append(process)
    process.start()

e = ios.__len__()
while True:
    for th in ios:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Multiprocess IO", time.time() - t)

Output: 12.509000062942505, 13.059000015258789

(11) Testing the concurrent execution of HTTP request-intensive operations by multiple processes

t = time.time()
httprs = []
t = time.time()
for x in range(10):
    process = Process(target=http_request)
    ios.append(process)
    process.start()

e = httprs.__len__()
while True:
    for th in httprs:
        if not th.is_alive():
            e -= 1
    if e <= 0:
        break
print("Multiprocess Http Request", time.time() - t)

Output: 0.5329999923706055, 0.4760000 705718994

Experimental results

Through the above results, we can see To:

Multi-threading does not seem to have a big advantage under IO-intensive operations (perhaps the advantages will be reflected if the IO operation tasks are heavier). Under CPU-intensive operations, it is obviously more linear than single thread execution. The performance is worse, but for operations such as network requests that are busy waiting to block threads, the advantages of multi-threading are very significant

Multi-processes are either CPU-intensive, IO-intensive or network request-intensive (thread blocking often occurs) operation), the performance advantages can be reflected. However, for network request-intensive operations, it is almost the same as multi-threading, but it takes up more resources such as CPU, so in this case, we can choose multi-threading to execute