PHP curl concurrency best practice code sharing_PHP tutorial

This article will discuss two specific implementation methods and make a simple performance comparison of different methods.

1. Classic cURL concurrency mechanism and its existing problems

The classic cURL implementation mechanism is easy to find online. For example, refer to the following implementation method in the PHP online manual:

First push all URLs into the concurrent queue, and then execute the concurrent process, Wait for all requests to be received before parsing the data and other subsequent processing. In the actual processing process, affected by network transmission, the content of some URLs will be returned before other URLs, but classic cURL concurrency must wait for the slowest URL to return Processing starts later, and waiting means CPU idleness and waste. If the URL queue is short, this idleness and waste are still within an acceptable range, but if the queue is long, this waiting and waste will become unacceptable. Accept.

2. Improved Rolling cURL concurrency method

After careful analysis, it is not difficult to find that there is still room for optimization in classic cURL concurrency. When a URL is optimized, After the request is completed, process it as quickly as possible, and wait for other URLs to return while processing, instead of waiting for the slowest interface to return before starting processing and other work, thereby avoiding CPU idleness and waste. Without further ado, below Paste the specific implementation:


3. Performance comparison of two concurrent implementations

The performance comparison test before and after the improvement was conducted on a LINUX host. The concurrent queue used during the test is as follows:

http://item.taobao.com/item.htm?id=14392877692
http://item.taobao.com/item.htm?id=16231676302
http://item.taobao .com/item.htm?id=17037160462
http://item.taobao.com/item.htm?id=5522416710
http://item.taobao.com/item.htm?id=16551116403
http://item.taobao.com/item.htm?id=14088310973

Briefly explain the principles of experimental design and the format of performance test results: To ensure the reliability of the results, each set of experiments is repeated 20 times, in a single experiment, given the same interface URL set, measure the time consuming (in seconds) of the two concurrency mechanisms: Classic (referring to the classic concurrency mechanism) and Rolling (referring to the improved concurrency mechanism). The one with the shortest time is the winner (Winner), and the time saved (Excellence, in seconds) and performance improvement ratio (Excel. %) are calculated. In order to be as close to the real request as possible while keeping the experiment simple, in the processing of the returned results The above only performed simple regular expression matching without performing other complex operations. In addition, in order to determine the impact of the result processing callback on the performance comparison test results, usleep can be used to simulate more responsible data processing logic in reality (such as extraction, word segmentation, writing to files or databases, etc.).

The callback function used in the performance test is:

When the data processing callback has no delay: Rolling Curl is slightly better, but the performance The improvement effect is not obvious.
The data processing callback delay is 5 milliseconds: Rolling Curl wins, and the performance is improved by about 40%.
Through the above performance comparison, Rolling cURL should be a better choice in application scenarios that handle URL queue concurrency. , when the amount of concurrency is very large (1000+), you can control the maximum length of the concurrent queue, such as 20. Whenever a URL is returned and processed, a URL that has not yet been requested is immediately added to the queue. The code written in this way will It is more robust and will not get stuck or crash if the number of concurrency is too large. For detailed implementation, please refer to: http://code.google.com/p/rolling-curl/