Home Backend Development PHP Tutorial php curl批处理-可控并发异步

php curl批处理-可控并发异步

Jun 13, 2016 pm 01:11 PM
curl multi while

php curl批处理--可控并发异步
在php的手册里面有一段代码:

$mrc = curl_multi_init();
//发出请求
.......
$active = null;
		do {
		    $mrc = curl_multi_exec($mh, $active);
		} while ($mrc == CURLM_CALL_MULTI_PERFORM);
		
		while ($active && $mrc == CURLM_OK) {
		    if (curl_multi_select($mh) != -1) {
		        do {
		            $mrc = curl_multi_exec($mh, $active);
		        } while ($mrc == CURLM_CALL_MULTI_PERFORM);
		    }
		}
//下面是处理请求返回的结果
Copy after login


这段do...while是用来控制并发的,效果是将$mrc所有的请求完毕,再执行response结果

但如果我有1000个请求,那么curl批处理将并发1000个请求,显然是不合理,所以应该要控制一个并发数,并且将剩余的连接添加到请求队列里:
参考:http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/

  $mh = curl_multi_init();
        $ch = array();
        $chunck = 10; //并发控制数
        $all = count($urls);//所有的请求url数组
        $chunck = $all > $chunck ? $chunck : $all;
		
		$options = array(
			CURLOPT_HEADER=>FALSE,
			CURLOPT_RETURNTRANSFER=>TRUE,
			CURLOPT_FOLLOWLOCATION=>TRUE,
			CURLOPT_MAXREDIRS=>5,
			CURLOPT_USERAGENT=>'Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20100101 Firefox/6.0'
		);
		
		for($i = 0 ; $i <br><br>使用下来效果很好,没有副作用,并发数可控,应用之处多多,自己发挥想象吧
 <div class="clear">
                 
              
              
        
            </div>
Copy after login
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. How to Fix Audio if You Can't Hear Anyone
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to realize the mutual conversion between CURL and python requests in python How to realize the mutual conversion between CURL and python requests in python May 03, 2023 pm 12:49 PM

Both curl and Pythonrequests are powerful tools for sending HTTP requests. While curl is a command-line tool that allows you to send requests directly from the terminal, Python's requests library provides a more programmatic way to send requests from Python code. The basic syntax for converting curl to Pythonrequestscurl command is as follows: curl[OPTIONS]URL When converting curl command to Python request, we need to convert the options and URL into Python code. Here is an example curlPOST command: curl-XPOST https://example.com/api

In C language, what is the difference between while(1) and while(0)? In C language, what is the difference between while(1) and while(0)? Aug 31, 2023 am 10:45 AM

We know that in C language, 'while' keyword is used to define a loop that works based on the condition passed to the loop. Now, since the condition can have two values, true or false, the code inside the while block will be executed repeatedly if the condition is true and will not be executed if the condition is false. Now, by passing parameters to the while loop, we can differentiate between while(1) and while(0) because while(1) is a loop where the condition is always considered true and hence the code inside the block will start executing repeatedly. Furthermore, we can state that it is not 1 that is passed to the loop that makes the condition true, but if any non-zero integer is passed to the while loop, then it will be considered as the true condition, so

Tutorial on updating curl version under Linux! Tutorial on updating curl version under Linux! Mar 07, 2024 am 08:30 AM

To update the curl version under Linux, you can follow the steps below: Check the current curl version: First, you need to determine the curl version installed in the current system. Open a terminal and execute the following command: curl --version This command will display the current curl version information. Confirm available curl version: Before updating curl, you need to confirm the latest version available. You can visit curl's official website (curl.haxx.se) or related software sources to find the latest version of curl. Download the curl source code: Using curl or a browser, download the source code file for the curl version of your choice (usually .tar.gz or .tar.bz2

PHP8.1 released: Introducing curl for concurrent processing of multiple requests PHP8.1 released: Introducing curl for concurrent processing of multiple requests Jul 08, 2023 pm 09:13 PM

PHP8.1 released: Introducing curl for concurrent processing of multiple requests. Recently, PHP officially released the latest version of PHP8.1, which introduced an important feature: curl for concurrent processing of multiple requests. This new feature provides developers with a more efficient and flexible way to handle multiple HTTP requests, greatly improving performance and user experience. In previous versions, handling multiple requests often required creating multiple curl resources and using loops to send and receive data respectively. Although this method can achieve the purpose

From start to finish: How to use php extension cURL to make HTTP requests From start to finish: How to use php extension cURL to make HTTP requests Jul 29, 2023 pm 05:07 PM

From start to finish: How to use php extension cURL for HTTP requests Introduction: In web development, it is often necessary to communicate with third-party APIs or other remote servers. Using cURL to make HTTP requests is a common and powerful way. This article will introduce how to use PHP to extend cURL to perform HTTP requests, and provide some practical code examples. 1. Preparation First, make sure that php has the cURL extension installed. You can execute php-m|grepcurl on the command line to check

How to handle 301 redirection of web pages in PHP Curl? How to handle 301 redirection of web pages in PHP Curl? Mar 08, 2024 am 11:36 AM

How to handle 301 redirection of web pages in PHPCurl? When using PHPCurl to send network requests, you will often encounter a 301 status code returned by the web page, indicating that the page has been permanently redirected. In order to handle this situation correctly, we need to add some specific options and processing logic to the Curl request. The following will introduce in detail how to handle 301 redirection of web pages in PHPCurl, and provide specific code examples. 301 redirect processing principle 301 redirect means that the server returns a 30

what is linux curl what is linux curl Apr 20, 2023 pm 05:05 PM

In Linux, curl is a very practical tool for transferring data to and from the server. It is a file transfer tool that uses URL rules to work under the command line; it supports file upload and download, and is a comprehensive transfer tool. . Curl provides a lot of very useful functions, including proxy access, user authentication, ftp upload and download, HTTP POST, SSL connection, cookie support, breakpoint resume and so on.

How to set cookies in php curl How to set cookies in php curl Sep 26, 2021 am 09:27 AM

How to set cookies in php curl: 1. Create a PHP sample file; 2. Set cURL transmission options through the "curl_setopt" function; 3. Pass the cookie in CURL.

See all articles