Home Backend Development PHP Tutorial 流方式实现多线程采集有关问题,请高手分析上

流方式实现多线程采集有关问题,请高手分析上

Jun 13, 2016 am 10:52 AM
array data quot sockets

流方式实现多线程采集问题,请高手分析下
采集内容速度慢,我一直很头大,最近在研究多线程采集,下面贴出比较代码,有两个问题,一是获取的结果长度有点不一致;二是效率是不是还不够高?大伙帮忙分析,测试!

PHP code
<!--Code highlighting produced by Actipro CodeHighlighter (freeware)http://www.CodeHighlighter.com/--><?php $timeStart = microtimeFloat();function microtimeFloat() {    list($usec, $sec) = explode(" ", microtime());    return ((float)$usec + (float)$sec);}$data = '';$urls = array('http://www.tzksgs.com/news/2012-09/article-217.html', 'http://www.tzksgs.com/news/2012-09/article-219.html', 'http://www.tzksgs.com/news/2012-09/article-222.html');foreach($urls as $url){    echo strlen(file_get_contents($url)),'<br>';}$timeEnd = microtimeFloat();echo sprintf("Spend time: %s second(s)\n", $timeEnd - $timeStart),'<br>';$timeStart = microtimeFloat();$timeout = 30;$status = array();$retdata = array();$sockets = array();$userAgent = $_SERVER['HTTP_USER_AGENT'];foreach($urls as $id => $url) {    $tmp = parse_url($url);    $host = $tmp['host'];    $path = isset($tmp['path'])?$tmp['path']:'/';    empty($tmp['query']) or $path .= '?' . $tmp['query'];    if (empty($tmp['port'])) {        $port = $tmp['scheme'] == 'https' ? 443 : 80;    } else $port = $tmp['port'];    $fp = stream_socket_client("$host:$port", $errno, $errstr, 30);    if (!$fp) {        $status[$id] = "failed, $errno $errstr";    } else {        $status[$id] = "in progress";        $retdata[$id] = '';        $sockets[$id] = $fp;        fwrite($fp, "GET $path HTTP/1.1\r\nHost: $host\r\nUser-Agent: $userAgent\r\nConnection: Close\r\n\r\n");    }}// Now, wait for the results to come back inwhile (count($sockets)) {    $read = $write = $sockets;    //This is the magic function - explained below    if (stream_select($read, $write = null, $e = null, $timeout)) {        //readable sockets either have data for us, or are failed connection attempts        foreach ($read as $r) {            $id = array_search($r, $sockets);            $data = fread($r, 8192);            if (strlen($data) == 0) {                if ($status[$id] == "in progress") {                    $status[$id] = "failed to connect";                }                fclose($r);                unset($sockets[$id]);            } else {                $retdata[$id] .= $data;            }        }    }}foreach($retdata as $data){    $data = trim(substr($data, strpos($data, "\r\n\r\n") + 4));    echo strlen($data),'<br>';}$timeEnd = microtimeFloat();echo sprintf("Spend time: %s second(s)\n", $timeEnd - $timeStart);?>
Copy after login


------解决方案--------------------
你可以尝试 curl_multi_.... 并发执行
这样可尽可能的减少 php 指令,至于楼上两位说的问题。绝不是php所能解决的

------解决方案--------------------
当然,file_get_contents()是阻塞型的,所以如果是执行多个抓取任务,当然会慢。
而socket_*(), fsockopen(), stream_*()都是非阻塞的。
------解决方案--------------------
慢到什么程度? 

试下加上这个:

$context = stream_context_create(array('http' => array('header'=>'Connection: close')));
file_get_contents(".....",false,$context);
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. How to Fix Audio if You Can't Hear Anyone
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Sort array using Array.Sort function in C# Sort array using Array.Sort function in C# Nov 18, 2023 am 10:37 AM

Title: Example of using the Array.Sort function to sort an array in C# Text: In C#, array is a commonly used data structure, and it is often necessary to sort the array. C# provides the Array class, which has the Sort method to conveniently sort arrays. This article will demonstrate how to use the Array.Sort function in C# to sort an array and provide specific code examples. First, we need to understand the basic usage of the Array.Sort function. Array.So

php提交表单通过后,弹出的对话框怎样在当前页弹出,该如何解决 php提交表单通过后,弹出的对话框怎样在当前页弹出,该如何解决 Jun 13, 2016 am 10:23 AM

php提交表单通过后,弹出的对话框怎样在当前页弹出php提交表单通过后,弹出的对话框怎样在当前页弹出而不是在空白页弹出?想实现这样的效果:而不是空白页弹出:------解决方案--------------------如果你的验证用PHP在后端,那么就用Ajax;仅供参考:HTML code

Simple and clear method to use PHP array_merge_recursive() function Simple and clear method to use PHP array_merge_recursive() function Jun 27, 2023 pm 01:48 PM

When programming in PHP, we often need to merge arrays. PHP provides the array_merge() function to complete array merging, but when the same key exists in the array, this function will overwrite the original value. In order to solve this problem, PHP also provides an array_merge_recursive() function in the language, which can merge arrays and retain the values ​​of the same keys, making the program design more flexible. array_merge

How to use the array_combine function in PHP to combine two arrays into an associative array How to use the array_combine function in PHP to combine two arrays into an associative array Jun 26, 2023 pm 01:41 PM

In PHP, there are many powerful array functions that can make array operations more convenient and faster. When we need to combine two arrays into an associative array, we can use PHP's array_combine function to achieve this operation. This function is actually used to combine the keys of one array as the values ​​of another array into a new associative array. Next, we will explain how to use the array_combine function in PHP to combine two arrays into an associative array. Learn about array_comb

What data is in the data folder? What data is in the data folder? May 05, 2023 pm 04:30 PM

The data folder contains system and program data, such as software settings and installation packages. Each folder in the Data folder represents a different type of data storage folder, regardless of whether the Data file refers to the file name Data or the extension. Named data, they are all data files customized by the system or program. Data is a backup file for data storage. Generally, it can be opened with meidaplayer, notepad or word.

Detailed explanation of PHP array_fill() function usage Detailed explanation of PHP array_fill() function usage Jun 27, 2023 am 08:42 AM

In PHP programming, array is a very important data structure that can handle large amounts of data easily. PHP provides many array-related functions, array_fill() is one of them. This article will introduce in detail the usage of the array_fill() function, as well as some tips in practical applications. 1. Overview of the array_fill() function The function of the array_fill() function is to create an array of a specified length and composed of the same values. Specifically, the syntax of this function is

Detailed explanation and practical guide of Python Socket programming Detailed explanation and practical guide of Python Socket programming Apr 22, 2023 pm 05:04 PM

In today's Internet, the Socket protocol is one of the most important foundations. This article covers all areas of dealing with Socket programming in Python. Why use SocketsSockets are the various communication protocols that make up today's networks and make it possible to transfer information between two different programs or devices. For example, when we open a browser, we as clients create a connection to the server to transfer information. Before delving into this communication principle, let us first clarify what Sockets are. What are Sockets Generally speaking, Sockets are internal application protocols built for sending and receiving data. A single network will have two Socks

What to do if mysql load data is garbled? What to do if mysql load data is garbled? Feb 16, 2023 am 10:37 AM

The solution to the garbled mysql load data: 1. Find the SQL statement with garbled characters; 2. Modify the statement to "LOAD DATA LOCAL INFILE "employee.txt" INTO TABLE EMPLOYEE character set utf8;".

See all articles