I encountered a problem today. I wrote a program to grab the addresses of 150,000 pictures from the Internet and stored them in a txt document. Each line saved the address of one picture. Now I want to download the picture and save it to my local machine, but the download speed is not very fast, so I use php to implement multi-threading under Linux. First, I split the txt file into 10 files of equal size, and each file is stored With 15,000 picture addresses and 10 programs running and saving pictures opened at the same time, the time may be only 1/10 of the original time. I’ll post the program below and let everyone know if there is a better way.
01
71 function for_save_img($num)
02
72 {
03
73 for($i = 0; $i <= $num; $i++)
04
74 {
05
75 system("/usr/local/bin/php index.php crawl save_img {$i} & > /tmp/null");
06
76 }
07
77 }
08
78
09
79 function save_img($num)
10
80 {
11
81 static $i = 0;
12
82 // Read the file into an array
13
83 $img_urllists = ROOTDIRPATH . "/static/image_{$num}.txt";
14
84 $arr_img_url = file($img_urllists);
15
85 foreach($arr_img_url as $imageurl)
16
86 {
17
87 $imageurl = trim($imageurl);
18
88 echo $imageurl;
19
89 $this->benchmark->mark("code_start");
20
90
21
91 $final_imageurl = "http://www.****.com/upload/UploadFile/" . $imageurl;
22
92 $img_open = file_get_contents($final_imageurl);
23
93 $ret = file_put_contents(ROOTDIRPATH . '/static/uploadimg/' . $imageurl, $img_open);
24
94 if($ret)
25
95 {
26
96 echo "Success...";
27
97 }
28
98 $this->benchmark->mark('code_end');
29
99
30
100 echo $this->benchmark->elapsed_time('code_start', 'code_end');
31
101 }
32
102 }
33
34
35
This program is run under the ci framework in the root directory of the website under the linux shell php index.php crawl for_save_img
36
If you have any good suggestions, welcome them and accept them with an open mind
Author: Yue Guanqun