PHP processing ideas for large files_PHP tutorial

Requirements: There is a log file of about 1G with about 5 million lines. Use PHP to return the contents of the last few lines.

In PHP, when reading files, the fastest way is to use some functions such as file and file_get_contents. A few simple lines of code can beautifully complete the functions we need. But when the file being operated is a relatively large file, these functions may be insufficient. The following will start with a requirement to explain the commonly used operating methods when reading large files.

1. Directly use the file function to operate

Since the file function reads all the contents into the memory at once, and PHP is limited to only The maximum memory that can be used is 16M. This is set by memory_limit = 16M in php.ini. If this value is set to -1, the memory usage is not limited.

The following is a piece of code that uses file to extract the last line of this file. The code execution takes about 2 minutes.

01 $fp = fopen($file, "r");

02 $num = 10;

03 $chunk = 4096;

04 $fs = sprintf("%u", filesize($file));

05 $max = (intval($fs) == PHP_INT_MAX) ? PHP_INT_MAX : filesize($file);

06 for ($len = 0; $len < $max; $len += $chunk) {

07 $seekSize = ($max - $len > $chunk) ? $chunk : $max - $len;

08 fseek($fp, ($len + $seekSize) * -1, SEEK_END);

09 $readData = fread($fp, $seekSize) . $readData;

10

11 if (substr_count($readData, "n") >= $num + 1) {

12 preg_match("!(.*?n){".($num)."}$!", $readData, $match);

13 $data = $match[0];

14 break;

15 }

16 }

17 fclose($fp);

18 echo $data;

My machine has 2G of memory. When I press F5 to run, the system turns gray and only recovers after almost 20 minutes. It can be seen that the consequences of reading such a large file directly into the memory are serious, so As a last resort, memory_limit cannot be set too high, otherwise the only option is to call the computer room to reset the machine.

2. Directly call the linux tail command to display the last few lines

In the Linux command line, you can directly use tail -n 10 access.log to easily display the last few lines of the log file. You can directly use php to call the tail command and execute the php code as follows. The entire code execution takes 0.0034 (s)

1 file = 'access.log';

2 $file = escapeshellarg($file); // Safely escape command line parameters

3 $line = `tail -n 1 $file`;

4 echo $line;

3. Directly use php’s fseek to perform file operations

This method is the most common method. It does not need to read all the contents of the file into the memory, but operates directly through pointers, so the efficiency is quite efficient. When using fseek to operate the file, there are also There are many different methods, and the efficiency may be slightly different. The following are two commonly used methods.

Method 1:

First find the last EOF of the file through fseek, then find the starting position of the last line, take the data of this line, then find the starting position of the next line, then take the position of this line, and so on until found $num rows.

view sourceprint?

01 function tail($fp,$n,$base=5)

02 {

03 assert($n>0);

04 $pos = $n+1;

05 $lines = array();

06 while(count($lines)< =$n){

07 try{

08 fseek($fp,-$pos,SEEK_END);

09 } catch (Exception $e){

10 fseek(0);

11 break;

12 }  

13 $pos *= $base;

14 while(!feof($fp)){

15 array_unshift($lines,fgets($fp));

16 }  

17 }

18 return array_slice($lines,0,$n);

19 }

20 var_dump(tail(fopen("access.log","r+"),10));

Method 2:

Still use fseek to read from the end of the file, but this time it is not reading bit by bit, but reading piece by piece. Every time a piece of data is read, the read data is placed in a buf. , and then use the number of newline characters (n) to determine whether the last $num rows of data have been read.

01 $fp = fopen($file, "r");

02 $line = 10;

03 $pos = -2;

04 $t = " ";

05 $data = "";

06 while ($line > 0) {

07 while ($t != "n") {

08 fseek($fp, $pos, SEEK_END);

09 $t = fgetc($fp);

10 $pos --;

11 }

12 $t = " ";

13 $data .= fgets($fp);

14 $line --;

15 }

16 fclose ($fp);

17 echo $data

Method three:

1 ini_set('memory_limit','-1');

2 $file = 'access.log';

3 $data = file($file);

4 $line = $data[count($data)-1];

5 echo $line;

http://www.bkjia.com/PHPjc/477747.htmlwww.bkjia.comtruehttp: //www.bkjia.com/PHPjc/477747.htmlTechArticleRequirements: There is a log file of about 1G with about 5 million lines. Use php to return the last few lines. content. In php, when reading files, the fastest way is to use...