Detailed explanation of how to read large files in PHP_PHP Tutorial

This article will introduce to you the detailed explanation of the implementation method of reading large files in PHP. Students who need to know more can refer to it for reference.

The requirements are as follows: There is a log file of about 1G with about 5 million lines. Use PHP to return the contents of the last few lines.

Implementation method:

1. Directly use the file function to operate
Note: Since the file function reads all the content into the memory at one time, PHP in order to prevent some poor writing The program takes up too much memory and causes insufficient system memory, causing the server to crash. Therefore, by default, the maximum memory usage is limited to 16M. This is set through memory_limit = 16M in php.ini. If this value is set -1, the memory usage is not limited.

The following is a piece of code that uses file to extract the last line of this file.
The entire code execution takes 116.9613 (s).

The code is as follows Copy code ini_set('memory_limit','-1');

代码如下	复制代码
ini_set('memory_limit','-1'); $file = 'access.log'; $data = file($file); $line = $data[count($data)-1]; echo $line;

$file = 'access.log';

$data = file($file);

$line = $data[count( $data)-1];

echo $line;

My machine has 2 G of memory. When I press F5 to run, the system directly It turned gray and recovered after almost 20 minutes. It can be seen that the consequences of reading such a large file directly into the memory are serious, so it is not a last resort. The memory_limit cannot be adjusted too high, otherwise the only choice is to call the computer room. Let the machine reset.

2. Directly call the linux tail command to display the last few lines

代码如下	复制代码
file = 'access.log'; $file = escapeshellarg($file); // 对命令行参数进行安全转义 $line = `tail -n 1 $file`; echo $line;

In the linux command line, you can directly use tail -n 10 access.log to easily display For the last few lines of the log file, you can directly use php to call the tail command. The execution of the php code is as follows.

The entire code execution takes 0.0034 (s)

The code is as follows	Copy code
file = 'access.log'; $file = escapeshellarg($file); // Safely escape command line parameters $line = `tail -n 1 $file`; echo $line;

3. Directly use php’s fseek to perform file operations

代码如下	复制代码
function tail($fp,$n,$base=5) {     assert($n>0);     $pos = $n+1;     $lines = array();     while(count($lines)< =$n){ try{ fseek($fp,-$pos,SEEK_END); } catch (Exception $e){ fseek(0); break; } $pos *= $base; while(!feof($fp)){ array_unshift($lines,fgets($fp)); } } return array_slice($lines,0,$n); } var_dump(tail(fopen("access.log","r+"),10));

This method is the most common method. It does not need to read all the contents of the file into the memory, but It operates directly through pointers, so the efficiency is quite efficient. When using fseek to operate files, there are many different methods, and the efficiency may be slightly different. The following are two commonly used methods. <🎜> method 1: <🎜> First find the last EOF of the file through fseek, then find the starting position of the last line, get the data of this line, then find the starting position of the next line, then take the position of this line, and so on. Until the $num row is found. <🎜>The implementation code is as follows<🎜><🎜>The entire code execution takes 0.0095 (s)<🎜>

The code is as follows	Copy code
<🎜>function tail($fp,$n,$base=5)<🎜>{ <🎜> assert($n>0); $pos = $n+1; $lines = array(); while(count($lines)< =$n){< 🎜> try{<🎜> fseek($fp,-$pos,SEEK_END);<🎜> } catch (Exception $e){<🎜> fseek(0);<🎜> break;<🎜> }<🎜> $pos *= $base;<🎜> while(!feof($fp)){<🎜> array_unshift($lines,fgets($fp));<🎜> }<🎜> }<🎜> return array_slice($ lines,0,$n);<🎜>}<🎜>var_dump(tail(fopen("access.log","r+"),10));<🎜>

Method 2:

Still use fseek to read from the end of the file, but this time it is not reading bit by bit, but reading piece by piece. Each time a piece of data is read, it will be read The fetched data is placed in a buf, and then the number of newline characters (n) is used to determine whether the last $num rows of data have been read.
The implementation code is as follows
The entire code execution takes 0.0009 (s ).

$t = " ";$data = "";while ($line > 0) {
while ($t != "n") { fseek($fp, $pos, SEEK_END);

$t = fgetc($fp);

$pos --; } $t = " "; $data .= fgets($fp); $line -- ;}fclose ($fp);echo $data Method 3:

The entire code execution takes time to complete 0.0003(s)

The code is as follows

代码如下	复制代码
$fp = fopen($file, "r"); $line = 10; $pos = -2; $t = " "; $data = ""; while ($line > 0) {     while ($t != "n") {         fseek($fp, $pos, SEEK_END);         $t = fgetc($fp);         $pos --;     }     $t = " ";     $data .= fgets($fp);     $line --; } fclose ($fp); echo $data

Copy code

$fp = fopen($file, "r");

$line = 10;

代码如下

复制代码

$fp = fopen($file, "r"); $num = 10; $chunk = 4096; $fs = sprintf("%u", filesize($file)); $max = (intval($fs) == PHP_INT_MAX) ? PHP_INT_MAX : filesize($file); for ($len = 0; $len < $max; $len += $chunk) { $seekSize = ($max - $len > $chunk) ? $chunk : $max - $len;     fseek($fp, ($len + $seekSize) * -1, SEEK_END);     $readData = fread($fp, $seekSize) . $readData;       if (substr_count($readData, "n") >= $num + 1) {         preg_match("!(.*?n){".($num)."}$!", $readData, $match);         $data = $match[0];         break;     } } fclose($fp); echo $data;

$pos = -2;

The code is as follows

Copy code

$fp = fopen($file, "r");$num = 10;$chunk = 4096;$fs = sprintf("% u", filesize($file));$max = (intval($fs) == PHP_INT_MAX) ? PHP_INT_MAX : filesize($file);for ($len = 0; $len < $ max; $len += $chunk) {<🎜> $seekSize = ($max - $len > $chunk) ? $chunk : $max - $len; fseek($fp, ($len + $ seekSize) * -1, SEEK_END); $readData = fread($fp, $seekSize) . $readData; if (substr_count($readData, "n") >= $num + 1) { preg_match("!(.*?n){".($num)."}$!", $readData, $match); $data = $match[0]; break; }}fclose($fp);echo $data; http://www.bkjia.com/PHPjc/444602.htmlwww.bkjia.comtruehttp: //www.bkjia.com/PHPjc/444602.htmlTechArticleThis article will introduce you to the detailed implementation method of reading large files in php. Students who need to know more can Go to Reference. The requirements are as follows: There is a log file of about 1G, about...