PHP implementation code for reading large files_PHP tutorial-PHP Tutorial-php.cn

PHP implementation code for reading large files_PHP tutorial

WBOY

Release： 2016-07-21 15:13:05

Original

661 people have browsed it

In PHP, when reading files, the fastest way is to use some functions such as file and file_get_contents. A few simple lines of code can beautifully complete the functions we need. But when the file being operated is a relatively large file, these functions may be insufficient. The following will start with a requirement to explain the commonly used operating methods when reading large files.
Requirements

There is an 800M log file with about 5 million lines. Use PHP to return the contents of the last few lines.

Implementation method

1. Directly use the file function to operate

Note: Since the file function reads all the contents into the memory at once, and In order to prevent some poorly written programs from taking up too much memory and causing insufficient system memory and causing the server to crash, PHP limits the maximum memory usage to 16M by default. This is done through memory_limit = 16M in php.ini. To set, if this value is set to -1, the memory usage is not limited.

The following is a piece of code that uses file to extract the last line of this file.
The entire code execution takes 116.9613 (s).

Copy code The code is as follows:

 
$fp = fopen($file, "r"); 
$num = 10; 
$chunk = 4096; 
$fs = sprintf("%u", filesize($file)); 
$max = (intval($fs) == PHP_INT_MAX ) ? PHP_INT_MAX : filesize($file); 
for ($len = 0; $len < $max; $len += $chunk) { 
$seekSize = ($max - $len > $ chunk) ? $chunk : $max - $len; 
fseek($fp, ($len + $seekSize) * -1, SEEK_END); 
$readData = fread($fp, $seekSize) . $ readData; 

if (substr_count($readData, "n") >= $num + 1) { 
preg_match("!(.*?n){".($num)." }$!", $readData, $match); 
$data = $match[0]; 
break; 
} 
} 
fclose($fp); 
echo $data; 

My machine has 2G of memory. When I press F5 to run, the system turns gray and recovers after almost 20 minutes. It can be seen that all such large files can be read directly. into the memory, the consequences are quite serious, so it is not a last resort. The memory_limit cannot be adjusted too high, otherwise the only choice is to call the computer room and reset the machine.

2. Directly call Linux The tail command displays the last few lines of the log file

Under the Linux command line, you can directly use tail -n 10 access.log to easily display the last few lines of the log file. You can directly use php to call the tail command. , execute the php code as follows.
The entire code execution takes 0.0034 (s)

Copy the code The code is as follows:

 
file = 'access.log'; 
$file = escapeshellarg($file); // Safely escape command line parameters
$line = `tail -n 1 $file`; 
echo $line; 

3. Directly use php’s fseek to perform file operations

This method is the most common method, it does not require Read all the contents of the file into the content, but operate directly through pointers, so the efficiency is quite efficient. When using fseek to operate the file, there are many different methods, and the efficiency may be slightly different, as follows There are two commonly used methods.

Method 1
First find the last EOF of the file through fseek, then find the starting position of the last line, and get the data of this line. Find the starting position of the next row, then take the position of this row, and so on, until the $num row is found.
The implementation code is as follows
The entire code execution takes 0.0095 (s)

Copy the code The code is as follows:

 
function tail($fp,$n,$base=5) 
{ 
assert($n>0); 
$pos = $n+1; 
$lines = array() ; 
while(count($lines)< =$n){ 
try{ 
fseek($fp,-$pos,SEEK_END); 
} catch (Exception $e){ 
fseek(0); 
break; 
} 
$pos *= $base; 
while(!feof($fp)){ 
array_unshift($lines,fgets($ fp)); 
} 
} 
return array_slice($lines,0,$n); 
} 
var_dump(tail(fopen("access.log","r+") ,10)); 

Method 2
Still use fseek to read from the end of the file, but this time it is not read one by one, but one block Read one block, each time a block of data is read, the read data is placed in a buf, and then the number of newline characters (n) is used to determine whether the last $num rows of data have been read.
Implementation code As follows
The entire code execution takes 0.0009(s).

Copy the code The code is as follows:

 
$fp = fopen($file, "r"); 
$line = 10; 
$pos = -2; 
$t = " "; 
$data = ""; 
while ( $line > 0) { 
while ($t != "n") { 
fseek($fp, $pos, SEEK_END); 
$t = fgetc($fp); 
 $pos --; 
} 
$t = " "; 
$data .= fgets($fp); 
$line --; 
} 
fclose ($fp ); 
echo $data 

Method Three
The entire code execution takes 0.0003(s)

Copy the code The code is as follows:

 
ini_set('memory_limit','-1'); 
$file = 'access.log'; 
$data = file($file); 
$line = $data [count($data)-1]; 
echo $line; 