How to read certain lines of large files in php

How to implement PHP to read certain lines of a large file: 1. Directly use the file function to operate; 2. Call the Linux tail command to display the last few lines; 3. Use PHP's fseek to perform file operations; 4. , reading through PHP's stream_get_line function.

The operating environment of this article: windows7 system, PHP7.1 version, DELL G3 computer

php How to read certain lines of a large file ?

Introduction to various methods of reading large files in PHP

Reading large files has always been a headache. We can use various methods to read small files using PHP development. Function implementation, but once you get to the big article, you will find that the commonly used methods cannot be used normally or take too long and are too stuck. Let's take a look at the solutions to the problem of reading large files in PHP. I hope the example can help you.

In PHP, when reading files, the fastest way is to use some functions such as file and file_get_contents. A few simple lines of code can accomplish what we need beautifully. function. But when the file being operated is a relatively large file, these functions may be insufficient. The following will start with a requirement to explain the commonly used operating methods for reading large files.

Requirements:

There is an 800M log file with about 5 million lines. Use PHP to return the contents of the last few lines.

Implementation method:

1. Directly use the file function to operate

Since the file function reads all the contents into the memory at one time, PHP in order to prevent some write comparisons Bad programs occupy too much memory and cause insufficient system memory, causing the server to crash. Therefore, by default, the maximum memory usage is limited to 16M. This is set through memory_limit = 16M in php.ini. If this value Set to -1 for unlimited memory usage.

The following is a piece of code that uses file to extract the last line of this file:

<?php
　　ini_set(&#39;memory_limit&#39;, &#39;-1&#39;);
　　$file = &#39;access.log&#39;;
　　$data = file($file);
　　$line = $data[count($data) - 1];
　　echo $line;
?>

The entire code execution takes 116.9613 (s).

My machine has 2 G of memory. When I press F5 to run, the system turns gray and recovers after almost 20 minutes. It can be seen that the consequences of reading such a large file directly into the memory are what It's serious, so it's not a last resort. The memory_limit thing cannot be adjusted too high. Otherwise, the only choice is to call the computer room and ask the machine to be reset.

2. Directly call the Linux tail command to display the last few lines

Under the Linux command line, you can directly use tail -n 10 access.log to easily display the last few lines of the log file. , you can directly use PHP to call the tail command, and execute the PHP code as follows:

<?php
  $file = &#39;access.log&#39;;
  $file = escapeshellarg($file); // 对命令行参数进行安全转义
  $line = `tail -n 1 $file`;
  echo $line;
?>

The entire code execution takes 0.0034 (s)

3. Directly use PHP’s fseek to perform file operations

This method is the most common method. It does not need to read all the contents of the file, but operates directly through pointers, so the efficiency is quite efficient. When using fseek to operate files, there are many different methods, and the efficiency may be slightly different. The following are two commonly used methods:

Method 1

First, use fseek Find the last EOF of the file, then find the starting position of the last line, get the data of this line, then find the starting position of the next line, then take the position of this line, and so on, until the $num line is found.

The implementation code is as follows

<?php
$fp = fopen($file, "r");
$line = 10;
$pos = -2;
$t = " ";
$data = "";
while ($line > 0)
{
 while ($t != "＼n")
 {
 fseek($fp, $pos, SEEK_END);
 $t = fgetc($fp);
 $pos--;
 }
 $t = " ";
 $data .= fgets($fp);
 $line--;
}
fclose($fp);
echo $data
?>

The entire code execution takes 0.0095 (s)

Method 2

Still use fseek to read from the end of the file , but this time it is not reading bit by bit, but reading piece by piece. Every time a piece of data is read, the read data is placed in a buf, and then the number of newline characters (\n) is used to read it. Determine whether the last $num rows of data have been read.

The implementation code is as follows

<?php
$fp = fopen($file, "r");
$num = 10;
$chunk = 4096;
$fs = sprintf("%u", filesize($file));
$max = (intval($fs) == PHP_INT_MAX) ? PHP_INT_MAX : filesize($file);
for ($len = 0; $len < $max; $len += $chunk)
{
 $seekSize = ($max - $len > $chunk) ? $chunk : $max - $len;
 fseek($fp, ($len + $seekSize) * -1, SEEK_END);
 $readData = fread($fp, $seekSize) . $readData;
 if (substr_count($readData, "＼n") >= $num + 1)
 {
 preg_match("!(.*?＼n){" . ($num) . "}$!", $readData, $match);
 $data = $match[0];
 break;
 }
}
fclose($fp);
echo $data;
?>

The entire code execution takes 0.0009(s).

Method 3

<?php
function tail($fp, $n, $base = 5)
{
 assert($n > 0);
 $pos = $n + 1;
 $lines = array();
 while (count($lines) <= $n)
 {
 try
 {
  fseek($fp, -$pos, SEEK_END);
 }
 catch (Exception $e)
 {
  fseek(0);
  break;
 }
 $pos *= $base;
 while (!feof($fp))
 {
  array_unshift($lines, fgets($fp));
 }
 }
 return array_slice($lines, 0, $n);
}
var_dump(tail(fopen("access.log", "r+"), 10));
?>

The entire code execution takes 0.0003(s)

Method 4, PHP’s stream_get_line function, reads quickly, and reads 500,000 pieces of data file, it will take about 20 seconds! The example code is as follows

$fp = fopen('./iis.log', 'r'); //文件 
while (!feof($fp)) { 
 //for($j=1;$j<=1000;$j++) {     //读取下面的1000行并存储到数组中 
 $logarray[] = stream_get_line($fp, 65535, "＼n"); 
    // break;
 // } 
  
 }

Recommended learning: "PHP Video Tutorial"

The above is the detailed content of How to read certain lines of large files in php. For more information, please follow other related articles on the PHP Chinese website!