How Can I Efficiently Read the Last N Lines of a File in Python?

Get Last n Lines of a File, Similar to Tail

Seeking to read the last n lines of a file is a common requirement, reminiscent of the tail -n command in Unix-like systems. To achieve this, we need to find an efficient method that can provide this functionality.

A Proposal for Tail Implementation

One proposed approach involves estimating an average line length and gradually increasing it until a sufficient number of lines are read. This method, while reasonable, relies on estimating the line length and can perform worse in certain scenarios.

An Alternative Approach

A more robust alternative method involves iterating through the file in blocks. The block size can be tuned for optimal performance, and the method does not rely on any assumptions about line length. It continues to read blocks until the total number of desired lines are obtained. This technique ensures consistent and reliable performance across different file sizes and line lengths.

Optimization Considerations

While using this block-based method, it is important to consider the file's size in relation to the system's operating system (OS) block size. If the file is smaller than a single OS block, the method may result in redundant reads and lower performance. In such cases, aligning the block size with the OS block size can yield improvements. However, for large files, this optimization may not have a significant impact.

Python Implementations

The proposed alternative approach can be implemented in Python as follows:

def tail(f, lines=20):
    """Reads the last n lines from a file."""
    BLOCK_SIZE = 1024
    f.seek(0, 2)
    block_end_byte = f.tell()
    lines_to_go = lines
    block_number = -1
    blocks = []
    while lines_to_go > 0 and block_end_byte > 0:
        if (block_end_byte - BLOCK_SIZE > 0):
            f.seek(block_number*BLOCK_SIZE, 2)
            blocks.append(f.read(BLOCK_SIZE))
        else:
            f.seek(0, 0)
            blocks.append(f.read(block_end_byte))
        lines_found = blocks[-1].count(b'\n')  # Edit for Python 3.2 and up
        lines_to_go -= lines_found
        block_end_byte -= BLOCK_SIZE
        block_number -= 1
    all_read_text = b''.join(reversed(blocks))  # Edit for Python 3.2 and up
    return b'\n'.join(all_read_text.splitlines()[-lines:])  # Edit for Python 3.2 and up

This implementation allows for the specification of the number of lines to read, making it a flexible and versatile solution. It prioritizes robustness and performance, avoiding assumptions about line length or file size.

The above is the detailed content of How Can I Efficiently Read the Last N Lines of a File in Python?. For more information, please follow other related articles on the PHP Chinese website!