How to Efficiently Read the Last Lines from a Large File in Go Every 10 Seconds?

How to Read Last Lines from a Big File with Go Every 10 Seconds

Challenge: Reading the last two lines of a large log file without loading the entire file into memory, repeating this process every 10 seconds.

Solution: Using Seek or ReadAt and Stat

Stat Function: To avoid loading the entire file into memory, the file's length can be obtained using the Stat() function of the os package. This provides the file size in bytes.

Seeking or Reading Forward:

• Seek() can be used to move the file pointer to the appropriate location. Reading forward a specific number of bytes from this position will fetch the desired lines.
• Alternatively, the ReadAt() function can be used to read a specific number of bytes starting at a particular position.

Consider the example provided in your question:

Code Snippet:

package main

import (
    "fmt"
    "os"
    "time"
)

const MYFILE = "logfile.log"

func main() {
    c := time.Tick(10 * time.Second)
    for _ = range c {
        readFile(MYFILE)
    }
}

func readFile(fname string) {
    file, err := os.Open(fname)
    if err != nil {
        panic(err)
    }
    defer file.Close()

    buf := make([]byte, 62)
    stat, statErr := file.Stat()
    if statErr != nil {
        panic(statErr)
    }
    start := stat.Size() - 62
    _, err = file.ReadAt(buf, start)
    if err == nil {
        fmt.Printf("%s\n", buf)
    }

}

In this example:

• The Stat() function is utilized to obtain the file size.
• ReadAt() is employed to read 62 bytes starting at a specific position (file size - 62).
• The last two lines of the log file, stored as bytes, are then printed.

The above is the detailed content of How to Efficiently Read the Last Lines from a Large File in Go Every 10 Seconds?. For more information, please follow other related articles on the PHP Chinese website!