How to read and write files concurrently in Golang?

Concurrent reading of files: Create goroutine to read data blocks concurrently, and use pipeline communication coordination; Concurrent writing of files: Use goroutine to write data, and use mutex locks to protect write operations.

#How to read and write files concurrently in Golang?

Concurrent processing of file I/O is a common technique to improve the performance of Golang programs. With concurrent access, we can read and write data simultaneously, rather than waiting for one operation to complete before starting another. This article will show how to use goroutine to read and write files concurrently in Golang.

Concurrent reading

In order to read files concurrently, we can create multiple goroutines, each goroutine reads data blocks from the file. To coordinate these goroutines, we will use pipes to communicate.

package main

import (
    "bufio"
    "os"
    "fmt"
    "sync"
    "time"
)

func readFile(filePath string, wg *sync.WaitGroup, ch chan<- string) {
    defer wg.Done()
    file, err := os.Open(filePath)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        ch <- scanner.Text()
    }
}

func main() {
    // 创建一个容量为 10 的管道
    ch := make(chan string, 10)

    // 等待组用于跟踪 goroutine
    var wg sync.WaitGroup

    filePath := "filename.txt"
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go readFile(filePath, &wg, ch)
    }

    // 从管道中读取数据，并停止管道以等待所有 goroutine 完成
    go func() {
        wg.Wait()
        close(ch)
    }()

    for data := range ch {
        fmt.Println(data)
    }

    // 等待管道数据完全输出
    time.Sleep(1 * time.Second)
}

Concurrent writing

For concurrent writing to files, we use a similar approach but use goroutines to write data and use mutexes to ensure exclusive access to writes Enter operation.

package main

import (
    "os"
    "fmt"
    "sync"
)

func writeFile(filePath string, wg *sync.WaitGroup, data <-chan string) {
    defer wg.Done()

    // 使用互斥锁保护文件写入
    lock := new(sync.Mutex)

    file, err := os.OpenFile(filePath, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0666)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    for dataString := range data {
        lock.Lock()
        _, err = file.WriteString(dataString + "\n")
        if err != nil {
            fmt.Println(err)
            continue
        }
        lock.Unlock()
    }
}

func main() {
    // 创建一个容量为 10 的管道
    ch := make(chan string, 10)

    // 等待组用于跟踪 goroutine
    var wg sync.WaitGroup

    filePath := "filename.txt"
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go writeFile(filePath, &wg, ch)
    }

    // 向管道写入数据，并停止管道以等待所有 goroutine 完成
    go func() {
        for i := 0; i < 100; i++ {
            ch <- fmt.Sprintf("Data point %d\n", i)
        }
        close(ch)
    }()

    wg.Wait()
}

The above is the detailed content of How to read and write files concurrently in Golang?. For more information, please follow other related articles on the PHP Chinese website!