How to Efficiently Stream and Decode Large JSON Arrays in Go?

Streaming JSON Arrays for Large Data

Decoding massive JSON arrays into memory can be problematic, leading to memory issues. To address this, streaming JSON elements is a more efficient solution.

The encoding/json package in Go provides a mechanism for streaming JSON data element by element. Here's an extended example:

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "strings"
)

func main() {
    const jsonStream = `
                [
                    {"Name": "Ed", "Text": "Knock knock."},
                    {"Name": "Sam", "Text": "Who's there?"},
                    {"Name": "Ed", "Text": "Go fmt."},
                    {"Name": "Sam", "Text": "Go fmt who?"},
                    {"Name": "Ed", "Text": "Go fmt yourself!"}
                ]
            `

    type Message struct {
        Name, Text string
    }

    dec := json.NewDecoder(strings.NewReader(jsonStream))

    // Read the opening bracket.
    t, err := dec.Token()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%T: %v\n", t, t)

    // Loop through the array elements.
    for dec.More() {
        var m Message

        // Decode the current element.
        err = dec.Decode(&m)
        if err != nil {
            log.Fatal(err)
        }

        fmt.Printf("%v: %v\n", m.Name, m.Text)
    }

    // Read the closing bracket.
    t, err = dec.Token()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%T: %v\n", t, t)
}

In this example, we explicitly handle the opening and closing brackets of the JSON array and decode each element individually, enabling us to stream through the JSON data efficiently without memory overhead.

The above is the detailed content of How to Efficiently Stream and Decode Large JSON Arrays in Go?. For more information, please follow other related articles on the PHP Chinese website!