Generator Concurrency Pattern in Go: A Comprehensive Guide

⚠️ How to go about this series?

1. Run Every Example: Don't just read the code. Type it out, run it, and observe the behavior.
2. Experiment and Break Things: Remove sleeps and see what happens, change channel buffer sizes, modify goroutine counts.
Breaking things teaches you how they work
3. Reason About Behavior: Before running modified code, try predicting the outcome. When you see unexpected behavior, pause and think why. Challenge the explanations.
4. Build Mental Models: Each visualization represents a concept. Try drawing your own diagrams for modified code.

In our previous post, we explored the basics of goroutines and channels, the building blocks of Go's concurrency. Read here:

Understanding and visualizing Goroutines and Channels in Golang

Souvik Kar Mahapatra ・ Dec 20

#go #programming #learning #tutorial

Now, let's look at how these primitives combine to form powerful patterns that solve real-world problems.

In this post we'll cover Generator Pattern and will try to visualize them. So let's gear up as we'll be hands on through out the process.

Generator Pattern

A generator is like a fountain that continuously produces values that we can consume whenever needed.

In Go, it's a function that produces a stream of values and sends them through a channel, allowing other parts of our program to receive these values on demand.

Let's look at an example:

// generateNumbers creates a generator that produces numbers from 1 to max
func generateNumbers(max int) chan int {
    // Create a channel to send numbers
    out := make(chan int)

    // Launch a goroutine to generate numbers
    go func() {
        // Important: Always close the channel when done
        defer close(out)

        for i := 1; i <= max; i++ {
            out <- i  // Send number to channel
        }
    }()

    // Return channel immediately
    return out
}

// Using the generator
func main() {
    // Create a generator that produces numbers 1-5
    numbers := generateNumbers(5)

    // Receive values from the generator
    for num := range numbers {
        fmt.Println("Received:", num)
    }
}

In this example, our generator function does three key things:

1. Creates a channel to send values
2. Launches a goroutine to generate values
3. Returns the channel immediately for consumers to use

Why Use Generators?

1. Separate value production from consumption
2. Generate values on-demand (lazy evaluation)
3. Can represent infinite sequences without consuming infinite memory
4. Allow concurrent production and consumption of values

Real-world Use Case

Reading large files line by line:

func generateLines(filename string) chan string {
    out := make(chan string)
    go func() {
        defer close(out)
        file, err := os.Open(filename)
        if err != nil {
            return
        }
        defer file.Close()

        scanner := bufio.NewScanner(file)
        for scanner.Scan() {
            out <- scanner.Text()
        }
    }()
    return out
}

Now you might be thinking, what's so special about it? we can do the same like generating sequence of data or read line by line without goroutines. Isn't it an overkill? Let's try to visualize both cases:

Without the goroutines

// Traditional approach
func getNumbers(max int) []int {
    numbers := make([]int, max)
    for i := 1; i <= max; i++ {
        numbers[i-1] = i
        // Imagine some heavy computation here
        time.Sleep(100 * time.Millisecond)
    }
    return numbers
}

Here you have to wait for everything to be ready before you can start processing.

With goroutines

// Generator approach
func generateNumbers(max int) chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for i := 1; i <= max; i++ {
            out <- i
            // Same heavy computation
            time.Sleep(100 * time.Millisecond)
        }
    }()
    return out
}

You can start processing the data while the data is still being generated.

Key Benefits of Generator Pattern:

1. Non-Blocking Execution: Generation and processing happen simultaneously

2. Memory Efficiency: Can generate and process one value at a time, no need to store in the memory right away

3. Infinite Sequences: Can generate infinite sequences without memory issues

4. Backpressure Handling: If your consumer is slow, the generator naturally slows down (due to channel blocking), preventing memory overload.
   

// generateNumbers creates a generator that produces numbers from 1 to max
func generateNumbers(max int) chan int {
    // Create a channel to send numbers
    out := make(chan int)

    // Launch a goroutine to generate numbers
    go func() {
        // Important: Always close the channel when done
        defer close(out)

        for i := 1; i <= max; i++ {
            out <- i  // Send number to channel
        }
    }()

    // Return channel immediately
    return out
}

// Using the generator
func main() {
    // Create a generator that produces numbers 1-5
    numbers := generateNumbers(5)

    // Receive values from the generator
    for num := range numbers {
        fmt.Println("Received:", num)
    }
}

Common Pitfalls and Solutions

1. Forgetting to Close Channels

func generateLines(filename string) chan string {
    out := make(chan string)
    go func() {
        defer close(out)
        file, err := os.Open(filename)
        if err != nil {
            return
        }
        defer file.Close()

        scanner := bufio.NewScanner(file)
        for scanner.Scan() {
            out <- scanner.Text()
        }
    }()
    return out
}

1. Not Handling Errors

// Traditional approach
func getNumbers(max int) []int {
    numbers := make([]int, max)
    for i := 1; i <= max; i++ {
        numbers[i-1] = i
        // Imagine some heavy computation here
        time.Sleep(100 * time.Millisecond)
    }
    return numbers
}

1. Resource Leaks: When using generators with resources (like files), ensure proper cleanup:

// Generator approach
func generateNumbers(max int) chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for i := 1; i <= max; i++ {
            out <- i
            // Same heavy computation
            time.Sleep(100 * time.Millisecond)
        }
    }()
    return out
}

That's all for the generator pattern. Up next is Pipeline concurrency pattern. Stay tuned to clear your concepts on Golang concurrency.

Did I miss something? Got questions? Got something interesting to share? All comments are welcomed.

The above is the detailed content of Generator Concurrency Pattern in Go: A Comprehensive Guide. For more information, please follow other related articles on the PHP Chinese website!