How to Maximize Concurrent HTTP Requests in Go?

How to Effectively "Max Out" Concurrent HTTP Requests?

When experimenting with Go's performance, you may encounter limitations while attempting to execute a high volume of HTTP requests concurrently. This article explores the challenges faced and provides a solution to achieve maximum concurrency.

The Problem

Your initial approach involves launching a large number of goroutines to send HTTP requests in parallel, expecting them to utilize all available CPUs. However, you encounter errors due to file descriptor limits.

The Solution

To overcome these limitations, consider the following approaches:

• Use a bounded semaphore channel to control concurrency: Implement a buffered channel that serves as a semaphore, limiting the number of concurrent requests. By adjusting the channel buffer size and tuning GOMAXPROCS, you can optimize concurrency for your system.
• Leverage a worker pool with a dispatcher: Employ a worker pool pattern where requests are queued into a channel. A dispatcher populates the channel, while a pool of workers processes requests one at a time. This approach ensures that the maximum number of concurrent requests is maintained.
• Utilize a dedicated consumer goroutine to process responses: To avoid blocking request processing, employ a dedicated goroutine to consume responses from the worker pool. This ensures continuous request execution, allowing your system to handle more concurrent requests.

Optimized Code

Here's a modified version of your code that incorporates these optimizations:

package main

import (
    "fmt"
    "net/http"
    "runtime"
    "sync"
    "time"
)

var (
    reqs       int
    concurrent int
    work       chan *http.Request
    results    chan *http.Response
)

func init() {
    reqs = 1000000
    concurrent = 200
}

func main() {
    runtime.GOMAXPROCS(runtime.NumCPU())
    work = make(chan *http.Request, concurrent)
    results = make(chan *http.Response)
    start := time.Now()

    // Create a semaphore channel to limit concurrency
    sem := make(chan struct{}, concurrent)

    // Create a dispatcher to populate the work channel
    go func() {
        for i := 0; i < reqs; i++ {
            req, _ := http.NewRequest("GET", "http://localhost/", nil)
            work <- req
        }
        close(work) // Signal to workers that no more requests are incoming
    }()

    // Create a worker pool to process requests
    for i := 0; i < concurrent; i++ {
        go func() {
            for req := range work {
                resp, err := http.DefaultClient.Do(req)
                if err != nil {
                    fmt.Println(err)
                }
                results <- resp

                // Release semaphore token to allow another worker to proceed
                <-sem
            }
        }()
    }

    // Consume responses from worker pool
    var (
        conns int64
        totalSize int64
        wg     sync.WaitGroup
    )

    wg.Add(1)
    go func() {
        defer wg.Done()
        for {
            select {
            case resp, ok := <-results:
                if ok {
                    conns++
                    totalSize += resp.ContentLength
                    resp.Body.Close()
                } else {
                    return
                }
            }
        }
    }()

    // Block until all responses are processed
    wg.Wait()

    elapsed := time.Since(start)
    fmt.Printf("Connections:\t%d\nConcurrent:\t%d\nTotal size:\t%d bytes\nElapsed:\t%s\n", conns, concurrent, totalSize, elapsed)
}

By adjusting the concurrent variable and observing the results, you can determine the optimal concurrency level for your system, "maxing out" its capacity for concurrent HTTP requests.

The above is the detailed content of How to Maximize Concurrent HTTP Requests in Go?. For more information, please follow other related articles on the PHP Chinese website!