In the Go crawler framework Colly, how does the thread count setting of Queue and request delay affect the concurrent processing of requests?

The number of Queue threads and request delay of Go language crawler framework Colly

Efficient concurrent request processing is crucial when using the Go crawler framework Colly. This article will dig into how thread count settings and request delays in queue in Colly affect concurrent processing and answer a common question.

Problem: Interaction between number of threads and request delay

Suppose we set queue 's number of threads to 2:

 q, _ := queue.New(2, storage)

And added 3 requests. Meanwhile, colly.Limit() is used to set the delay of each request to 5 seconds. It is expected that two requests are issued almost simultaneously and respond after 5 seconds, and the third request is delayed by another 5 seconds. However, the actual result is:

1. Two requests are created.
2. After 5 seconds, the first request responds and a third request is created.
3. After 5 seconds, the second request responds.
4. After 5 seconds, the third request responds.

This is not processed in parallel. Why does the number of threads of queue seem to fail? Does colly.Limit() affect the concurrency of queue ? Is onrequest callback function just creating a request, not actually making a request?

Analysis: Independence between number of threads and request delay

Colly's queue manages the number of concurrent requests, while colly.Limit() sets the delay for each request. The two are independent mechanisms.

The number of threads of queue limits the number of requests processed simultaneously. colly.Limit() applies a delay before each request is issued.

In the above case:

1. queue creates two requests, but colly.Limit() makes them both wait for 5 seconds.
2. The first request is issued after the delay is over. After the response, queue releases a thread and creates a third request.
3. The second request is also sent and responded after waiting for 5 seconds.
4. The third request is also sent and responded after waiting for 5 seconds.

Therefore, the request delay masks the concurrency of queue .

onrequest callback and request issuance time

onrequest callback function is fired when the request is added to queue , not when the request is actually issued. It is used to perform some preprocessing operations before the request is issued.

Conclusion: Coordinate the number of threads and request delays

The delay of colly.Limit() will affect the concurrency effect of the number of queue threads. To achieve true concurrency, careful coordination of thread count and request delay settings is required. If high concurrency is required, the delay set by colly.Limit() should be minimized or removed, or a finer concurrency control mechanism should be considered. If you need to control the crawl speed, it is recommended to use a finer granular control method instead of relying on colly.Limit() .

The above is the detailed content of In the Go crawler framework Colly, how does the thread count setting of Queue and request delay affect the concurrent processing of requests?. For more information, please follow other related articles on the PHP Chinese website!