Python error message: [Errno 24] Analysis and solution of Too many open files

Background

I recently discovered an error at work. When executing a multi-thread scanning script, the following error frequently occurs

HTTPConnectionPool(host=‘t.tips&#39;, port=80): Max retries exceeded with url: /index.php (Caused by NewConnectionError(&#39;<requests.packages.urllib3.connection.HTTPConnection object at 0x7f29d4081150>: Failed to establish a new connection: [Errno 24] Too many open files&#39;,))

It’s weird...

Analysis

Some small projects in the blogger’s own hands often require multi-threaded batch processing Scan to collect data. For functions like collecting data, the more threads the better, so usually I use 3000 threads for scanning. It will still occur when the bandwidth is sufficient and the system hardware configuration is high enough. The above problems make me think about whether it is due to system limitations.

Through the search engine, I found an introduction to the system restrictions. When I executed

$ ulimit -n
1024

, the result was 1024. The system limit is to open 1024 at the same time. files, which is obviously too few.

Solution

The solution to this problem is very simple, directly modify the following file

sudo vim /etc/security/limits.conf

Add two lines of code at the end of this file

* soft nofile 10240
* hard nofile 10240

Of course, you can modify this number according to your needs. After saving, log out and log in again. alright.

More Python error prompts: [Errno 24] Analysis and resolution of Too many open files For related articles, please pay attention to the PHP Chinese website!