Python - How to quickly confirm the status code of a web page with 200 million++ URLs?

Question

I used requests to write a multi-thread, but it feels a bit slow. Are there any other methods?

Answer 1

Use Tornado’s curl client support to close the connection after reading the request header. (I haven’t tried it yet. If the HTTP client it provides does not support closing the connection midway, you can use TCP and then use http-parser to parse it like I did.)

Okay, actually you can just add an extension to fetchtitle to get the status code... (remember to install pycurl)

Answer 2

Python is inherently slow. If you want to be fast, just write the tcp request directly and then read the reply. After reading the status, close the socket.

Answer 3

Using grequests, requests are encapsulated concurrently

https://github.com/kennethrei...

Answer 4

In this case, you can consider using gevent, tornado, scrapy-redis, asyncio!

Answer 5

Using Head to request can it be faster?