How long does it take for Robots to take effect?

Sometimes we forget to write the robots.txt file in the early stage of building the website, and a lot of content that we don’t want search engines to include is included. At this time, we will find that it seems that a long time has passed. Pages that you do not want to be included are still included. How long does it take for robots to take effect?

Baidu Lee once said in public: Different sites are slightly different, but they usually take effect within a few days.

In fact, we can imagine the process of robots taking effect:

1. The spider discovers that the robots file has been modified and sends the information back

2 .Analyze the robots files before and after modification

3. Delete the index data that is not allowed to be included

There may be other steps in this process. However, due to the frequency of spider crawling on each website Different weights of the website (importance of the website) lead to a delay in the time for spiders to discover changes in robots or different processing priorities. The final result is that the effective time of robots is different.

Through the above analysis, we can conclude that the effective time of robots is not certain. If your robots do not take effect for a long time after being changed, you can complain here: tousu.baidu.com

The above is the detailed content of How long does it take for Robots to take effect?. For more information, please follow other related articles on the PHP Chinese website!