python - 禁止自己的网站被爬虫爬去？

Question

禁止自己的网站被爬虫爬去？有什么方法啊

Answer 1

If you are defending against targeted crawlers, you can set some access restrictions, such as access frequency, add verification codes, etc.

Answer 2

1. Important content is dynamically added using js

2. limit http_referer

3. Different interfaces can consider different templates, the kind that a set of regular expressions cannot perfectly match

4. Add some random copyright information to content that may be crawled

5. . You can only visit after logging in

6. Record access log

That’s all I can think of, but if you really want to catch it, these will just make it a little more difficult to catch it

Answer 3

You can modify robots.txt to disable search engine crawling
It is a bit difficult to disable personal crawling. You can only increase the difficulty, such as adding more complex verification codes, access frequency, regular style/data format changes, etc.