Configuring the User-Agent method of filtering crawlers in Nginx

It’s hard to see, it’s basically a user agent called “yisouspider” that swiped the screen. I don’t know where the spider is from at first glance. It’s so shameless. .
Find the root directory configuration area, add the user agent filter judgment statement, and find that the one called "yisouspider" will directly return 403

Note 1: If you need to add multiple filters, do this

 ($http_user_agent ~* "spider1|spider2|spider3|spider4")

, just separate it with |

Note 2: If you are using a subdirectory blog, like mine, then you need to find a section like "location /blog/" to modify it

location / {
......其它配置
if ($http_user_agent ~* "yisouspider") {
return 403;
}
}

After completing the configuration and saving wq, reload nginx, then use the following command to test yourself and change the address yourself. If curl is not installed, I have no choice but to install it myself with apt or yum. It comes with a magic tool.

curl -i -a "yisouspider" www.slyar.com/blog/

Just see 403 returned, indicating that the configuration is successful

The above is the detailed content of Configuring the User-Agent method of filtering crawlers in Nginx. For more information, please follow other related articles on the PHP Chinese website!