php - Baidu and Google crawlers return 403 when accessing the laravel5.4 framework

Question

csrf verification is closed, and get, post access is allowed. The website can be accessed normally. The server does not use any security software. The firewall does not restrict any IP addresses. Has anyone encountered related problems?

Answer 1

There is a robots.txt file under public. You can see how it is set up. This is to control whether crawlers can crawl it.