How to write robots.txt

The writing method of robots.txt is something that SEO personnel must know (what is robots.txt), but how to write it, what is prohibited and what is allowed, we have to set it ourselves.

Baidu Spider is a machine. It only recognizes numbers, letters and Chinese characters, and robots.txt is the most important and first "dialogue" with Baidu.

When our website is not built yet, we don’t want Baidu to crawl our website, and some people often prohibit Baidu from crawling it. However, this approach is very bad, as it will make it difficult for Baidu spiders to come to your website again. Therefore, we must build the website locally now, and then buy the domain name and space after everything is done. Otherwise, repeated modifications of a website will have certain adverse effects on your website.

The initial robots.txt of our website is written as follows:

User-agent: *

Disallow: /wp-admin/

Disallow: / wp-includes/

User-agent: * means, allow all engines to crawl.

Disallow: /wp-admin/ and Disallow: /wp-includes/ prohibit Baidu from crawling our privacy, including user passwords, databases, etc. This way of writing not only protects our privacy, but also maximizes Baidu Spider’s crawling.

If you want to prohibit Baidu Spider from crawling a certain page, such as 123.html, then add the code "Disallow: /123.html/".

After writing robots.txt, you only need to upload it to the root directory of the website.

The above is the detailed content of How to write robots.txt. For more information, please follow other related articles on the PHP Chinese website!