Home Backend Development PHP Tutorial SEO关于robots.txt_PHP

SEO关于robots.txt_PHP

Jun 01, 2016 pm 12:34 PM
about search engine document

SEO

robots.txt和Robots META标签

平文胜 2003-10-29

我们知道,搜索引擎都有自己的“搜索机器人”(ROBOTS),并通过这些ROBOTS在网络上沿着网页上的链接(一般是http和src链接)不断抓取资料建立自己的数据库。 对于网站管理者和内容提供者来说,有时候会有一些站点内容,不希望被ROBOTS抓取而公开。为了解决这个问题,ROBOTS开发界提供了两个办法:一个是robots.txt,另一个是The Robots META标签。
一、 robots.txt
1、 什么是robots.txt? robots.txt是一个纯文本文件,通过在这个文件中声明该网站中不想被robots访问的部分,这样,该网站的部分或全部内容就可以不被搜索引擎收录了,或者指定搜索引擎只收录指定的内容。 当一个搜索机器人访问一个站点时,它会首先检查该站点根目录下是否存在robots.txt,如果找到,搜索机器人就会按照该文件中的内容来确定访问的范围,如果该文件不存在,那么搜索机器人就沿着链接抓取。 robots.txt必须放置在一个站点的根目录下,而且文件名必须全部小写。 网站 URL
相应的 robots.txt的 URL

http://www.w3.org/
http://www.w3.org/robots.txt
http://www.w3.org:80/
http://www.w3.org:80/robots.txt
http://www.w3.org:1234/
http://www.w3.org:1234/robots.txt
http://w3.org/
http://w3.org/robots.txt

2、 robots.txt的语法

"robots.txt"文件包含一条或更多的记录,这些记录通过空行分开(以CR,CR/NL, or NL作为结束符),每一条记录的格式如下所示:
":"。 在该文件中可以使用#进行注解,具体使用方法和UNIX中的惯例一样。该文件中的记录通常以一行或多行User-agent开始,后面加上若干Disallow行,详细情况如下:

User-agent:
该项的值用于描述搜索引擎robot的名字,在"robots.txt"文件中,如果有多条User-agent记录说明有多个robot会受到该协议的限制,对该文件来说,至少要有一条User-agent记录。如果该项的值设为*,则该协议对任何机器人均有效,在"robots.txt"文件中, "User-agent:*"这样的记录只能有一条。

Disallow :
该项的值用于描述不希望被访问到的一个URL,这个URL可以是一条完整的路径,也可以是部分的,任何以Disallow 开头的URL均不会被robot访问到。例如"Disallow: /help"对/help.phpl 和/help/index.phpl都不允许搜索引擎访问,而"Disallow: /help/"则允许robot访问/help.phpl,而不能访问/help/index.phpl。

任何一条Disallow记录为空,说明该网站的所有部分都允许被访问,在"/robots.txt"文件中,至少要有一条Disallow记录。如果 "/robots.txt"是一个空文件,则对于所有的搜索引擎robot,该网站都是开放的。

下面是一些robots.txt基本的用法:
l 禁止所有搜索引擎访问网站的任何部分:
User-agent: *
Disallow: / l 允许所有的robot访问
User-agent: *
Disallow:
或者也可以建一个空文件 "/robots.txt" file l 禁止所有搜索引擎访问网站的几个部分(下例中的cgi-bin、tmp、private目录)
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/ l 禁止某个搜索引擎的访问(下例中的BadBot)
User-agent: BadBot
Disallow: / l 只允许某个搜索引擎的访问(下例中的WebCrawler)
User-agent: WebCrawler
Disallow: User-agent: *
Disallow: /

3、 常见搜索引擎机器人Robots名字
名称 搜索引擎
Baiduspider http://www.baidu.com Scooter http://www.altavista.com ia_archiver http://www.alexa.com Googlebot http://www.google.com FAST-WebCrawler http://www.alltheweb.com Slurp http://www.inktomi.com MSNBOT http://search.msn.com 

4、 robots.txt举例

下面是一些著名站点的robots.txt:

http://www.cnn.com/robots.txt

http://www.google.com/robots.txt

http://www.ibm.com/robots.txt

http://www.sun.com/robots.txt

http://www.eachnet.com/robots.txt

5、 常见robots.txt错误

l 颠倒了顺序:
错误写成
User-agent: *
Disallow: GoogleBot

正确的应该是:
User-agent: GoogleBot
Disallow: *

l 把多个禁止命令放在一行中:
例如,错误地写成
Disallow: /css/ /cgi-bin/ /../images/

正确的应该是
Disallow: /css/
Disallow: /cgi-bin/
Disallow: /../images/

l 行前有大量空格
例如写成
Disallow: /cgi-bin/
尽管在标准没有谈到这个,但是这种方式很容易出问题。

l 404重定向到另外一个页面:
当Robot访问很多没有设置robots.txt文件的站点时,会被自动404重定向到另外一个Html页面。这时Robot常常会以处理robots.txt文件的方式处理这个Html页面文件。虽然一般这样没有什么问题,但是最好能放一个空白的robots.txt文件在站点根目录下。

l 采用大写。例如
USER-AGENT: EXCITE
DISALLOW:
虽然标准是没有大小写的,但是目录和文件名应该小写:
user-agent:GoogleBot
disallow:

l 语法中只有Disallow,没有Allow!
错误的写法是:
User-agent: Baiduspider
Disallow: /john/
allow: /jane/

l 忘记了斜杠/
错误的写做:
User-agent: Baiduspider
Disallow: css

正确的应该是
User-agent: Baiduspider
Disallow: /css/

下面一个小工具专门检查robots.txt文件的有效性:http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

二、 Robots META标签

1、什么是Robots META标签

Robots.txt文件主要是限制整个站点或者目录的搜索引擎访问情况,而Robots META标签则主要是针对一个个具体的页面。和其他的META标签(如使用的语言、页面的描述、关键词等)一样,Robots META标签也是放在页面的

中,专门用来告诉搜索引擎ROBOTS如何抓取该页的内容。具体的形式类似(见黑体部分):

时代营销--网络营销专业门户

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to recover expired WeChat files? Can expired WeChat files be recovered? How to recover expired WeChat files? Can expired WeChat files be recovered? Feb 22, 2024 pm 02:46 PM

Open WeChat, select Settings in Me, select General and then select Storage Space, select Management in Storage Space, select the conversation in which you want to restore files and select the exclamation mark icon. Tutorial Applicable Model: iPhone13 System: iOS15.3 Version: WeChat 8.0.24 Analysis 1 First open WeChat and click the Settings option on the My page. 2 Then find and click General Options on the settings page. 3Then click Storage Space on the general page. 4 Next, click Manage on the storage space page. 5Finally, select the conversation in which you want to recover files and click the exclamation mark icon on the right. Supplement: WeChat files generally expire in a few days. If the file received by WeChat has not been clicked, the WeChat system will clear it after 72 hours. If the WeChat file has been viewed,

Photos cannot open this file because the format is not supported or the file is corrupted Photos cannot open this file because the format is not supported or the file is corrupted Feb 22, 2024 am 09:49 AM

In Windows, the Photos app is a convenient way to view and manage photos and videos. Through this application, users can easily access their multimedia files without installing additional software. However, sometimes users may encounter some problems, such as encountering a "This file cannot be opened because the format is not supported" error message when using the Photos app, or file corruption when trying to open photos or videos. This situation can be confusing and inconvenient for users, requiring some investigation and fixes to resolve the issues. Users see the following error when they try to open photos or videos on the Photos app. Sorry, Photos cannot open this file because the format is not currently supported, or the file

Can Tmp format files be deleted? Can Tmp format files be deleted? Feb 24, 2024 pm 04:33 PM

Tmp format files are a temporary file format usually generated by a computer system or program during execution. The purpose of these files is to store temporary data to help the program run properly or improve performance. Once the program execution is completed or the computer is restarted, these tmp files are often no longer necessary. Therefore, for Tmp format files, they are essentially deletable. Moreover, deleting these tmp files can free up hard disk space and ensure the normal operation of the computer. However, before deleting Tmp format files, we need to

How to transfer files from Quark Cloud Disk to Baidu Cloud Disk? How to transfer files from Quark Cloud Disk to Baidu Cloud Disk? Mar 14, 2024 pm 02:07 PM

Quark Netdisk and Baidu Netdisk are currently the most commonly used Netdisk software for storing files. If you want to save the files in Quark Netdisk to Baidu Netdisk, how do you do it? In this issue, the editor has compiled the tutorial steps for transferring files from Quark Network Disk computer to Baidu Network Disk. Let’s take a look at how to operate it. How to save Quark network disk files to Baidu network disk? To transfer files from Quark Network Disk to Baidu Network Disk, you first need to download the required files from Quark Network Disk, then select the target folder in the Baidu Network Disk client and open it. Then, drag and drop the files downloaded from Quark Cloud Disk into the folder opened by the Baidu Cloud Disk client, or use the upload function to add the files to Baidu Cloud Disk. Make sure to check whether the file was successfully transferred in Baidu Cloud Disk after the upload is completed. That's it

What to do if the 0x80004005 error code appears. The editor will teach you how to solve the 0x80004005 error code. What to do if the 0x80004005 error code appears. The editor will teach you how to solve the 0x80004005 error code. Mar 21, 2024 pm 09:17 PM

When deleting or decompressing a folder on your computer, sometimes a prompt dialog box "Error 0x80004005: Unspecified Error" will pop up. How should you solve this situation? There are actually many reasons why the error code 0x80004005 is prompted, but most of them are caused by viruses. We can re-register the dll to solve the problem. Below, the editor will explain to you the experience of handling the 0x80004005 error code. Some users are prompted with error code 0X80004005 when using their computers. The 0x80004005 error is mainly caused by the computer not correctly registering certain dynamic link library files, or by a firewall that does not allow HTTPS connections between the computer and the Internet. So how about

What is hiberfil.sys file? Can hiberfil.sys be deleted? What is hiberfil.sys file? Can hiberfil.sys be deleted? Mar 15, 2024 am 09:49 AM

Recently, many netizens have asked the editor, what is the file hiberfil.sys? Can hiberfil.sys take up a lot of C drive space and be deleted? The editor can tell you that the hiberfil.sys file can be deleted. Let’s take a look at the details below. hiberfil.sys is a hidden file in the Windows system and also a system hibernation file. It is usually stored in the root directory of the C drive, and its size is equivalent to the size of the system's installed memory. This file is used when the computer is hibernated and contains the memory data of the current system so that it can be quickly restored to the previous state during recovery. Since its size is equal to the memory capacity, it may take up a larger amount of hard drive space. hiber

How to install GHO files How to install GHO files Feb 19, 2024 pm 10:06 PM

The gho file is a GhostImage image file, which is usually used to back up the entire hard disk or partition data into a file. In some specific cases, we need to reinstall this gho file back to the hard drive to restore the hard drive or partition to its previous state. The following will introduce how to install the gho file. First, before installation, we need to prepare the following tools and materials: Entity gho file: Make sure you have a complete gho file, which usually has a .gho suffix and contains a backup

Different uses of slashes and backslashes in file paths Different uses of slashes and backslashes in file paths Feb 26, 2024 pm 04:36 PM

A file path is a string used by the operating system to identify and locate a file or folder. In file paths, there are two common symbols separating paths, namely forward slash (/) and backslash (). These two symbols have different uses and meanings in different operating systems. The forward slash (/) is a commonly used path separator in Unix and Linux systems. On these systems, file paths start from the root directory (/) and are separated by forward slashes between each directory. For example, the path /home/user/Docume

See all articles