Search Engine Technology Core Revealed (PHP Version)

When most people think of web search engines, they think of Yahoo. Indeed, Yahoo created an era of Internet search. However, the technology Yahoo currently uses to search the web was not originally developed by the company. In August 2000, Yahoo adopted technology from Google (www.google.com), a venture company founded by Stanford University students. The reason is very simple. Google's search engine can search for the required information faster and more accurately than the technology previously used by Yahoo.

It is probably impossible for us to design and develop a powerful and efficient search engine and database ourselves in terms of technology and funds in a short period of time. However, since Yahoo is using other people's technology, can we also use other people's ready-made technologies? What about the search engine website?

Analysis of programming ideas

We can imagine this: simulate a query, issue a search command in a corresponding format to a search engine website, then return the search results, analyze the HTML code of the results, strip off redundant characters and codes, and finally display them in the required format. within our own website pages.

In this way, the key to the problem is that we need to select a search that has accurate information (so that our search will be more meaningful), fast (because we need extra time to analyze the search results and display them), and the search results are concise (convenient for convenience). (HTML source code analysis and stripping) search website, due to various excellent features of the new generation search engine Google, here we choose it as an example to see how to use PHP to implement background search for Google (www.google.com), Front desk personalization displays this process.

Let’s first take a look at the structure of Google’s query command. Enter the www.google.com website, enter "abcd" in the query bar, and click the query button. We can find that the browser's address bar changes to: "http://www.google.com/search?q=abcd&btnG=Google %CB%D1%CB%F7&hl=zh-CN&lr=", it can be seen that Google passes the query parameters and submits the query command through the get method of the form. We can use the file() function in PHP to simulate this query process.

Understand the File() function

Syntax: array file(string filename);

The return value is an array, and all files are read into the array variable. The file here can be local or remote. The remote file must indicate the protocol used. For example: result=file("http://www.google.com/search?q=a ... mp;hl=zh-CN&lr="), this statement will simulate our query for the word "abcd" on Google process, and transfer the search results back to the array variable result in the form of elements per row. Because the file read here is remote, the protocol name "http://" cannot be missing.

If we want users to enter search characters for any search, we can make an input text box and submit button, and replace the searched character "abcd" above with a variable:

echo

echo
; file://没有参数的form，默认提交方式为get，提交到本身
echo ; file://构造一个文本输入框
echo ; file://构造一个提交查询按钮
echo
;

if (isset( keywords)) file://提交后PHP会生成变量 kwywords,即要求下面的程序在提交后运行
{
urlencode( keywords); file://对用户输入内容进行URL编码
result=file("http://www.google.com/search?q=". keywords."&btnG=Google%CB%D1%CB%F7&hl=zh-CN&lr=");
file://对查询语句进行变量替换,将查询结果保存在数组变量 result中
result_string=join(" ", result); file://将数组$result合并成字符串，各数组元素之间用空格粘和
... file://进一步处理
}
?＞

; file://form without parameters, the default submission method is get, submitted to itself echo ; file://Construct a text input box echo ; file://Construct a submit query button echo ; if (isset( keywords)) file://After submission, PHP will generate the variable kwywords, which requires the following program to run after submission { urlencode(keywords); file:// URL-encodes user input result=file("http://www.google.com/search?q=". keywords."&btnG=Google%CB%D1%CB%F7&hl=zh-CN&lr="); file:// performs variable substitution on the query statement and saves the query results in the array variable result result_string=join(" ", result); file://Combines the array $result into a string, and glues the array elements with spaces ... file://further processing } ?＞

The above program can already query based on user input and synthesize the returned results into a string variable $result_string. Please note that you must use the urlencode() function to URL-encode the user input so that the input Chinese characters, spaces, and other special characters can be queried normally. This also simulates Google's query commands as realistically as possible to ensure the search results are accurate. Correctness.

Analysis of Google

To make it easier to understand, let’s assume that what we really need is: the title of the search result. URL and introduction, etc. This is a simple and typical requirement. In this way, all we have to do is: remove the header and footer of Google search results, including a Google logo, input box for re-search, search result description, etc., and strip the original HTML from the remaining search result items. Format tag, replace it with the format we want.

To do this, we must carefully analyze the HTML source code of Google search results and find the patterns. It is not difficult to find that the text of Google search results is always included in the first
of the source code
Mark and penultimate

between tags, and the penultimate

The mark is followed by the table character, and this combination "

All the following procedures are continued in the "further processing" of the above procedures.

result_string = strstr( result_string, " ");

　　result_string = strstr( result_string, " ");
　　file://取 result_string从第一个开始后的字符串，以去除Google台头
　　position= strpos( result_string," table符号的位置
　　result_string= substr( result_string,0, position);//截取第一个table符号之前的字符串,以去除脚注

file://Get the string after result_string starting from the first one to remove the Google header
position= strpos( result_string," The position of the table symbol result_string= substr(result_string,0, position);//Intercept the string before the first table symbol to remove the footnote

Application and Implementation

Now that we have a useful backbone of HTML source code, the remaining question is how to display this content autonomously. Let's analyze these search result entries again and find that each entry is also separated very regularly, that is, each entry is a paragraph. According to this feature, we use the explode() function to cut each entry:

Syntax: explode(string separator, string string);

Returns an array, and each small string divided by separator is saved in the array.

So:

result_array=explode(" ", result_string); file://用字串" "把结果切开

We get an array result_array, where each element is a search result entry. All we have to do is study each entry and its HTML display format code, and then replace it as required. The following uses a loop to process each entry in result_array.

for( i=0; i {
... file://处理每个条目
}

for( i=0; i {
... file:// handles each entry
}

For each entry, we can easily find some characteristics: each entry is composed of title, abstract, introduction, category, URL, etc., and each part is line-wrapped, that is, it contains

every_item=explode(" ", result_array[ i]);

Mark, so split again: (The following handler is placed in the loop above)

echo eregi_replace( {
... file://处理每个条目中除去第一项（第一项为标题，已经显示）的每一项
... file://更多格式修改
}

In this way, we get an array every_item, where every_item[0] is the title, every_item[1] and every_item[2] are two lines of summary, if the headers of every_item[3], every_item[4], etc. contain "Introduction:", " ＜ font size=-1 color=#6f6f6f＞Category: ＜ /font＞” character, it is an introduction or category (because some result entries do not have this item), if the header contains "＜ font color=green＞", it is sure It is the URL. We often use regular expressions (omitted) for this kind of comparison and judgment. It is also very convenient if you want to replace it. For example, $every_item[0] containing the title has a link itself. We want to modify this link attribute so that It opens the link in a new window:

echo eregi_replace( { ... file://processes every item in each entry except the first item (the first item is the title, which is already displayed) ... file://More format modifications }

In this way, the link attributes are modified, and many other display format modifications, stripping, and replacements can be completed using regular replacement eregi_replace().

So far we have obtained each item of each search item, and can modify the format of each item arbitrarily, and even put a beautiful table on it. However, a good program should be able to adapt to various operating environments, and here is no exception. In fact, we have only discussed a framework method for HTML stripping of search results. To really do it perfectly, there are many things to consider, such as It displays the total number of search results, how many pages it is divided into, etc. It can even remove the "category", "introduction" and other codes related to Google, so that customers cannot see the original website at all. However, we can all extract these contents and requirements by analyzing HTML. Now everyone can do it themselves and build a highly personalized search engine.

Search Engine Technology Core Revealed (PHP Version)_PHP Tutorial