我有一個自己的筆記博客,經常在上面做一些技術文章分析。在查詢一些文章的時候,以前只能將要查詢的內容去mysql中用like模糊匹配。在文章多了的情況下,這樣做法效率肯定不行。於是我把目標投向了中文搜尋插件coreseek,並成功的把它使用到了我的專案當中。
效果圖:
希望透過這篇分析,讓有興趣的同學少走些彎路。
sphinx是開源的搜尋引擎,它支援英文的全文檢索。但英文的天然分詞符是空格,而中文則有比較複雜的分詞要求。中國人提供了一個可供企業使用的,基於Sphinx的中文全文檢索引擎。也就是說Coreseek實際上的核心還是Sphinx。但其最大的不同是coreseek有一個中文分詞利器mmseg。
系統:Ubuntu
http服務:Apache/2.2.22
Mysql: Ver 14.14 Distrib 5.5.41
PHP: PHP 5.3.10
安裝步驟
下載coreseek-3.2.14.tar.gz,並放置在/usr/local/src
首先,為了避免安裝中出現依賴包缺失,你需要補一下血
apt-get <span style="color: #0000ff;">install</span> <span style="color: #0000ff;">make</span> <span style="color: #0000ff;">gcc</span> g++ automake libtool mysql-client libmysqlclient15-dev libxml2-dev libexpat1-dev
上面那個指令你執行就行,不然可能會因為軟體包不夠新出現各種奇葩問題。例如我就更新了159M的軟體包。 (我遇到各種坑洞後才回頭補的血)
1,安裝mmseg分詞模組
cd /usr/local/<span style="color: #000000;">src </span><span style="color: #0000ff;">tar</span> zxvf coreseek-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span>.<span style="color: #0000ff;">tar</span><span style="color: #000000;">.gz #解压 cd coreseek</span>-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span><span style="color: #000000;"> cd mmseg</span>-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span><span style="color: #000000;"> .</span>/<span style="color: #000000;">bootstrap #输出的warning信息可以忽略,如果出现error则需要解决 .</span>/configure --prefix=/usr/local/<span style="color: #000000;">mmseg3 #配置 </span><span style="color: #0000ff;">make</span><span style="color: #000000;"> #编译 </span><span style="color: #0000ff;">make</span> <span style="color: #0000ff;">install</span> #安装
1.1)可能遇到的問題與解決方法:
在執行./bootstrap時出現./bootstrap: 27: ./bootstrap: autoconf: not found 的錯誤,
原因:因為沒有安裝automake 工具, (ubuntu 10.04)用下面的指令安裝好就可以了。
sudo apt-get install autoconf automake libtool
1.2)可能遇到的問題:在安裝分詞模組 mmseg ,編譯安裝到最後,出現annot find input file: src/Makefile.in 的錯誤
然後查了一下,只找到解決方法,如下:
aclocal //是一個perl 腳本程序,它的定義是:「aclocal - create aclocal.m4 by scanning configure.ac」
libtoolize --force <span style="color: #008000;">//</span><span style="color: #008000;">运行后有一个错误,不用管它。<br></span> automake --add-<span style="color: #000000;">missing<br> autoconf<br> autoheader<br> </span><span style="color: #0000ff;">make</span> clean
然後在重新編譯
./configure --prefix=/usr/local/mmseg3
make && make install
編譯安裝成功
總結:其實這個報錯原因我也沒查出來。反正按照解決辦法就成功了,誰若知道拜託請留言,謝謝。
2、安裝CoreSeek
cd /usr/local/<span style="color: #000000;">src cd coreseek</span>-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span><span style="color: #000000;"> cd csft</span>-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span> <span style="color: #0000ff;">sh</span> buildconf.<span style="color: #0000ff;">sh</span><span style="color: #000000;"> #输出的warning信息可以忽略,如果出现error则需要解决 .</span>/configure --prefix=/usr/local/coreseek --without-unixodbc --with-mmseg --with-mmseg-includes=/usr/local/mmseg3/include/mmseg/ --with-mmseg-libs=/usr/local/mmseg3/lib/ --with-<span style="color: #000000;">mysql #配置 </span><span style="color: #0000ff;">make</span><span style="color: #000000;"> #编译 </span><span style="color: #0000ff;">make</span> <span style="color: #0000ff;">install</span> #安装
3,測試mmseg分詞、coreseek搜尋、MySQL資料來源
cd /usr/local/src
cd coreseek-3.2.14
cd testpack
cat /usr/local/src/coreseek-3.2.14/testpack/var/test/test.xml #此時應該正確顯示中文,如下圖所示
/usr/local/mmseg3/bin/mmseg -d /usr/local/mmseg3/bin/mmseg -d /usr/local/mmseg3/etc /usr/local/src/coreseek-3.2.14/testpack/var/test/test.xml
/usr/local/coreseek/bin/indexer -c /usr/local/src/coreseek-3.2.14/testpack/etc/csft.conf --all
/usr/local/coreseek/bin/search -c /usr/local/src/coreseek-3.2.14/testpack/etc/csft.conf 網路搜尋
/usr/local/coreseek/bin/indexer -c /usr/local/coreseek/etc/sphinx-min.conf.dist
/usr/local/coreseek/bin/indexer -c /usr/local/src/coreseek-3.2.14/testpack/etc/csft.conf --all --rotate #啟動服務,更新索引
如圖,且無報錯。則表示你的coreseek已經能夠正常運作了。
3.1)可能遇到的問題與解決方法:
在輸入/usr/local/coreseek/bin/indexer -c etc/csft.conf --all,報xmlpipe2 support NOT compiled in. To use xmlpipe2, install missing錯誤
原因:
缺少xmlpipe2函式庫,解決:
apt-get install expat-*
然後重新編譯coreseek,記得make clean
4,coreseek配置與使用
<span style="color: #0000ff;">cp</span> /usr/local/src/coreseek-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span>/testpack/etc/csft_mysql.conf /usr/local/coreseek/etc/<span style="color: #000000;">csft_mysql.conf #拷贝MySQL数据源配置文件 </span><span style="color: #0000ff;">ln</span> -s /usr/local/coreseek/etc/csft_mysql.conf /etc/<span style="color: #000000;">csft_mysql.conf #添加软连接 vim </span>/etc/csft_mysql.conf #编辑,修改
以我自己的配置文件为例:
/usr/local/coreseek/etc/csft_mysql.conf
<span style="color: #000000;">#索引源定义 source mysql { type </span>=<span style="color: #000000;"> mysql sql_host </span>=<span style="color: #000000;"> localhost sql_user </span>=<span style="color: #000000;">xxxx sql_pass </span>=<span style="color: #000000;">xxxx sql_db </span>=<span style="color: #000000;">xxxx sql_port </span>= <span style="color: #800080;">3306</span><span style="color: #000000;"> sql_query_pre </span>=<span style="color: #000000;"> SET NAMES utf8 sql_query </span>= SELECT <span style="color: #0000ff;">id</span>,<span style="color: #0000ff;">id</span><span style="color: #000000;">,uid,title,data FROM notebook_notepad #sql_query第一列id需为整数 #title、data作为字符串</span>/<span style="color: #000000;">文本字段,被全文索引 sql_attr_uint </span>= <span style="color: #0000ff;">id</span><span style="color: #000000;"> #从SQL读取到的值必须为整数 #sql_attr_timestamp </span>= <span style="color: #0000ff;">time</span><span style="color: #000000;"> #从SQL读取到的值必须为整数,作为时间属性 sql_attr_uint </span>=<span style="color: #000000;"> uid sql_query_info_pre </span>=<span style="color: #000000;"> SET NAMES utf8 #命令行查询时,设置正确的字符集 sql_query_info </span>= SELECT * FROM notebook_notepad WHERE <span style="color: #0000ff;">id</span>=$<span style="color: #0000ff;">id</span><span style="color: #000000;"> #命令行查询时,从数据库读取原始数据信息 } #index定义 index mysql { source </span>=<span style="color: #000000;"> mysql #对应的source名称 path </span>=/usr/local/coreseek/var/data/mysql #请修改为实际使用的绝对路径,例如:/usr/local/coreseek/var/<span style="color: #000000;">... docinfo </span>=<span style="color: #000000;"> extern mlock </span>= <span style="color: #800080;">0</span><span style="color: #000000;"> morphology </span>=<span style="color: #000000;"> none min_word_len </span>= <span style="color: #800080;">1</span><span style="color: #000000;"> html_strip </span>= <span style="color: #800080;">0</span><span style="color: #000000;"> #中文分词配置,详情请查看:http:</span><span style="color: #008000;">//</span><span style="color: #008000;">www.coreseek.cn/products-install/coreseek_mmseg/</span> charset_dictpath = /usr/local/mmseg3/etc/ #BSD、Linux环境下设置,/<span style="color: #000000;">符号结尾 #charset_dictpath </span>= etc/ #Windows环境下设置,/符号结尾,最好给出绝对路径,例如:C:/usr/local/coreseek/etc/<span style="color: #000000;">... charset_type </span>= zh_cn.utf-<span style="color: #800080;">8</span><span style="color: #000000;"> } #全局index定义 indexer { mem_limit </span>=<span style="color: #000000;"> 128M } #searchd服务定义 searchd { listen </span>= <span style="color: #800080;">9312</span><span style="color: #000000;"> read_timeout </span>= <span style="color: #800080;">5</span><span style="color: #000000;"> max_children </span>= <span style="color: #800080;">30</span><span style="color: #000000;"> max_matches </span>= <span style="color: #800080;">1000</span><span style="color: #000000;"> seamless_rotate </span>= <span style="color: #800080;">0</span><span style="color: #000000;"> preopen_indexes </span>= <span style="color: #800080;">0</span><span style="color: #000000;"> unlink_old </span>= <span style="color: #800080;">1</span><span style="color: #000000;"> pid_file </span>=/usr/local/coreseek/var/log/searchd_mysql.pid #请修改为实际使用的绝对路径,例如:/usr/local/coreseek/var/<span style="color: #000000;">... log </span>=/usr/local/coreseek/var/log/searchd_mysql.log #请修改为实际使用的绝对路径,例如:/usr/local/coreseek/var/<span style="color: #000000;">... query_log </span>=/usr/local/coreseek/var/log/query_mysql.log #请修改为实际使用的绝对路径,例如:/usr/local/coreseek/var/<span style="color: #000000;">... }</span>
这样搜索的话就会从索引文件中查出id,uid,title,data字段。
OK,配置完成以后,重启Coreseek 服务就能从生产你想要的查询索引,以后你就可以摆脱mysql的桎梏,什么中文,英文都可以,还自带分词。怎么样,是不是打开了新世界的大门。
下面讲一下重建索引可能出错的地方,以及解决办法。有兴趣的同学看一下,否则可以跳到下一节:PHP测试Coreseek 。
重建索引时报错误:WARNING: failed to open pid_file '/usr/local/coreseek/var/log/searchd_mysql.pid'.
解决方法:
试着停止一下coreseek服务
/usr/local/coreseek/bin/searchd -c /usr/local/coreseek/etc/csft_mysql.conf --stop 停止服务
然后重启
/usr/local/coreseek/bin/searchd -c /usr/local/coreseek/etc/csft_mysql.conf 启动服务
再次建立索引
/usr/local/coreseek/bin/indexer -c /usr/local/coreseek/etc/csft_mysql.conf --all 建立索引
如果提示:FATAL: failed to lock /usr/local/coreseek/var/data/xxxx.spl: Resource temporarily unavailable, will not index. Try --rotate option.
则尝试重建索引
/usr/local/coreseek/bin/indexer -c /usr/local/coreseek/etc/csft_mysql.conf --all --rotate 重建索引
1,将sphinxapi.php放到测试目录下
cp /usr/local/src/coreseek-3.2.14/testpack/api/sphinxapi.php ./
vim test.php
<span style="color: #008080;">header</span>("Content-type: text/html; charset=utf-8"<span style="color: #000000;">); </span><span style="color: #008000;">//</span><span style="color: #008000;">require("./ ");</span> <span style="color: #800080;">$s</span> = <span style="color: #0000ff;">new</span><span style="color: #000000;"> SphinxClient; </span><span style="color: #800080;">$s</span>->setServer("127.0.0.1", 9312<span style="color: #000000;">); </span><span style="color: #008000;">//</span><span style="color: #008000;">SPH_MATCH_ALL, 匹配所有查询词(默认模式); SPH_MATCH_ANY, 匹配查询词中的任意一个; SPH_MATCH_EXTENDED2, 支持特殊运算符查询</span> <span style="color: #800080;">$s</span>-><span style="color: #000000;">setMatchMode(SPH_MATCH_ALL); </span><span style="color: #800080;">$s</span>->setMaxQueryTime(30); <span style="color: #008000;">//</span><span style="color: #008000;">设置最大搜索时间</span> <span style="color: #800080;">$s</span>->SetArrayResult(<span style="color: #0000ff;">false</span>); <span style="color: #008000;">//</span><span style="color: #008000;">是否将Matches的key用ID代替</span> <span style="color: #800080;">$s</span>->SetSelect ( "*" ); <span style="color: #008000;">//</span><span style="color: #008000;">设置返回信息的内容,等同于SQL</span> <span style="color: #800080;">$s</span>-><span style="color: #000000;">SetRankingMode(SPH_RANK_BM25); </span><span style="color: #800080;">$s</span>->SetLimits ( 0, 30, 1000, 0 ); <span style="color: #008000;">//</span><span style="color: #008000;">设置结果集偏移量 SetLimits </span> <span style="color: #800080;">$res</span> = <span style="color: #800080;">$s</span>->query('coreseek','mysql','--single-0-query--'); <span style="color: #008000;">#</span><span style="color: #008000;">[coreseek]关键字,[mysql]数据源source</span> <span style="color: #800080;">$err</span> = <span style="color: #800080;">$s</span>-><span style="color: #000000;">GetLastError(); </span><span style="color: #0000ff;">echo</span> '<pre class="brush:php;toolbar:false">'<span style="color: #000000;">; </span><span style="color: #008080;">var_dump</span>(<span style="color: #800080;">$res</span><span style="color: #000000;">); </span><span style="color: #008080;">var_dump</span>(<span style="color: #800080;">$res</span>['matches'<span style="color: #000000;">]); </span><span style="color: #008080;">var_export</span>(<span style="color: #800080;">$err</span><span style="color: #000000;">); </span><span style="color: #0000ff;">echo</span> '';
php5 test.php
运行结果:matches为匹配后的结果集
1,Sphinx扩展安装安装
Coreseek官方教程中建议php使用直接include一个php文件进行操作,事实上php有独立的sphinx模块可以直接操作coreseek(coreseek就是sphinx!)已经进入了php的官方函数库,而且效率更高!但php模块依赖于libsphinxclient包。我是按照以下这篇文章的步骤安装了Sphinx扩展。
感谢:http://blog.csdn.net/e421083458/article/details/21529969
[第一步] 安装依赖libsphinxclient
# cd /var/<span style="color: #0000ff;">install</span>/coreseek-<span style="color: #800080;">4.1</span>-beta/csft-<span style="color: #800080;">4.1</span>/api/libsphinxclient/<span style="color: #000000;"> # .</span>/configure --prefix=/usr/local/<span style="color: #000000;">sphinxclient configure: creating .</span>/<span style="color: #000000;">config.status config.status: creating Makefile config.status: error: cannot </span><span style="color: #0000ff;">find</span> input <span style="color: #0000ff;">file</span>: Makefile.<span style="color: #0000ff;">in</span><span style="color: #000000;"> #报错configure失败 </span><span style="color: #008000;">//</span><span style="color: #008000;">处理configure报错</span> 编译过程中报了一个config.status: error: cannot <span style="color: #0000ff;">find</span> input <span style="color: #0000ff;">file</span>: src/<span style="color: #000000;">Makefile.in这个的错误,然后运行下列指令再次编译就能通过了: # aclocal # libtoolize </span>--<span style="color: #000000;">force # automake </span>--add-<span style="color: #000000;">missing # autoconf # autoheader # </span><span style="color: #0000ff;">make</span><span style="color: #000000;"> clean </span><span style="color: #008000;">//</span><span style="color: #008000;">从新configure编译</span> # ./<span style="color: #000000;">configure # </span><span style="color: #0000ff;">make</span> && <span style="color: #0000ff;">make</span> <span style="color: #0000ff;">install</span>
[第二步] 安装sphinx的PHP扩展
http:<span style="color: #008000;">//</span><span style="color: #008000;">pecl.php.net/package/sphinx</span> # <span style="color: #0000ff;">wget</span> http:<span style="color: #008000;">//</span><span style="color: #008000;">pecl.php.net/get/sphinx-1.3.0.tgz</span> # <span style="color: #0000ff;">tar</span> zxvf sphinx-<span style="color: #800080;">1.3</span>.<span style="color: #800080;">0</span><span style="color: #000000;">.tgz # cd sphinx</span>-<span style="color: #800080;">1.3</span>.<span style="color: #800080;">0</span><span style="color: #000000;"> # phpize # .</span>/configure --with-php-config=/usr/bin/php-config --with-sphinx=/usr/local/<span style="color: #000000;">sphinxclient # </span><span style="color: #0000ff;">make</span> && <span style="color: #0000ff;">make</span> <span style="color: #0000ff;">install</span><span style="color: #000000;"> # cd </span>/etc/php.d/<span style="color: #000000;"> # </span><span style="color: #0000ff;">cp</span><span style="color: #000000;"> gd.ini sphinx.ini # </span><span style="color: #0000ff;">vi</span><span style="color: #000000;"> sphinx.ini extension</span>=<span style="color: #000000;">sphinx.so # service php</span>-fpm restart
安装完PHP的Sphinx扩展后,就可以直接使用$coreseek = new SphinxClient();而无需引入源文件了。
简单说一下我在TP里使用coreseek查询,并高亮关键词的思路:
1,通过sphinx查出id,uid的集合
2,然后$sql = "select * from post where id in($ids)";$res = mysql_query($sql);获取到数据库的真实数据
3,用BuildExcerpts将title和data的关键字高亮,然后分页展示
关键代码:
<span style="color: #800080;"> $where</span> = <span style="color: #0000ff;">array</span><span style="color: #000000;">(); </span><span style="color: #800080;">$where</span>['uid']=<span style="color: #800080;">$uid</span><span style="color: #000000;">; </span><span style="color: #0000ff;">if</span>(!<span style="color: #0000ff;">empty</span>(<span style="color: #800080;">$search</span>)){ <span style="color: #008000;">//</span><span style="color: #008000;">有需要查找的内容,则去 coreseek 忠查出对应的id</span> <span style="color: #800080;">$coreseek</span> = <span style="color: #0000ff;">new</span><span style="color: #000000;"> \SphinxClient(); </span><span style="color: #800080;">$coreseek</span>->setServer("127.0.0.1", 9312<span style="color: #000000;">); </span><span style="color: #008000;">//</span><span style="color: #008000;">SPH_MATCH_ALL, 匹配所有查询词(默认模式); SPH_MATCH_ANY, 匹配查询词中的任意一个; SPH_MATCH_EXTENDED2, 支持特殊运算符查询</span> <span style="color: #800080;">$coreseek</span>-><span style="color: #000000;">setMatchMode(SPH_MATCH_ALL); </span><span style="color: #800080;">$coreseek</span>->setMaxQueryTime(30); <span style="color: #008000;">//</span><span style="color: #008000;">设置最大搜索时间</span> <span style="color: #800080;">$coreseek</span>->SetArrayResult(<span style="color: #0000ff;">false</span>); <span style="color: #008000;">//</span><span style="color: #008000;">是否将Matches的key用ID代替</span> <span style="color: #800080;">$coreseek</span>->SetSelect ( "*" ); <span style="color: #008000;">//</span><span style="color: #008000;">设置返回信息的内容,等同于SQL</span> <span style="color: #800080;">$coreseek</span>->SetLimits ( 0, 30, 1000, 0 ); <span style="color: #008000;">//</span><span style="color: #008000;">设置结果集偏移量 SetLimits</span> <span style="color: #800080;">$res</span> = <span style="color: #800080;">$coreseek</span>->query(<span style="color: #800080;">$search</span>,'mysql','--single-0-query--'<span style="color: #000000;">); </span><span style="color: #800080;">$key</span> = <span style="color: #008080;">array_keys</span>(<span style="color: #800080;">$res</span>['matches'<span style="color: #000000;">]); </span><span style="color: #800080;">$where</span>['id']=<span style="color: #0000ff;">array</span>('in',<span style="color: #800080;">$key</span><span style="color: #000000;">); </span><span style="color: #800080;">$coreseek</span>-><span style="color: #000000;">close(); }</span><span style="color: #0000ff;">else</span><span style="color: #000000;">{ } </span><span style="color: #008000;">//</span><span style="color: #008000;">获取总数据条数</span> <span style="color: #800080;">$total</span>=<span style="color: #800080;">$mod</span>->where(<span style="color: #800080;">$where</span>)-><span style="color: #008080;">count</span>();
高亮的关键代码:
<span style="color: #0000ff;"> if</span>(!<span style="color: #0000ff;">empty</span>(<span style="color: #800080;">$search</span><span style="color: #000000;">)){ </span><span style="color: #800080;">$page</span>->parameter['search']=<span style="color: #800080;">$search</span><span style="color: #000000;">; </span><span style="color: #008000;">//</span><span style="color: #008000;">代码高亮</span> <span style="color: #800080;">$opt</span> = <span style="color: #0000ff;">array</span>("before_match"=>"<font style='font-weight:bold;color:#f00'>","after_match"=>"</font>"<span style="color: #000000;">); </span><span style="color: #800080;">$coreseek1</span> = <span style="color: #0000ff;">new</span><span style="color: #000000;"> \SphinxClient(); </span><span style="color: #800080;">$coreseek1</span>->setServer("127.0.0.1", 9312<span style="color: #000000;">); </span><span style="color: #800080;">$coreseek1</span>-><span style="color: #000000;">SetMatchMode(SPH_MATCH_ALL); </span><span style="color: #800080;">$i</span>=0<span style="color: #000000;">; </span><span style="color: #800080;">$tags_title</span>=<span style="color: #0000ff;">array</span><span style="color: #000000;">(); </span><span style="color: #0000ff;">foreach</span>(<span style="color: #800080;">$info</span> <span style="color: #0000ff;">as</span> <span style="color: #800080;">$key</span>=><span style="color: #800080;">$row</span><span style="color: #000000;">){ </span><span style="color: #800080;">$tags_title</span>[]=<span style="color: #800080;">$row</span>['title'<span style="color: #000000;">]; } </span><span style="color: #800080;">$replace_title</span>=<span style="color: #800080;">$coreseek1</span>->BuildExcerpts(<span style="color: #800080;">$tags_title</span>,'mysql',<span style="color: #800080;">$search</span>,<span style="color: #800080;">$opt</span><span style="color: #000000;">); </span><span style="color: #0000ff;">foreach</span>(<span style="color: #800080;">$info</span> <span style="color: #0000ff;">as</span> <span style="color: #800080;">$key</span>=>&<span style="color: #800080;">$row</span><span style="color: #000000;">){ </span><span style="color: #800080;">$info</span>[<span style="color: #800080;">$key</span>]['title']=<span style="color: #800080;">$replace_title</span>[<span style="color: #800080;">$key</span><span style="color: #000000;">]; } </span><span style="color: #800080;">$coreseek1</span>-><span style="color: #000000;">close(); }</span>
OK,至此,coreseek已经能够在TP里完美运行了。这篇分享的文章也可以画上句号了。以上是我一步步安装时的细节,记录下怕以后自己忘记,也希望能给感兴趣的同学一些帮助。文章信息量偏大,如果有哪些疏漏,万望指正!