我有一个自己的笔记博客,经常在上面做一些技术文章分析。在查询一些文章的时候,以前只能将要查询的内容去mysql中用like模糊匹配。在文章多了的情况下,这样做法效率肯定不行。于是我把目标投向了中文搜索插件coreseek,并成功的把它使用到了我的项目当中。
效果图:
希望通过这篇分析,让有兴趣的同学少走些弯路。
sphinx是开源的搜索引擎,它支持英文的全文检索。但英文的天然分词符是空格,而中文则有比较复杂的分词要求。中国人提供了一个可供企业使用的,基于Sphinx的中文全文检索引擎。也就是说Coreseek实际上的内核还是Sphinx。但其最大的不同是coreseek有一个中文分词利器mmseg。
系统:Ubuntu
http服务:Apache/2.2.22
Mysql: Ver 14.14 Distrib 5.5.41
PHP: PHP 5.3.10
安装步骤
下载coreseek-3.2.14.tar.gz,并放置在/usr/local/src
首先,为了避免安装中出现依赖包缺失,你需要补一下血
apt-get <span style="color: #0000ff;">install</span> <span style="color: #0000ff;">make</span> <span style="color: #0000ff;">gcc</span> g++ automake libtool mysql-client libmysqlclient15-dev libxml2-dev libexpat1-dev
上面那个命令你执行就行,不然可能会因为软件包不够新出现各种奇葩问题。比如我就更新了159M的软件包。(我在遇到各种坑后才回头补的血)
1,安装mmseg分词模块
cd /usr/local/<span style="color: #000000;">src </span><span style="color: #0000ff;">tar</span> zxvf coreseek-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span>.<span style="color: #0000ff;">tar</span><span style="color: #000000;">.gz #解压 cd coreseek</span>-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span><span style="color: #000000;"> cd mmseg</span>-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span><span style="color: #000000;"> .</span>/<span style="color: #000000;">bootstrap #输出的warning信息可以忽略,如果出现error则需要解决 .</span>/configure --prefix=/usr/local/<span style="color: #000000;">mmseg3 #配置 </span><span style="color: #0000ff;">make</span><span style="color: #000000;"> #编译 </span><span style="color: #0000ff;">make</span> <span style="color: #0000ff;">install</span> #安装
1.1)可能遇到的问题和解决办法:
在执行./bootstrap时出现./bootstrap: 27: ./bootstrap: autoconf: not found 的错误,
原因:因为没有安装automake 工具, (ubuntu 10.04)用下面的命令安装好就可以了。
sudo apt-get install autoconf automake libtool
1.2)可能遇到的问题:在安装分词模块 mmseg ,编译安装到最后,出现annot find input file: src/Makefile.in 的错误
然后查了一下,只找到解决办法,如下:
aclocal //是一个perl 脚本程序,它的定义是:“aclocal - create aclocal.m4 by scanning configure.ac”
libtoolize --force <span style="color: #008000;">//</span><span style="color: #008000;">运行后有一个错误,不用管它。<br></span> automake --add-<span style="color: #000000;">missing<br> autoconf<br> autoheader<br> </span><span style="color: #0000ff;">make</span> clean
然后在重新编译
./configure --prefix=/usr/local/mmseg3
make && make install
编译安装成功
总结:其实这个报错原因我也没查出来。反正按照解决办法就成功了,谁若知道拜托请留言,谢谢。
2、安装CoreSeek
cd /usr/local/<span style="color: #000000;">src cd coreseek</span>-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span><span style="color: #000000;"> cd csft</span>-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span> <span style="color: #0000ff;">sh</span> buildconf.<span style="color: #0000ff;">sh</span><span style="color: #000000;"> #输出的warning信息可以忽略,如果出现error则需要解决 .</span>/configure --prefix=/usr/local/coreseek --without-unixodbc --with-mmseg --with-mmseg-includes=/usr/local/mmseg3/include/mmseg/ --with-mmseg-libs=/usr/local/mmseg3/lib/ --with-<span style="color: #000000;">mysql #配置 </span><span style="color: #0000ff;">make</span><span style="color: #000000;"> #编译 </span><span style="color: #0000ff;">make</span> <span style="color: #0000ff;">install</span> #安装
3,测试mmseg分词、coreseek搜索、MySQL数据源
cd /usr/local/src
cd coreseek-3.2.14
cd testpack
cat /usr/local/src/coreseek-3.2.14/testpack/var/test/test.xml #此时应该正确显示中文,如下图所示
/usr/local/mmseg3/bin/mmseg -d /usr/local/mmseg3/etc /usr/local/src/coreseek-3.2.14/testpack/var/test/test.xml
/usr/local/coreseek/bin/indexer -c /usr/local/src/coreseek-3.2.14/testpack/etc/csft.conf --all
/usr/local/coreseek/bin/search -c /usr/local/src/coreseek-3.2.14/testpack/etc/csft.conf 网络搜索
/usr/local/coreseek/bin/indexer -c /usr/local/coreseek/etc/sphinx-min.conf.dist
/usr/local/coreseek/bin/indexer -c /usr/local/src/coreseek-3.2.14/testpack/etc/csft.conf --all --rotate #启动服务,更新索引
如图,且无报错。则说明你的coreseek已经能够正常运行了。
3.1)可能遇到的问题与解决办法:
在输入/usr/local/coreseek/bin/indexer -c etc/csft.conf --all,报xmlpipe2 support NOT compiled in. To use xmlpipe2, install missing错误
原因:
缺少xmlpipe2库,解决:
apt-get install expat-*
然后重新编译coreseek,记得make clean
4,coreseek配置和使用
<span style="color: #0000ff;">cp</span> /usr/local/src/coreseek-<span style="color: #800080;">3.2</span>.<span style="color: #800080;">14</span>/testpack/etc/csft_mysql.conf /usr/local/coreseek/etc/<span style="color: #000000;">csft_mysql.conf #拷贝MySQL数据源配置文件 </span><span style="color: #0000ff;">ln</span> -s /usr/local/coreseek/etc/csft_mysql.conf /etc/<span style="color: #000000;">csft_mysql.conf #添加软连接 vim </span>/etc/csft_mysql.conf #编辑,修改
以我自己的配置文件为例:
/usr/local/coreseek/etc/csft_mysql.conf
<span style="color: #000000;">#索引源定义 source mysql { type </span>=<span style="color: #000000;"> mysql sql_host </span>=<span style="color: #000000;"> localhost sql_user </span>=<span style="color: #000000;">xxxx sql_pass </span>=<span style="color: #000000;">xxxx sql_db </span>=<span style="color: #000000;">xxxx sql_port </span>= <span style="color: #800080;">3306</span><span style="color: #000000;"> sql_query_pre </span>=<span style="color: #000000;"> SET NAMES utf8 sql_query </span>= SELECT <span style="color: #0000ff;">id</span>,<span style="color: #0000ff;">id</span><span style="color: #000000;">,uid,title,data FROM notebook_notepad #sql_query第一列id需为整数 #title、data作为字符串</span>/<span style="color: #000000;">文本字段,被全文索引 sql_attr_uint </span>= <span style="color: #0000ff;">id</span><span style="color: #000000;"> #从SQL读取到的值必须为整数 #sql_attr_timestamp </span>= <span style="color: #0000ff;">time</span><span style="color: #000000;"> #从SQL读取到的值必须为整数,作为时间属性 sql_attr_uint </span>=<span style="color: #000000;"> uid sql_query_info_pre </span>=<span style="color: #000000;"> SET NAMES utf8 #命令行查询时,设置正确的字符集 sql_query_info </span>= SELECT * FROM notebook_notepad WHERE <span style="color: #0000ff;">id</span>=$<span style="color: #0000ff;">id</span><span style="color: #000000;"> #命令行查询时,从数据库读取原始数据信息 } #index定义 index mysql { source </span>=<span style="color: #000000;"> mysql #对应的source名称 path </span>=/usr/local/coreseek/var/data/mysql #请修改为实际使用的绝对路径,例如:/usr/local/coreseek/var/<span style="color: #000000;">... docinfo </span>=<span style="color: #000000;"> extern mlock </span>= <span style="color: #800080;">0</span><span style="color: #000000;"> morphology </span>=<span style="color: #000000;"> none min_word_len </span>= <span style="color: #800080;">1</span><span style="color: #000000;"> html_strip </span>= <span style="color: #800080;">0</span><span style="color: #000000;"> #中文分词配置,详情请查看:http:</span><span style="color: #008000;">//</span><span style="color: #008000;">www.coreseek.cn/products-install/coreseek_mmseg/</span> charset_dictpath = /usr/local/mmseg3/etc/ #BSD、Linux环境下设置,/<span style="color: #000000;">符号结尾 #charset_dictpath </span>= etc/ #Windows环境下设置,/符号结尾,最好给出绝对路径,例如:C:/usr/local/coreseek/etc/<span style="color: #000000;">... charset_type </span>= zh_cn.utf-<span style="color: #800080;">8</span><span style="color: #000000;"> } #全局index定义 indexer { mem_limit </span>=<span style="color: #000000;"> 128M } #searchd服务定义 searchd { listen </span>= <span style="color: #800080;">9312</span><span style="color: #000000;"> read_timeout </span>= <span style="color: #800080;">5</span><span style="color: #000000;"> max_children </span>= <span style="color: #800080;">30</span><span style="color: #000000;"> max_matches </span>= <span style="color: #800080;">1000</span><span style="color: #000000;"> seamless_rotate </span>= <span style="color: #800080;">0</span><span style="color: #000000;"> preopen_indexes </span>= <span style="color: #800080;">0</span><span style="color: #000000;"> unlink_old </span>= <span style="color: #800080;">1</span><span style="color: #000000;"> pid_file </span>=/usr/local/coreseek/var/log/searchd_mysql.pid #请修改为实际使用的绝对路径,例如:/usr/local/coreseek/var/<span style="color: #000000;">... log </span>=/usr/local/coreseek/var/log/searchd_mysql.log #请修改为实际使用的绝对路径,例如:/usr/local/coreseek/var/<span style="color: #000000;">... query_log </span>=/usr/local/coreseek/var/log/query_mysql.log #请修改为实际使用的绝对路径,例如:/usr/local/coreseek/var/<span style="color: #000000;">... }</span>
这样搜索的话就会从索引文件中查出id,uid,title,data字段。
OK,配置完成以后,重启Coreseek 服务就能从生产你想要的查询索引,以后你就可以摆脱mysql的桎梏,什么中文,英文都可以,还自带分词。怎么样,是不是打开了新世界的大门。
下面讲一下重建索引可能出错的地方,以及解决办法。有兴趣的同学看一下,否则可以跳到下一节:PHP测试Coreseek 。
重建索引时报错误:WARNING: failed to open pid_file '/usr/local/coreseek/var/log/searchd_mysql.pid'.
解决方法:
试着停止一下coreseek服务
/usr/local/coreseek/bin/searchd -c /usr/local/coreseek/etc/csft_mysql.conf --stop 停止服务
然后重启
/usr/local/coreseek/bin/searchd -c /usr/local/coreseek/etc/csft_mysql.conf 启动服务
再次建立索引
/usr/local/coreseek/bin/indexer -c /usr/local/coreseek/etc/csft_mysql.conf --all 建立索引
如果提示:FATAL: failed to lock /usr/local/coreseek/var/data/xxxx.spl: Resource temporarily unavailable, will not index. Try --rotate option.
则尝试重建索引
/usr/local/coreseek/bin/indexer -c /usr/local/coreseek/etc/csft_mysql.conf --all --rotate 重建索引
1,将sphinxapi.php放到测试目录下
cp /usr/local/src/coreseek-3.2.14/testpack/api/sphinxapi.php ./
vim test.php
<span style="color: #008080;">header</span>("Content-type: text/html; charset=utf-8"<span style="color: #000000;">); </span><span style="color: #008000;">//</span><span style="color: #008000;">require("./ ");</span> <span style="color: #800080;">$s</span> = <span style="color: #0000ff;">new</span><span style="color: #000000;"> SphinxClient; </span><span style="color: #800080;">$s</span>->setServer("127.0.0.1", 9312<span style="color: #000000;">); </span><span style="color: #008000;">//</span><span style="color: #008000;">SPH_MATCH_ALL, 匹配所有查询词(默认模式); SPH_MATCH_ANY, 匹配查询词中的任意一个; SPH_MATCH_EXTENDED2, 支持特殊运算符查询</span> <span style="color: #800080;">$s</span>-><span style="color: #000000;">setMatchMode(SPH_MATCH_ALL); </span><span style="color: #800080;">$s</span>->setMaxQueryTime(30); <span style="color: #008000;">//</span><span style="color: #008000;">设置最大搜索时间</span> <span style="color: #800080;">$s</span>->SetArrayResult(<span style="color: #0000ff;">false</span>); <span style="color: #008000;">//</span><span style="color: #008000;">是否将Matches的key用ID代替</span> <span style="color: #800080;">$s</span>->SetSelect ( "*" ); <span style="color: #008000;">//</span><span style="color: #008000;">设置返回信息的内容,等同于SQL</span> <span style="color: #800080;">$s</span>-><span style="color: #000000;">SetRankingMode(SPH_RANK_BM25); </span><span style="color: #800080;">$s</span>->SetLimits ( 0, 30, 1000, 0 ); <span style="color: #008000;">//</span><span style="color: #008000;">设置结果集偏移量 SetLimits </span> <span style="color: #800080;">$res</span> = <span style="color: #800080;">$s</span>->query('coreseek','mysql','--single-0-query--'); <span style="color: #008000;">#</span><span style="color: #008000;">[coreseek]关键字,[mysql]数据源source</span> <span style="color: #800080;">$err</span> = <span style="color: #800080;">$s</span>-><span style="color: #000000;">GetLastError(); </span><span style="color: #0000ff;">echo</span> '<pre class="brush:php;toolbar:false">'<span style="color: #000000;">; </span><span style="color: #008080;">var_dump</span>(<span style="color: #800080;">$res</span><span style="color: #000000;">); </span><span style="color: #008080;">var_dump</span>(<span style="color: #800080;">$res</span>['matches'<span style="color: #000000;">]); </span><span style="color: #008080;">var_export</span>(<span style="color: #800080;">$err</span><span style="color: #000000;">); </span><span style="color: #0000ff;">echo</span> '
php5 test.php
运行结果:matches为匹配后的结果集
1,Sphinx扩展安装安装
Coreseek官方教程中建议php使用直接include一个php文件进行操作,事实上php有独立的sphinx模块可以直接操作coreseek(coreseek就是sphinx!)已经进入了php的官方函数库,而且效率更高!但php模块依赖于libsphinxclient包。我是按照以下这篇文章的步骤安装了Sphinx扩展。
感谢:http://blog.csdn.net/e421083458/article/details/21529969
[第一步] 安装依赖libsphinxclient
# cd /var/<span style="color: #0000ff;">install</span>/coreseek-<span style="color: #800080;">4.1</span>-beta/csft-<span style="color: #800080;">4.1</span>/api/libsphinxclient/<span style="color: #000000;"> # .</span>/configure --prefix=/usr/local/<span style="color: #000000;">sphinxclient configure: creating .</span>/<span style="color: #000000;">config.status config.status: creating Makefile config.status: error: cannot </span><span style="color: #0000ff;">find</span> input <span style="color: #0000ff;">file</span>: Makefile.<span style="color: #0000ff;">in</span><span style="color: #000000;"> #报错configure失败 </span><span style="color: #008000;">//</span><span style="color: #008000;">处理configure报错</span> 编译过程中报了一个config.status: error: cannot <span style="color: #0000ff;">find</span> input <span style="color: #0000ff;">file</span>: src/<span style="color: #000000;">Makefile.in这个的错误,然后运行下列指令再次编译就能通过了: # aclocal # libtoolize </span>--<span style="color: #000000;">force # automake </span>--add-<span style="color: #000000;">missing # autoconf # autoheader # </span><span style="color: #0000ff;">make</span><span style="color: #000000;"> clean </span><span style="color: #008000;">//</span><span style="color: #008000;">从新configure编译</span> # ./<span style="color: #000000;">configure # </span><span style="color: #0000ff;">make</span> && <span style="color: #0000ff;">make</span> <span style="color: #0000ff;">install</span>
[第二步] 安装sphinx的PHP扩展
http:<span style="color: #008000;">//</span><span style="color: #008000;">pecl.php.net/package/sphinx</span> # <span style="color: #0000ff;">wget</span> http:<span style="color: #008000;">//</span><span style="color: #008000;">pecl.php.net/get/sphinx-1.3.0.tgz</span> # <span style="color: #0000ff;">tar</span> zxvf sphinx-<span style="color: #800080;">1.3</span>.<span style="color: #800080;">0</span><span style="color: #000000;">.tgz # cd sphinx</span>-<span style="color: #800080;">1.3</span>.<span style="color: #800080;">0</span><span style="color: #000000;"> # phpize # .</span>/configure --with-php-config=/usr/bin/php-config --with-sphinx=/usr/local/<span style="color: #000000;">sphinxclient # </span><span style="color: #0000ff;">make</span> && <span style="color: #0000ff;">make</span> <span style="color: #0000ff;">install</span><span style="color: #000000;"> # cd </span>/etc/php.d/<span style="color: #000000;"> # </span><span style="color: #0000ff;">cp</span><span style="color: #000000;"> gd.ini sphinx.ini # </span><span style="color: #0000ff;">vi</span><span style="color: #000000;"> sphinx.ini extension</span>=<span style="color: #000000;">sphinx.so # service php</span>-fpm restart
安装完PHP的Sphinx扩展后,就可以直接使用$coreseek = new SphinxClient();而无需引入源文件了。
简单说一下我在TP里使用coreseek查询,并高亮关键词的思路:
1,通过sphinx查出id,uid的集合
2,然后$sql = "select * from post where id in($ids)";$res = mysql_query($sql);获取到数据库的真实数据
3,用BuildExcerpts将title和data的关键字高亮,然后分页展示
关键代码:
<span style="color: #800080;"> $where</span> = <span style="color: #0000ff;">array</span><span style="color: #000000;">(); </span><span style="color: #800080;">$where</span>['uid']=<span style="color: #800080;">$uid</span><span style="color: #000000;">; </span><span style="color: #0000ff;">if</span>(!<span style="color: #0000ff;">empty</span>(<span style="color: #800080;">$search</span>)){ <span style="color: #008000;">//</span><span style="color: #008000;">有需要查找的内容,则去 coreseek 忠查出对应的id</span> <span style="color: #800080;">$coreseek</span> = <span style="color: #0000ff;">new</span><span style="color: #000000;"> \SphinxClient(); </span><span style="color: #800080;">$coreseek</span>->setServer("127.0.0.1", 9312<span style="color: #000000;">); </span><span style="color: #008000;">//</span><span style="color: #008000;">SPH_MATCH_ALL, 匹配所有查询词(默认模式); SPH_MATCH_ANY, 匹配查询词中的任意一个; SPH_MATCH_EXTENDED2, 支持特殊运算符查询</span> <span style="color: #800080;">$coreseek</span>-><span style="color: #000000;">setMatchMode(SPH_MATCH_ALL); </span><span style="color: #800080;">$coreseek</span>->setMaxQueryTime(30); <span style="color: #008000;">//</span><span style="color: #008000;">设置最大搜索时间</span> <span style="color: #800080;">$coreseek</span>->SetArrayResult(<span style="color: #0000ff;">false</span>); <span style="color: #008000;">//</span><span style="color: #008000;">是否将Matches的key用ID代替</span> <span style="color: #800080;">$coreseek</span>->SetSelect ( "*" ); <span style="color: #008000;">//</span><span style="color: #008000;">设置返回信息的内容,等同于SQL</span> <span style="color: #800080;">$coreseek</span>->SetLimits ( 0, 30, 1000, 0 ); <span style="color: #008000;">//</span><span style="color: #008000;">设置结果集偏移量 SetLimits</span> <span style="color: #800080;">$res</span> = <span style="color: #800080;">$coreseek</span>->query(<span style="color: #800080;">$search</span>,'mysql','--single-0-query--'<span style="color: #000000;">); </span><span style="color: #800080;">$key</span> = <span style="color: #008080;">array_keys</span>(<span style="color: #800080;">$res</span>['matches'<span style="color: #000000;">]); </span><span style="color: #800080;">$where</span>['id']=<span style="color: #0000ff;">array</span>('in',<span style="color: #800080;">$key</span><span style="color: #000000;">); </span><span style="color: #800080;">$coreseek</span>-><span style="color: #000000;">close(); }</span><span style="color: #0000ff;">else</span><span style="color: #000000;">{ } </span><span style="color: #008000;">//</span><span style="color: #008000;">获取总数据条数</span> <span style="color: #800080;">$total</span>=<span style="color: #800080;">$mod</span>->where(<span style="color: #800080;">$where</span>)-><span style="color: #008080;">count</span>();
高亮的关键代码:
<span style="color: #0000ff;"> if</span>(!<span style="color: #0000ff;">empty</span>(<span style="color: #800080;">$search</span><span style="color: #000000;">)){ </span><span style="color: #800080;">$page</span>->parameter['search']=<span style="color: #800080;">$search</span><span style="color: #000000;">; </span><span style="color: #008000;">//</span><span style="color: #008000;">代码高亮</span> <span style="color: #800080;">$opt</span> = <span style="color: #0000ff;">array</span>("before_match"=>"<font style="font-weight:bold;color:#f00">","after_match"=>"</font>"<span style="color: #000000;">); </span><span style="color: #800080;">$coreseek1</span> = <span style="color: #0000ff;">new</span><span style="color: #000000;"> \SphinxClient(); </span><span style="color: #800080;">$coreseek1</span>->setServer("127.0.0.1", 9312<span style="color: #000000;">); </span><span style="color: #800080;">$coreseek1</span>-><span style="color: #000000;">SetMatchMode(SPH_MATCH_ALL); </span><span style="color: #800080;">$i</span>=0<span style="color: #000000;">; </span><span style="color: #800080;">$tags_title</span>=<span style="color: #0000ff;">array</span><span style="color: #000000;">(); </span><span style="color: #0000ff;">foreach</span>(<span style="color: #800080;">$info</span> <span style="color: #0000ff;">as</span> <span style="color: #800080;">$key</span>=><span style="color: #800080;">$row</span><span style="color: #000000;">){ </span><span style="color: #800080;">$tags_title</span>[]=<span style="color: #800080;">$row</span>['title'<span style="color: #000000;">]; } </span><span style="color: #800080;">$replace_title</span>=<span style="color: #800080;">$coreseek1</span>->BuildExcerpts(<span style="color: #800080;">$tags_title</span>,'mysql',<span style="color: #800080;">$search</span>,<span style="color: #800080;">$opt</span><span style="color: #000000;">); </span><span style="color: #0000ff;">foreach</span>(<span style="color: #800080;">$info</span> <span style="color: #0000ff;">as</span> <span style="color: #800080;">$key</span>=>&<span style="color: #800080;">$row</span><span style="color: #000000;">){ </span><span style="color: #800080;">$info</span>[<span style="color: #800080;">$key</span>]['title']=<span style="color: #800080;">$replace_title</span>[<span style="color: #800080;">$key</span><span style="color: #000000;">]; } </span><span style="color: #800080;">$coreseek1</span>-><span style="color: #000000;">close(); }</span>
OK,至此,coreseek已经能够在TP里完美运行了。这篇分享的文章也可以画上句号了。以上是我一步步安装时的细节,记录下怕以后自己忘记,也希望能给感兴趣的同学一些帮助。文章信息量偏大,如果有哪些疏漏,万望指正!