How to integrate Kafka with Nginx

Background

nginx-kafka-module is a plug-in for nginx. It can integrate kafka into nginx to facilitate the collection of front-end page buried point data in web projects. If the front-end page has set buried points, that is Some of the user's access and request data can be sent directly to the message middleware kafka through http requests. The backend can consume the messages in kafka through the program to perform real-time calculations. For example, sparkstream is used to consume data in Kafka in real time to analyze user pv, uv, some user behaviors and the funnel model conversion rate of the page, so as to better optimize the system or conduct real-time dynamic analysis of visiting users.

Specific integration steps

1. Install git

yum install -y git

2. Switch to the /usr/local/src directory, and then clone the kafka c client source code Go to local

cd /usr/local/src
git clone https://github.com/edenhill/librdkafka

3. Enter librdkafka, and then compile it

cd librdkafka
yum install -y gcc gcc-c++ pcre-devel zlib-devel
./configure
make && make install

4. Install nginx integration kafka plug-in, enter /usr/local/src, clone nginx integrates the source code of kafka

cd /usr/local/src
git clone https://github.com/brg-liuwei/ngx_kafka_module

5. Enter the nginx source package directory (compile nginx, and then compile the plug-in at the same time)

cd /usr/local/src/nginx-1.12.2
./configure --add-module=/usr/local/src/ngx_kafka_module/
make && make install

6. Modify nginx Configuration file: Set a location and kafaka topic

#添加配置(2处)
kafka;
kafka_broker_list f1:9092 f2:9092 f3:9092;

location = /kafka/access {
kafka_topic access888;
}

as shown below:

7. Start zk and kafka clusters (create topic)

zkserver.sh start
kafka-server-start.sh -daemon config/server.properties

8. Start nginx and report an error. The file kafka.so.1 cannot be found

error while loading shared libraries: librdkafka.so.1: cannot open shared object file: no such file or directory

##9. Loading so library

#开机加载/usr/local/lib下面的库
echo "/usr/local/lib" >> /etc/ld.so.conf
#手动加载
ldconfig

10. Test, write data to nginx, and then observe whether the kafka consumer can consume the data

curl http://localhost/kafka/access -d "message send to kafka topic"
curl http://localhost/kafka/access -d "小伟666"测试

You can also simulate the page buried request interface to send information:

The background kafka consumption information is as shown in the figure:

The above is the detailed content of How to integrate Kafka with Nginx. For more information, please follow other related articles on the PHP Chinese website!