Kibana Logstash Elasticsearch log query system, kibanalogstash
Kibana Logstash Elasticsearch log query system, kibanalogstash
The purpose of building this platform is to facilitate log query for operation, maintenance and research and development. Kibana is a free web shell; Logstash integrates various log collection plug-ins and is an excellent regular cutting log tool; Elasticsearch is an open source search engine framework (supports cluster architecture).
1 Installation requirements
1.1 Theoretical Topology
1.2 Installation environment
1.2.1 Hardware environment
192.168.50.62 (HP DL 385 G7, RAM: 12G, CPU: AMD 6128, DISK: SAS 146*4)
192.168.50.98 (HP DL 385 G7, RAM: 12G, CPU: AMD 6128, DISK: SAS 146*6)
192.168.10.42 (Xen virtual machine, RAM: 8G, CPU: ×4, DISK: 100G)
1.2.2 Operating System
CentOS 5.6 X64
1.2.3 Web-server basic environment
Nginx php (the installation process is skipped)
1.2.4 Software List
JDK 1.6.0_25
logstash-1.1.0-monolithic.jar
elasticsearch-0.18.7.zip
redis-2.4.12.tar.gz
kibana
1.3 How to obtain
1.3.1 Jdk acquisition path
http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u25-download-346242.html
1.3.2 Logstash acquisition path
http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
1.3.3 Elasticsearch acquisition path
https://github.com/downloads/elasticsearch/elasticsearch/ elasticsearch-0.18.7.zip
1.3.4 Kibana Get Path
http://github.com/rashidkpc/Kibana/tarball/master
2 Installation steps
2.1 JDK download and installation
Basic installation
wget http://download.oracle.com/otn-pub/java/jdk/6u25-b06/jdk-6u25-linux-x64.bin
sh jdk-6u25-linux-x64.bin
mkdir -p /usr/java
mv ./jdk1.6.0_25 /usr/java
ln –s /usr/java/jdk1.6.0_25 /usr/java/default
Edit the /etc/profile file and add the following lines
export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH
Refresh environment variables
source /etc/profile
2.2 Redis download and installation
wget http://redis.googlecode.com/files/redis-2.4.14.tar.gz
make –j24
make install
mkdir -p /data/redis
cd /data/redis/
mkdir {db,log,etc}
2.3 Elasticsearch download and installation
cd /data/
mkdir –p elasticsearch && cd elasticsearch
wget --no-check-certificate https://github.com/downloads/elasticsearch/elasticsearch/ elasticsearch-0.18.7.zip
unzip elasticsearch-0.18.7.zip
2.4 Logstash download and installation
mkdir –p /data/logstash/ && cd /data/logstash
wget http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
2.5 Kibana download and installation
wget http://github.com/rashidkpc/Kibana/tarball/master --no-check-certificate
tar zxvf master
3 Related configuration and startup
3.1 Redis configuration and startup
3.1.1 Configuration File
vim /data/redis/etc/redis.conf
#--------------------------------------------- ------
#this is the config file for redis
pidfile /var/run/redis.pid
port 6379
timeout 0
loglevel verbose
logfile /data/redis/log/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename dump.rdb
dir /data/redis/db/
slave-serve-stale-data yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
slowlog-log-slower-than 10000
slowlog-max-len 128
vm-enabled no
vm-swap-file /tmp/redis.swap
vm-max-memory 0
vm-page-size 32
vm-pages 134217728
vm-max-threads 4
hash-max-zipmap-entries 512
hash-max-zipmap-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
3.1.2 Redis startup
[logstash@Logstash_2 redis]# redis-server /data/redis/etc/redis.conf &
3.2 Elasticsearch configuration and startup
3.2.1 Elasticsearch startup
[logstash@Logstash_2 redis]# /data/elasticsearch/elasticsearch-0.18.7/bin/elasticsearch –p ../esearch.pid &
3.2.2 Elasticsearch cluster configuration
curl 127.0.0.1:9200/_cluster/nodes/192.168.50.62
3.3 Logstash configuration and startup
3.3.1 Logstash configuration file
input {
redis {
host => "192.168.50.98"
data_type =>"list"
key => "logstash:redis"
type => "redis-input"
}
}
filter {
grok {
type => "linux-syslog"
pattern => "%{SYSLOGLINE}"
}
grok {
type => "nginx-access"
pattern => "%{NGINXACCESSLOG}"
}
}
output {
elasticsearch {
host =>"192.168.50.62"
}
}
3.3.2 Logstash starts as Index
java -jar logstash.jar agent -f my.conf &
3.3.3 Logstash starts as agent
Configuration file
input {
file{
type => "linux-syslog"
path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
}
file {
type => "nginx-access"
path => "/usr/local/nginx/logs/access.log"
}
file {
type => "nginx-error"
path => "/usr/local/nginx/logs/error.log"
}
}
output {
redis {
host => "192.168.50.98"
data_type =>"list"
key => "logstash:redis"
}
}
Agent starts
java -jar logstash-1.1.0-monolithic.jar agent -f shipper.conf &
3.3.4 kibana configuration
First add site configuration in nginx
server {
listen 80;
server_name logstash.test.com;
index index.php;
root /usr/local/nginx/html;
#charset koi8-r;
#access_log logs/host.access.log main;
location ~ .*.(php|php5)$
{
#fastcgi_pass unix:/tmp/php-cgi.sock;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
include fastcgi.conf;
}
}
4 Performance Tuning
4.1 Elasticsearch Tuning
4.1.1 JVM Tuning
Edit the Elasticsearch.in.sh file
ES_CLASSPATH=$ES_CLASSPATH:$ES_HOME/lib/*:$ES_HOME/lib/sigar/*
if [ "x$ES_MIN_MEM" = "x" ]; then
ES_MIN_MEM=4g
fi
if [ "x$ES_MAX_MEM" = "x" ]; then
ES_MAX_MEM=4g
fi
4.1.2 Elasticsearch Index Compression
vim index_elastic.sh
#!/bin/bash
#comperssion the data for elasticsearch now
date=` date %Y.%m.%d `
# compression the new index;
/usr/bin/curl -XPUT http://localhost:9200/logstash-$date/nginx-access/_mapping -d '{"nginx-access" : {"_source" : { "compress " : true }}}'
echo ""
/usr/bin/curl -XPUT http://localhost:9200/logstash-$date/nginx-error/_mapping -d '{"nginx-error" : {"_source" : { "compress " : true }}}'
echo ""
/usr/bin/curl -XPUT http://localhost:9200/logstash-$date/linux-syslog/_mapping -d '{"linux-syslog" : {"_source" : { "compress " : true }}}'
echo ""
Save the script and execute it
sh index_elastic.sh
5 Use
5.1 Logstash query page
Use Firefox or Google Chrome to visit http://logstash.test.com
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1386
52
What is event ID 6013 in win10?
Jan 09, 2024 am 10:09 AM
The logs of win10 can help users understand the system usage in detail. Many users must have encountered log 6013 when looking for their own management logs. So what does this code mean? Let’s introduce it below. What is win10 log 6013: 1. This is a normal log. The information in this log does not mean that your computer has been restarted, but it indicates how long the system has been running since the last startup. This log will appear once every day at 12 o'clock sharp. How to check how long the system has been running? You can enter systeminfo in cmd. There is one line in it.
Logger buffer size what is log used for
Mar 13, 2023 pm 04:27 PM
The function is to provide engineers with feedback on usage information and records to facilitate problem analysis (used during development); because users themselves do not often generate upload logs, they are useless to users. The logging buffer is a small, temporary area used for short-term storage of change vectors for redo logs to be written to disk. A log buffer write to disk is a batch of change vectors from multiple transactions. Even so, the change vector in the log buffer is written to disk in near real-time, and when the session issues a COMMIT statement, the log buffer write operation is performed in real time.
Troubleshooting Event 7034 Error Log Issues in Win10
Jan 11, 2024 pm 02:06 PM
The logs of win10 can help users understand the system usage in detail. Many users must have seen a lot of error logs when looking for their own management logs. So how to solve them? Let’s take a look below. . How to solve win10 log event 7034: 1. Click "Start" to open "Control Panel" 2. Find "Administrative Tools" 3. Click "Services" 4. Find HDZBCommServiceForV2.0, right-click "Stop Service" and change it to "Manual Start "
How to use logging in ThinkPHP6
Jun 20, 2023 am 08:37 AM
With the rapid development of the Internet and Web applications, log management is becoming more and more important. When developing web applications, how to find and locate problems is a very critical issue. A logging system is a very effective tool that can help us achieve these tasks. ThinkPHP6 provides a powerful logging system that can help application developers better manage and track events that occur in applications. This article will introduce how to use the logging system in ThinkPHP6 and how to utilize the logging system
How to view your medication log history in the Health app on iPhone
Nov 29, 2023 pm 08:46 PM
iPhone lets you add medications to the Health app to track and manage the medications, vitamins and supplements you take every day. You can then log medications you've taken or skipped when you receive a notification on your device. After you log your medications, you can see how often you took or skipped them to help you track your health. In this post, we will guide you to view the log history of selected medications in the Health app on iPhone. A short guide on how to view your medication log history in the Health App: Go to the Health App>Browse>Medications>Medications>Select a Medication>Options&a
Detailed explanation of log viewing command in Linux system!
Mar 06, 2024 pm 03:55 PM
In Linux systems, you can use the following command to view the contents of the log file: tail command: The tail command is used to display the content at the end of the log file. It is a common command to view the latest log information. tail [option] [file name] Commonly used options include: -n: Specify the number of lines to be displayed, the default is 10 lines. -f: Monitor the file content in real time and automatically display the new content when the file is updated. Example: tail-n20logfile.txt#Display the last 20 lines of the logfile.txt file tail-flogfile.txt#Monitor the updated content of the logfile.txt file in real time head command: The head command is used to display the beginning of the log file
Understand the meaning of event ID455 in win10 logs
Jan 12, 2024 pm 09:45 PM
The logs of win10 have a lot of rich content. Many users must have seen the event ID455 display error when looking for their own management logs. So what does it mean? Let’s take a look below. What is event ID455 in the win10 log: 1. ID455 is the error <error> that occurred in <file> when the information store opened the log file.
Different types of Linux log files and setting steps
Feb 26, 2024 pm 10:54 PM
Types of Linux log files and configuration methods In Linux systems, log files are very important. They record the running status of the system, user operations, and the occurrence of various events. By checking log files, system administrators can discover problems in time and handle them accordingly. This article will introduce the common types of log files in Linux systems and how to configure logging. 1. Types of log files System log: System log is a log file that records the operating status of the system, including system startup, shutdown, service startup and stop, etc.
