How to use Nginx for logging and analysis of HTTP requests

How to use Nginx to log and analyze HTTP requests

Introduction:
In daily web development, we often need to log and analyze HTTP requests in order to monitor website access. , troubleshoot issues and optimize performance. As a powerful HTTP server and reverse proxy server, Nginx provides rich logging functions, and its performance is better than other servers. This article will introduce how to configure and use Nginx for logging and analysis of HTTP requests.

1. Configure Nginx logging format

In order to record more detailed information, we can customize the Nginx logging format. In the Nginx configuration file, find the "http" context and add the following content:

http {
    log_format my_log_format '$remote_addr - $remote_user [$time_local] "$request" '
                             '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
                             '"$gzip_ratio"';

    access_log /var/log/nginx/access.log my_log_format;
}

Here we define a logging format named "my_log_format", including remote address, remote user, Access time, request content, status code, number of bytes sent, source URL, User-Agent, Gzip ratio and other information. This format is then applied to Nginx’s access log files.

2. Configure Nginx for log segmentation

As the number of website visits increases, log files tend to become very large. In order to facilitate log management and analysis, we can configure Nginx to split logs. Add the following content to the Nginx configuration file:

http {
    access_log /var/log/nginx/access.log my_log_format;
    error_log /var/log/nginx/error.log;

    logrotate daily;
    rotate 7;
    size 10M;
    missingok;
    notifempty;
    compress;
    delaycompress;
}

Here we configure Nginx to split logs every day and retain the log files of the last 7 days. Each file has a maximum size of 10MB. Once the upper limit is reached, it will be compressed and a new log file will be generated. Additionally, logging will continue if the file does not exist, and empty files will not trigger log splitting.

3. Use ELK Stack for log analysis

ELK Stack is an open source log analysis solution consisting of three tools: Elasticsearch, Logstash and Kibana. We can use ELK Stack to analyze and visualize Nginx logs. The following are the configuration steps:

1. Install Elasticsearch

First, you need to install and configure Elasticsearch as a database for log storage and indexing. For specific installation steps, please refer to the official Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

2. Install Logstash

Secondly, Install and configure Logstash as a data collection and processing tool. For specific installation steps, please refer to the official Logstash documentation: https://www.elastic.co/guide/en/logstash/current/index.html

3. Write the Logstash configuration file

Create one New configuration file (for example nginx.conf) and add the following content:

input {
    file {
        path => "/var/log/nginx/access.log"
        start_position => "beginning"
    }
}

filter {
    grok {
        match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "%{URI:referrer}" "%{DATA:useragent}" "%{DATA:gzipratio}"" }
    }
}

output {
    elasticsearch {
        hosts => "localhost"
        index => "nginx-logs-%{+YYYY.MM.dd}"
    }
}

Here we specify the path to the log file, the log format and the address of Elasticsearch, and store the logs to an index suffixed by date middle.

4. Start Logstash

Start Logstash and load the configuration file:

$ bin/logstash -f nginx.conf

5. Install Kibana

Finally, install and configure Kibana for log visualization tool. For specific installation steps, please refer to Kibana’s official documentation: https://www.elastic.co/guide/en/kibana/current/index.html

After the configuration is completed, open Kibana’s web interface and search and filter Function, you can view and analyze Nginx access logs in real time.

Conclusion:
By configuring Nginx to log and analyze HTTP requests, we can monitor website access, troubleshoot problems and optimize performance. At the same time, combined with ELK Stack, centralized storage, analysis and visualization of logs can be achieved, improving operation and maintenance efficiency and the accuracy of website performance analysis. I hope this article can be helpful to everyone.

The above is the detailed content of How to use Nginx for logging and analysis of HTTP requests. For more information, please follow other related articles on the PHP Chinese website!