How to monitor memory changes in the local environment and production environment in the Node service?

When using Node as a server language in a production environment, excessive concurrency or code problems cause OOM (out of memory) or the CPU is fully loaded. These are common problems in servers. At this time, it is easy to find problems by monitoring CPU and memory, combined with logs and Release.

[Video tutorial recommendation: node js tutorial]

This chapter will introduce how to monitor memory changes in the local environment and production environment

A Node application instance

So, how to dynamically monitor the memory changes of a Node process?

The following is an example of a Node Server, and it is an example with a memory leak problem, and it is a streamlined version of the problem that Shanyue has been positioning in the production environment for a long time.

In that memory leak problem, the memory in a single container skyrocketed from the original 400M to 700M. Under the 800M container resource limit, OOM would occasionally occur, causing a restart. The problem was not located for a while (the problem was discovered too late, the time series data half a month ago had been swallowed up, so Release was not located), so the resource limit was raised to 1000M. Later it was found that it was caused by ctx.request mounting a large field in the database

const Koa = require('koa')
const app = new Koa()

function getData () {
  return Array.from(Array(1000)).map(x => 10086)
}

app.use(async (ctx, next) => {
  ctx.data = getData()
  await next()
})

app.use(ctx => {
  ctx.body = 'hello, world'
})

app.listen(3200, () => console.log('Port: 3200'))

Process memory monitoring

Some problems need to be eliminated in time in the local and test environments , to avoid greater impact on the production environment. Then it is crucial to understand how to monitor memory locally.

pidstat is a package of the sysstat series of Linux performance debugging tools. It is actually used to debug Linux performance problems, including memory, network, IO, CPU, etc.

This is not only tried with node, but also applies to all processes, including python, java and go

# -r: 指输出内存指标
# -p: 指定 pid
# 1: 每一秒输出一次
# 100: 输出100次
$ pidstat -r -p pid 1 100

Before using pidstat, you need to find the pid of the process pid

How to find the pid of the Node process

In node you can find the process’ pid<div class="code" style="position:relative; padding:0px; margin:0px;"><pre class="brush:js;toolbar:false">&gt; process.pid 16425</pre><div class="contentsignin">Copy after login</div></div> through

process.pid

although by writing The code can find pid, but is intrusive and not very practical. So how to find pid through non-invasive means? There are two ways

1. Combined with redundant parameterspsLocate the process
2. Combined with the port numberlsofLocate the process

$ node index.js shanyue

# 第一种方法：通过多余的参数快速定位 pid
$ ps -ef | grep shanyue
root     31796 23839  1 16:38 pts/5    00:00:00 node index.js shanyue

# 第二种方法：通过端口号定位 pid
lsof -i:3200
COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
node    31796 root   20u  IPv6 235987334      0t0  TCP *:tick-port (LISTEN)

Use pidstat to monitor memory

From the above code, we can know that the pid of the node service is 31796 , in order to observe the dynamic changes of memory, apply a stress test

$ ab -c 10000 -n 1000000 http://localhost:3200/

# -r: 指输出内存指标
# -p: 指定 pid
# 1: 每一秒输出一次
# 100: 输出100次
$ pidstat -r -p 31796 1 100
Linux 3.10.0-957.21.3.el7.x86_64 (shuifeng)     2020年07月02日  _x86_64_        (2 CPU)

             UID       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
19时20分39秒     0     11401      0.00      0.00  566768  19800   0.12  node
19时20分40秒     0     11401      0.00      0.00  566768  19800   0.12  node
19时20分41秒     0     11401   9667.00      0.00  579024  37792   0.23  node
19时20分42秒     0     11401  11311.00      0.00  600716  59988   0.37  node
19时20分43秒     0     11401   5417.82      0.00  611420  70900   0.44  node
19时20分44秒     0     11401   3901.00      0.00  627292  85928   0.53  node
19时20分45秒     0     11401   1560.00      0.00  621660  81208   0.50  node
19时20分46秒     0     11401   2390.00      0.00  623964  83696   0.51  node
19时20分47秒     0     11401   1764.00      0.00  625500  85204   0.52  node

The meaning of the output indicators is as follows

• ##RSS: Resident Set Size, resident memory set, can be understood as memory, this is the memory indicator we need to monitor
• VSZ: virtual size, virtual memory

As can be seen from the output,

After the stress test was applied, the memory increased from 19M to 85M.

Use top to monitor memory

pidstat is a Linux performance tool under sysstat, but in mac, how to locate the memory Variety?

You can use

top/htop

$ htop -p 31796

Production environment memory monitoring

Since most of the current production environment Deployed in

k8s, So the memory monitoring of a certain application in the production environment is essentially the memory monitoring of a certain workload/deployment by k8s, about memory monitoring# The data flow direction of ##metric is roughly as follows:

k8s

-> metric server -> prometheus -> grafanaThe architecture diagram is as follows:

##The above pictures are taken from the following articles

Kubernetes Monitoring with Prometheus

• Kubernetes monitoring architecture
Finally be able to A real-time graph of memory monitoring of an application collected in #grafana

:

Since there is too much design content in this part, I will introduce it in the following chapters

This is not only applicable to node services, but also applicable to all k8sworkload

Summary

This chapter introduces the monitoring of Node service memory in the local environment and production environment

1. Local usehtop/top or pidstat Monitor process memory

2. Use in production environmentk8s/metric-server/prometheus/grafana Monitor the memory of the entire node application

When a memory leak is detected in a certain service, how to solve the problem? Therefore, the following article will talk about

1. How the production environment monitors the memory of the entire application

2. When OOM occurs in the production environment, how to quickly locate

3. Several examples of OOM positioning in real production environments

For more programming-related knowledge, please visit:Introduction to Programming! !

The above is the detailed content of How to monitor memory changes in the local environment and production environment in the Node service?. For more information, please follow other related articles on the PHP Chinese website!