A brief analysis of the memory leak problem in Node.js_node.js

This article is the first in a series of A Node.JS Holiday Season articles brought to you by Mozilla’s Identity team, which released the first beta version of Persona last month. When developing Persona we built a series of tools, from debugging, to localization, to dependency management and more. In this series of articles we will share our experiences and these tools with the community, which will be useful to anyone who wants to build a high-availability service with Node.js. We hope you enjoy these articles and look forward to seeing your thoughts and contributions.

We will start with a topical article about a substantive problem in Node.js: memory leaks. We’ll introduce node-memwatch — a library that helps find and isolate memory leaks in Node.

Why are you asking for trouble?

The most frequently asked question about tracking memory leaks is, "Why bother yourself?". Aren’t there more pressing issues that need to be addressed first? Why not choose to restart the service from time to time, or allocate more RAM to it? To answer these questions, we propose the following three suggestions:

1. Maybe you don't care about the growing memory footprint, but V8 does (V8 is the engine of the Node runtime). As memory leaks grow, V8 becomes more aggressive with the garbage collector, which can make your application run slower. Therefore, on Node, memory leaks will harm program performance.

2. Memory leaks can trigger other types of failures. Code that leaks memory may continually reference limited resources. You might run out of file descriptors; you might also suddenly be unable to make new database connections. This type of problem may surface long before your app runs out of memory, but it can still get you into trouble.

3． Eventually, your app will crash sooner or later, and it will definitely happen as your app gains popularity. Everyone will laugh at you and ridicule you on Hacker News, which will make you a tragedy.

Where is the ant nest that broke the embankment of thousands of miles?

When building complex applications, memory leaks may occur in many places. Closures are probably the most well-known and infamous. Because closures retain references to things within their scope, this is where memory leaks usually come from.

Closure leaks are often discovered only when someone looks for them. But in the asynchronous world of Node, we constantly generate closures through callback functions anytime and anywhere. If these callback functions are not used immediately after creation, the allocated memory will continue to grow, and code that does not appear to have memory leaks will leak. And this kind of problem is harder to find.

Your application may also cause memory leaks due to problems in upstream code. Maybe you can locate the code that leaked memory, but you may just stare at your perfect code and wonder how it leaked!

It's these hard-to-locate memory leaks that make us want a tool like node-memwatch. Legend has it that a few months ago, our own Lloyd Hilaiel locked himself in a small room for two days, trying to track down a memory leak that became apparent under stress testing. (By the way, stay tuned for Lloyd’s upcoming article on load testing)

After two days of hard work, he finally discovered the culprit in the Node kernel: the event listener in http.ClientRequest was not released. (The patch that eventually fixed the problem was only two but crucial letters). It was this painful experience that prompted Lloyd to write a tool that could help find memory leaks.

Memory leak locating tool

There are many useful and constantly improving tools for locating memory leaks in Node.js applications. Here are some of them:

• Jimb Esser’s node-mtrace, which uses GCC’s mtrace tool to analyze heap usage.
• Dave Pacheco’s node-heap-dump takes a snapshot of the V8 heap and serializes everything into a huge JSON file. It also contains JavaScript tools for analyzing study snapshot results.
• Danny Coates's v8-profiler and node-inspector provide a V8 profiler bundled in Node and a debug interface based on WebKit Web Inspector.
• Felix Gnass’s does not disable the keeper chart branch .
• Felix Geisendorfer's Node Memory Leak Tutorial is a short and cool tutorial on how to use v8-profiler and node-debugger. It is also the most advanced Node.js memory leak debugging technical guide.
• Joyent’s SmartOS platform, which provides a large number of tools for debugging Node.js memory leaks.

We all like the tools above, but none of them apply to our scenario. Web Inspector is great for developing applications, but difficult to use in hot deployment scenarios, especially when multiple servers and sub-processes are involved. Similarly, memory leaks that occur during long-term high-load operations are also difficult to reproduce. Tools like dtrace and libumem, while impressive, are not available on all operating systems.

Enternode-memwatch

We need a cross-platform debugging library that does not require the device to tell us when our program may have a memory leak, and will help us find where the leak exists. So we implemented node-memwatch.

It provides us with three things:

A ‘leak’ event emitter
 

   memwatch.on('leak', function(info) {
  // look at info to find out about what might be leaking
  });

A ‘status event emitter

       

  var memwatch = require('memwatch');
  memwatch.on('stats', function(stats) {
  // do something with post-gc memory usage stats
  });

A heap memory area classification

  var hd = new memwatch.HeapDiff();
  // your code here ...
  var diff = hd.end();

And there is also a function that can trigger the garbage collector that is very useful during testing. Okay, four in total.
 

 var stats = memwatch.gc();

memwatch.on('stats', ...): Post-GC heap statistics

node-memwatch can emit a memory usage sample following a complete garbage collection and memory compaction before any JS object is allocated. (It uses V8’s post-gc hook, V8::AddGCEpilogueCallback, to collect heap usage information every time a garbage collection is triggered)

Statistics include:

• usage_trend (usage trend)
• current_base (current base)
• estimated_base (expected base)
• num_full_gc (number of complete garbage collections)
• num_inc_gc (increased number of garbage collections)
• heap_compactions (number of memory compressions)
• min (minimum)
• max (maximum)

Here is an example of what the data for an application with a memory leak looks like. The chart below tracks memory usage over time. The crazy green line shows what process.memoryUsage() reports. The red line shows the current_base reported by node_memwatch. The box on the lower left shows additional information.

Note that Incr GCs are very high. That means V8 is desperately trying to clear memory.

memwatch.on('leak', ...): Heap allocation trend

We have defined a simple detection algorithm to alert you that your application may have a memory leak. That is, if after five consecutive GCs, the memory is still allocated but not released, node-memwatch will issue a leak event. The specific information format of the event is clear and easy to read, like this:

{ start: Fri, 29 Jun 2012 14:12:13 GMT,
 end: Fri, 29 Jun 2012 14:12:33 GMT,
 growth: 67984,
 reason: 'heap growth over 5 consecutive GCs (20s) - 11.67 mb/hr' }

memwatch.HeapDiff(): 查找泄漏元凶

最后，node-memwatch能比较堆上对象的名称和分配数量的快照，其对比前后的差异可以帮助找出导致内存泄漏的元凶。

var hd = new memwatch.HeapDiff();
 
// Your code here ...
 
var diff = hd.end();

对比产生的内容就像这样：

{
 "before": {
  "nodes": 11625,
  "size_bytes": 1869904,
  "size": "1.78 mb"
 },
 "after": {
  "nodes": 21435,
  "size_bytes": 2119136,
  "size": "2.02 mb"
 },
 "change": {
  "size_bytes": 249232,
  "size": "243.39 kb",
  "freed_nodes": 197,
  "allocated_nodes": 10007,
  "details": [
   {
    "what": "Array",
    "size_bytes": 66688,
    "size": "65.13 kb",
    "+": 4,
    "-": 78
   },
   {
    "what": "Code",
    "size_bytes": -55296,
    "size": "-54 kb",
    "+": 1,
    "-": 57
   },
   {
    "what": "LeakingClass",
    "size_bytes": 239952,
    "size": "234.33 kb",
    "+": 9998,
    "-": 0
   },
   {
    "what": "String",
    "size_bytes": -2120,
    "size": "-2.07 kb",
    "+": 3,
    "-": 62
   }
  ]
 }
}

HeapDiff方法在进行数据采样前会先进行一次完整的垃圾回收，以使得到的数据不会充满太多无用的信息。memwatch的事件处理会忽略掉由HeapDiff触发的垃圾回收事件，所以在stats事件的监听回调函数中你可以安全地调用HeapDiff方法。