Detailed explanation of the Buffer module in NodeJS_node.js

1, opening analysis

The so-called buffer Buffer means "temporary storage area", which is a section of memory that temporarily stores input and output data.

The JS language itself only has a string data type and no binary data type, so NodeJS provides a global constructor Buffer equivalent to String to provide operations on binary data. In addition to reading the file to obtain an instance of the Buffer, it can also be constructed directly, for example:

Copy code The code is as follows:

var buffer = new Buffer([ 0x68, 0x65, 0x6c, 0x6c, 0x6f ]) ;

Buffer is similar to a string. In addition to using the .length attribute to get the byte length, you can also use the [index] method to read the bytes at the specified position, for example:

Copy code The code is as follows:

buffer[0] ; // 0x68;

Buffer and string can be converted to each other. For example, binary data can be converted into string using specified encoding:

Copy code The code is as follows:

var str = buffer.toString("utf-8"); // hello

Convert the string to binary data in the specified encoding:

Copy code The code is as follows:

var buffer= new Buffer("hello", "utf-8") ; //

A little difference:

There is an important difference between Buffer and string. The string is read-only, and any modification to the string results in a new string, while the original string remains unchanged.

As for Buffer, it is more like a C language array that can perform pointer operations. For example, you can use the [index] method to directly modify the bytes at a certain position.

-------------------------------------------------- -------------------------------------------------- -------------------------------------------------- -------------------------------------------------- --------------------------------

The

slice method does not return a new Buffer, but rather returns a pointer to a position in the middle of the original Buffer, as shown below.

[ 0x68, 0x65, 0x6c, 0x6c, 0x6f ]
                                                               |
bin bin.slice(2)
Therefore, modifications to the Buffer returned by the slice method will affect the original Buffer, for example:

Copy code The code is as follows:

var buffer= new Buffer([ 0x68, 0x65, 0x6c, 0x6c, 0x6f ]) ;
var sub = bin.slice(2) ;
sub[0] = 0x65;
console.log(buffer); //

If you want to copy a Buffer, you must first create a new Buffer and copy the data in the original Buffer through the .copy method.

This is similar to applying for a new memory and copying the data in the existing memory. Below is an example.

Copy code The code is as follows:

var buffer= new Buffer([ 0x68, 0x65, 0x6c, 0x6c, 0x6f ]) ;
var dup = new Buffer(bin.length) ;
buffer.copy(dup) ;
dup[0] = 0x48 ;
console.log(buffer); //
console.log(dup) ; //

In short, Buffer extends the data processing capabilities of JS from strings to arbitrary binary data.

The above is a brief introduction to what Buffer is. Let’s talk about how to use it and the specific usage scenarios.

Two, let’s talk about Buffer

JavaScript is very friendly to string processing. Whether it is a wide byte or a single-byte string, it is considered a string. Node needs to process network protocols, operate databases, process pictures, file uploads, etc. It also needs to process a large amount of binary data. The built-in strings are far from meeting these requirements, so Buffer came into being.

Buffer structure

Buffer is a typical module that combines Javascript and C. The performance-related parts are implemented in C, and the non-performance-related parts are implemented in JavaScript.

Node has already loaded the Buffer into the memory when the process starts and puts it into the global object, so there is no need to require

Buffer object: similar to an array, its elements are two-digit hexadecimal digits.

Buffer memory allocation

The memory allocation of the Buffer object is not in the heap memory of V8. The memory application is implemented at the C level of Node.

In order to efficiently use the requested memory, Node adopts the slab allocation mechanism. Slab is a dynamic memory management mechanism that applies various *nix operating systems. Slab has three states:

(1) full: fully allocated status

(2) partial: partial allocation status

(3) empty: no assigned status

Buffer conversion

Buffer objects can be converted to and from strings. The supported encoding types are as follows:

ASCII, UTF-8, UTF-16LE/UCS-2, Base64, Binary, Hex

String to Buffer

new Buffer(str, [encoding]), default UTF-8
buf.write(string, [offset], [length], [encoding])

Buffer to string

buf.toString([encoding], [start], [end])

Encoding type not supported by Buffer

Determine whether it is supported by Buffer.isEncoding(encoding)

iconv-lite: pure JavaScript implementation, lighter weight, better performance without C to javascript conversion

iconv: Call C’s libiconv library to complete

Buffer splicing

Pay attention to "res.on('data', function(chunk) {})". The parameter chunk is a Buffer object. Direct splicing will automatically convert it into a string. For wide-byte characters, garbled characters may be generated. ,

Solution:

(1) Through the setEncoding() method in the readable stream, this method allows the data event to be transmitted no longer as a Buffer object, but as an encoded string, which uses the StringEncoder module internally.

(2) Temporarily store the Buffer object into an array, and finally assemble it into a large Buffer and then encode it into a string for output.

Buffer is widely used in file I/O and network I/O. Its performance is very important, and its performance is much higher than that of ordinary strings.

In addition to the performance loss in converting strings, the use of Buffer has a highWaterMark setting that is crucial to performance when reading files.

a. The highWaterMark setting has a certain impact on the allocation and use of Buffer memory.

b. Setting highWaterMark too small may result in too many system calls.

When to use buffer and when not to use it ------ Pure javascript supports unicode code but not binary. When dealing with TCP stream or file stream, it is necessary to process the stream. When we save non-utf-8 strings, binary and other formats, we must use "Buffer".

3. Introduction of examples

Copy code The code is as follows:

var buf = new Buffer("this is text concat test !") ,str = "this is text concat test !" ;
console.time("buffer concat test !");
var list = [] ;
var len = 100000 * buf.length ;
for(var i=0;i<100000;i ){
List.push(buf) ;
len = buf.length ;
}
var s1 = Buffer.concat(list, len).toString() ;
console.timeEnd("buffer concat test !") ;
console.time("string concat test !") ;
var list = [] ;
for (var i = 100000; i >= 0; i--) {
List.push(str) ;
}
var s2 = list.join("") ;
console.timeEnd("string concat test !") ;

The following are the results:

The reading speed of string is definitely faster, and the buffer also needs toString() operation. So when we save a string, we should still use string. Even if a large string is spliced ​​into a string, it will not be slower than the buffer.

So when do we need to use buffer again? When there is no other way, when we save non-utf-8 strings, binary and other formats, we must use it.

Four, summary

(1), JavaScript is suitable for processing Unicode encoded data, but it is not friendly to the processing of binary data.
(2), so when processing TCP streams or file systems, it is necessary to process octet streams.
(3), Node has several methods for processing, creating and consuming octet streams.
(4) The original data is stored in a Buffer instance. A Buffer is similar to an integer array, but its memory is allocated outside the V8 stack. The size of a Buffer cannot be changed.
(5), the encoding types processed are: ascii, utf8, utf16le, ucs2 (alias of utf16le), base64, binary, hex.
(6), Buffer is a global element, and you can get a Buffer instance directly by using new Buffer().