A brief discussion of duplex streams in Nodejs

Related recommendations: "node js tutorial"

Duplex stream is a stream that implements both Readable and Writable, that is, it can As upstream production data, it can also be used as downstream consumption data, so that it can be in the middle part of the data flow pipeline, that is,

 rs.pipe(rws1).pipe(rws2).pipe(rws3).pipe(ws);

There are two commonly used duplex streams in NodeJS

• Duplex

• Transform

Duplex

Implement Duplex

Similar to the implementation methods of Readable and Writable, implementing Duplex stream is very simple, but Duplex implements both Readable and Writable. NodeJS does not support multiple inheritance, so we need to inherit the Duplex class

• Inherit the Duplex class

• Implement the _read() method

• Implement the _write() method

I believe everyone is familiar with the implementation of the read() and write() methods, because they are exactly the same as Readable and Writable.

const Duplex = require('stream').Duplex;

const myDuplex = new Duplex({
  read(size) {
    // ...
  },
  write(chunk, encoding, callback) {
    // ...
  }
});

Constructor parameters

The Duplex instance contains both readable and writable streams. Several parameters can be passed when instantiating the Duplex class

• readableObjectMode: Whether the readable stream is set to ObjectMode, the default is false
• writableObjectMode: Whether the writable stream is set to ObjectMode, the default is false
• allowHalfOpen: Default is true. If set to false, when the writing end ends, the stream will automatically end the reading end, and vice versa.

Small example

After understanding Readable and Writable, it is very simple to see Duplex. Just use an example from the official website

 const Duplex = require('stream').Duplex;
const kSource = Symbol('source');

class MyDuplex extends Duplex {
  constructor(source, options) {
    super(options);
    this[kSource] = source;
  }

  _write(chunk, encoding, callback) {
    // The underlying source only deals with strings
    if (Buffer.isBuffer(chunk))
      chunk = chunk.toString();
    this[kSource].writeSomeData(chunk);
    callback();
  }

  _read(size) {
    this[kSource].fetchSomeData(size, (data, encoding) => {
      this.push(Buffer.from(data, encoding));
    });
  }
}

Of course, this is pseudo code that cannot be executed, but the role of Duplex can be seen. It can produce data and consume data, so it can be in the middle of the data flow pipeline. Common Duplex flows are

• Tcp Scoket
• Zlib
• Crypto

Transform

Transform is also a duplex stream. It seems to be a duplicate of Duplex, but both There is an important difference: although Duplex has readable streams and writable streams, they are relatively independent; the data in Transform's readable stream will automatically enter the writable stream after certain processing.

Although it will enter the writable stream from the readable stream, it does not mean that the amount of data between the two is the same. The certain processing logic mentioned above will determine whether to transform the readable stream and then put it into the writable stream. , the original meaning of transform is transformation, which describes the Transform flow function very appropriately.

Our most common zlib for compression and decompression is the Transform stream. The amount of data before and after compression and decompression is obviously different. The function of the transform stream is to input a zip package and a decompressed file or vice versa. Most of the duplex streams we usually use are Transform.

Implementing Tranform

The Tranform class internally inherits Duplex and implements the writable.write() and readable._read() methods. We want to customize a Transform stream. Just

• Inherit the Transform class

• implement the _transform() method

• implement _flush () method (optional)

_transform(chunk, encoding, callback) method is used to receive data and generate output. We are already familiar with the parameters, which are the same as Writable, chunk The default is Buffer, unless decodeStrings is set to false.

You can call this.push(data) inside the _transform() method to produce data and hand it over to the writable stream, or you can not call it, which means that input will not produce output.

When the data is processed, callback(err, data) must be called. The first parameter is used to pass error information. The second parameter can be omitted. If it is passed in, the effect is the same as this.push(data) Same

 transform.prototype._transform = function (data, encoding, callback) {
  this.push(data);
  callback();
};

transform.prototype._transform = function (data, encoding, callback) {
  callback(null, data);
};

Sometimes, the transform operation may need to write more data to the writable stream at the end of the stream. For example, Zlib streams store some internal state in order to optimize compressed output. In this case, the _flush() method can be used, which will be called before all written data is consumed, triggering 'end'.

Transform event

Transform stream has two commonly used events

• finish from Writable

• end from Readable

When transform.end() is called and the data is processed by _transform(), finish will be triggered. After _flush is called, all data will be output and end will be triggered. event.

Comparison

After understanding Readable and Writable, it is very natural to understand duplex streams, but the difference between the two will confuse some beginners. A simple distinction: Duplex's readable stream and readable stream There is no direct relationship between write streams. The data in the readable stream in Transform will be automatically put into the writable stream after processing.

You can intuitively understand the difference between Duplex and Transform by looking at two simple examples

TCP socket

net 模块可以用来创建 socket，socket 在 NodeJS 中是一个典型的 Duplex，看一个 TCP 客户端的例子

var net = require('net');

//创建客户端
var client = net.connect({port: 1234}, function() {
    console.log('已连接到服务器');
    client.write('Hi!');
});

//data事件监听。收到数据后，断开连接
client.on('data', function(data) {
    console.log(data.toString());
    client.end();
});

//end事件监听，断开连接时会被触发
client.on('end', function() {
    console.log('已与服务器断开连接');
});

可以看到 client 就是一个 Duplex，可写流用于向服务器发送消息，可读流用于接受服务器消息，两个流内的数据并没有直接的关系。

gulp

gulp 非常擅长处理代码本地构建流程，看一段官网的示例代码

gulp.src('client/templates/*.jade')
  .pipe(jade())
  .pipe(minify())
  .pipe(gulp.dest('build/minified_templates'));

其中 jada() 和 minify() 就是典型的 Transform，处理流程大概是

.jade 模板文件 -> jada() -> html 文件 -> minify -> 压缩后的 html

可以看出来，jade() 和 minify() 都是对输入数据做了些特殊处理，然后交给了输出数据。

这样简单的对比就能看出 Duplex 和 Transform 的区别，在平时实用的时候，当一个流同事面向生产者和消费者服务的时候我们会选择 Duplex，当只是对数据做一些转换工作的时候我们便会选择使用 Tranform。

更多编程相关知识，请访问：编程学习网站！！

The above is the detailed content of A brief discussion of duplex streams in Nodejs. For more information, please follow other related articles on the PHP Chinese website!