Home Web Front-end Front-end Q&A nodejs character set conversion

nodejs character set conversion

May 13, 2023 pm 05:31 PM

Node.js is a very popular server-side JavaScript running environment. When writing Node.js applications, character set conversion is inevitably required. When dealing with Chinese, Japanese, Korean and other systems, character set conversion is necessary, which allows us to correctly handle text in various encoding formats.

In this article, we will introduce character set conversion in Node.js and how to use the iconv library for encoding conversion.

1. Overview of Character Sets

In computers, a character set is a scheme that maps characters to numerical codes. In practical applications, we usually use character encodings such as ASCII, Unicode, and UTF-8.

ASCII (American Standard Code for Information Interchange, American Standard Code for Information Interchange) is one of the earliest character encodings. It only supports English, numbers and a small number of special characters. ASCII encoding has a total of 128 characters, which are represented by 7 binary bits, with the highest bit being 0. ASCII encoding was later expanded to support more character sets, such as Latin-1, ISO-8859 series, etc.

Unicode is a globally unified character encoding scheme that includes all characters in various languages ​​around the world and assigns them unique codes. Unicode encoding has a total of 1,114,112 characters, of which the characters between 0x0000 and 0xFFFF correspond to a 16-bit encoding, called Basic Multilingual Plane (BMP) characters; and the characters between 0x010000 and 0x10FFFF correspond to a 32-bit encoding, called Auxiliary flat characters. Unicode encoding has multiple representation methods, such as UTF-8, UTF-16, etc.

UTF-8 (Unicode Transformation Format-8, Unicode Transformation Format 8-bit) is a variable-length Unicode encoding that can use 1 to 4 bytes to represent a character. It supports all Unicode characters . In UTF-8 encoding, ASCII characters are still represented by one byte, while non-ASCII characters such as Chinese characters are represented by 2 to 3 bytes. Therefore, it is compatible with ASCII encoding and can also save characters in various languages.

2. Node.js character set conversion

Node.js natively provides the Buffer class, which can store binary data in memory and is usually used in scenarios such as network transmission and file reading and writing. When we read binary data from external resources, we need to decode the character set and convert the binary data into a readable string; when we convert the string into binary data, we need to encode it and convert the characters is binary data.

Node.js provides some built-in character set conversion modules, such as buffer.toString(), Buffer.from(), etc., which can perform partial encoding conversion of binary data. For example:

const buf = Buffer.from('中文');
console.log(buf.toString('utf8')); // 中文
console.log(buf.toString('ascii')); // ��w�
console.log(buf.toString('base64')); // 5Lit5paH
Copy after login

In the above example, we convert a string 'Chinese' into a Buffer object and use the toString() method to convert it into other encoding formats. The utf8 encoding result is correct, but the ascii encoding and base64 encoding results are incorrect.

Node.js uses utf8 encoding by default for string conversion. If the character set to be converted is incompatible with utf8, you need to use a third-party library such as iconv for character set conversion.

3. Use the iconv library for character set conversion

iconv is a free open source C library used to convert between different character sets. There is an NPM module called iconv-lite in Node.js, which is a package of the iconv library and provides high-performance character set conversion functions.

Next, we will introduce how to use iconv-lite for character set conversion.

1. Install iconv-lite

First, we need to install iconv-lite.

npm install iconv-lite
Copy after login

2. String encoding conversion

Using iconv-lite to perform encoding conversion is very simple, we only need to call the corresponding method. For example, we convert a GB2312-encoded string to UTF-8 encoding:

const iconv = require('iconv-lite');
const str = '中文字符串';
const buf = iconv.encode(str, 'gb2312');
const str2 = iconv.decode(buf, 'utf8');
console.log(str2); // 中文字符串
Copy after login

In the above example, we first use the iconv.encode method to convert the GB2312-encoded string into a Buffer object, and then use The iconv.decode method converts the Buffer object into a UTF-8 encoded string. The converted string is a correct Chinese string.

3. Stream encoding conversion

When processing large files, you may need to use streams for encoding conversion. Here we can use the iconv-lite-stream library provided by iconv-lite to easily implement stream encoding conversion. For example, we convert a Shift_JIS encoded file to UTF-8 encoding:

const iconv = require('iconv-lite');
const fs = require('fs');
const iconvStream = iconv.decodeStream('Shift_JIS');
const inputStream = fs.createReadStream('shift-jis-file.txt');
const outputStream = fs.createWriteStream('utf8-file.txt');
inputStream.pipe(iconvStream).pipe(outputStream);
Copy after login

In the above example, we first create a readable stream, read the Shift_JIS encoded file, and then create a writable stream, Writes a UTF-8 encoded file. At the same time, we created a stream encoded as Shift_JIS through the iconv.decodeStream method, and chained it with the readable stream and the writable stream to achieve stream encoding conversion.

4. Summary

In this article, we introduced character set conversion in Node.js and how to use the iconv-lite library for encoding conversion. Character set conversion is a very necessary function when dealing with Chinese, Japanese, Korean and other systems. Correct use of character set conversion can effectively avoid problems such as garbled characters. We hope this article helps readers better understand character set conversion in Node.js.

The above is the detailed content of nodejs character set conversion. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
1 months ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

What is useEffect? How do you use it to perform side effects? What is useEffect? How do you use it to perform side effects? Mar 19, 2025 pm 03:58 PM

The article discusses useEffect in React, a hook for managing side effects like data fetching and DOM manipulation in functional components. It explains usage, common side effects, and cleanup to prevent issues like memory leaks.

Explain the concept of lazy loading. Explain the concept of lazy loading. Mar 13, 2025 pm 07:47 PM

Lazy loading delays loading of content until needed, improving web performance and user experience by reducing initial load times and server load.

What are higher-order functions in JavaScript, and how can they be used to write more concise and reusable code? What are higher-order functions in JavaScript, and how can they be used to write more concise and reusable code? Mar 18, 2025 pm 01:44 PM

Higher-order functions in JavaScript enhance code conciseness, reusability, modularity, and performance through abstraction, common patterns, and optimization techniques.

How does currying work in JavaScript, and what are its benefits? How does currying work in JavaScript, and what are its benefits? Mar 18, 2025 pm 01:45 PM

The article discusses currying in JavaScript, a technique transforming multi-argument functions into single-argument function sequences. It explores currying's implementation, benefits like partial application, and practical uses, enhancing code read

How does the React reconciliation algorithm work? How does the React reconciliation algorithm work? Mar 18, 2025 pm 01:58 PM

The article explains React's reconciliation algorithm, which efficiently updates the DOM by comparing Virtual DOM trees. It discusses performance benefits, optimization techniques, and impacts on user experience.Character count: 159

How do you prevent default behavior in event handlers? How do you prevent default behavior in event handlers? Mar 19, 2025 pm 04:10 PM

Article discusses preventing default behavior in event handlers using preventDefault() method, its benefits like enhanced user experience, and potential issues like accessibility concerns.

What is useContext? How do you use it to share state between components? What is useContext? How do you use it to share state between components? Mar 19, 2025 pm 03:59 PM

The article explains useContext in React, which simplifies state management by avoiding prop drilling. It discusses benefits like centralized state and performance improvements through reduced re-renders.

What are the advantages and disadvantages of controlled and uncontrolled components? What are the advantages and disadvantages of controlled and uncontrolled components? Mar 19, 2025 pm 04:16 PM

The article discusses the advantages and disadvantages of controlled and uncontrolled components in React, focusing on aspects like predictability, performance, and use cases. It advises on factors to consider when choosing between them.

See all articles