简体中文(ZH-CN) English(EN) 繁体中文(ZH-TW) 日本語(JA) 한국어(KO) Melayu(MS) Français(FR) Deutsch(DE)
Home > Web Front-end > JS Tutorial > body text

nodejs爬虫抓取数据之编码问题_node.js

WBOY
Release： 2016-05-16 15:51:39
Original
1389 people have browsed it

cheerio DOM化并解析的时候

1.假如使用了 .text()方法，则一般不会有html实体编码的问题出现

2.如果使用了 .html()方法，则很多情况下(多数是非英文的时候）都会出现，这时，可能就需要转义一番了

类似这些 因为需要作数据存储，所有需要转换

复制代码 代码如下:

Халк крушит. Новый способ исполнен

大多数都是(x)?\w+的格式

所以就用正则转换一番

var body = ....//这里就是请求后获得的返回数据，或者那些 .html()后获取的

//一般可以先转换为标准unicode格式（有需要就添加：当返回的数据呈现太多\\u 之类的时）
body=unescape(body.replace(/\\u/g,"%u"));
//再对实体符进行转义
//有x则表示是16进制，$1就是匹配是否有x ，$2就是匹配出的第二个括号捕获到的内容，将$2以对应进制表示转换
body = body.replace(/&#(x)?(\w+);/g,function($,$1,$2){
        return String.fromCharCode(parseInt($2,$1?16:10));
       });
Copy after login

ok ～

当然了，网上也有很多个转换的版本，适用的就行了

后记：

当使用爬虫抓取网页数据时，cheerio模块是经常使用到底，它像jq那样方便快捷

（但有些功能并未支持或者换了某种形式，比如 jq的 jQuery('.myClass').prop('outerHTML') ，cheerio则等价于 jQuery.html('.myClass')http://www.mgenware.com/blog/?p=2514）

Related labels：
nodejs reptile coding
source：php.cn
Previous article：javascript表格的渲染组件_javascript技巧 Next article：解决jquery实现的radio重新选中的问题_jquery
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Latest Articles by Author
Latest Issues
Ways to resolve "Error: error:0308010C:digital envelope routines::unsupported" error in Nodejs 18 I need help with my NuxtJS application. Recently, I got an ESLint conflict in the applicat...
From 2023-10-18 15:53:45
0
2
220
How to connect to the database with node.js How to connect nodejs to database, express connection? How to connect nodejs to database, ...
From 2023-10-10 10:06:27
0
9
184
Unserializable error when returning response from selecting documents from MongoDb collection. React and NodeJS My goal is just to return complaints that belong to this user. I have two collections in M...
From 2023-09-16 18:20:04
0
1
205
In nodeJS I want to set a default folder for my Box API download function I'm using BoxSDK for NodeJS and I have a function to download a file. I just need to set t...
From 2023-09-16 16:35:25
0
1
435
Nodejs: Correct way to return non-HTML data to browser? User JWT is sent to the browser as an HTTP-only cookie. My question is, what is the best w...
From 2023-09-15 16:04:49
0
1
736
Related Topics
More>
Popular Recommendations
Popular Tutorials
More>
Related Tutorials
Popular Recommendations
Latest courses
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template
About us Disclaimer Sitemap
php.cn：Public welfare online PHP training，Help PHP learners grow quickly！