nodejs implements the function of crawling website images

nodejs implements the function of crawling website images_node.js

韦小宝

Release： 2017-12-16 09:15:53

Original

2108 people have browsed it

I’ll give you an example to teach you how to use nodejs to implement the pictures function of crawling websites. Friends who are interested can save it.

I will explain to you how nodejs implements the function of crawling website images through examples. The following is the full content:

Principle:

Crawler is the most powerful For obvious IO-intensive application scenarios, it is obvious to use node, which makes data mining with small I/O waiting overhead more convenient.

Use the express module to build node services

and use the request module to obtain the target page html code

Download the cheerio module to process the html code (cheerio has a syntax similar to jQuery, so it is easy to use and convenient)

Environment configuration:

npm install express request cheerio --save

Copy after login

(1)Introduce each module

var http = require(&#39;http&#39;);
var request = require(&#39;request);
var cheerio = require(&#39;cheerio&#39;);
var fs = require(&#39;fs&#39;); //用来操作文件
var url = &#39;https://movie.douban.com/cinema/nowplaying/beijing/&#39; //定义要爬的页面

Copy after login

(2) Send a request

http.get(function(res){
  var html = &#39;&#39;;
  var titles = [];
  res.setEncoding(&#39;utf-8&#39;) //防止中文乱码
  res.on(&#39;data&#39;,function(chunk){
    html += chrunk;    //监听data事件 每次取一块数据
  })
  res.on(&#39;end&#39;,function(){
    var $ = cheerio.load(html);  //获取数据完成后，解析html
    //将获取的图片存到images文件夹中
    $(&#39;.mod-bd img&#39;).each(function(index, item){
      //获取图片属性
      var imgName = $(this).parent().next().text().trimg()
      var imgfile = imgName + &#39;.jpeg&#39;;
      var imgSrc = $(this).attr(&#39;src&#39;)
      //采用request模块，向服务器发起请求 获取图片资源
      request.head(imgSrc, function(error, res,body){
        if(error){
          console.log(&#39;失败了&#39;)
        }
      });
      //通过管道的方式用fs模块将图片写到本地的images文件下
      request(imgSrc).pipe.(fs.createWriteStream(&#39;./images/&#39; + imgfile));
    })
    
  })
})

Copy after login

The above is all the content of this article, I hope it will be helpful to my friends! !

Related recommendations:

NodeJS crawler instance encyclopedia of embarrassing things_node.js

NodeJs’s solution to database exception handling

How to use nodejs to implement chat function

The above is the detailed content of nodejs implements the function of crawling website images_node.js. For more information, please follow other related articles on the PHP Chinese website!