How to implement information crawler using Node.js (detailed tutorial)

This article mainly introduces the process of developing information crawlers using Node.js. The crawler process can be summarized as downloading the HTML of the target website to the local and then extracting the data. Please refer to this article for specific content details

The recent project needs some information, because the project is written in Node.js, so it is natural to use Node.js to write the crawler

Project address ：github.com/mrtanweijie…, the project crawls the information content of Readhub, Open Source China, Developer Headlines, and 36Kr. Multiple pages are not processed for the time being, because the crawler will run once a day, and now it is obtained every time The latest one can meet the needs, and will be improved later.

The crawler process can be summarized as downloading the HTML of the target website to the local and then extracting the data.

1. Download page

Node.js has many http request libraries. request is used here. The main code is as follows:

requestDownloadHTML () {
 const options = {
  url: this.url,
  headers: {
  'User-Agent': this.randomUserAgent()
  }
 }
 return new Promise((resolve, reject) => {
  request(options, (err, response, body) => {
  if (!err && response.statusCode === 200) {
   return resolve(body)
  } else {
   return reject(err)
  }
  })
 })
 }

Use Promise for packaging so that async/await can be used later. Because many websites are rendered on the client, the downloaded pages may not necessarily contain the desired HTML content. We can use Google's puppeteer to download client-rendered website pages. As we all know, when using npm i, puppeteer may fail to install because it needs to download the Chrome kernel. Just try a few more times:)

puppeteerDownloadHTML () {
 return new Promise(async (resolve, reject) => {
  try {
  const browser = await puppeteer.launch({ headless: true })
  const page = await browser.newPage()
  await page.goto(this.url)
  const bodyHandle = await page.$('body')
  const bodyHTML = await page.evaluate(body => body.innerHTML, bodyHandle)
  return resolve(bodyHTML)
  } catch (err) {
  console.log(err)
  return reject(err)
  }
 })
 }

Of course, it is best to use the interface request directly for pages rendered by the client. way, so that the subsequent HTML parsing is not needed, just do a simple encapsulation, and then you can use it like this: #Funny:)

await new Downloader('http://36kr.com/newsflashes', DOWNLOADER.puppeteer).downloadHTML()

2. HTML content extraction

HTML content extraction is of course using the artifact cheerio. Cheerio exposes the same interface as jQuery and is very simple to use. Open the page F12 in the browser to view the extracted page element nodes, and then extract the content according to the needs

readHubExtract () {
 let nodeList = this.$('#itemList').find('.enableVisited')
 nodeList.each((i, e) => {
  let a = this.$(e).find('a')
  this.extractData.push(
  this.extractDataFactory(
   a.attr('href'),
   a.text(),
   '',
   SOURCECODE.Readhub
  )
  )
 })
 return this.extractData
 }

3. Scheduled tasks

cron 每天跑一跑 
function job () {
 let cronJob = new cron.CronJob({
 cronTime: cronConfig.cronTime,
 onTick: () => {
  spider()
 },
 start: false
 })
 cronJob.start()
}

4. Data persistence

Theoretically, data persistence should not be within the scope of crawler concern. Use mongoose to create Model

import mongoose from 'mongoose'
const Schema = mongoose.Schema
const NewsSchema = new Schema(
 {
 title: { type: 'String', required: true },
 url: { type: 'String', required: true },
 summary: String,
 recommend: { type: Boolean, default: false },
 source: { type: Number, required: true, default: 0 },
 status: { type: Number, required: true, default: 0 },
 createdTime: { type: Date, default: Date.now }
 },
 {
 collection: 'news'
 }
)
export default mongoose.model('news', NewsSchema)

Basic operations

import { OBJ_STATUS } from '../../Constants'
class BaseService {
 constructor (ObjModel) {
 this.ObjModel = ObjModel
 }

 saveObject (objData) {
 return new Promise((resolve, reject) => {
  this.ObjModel(objData).save((err, result) => {
  if (err) {
   return reject(err)
  }
  return resolve(result)
  })
 })
 }
}
export default BaseService

Information

import BaseService from './BaseService'
import News from '../models/News'
class NewsService extends BaseService {}
export default new NewsService(News)

Happily save data

await newsService.batchSave(newsListTem)

For more information, just go to Github and clone the project to see it.

Summary

The above is what I compiled for everyone. I hope it will be helpful to everyone in the future.

Related articles:

How to build a d3 force-directed graph using react (detailed tutorial)

How to implement instant messaging using nodejs

About axios issues related to Vue.use

The above is the detailed content of How to implement information crawler using Node.js (detailed tutorial). For more information, please follow other related articles on the PHP Chinese website!