node.js - nodejs superagent读取网页内容，怎么控制请求的频率

Question

nodejs superagent读取网页内容，怎么控制请求的频率，比如说1秒一次请求，因为做数据抓取，不能太快，我想到的是用settimeout之类的，但是感觉不是特别合适，有没有比较好的实现方法？

Answer 1

node-schedule - Cron style timer

*    *    *    *    *    *
┬    ┬    ┬    ┬    ┬    ┬
│    │    │    │    │    |
│    │    │    │    │    └ day of week (0 - 7) (0 or 7 is Sun)
│    │    │    │    └───── month (1 - 12)
│    │    │    └────────── day of month (1 - 31)
│    │    └─────────────── hour (0 - 23)
│    └──────────────────── minute (0 - 59)
└───────────────────────── second (0 - 59, OPTIONAL)

// 每天 6:30,10:30,14:30,18:30,21:30 定时dosomething...
schedule.scheduleJob('0 30 6,10,14,18,21 * * *', function() {
    // do something....
});

// 每隔十分钟dosomething...
schedule.scheduleJob('0 0,10,20,30,40,50 * * * *', function() {
    // do something...
});

https://github.com/xCss/bing/... Here are examples of usage

Answer 2

Some websites have limits on the number of concurrent connections. If the request is sent too fast, it will return empty or report an error. Therefore, it is recommended that you use async to control concurrency.
https://github.com/caolan/async