连续的分页采集问题
目标站点:http://news.xinhuanet.com/photo/2015-01/29/c_127433852.htm
获取内容:正文图片和文字说明;
本人实现方法:
因为分页地址是连续的,我是在目标地址上做循环
http://news.xinhuanet.com/photo/2015-01/29/c_127433852_2.htm
用get_headers
访问后 获取返回是否为200,是的话进行内容抓取
1、curl获取当前地址html后,截取所需内容
2、_2这个就是变量,自增后继续判断下一页是否可以访问,可以访问就继续采集
以上是可以实现采集抓取的,但是好像速度很慢,而且每次都是curl去抓取对服务器貌似也有压力
问题:如何改进,大家是如何用php做的抓取,求思路和方向
回复内容:
目标站点:http://news.xinhuanet.com/photo/2015-01/29/c_127433852.htm
获取内容:正文图片和文字说明;
本人实现方法:
因为分页地址是连续的,我是在目标地址上做循环
http://news.xinhuanet.com/photo/2015-01/29/c_127433852_2.htm
用get_headers
访问后 获取返回是否为200,是的话进行内容抓取
1、curl获取当前地址html后,截取所需内容
2、_2这个就是变量,自增后继续判断下一页是否可以访问,可以访问就继续采集
以上是可以实现采集抓取的,但是好像速度很慢,而且每次都是curl去抓取对服务器貌似也有压力
问题:如何改进,大家是如何用php做的抓取,求思路和方向
下面不是有分页么,直接获取<div id="div_currpage"></div>
里面的链接存到数组抓吧
- 直接抓取,抓取返回结果的时候就先判断状态码是否为200
- Curl并发

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

In this chapter, we will understand the Environment Variables, General Configuration, Database Configuration and Email Configuration in CakePHP.

PHP 8.4 brings several new features, security improvements, and performance improvements with healthy amounts of feature deprecations and removals. This guide explains how to install PHP 8.4 or upgrade to PHP 8.4 on Ubuntu, Debian, or their derivati

To work with date and time in cakephp4, we are going to make use of the available FrozenTime class.

To work on file upload we are going to use the form helper. Here, is an example for file upload.

CakePHP is an open-source framework for PHP. It is intended to make developing, deploying and maintaining applications much easier. CakePHP is based on a MVC-like architecture that is both powerful and easy to grasp. Models, Views, and Controllers gu

In this chapter, we are going to learn the following topics related to routing ?

Working with database in CakePHP is very easy. We will understand the CRUD (Create, Read, Update, Delete) operations in this chapter.

Validator can be created by adding the following two lines in the controller.
