Table of Contents
Reply content:
Home Backend Development PHP Tutorial Problem with crawler obtaining AJax data

Problem with crawler obtaining AJax data

Jul 06, 2016 pm 01:52 PM
http java php python web crawler

I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
Problem with crawler obtaining AJax data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Problem with crawler obtaining AJax data

Please tell me how to get json data

Reply content:

I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
Problem with crawler obtaining AJax data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Problem with crawler obtaining AJax data

Please tell me how to get json data

This is controlled by header
Problem with crawler obtaining AJax data

Just add the corresponding request header. Attached is my code

<code>#coding=utf-8

import requests as RQ

hburl = "http://huaban.com/boards/28195582/?ipva9fpx&max=759391397&limit=20&wfl=1"
heads = {
'Host':"huaban.com",
'Referer':"{http://huaban.com/boards/28195582/",
'User-Agent':"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36",
'X-Request':"JSON",
'X-Requested-With':"XMLHttpRequest"
}


print RQ.get(hburl, headers=heads).content</code>
Copy after login

Referer is a common header used to prevent web pages from being obtained outside this site. When doing crawlers, be sure to ensure that the Referer is set

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Roblox: Bubble Gum Simulator Infinity - How To Get And Use Royal Keys
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Nordhold: Fusion System, Explained
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1664
14
PHP Tutorial
1269
29
C# Tutorial
1248
24
Python vs. JavaScript: Development Environments and Tools Python vs. JavaScript: Development Environments and Tools Apr 26, 2025 am 12:09 AM

Both Python and JavaScript's choices in development environments are important. 1) Python's development environment includes PyCharm, JupyterNotebook and Anaconda, which are suitable for data science and rapid prototyping. 2) The development environment of JavaScript includes Node.js, VSCode and Webpack, which are suitable for front-end and back-end development. Choosing the right tools according to project needs can improve development efficiency and project success rate.

The Compatibility of IIS and PHP: A Deep Dive The Compatibility of IIS and PHP: A Deep Dive Apr 22, 2025 am 12:01 AM

IIS and PHP are compatible and are implemented through FastCGI. 1.IIS forwards the .php file request to the FastCGI module through the configuration file. 2. The FastCGI module starts the PHP process to process requests to improve performance and stability. 3. In actual applications, you need to pay attention to configuration details, error debugging and performance optimization.

Python vs. C  : Understanding the Key Differences Python vs. C : Understanding the Key Differences Apr 21, 2025 am 12:18 AM

Python and C each have their own advantages, and the choice should be based on project requirements. 1) Python is suitable for rapid development and data processing due to its concise syntax and dynamic typing. 2)C is suitable for high performance and system programming due to its static typing and manual memory management.

Golang vs. Python: The Pros and Cons Golang vs. Python: The Pros and Cons Apr 21, 2025 am 12:17 AM

Golangisidealforbuildingscalablesystemsduetoitsefficiencyandconcurrency,whilePythonexcelsinquickscriptinganddataanalysisduetoitssimplicityandvastecosystem.Golang'sdesignencouragesclean,readablecodeanditsgoroutinesenableefficientconcurrentoperations,t

Laravel vs. Python (with Frameworks): A Comparative Analysis Laravel vs. Python (with Frameworks): A Comparative Analysis Apr 21, 2025 am 12:15 AM

Laravel is suitable for projects that teams are familiar with PHP and require rich features, while Python frameworks depend on project requirements. 1.Laravel provides elegant syntax and rich features, suitable for projects that require rapid development and flexibility. 2. Django is suitable for complex applications because of its "battery inclusion" concept. 3.Flask is suitable for fast prototypes and small projects, providing great flexibility.

Does Python projects need to be layered? Does Python projects need to be layered? Apr 19, 2025 pm 10:06 PM

Discussion on Hierarchical Structure in Python Projects In the process of learning Python, many beginners will come into contact with some open source projects, especially projects using the Django framework...

What happens if session_start() is called multiple times? What happens if session_start() is called multiple times? Apr 25, 2025 am 12:06 AM

Multiple calls to session_start() will result in warning messages and possible data overwrites. 1) PHP will issue a warning, prompting that the session has been started. 2) It may cause unexpected overwriting of session data. 3) Use session_status() to check the session status to avoid repeated calls.

Python vs. C  : Which Language to Choose for Your Project? Python vs. C : Which Language to Choose for Your Project? Apr 21, 2025 am 12:17 AM

Choosing Python or C depends on project requirements: 1) If you need rapid development, data processing and prototype design, choose Python; 2) If you need high performance, low latency and close hardware control, choose C.

See all articles