Problem with crawler obtaining AJax data
I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Please tell me how to get json data
Reply content:
I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Please tell me how to get json data
This is controlled by header
Just add the corresponding request header. Attached is my code
<code>#coding=utf-8 import requests as RQ hburl = "http://huaban.com/boards/28195582/?ipva9fpx&max=759391397&limit=20&wfl=1" heads = { 'Host':"huaban.com", 'Referer':"{http://huaban.com/boards/28195582/", 'User-Agent':"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36", 'X-Request':"JSON", 'X-Requested-With':"XMLHttpRequest" } print RQ.get(hburl, headers=heads).content</code>
Referer is a common header used to prevent web pages from being obtained outside this site. When doing crawlers, be sure to ensure that the Referer is set

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics











Both Python and JavaScript's choices in development environments are important. 1) Python's development environment includes PyCharm, JupyterNotebook and Anaconda, which are suitable for data science and rapid prototyping. 2) The development environment of JavaScript includes Node.js, VSCode and Webpack, which are suitable for front-end and back-end development. Choosing the right tools according to project needs can improve development efficiency and project success rate.

IIS and PHP are compatible and are implemented through FastCGI. 1.IIS forwards the .php file request to the FastCGI module through the configuration file. 2. The FastCGI module starts the PHP process to process requests to improve performance and stability. 3. In actual applications, you need to pay attention to configuration details, error debugging and performance optimization.

Python and C each have their own advantages, and the choice should be based on project requirements. 1) Python is suitable for rapid development and data processing due to its concise syntax and dynamic typing. 2)C is suitable for high performance and system programming due to its static typing and manual memory management.

Golangisidealforbuildingscalablesystemsduetoitsefficiencyandconcurrency,whilePythonexcelsinquickscriptinganddataanalysisduetoitssimplicityandvastecosystem.Golang'sdesignencouragesclean,readablecodeanditsgoroutinesenableefficientconcurrentoperations,t

Laravel is suitable for projects that teams are familiar with PHP and require rich features, while Python frameworks depend on project requirements. 1.Laravel provides elegant syntax and rich features, suitable for projects that require rapid development and flexibility. 2. Django is suitable for complex applications because of its "battery inclusion" concept. 3.Flask is suitable for fast prototypes and small projects, providing great flexibility.

Discussion on Hierarchical Structure in Python Projects In the process of learning Python, many beginners will come into contact with some open source projects, especially projects using the Django framework...

Multiple calls to session_start() will result in warning messages and possible data overwrites. 1) PHP will issue a warning, prompting that the session has been started. 2) It may cause unexpected overwriting of session data. 3) Use session_status() to check the session status to avoid repeated calls.

Choosing Python or C depends on project requirements: 1) If you need rapid development, data processing and prototype design, choose Python; 2) If you need high performance, low latency and close hardware control, choose C.
