抓取时ip被封禁问题
最近需要抓取一个站点的内容，我是用的snoopy来抓的，一开始发现会封ip后，我按照网上的解决方案将user-agent换成了google的蜘蛛的，并且用snoopy进行了伪造了ip（每抓一条就换一个随机ip)但是抓取一百多个页面后还是被封ip导致无法抓取，有什么好的解决方法吗？
回复讨论(解决方案)
应该是访问太频繁了。
应该是访问太频繁了。 那该如何解决呢，sleep吗，但是要抓取的数据量很大，如果sleep的话根本没那么多时间抓
并且用snoopy进行了伪造了ip
这个是忽悠你的..
能伪造ip？这。。。
抓太快管你是谁啊，照封
并且用snoopy进行了伪造了ip
这个是忽悠你的.. 额，原来如此
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1386
52
Alipay PHP SDK transfer error: How to solve the problem of 'Cannot declare class SignData'?
Apr 01, 2025 am 07:21 AM
Alipay PHP...
Explain JSON Web Tokens (JWT) and their use case in PHP APIs.
Apr 05, 2025 am 12:04 AM
JWT is an open standard based on JSON, used to securely transmit information between parties, mainly for identity authentication and information exchange. 1. JWT consists of three parts: Header, Payload and Signature. 2. The working principle of JWT includes three steps: generating JWT, verifying JWT and parsing Payload. 3. When using JWT for authentication in PHP, JWT can be generated and verified, and user role and permission information can be included in advanced usage. 4. Common errors include signature verification failure, token expiration, and payload oversized. Debugging skills include using debugging tools and logging. 5. Performance optimization and best practices include using appropriate signature algorithms, setting validity periods reasonably,
Describe the SOLID principles and how they apply to PHP development.
Apr 03, 2025 am 12:04 AM
The application of SOLID principle in PHP development includes: 1. Single responsibility principle (SRP): Each class is responsible for only one function. 2. Open and close principle (OCP): Changes are achieved through extension rather than modification. 3. Lisch's Substitution Principle (LSP): Subclasses can replace base classes without affecting program accuracy. 4. Interface isolation principle (ISP): Use fine-grained interfaces to avoid dependencies and unused methods. 5. Dependency inversion principle (DIP): High and low-level modules rely on abstraction and are implemented through dependency injection.
How to automatically set permissions of unixsocket after system restart?
Mar 31, 2025 pm 11:54 PM
How to automatically set the permissions of unixsocket after the system restarts. Every time the system restarts, we need to execute the following command to modify the permissions of unixsocket: sudo...
How to debug CLI mode in PHPStorm?
Apr 01, 2025 pm 02:57 PM
How to debug CLI mode in PHPStorm? When developing with PHPStorm, sometimes we need to debug PHP in command line interface (CLI) mode...
Explain the concept of late static binding in PHP.
Mar 21, 2025 pm 01:33 PM
Article discusses late static binding (LSB) in PHP, introduced in PHP 5.3, allowing runtime resolution of static method calls for more flexible inheritance.Main issue: LSB vs. traditional polymorphism; LSB's practical applications and potential perfo
How to send a POST request containing JSON data using PHP's cURL library?
Apr 01, 2025 pm 03:12 PM
Sending JSON data using PHP's cURL library In PHP development, it is often necessary to interact with external APIs. One of the common ways is to use cURL library to send POST�...
Explain late static binding in PHP (static::).
Apr 03, 2025 am 12:04 AM
Static binding (static::) implements late static binding (LSB) in PHP, allowing calling classes to be referenced in static contexts rather than defining classes. 1) The parsing process is performed at runtime, 2) Look up the call class in the inheritance relationship, 3) It may bring performance overhead.
