Table of Contents
回复讨论(解决方案)
Home Backend Development PHP Tutorial curl的一些小问题

curl的一些小问题

Jun 23, 2016 pm 01:51 PM
curl

怎样用curl抓取最新的数据?
比如,我抓取了一个网站的全部信息,然后那个网站更新了,我怎样才能抓取更新的那几条,而不重复抓取。


回复讨论(解决方案)

抓取到了之后保存到本地数据库。 然后找下,最新那几条有什么规律。如果能区分开来,则之前本地数据库有的就修改一下。 实在是区别不来,把所有抓取的都保存一次,如果保存成功了,清理掉以前的数据即可。

只有看到了,才知道是否是新的

curl 用属性 CURLOPT_WRITEFUNCTION
可在回调函数中判断抓取是否可以提前结束

哦,取数据库最大的一个id,然后得到那条信息的id,然后网页得到的id比数据库取出来大的就是最新的。
可是这样不是所有数据都要判断一次,效率不是很低?

我懂了,这些时间可以忽略不计,结贴,感谢

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
1 months ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to realize the mutual conversion between CURL and python requests in python How to realize the mutual conversion between CURL and python requests in python May 03, 2023 pm 12:49 PM

Both curl and Pythonrequests are powerful tools for sending HTTP requests. While curl is a command-line tool that allows you to send requests directly from the terminal, Python's requests library provides a more programmatic way to send requests from Python code. The basic syntax for converting curl to Pythonrequestscurl command is as follows: curl[OPTIONS]URL When converting curl command to Python request, we need to convert the options and URL into Python code. Here is an example curlPOST command: curl-XPOST https://example.com/api

Tutorial on updating curl version under Linux! Tutorial on updating curl version under Linux! Mar 07, 2024 am 08:30 AM

To update the curl version under Linux, you can follow the steps below: Check the current curl version: First, you need to determine the curl version installed in the current system. Open a terminal and execute the following command: curl --version This command will display the current curl version information. Confirm available curl version: Before updating curl, you need to confirm the latest version available. You can visit curl's official website (curl.haxx.se) or related software sources to find the latest version of curl. Download the curl source code: Using curl or a browser, download the source code file for the curl version of your choice (usually .tar.gz or .tar.bz2

PHP8.1 released: Introducing curl for concurrent processing of multiple requests PHP8.1 released: Introducing curl for concurrent processing of multiple requests Jul 08, 2023 pm 09:13 PM

PHP8.1 released: Introducing curl for concurrent processing of multiple requests. Recently, PHP officially released the latest version of PHP8.1, which introduced an important feature: curl for concurrent processing of multiple requests. This new feature provides developers with a more efficient and flexible way to handle multiple HTTP requests, greatly improving performance and user experience. In previous versions, handling multiple requests often required creating multiple curl resources and using loops to send and receive data respectively. Although this method can achieve the purpose

From start to finish: How to use php extension cURL to make HTTP requests From start to finish: How to use php extension cURL to make HTTP requests Jul 29, 2023 pm 05:07 PM

From start to finish: How to use php extension cURL for HTTP requests Introduction: In web development, it is often necessary to communicate with third-party APIs or other remote servers. Using cURL to make HTTP requests is a common and powerful way. This article will introduce how to use PHP to extend cURL to perform HTTP requests, and provide some practical code examples. 1. Preparation First, make sure that php has the cURL extension installed. You can execute php-m|grepcurl on the command line to check

what is linux curl what is linux curl Apr 20, 2023 pm 05:05 PM

In Linux, curl is a very practical tool for transferring data to and from the server. It is a file transfer tool that uses URL rules to work under the command line; it supports file upload and download, and is a comprehensive transfer tool. . Curl provides a lot of very useful functions, including proxy access, user authentication, ftp upload and download, HTTP POST, SSL connection, cookie support, breakpoint resume and so on.

How to handle 301 redirection of web pages in PHP Curl? How to handle 301 redirection of web pages in PHP Curl? Mar 08, 2024 am 11:36 AM

How to handle 301 redirection of web pages in PHPCurl? When using PHPCurl to send network requests, you will often encounter a 301 status code returned by the web page, indicating that the page has been permanently redirected. In order to handle this situation correctly, we need to add some specific options and processing logic to the Curl request. The following will introduce in detail how to handle 301 redirection of web pages in PHPCurl, and provide specific code examples. 301 redirect processing principle 301 redirect means that the server returns a 30

How to set cookies in php curl How to set cookies in php curl Sep 26, 2021 am 09:27 AM

How to set cookies in php curl: 1. Create a PHP sample file; 2. Set cURL transmission options through the "curl_setopt" function; 3. Pass the cookie in CURL.

Solution to PHP Fatal error: Call to undefined function curl_setopt() Solution to PHP Fatal error: Call to undefined function curl_setopt() Jun 23, 2023 am 08:18 AM

PHP is a widely used open source scripting language used by many websites. However, sometimes you may encounter the problem PHPFatalerror:Calltoundefinedfunctioncurl_setopt(), which may prevent your website from working properly. So what exactly causes this problem? In PHP, curl_setopt() is a very important function, which is used to extend the library through curl

See all articles