


Use python to make an automatic image downloader in 5 minutes
Python crawler practice - automatic image downloader
Basic steps for making a crawler
By the way, through this small example, you can master some basic steps about making a crawler.
Generally speaking, making a crawler requires the following steps:
Analyze requirements (yes, requirement analysis is very important, don’t tell me your teacher didn’t teach you )
Analyze the web page source code and cooperate with F12 (the web page source code is not as messy as F12, do you want to see me to death?)
Prepared Regular expression or XPath expression (the artifact mentioned earlier)
Formally write python crawler code
Effect
Run:
Well, let me enter keywords and let me think about it, what should I enter? It seems to be a bit exposed.
Enter
It seems that the download has started! Great! , I looked at the downloaded pictures, and wow, instantly I felt like I had added a lot more emoticons....
Okay, that’s pretty much it.
Requirement Analysis
"I want pictures, but I don’t want to search online"
"It’s best to download them automatically"
......
This is Requirements, okay, let's start analyzing the requirements. At least two functions must be implemented, one is to search for pictures, and the other is to automatically download.
First of all, when searching for pictures, the easiest thing to think of is the result of crawling Baidu pictures. Okay, then let’s go to Baidu pictures and take a look.
That’s basically it. , quite beautiful.
Let's try to search for something. I type a bad word and a series of search results come out. What does this mean...
Just find one Enter
#Okay, we have seen a lot of pictures, it would be great if we could climb down all the pictures here. We saw that there is keyword information in the URL
# We tried to change the keyword directly in the URL, did it jump?
In this way, you can search for pictures with specific keywords through this URL, so in theory, we can search for specific pictures without opening the web page. The next question is how to implement automatic downloading. In fact, using previous knowledge, we know that we can use request to get the URL of the image, then crawl it down and save it as a .jpg.
So this project should be completed.
Analyzing web pages
Okay, let’s start with the next step, analyzing the web page source code. Here I first switch back to the traditional page. Why do I do this? Because Baidu pictures currently use the waterfall flow mode, dynamically loading pictures, which is very troublesome to process. The traditional page turning interface is much better.
Another tip here is: if you can crawl the mobile version, don’t crawl the computer version, because the code of the mobile version is very clear and it is easy to get the required content.
Okay, I switched back to the traditional version, but it still has page numbers and is comfortable to read.
Let’s right-click and view the source code
What the hell is this, how can we see it clearly! !
At this time, it’s time to use F12, the developer tool! Let's go back to the previous page, press F12, and the toolbar below will come out. What we need to use is the thing in the upper left corner. One is mouse following, and the other is switching mobile versions, both of which are very useful to us. We use the first
here
然后选择你想看源代码的地方,就可以发现,下面的代码区自动定位到了这个位置,是不是很NB!
我们复制这个地址
然后到刚才的乱七八糟的源代码里搜索一下,发现它的位置了!(小样!我还找不到你!)但是这里我们又疑惑了,这个图片怎么有这么多地址,到底用哪个呢?我们可以看到有thumbURL,middleURL,hoverURL,objURL
通过分析可以知道,前面两个是缩小的版本,hover是鼠标移动过后显示的版本,objURL应该是我们需要的,不信可以打开这几个网址看看,发现obj那个最大最清晰。
好了,找到了图片位置,我们就开始分析它的代码。我看看是不是所有的objURL全是图片
貌似都是以.jpg格式结尾的,那应该跑不了了,我们可以看到搜索出61条,说明应该有61个图片
编写正则表达式
通过前面的学习,写出如下的一条正则表达式不难把?
pic_url = re.findall('"objURL":"(.*?)",',html,re.S)
编写爬虫代码
好了,正式开始编写爬虫代码了。这里我们就用了2个包,一个是正则,一个是requests包,之前也介绍过了,没看的回去看!
#-*- coding:utf-8 -*-import reimport requests
然后我们把刚才的网址粘过来,传入requests,然后把正则表达式写好
url = 'http://image.baidu.com/search/flip?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1&fm=result&fr=&sf=1&fmq=1460997499750_R&pv=&ic=0&nc=1&z=&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&ie=utf-8&word=%E5%B0%8F%E9%BB%84%E4%BA%BA' html = requests.get(url).text pic_url = re.findall('"objURL":"(.*?)",',html,re.S)
理论有很多图片,所以要循环,我们打印出结果来看看,然后用request获取网址,这里由于有些图片可能存在网址打不开的情况,加个5秒超时控制。
pic_url = re.findall('"objURL":"(.*?)",',html,re.S) i = 0for each in pic_url:print eachtry: pic= requests.get(each, timeout=10)except requests.exceptions.ConnectionError:print '【错误】当前图片无法下载'continue
好了,再就是把网址保存下来,我们在事先在当前目录建立一个picture目录,把图片都放进去,命名的时候,用数字命名把
string = 'pictures\\'+str(i) + '.jpg' fp = open(string,'wb') fp.write(pic.content) fp.close() i += 1
整个代码就是这样:
#-*- coding:utf-8 -*- import re import requests url = 'http://image.baidu.com/search/flip?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1&fm=result&fr=&sf=1&fmq=1460997499750_R&pv=&ic=0&nc=1&z=&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&ie=utf-8&word=%E5%B0%8F%E9%BB%84%E4%BA%BA'html = requests.get(url).text pic_url = re.findall('"objURL":"(.*?)",',html,re.S)i = 0for each in pic_url: print each try: pic= requests.get(each, timeout=10) except requests.exceptions.ConnectionError: print '【错误】当前图片无法下载' continue string = 'pictures\\'+str(i) + '.jpg' fp = open(string,'wb') fp.write(pic.content) fp.close()i += 1
我们运行一下,看效果(什么你说这是什么IDE感觉很炫!?赶紧去装Pycharm,Pycharm的配置和使用看这个文章!)!
好了我们下载了58个图片,咦刚才不是应该是61个吗?
我们看,运行中出现了有一些图片下载不了
我们还看到有图片没显示出来,打开网址看,发现确实没了。
所以,百度有些图片它缓存到了自己的机器上,所以你还能看见,但是实际连接已经失效
好了,现在自动下载问题解决了,那根据关键词搜索图片呢?只要改url就行了,我这里把代码写下来了
word = raw_input("Input key word: ") url = 'http://image.baidu.com/search/flip?tn=baiduimage&ie=utf-8&word='+word+'&ct=201326592&v=flip'result = requests.get(url)
好了,享受你第一个图片下载爬虫吧!!当然不只能下载百度的图片拉,依葫芦画瓢,你现在应该做很多事情了,比如爬取头像,爬淘宝展示图,或是...美女图片,捂脸。一切都凭客官你的想象了,当然,作为爬虫的第一个实例,虽然纯用request已经能解决很多问题了,但是效率还是不够高,如果想要高效爬取大量数据,还是用scrapy吧
The above is the detailed content of Use python to make an automatic image downloader in 5 minutes. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics





PHP is mainly procedural programming, but also supports object-oriented programming (OOP); Python supports a variety of paradigms, including OOP, functional and procedural programming. PHP is suitable for web development, and Python is suitable for a variety of applications such as data analysis and machine learning.

PHP is suitable for web development and rapid prototyping, and Python is suitable for data science and machine learning. 1.PHP is used for dynamic web development, with simple syntax and suitable for rapid development. 2. Python has concise syntax, is suitable for multiple fields, and has a strong library ecosystem.

VS Code can be used to write Python and provides many features that make it an ideal tool for developing Python applications. It allows users to: install Python extensions to get functions such as code completion, syntax highlighting, and debugging. Use the debugger to track code step by step, find and fix errors. Integrate Git for version control. Use code formatting tools to maintain code consistency. Use the Linting tool to spot potential problems ahead of time.

VS Code can run on Windows 8, but the experience may not be great. First make sure the system has been updated to the latest patch, then download the VS Code installation package that matches the system architecture and install it as prompted. After installation, be aware that some extensions may be incompatible with Windows 8 and need to look for alternative extensions or use newer Windows systems in a virtual machine. Install the necessary extensions to check whether they work properly. Although VS Code is feasible on Windows 8, it is recommended to upgrade to a newer Windows system for a better development experience and security.

VS Code extensions pose malicious risks, such as hiding malicious code, exploiting vulnerabilities, and masturbating as legitimate extensions. Methods to identify malicious extensions include: checking publishers, reading comments, checking code, and installing with caution. Security measures also include: security awareness, good habits, regular updates and antivirus software.

Python is more suitable for beginners, with a smooth learning curve and concise syntax; JavaScript is suitable for front-end development, with a steep learning curve and flexible syntax. 1. Python syntax is intuitive and suitable for data science and back-end development. 2. JavaScript is flexible and widely used in front-end and server-side programming.

PHP originated in 1994 and was developed by RasmusLerdorf. It was originally used to track website visitors and gradually evolved into a server-side scripting language and was widely used in web development. Python was developed by Guidovan Rossum in the late 1980s and was first released in 1991. It emphasizes code readability and simplicity, and is suitable for scientific computing, data analysis and other fields.

In VS Code, you can run the program in the terminal through the following steps: Prepare the code and open the integrated terminal to ensure that the code directory is consistent with the terminal working directory. Select the run command according to the programming language (such as Python's python your_file_name.py) to check whether it runs successfully and resolve errors. Use the debugger to improve debugging efficiency.
