Example of implementing simple crawler function in python

When we browse the Internet every day, we often see some good-looking pictures, and we want to save and download these pictures, or use them as desktop wallpapers, or as design materials.

Our most common method is to right-click the mouse and select Save As. However, some pictures do not have a save as option when you right-click the mouse. Another way is to capture them with a screenshot tool, but this will reduce the clarity of the picture. Okay~! In fact, you are very good. Right-click to view the page source code.

We can use python to implement such a simple crawler function and crawl the code we want locally. Let's take a look at how to use python to implement such a function.

One, get the entire page data

First we can get the entire page information of the image to be downloaded.
getjpg.py

#coding=utf-8
import urllib
def getHtml(url):
  page = urllib.urlopen(url)
  html = page.read()
  return html

html = getHtml("http://tieba.baidu.com/p/2738151262")
print html

import re
import urllib

def getHtml(url):
  page = urllib.urlopen(url)
  html = page.read()
  return html

def getImg(html):
  reg = r'src="(.+?\.jpg)" pic_ext'
  imgre = re.compile(reg)
  imglist = re.findall(imgre,html)
  return imglist   
  
html = getHtml("http://tieba.baidu.com/p/2460150866")
print getImg(html)

#coding=utf-8
import urllib
import re

def getHtml(url):
  page = urllib.urlopen(url)
  html = page.read()
  return html

def getImg(html):
  reg = r'src="(.+?\.jpg)" pic_ext'
  imgre = re.compile(reg)
  imglist = re.findall(imgre,html)
  x = 0
  for imgurl in imglist:
    urllib.urlretrieve(imgurl,'%s.jpg' % x)
    x+=1


html = getHtml("http://tieba.baidu.com/p/2460150866")

print getImg(html)