Python method to crawl APP download link

First is the preparation work

Python 2.7.11: Download python

Pycharm: Download Pycharm

where python2 and python3 are currently Synchronous release, I am using python2 as the environment here. Pycharm is a relatively efficient Python IDE, but it requires payment.

Basic ideas for implementation

First of all, our target website: Android market

Click [App] to enter our Key page:

After jumping to the application interface, we need to pay attention to three places. The red box in the picture below indicates:

First pay attention to the URL in the address bar, then pay attention to the free download button, and then pay attention to the page turning options at the bottom. Clicking the "Free Download" button will immediately download the corresponding APP, so our idea is to get the click-to-download link and download the APP directly.

Writing a crawler

The first point that needs to be solved: How do we get the download link mentioned above? Here I have to introduce the basic principles of how browsers display web pages. To put it simply, the browser is a tool similar to a parser. When it gets HTML and other codes, it will parse and render according to the corresponding rules, so that we can see the page.

I am using Google Chrome here. Right-click on the page and click "Inspect" to see the original HTML code of the webpage:

Look Don’t worry if you encounter dazzling HTML codes. Google Chrome’s review element has a useful little function that can help us locate the HTML codes corresponding to page controls.

Location:

As shown in the picture above, click the small arrow in the rectangular box above, click the corresponding position on the page, and the HTML code on the right will be automatically positioned and highlighted.

Next we locate the HTML code corresponding to the download button:

You can see that in the code corresponding to the button, there is a corresponding download link: [/ appdown/com.tecent.mm], plus the prefix, the complete download link is http://apk.hiapk.com/appdown/com.tecent.mm

First use python Getting the HTML of the entire page is very simple, just use "requests.get(url) " and fill in the corresponding URL.

Next, when crawling the key information of the page, adopt the idea of ​​​​"grab the big ones first, then the small ones". You can see that there are 10 APPs on a page, corresponding to 10 items in the HTML code:

And each li tag contains the attributes (name) of the respective APP. , download link, etc.). So in the first step, we extract these 10 li tags:

def geteveryapp(self,source):
  everyapp = re.findall(&#39;(<li class="list_item".*?</li>)&#39;,source,re.S)
  #everyapp2 = re.findall(&#39;(<p class="button_bg button_1 right_mt">.*?</p>)&#39;,everyapp,re.S)
  return everyapp

Simple regular expression knowledge is used here

Extract the download link in the li tag:

def getinfo(self,eachclass):
  info = {}
  str1 = str(re.search(&#39;<a href="(.*?)">&#39;, eachclass).group(0))
  app_url = re.search(&#39;"(.*?)"&#39;, str1).group(1)
  appdown_url = app_url.replace(&#39;appinfo&#39;, &#39;appdown&#39;)
  info[&#39;app_url&#39;] = appdown_url
  print appdown_url
  return info

The next difficulty is turning pages , after clicking the page turning button below, we can see that the address bar has changed as follows:

豁然开朗，我们可以在每次的请求中替换URL中对应的id值实现翻页。

def changepage(self,url,total_page):
  now_page = int(re.search('pi=(\d)', url).group(1))
  page_group = []
  for i in range(now_page,total_page+1):
   link = re.sub('pi=\d','pi=%s'%i,url,re.S)
   page_group.append(link)
  return page_group

爬虫效果

关键位置说完了，我们先看下最后爬虫的效果：

在TXT文件中保存结果如下：

直接复制进迅雷就可以批量高速下载了。

附上全部代码

#-*_coding:utf8-*-
import requests
import re
import sys
reload(sys)
sys.setdefaultencoding("utf-8")

class spider(object):
 def __init__(self):
  print u'开始爬取内容'
 def getsource(self,url):
  html = requests.get(url)
  return html.text

 def changepage(self,url,total_page):
  now_page = int(re.search('pi=(\d)', url).group(1))
  page_group = []
  for i in range(now_page,total_page+1):
   link = re.sub('pi=\d','pi=%s'%i,url,re.S)
   page_group.append(link)
  return page_group

 def geteveryapp(self,source):
  everyapp = re.findall('(<li class="list_item" .*?<="" li="">)',source,re.S)
  return everyapp

 def getinfo(self,eachclass):
  info = {}
  str1 = str(re.search(&#39;<a href="(.*?)">&#39;, eachclass).group(0))
  app_url = re.search(&#39;"(.*?)"&#39;, str1).group(1)
  appdown_url = app_url.replace(&#39;appinfo&#39;, &#39;appdown&#39;)
  info[&#39;app_url&#39;] = appdown_url
  print appdown_url
  return info

 def saveinfo(self,classinfo):
  f = open('info.txt','a')
  str2 = "http://apk.hiapk.com"
  for each in classinfo:
   f.write(str2)
   f.writelines(each['app_url'] + '\n')
  f.close()

if __name__ == '__main__':

 appinfo = []
 url = 'http://apk.hiapk.com/apps/MediaAndVideo?sort=5&pi=1'
 appurl = spider()
 all_links = appurl.changepage(url, 5)
 for link in all_links:
  print u'正在处理页面' + link
  html = appurl.getsource(link)
  every_app = appurl.geteveryapp(html)
  for each in every_app:
   info = appurl.getinfo(each)
   appinfo.append(info)
 appurl.saveinfo(appinfo)</li>

总结

选取的目标网页相对结构清晰简单，这是一个比较基本的爬虫。代码写的比较乱请见谅，以上就是这篇文章的全部内容了，希望能对大家的学习或者工作带来一定的帮助，如果有问题大家可以留言交流。

更多Python method to crawl APP download link相关文章请关注PHP中文网！