Introducing three methods of python data capture

Free learning recommendation: python video tutorial

Three methods of data capture

1. Regular expression (re library)
2. BeautifulSoup (bs4)
3. lxml

*Use the previously built download web page function to obtain the html of the target web page. We take https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/ as an example to obtain the html.

from get_html import download

url = 'https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/'page_content = download(url)

* Suppose we need to crawl the country name and profile in this web page, we use these three data crawling methods in turn to achieve data crawling.
1. Regular expression

from get_html import downloadimport re

url = 'https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/'page_content = download(url)country = re.findall('class="h2dabiaoti">(.*?)', page_content) #注意返回的是listsurvey_data = re.findall('<tr><td>(.*?)</td></tr>', page_content)survey_info_list = re.findall('<p>　　(.*?)</p>', survey_data[0])survey_info = ''.join(survey_info_list)print(country[0],survey_info)

2.BeautifulSoup(bs4)

from get_html import downloadfrom bs4 import BeautifulSoup

url = 'https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/'html = download(url)#创建 beautifulsoup 对象soup = BeautifulSoup(html,"html.parser")#搜索country = soup.find(attrs={'class':'h2dabiaoti'}).text
survey_info = soup.find(attrs={'id':'wzneirong'}).textprint(country,survey_info)

3.lxml

from get_html import downloadfrom lxml import etree #解析树url = 'https://guojiadiqu.bmcx.com/AFG__guojiayudiqu/'page_content = download(url)selector = etree.HTML(page_content)#可进行xpath解析country_select = selector.xpath('//*[@id="main_content"]/h2') #返回列表for country in country_select:
    print(country.text)survey_select = selector.xpath('//*[@id="wzneirong"]/p')for survey_content in survey_select:
    print(survey_content.text,end='')

Run result:

Finally, quote the performance comparison of the three methods in "Writing a Web Crawler with Python", as shown below:

For reference only.

Related free learning recommendations: python tutorial(Video)

The above is the detailed content of Introducing three methods of python data capture. For more information, please follow other related articles on the PHP Chinese website!