Introduction to the installation and use guide of the Python crawler auxiliary tool PyQuery module-Python Tutorial-php.cn

Home

Backend Development

Python Tutorial

Introduction to the installation and use guide of the Python crawler auxiliary tool PyQuery module

高洛峰

Mar 04, 2017 pm 04:04 PM

This article mainly introduces the installation and use guide of the Python crawler auxiliary tool PyQuery module. PyQuery can be easily used to parse HTML content, making it a favorite of many crawler program developers. Friends who need it can refer to it

Installation under Windows:
Download address: https://pypi.python.org/pypi/pyquery/#downloads

Download Post-installation:

C:\Python27>easy_install E:\python\pyquery-1.2.4.zip

Copy after login

You can also install directly online:

C:\Python27>easy_install pyquery

Copy after login

pyquery is a python library similar to jquery. You can use syntax like jquery to extract any data in the web page. This is used for data extraction and mining of html web pages. A very good third-party library. Let's take a look at the uses of pyquery.

Extract information from html string

#!/usr/bin/python
# -*- coding: utf-8 -*-
 
from pyquery import PyQuery as pq
html = &#39;&#39;&#39;
<html>
<head>
 <title>this is title</title>
</head>
<body>
 <p id="hi">Hello, World</p>
 <p id="hi2">Nihao</p>
 <div class="class1">
  <img src="1.jpg" />
 </div>
 <ul>
  <li>list1</li>
  <li>list2</li>
 </ul>
</body>
</html>
&#39;&#39;&#39;
d=pq(html)
 
print d(&#39;title&#39;) # 相当于css选择器，根据html标签获取元素
print d(&#39;title&#39;).text() # text()方法获取当前选中的文本块
 
print d(&#39;#hi&#39;).text() # 相当于id选择器，直接根据id名获取元素
print d(&#39;p&#39;).filter(&#39;#hi2&#39;).text() # 可以根据id或class得到指定元素
print d(&#39;.class1&#39;) # 相当于class选择器
print d(&#39;.class1&#39;).html() # html()方法获取当前选中的html块
print d(&#39;.class1&#39;).find(&#39;img&#39;).attr(&#39;src&#39;) # 查找嵌套元素，并选中属性
print d(&#39;ul&#39;).find(&#39;li&#39;).eq(0).text() # 根据索引号获取多个相同html元素中的某一个
print d(&#39;ul&#39;).children() # 获取所有子元素
print d(&#39;ul&#39;).children().eq(0) #根据索引获取子元素
print d(&#39;img&#39;).parents() # 获取父元素
print d(&#39;#hi&#39;).next() # 获取下一个元素
print d(&#39;#hi&#39;).nextAll() #获取后面全部元素块
print d(&#39;p&#39;).not_(&#39;#hi2&#39;) # 返回不匹配选择器的元素
# 遍历所有匹配的元素
for i in d.items(&#39;li&#39;):
 print i.text()
print [i.text() for i in d.items(&#39;li&#39;)] # 遍历用于列表推倒
print d.make_links_absolute(base_url=&#39;http://www.baidu.com&#39;) # 把html文档中的相对路径变为绝对路径

Copy after login

The above code snippet gives Learn the commonly used operating methods of pyquery. We first defined a piece of HTML code, and then used a series of methods of pyquery to operate on the HTML code, mainly to obtain specific elements and text. Of course, pyquery can not only obtain elements, but also set element attributes, add elements and other functions. Since the most commonly used method is the method used in the above code, other methods will not be introduced here.

Extract information from url or local html file

Of course, pyquery can not only parse html strings like the above, but also like this:

d = pq(url=&#39;http://www.baidu.com/&#39;)

Copy after login

We can load a URL directly, there is no difference from the above operation method. This method uses the urllib module to make http requests by default, but if requests are installed in your system, requests will be used to make http requests, which means you can use any parameters of requests, such as:

pq(&#39;http://www.baidu.com/&#39;, headers={&#39;user-agent&#39;: &#39;pyquery&#39;})

Copy after login

Or, if you already have the corresponding html file in your local area, you can also do this:

d = pq(filename=path_to_html_file)

Copy after login

The above writing method directly specifies the local html file, and the operation method is still the same as above. same.
As you can see, pyquery provides us with full convenience to select any element, just like jquery.

Use pyquery to grab the top 250 Douban movies

After reading the syntax of pyquery, let’s look at an example to grab the top 250 Douban movies.
Because Douban’s anti-crawler is very powerful, I couldn’t catch it after running it a few times. I had to use requests to download the page first, and directly use pyquery to analyze the page to extract the information:

from pyquery import PyQuery as pq
import requests
 
head_req = {
 &#39;User-Agent&#39;:&#39;Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36&#39;,
 &#39;Referer&#39;:&#39;https://movie.douban.com/top250?start=0&#39;,
 
}
r=requests.get("https://movie.douban.com/top250?start=0",headers=head_req)
with open("1.html","wb") as html:
 html.write(r.content)
 
d=pq(filename="1.html")
 
# print d(&#39;ol&#39;).find(&#39;li&#39;).html()
for data in d(&#39;ol&#39;).items(&#39;li&#39;):
 print data.find(&#39;.hd&#39;).find(&#39;.title&#39;).eq(0).text()
 print data.find(&#39;.star&#39;).find(&#39;.rating_num&#39;).text()
 print data.find(&#39;.quote&#39;).find(&#39;.inq&#39;).text()
 print

Copy after login

Run it and see the result:

肖申克的救赎
9.6
希望让人自由。

这个杀手不太冷
9.4
怪蜀黍和小萝莉不得不说的故事。

阿甘正传
9.4
一部美国近现代史。

霸王别姬
9.4
风华绝代。

美丽人生
9.5
最美的谎言。

千与千寻
9.2
最好的宫崎骏，最好的久石让。

辛德勒的名单
9.4
拯救一个人，就是拯救整个世界。

海上钢琴师
9.2
每个人都要走一条自己坚定了的路，就算是粉身碎骨。

机器人总动员
9.3
小瓦力，大人生。

盗梦空间
9.2
诺兰给了我们一场无法盗取的梦。

泰坦尼克号
9.1
失去的才是永恒的。

三傻大闹宝莱坞
9.1
英俊版憨豆，高情商版谢耳朵。

放牛班的春天
9.2
天籁一般的童声，是最接近上帝的存在。

忠犬八公的故事
9.2
永远都不能忘记你所爱的人。

龙猫
9.1
人人心中都有个龙猫，童年就永远不会消失。

大话西游之大圣娶亲
9.1
一生所爱。

教父
9.2
千万不要记恨你的对手，这样会让你失去理智。

乱世佳人
9.2
Tomorrow is another day.

天堂电影院
9.1
那些吻戏，那些青春，都在影院的黑暗里被泪水冲刷得无比清晰。

当幸福来敲门
8.9
平民励志片。

搏击俱乐部
9.0
邪恶与平庸蛰伏于同一个母体，在特定的时间互相对峙。

楚门的世界
9.0
如果再也不能见到你，祝你早安，午安，晚安。

触不可及
9.1
满满温情的高雅喜剧。

指环王3：王者无敌
9.1
史诗的终章。

罗马假日
8.9
爱情哪怕只有一天。

Copy after login

Of course, this is only the 25 items on the first page. We already know the URL of the top 250 Douban movies.

https://movie.douban.com/top250?start=0
The start parameter starts from 0 and increases by 25 each time until

https://movie. douban.com/top250?start=225
So you can write a loop to catch them all.

For more related articles on the installation and use of the Python crawler auxiliary tool PyQuery module, please pay attention to the PHP Chinese website!

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)

2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hello Kitty Island Adventure: How To Get Giant Seeds

1 months ago By 尊渡假赌尊渡假赌尊渡假赌

How Long Does It Take To Beat Split Fiction?

4 weeks ago By DDD

R.E.P.O. Save File Location: Where Is It & How to Protect It?

4 weeks ago By DDD

Two Point Museum: All Exhibits And Where To Find Them

1 months ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Where is the login entrance for gmail email?

7375

Java Tutorial

1628

CakePHP Tutorial

1355

Laravel Tutorial

1267

PHP Tutorial

1216

Related knowledge

How to Use Python to Find the Zipf Distribution of a Text File Mar 05, 2025 am 09:58 AM

This tutorial demonstrates how to use Python to process the statistical concept of Zipf's law and demonstrates the efficiency of Python's reading and sorting large text files when processing the law. You may be wondering what the term Zipf distribution means. To understand this term, we first need to define Zipf's law. Don't worry, I'll try to simplify the instructions. Zipf's Law Zipf's law simply means: in a large natural language corpus, the most frequently occurring words appear about twice as frequently as the second frequent words, three times as the third frequent words, four times as the fourth frequent words, and so on. Let's look at an example. If you look at the Brown corpus in American English, you will notice that the most frequent word is "th

How Do I Use Beautiful Soup to Parse HTML? Mar 10, 2025 pm 06:54 PM

This article explains how to use Beautiful Soup, a Python library, to parse HTML. It details common methods like find(), find_all(), select(), and get_text() for data extraction, handling of diverse HTML structures and errors, and alternatives (Sel

How to Perform Deep Learning with TensorFlow or PyTorch? Mar 10, 2025 pm 06:52 PM

This article compares TensorFlow and PyTorch for deep learning. It details the steps involved: data preparation, model building, training, evaluation, and deployment. Key differences between the frameworks, particularly regarding computational grap

Mathematical Modules in Python: Statistics Mar 09, 2025 am 11:40 AM

Python's statistics module provides powerful data statistical analysis capabilities to help us quickly understand the overall characteristics of data, such as biostatistics and business analysis. Instead of looking at data points one by one, just look at statistics such as mean or variance to discover trends and features in the original data that may be ignored, and compare large datasets more easily and effectively. This tutorial will explain how to calculate the mean and measure the degree of dispersion of the dataset. Unless otherwise stated, all functions in this module support the calculation of the mean() function instead of simply summing the average. Floating point numbers can also be used. import random import statistics from fracti

Serialization and Deserialization of Python Objects: Part 1 Mar 08, 2025 am 09:39 AM

Serialization and deserialization of Python objects are key aspects of any non-trivial program. If you save something to a Python file, you do object serialization and deserialization if you read the configuration file, or if you respond to an HTTP request. In a sense, serialization and deserialization are the most boring things in the world. Who cares about all these formats and protocols? You want to persist or stream some Python objects and retrieve them in full at a later time. This is a great way to see the world on a conceptual level. However, on a practical level, the serialization scheme, format or protocol you choose may determine the speed, security, freedom of maintenance status, and other aspects of the program

What are some popular Python libraries and their uses? Mar 21, 2025 pm 06:46 PM

The article discusses popular Python libraries like NumPy, Pandas, Matplotlib, Scikit-learn, TensorFlow, Django, Flask, and Requests, detailing their uses in scientific computing, data analysis, visualization, machine learning, web development, and H

How to Create Command-Line Interfaces (CLIs) with Python? Mar 10, 2025 pm 06:48 PM

This article guides Python developers on building command-line interfaces (CLIs). It details using libraries like typer, click, and argparse, emphasizing input/output handling, and promoting user-friendly design patterns for improved CLI usability.

Scraping Webpages in Python With Beautiful Soup: Search and DOM Modification Mar 08, 2025 am 10:36 AM

This tutorial builds upon the previous introduction to Beautiful Soup, focusing on DOM manipulation beyond simple tree navigation. We'll explore efficient search methods and techniques for modifying HTML structure. One common DOM search method is ex

See all articles