What needs to be installed for python crawler
80% of the world's crawlers are developed based on Python. Learning crawler skills can provide important data sources for subsequent big data analysis, mining, machine learning, etc.
Python crawler needs to install related libraries:
Libraries involved in python crawler:
Request library, parsing library, storage library, tool library
1. Request library: urllib/re/requests
(1) urllib/re is the library that comes with python by default and can be verified by the following command:
No error message is output, indicating that the environment is normal
(2) requests installation
2.1 Open CMD and enter
pip3 install requests
2.2 Wait for installation and verify
(3) Selenium installation (drives the browser for website access behavior)
3.1 Open CMD and enter
pip3 install selenium
3.2 Install chromedriver
Website: https://npm.taobao.org/
Decompress the downloaded compressed package and put the exe into D:\Python3.6.0\Scripts\
This path only needs to be in the PATH variable
3.3 After the installation is completed, verify
Press Enter and the chrome browser interface will pop up
3.4 Install other browsers
Interfaceless browser phantomjs
Download URL: http://phantomjs.org/
After downloading, unzip and put the entire directory Go to D:\Python3.6.0\Scripts\ and add the path to the bin directory to the PATH variable
Verification:
Open CMD
phantomjs console.log('phantomjs') CTRL+C python from selenium import webdriver driver = webdriver.PhantomJS() dirver.get('http://www.baidu.com') driver.page_source
2. Parsing library:
2.1 lxml (XPATH)
Open CMD
pip3 install lxml
or download from https://pypi.python.org, for example, lxml-4.1.1-cp36- cp36m-win_amd64.whl (md5), first download the whl file
pip3 install 文件名.whl
2.2 beautifulsoup
Open CMD, you need to install lxml
pip3 install beautifulsoup4
Verification
python from bs4 import BeautifulSoup soup = BeautifulSoup('<html></html>','lxml')
2.3 pyquery (similar to jquery syntax)
Open CMD
pip3 install pyquery
Verify the installation results
python from pyquery import PyQuery as pq doc = pq('<html>hi</html>') result = doc('html').text() result
3. Repository
3.1 pymysql (operating MySQL, relational database)
Installation:
pip3 install pymysql
Test after installation:
3.2 pymongo (operating MongoDB, key-value)
Installation
pip3 install pymongo
Verification
python
import pymongo
client = pymongo.MongoClient('localhost')
db = client['testdb']
db['table'].insert({'name':'bob'})
db['table'].find_one({'name':'bob'}) 
3.3 redis (distributed crawler, maintain crawling queue)
Installation:
pip3 install redis
Verification:
4. Tool library
4.1 flask (WEB library)
pip3 install flask
4.2 Django (distributed crawler maintenance system)
pip3 install django
4.3 jupyter (notepad running on the web page, supports markdown, and can run code on the web page)
pip3 install jupyter
Verification:
After opening CMD
jupyter notebook
, you can directly run the code on the web page Create notepad, code blocks and Markdown blocks, support printing
[Related recommendations]
1. Python crawler library and related tools
2. Python crawler introductory tutorial
The above is the detailed content of What needs to be installed for python crawler. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
Is there any mobile app that can convert XML into PDF?
Apr 02, 2025 pm 08:54 PM
An application that converts XML directly to PDF cannot be found because they are two fundamentally different formats. XML is used to store data, while PDF is used to display documents. To complete the transformation, you can use programming languages and libraries such as Python and ReportLab to parse XML data and generate PDF documents.
Is the conversion speed fast when converting XML to PDF on mobile phone?
Apr 02, 2025 pm 10:09 PM
The speed of mobile XML to PDF depends on the following factors: the complexity of XML structure. Mobile hardware configuration conversion method (library, algorithm) code quality optimization methods (select efficient libraries, optimize algorithms, cache data, and utilize multi-threading). Overall, there is no absolute answer and it needs to be optimized according to the specific situation.
How to convert XML files to PDF on your phone?
Apr 02, 2025 pm 10:12 PM
It is impossible to complete XML to PDF conversion directly on your phone with a single application. It is necessary to use cloud services, which can be achieved through two steps: 1. Convert XML to PDF in the cloud, 2. Access or download the converted PDF file on the mobile phone.
How to control the size of XML converted to images?
Apr 02, 2025 pm 07:24 PM
To generate images through XML, you need to use graph libraries (such as Pillow and JFreeChart) as bridges to generate images based on metadata (size, color) in XML. The key to controlling the size of the image is to adjust the values of the <width> and <height> tags in XML. However, in practical applications, the complexity of XML structure, the fineness of graph drawing, the speed of image generation and memory consumption, and the selection of image formats all have an impact on the generated image size. Therefore, it is necessary to have a deep understanding of XML structure, proficient in the graphics library, and consider factors such as optimization algorithms and image format selection.
How to open xml format
Apr 02, 2025 pm 09:00 PM
Use most text editors to open XML files; if you need a more intuitive tree display, you can use an XML editor, such as Oxygen XML Editor or XMLSpy; if you process XML data in a program, you need to use a programming language (such as Python) and XML libraries (such as xml.etree.ElementTree) to parse.
Recommended XML formatting tool
Apr 02, 2025 pm 09:03 PM
XML formatting tools can type code according to rules to improve readability and understanding. When selecting a tool, pay attention to customization capabilities, handling of special circumstances, performance and ease of use. Commonly used tool types include online tools, IDE plug-ins, and command-line tools.
What is the process of converting XML into images?
Apr 02, 2025 pm 08:24 PM
To convert XML images, you need to determine the XML data structure first, then select a suitable graphical library (such as Python's matplotlib) and method, select a visualization strategy based on the data structure, consider the data volume and image format, perform batch processing or use efficient libraries, and finally save it as PNG, JPEG, or SVG according to the needs.
Is there a mobile app that can convert XML into PDF?
Apr 02, 2025 pm 09:45 PM
There is no APP that can convert all XML files into PDFs because the XML structure is flexible and diverse. The core of XML to PDF is to convert the data structure into a page layout, which requires parsing XML and generating PDF. Common methods include parsing XML using Python libraries such as ElementTree and generating PDFs using ReportLab library. For complex XML, it may be necessary to use XSLT transformation structures. When optimizing performance, consider using multithreaded or multiprocesses and select the appropriate library.
