Backend Development
Python Tutorial
[Python] Web Crawler (11): Show your sword! The crawler framework Scrapy makes its debut!
[Python] Web Crawler (11): Show your sword! The crawler framework Scrapy makes its debut!
The previous ten chapters of crawler notes have successively recorded some simple Python crawler knowledge,
can be used to solve simple Tieba downloads, and the calculation of grade points is naturally a no-brainer.
But if you want to download a large amount of content in batches, such as all the questions and answers on Zhihu, it seems a bit difficult.
Ever since, the crawler framework Scrapy has appeared!
Scrapy = Scrach+Python. The word Scrach means scraping.
For the time being, we can call it: small scratch.
Xiaozhuazhao’s official website address: Click me, click me.
#Then let’s briefly demonstrate the installation process of Scrapy.
For specific procedures, please refer to: Official website tutorial
Friendly reminder: Be sure to download according to the Python version, otherwise you will be reminded that Python cannot be found during installation. It is recommended that you install 32-bit because some 64-bit versions of essential software are difficult to find.
1. Install Python (32-bit recommended)
It is recommended to install Python2.7.x, 3.x does not seem to be supported yet.
After installation, remember to configure the environment and add the python directory and the Scripts directory under the python directory to the Path of the system environment variable.
Enter python in cmd. If version information appears, the configuration is complete.
2. Install lxml
lxml is a library written in Python that can process XML quickly and flexibly. Click here to select the corresponding Python version to install.
3. Install setuptools
is used to install the egg file. Click here to download the corresponding version of setuptools for python2.7.
4. Install zope.interface
You can use the setuptools downloaded in the third step to install the egg file. There is also an exe version now. Click here to download.
5. Install Twisted
Twisted is an event-driven network engine framework implemented in Python. Click here to download.
6. Install pyOpenSSL
pyOpenSSL is Python’s OpenSSL interface, click here to download.
7. Install win32py
Provides win32api, click here to download
8 .Install Scrapy
Finally it’s time to get excited! After installing so many widgets, it’s finally the protagonist’s turn.
Directly enter easy_install scrapy in cmd and press Enter.
9. Check the installation
Open a cmd window and execute the scrapy command anywhere. If you get the following page, it means the environment configuration is successful.
![[Python] Web Crawler (11): Show your sword! The crawler framework Scrapy makes its debut!](https://img.php.cn//upload/image/570/136/515/1484981010714426.jpg "1484981010714426.jpg")
The above is [Python] Web Crawler (11): Show your sword! The crawler framework Scrapy makes its debut! For more related content, please pay attention to the PHP Chinese website (www.php.cn)!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1386
52
PHP and Python: Code Examples and Comparison
Apr 15, 2025 am 12:07 AM
PHP and Python have their own advantages and disadvantages, and the choice depends on project needs and personal preferences. 1.PHP is suitable for rapid development and maintenance of large-scale web applications. 2. Python dominates the field of data science and machine learning.
Python vs. JavaScript: Community, Libraries, and Resources
Apr 15, 2025 am 12:16 AM
Python and JavaScript have their own advantages and disadvantages in terms of community, libraries and resources. 1) The Python community is friendly and suitable for beginners, but the front-end development resources are not as rich as JavaScript. 2) Python is powerful in data science and machine learning libraries, while JavaScript is better in front-end development libraries and frameworks. 3) Both have rich learning resources, but Python is suitable for starting with official documents, while JavaScript is better with MDNWebDocs. The choice should be based on project needs and personal interests.
Detailed explanation of docker principle
Apr 14, 2025 pm 11:57 PM
Docker uses Linux kernel features to provide an efficient and isolated application running environment. Its working principle is as follows: 1. The mirror is used as a read-only template, which contains everything you need to run the application; 2. The Union File System (UnionFS) stacks multiple file systems, only storing the differences, saving space and speeding up; 3. The daemon manages the mirrors and containers, and the client uses them for interaction; 4. Namespaces and cgroups implement container isolation and resource limitations; 5. Multiple network modes support container interconnection. Only by understanding these core concepts can you better utilize Docker.
How to run programs in terminal vscode
Apr 15, 2025 pm 06:42 PM
In VS Code, you can run the program in the terminal through the following steps: Prepare the code and open the integrated terminal to ensure that the code directory is consistent with the terminal working directory. Select the run command according to the programming language (such as Python's python your_file_name.py) to check whether it runs successfully and resolve errors. Use the debugger to improve debugging efficiency.
Python: Automation, Scripting, and Task Management
Apr 16, 2025 am 12:14 AM
Python excels in automation, scripting, and task management. 1) Automation: File backup is realized through standard libraries such as os and shutil. 2) Script writing: Use the psutil library to monitor system resources. 3) Task management: Use the schedule library to schedule tasks. Python's ease of use and rich library support makes it the preferred tool in these areas.
What is vscode What is vscode for?
Apr 15, 2025 pm 06:45 PM
VS Code is the full name Visual Studio Code, which is a free and open source cross-platform code editor and development environment developed by Microsoft. It supports a wide range of programming languages and provides syntax highlighting, code automatic completion, code snippets and smart prompts to improve development efficiency. Through a rich extension ecosystem, users can add extensions to specific needs and languages, such as debuggers, code formatting tools, and Git integrations. VS Code also includes an intuitive debugger that helps quickly find and resolve bugs in your code.
Can vs code run in Windows 8
Apr 15, 2025 pm 07:24 PM
VS Code can run on Windows 8, but the experience may not be great. First make sure the system has been updated to the latest patch, then download the VS Code installation package that matches the system architecture and install it as prompted. After installation, be aware that some extensions may be incompatible with Windows 8 and need to look for alternative extensions or use newer Windows systems in a virtual machine. Install the necessary extensions to check whether they work properly. Although VS Code is feasible on Windows 8, it is recommended to upgrade to a newer Windows system for a better development experience and security.
Can visual studio code be used in python
Apr 15, 2025 pm 08:18 PM
VS Code can be used to write Python and provides many features that make it an ideal tool for developing Python applications. It allows users to: install Python extensions to get functions such as code completion, syntax highlighting, and debugging. Use the debugger to track code step by step, find and fix errors. Integrate Git for version control. Use code formatting tools to maintain code consistency. Use the Linting tool to spot potential problems ahead of time.
