Modern Web Scraping With Beautiful Soup and Selenium

Web scraping: efficiently extracting data from web pages

This tutorial explains how web page content is rendered and how to scrape it using Python, requests, Beautiful Soup, and Selenium. We'll focus on scraping dynamic content, specifically comments.

When is web scraping necessary?

Web scraping automatically retrieves, parses, and extracts information from web pages—often those designed for human interaction. It's a last resort when APIs aren't available. Consider these drawbacks:

1. Fragility: Web pages change frequently, breaking scrapers.
2. Restrictions: Many websites prohibit scraping.
3. Performance: Scraping large volumes of data can be slow and expensive.

Understanding modern web pages

Let's examine the structure of typical web applications. We'll use the "Introduction to Vagrant" article as an example. To scrape content, we must first locate the relevant HTML elements.

Viewing page source

Browsers allow viewing the HTML source. The "Introduction to Vagrant" source reveals a significant portion of minified JavaScript unrelated to the article content itself. A small excerpt is shown below:

Here's a sample of the actual HTML:

Static vs. dynamic scraping

Static scraping ignores JavaScript, fetching the raw server-side HTML. This works if the target content is directly in the source. However, for content generated by JavaScript (dynamic content), this method fails. Dynamic scraping uses tools like Selenium to interact with the page as a browser would, rendering the JavaScript and making the dynamic content available.

Scraping dynamic comments with Selenium

Let's scrape comments from a website (example: Codecanyon comments). We'll use Selenium to navigate to the comments URL:

from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://codecanyon.net/item/whatshelp-whatsapp-help-and-support-plugin-for-javascript/42202303/comments')

Locating the comments element requires inspecting the page (right-click, "Inspect"). Selenium's WebDriverWait helps handle asynchronous loading:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
url = 'http://www.c2.com/loading-page'
driver.get(url)

element = WebDriverWait(driver, 5).until(
    EC.presence_of_element_located((By.ID, "loaded_element"))
)

Conclusion

Web scraping is valuable when necessary data isn't readily available via APIs. While challenging with modern web applications, tools like requests, Beautiful Soup, and Selenium simplify the process.

This tutorial incorporates contributions from Esther Vaati, a software developer and writer for Envato Tuts .

The above is the detailed content of Modern Web Scraping With Beautiful Soup and Selenium. For more information, please follow other related articles on the PHP Chinese website!