How Can I Efficiently Detect When a Web Page (with Infinite Scrolling) Has Fully Loaded Using Selenium WebDriver in Python?

How to Wait for Page to Load Using Selenium WebDriver for Python

When scraping data from web pages implemented with infinite scrolling, it's crucial to efficiently detect when the page has fully loaded new content. This article explores a more time-efficient approach compared to waiting a fixed time interval after each scroll.

Default Page Loading Mechanism

By default, the Selenium WebDriver's .get() method waits for the page to load completely. However, this doesn't account for loading within frames or via AJAX requests.

Using WebDriverWait for Specific Elements

To wait for a specific element to become available on the page, use WebDriverWait() with expected_conditions. For example:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

delay = 3 # seconds
driver = webdriver.Firefox()
driver.get("url")
try:
    myElem = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'id-of-element')))
    print("Page is ready!")
except TimeoutException:
    print("Loading took too much time!")

Considerations

Note that WebDriverWait only waits for the specified element or condition to become true. If waiting for additional elements or conditions, you may need to use custom code to detect page load completion.

The above is the detailed content of How Can I Efficiently Detect When a Web Page (with Infinite Scrolling) Has Fully Loaded Using Selenium WebDriver in Python?. For more information, please follow other related articles on the PHP Chinese website!