Example of how Python3 captures html web pages dynamically generated by JS

This article mainly introduces the function of Python3 to capture HTML webpages dynamically generated by JavaScript, and analyzes the relevant operating skills of Python3 using the selenium library to capture HTML webpage elements dynamically generated by JavaScript, combined with examples. Friends who need it can Refer to the following

The example of this article describes the function of Python3 to capture html web pages dynamically generated by javascript. Share it with everyone for your reference, the details are as follows:

Using urllib to crawl web pages can only read the static source files of the web page, but cannot capture the content generated by javascript.

The reason is that urllib crawls instantly and does not wait for the loading delay of javascript, so the content generated by javascript in the page cannot be read by urllib.

Is it really impossible to read the content generated by javascript? No!

Here we will introduce a python library: selenium. The version used in this article is 2.44.0

Install first:

pip install -U selenium

The following three examples are used to illustrate its usage:

【Example 0】

Open a Firefox browser
Load the page with the given url address

from selenium import webdriver
browser = webdriver.Firefox()
browser.get('http://www.baidu.com/')

【Example 1 】

Open a Firefox browser
Load Baidu homepage
Search for "seleniumhq"
Close the browser

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser = webdriver.Firefox()
browser.get('http://www.baidu.com')
assert '百度' in browser.title
elem = browser.find_element_by_name('p') # Find the search box
elem.send_keys('seleniumhq' + Keys.RETURN) # 模拟按键
browser.quit()

[Example 2]

Selenium WebDriver is often used for testing network programs. Here is an example using the Python standard library unittest:

import unittest
class BaiduTestCase(unittest.TestCase):
  def setUp(self):
    self.browser = webdriver.Firefox()
    self.addCleanup(self.browser.quit)
  def testPageTitle(self):
    self.browser.get('http://www.baidu.com')
    self.assertIn('百度', self.browser.title)
if __name__ == '__main__':
  unittest.main(verbosity=2)

The above is the detailed content of Example of how Python3 captures html web pages dynamically generated by JS. For more information, please follow other related articles on the PHP Chinese website!