Extract titles from web pages using Python

In Python, we can extract titles from web pages using web scraping. Web scraping is the process of extracting data from a website or web page. In this article, we will scrap the title of a web page using the Requests and BeautifulSoup libraries in Python.

Extract title from web page

Method 1: Using Request and Beautiful Soup libraries

We can use Python's request and Beautiful Soup libraries to extract titles from web pages. The requests library is used to send HTTP requests to websites and get their responses. We then use the response object to extract the HTML content of the web page.

Example

In the example below, we extract the title of the Wikipedia home page. We use the requests library to send a GET request to the Wikipedia page URL and store the response object in the response variable.

We can then use the Beautiful Soup object to parse the HTML content received in the response object and extract the title tag of the web page using the soup.title property. We can then extract the string attribute and store it in the title variable.

import requests
from bs4 import BeautifulSoup

url = 'https://www.wikipedia.org/'
response = requests.get(url)

soup = BeautifulSoup(response.content, 'html.parser')
title = soup.title.string

print(title)

Output

Wikipedia

Method 2: Use urllib and BeautifulSoup to extract the title

The urllib and BeautifulSoup methods are used to extract the title from a web page by opening the URL and retrieving the HTML content of the web page using the urllib library. Create a BeautifulSoup object with the HTML content, and use the 'soup.title' property to extract the page's title tag.

Example

In the following example, we use the urllib library to open a URL and retrieve the HTML content of the web page. We then use the 'html.parser' parser to create a BeautifulSoup object using the HTML content of the web page.

Then we can use the 'soup.title' attribute to extract the title tag of the web page. Finally, we use the "string" attribute to extract the string content of the title tag and store it in the "title" variable. Then we print the title of the web page to the console.

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = 'https://www.wikipedia.org/'
html_page = urlopen(url)
soup = BeautifulSoup(html_page, 'html.parser')
title = soup.title.string

print(title)

Output

Wikipedia

Method 3: Extract titles using selenium and BeautifulSoup

The selenium and BeautifulSoup methods are used to extract the title from a web page by opening the URL using the selenium library and retrieving the HTML content of the web page. Create a Chrome Webdriver and use it to navigate to web pages. Retrieve the HTML content of the web page using the webdriver's "page_source" attribute. Create a BeautifulSoup object with the HTML content, and use the 'soup.title' property to extract the page's title tag.

Example

In the following example, we use the selenium library to open a URL and retrieve the HTML content of the web page. We create a Chrome Webdriver and use it to navigate to web pages. We then retrieve the HTML content of the web page using the web driver's "page_source" attribute.

We use the "html.parser" parser to create a BeautifulSoup object using the HTML content of the web page. We can then extract the title tag of the web page using the "soup.title" property. Finally, we use the "string" attribute to extract the string content of the title tag and store it in the "title" variable. Then we print the title of the web page to the console.

from selenium import webdriver
from bs4 import BeautifulSoup

url = 'https://www.wikipedia.org/'
driver = webdriver.Chrome()
driver.get(url)

html_page = driver.page_source
soup = BeautifulSoup(html_page, 'html.parser')
title = soup.title.string

print(title)

driver.quit()

Output

Wikipedia

Method 4: Use regular expressions to extract titles

The regular expression method is used to extract headers from a web page by using the requests library to send a GET request to the URL and store the response object. The HTML content of the web page is then decoded and stored in variables. Define a regular expression pattern to match the title tag of a web page. The Search method of a regular expression pattern finds the first occurrence of the pattern in the HTML content of the web page. The group(1) method can be used to extract the string content of the first matching group to obtain the title of the web page.

Example

In the example below, we use regular expressions to extract the title of the web page. We use the requests library to send a GET request to a URL and store the response object in the "response" variable.

We then decode the HTML content of the web page using "utf-8" encoding and store it in the "html_content" variable. We define a regular expression pattern to match the title tag of a web page.

We use the "search" method of a regular expression pattern to find the first occurrence of that pattern in the HTML content of the web page. We use the "group(1)" method to extract the string content of the first matching group and store it in the "title" variable. Then we print the title of the web page to the console.

import requests

url = 'https://www.wikipedia.org/'
response = requests.get(url)
html_content = response.content.decode('utf-8')

title_pattern = re.compile('(.+?)')
match = title_pattern.search(html_content)
title = match.group(1)

print(title)

Output

Wikipedia

in conclusion

In this article, we discussed the process of extracting titles from web pages using requests and the Beautiful Soup library in Python. The requests library is used to send HTTP requests to website URLs and get HTML content in response. The Beautiful Soup library is then used to parse the HTML content and extract the required headers from the HTML content.

The above is the detailed content of Extract titles from web pages using Python. For more information, please follow other related articles on the PHP Chinese website!