How to Scrape Product Pages(Etsy, Amazon, Ebay) in Python Using Scrapy and Request Callbacks

Patricia Arquette
Release: 2024-11-06 10:58:02
Original
634 people have browsed it

How to Scrape Product Pages(Etsy, Amazon, Ebay) in Python Using Scrapy and Request Callbacks

(I'm not condoning anything illegal, this is for educational purposes only)

Introduction

Scrapy is one of the best web scraping frameworks in Python, it's easy to use, fast and packed with features.

But what if you we wanted to scrape multiple pages recursively? Such as product pages.

Well the easiest way is by adding a simple callback to a Request function.

Here’s a code snippet inside a Scrapy project crawling a website with listed products such as Amazon, eBay and Etsy:

def parse(self, response):
    links = response.css('a.s-item__link::attr(href)').getall()

    for link in links:
        yield Request(url=link, callback=self.parse_item)

    next_page = response.css('a.pagination___next.icon-link::attr(href)').get()
    if next_page:
        print('Next page: %s' % next_page)
        yield Request(url=next_page, callback=self.parse)

def parse_item(self, response):
    title = response.xpath('//h1[@class="x-item-title___mainTitle"]/span/text()').get()
    price = response.xpath('//span[@id="prcIsum"]/text()').get()

    yield {'title':title,
           'price':price}
Copy after login

How it works?

First, it grabs the links of each items listed on a products page using this line of code:

links = response.css('a.s-item__link::attr(href)').getall()
Copy after login

It then loops through each one of those links, sends a request to get the page source and does a callback to parse_item:

for link in links:
     yield Request(url=link,callback=self.parse_item)
Copy after login

Inside the parse_item function, it grabs the title and price of the item:

def parse_item(self, response):
      title = response.xpath('//h1[@class="x-item-title__mainTitle"]/span/text()').get()
      price = response.xpath('//span[@id="prcIsum"]/text()').get()

      yield {'title':title,
             'price':price}
Copy after login

Our code then grabs the link to the next page, requests it with a callback to itself (i.e the parse function) and starts all over again:

next_page = response.css('a.pagination__next.icon-link::attr(href)').get()
  if next_page:
        print('Next page:%s' % next_page)
        yield Request(url=next_page,callback=self.parse)
Copy after login

Conclusion

There you go, it's that simple!

Scraping product pages recursively with Scrapy can be implemented as easily as adding a callback to a Request function.

The above is the detailed content of How to Scrape Product Pages(Etsy, Amazon, Ebay) in Python Using Scrapy and Request Callbacks. For more information, please follow other related articles on the PHP Chinese website!

source:dev.to
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Latest Articles by Author
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template