Building an Async E-Commerce Web Scraper with Pydantic, Crawl & Gemini

In short: This guide demonstrates building an e-commerce scraper using crawl4ai's AI-powered extraction and Pydantic data models. The scraper asynchronously retrieves both product listings (names, prices) and detailed product information (specifications, reviews).

Access the complete code on Google Colab

---

Tired of the complexities of traditional web scraping for e-commerce data analysis? This tutorial simplifies the process using modern Python tools. We'll leverage crawl4ai for intelligent data extraction and Pydantic for robust data modeling and validation.

Why Choose Crawl4AI and Pydantic?

• crawl4ai: Streamlines web crawling and scraping using AI-driven extraction methods.
• Pydantic: Provides data validation and schema management, ensuring structured and accurate scraped data.

Why Target Tokopedia?

Tokopedia, a major Indonesian e-commerce platform, serves as our example. (Note: The author is Indonesian and a user of the platform, but not affiliated.) The principles apply to other e-commerce sites. This scraping approach is beneficial for developers interested in e-commerce analytics, market research, or automated data collection.

What Sets This Approach Apart?

Instead of relying on complex CSS selectors or XPath, we utilize crawl4ai's LLM-based extraction. This offers:

• Enhanced resilience to website structure changes.
• Cleaner, more structured data output.
• Reduced maintenance overhead.

Setting Up Your Development Environment

Begin by installing necessary packages:

<code class="language-bash">%pip install -U crawl4ai
%pip install nest_asyncio
%pip install pydantic</code>

For asynchronous code execution in notebooks, we'll also use nest_asyncio:

<code class="language-python">import crawl4ai
import asyncio
import nest_asyncio
nest_asyncio.apply()</code>

Defining Data Models with Pydantic

We use Pydantic to define the expected data structure. Here are the models:

<code class="language-python">from pydantic import BaseModel, Field
from typing import List, Optional

class TokopediaListingItem(BaseModel):
    product_name: str = Field(..., description="Product name from listing.")
    product_url: str = Field(..., description="URL to product detail page.")
    price: str = Field(None, description="Price displayed in listing.")
    store_name: str = Field(None, description="Store name from listing.")
    rating: str = Field(None, description="Rating (1-5 scale) from listing.")
    image_url: str = Field(None, description="Primary image URL from listing.")

class TokopediaProductDetail(BaseModel):
    product_name: str = Field(..., description="Product name from detail page.")
    all_images: List[str] = Field(default_factory=list, description="List of all product image URLs.")
    specs: str = Field(None, description="Technical specifications or short info.")
    description: str = Field(None, description="Long product description.")
    variants: List[str] = Field(default_factory=list, description="List of variants or color options.")
    satisfaction_percentage: Optional[str] = Field(None, description="Customer satisfaction percentage.")
    total_ratings: Optional[str] = Field(None, description="Total number of ratings.")
    total_reviews: Optional[str] = Field(None, description="Total number of reviews.")
    stock: Optional[str] = Field(None, description="Stock availability.")</code>

These models serve as templates, ensuring data validation and providing clear documentation.

The Scraping Process

The scraper operates in two phases:

1. Crawling Product Listings

First, we retrieve search results pages:

<code class="language-python">async def crawl_tokopedia_listings(query: str = "mouse-wireless", max_pages: int = 1):
    # ... (Code remains the same) ...</code>

2. Fetching Product Details

Next, for each product URL, we fetch detailed information:

<code class="language-python">async def crawl_tokopedia_detail(product_url: str):
    # ... (Code remains the same) ...</code>

Combining the Stages

Finally, we integrate both phases:

<code class="language-python">async def run_full_scrape(query="mouse-wireless", max_pages=2, limit=15):
    # ... (Code remains the same) ...</code>

Running the Scraper

Here's how to execute the scraper:

<code class="language-bash">%pip install -U crawl4ai
%pip install nest_asyncio
%pip install pydantic</code>

Pro Tips

1. Rate Limiting: Respect Tokopedia's servers; introduce delays between requests for large-scale scraping.
2. Caching: Enable crawl4ai's caching during development (cache_mode=CacheMode.ENABLED).
3. Error Handling: Implement comprehensive error handling and retry mechanisms for production use.
4. API Keys: Store Gemini API keys securely in environment variables, not directly in the code.

Next Steps

This scraper can be extended to:

• Store data in a database.
• Monitor price changes over time.
• Analyze product trends and patterns.
• Compare prices across multiple stores.

Conclusion

crawl4ai's LLM-based extraction significantly improves web scraping maintainability compared to traditional methods. The integration with Pydantic ensures data accuracy and structure.

Always adhere to a website's robots.txt and terms of service before scraping.

---

Important Links:

Crawl4AI

• Official Website: https://www.php.cn/link/1026d8c97a822ee171c6cbf939fe4aca
• GitHub Repository: https://www.php.cn/link/62c1b075041300455ec2b54495d93c99
• Documentation: https://www.php.cn/link/1026d8c97a822ee171c6cbf939fe4aca/mkdocs/core/installation/

Pydantic

• Official Documentation: https://www.php.cn/link/a4d4ec4aa3c45731396ed6e65fee40b9
• PyPI Page: https://www.php.cn/link/4d8ab89733dd9a88f1a9d130ca675c2e
• GitHub Repository: https://www.php.cn/link/22935fba49f7d80d5adf1cfa6b0344f4

---

Note: The complete code is available in the Colab notebook. Feel free to experiment and adapt it to your specific needs.

The above is the detailed content of Building an Async E-Commerce Web Scraper with Pydantic, Crawl & Gemini. For more information, please follow other related articles on the PHP Chinese website!