Python developers know the drill: you need reliable company data, and Crunchbase has it. This guide shows you how to build an effective Crunchbase scraper in Python that gets you the data you need.
Crunchbase tracks details that matter: locations, business focus, founders, and investment histories. Manual extraction from such a large dataset isn't practical -automation is essential for transforming this information into an analyzable format.
By the end of this blog, we'll explore three different ways to extract data from Crunchbase using Crawlee for Python. We'll fully implement two of them and discuss the specifics and challenges of the third. This will help us better understand how important it is to properly choose the right data source.
Note: This guide comes from a developer in our growing community. Have you built interesting projects with Crawlee? Join us on Discord to share your experiences and blog ideas - we value these contributions from developers like you.
Key steps we'll cover:
Before we start scraping, we need to set up our project. In this guide, we won't be using crawler templates (Playwright and Beautifulsoup), so we'll set up the project manually.
Install Poetry
pipx install poetry
Create and navigate to the project folder.
mkdir crunchbase-crawlee && cd crunchbase-crawlee
Initialize the project using Poetry, leaving all fields empty.
poetry init
When prompted:
Add and install Crawlee with necessary dependencies to your project using Poetry.
poetry add crawlee[parsel,curl-impersonate]
Complete the project setup by creating the standard file structure for Crawlee for Python projects.
mkdir crunchbase-crawlee && touch crunchbase-crawlee/{__init__.py,__main__.py,main.py,routes.py}
After setting up the basic project structure, we can explore different methods of obtaining data from Crunchbase.
While we can extract target data directly from the company page, we need to choose the best way to navigate the site.
A careful examination of Crunchbase's structure shows that we have three main options for obtaining data:
Let's examine each of these approaches in detail.
Sitemap is a standard way of site navigation used by crawlers like Google, Ahrefs, and other search engines. All crawlers must follow the rules described in robots.txt.
Let's look at the structure of Crunchbase's Sitemap:
As you can see, links to organization pages are located inside second-level Sitemap files, which are compressed using gzip.
The structure of one of these files looks like this:
The lastmod field is particularly important here. It allows tracking which companies have updated their information since the previous data collection. This is especially useful for regular data updates.
To work with the site, we'll use CurlImpersonateHttpClient, which impersonates a Safari browser. While this choice might seem unexpected for working with a sitemap, it's necessitated by Crunchbase's protection features.
The reason is that Crunchbase uses Cloudflare to protect against automated access. This is clearly visible when analyzing traffic on a company page:
An interesting feature is that challenges.cloudflare is executed after loading the document with data. This means we receive the data first, and only then JavaScript checks if we're a bot. If our HTTP client's fingerprint is sufficiently similar to a real browser, we'll successfully receive the data.
Cloudflare also analyzes traffic at the sitemap level. If our crawler doesn't look legitimate, access will be blocked. That's why we impersonate a real browser.
To prevent blocks due to overly aggressive crawling, we'll configure ConcurrencySettings.
When scaling this approach, you'll likely need proxies. Detailed information about proxy setup can be found in the documentation.
We'll save our scraping results in JSON format. Here's how the basic crawler configuration looks:
pipx install poetry
Sitemap navigation happens in two stages. In the first stage, we need to get a list of all files containing organization information:
pipx install poetry
In the second stage, we process second-level sitemap files stored in gzip format. This requires a special approach as the data needs to be decompressed first:
mkdir crunchbase-crawlee && cd crunchbase-crawlee
Each company page contains a large amount of information. For demonstration purposes, we'll focus on the main fields: Company Name, Short Description, Website, and Location.
One of Crunchbase's advantages is that all data is stored in JSON format within the page:
This significantly simplifies data extraction - we only need to use one Xpath selector to get the JSON, and then apply jmespath to extract the needed fields:
poetry init
The collected data is saved in Crawlee for Python's internal storage using the context.push_data method. When the crawler finishes, we export all collected data to a JSON file:
poetry add crawlee[parsel,curl-impersonate]
With all components in place, we need to create an entry point for our crawler:
mkdir crunchbase-crawlee && touch crunchbase-crawlee/{__init__.py,__main__.py,main.py,routes.py}
Execute the crawler using Poetry:
# main.py
from crawlee import ConcurrencySettings, HttpHeaders
from crawlee.crawlers import ParselCrawler
from crawlee.http_clients import CurlImpersonateHttpClient
from .routes import router
async def main() -> None:
"""The crawler entry point."""
concurrency_settings = ConcurrencySettings(max_concurrency=1, max_tasks_per_minute=50)
http_client = CurlImpersonateHttpClient(
impersonate='safari17_0',
headers=HttpHeaders(
{
'accept-language': 'en',
'accept-encoding': 'gzip, deflate, br, zstd',
}
),
)
crawler = ParselCrawler(
request_handler=router,
max_request_retries=1,
concurrency_settings=concurrency_settings,
http_client=http_client,
max_requests_per_crawl=30,
)
await crawler.run(['https://www.crunchbase.com/www-sitemaps/sitemap-index.xml'])
await crawler.export_data_json('crunchbase_data.json')
<h3>
5. Finally, characteristics of using the sitemap crawler
</h3>
<p>The sitemap approach has its distinct advantages and limitations. It's ideal in the following cases:</p>
<ul>
<li>When you need to collect data about all companies on the platform</li>
<li>When there are no specific company selection criteria</li>
<li>If you have sufficient time and computational resources</li>
</ul>
<p>However, there are significant limitations to consider:</p>
<ul>
<li>Almost no ability to filter data during collection</li>
<li>Requires constant monitoring of Cloudflare blocks</li>
<li>Scaling the solution requires proxy servers, which increases project costs</li>
</ul>
<h2>
Using search for scraping Crunchbase
</h2>
<p>The limitations of the sitemap approach might point to search as the next solution. However, Crunchbase applies tighter security measures to its search functionality compared to its public pages.</p>
<p>The key difference lies in how Cloudflare protection works. While we receive data before the challenges.cloudflare check when accessing a company page, the search API requires valid cookies that have passed this check.</p>
<p>Let's verify this in practice. Open the following link in Incognito mode:<br>
</p>
<pre class="brush:php;toolbar:false"># routes.py
from crawlee.crawlers import ParselCrawlingContext
from crawlee.router import Router
from crawlee import Request
router = Router[ParselCrawlingContext]()
@router.default_handler
async def default_handler(context: ParselCrawlingContext) -> None:
"""Default request handler."""
context.log.info(f'default_handler processing {context.request} ...')
requests = [
Request.from_url(url, label='sitemap')
for url in context.selector.xpath('//loc[contains(., "sitemap-organizations")]/text()').getall()
]
# Since this is a tutorial, I don't want to upload more than one sitemap link
await context.add_requests(requests, limit=1)
When analyzing the traffic, we'll see the following pattern:
The sequence of events here is:
Automating this process would require a headless browser capable of bypassing Cloudflare Turnstile. The current version of Crawlee for Python (v0.5.0) doesn't provide this functionality, although it's planned for future development.
You can extend the capabilities of Crawlee for Python by integrating Camoufox following this example.
Crunchbase provides a free API with basic functionality. Paid subscription users get expanded data access. Complete documentation for available endpoints can be found in the official API specification.
To start working with the API, follow these steps:
Although the documentation states that key activation may take up to an hour, it usually starts working immediately after creation.
An important API feature is the limit - no more than 200 requests per minute, but in the free version, this number is significantly lower. Taking this into account, let's configure ConcurrencySettings. Since we're working with the official API, we don't need to mask our HTTP client. We'll use the standard 'HttpxHttpClient' with preset headers.
First, let's save the API key in an environment variable:
pipx install poetry
Here's how the crawler configuration for working with the API looks:
mkdir crunchbase-crawlee && cd crunchbase-crawlee
For working with the API, we'll need two main endpoints:
First, let's implement search results processing:
poetry init
After getting the list of companies, we extract detailed information about each one:
poetry add crawlee[parsel,curl-impersonate]
If you need more flexible search capabilities, the API provides a special search endpoint. Here's an example of searching for all companies in Prague:
mkdir crunchbase-crawlee && touch crunchbase-crawlee/{__init__.py,__main__.py,main.py,routes.py}
For processing search results and pagination, we use the following handler:
# main.py
from crawlee import ConcurrencySettings, HttpHeaders
from crawlee.crawlers import ParselCrawler
from crawlee.http_clients import CurlImpersonateHttpClient
from .routes import router
async def main() -> None:
"""The crawler entry point."""
concurrency_settings = ConcurrencySettings(max_concurrency=1, max_tasks_per_minute=50)
http_client = CurlImpersonateHttpClient(
impersonate='safari17_0',
headers=HttpHeaders(
{
'accept-language': 'en',
'accept-encoding': 'gzip, deflate, br, zstd',
}
),
)
crawler = ParselCrawler(
request_handler=router,
max_request_retries=1,
concurrency_settings=concurrency_settings,
http_client=http_client,
max_requests_per_crawl=30,
)
await crawler.run(['https://www.crunchbase.com/www-sitemaps/sitemap-index.xml'])
await crawler.export_data_json('crunchbase_data.json')
The free version of the API has significant limitations:
Consider a paid subscription for production-level work. The API provides the most reliable way to access Crunchbase data, even with its rate constraints.
We've explored three different approaches to obtaining data from Crunchbase:
Each method has its advantages, but for most projects, I recommend using the official API despite its limitations in the free version.
The complete source code is available in my repository. Have questions or want to discuss implementation details? Join our Discord - our community of developers is there to help.
The above is the detailed content of How to scrape Crunchbase using Python in Easy Guide). For more information, please follow other related articles on the PHP Chinese website!
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
