crawlee-python
Build reliable web scrapers and browser automation workflows in Python.
Extract data for AI, LLMs, RAG, and GPTs using BeautifulSoup or Playwright.
Crawlee-python is a web scraping and browser automation library designed to extract data for AI applications. It gives you a unified interface for HTTP and headless browser crawling with built-in proxy rotation.
What it is
It handles end-to-end web crawling and scraping, letting you download HTML, PDF, and image files while persistently storing tabular data. You can switch between BeautifulSoupCrawler for fast HTTP requests and PlaywrightCrawler for JavaScript-heavy pages.
Who it's for
It suits Python developers and AI engineers building data pipelines for RAG or LLMs who need resilient bot-evading scrapers. It does not suit those looking for a no-code visual scraping GUI.
Key features
Unified HTTP & Browser API
Easily switch between BeautifulSoup for speed and Playwright for JavaScript rendering using a similar interface.
Integrated Proxy Rotation
Manage sessions and rotate proxies automatically to fly under the radar of modern bot protections.
Asyncio-Based Architecture
Leverage standard Python asyncio for high-performance parallel crawling and modern integration.
State Persistence
Save scraping state during interruptions so you never have to restart pipelines from scratch.
Up and running in 3 steps
Our honest verdict
What's great
- Complete type hint coverage for better IDE autocompletion and fewer bugs
- Built-in automatic retries on errors and blocking protection
- Organized data storage options including datasets and key-value stores
- CLI tool with prepared project templates using uv or standard pip
Where it falls short
- Requires writing Python code and managing async workflows
- Playwright extra requires installing separate browser binaries
- Running heavy browser-based automation consumes significant system memory
Pricing & license
Crawlee is free under the Apache-2.0 license, and since it runs locally or on your own infrastructure, the only cost is running it yourself.
Open remote AI-training roles
880 open remote AI-training roles, updated November 15, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
Use it, or have us set it up
FAQ
Should I use BeautifulSoupCrawler or PlaywrightCrawler?
Use BeautifulSoupCrawler for high-performance static HTML scraping, and PlaywrightCrawler when you need to execute client-side JavaScript.
Can I run Crawlee-python on the Apify platform?
Yes, although it is open-source and runs anywhere, it integrates easily with the Apify cloud platform.
