AI tools, scored for your job
Get paid to train AI
🤝 You're in the Free & Open Source library — community-built tools you can use without paying. A different concept from our scored AI directory.
Home › Free & Open Source › Pro Services › crawlee
✦ Free & open sourcePro Services

crawlee

Web scraping and browser automation library for Node.js and TypeScript.

Reliable web scraping and browser automation

Build robust crawlers with integrated proxy rotation and anti-bot evasion.

Free forever · open source · Apache-2.0
✓ Light setup✓ Browser-based✓ Real-time collaboration
7.9
ZEKAI
Essential Node.js scraper
Free & open source · Pro Services
Ease of setup4
Usefulness9
Maintenance9.5
Community9.2
★ 26,090
GITHUB
Oct 2026
UPDATED
Apache-2.0
LICENSE
🛠 Don't want to self-host? We can set it up →
⚡ Quick answer

Crawlee is an end-to-end web scraping and browser automation library for JavaScript and TypeScript. It helps you extract clean data for AI, LLMs, and RAG pipelines while bypassing modern bot protections. Professionals building automated data collection workflows will find it exceptionally capable.

Runs on
Cross-platform
Hosting
Self-host
Languages
TypeScript
License
Apache-2.0

What it is

The library provides a unified interface for both raw HTTP crawling and full headless browser automation using Playwright or Puppeteer. It features automatic session management, proxy rotation, fingerprint generation, and persistent request queues. You can easily save extracted data to disk or cloud storage in tabular or file formats.

Who it's for

It suits developers and data engineers building production-grade web scrapers in Node.js who need to bypass bot detection. It does not suit teams looking for a zero-code GUI tool or those working primarily outside of the JavaScript and TypeScript ecosystem.

Key features

Unified Crawling Interface

Switch seamlessly between HTTP requests and headless browser automation using the same API.

Automatic Proxy Rotation

Manage sessions and rotate proxies natively to avoid getting blocked by target websites.

Human-Like Fingerprints

Generate browser-like headers and TLS fingerprints automatically to bypass modern bot protections.

Pluggable Storage

Store scraped tabular data and downloaded files locally or configure cloud storage backends.

How to start

Up and running in 3 steps

1
Run `npx crawlee create my-crawler` via the CLI to generate a new project with boilerplate code.
2
Install required dependencies such as Playwright if you are using browser-based crawling.
3
Implement your custom crawling logic inside the request handler and start the scraper.
Real talk

Our honest verdict

What's great

  • Supports both Cheerio/JSDOM for fast HTTP scraping and Playwright/Puppeteer for full browser rendering.
  • Built-in automatic proxy rotation and session management handle rate limits gracefully.
  • Generates human-like browser fingerprints out of the box to evade strict bot detection.
  • Written fully in TypeScript with strong generics for type-safe data extraction.

Where it falls short

  • Requires Node.js 22.13 or higher to run properly.
  • Heavy browser automation via Playwright or Puppeteer requires significant RAM and CPU resources at scale.
  • You must write and maintain your own JavaScript or TypeScript extraction logic.
Cost

Pricing & license

Crawlee is free under its open-source license. The only cost is running it yourself on your infrastructure or local machine, plus any optional proxy services you choose to integrate.

Paid work

Open remote AI-training roles

880 open remote AI-training roles, updated November 15, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.

See all open roles →
Compare

Alternatives

Get more

Use it, or have us set it up

📄 Free PDF: crawlee for Pro Services
A plain-language guide to getting your first result. Sent to your inbox.
🔔 64 people want a full video tutorial — add your vote
🛠 Don't want to set it up yourself?
We'll install, host & configure crawlee for you — and train your team.
Get it done for you ↗
Questions

FAQ

Can I use Crawlee with Python?

The primary Node.js library is JavaScript and TypeScript, but the maintainers also provide a separate Crawlee for Python repository.

Does Crawlee come with browsers pre-installed?

No, you need to install browser automation libraries like Playwright or Puppeteer separately to keep installation sizes smaller.

How does Crawlee handle data storage by default?

By default, Crawlee stores scraped datasets, key-value pairs, and request queues locally in a `./storage` directory.

Radar

Latest AI news

✉️ The weekly free toolkit
The best new free & open-source tools for Pro Services — one short email a week, in your language.
Free · 1 email/week · unsubscribe anytime
Get paid to train AI →

See Zekai first in Google