Web scraping and browser automation library for Node.js and TypeScript.
Build robust crawlers with integrated proxy rotation and anti-bot evasion.
Crawlee is an end-to-end web scraping and browser automation library for JavaScript and TypeScript. It helps you extract clean data for AI, LLMs, and RAG pipelines while bypassing modern bot protections. Professionals building automated data collection workflows will find it exceptionally capable.
The library provides a unified interface for both raw HTTP crawling and full headless browser automation using Playwright or Puppeteer. It features automatic session management, proxy rotation, fingerprint generation, and persistent request queues. You can easily save extracted data to disk or cloud storage in tabular or file formats.
It suits developers and data engineers building production-grade web scrapers in Node.js who need to bypass bot detection. It does not suit teams looking for a zero-code GUI tool or those working primarily outside of the JavaScript and TypeScript ecosystem.
Switch seamlessly between HTTP requests and headless browser automation using the same API.
Manage sessions and rotate proxies natively to avoid getting blocked by target websites.
Generate browser-like headers and TLS fingerprints automatically to bypass modern bot protections.
Store scraped tabular data and downloaded files locally or configure cloud storage backends.
Crawlee is free under its open-source license. The only cost is running it yourself on your infrastructure or local machine, plus any optional proxy services you choose to integrate.
880 open remote AI-training roles, updated November 15, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
The primary Node.js library is JavaScript and TypeScript, but the maintainers also provide a separate Crawlee for Python repository.
No, you need to install browser automation libraries like Playwright or Puppeteer separately to keep installation sizes smaller.
By default, Crawlee stores scraped datasets, key-value pairs, and request queues locally in a `./storage` directory.
See Zekai first in Google