Cockroach Crawler

Cockroach Crawler is an advanced open-source web crawler designed to give AI agents structured access to the public web without the need for an unrestricted browser. This TypeScript toolkit enables users to crawl, render JavaScript content, search, and normalize evidence efficiently, creating clean Markdown, JSON, and JSONL records from explicitly defined public URLs and supported sources.

Key Features

• Comprehensive Crawling: Allows for the complete crawling of websites while managing requests through browsers in a controlled manner, ensuring that the AI agent operates within specified guidelines.
• Evidence Extraction: Extracts structured data, including the ability to scrape PDFs or generate screenshots, providing a reliable method for indexing and research.
• Traverse Strategies: Supports multiple crawling methods including Breadth-First Search (BFS), Depth-First Search (DFS), best-first traversal, and adaptive strategies to maximize efficiency under predefined constraints.
• Compliant and Safe: Designed to handle only explicitly permitted public web content while maintaining necessary boundaries and compliance. This ensures that all operations respect robots.txt files and other site regulations.

Usage Scenarios

Cockroach Crawler is suitable for a variety of use cases:

• Documentation Indexing: Streamlining the process of collecting and structuring documentation data from multiple sources for easy access and reviews.
• Research and Content Inventory: Gathering and preserving public web content for analysis, audits, or quality assurance in projects.
• AI Integration: Ideal for feeding research agents and Retrieval-Augmented Generation (RAG) pipelines with high-quality, normalized data records.

Example Code Snippet

Utilizing the powerful API is straightforward:

import { crawl } from "cockroach-crawler";

const pages = await crawl({
  seeds: ["https://docs.example.com"],
  maxPages: 50,
  traversal: {
    mode: "adaptive",
    query: "authentication migration"
  }
});

This package streamlines the management of crawling tasks and enhances the capabilities of AI models without compromising web integrity or security. It competes favorably with existing crawling solutions by offering precise control over each action and a full audit trail of operations undertaken.