jevcrawl

jevcrawl: Smart Web Crawler for AI Agents

jevcrawl is a powerful web crawler designed specifically to enhance AI agents' ability to find relevant answers quickly and cost-effectively. Unlike traditional web crawlers that indiscriminately fetch every page, jevcrawl intelligently selects links and passages, ensuring that your AI's language model (LLM) interacts with only the content needed to provide accurate answers. This targeted approach not only improves efficiency but also significantly reduces costs per query.

Key Benefits

• Enhanced Answer Discovery: Achieves a 54% success rate in locating answer pages, outperforming standard LLM approaches which hover around 33% accuracy. It is also 2x more effective than similar tools like Crawl4AI.
• Cost Efficiency: Operating at approximately $0.025 per task, jevcrawl is up to 10x cheaper per correct answer when compared to other methods.
• Improved Speed: Capable of processing four pages concurrently, enabling faster results and cutting down response times.
• Accuracy with Source Integrity: Returns verbatim passages along with their source URLs, ensuring that the responses are reliable and verifiable.
• Safety Features: Designed to be secure against prompt injection attacks, as it scores page text without executing any link instructions.
• Robust Compatibility: Functions seamlessly with various MCP-compatible environments such as Claude Code, Claude Desktop, and others.

Usage Overview

Using jevcrawl is straightforward. After obtaining an API key from console.typesafe.ai and configuring it within the desired MCP client, agents can query with specific goals. Here’s an example of how it can be invoked in a command line:

jevcrawl crawl <starting_url> --goal "<your question>"

Performance Metrics

jevcrawl is benchmarked against similar systems, demonstrating its superiority in both cost efficiency and success rates. For instance:

• Cost per correct answer: jevcrawl at $0.06 vs. Claude’s method at $0.65.
• Found answer pages: jevcrawl successfully identified pages 10 times in tests, while Crawl4AI achieved this 5 times.

Example Case Study

In practical trials, jevcrawl demonstrated its capability to distill extensive documentation into concise answers. For instance, when querying about the walrus operator in Python, it efficiently reduced 80,034 tokens of content to just 1,873 relevant tokens within seconds, costing mere fractions of a cent.

Understanding the Mechanism

jevcrawl operates by initiating from a starting point (a website’s home page), fetching the best links, and using its decision model to evaluate and score passages based on relevance to the asked question. Each request is tracked and managed to ensure budget and performance expectations are met.

Responsible Crawling

It’s important to note that jevcrawl respects web protocols, does not scrape beyond defined boundaries, and maintains compliance with site terms of service.

In summary, jevcrawl represents a sophisticated solution for AI applications that require efficient and economical web crawling capabilities, providing robust performance without compromising on reliability.