Bellwether

Bellwether is an open-source testing tool specifically designed for Model Context Protocol (MCP) servers. This innovative tool helps detect breaking schema changes before they impact production environments, ensuring that AI workflows remain functional and reliable. With deterministic validation and optional large language model (LLM)-powered behavioral exploration, Bellwether seamlessly integrates into CI/CD pipelines, offering optimal support through GitHub Actions.

What is Model Context Protocol (MCP)?

Model Context Protocol serves as the bridge for AI assistants, enabling connections to external tools that allow interactions such as reading files, querying databases, and calling APIs. Changes in the underlying tool schemas can often lead to silent failures in AI workflows, making early detection crucial.

Key Advantages of Bellwether

• Drift Detection: Automatically identifies schema changes during continuous integration (CI) processes, preventing breaking changes from reaching production.
• Native MCP Support: Understanding of tools, prompts, and resources helps automate the testing process without requiring extensive manual intervention.
• Automated Exploration: Covers edge cases that manual testing might overlook, enhancing overall reliability.
• Real-time Documentation: Generates CONTRACT.md documentation based on actual tool behavior, ensuring that documentation remains up-to-date and reflective of system capabilities.

Quick and Efficient Setup

Embark on testing with Bellwether by executing simple commands directly in your CLI:

npm install -g @dotsetlabs/bellwether
bellwether init npx @mcp/your-server
bellwether check

This streamlined process eliminates the need for API keys and incurs no LLM costs, running efficiently in seconds.

Modes of Operation

Bellwether operates in two distinct modes:

• check: Utilized for schema drift detection; this mode is free and should be run during every pull request to ensure the integrity of the schema.
• explore: Enables LLM-powered behavioral testing for thorough analysis. While this mode may incur LLM API costs, it provides deeper insights for local development.

CI/CD Integration

Integrating Bellwether into a CI/CD workflow is straightforward. Firstly, establish a baseline using the following commands:

bellwether init npx @mcp/your-server
bellwether check
bellwether baseline save

The generated baseline can then be included in CI checks to continuously ensure compliance with expected schema definitions:

# .github/workflows/bellwether.yml
name: MCP Drift Detection
on: [pull_request]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx @dotsetlabs/bellwether check --fail-on-drift

What Changes Does Bellwether Detect?

Bellwether is capable of identifying various types of changes, including:

• Adding or removing tools: e.g., introduction or removal of delete_file.
• Schema modifications: e.g., changing a parameter to required status.
• Renaming parameters: e.g., changing path to file_path.
• Documentation changes: Updates in tool help text may issue a warning.
• Performance regressions: Notable increases in latency are flagged as warnings.

Essential Commands for Users

Some essential commands include:

bellwether init <server-command>   # Create configuration
bellwether check                   # Detect schema drift
bellwether baseline save           # Secure baseline state
bellwether baseline compare        # Compare current state with baseline

Additional Resources

For further assistance and enhanced usage of Bellwether, consult the following resources:

• Documentation: docs.bellwether.sh - Comprehensive reference for configuration and command usage.
• Community: Engage via GitHub Discussions for inquiries or share ideas, and report issues through GitHub Issues.

By implementing Bellwether, organizations can enhance their AI applications' resilience, ensuring that schema changes are detected and managed effectively, thus preserving the integrity of AI functionalities.