Echomine

Echomine is a powerful Python library and command-line interface (CLI) tool designed for parsing, searching, and exporting AI conversation exports. With its innovative multi-provider adapter pattern, Echomine currently supports exports from major AI platforms such as OpenAI ChatGPT and Anthropic Claude, while allowing for future extensions to other platforms like Gemini.

Key Features

• Memory Efficient: Utilizes stream-based parsing, enabling the handling of large files (1GB+) with constant memory usage.
• Advanced Search Capabilities: Incorporates BM25 relevance ranking, enabling features such as exact phrase matching, boolean logic, role filtering, and keyword exclusion.
• Dynamic Snippets: Automatically generates previews for search results, providing context for matches.
• Analytics & Statistics: Computes export statistics, conversation metrics, and identifies temporal patterns for better insight.
• Rich Command-Line Output: Features color-coded terminal displays, formatted tables, progress bars, and syntax highlighting.
• Versatile Export Formats: Supports exporting to Markdown (with YAML frontmatter), JSON, or CSV formats.
• Type Safety: Adheres to strict typing using Pydantic v2 and achieves compliance with mypy --strict to ensure reliability.
• Library-First Approach: All CLI functionalities are available as an importable Python library, reinforcing modularity and ease of use.
• Multi-Provider Support: Automatically detects and adapts to the provider used for the conversation exports, enhancing user experience.

Design Principles

Echomine is crafted with a focus on the following principles:

1. Library-First Architecture: The CLI interfaces are built on top of the library, allowing seamless integration.
2. Strict Type Safety: Ensures no Any types are present in the public API through rigorous typing practices.
3. Memory Efficiency: Implements stream-based parsing to prevent loading entire files into memory, enhancing performance.
4. Test-Driven Development: Features are validated through a test-first approach, ensuring high quality and reliability.
5. YAGNI (You Aren't Gonna Need It): Focuses on providing simple solutions without speculative features.

Quick Start Examples

Library API Usage:

from echomine import OpenAIAdapter, SearchQuery
from pathlib import Path

# Initialize the OpenAI adapter and specify the export file
adapter = OpenAIAdapter()
export_file = Path("conversations.json")

# List all conversations
for conversation in adapter.stream_conversations(export_file):
    print(f"[{conversation.created_at.date()}] {conversation.title}")

# Execute a search with keywords
query = SearchQuery(keywords=["algorithm", "design"], limit=10)
for result in adapter.search(export_file, query):
    print(f"{result.conversation.title} (score: {result.score:.2f})")

CLI Usage Example:

# Automatically detect provider and list conversations
echomine list export.json

# Search conversations by keyword
echomine search export.json --keywords "algorithm,design" --limit 10

Performance

Echomine is engineered for optimal memory efficiency and speed, capable of handling large data sets while maintaining performance. For instance, it can process 1.6GB files with 10,000 conversations and 50,000 messages in under 30 seconds.

Documentation and Support

Full documentation, including comprehensive guides, API references, and examples, can be found at the Echomine Documentation. This resource serves as an invaluable tool for developers looking to integrate or utilize Echomine in their projects.