StreamLens: A Comprehensive Visualization Tool for Apache Kafka Topologies

StreamLens is a powerful full-stack application designed to visualize Apache Kafka topologies, including topics, producers, consumers, streams, schemas, connectors, and access control lists (ACLs). This tool enhances the user experience by incorporating an optional AI assistant, StreamPilot, to aid in understanding and navigating complex Kafka environments.

Key Features

• Live Topology Visualization: Interactive graphs display the structure of your Kafka cluster, powered by React Flow for real-time updates.
• Auto-discovery of Components: Automatically detects topics, consumer groups, producers, connectors, schemas, and ACLs from the live cluster, ensuring that users see the most current data.
• Schema Grouping: Merges schemas that share the same Schema Registry ID into a single node linked to all related topics for clearer organization.
• Consumer Lag Metrics: Provides detailed insights into consumer lag on a per-partition basis by clicking on consumer nodes.
• Detailed Topic Insights: Access configuration settings, recent messages, and generate sample client code in Java or Python directly from topic nodes.
• Connector Configuration View: Inspect connector node configurations while sensitive values remain masked for security.
• Producer Detection Features: Detect producers through various methods such as JMX metrics or offset-change detection.
• Intuitive Search and Navigation: Users can easily find nodes by name or type, with features like auto-zoom and keyboard navigation for efficient exploration of large clusters.
• Incremental Topic Loading: Enhances performance in large clusters by loading topics in stages, starting with connected topics, and providing a comprehensive search functionality.
• Message Production Capability: Allows optional message production directly from the UI, which can be enabled on a per-cluster basis.
• AI Assistant (StreamPilot): Engage with an AI-powered assistant to ask questions about the topology, with functionality to highlight and zoom into relevant nodes. It supports integrations with OpenAI, Gemini, Anthropic, and Ollama.
• Customizable Themes: Users can toggle between dark and light themes to suit their preferences.

Project Structure

StreamLens's architecture is organized into several components:

• client/: Contains the React-based frontend using Vite, TypeScript, and Tailwind CSS.
• server/: Houses the Python backend implemented with FastAPI for efficient API handling.
• container/: Provides Docker files for deployment and testing.
• docs/: Contains additional helpful documentation, including guides for AI setup and detailed infrastructure layouts.

Configuration and Usage

Clusters can easily be managed through the configuration file, server/data/clusters.json, allowing for seamless adjustments and integration with existing Kafka setups. Users can also configure various protocols and enable advanced features such as JMX and AI integrations.

StreamLens stands out as an essential tool for teams working with Kafka. Its robust visualization capabilities, coupled with AI support and real-time data retrieval, optimize the management and analysis of Kafka topologies, making it easier to maintain and troubleshoot complex streaming infrastructures.