LLM Sentinel

LLM Sentinel is a lightweight, cross-platform tool designed to monitor locally-running Large Language Model (LLM) processes for unexpected external network connections. It provides crucial visibility into when AI models may be making unauthorized network calls, ensuring that data security is maintained in increasingly complex LLM environments.

Why Use LLM Sentinel?

As local LLMs incorporate various components, such as inference servers, frontends, and Python wrappers, the potential for unauthorized network activity increases. LLM Sentinel actively observes all connections made by these processes, alerting users in real-time to any suspicious activity.

Key Features

• Reliable Process Detection: Employs a multi-signal scoring system to effectively identify LLM runtimes, regardless of process naming conventions. Detection signals include:

  • Opening specific model weight files (e.g., .gguf, .safetensors)
  • ML inference libraries present in memory (Linux only)
  • Recognized name or command line patterns (e.g., Ollama, vLLM)
  • A significant memory footprint as an additional indicator

• External Connection Alerting: Classifies connections as local (safe) or external (potentially harmful), utilizing reverse DNS resolution and known organization labeling (e.g., AWS, GCP, Cloudflare).

• Live Terminal Dashboard: Features a rich TUI powered by the rich library, displaying real-time scan results.

• Web Dashboard: Built with FastAPI, this dashboard offers an enterprise-style UI with both light and dark themes. Key features include:

  • Metrics tiles and live process & connection tables
  • Comprehensive session history with the ability to review past connection events
  • Secure login with hashing for passwords and session management

• Session Persistence: Utilizes SQLite to maintain a complete record of connection history across restarts.

• Insights Panel: Provides valuable analytics including session age, unique destination counts, most contacted hosts, and connection phases.

• Zero External Auth Dependencies: Implements password hashing using Python's built-in hashlib.scrypt, ensuring security without relying on external libraries.

Supported Tools

LLM Sentinel can detect a variety of tools based on their name, command line pattern, or the model files and libraries they utilize. Some of the supported tools include:

Quickstart Usage

To launch the terminal dashboard (default mode):

python main.py

For additional functionality, such as launching the web UI:

python main.py --web

Architecture Overview

The architecture of LLM Sentinel is modular, incorporating various components focused on monitoring, alerting, and user interface management. Key files include:

• process_monitor.py: For process detection
• network_monitor.py: For connection classification
• session_log.py: Manages SQLite persistence

Testing Support

Extensive testing is supported through pytest, ensuring code reliability across various functionalities, including process detection, network classification, and web endpoints.

In summary, LLM Sentinel equips users with a robust solution to monitor and secure their local LLM environments, providing real-time insights into network activities to prevent unauthorized data interactions.