Pitch

Description

Comments

LogSentinelAI - Declarative LLM analyzer for security and system logs; extracts and visualizes data in ELK.

LogSentinelAI

combined_maroon_calida

Declarative LLM analyzer for security and system logs; extracts and visualizes data in ELK.

Visit project

Pitch

LogSentinelAI utilizes advanced language models to analyze security events, anomalies, and errors from various log sources. By simply defining the desired result structure, users can extract structured data for visualization in Elasticsearch and Kibana without the need for complex parsing. This declarative approach streamlines log management and enhances security monitoring.

Description

LogSentinelAI is a declarative, LLM-powered analyzer specifically designed for security events and system logs. This innovative tool transforms raw log data from various sources—such as Apache and Linux—into structured information that can be conveniently visualized and analyzed using Elasticsearch and Kibana. By utilizing a declarative extraction approach, users can define the desired data structure through Pydantic classes, allowing the AI to automatically parse logs and return JSON results that adhere to the specified schema.

Core Functionality

Log Sources: Collects data from multiple origins including local files, remote SSH connections, HTTP endpoints, Apache error logs, system logs, and TCPDump outputs.
Declarative Extraction: Users declare their expected result structure, and the LLM takes on the complexity of log analysis, enabling a focus on the presentation of data rather than the intricacies of parsing.

Example of Result Structure Definition

# Define the desired structure for HTTP Access log analysis
from pydantic import BaseModel

class MyAccessLogResult(BaseModel):
    ip: str
    url: str
    is_attack: bool

Key Features

AI-Powered Analysis: Automatically analyzes logs based on defined Pydantic structures, eliminating intricate log parsing tasks.
Threat Detection Capabilities: Identifies vulnerabilities including SQL Injection, XSS, Brute Force attacks, and Network Anomalies.
Flexible Output: Provides structured JSON outputs validated by Pydantic, enhancing data reliability.
Adaptive Sensitivity: The LLM model adjusts detection sensitivity based on log types and other factors for improved accuracy.

Processing Modes

Batch Processing: Allows bulk analysis of historical logs.
Real-Time Monitoring: Enables live analysis of logs through a sampling approach.

Data Enrichment and Integration

Geolocation: Incorporates GeoIP for enhanced data context, compatible with Kibana visualizations.
Enterprise Storage: Integrates seamlessly with Elasticsearch for efficient data indexing and retrieval.

Visualization

LogSentinelAI supports sophisticated dashboards via Kibana, providing users with immediate insights into security events and operational data through customizable views.

Dashboard Example

Kibana Dashboard

Sample Log Analysis Outputs

LogSentinelAI effectively analyzes various log types:

HTTP Access Log Sample

54.36.149.41 - - [22/Jan/2019:03:56:14 +0330] "GET /filter/27 HTTP/1.1" 200 30577

Apache Server Log Sample

[Thu Jun 09 06:07:04 2005] [notice] LDAP: SSL support unavailable

User-Friendly CLI

LogSentinelAI provides a command-line interface to simplify usage:

logsentinelai-httpd-access → analyzers/httpd_access.py
logsentinelai-linux-system → analyzers/linux_system.py

Conclusion

LogSentinelAI empowers users to leverage AI for efficient log analysis, ensuring that important security insights are accessible and actionable. For detailed instructions on installation and usage, refer to the provided documentation.

More Details

https://call518.medium.com/logsentinelai-practical-llm-based-ai-log-analysis-for-security-engineering-fed4c941dee2

Demo Installation & Usage Guide (RHEL & Ubuntu)

This document provides a detailed step-by-step guide for installing, configuring, and testing LogSentinelAI on RHEL (RockyLinux, CentOS, etc.) and Ubuntu environments. Each step includes clear commands, notes, and practical examples.

1. System Requirements

OS: RHEL 8/9, RockyLinux 8/9, CentOS 8/9, Ubuntu 20.04/22.04 (including WSL2)
Python: 3.11+ (3.12 recommended)
Memory: Minimum 4GB (8GB+ recommended for local LL## 13. Reference Links & Contact*: At least 2GB free space
Network: Access to PyPI, GitHub, OpenAI, Ollama/vLLM, etc.
(Optional) Docker: Required for running Elasticsearch/Kibana, vLLM, Ollama containers

2. Install uv & Create Virtual Environment

2.1 Install uv

# Install uv in Python 3.9+ environment
python3 -m pip install --user uv
# or
pip3 install --user uv

# Add PATH if needed
export PATH="$HOME/.local/bin:$PATH"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

# Verify installation
uv --version

2.2 Install Python 3.11+ & Create Virtual Environment

# Install Python 3.11
uv python install 3.11

# Create and activate virtual environment
uv venv --python=3.11 logsentinelai-venv
source logsentinelai-venv/bin/activate

# Ensure prompt changes to (logsentinelai-venv)

3. Install LogSentinelAI

3.1 Install from PyPI (Recommended)

# Install using uv
uv sync
uv pip install -U logsentinelai

3.2 Install from GitHub Source (Development/Latest)

git clone https://github.com/call518/LogSentinelAI.git
cd LogSentinelAI
uv sync
uv pip install .

4. Install Required External Tools

4.1 (Optional) Install Docker

Official Docker Install Guide
Refer to official docs for both RHEL/Ubuntu

4.2 (Optional) Install Ollama (Local LLM)

Ollama Official Install

curl -fsSL https://ollama.com/install.sh | sh
systemctl start ollama
ollama pull gemma3:1b

4.3 (Optional) Install vLLM (Local GPU LLM)

# Docker-based vLLM install & model download example
git clone https://github.com/call518/vLLM-Tutorial.git
cd vLLM-Tutorial
uv pip install huggingface_hub
huggingface-cli download lmstudio-community/Qwen2.5-3B-Instruct-GGUF Qwen2.5-3B-Instruct-Q4_K_M.gguf --local-dir ./models/Qwen2.5-3B-Instruct/
huggingface-cli download Qwen/Qwen2.5-3B-Instruct generation_config.json --local-dir ./config/Qwen2.5-3B-Instruct
# Run vLLM with Docker
./run-docker-vllm---Qwen2.5-1.5B-Instruct.sh
# Check API is working
curl -s -X GET http://localhost:5000/v1/models | jq

vLLM generation_config.json example (recommended values)

{
  "temperature": 0.1,
  "top_p": 0.5,
  "top_k": 20
}

5. Prepare Config File & Main Options

cd ~/LogSentinelAI  # If installed from source
curl -o config https://raw.githubusercontent.com/call518/LogSentinelAI/main/config.template
nano config  # or vim config
# Enter required fields such as OPENAI_API_KEY

Example config main items

# LLM Provider & Model
LLM_PROVIDER=openai   # openai/ollama/vllm/gemini
LLM_MODEL_OPENAI=gpt-4o-mini
LLM_MODEL_OLLAMA=gemma3:1b
LLM_MODEL_VLLM=Qwen/Qwen2.5-1.5B-Instruct
LLM_MODEL_GEMINI=gemini-1.5-pro

# OpenAI API Key
OPENAI_API_KEY=sk-...

# Gemini API Key (required if using Gemini provider)
GEMINI_API_KEY=YOUR_GEMINI_API_KEY_HERE

# Response language
RESPONSE_LANGUAGE=korean   # or english

# Analysis mode
ANALYSIS_MODE=batch        # batch/realtime

# Log file paths (defaults when --log-path not specified)
LOG_PATH_HTTPD_ACCESS=sample-logs/access-10k.log
LOG_PATH_HTTPD_SERVER=sample-logs/apache-10k.log
LOG_PATH_LINUX_SYSTEM=sample-logs/linux-2k.log
LOG_PATH_GENERAL_LOG=sample-logs/general.log

# chunk size (analysis unit)
CHUNK_SIZE_HTTPD_ACCESS=10
CHUNK_SIZE_HTTPD_SERVER=10
CHUNK_SIZE_LINUX_SYSTEM=10
CHUNK_SIZE_GENERAL_LOG=10

# Realtime mode options
REALTIME_POLLING_INTERVAL=5
REALTIME_MAX_LINES_PER_BATCH=50
REALTIME_BUFFER_TIME=2
REALTIME_PROCESSING_MODE=full     # full/sampling
REALTIME_SAMPLING_THRESHOLD=100

# GeoIP options
GEOIP_ENABLED=true
GEOIP_DATABASE_PATH=~/.logsentinelai/GeoLite2-City.mmdb
GEOIP_FALLBACK_COUNTRY=Unknown
GEOIP_INCLUDE_PRIVATE_IPS=false

# Elasticsearch integration options (optional)
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
ELASTICSEARCH_USER=elastic
ELASTICSEARCH_PASSWORD=changeme

6. GeoIP DB Auto/Manual Install & Usage

On first run, GeoIP City DB is automatically downloaded to ~/.logsentinelai/ (recommended)
For manual download:

logsentinelai-geoip-download
# or
logsentinelai-geoip-download --output-dir ~/.logsentinelai/

GeoIP main features

City/country/coordinates (geo_point) auto assignment, Kibana map visualization supported
Private IPs are excluded from geo_point
Analysis works even if DB is missing (GeoIP enrich is skipped)

7. Prepare Sample Log Files (for testing)

# If you already cloned the repo, skip this
git clone https://github.com/call518/LogSentinelAI.git
cd LogSentinelAI/sample-logs
ls *.log  # Check various sample logs

8. Install & Integrate Elasticsearch & Kibana (Optional)

8.1 Install ELK Stack via Docker

git clone https://github.com/call518/Docker-ELK.git
cd Docker-ELK
docker compose up setup
docker compose up kibana-genkeys  # Key generation (recommended)
docker compose up -d
# Access http://localhost:5601, elastic/changeme

8.2 Set Elasticsearch Index/Policy/Template

Run the following commands in the terminal when Kibana/Elasticsearch is running (default: http://localhost:5601, http://localhost:9200). Default account: elastic/changeme.

1) Create ILM Policy (7 days retention, 10GB/1d rollover)

curl -X PUT "localhost:9200/_ilm/policy/logsentinelai-analysis-policy" \
-H "Content-Type: application/json" \
-u elastic:changeme \
-d '{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "10gb",
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

2) Create Index Template

curl -X PUT "localhost:9200/_index_template/logsentinelai-analysis-template" \
-H "Content-Type: application/json" \
-u elastic:changeme \
-d '{
  "index_patterns": ["logsentinelai-analysis-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logsentinelai-analysis-policy",
      "index.lifecycle.rollover_alias": "logsentinelai-analysis",
      "index.mapping.total_fields.limit": "10000"
    },
    "mappings": {
      "properties": {
        "events": {
          "type": "object",
          "properties": {
            "source_ips": {
              "type": "object",
              "properties": {
                "ip": { "type": "ip" },
                "location": { "type": "geo_point" }
              }
            },
            "dest_ips": {
              "type": "object",
              "properties": {
                "ip": { "type": "ip" },
                "location": { "type": "geo_point" }
              }
            }
          }
        }
      }
    }
  }
}'

3) Create Initial Index & Write Alias

curl -X PUT "localhost:9200/logsentinelai-analysis-000001" \
-H "Content-Type: application/json" \
-u elastic:changeme \
-d '{
  "aliases": {
    "logsentinelai-analysis": {
      "is_write_index": true
    }
  }
}'

4) Import Kibana Dashboard/Settings

Access http://localhost:5601 (elastic/changeme)
Stack Management → Saved Objects → Import
Import Kibana-9.0.3-Advanced-Settings.ndjson then Kibana-9.0.3-Dashboard-LogSentinelAI.ndjson
Check results in Analytics > Dashboard > LogSentinelAI Dashboard

9. LogSentinelAI Main Commands & Test

9.1 List All Commands

logsentinelai --help

9.2 Main Analysis Command Examples

# HTTP Access log analysis (batch)
logsentinelai-httpd-access --log-path sample-logs/access-10k.log
# Apache Error log analysis
logsentinelai-httpd-server --log-path sample-logs/apache-10k.log
# Linux System log analysis
logsentinelai-linux-system --log-path sample-logs/linux-2k.log
# Realtime monitoring (local)
logsentinelai-linux-system --mode realtime
# Manual GeoIP DB download/path
logsentinelai-geoip-download --output-dir ~/.logsentinelai/

9.3 CLI Option Summary

Option	Description	config default	CLI override
--log-path	Log file path to analyze	LOG_PATH_*	Y
--mode	batch/realtime analysis mode	ANALYSIS_MODE	Y
--chunk-size	Analysis unit (lines)	CHUNK_SIZE_*	Y
--processing-mode	Realtime processing (full/sampling)	REALTIME_PROCESSING_MODE	Y
--sampling-threshold	Sampling threshold	REALTIME_SAMPLING_THRESHOLD	Y
--remote	Enable SSH remote analysis	REMOTE_LOG_MODE	Y
--ssh <user@host:port>	SSH connection info	REMOTE_SSH_*	Y
--ssh-key	SSH key path	REMOTE_SSH_KEY_PATH	Y
--help	Help	-	-

CLI options always override config file

9.8 SSH Remote Log Analysis

logsentinelai-linux-system --remote --ssh admin@192.168.1.100 --ssh-key ~/.ssh/id_rsa --log-path /var/log/messages

Tip: Register the target server in known_hosts in advance (ssh-keyscan -H <host> >> ~/.ssh/known_hosts)

9.9 Manual GeoIP DB Download/Path

logsentinelai-geoip-download --output-dir ~/.logsentinelai/

10. Declarative Extraction Usage

LogSentinelAI's biggest feature is Declarative Extraction. By declaring only the desired result structure (Pydantic class) in each analyzer, the LLM automatically analyzes logs according to that structure and returns results in JSON format. Without complex parsing/post-processing, you just declare the desired fields and AI fills in the results.

10.1 Basic Usage

In your analyzer script, declare the result structure (Pydantic class) you want to receive.
When you run the analysis command, the LLM automatically generates JSON matching that structure.

Example: Customizing HTTP Access Log Analyzer

from pydantic import BaseModel

class MyAccessLogResult(BaseModel):
    ip: str
    url: str
    is_attack: bool

Just define the fields you want, and the LLM will generate results like:

{
  "ip": "192.168.0.1",
  "url": "/admin.php",
  "is_attack": true
}

Example: Customizing Apache Error Log Analyzer

from pydantic import BaseModel

class MyApacheErrorResult(BaseModel):
    log_level: str
    event_message: str
    is_critical: bool

Example: Customizing Linux System Log Analyzer

from pydantic import BaseModel

class MyLinuxLogResult(BaseModel):
    event_type: str
    user: str
    is_anomaly: bool

By declaring only the result structure you want in each analyzer, the LLM automatically returns results in that structure—no manual parsing required.

11. Advanced Usage Examples

11.1 Set Defaults in config & Override with CLI

# Set CHUNK_SIZE_LINUX_SYSTEM=20 in config
logsentinelai-linux-system --chunk-size 10  # CLI option takes precedence

11.2 Realtime Mode Auto Sampling Operation & Principle

Realtime Mode Key Features

Starts from the current End of File (EOF) and processes only newly added logs

Existing logs in the file are excluded from analysis (true real-time monitoring)

Even after program interruption and restart, past logs are not processed (always starts from current time)

logsentinelai-httpd-access --mode realtime --processing-mode full --sampling-threshold 100
# Check auto-switch to sampling mode on heavy log inflow

Sampling Operation Example

Normal: 15 lines in → FULL mode (below threshold), analyze by chunk_size
Traffic spike: 250 lines in → Exceeds threshold (100), auto-switch to SAMPLING mode, analyze only latest 10 lines, skip the rest (original logs preserved)
Traffic normalizes: Back to FULL mode

Sampling Strategy

FIFO buffer, if threshold exceeded, only latest chunk_size analyzed
No severity/pattern-based priority (purely time order)
Possible analysis omission (original logs preserved)

11.3 Import Kibana Dashboard

Access http://localhost:5601 (elastic/changeme)
Stack Management → Saved Objects → Import
Import Kibana-9.0.3-Advanced-Settings.ndjson then Kibana-9.0.3-Dashboard-LogSentinelAI.ndjson
Check results in Analytics > Dashboard > LogSentinelAI Dashboard

12. Troubleshooting FAQ

Permission denied on pip install: Activate virtualenv or use pip install --user
Python 3.11 not found: Check install path, use python3.11 directly
Cannot access Elasticsearch/Kibana: Check Docker status, port conflicts, firewall
GeoIP DB download failed: Download manually and set path in config
SSH remote analysis error: Check SSH key permissions, known_hosts, firewall, port
LLM API error: Check OPENAI_API_KEY, Ollama/vLLM server status, network

13. Reference/Recommended Links & Contact

Contact/Feedback: GitHub Issue, Discussions, Pull Request welcome

13. Reference Links & Contact

Contact/Feedback: GitHub Issue, Discussions, Pull Request welcome

Appendix

A. Real-time Auto-Sampling Detailed Mechanism

This section provides a detailed explanation of how the system automatically switches processing modes when large volumes of logs are ingested in real-time mode.

A.1 Related Parameters and Their Roles

Parameter	Default	Role	Impact
`REALTIME_PROCESSING_MODE`	`full`	Base processing mode (full/sampling)	Determines initial processing method
`REALTIME_SAMPLING_THRESHOLD`	`100`	Auto-sampling switch threshold	Based on number of pending log lines
`CHUNK_SIZE_*`	`10`	LLM analysis unit	Number of log lines to analyze at once
`REALTIME_POLLING_INTERVAL`	`5`	Polling interval (seconds)	Log file check frequency
`REALTIME_MAX_LINES_PER_BATCH`	`50`	Read limit	Maximum lines to read at once
`REALTIME_BUFFER_TIME`	`2`	Buffer time (seconds)	Prevents incomplete log line processing

A.2 Auto-Sampling Trigger Scenarios

Scenario 1: Normal Log Processing (FULL mode maintained)

Configuration:
- CHUNK_SIZE_HTTPD_ACCESS = 10
- REALTIME_SAMPLING_THRESHOLD = 100
- REALTIME_POLLING_INTERVAL = 5
- REALTIME_MAX_LINES_PER_BATCH = 50

Process:
1. Check /var/log/apache2/access.log every 5 seconds
2. Find 15 new log lines → Add to internal pending buffer
3. Pending buffer: 15 lines (below threshold of 100)
4. Process CHUNK_SIZE(10): Analyze 10 lines with LLM, 5 lines remain pending
5. Check for additional logs in next polling cycle

Scenario 2: Traffic Spike - Auto-switch to Sampling

Configuration: Same as above

Spike situation:
1. Poll every 5 seconds, reading 50 lines each time (MAX_LINES_PER_BATCH)
2. Continuous heavy log influx over multiple polling cycles
3. Pending buffer accumulation: 20 lines → 45 lines → 85 lines → 125 lines
4. 125 lines > threshold(100) ▶️ Auto-switch to SAMPLING mode

SAMPLING mode operation:
- System outputs "AUTO-SWITCH: Pending lines (125) exceed threshold (100)"
- Displays "SWITCHING TO SAMPLING MODE" message
- Select only latest 10 lines (CHUNK_SIZE) from 125 pending lines
- Discard remaining 115 lines (original log file preserved)
- Output "SAMPLING: Discarded 115 older lines, keeping latest 10" message
- Send only latest 10 lines to LLM for analysis
- Limit memory usage, prevent system overload

Scenario 3: Traffic Normalization - Return to FULL mode

Normalization process:
1. Log influx decreases: around 5-15 lines per polling cycle
2. Pending buffer drops below threshold (100)
3. Automatically return to FULL mode
4. Resume sequential processing of all logs

A.3 Real-world Usage Examples

Web Server DDoS Attack Scenario

# Configuration: In config file
CHUNK_SIZE_HTTPD_ACCESS=15
REALTIME_SAMPLING_THRESHOLD=200
REALTIME_POLLING_INTERVAL=3

# Execution
logsentinelai-httpd-access --mode realtime

# Situational behavior:
# Normal: 10-20 requests/sec → FULL mode analyzes all logs
# Attack: 500+ requests/sec → SAMPLING mode when pending buffer exceeds 200 lines
# SAMPLING: Analyze only latest 15 lines, ignore rest to protect system
# Attack ends: Request volume normalizes → Auto return to FULL mode

System Log Mass Generation Scenario

# Configuration
CHUNK_SIZE_LINUX_SYSTEM=20
REALTIME_SAMPLING_THRESHOLD=150

# Situation: System error generating 100 error log lines per second
# Within 1-2 minutes, pending buffer exceeds 150 lines
# → Auto SAMPLING: Analyze only latest 20 lines
# → Protect system resources, prioritize latest errors

A.4 Sampling Strategy Features and Limitations

Advantages

Automation: Control system load without user intervention
Memory Protection: Prevent unlimited buffer growth
Recency Guarantee: Focus on most recent logs
Original Preservation: Log files themselves remain intact

Limitations

Analysis Gaps: Some logs skipped during sampling
Time-based Only: Chronological processing, no severity-based priority
Temporary Blind Spots: Limited pattern analysis during spike periods

Recommended Tuning Methods

# High-performance system (sufficient memory)
REALTIME_SAMPLING_THRESHOLD=500
CHUNK_SIZE_*=25

# Low-spec system (memory constrained)
REALTIME_SAMPLING_THRESHOLD=50
CHUNK_SIZE_*=5

# Critical logs (minimize gaps)
REALTIME_SAMPLING_THRESHOLD=1000
REALTIME_POLLING_INTERVAL=2

# General monitoring (efficiency priority)
REALTIME_SAMPLING_THRESHOLD=100
REALTIME_POLLING_INTERVAL=10

Through this auto-sampling mechanism, LogSentinelAI provides stable real-time analysis even in unpredictable log traffic situations.

0 comments

No comments yet.

New comment