OpenClaw Self-Healing System

"The system that heals itself — or calls for help when it can't."

The OpenClaw Self-Healing System offers a production-ready, 4-tier autonomous recovery solution for the OpenClaw Gateway. This innovative system leverages AI-powered diagnosis and repair capabilities through Claude Code, the world’s first emergency doctor for OpenClaw.

🎬 Demo

Watch the 4-tier recovery system in action: Watchdog → Health Check → Claude Doctor → Alert.

🌟 Purpose

OpenClaw Gateway can experience crashes and health check failures, leaving developers in the dark. This self-healing system proactively manages failures by:

1. Restarting the Gateway (Level 1-2, seconds)
2. Diagnosing issues (Level 3, AI-powered)
3. Fixing root causes (Level 3, autonomous)
4. Alerting users (Level 4, as a last resort)

Unlike standard watchdogs that merely restart processes, this system provides insights into why failures occur and how to address them, with Claude Code acting as a virtual emergency doctor.

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│ Level 1: Watchdog (180s interval)                       │
│ ├─ LaunchAgent: ai.openclaw.watchdog                    │
│ └─ Process exists? No → Restart                         │
└─────────────────────────────────────────────────────────┘
                         ↓ (process alive but unresponsive)
┌─────────────────────────────────────────────────────────┐
│ Level 2: Health Check (300s interval)                   │
│ ├─ HTTP 200 check on localhost:18789                    │
│ ├─ 3 retries with 30s delay                             │
│ └─ Still failing? → Level 3 escalation                  │
└─────────────────────────────────────────────────────────┘
                         ↓ (5 minutes of failure)
┌─────────────────────────────────────────────────────────┐
│ Level 3: Claude Emergency Recovery (30m timeout) 🧠     │
│ ├─ Launch Claude Code in tmux PTY session               │
│ ├─ Automated diagnosis:                                 │
│ │   - OpenClaw status                                   │
│ │   - Log analysis                                      │
│ │   - Config validation                                 │
│ │   - Port conflict detection                           │
│ │   - Dependency check                                  │
│ ├─ Autonomous repair (config fixes, restarts)           │
│ ├─ Generate recovery report                             │
│ └─ Success/failure verdict (HTTP 200 check)             │
└─────────────────────────────────────────────────────────┘
                         ↓ (Claude recovery failed)
┌─────────────────────────────────────────────────────────┐
│ Level 4: Discord Notification (300s monitoring) 🚨      │
│ ├─ Monitor emergency-recovery logs                      │
│ ├─ Pattern match: "MANUAL INTERVENTION REQUIRED"        │
│ └─ Alert human via Discord (with detailed logs)         │
└─────────────────────────────────────────────────────────┘

✨ Unique Features

1. AI-Powered Diagnosis 🧠

• Claude Code serves as an emergency doctor
• Conducts a 30-minute autonomous troubleshooting session
• Produces human-readable recovery reports
• First of its kind for OpenClaw

2. Production-Tested ✅

• Level 2 and Level 3 verifications demonstrate system effectiveness in real deployment scenarios

3. Meta-Level Self-Healing 🔄

• "AI heals AI" — OpenClaw autonomously resolves its own issues
• Focused on the agent itself to minimize false alarms

4. Safe by Design 🔒

• Engineered without hard-coded secrets; utilizes environment variables
• Lock files to prevent race conditions
• Atomic writes ensure alert tracking integrity

5. Elegant Simplicity 🎨

• Core functionality encapsulated in just 3 bash scripts
• Minimal external dependencies needed for operation

📚 Documentation

Comprehensive guides and documents are available to ease usage, including:

• Quick Start Guide
• Architecture Deep Dive
• Troubleshooting

🐛 Known Limitations

Currently, the system is designed exclusively for macOS environments and does not support multi-node configurations yet. Also, it requires a stable internet connection for AI functionalities.

🤝 Contributing

Contributions to the OpenClaw Self-Healing System are welcomed. Guidelines are provided in the CONTRIBUTING.md.

This self-healing system monitors, diagnoses, and resolves potential issues with OpenClaw Gateway efficiently, ensuring seamless operations for developers and end-users alike.