abdiff is a powerful tool designed to analyze the impact of modifications made to CLAUDE.md on the behavior of Claude, an AI model. This utility allows users to validate whether a new rule alters Claude’s functionality or output, ensuring that updates produce the intended results.

Key Features

• Assess whether an added rule genuinely modifies Claude’s behavior or output.
• Determine if a rule slated for deletion had any influence on Claude’s operations.
• Verify whether reference documentation is properly read and utilized by Claude.
• Compare the consistency of Claude's responses to two different phrasings of the same rule.

How to Use

To utilize abdiff, simply follow these steps:

/plugin marketplace add 2ykwang/abdiff
/plugin install abdiff@2ykwang

1. Edit the desired rule in CLAUDE.md without committing the change.
2. Execute the command /abdiff:abdiff <what you want to test> and respond to the questionnaire with "go".
3. Review the results in the generated report found at .abdiff/<experiment>/report.html. Use Claude Code to open the report directly: !open <path>.

Demonstrations

Explore the following experiments to see the capabilities of abdiff:

• fluent-korean: Evaluates the impact of a Korean output style on a code-analysis write-up.
• payment-docs: Investigates whether Claude adheres to policy documents once they are established.
• fastapi-architecture: Analyzes the effect of an architecture document on the placement of new endpoint code.
• doc-sentence: Tests the reach of a change in a specification number on the corresponding code.

Additional Information

• The total number of runs consists of 2 × repeats × test cases. Time and costs are influenced by the model being tested and the nature of the task.
• Experiments are conducted in a disposable copy of the project, allowing files to be accessed without sandbox restrictions. For untrusted projects, consider using allowlist mode per the instructions in skills/abdiff/references/protocol.md.
• User-level settings are not implemented; only files within the project can be manipulated.
• Compatibility requires Claude Code CLI, git, and Python 3.9 or higher.