abdiff — A tool for verifying rule changes in Claude's…

Pitch

Description

Comments

About this project

Abdiff enables users to determine the impact of changes in CLAUDE.md on Claude's output. By executing tasks before and after modifications, it clearly illustrates the effects of new rules or deletions by providing side-by-side comparisons of results, ensuring precise assessments of Claude's behavior.

abdiff is a powerful tool designed to analyze the impact of modifications made to CLAUDE.md on the behavior of Claude, an AI model. This utility allows users to validate whether a new rule alters Claude’s functionality or output, ensuring that updates produce the intended results.

Key Features

Assess whether an added rule genuinely modifies Claude’s behavior or output.
Determine if a rule slated for deletion had any influence on Claude’s operations.
Verify whether reference documentation is properly read and utilized by Claude.
Compare the consistency of Claude's responses to two different phrasings of the same rule.

How to Use

To utilize abdiff, simply follow these steps:

/plugin marketplace add 2ykwang/abdiff
/plugin install abdiff@2ykwang

Edit the desired rule in CLAUDE.md without committing the change.
Execute the command /abdiff:abdiff <what you want to test> and respond to the questionnaire with "go".
Review the results in the generated report found at .abdiff/<experiment>/report.html. Use Claude Code to open the report directly: !open <path>.

Demonstrations

Explore the following experiments to see the capabilities of abdiff:

fluent-korean: Evaluates the impact of a Korean output style on a code-analysis write-up.
payment-docs: Investigates whether Claude adheres to policy documents once they are established.
fastapi-architecture: Analyzes the effect of an architecture document on the placement of new endpoint code.
doc-sentence: Tests the reach of a change in a specification number on the corresponding code.

Additional Information

The total number of runs consists of 2 × repeats × test cases. Time and costs are influenced by the model being tested and the nature of the task.
Experiments are conducted in a disposable copy of the project, allowing files to be accessed without sandbox restrictions. For untrusted projects, consider using allowlist mode per the instructions in skills/abdiff/references/protocol.md.
User-level settings are not implemented; only files within the project can be manipulated.
Compatibility requires Claude Code CLI, git, and Python 3.9 or higher.