abdiff

Pitch

Description

Comments

About this project

Abdiff is a tool designed to verify whether modifications in CLAUDE.md affect Claude's performance. By running experiments before and after rule edits, it provides side-by-side comparisons to reveal any changes in behavior, ensuring effective rule management.

diff is an innovative tool designed to determine whether modifications made to CLAUDE.md result in changes to Claude's behavior. Users can easily add rules to CLAUDE.md and test their effectiveness. This tool compares the outputs of tasks executed before and after rule modifications, providing clear insights into their impact.

Key Features

Impact Assessment: Quickly identify if a new rule alters Claude's behavior or outputs.
Rule Analysis: Understand the effectiveness of rules that are considered for deletion.
Reference Document Compliance: Verify whether instructions in reference documents are being adhered to or ignored.
Wording Consistency: Discover which phrasing of a rule is followed more reliably by Claude.

How to Use

To utilize abdiff, follow these steps:

/plugin marketplace add 2ykwang/abdiff
/plugin install abdiff@2ykwang

Edit the desired rule in CLAUDE.md without committing changes.
Execute the command: /abdiff:abdiff <what you want to test>, proceed by confirming the questionnaire.
Open the report located at .abdiff/<experiment>/report.html to view the results.

Demonstrations

Explore the following experiments to see abdiff in action. Each link directs to a report detailing the experiment's setup and the specific changes tested:

Fluent Korean: Analysis of whether activating a Korean output style modifies code-analysis write-ups.
Payment Docs: Examines how Claude interacts with policy documents once they are implemented.
FastAPI Architecture: Investigates whether an architecture document influences the placement of new endpoint code.
Document Sentence: Assesses if modifying a single number in an imported specification impacts the code.

Important Information

Each test run consists of 2 × repeats × test cases, where time and costs will vary based on the model and task complexity.
Tests execute on a temporary copy of the project to prevent changes to the source files, while still allowing Bash access to the external filesystem.
User-level settings are not applied during execution, maintaining a streamlined testing environment.
Prerequisites include Claude Code CLI, git, and Python 3.9 or higher.