pr-witness

pr-witness: Ensuring Integrity in Pull Requests

The pr-witness tool checks the integrity of pull requests by comparing the tests from a base branch against the code of the pull request. It aims to catch potential regressions that may not be evident from the Continuous Integration (CI) results alone. In essence, a test suite represents a claim that certain behaviors are guaranteed. However, modifications made in a pull request can obscure these claims by altering, skipping, or rewriting tests. This tool runs the base branch's test suite against the new code in a pull request, ensuring that any test previously passing does not fail unnoticed due to changes hidden within the pull request.

Key Features

• Hidden Regression Detection: Detects when a test from the base branch that has not been touched by the pull request fails, signaling a possible regression.
• Expectation Change Monitoring: Identifies tests whose assertions have been rewritten, providing information on both the old and new assertions — ensuring clarity on whether a change is legitimate or a potential artifact of a bug fix.
• Behavior Coverage Checks: Monitors deleted tests that still pass against the pull request’s code, indicating that important behavior may have been retained but coverage is lost.
• Critical Analysis Signals: Flags various scenarios such as unexpected behavior changes, skipped tests, and unverifiable claims, allowing developers to address them effectively.

GitHub Action Integration

pr-witness can be integrated easily into GitHub Actions. The action posts comments on each pull request with live updates as changes are made. The following provides a quick setup example:

name: pr-witness
on: pull_request
permissions:
  contents: read
  pull-requests: write      # for the comment
jobs:
  witness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with: { fetch-depth: 0 }
      - uses: actions/setup-python@v7
        with: { python-version: "3.12" }
      - run: pip install -e . pytest
      - uses: talhayme/pr-witness@v0.3
        with:
          test-command: pytest
          mode: report

The action supports multiple modes of operation, including a reporting mode that doesn’t fail the job automatically, requiring explicit acknowledgment of critical flags.

Signed Evidence for Enhanced Transparency

In addition to running tests, pr-witness generates signed evidence using GitHub Artifact Attestations. This feature enhances the trustworthiness of the reports by ensuring they are verifiable and produced directly by the CI workflow, preserving their integrity.

Integration with Claude Code

Developers using Claude Code can leverage pr-witness as a plugin for deeper integration into their development workflows, promoting sound practices while managing test modifications or deletions.

Usage Beyond CI

Developers can also interact with pr-witness directly through the command line, running checks against a defined base branch to validate the current state of code changes and ensure compliance with expected behaviors.

Summary

pr-witness serves as a valuable tool for software development teams seeking to maintain rigorous standards in pull request integrity. By actively monitoring the behaviors asserted by their test suites and cross-checking them with code changes, teams can ensure a higher quality and reliability in their software development processes.