ProofFrame

ProofFrame is an advanced tool designed for data quality assurance, leveraging the power of Rust and Python to deliver sophisticated data validation capabilities. By utilizing Arrow-native structures, ProofFrame provides a robust solution for managing data contracts, validating datasets, and generating cryptographic evidence for data integrity.

Key Features

• Comprehensive Data Profiles: Each dataset is profiled to produce a deterministic BLAKE3 fingerprint, enabling precise identification of checked datasets.
• Bounded Evidence Generation: Unlike traditional pass/fail checks, validation returns detailed findings, including row-level errors and total violation counts, ensuring transparency.
• Seamless Integration: Built with a Rust core and Python bindings, the engine processes inputs from PyArrow, Pandas, and Polars without materializing rows in Python, enhancing performance for data-heavy applications.
• Contract-Based Validation: Utilize a deterministic JSON structure for validation results, facilitating clear interpretation of data integrity outcomes alongside explicit exit codes for CI integration.
• Efficient Change Detection: ProofFrame allows for easy comparison of datasets to identify changes in key columns through an intuitive diff mechanism that efficiently handles memory management.
• Robust PII and Leakage Scans: Built-in functionality for scanning personally identifiable information and detecting potential data leakage, ensuring that sensitive information is properly managed and secured.
• Signed Proof Receipts: Generate and verify proof receipts with Ed25519 signatures, providing cryptographic assurance that the validation results haven't been tampered with.

Example Usage

ProofFrame offers an intuitive API for data validation. Here’s a brief example demonstrating its capabilities:

import pyarrow as pa
import proofframe as pf

users = pa.table({
    "id": [1, 1, 3],
    "email": ["a@example.com", None, "not-an-email"],
    "score": [0.91, 1.40, 0.73],
})

report = pf.validate(users, {
    "columns": {
        "id": {"required": True, "unique": True},
        "email": {"not_null": True, "pattern": r"^[^@]+@[^@]+$"},
        "score": {"min": 0, "max": 1},
    }
})

assert not report["valid"]
for finding in report["findings"]:
    print(finding)

This script validates user data, checking for duplicates, null values, and out-of-bounds scores. Findings are reported with specifics, allowing for quick resolution of data quality issues.

In addition to validation, ProofFrame supports profiling datasets to create fingerprints that can be stored in CI metadata for future reference, ensuring reproducibility and trust in data handling.

Conclusion

ProofFrame supports teams in processing high-quality data while adhering to strict validation requirements. Its innovative design and comprehensive validation features make it a valuable asset in any data-centric workflow.