kv-cli

kv-cli: Enhance Data Pipeline Compliance

kv-cli is a versatile tool designed to enforce parameter contracts across various Python data pipelines, including frameworks such as Airflow, dbt, PySpark, Databricks, Snowpark, Flink, and Polars. By implementing robust contract validations, kv-cli helps to ensure that all governed functions explicitly declare required schema arguments—such as target_environment—and do not hardcode sensitive credentials or pin infrastructure identifiers to specific environments.

Key Features:

• Sub-200ms Compliance Checks: Conduct local scans of pipelines without the need for external accounts or network calls, ensuring that the audit process is swift and efficient.
• Parameter Contract Enforcement: Identify parameter contract violations (KV001), detect hardcoded credentials (KV002), and highlight pinned infrastructure identifiers (KV003) before deploying code.
• Automation and Fixes: Utilize the init command to automatically generate a contract file and initiate the audit process. The fix command rewrites problematic code sections—a seamless way to enhance compliance without manual intervention.

How It Works:

The tool employs a tree-sitter AST (Abstract Syntax Tree) for local analysis, focusing on parameter contract breaks and ensuring that sensitive information is not embedded within the codebase. The output includes comprehensive details such as the number of findings and the overall compliance status, clearly indicating any required actions.

Example Usage:

# Initialize and audit your pipeline repository
./kv-cli init  # This generates a .kovallent.yaml contract file
./kv-cli audit ~/your-pipeline-repo  # Scan for compliance issues

Framework Support:

• Airflow: Supports annotations for DAG and task decorations while ensuring sensitive parameters are handled appropriately.
• dbt: Validates model signatures and credentials management in profiles.yml files.
• Databricks and PySpark: Ensures compliance for data functions used within these frameworks.

Reason to Use kv-cli:

Before a developer's changes are merged, kv-cli acts as a gatekeeper in Continuous Integration (CI) environments, preventing non-compliant code from proceeding into production. By addressing potential issues at the development stage, it mitigates the risk of operational failures and security leaks, delivering a streamlined approach to code governance before it is executed in various environments.

Join the Community:

Engage with the community for ideas, feature requests, and discussions to continuously improve your data pipeline governance strategies. For enterprise solutions, consider exploring Kovallent's offerings for organization-wide policy governance.

For more detailed information and contributions, refer to the full documentation in the kv-cli GitHub repository.