Aqueduct

Aqueduct is an innovative framework designed for creating intelligent, self-healing data pipelines that leverage a declarative approach. It enables developers to build pipelines that are not only autonomous but also observable, enhancing the efficiency and reliability of data workflows.

Key Features

• Self-Healing Mechanism: Aqueduct automatically generates patches for failures, allowing seamless recovery without manual intervention. When a pipeline encounters an error, the system diagnoses the issue and produces a structured, Git-diffable patch, ready for review. Thus, instead of waking up to error messages, users can greet a pending patch that resolves the problem.

• Declarative Pipelines: Pipelines are defined using YAML blueprints, eliminating the need for cumbersome boilerplate code associated with frameworks like PySpark. Users can bring their own schedulers while Aqueduct acts as the control plane, facilitating better management of data workflows.

• Robust Observability: Every execution, healing attempt, and lineage change is captured in a queryable store, enhancing transparency and enabling thorough analysis of pipeline operations.

• Engine Agnostic and Local First: Aqueduct supports multiple engines, including Spark and DuckDB, allowing users to choose the most suitable for their needs while ensuring that all operations occur within their data environment. This keeps data local and secure, maintaining compliance with data privacy standards.

• Deterministic Operation Logic: While the LLM handles the diagnosis and proposal of fixes, the actual application of changes follows deterministic rules, ensuring all operations are predictable and subject to validation gates.

Supported Engines

Aqueduct provides support for both Apache Spark and DuckDB, allowing pipelines to be executed on either engine while adhering to a consistent blueprint format.

modules:
  - id: extract        # heavy join across two warehouses
    type: Channel
    engine: spark
    config: { op: sql, query: "..." }

  - id: aggregate      # small result, no cluster needed
    type: Channel
    engine: duckdb
    config: { op: sql, query: "SELECT region, sum(amount) FROM extract GROUP BY region" }

Comprehensive Healing Flow

Aqueduct streamlines the error handling process through a clearly defined healing flow, which involves multiple validation gates before applying patches. These include checks for guardrails, compiling errors, lineage integrity, and sandbox testing, ensuring that all changes are safe and effective.

Observability Dashboard

The aqueduct dashboard command launches a local, read-only Streamlit interface, providing insights into fleet health, run metrics, lineage, and the status of healing patches. This dashboard offers a centralized view of pipeline performance without impacting operational efficiency.

Conclusion

In a landscape where data pipelines are critical to business intelligence, Aqueduct represents a significant advancement by simplifying the management of data workflows, minimizing downtime, and enhancing overall pipeline reliability. By combining a self-healing architecture with a user-friendly blueprint approach, Aqueduct empowers users to maintain efficient and effective data operations.