clink

clink is a high-performance embedded-first, Arrow-native stream processing engine developed in modern C++ (C++23). It enables stateful stream processing with advanced engine-grade capabilities including SQL support, event time handling, keyed state management, and exactly-once checkpointing. With clink, pipelines can be executed quickly either in-process or distributed, maintaining efficiency and reliability.

Key Features:

• In-Process Execution: Execute SQL pipelines directly with minimal startup time. For instance, the command clink run pipeline.sql produces results in approximately 155 ms.
• Embedded as a Library: The engine can be seamlessly embedded in applications using the pure-C ABI (libclink). Additionally, results can be collected as Arrow C streams or parsed in Python notebooks using pyclink.
• Distributed Processing: Run the same SQL file on a distributed Coordinator/Worker cluster, ensuring parallelism, fault tolerance, and scalability without making changes to the SQL query.

Innovative Capabilities:

• Open Dataset State: Manage state effectively with documented Arrow IPC. Snapshots can directly integrate with tools such as pyarrow, DuckDB, or Polars, facilitating easy exports to file formats like Parquet and Iceberg. The live state can be accessed via point lookups and Arrow scans over HTTP, eliminating unnecessary sink round-trips.
• Deterministic Replay of Incidents: Utilize a flight recorder to log operator consumption per checkpoint. The command clink replay allows for the re-execution of operations over captured records, maintaining the exact state of events for regression tests.
• No Management Overhead: Deploy as a single static binary with no dependency on managed runtimes, ensuring rapid cold starts and consistent behavior across different environments.

Performance Metrics:

clink has demonstrated superior efficiency, processing events with significantly lower CPU usage compared to traditional JVM stream processors. For example, in the 17-query Nexmark suite over a five-node cluster, clink achieved a median performance improvement of 2.45 times in CPU utilization for equivalent output.

Easy to Start:

To experiment with clink, simply run:

docker run --rm ghcr.io/orhaugh/clink-runtime:latest run /opt/clink/examples/sql/hello.sql

This runs a predefined SQL pipeline that aggregates device readings, showcasing clink's capability to handle real-time data streams effectively.

Supported Platforms:

clink is designed for macOS and Linux environments, specifically Debian family distributions, and is optimized for Apple Silicon. It supports CI compatibility and offers a straightforward build process, allowing users to compile from source or use prebuilt binaries.

Documentation and Resources:

Extensive documentation is available at orhaugh.github.io/clink, covering capability catalogs, SQL references, and detailed descriptions of supported connectors and internal structures. For users looking to contribute or seek assistance, the repository includes resources for reporting issues and discussions.