hashsync

Overview

hashsync is a powerful utility designed to optimize command execution on remote hosts over SSH by deduplicating input and output data streams. This innovative approach minimizes bandwidth usage significantly by replacing duplicated content with block references derived from files already present on the receiving host. By utilizing efficient techniques such as block-level deduplication and zstd compression, hashsync ensures that data transfers are both effective and fast.

Key Features

• SSH Compatibility: Operates seamlessly with SSH, allowing existing configurations and keys to be utilized without additional setup.
• Input and Output Deduplication: Offers flexibility by providing options to deduplicate either the command's output against local files or the standard input against remote files, enabling effective data management.
• Enhanced rsync Functionality: Enhances rsync operations by allowing file transfers to deduplicate against the receiving host's file index, thus reducing redundant data transfer when files are renamed or moved.
• Accuracy Assurance: Employs a cryptographically strong hash mechanism to verify block references, assuring that the output remains consistent with the original stream.
• Dynamic Index Maintenance: Keeps the receiver's index updated before and after data transfer, enhancing the accuracy and efficiency of future operations.
• No Network Daemon Required: The setup is simple, relying on the same binary for both sender and receiver, eliminating the need for additional services.
• Optimized Compression: Non-duplicated data is compressed using zstd, applying advanced compression techniques for maximum efficiency.

Usage Example

A practical example of using hashsync with rsync would look like this:

# Copying data with deduplication against existing files
rsync -a --copy-dest=../v1 -e 'hashsync ssh' nas:/srv/data/v2/ /srv/data/v2/

In this example, any block of data that exists in /srv/data/v1 is referenced rather than resent, leading to significant bandwidth savings.

Operational Workflow

1. Establish Connection: The sender connects to the receiver using hashsync via SSH, transferring commands and data streams.
2. Hash List Exchange: The receiver sends a hash list of its index, allowing the sender to compute rolling hashes as it streams data.
3. Block Verification: The receiver verifies received blocks against stored data using SHA hashes, sending back only unique data or references.
4. Data Reconstruction: The original data stream is reassembled on the receiver’s end, providing a complete and accurate result without unnecessary transfers.

Conclusion

hashsync represents a significant advancement in the realm of data transfer over SSH, offering enhanced efficiency and performance. Its ability to deduplicate data intelligently makes it a valuable tool for system administrators and developers who manage large data sets across various environments. For more details, refer to the man page or the full documentation available in the repository.