openxet

OpenXet is an open-source server and client designed for the Xet protocol, which provides a robust solution for chunk-deduplicated storage of large, versioned files. This system is particularly beneficial for managing models and datasets that frequently change, ensuring efficient storage and data management.

Key Features

• Integration with Official Tools: OpenXet allows seamless integration with the official tools such as hf, huggingface_hub, hf_xet, git-xet, and git-lfs, enabling their use against custom infrastructure without modifications. This facilitates data management while keeping ownership and security intact.
• Optimized File Handling: Only changed chunks of files are transferred, minimizing data movement and ensuring efficiency in file storage and retrieval.
• Performance and Speed: The system runs locally with a SQLite index to deliver impressive performance, responding to queries for a 100k-term file in under 50 ms on an idle node.

Current Status

The server component is operational, with the next milestone focusing on the Hub API, following the recently achieved Milestone M1, which established the core functionality. The openXet storage server can serve as a Content Addressable Storage (CAS) for hf_xet version 1.7.0 and newer, offering features such as global deduplication, verified uploads, scoped tokens, and signed range-bound download URLs. Future milestones (M2 and M3) will enhance capabilities by integrating additional APIs and supporting various storage solutions like S3, GCS, Azure, and PostgreSQL.

Getting Started

To explore OpenXet, refer to the Quickstart Guide for instructions on starting the server and performing file operations using hf_xet and the openxet command line tool. Once built, the entire process can be completed in under a second, making it practical for continuous integration testing.

Further Documentation

Comprehensive documentation is available, including:

• Server Configuration
• Command Line Interface
• Exit Codes
• Security Tests ensuring compliance with security requirements
• Benchmark Results highlighting latency gates and performance metrics
• Changelog for tracking changes and updates
• Design Specifications and Architectural Decisions

Explore the potential of OpenXet to revolutionize file management and storage solutions in your projects.