Innovative zero-prior lossless compression using online context mixing.
Project details
UCZ is an experimental compression engine that operates without any external dictionaries, pretrained corpora, or format-specific decoders. By leveraging an online context-mixing approach, it learns from incoming data directly, providing novel insights into data compression efficiencies. Ideal for researchers exploring the limits of compression technology.
UCZ: Universal Context-Mixing Zero-Prior Compression Engine
UCZ is an experimental lossless compression engine designed to explore the boundaries of data compression without relying on any external knowledge, such as predefined dictionaries or training sets. Written in TypeScript, it operates entirely through a self-sufficient mechanism, learning the patterns of a data stream in real time.
The term "zero-prior" signifies that UCZ does not depend on:
The design of UCZ emerged from a thought experiment that challenges conventional wisdom in compression technology. Traditional tools leverage extensive knowledge to achieve high compression ratios. UCZ, however, directly engages with the data, continuously learning from its structure and characteristics without preconceived notions. This approach aims to uncover the true potential of universal online learning in data compression.
UCZ's compression process follows a straightforward flow:
UCZ is designed to handle various data formats, with distinct methodologies for different types:
UCZ has demonstrated impressive compression ratios and performance metrics in various tests, establishing itself as a contender among well-known compression tools like gzip and zstd. For example, in experiments with the Canterbury corpus:
In addition to its compression capabilities, UCZ provides two robust tools:
ucz inspect: A residual entropy microscope that analyzes the data to assess its compressibility and reveals statistical properties.ucz ncd: Computes a normalized compression distance to measure dissimilarity between different data inputs for comparative analysis.Future releases will focus on performance optimization, including a C++ backend for enhanced speed and efficiency. The roadmap includes plans for streamable variants and improved encoding methods.
In summary, UCZ offers a novel approach to data compression, emphasizing universal learning and real-time adaptability. It provides both a practical engine for compression and a scientific tool for measuring the efficiency and efficacy of compression methods across different formats.
Comments
0Start the conversation
Share the first comment.