cmzip: Experimental Compressor for Context Mixing

cmzip is a general-purpose data compressor crafted from scratch, featuring a blend of advanced technologies including a binary arithmetic coder, various order-N context models, a match model similar to LZP, a logistic mixer, and dual APM calibration. This innovative compressor consistently outperforms commonly used compression methods such as zlib (ZIP) and bz2, achieving comparable performance to high-end lzma (7z/RAR) compression, occasionally exceeding it in efficiency.

Dual Engines

cmzip operates with two engines within a single file, cmzip.py:

• Python Engine: A pure implementation without dependencies, providing exact context tables for superior compression ratios.
• Numba Engine: A significantly faster alternative, achieving speeds approximately 40-45 times greater than the Python engine but with a slightly lower compression ratio (~5% less). To utilize the Numba engine, install it via pip install numpy numba.

By default, cmzip selects the Numba engine if available, reverting to the Python engine when it is not detected. The engine utilized is recorded in the .cmz header, ensuring that decompression is seamless and requires no additional information about the compression process.

Repo Structure

The repository includes:

cmzip.py          <- Standalone file, ready to use
src/               <- Modular source code for ease of reading and understanding
  arith.py          <- Binary arithmetic coder
  models.py         <- Context models, logistic mixer, match model, APM
  compressor.py     <- Pure Python engine (integrates all components)
  numba_compressor.py  <- Numba engine (same algorithm, fixed-size arrays)
  imagefilter.py    <- Paeth filter (spatial predictor, similar trick to PNG)
tests/
  test_arith.py     <- Round-trip test for the arithmetic coder

Usage Examples

Here are some simple commands to get started with cmzip:

pip install numpy numba   # optional for faster engine

python3 cmzip.py c input.txt output.cmz              # compress (automatic)
python3 cmzip.py d output.cmz restored.txt           # decompress

python3 cmzip.py c input.txt output.cmz --engine python   # force pure engine
python3 cmzip.py c image.raw output.cmz --image 120x120x3 # image mode (Paeth filter)

The first compression using the Numba engine compiles code (taking a few seconds), but it caches the result on disk, making subsequent runs much quicker.

How It Works

The cmzip compression process predicts the probability of each bit in the file using several order-N models based on the last N bytes seen. A match model identifies repetitive sequences, and a logistic mixer combines these predictions while learning online how much to trust each source. A final calibration stage (APM/SSE) adjusts the resulting probability. The arithmetic coder encodes the actual bit with this probability—better predictions lead to fewer bits required, achieving adaptiveness without transmitting any tables, allowing the decoder to reconstruct the same learning bit by bit in sync with the encoder.

Performance Overview

cmzip has been tested with the following results, confirming its efficiency across various use cases:

Compression Ratio Results (verified round-trip without loss)

Image Compression (using Paeth filter)

Speed Comparison (compressing 60,000 bytes)

Honest Limitations

• The Numba engine generally achieves a ~5% drop in compression ratio compared to the Python engine. This discrepancy is likely attributed to cumulative floating-point rounding errors rather than hash table collisions.
• Extensive testing has been conducted on text, code, and a small photo, including benchmarks that are reproducible. Real user testing with a mixed binary file of 921 MB demonstrated effective compression to ~300 MB within ~50 minutes via the Numba engine, with flawless decompression in ~10 minutes.
• In environments with limited available RAM, the compression buffer (up to 2x the input size) could become a bottleneck.
• When utilizing the --image mode, users must specify the dimensions and channels of the raw file, as it does not parse headers from formats like PNG or JPEG, which are already compressed.