GPT Watermark

The gpt-watermark project is a cutting-edge experiment in watermarking generated text, developed on the foundation of Andrej Karpathy's minimal microgpt. This innovative approach demonstrates how a secret-keyed sampler can create a statistically detectable watermark within the text, without altering the model's inherent token probabilities.

Key Features

• Pure Python Implementation: This project utilizes only Python, ensuring ease of use and simplicity.
• No External Dependencies: There are no library dependencies beyond the standard Python 3 library, making it lightweight and straightforward.
• Minimal Codebase: The implementation consists of only a few lines of code, facilitating easy understanding and utilization.

How It Works

In the realm of large language models (LLMs), text generation involves sampling the next token based on a probability distribution. There are two modes of generation:

• Normal Generation:
  GPT → probabilities → random sampling → next token
• Watermarked Generation:
  GPT → same probabilities → keyed Gumbel sampling → next token

The watermark does not manifest itself as a visible marker in the text. Instead, it emerges statistically through numerous token selections and can be detected with the corresponding secret key.

Usage Example

To experiment with the watermarking functionality, clone the repository and run the script:

git clone https://github.com/berba-q/gpt-watermark
cd gpt-watermark
python3 microgpt_watermark.py

Demonstrated Results

Both text generation methods yield outputs that appear equally natural:

Normal samples:      | Watermarked samples:
--------------------|--------------------
yuh | yuha           | yuhan
xav | xavinn         | xavia
jua | juan           | juale

However, when subjected to detection, the watermark's significance is evident:

Normal       n=308  mean=0.962  p=7.459e-01
Watermarked  n=322  mean=2.105  p=6.844e-53
Wrong key    n=322  mean=0.984  p=6.094e-01

The watermarked text produces a p-value of approximately ~6.8×10⁻⁵³, demonstrating a significant difference from ordinary text, which scores around p≈0.6-0.75, indistinguishable from random chance.

For a comprehensive explanation, refer to Watermarking a tiny GPT: 91 lines, NO Frameworks.

Acknowledgments

This project exists thanks to the foundational work of Andrej Karpathy's microgpt and is inspired by Scott Aaronson's research on keyed Gumbel sampling for LLM output watermarking.