Story Crowdsource Preference System

The Story Crowdsource Preference System is an innovative open-source platform designed to collect, analyze, and learn from preferences regarding story variations. By combining cutting-edge AI models with valuable human feedback, this system aims to significantly enhance story generation and preference learning.

Key Components

• Story Generation: Leverages advanced AI models such as GPT-4 and Gemini to produce diverse story variations from initial prompts.
• Feedback Collection: Features an engaging web interface for gathering user preferences between different story versions.
• Embedding Processing: Efficiently generates and stores embeddings related to the story variations.
• Reward Model Training: Implements a robust training system for a reward model based on user feedback.
• Dataset Generation: Crafts curated preference datasets apt for Direct Preference Optimization (DPO).

Highlights

• Open Source Dataset: A commitment to sharing the resulting dataset with the community, benefiting both technology enthusiasts and writers.
• Live Demo: Engage with the platform through the Story Preference Collection App and contribute preferences to shape the dataset.
• Modern Architecture: Built using Streamlit for user interaction and Supabase for efficient data management.
• Benchmarking Tool: Serves as a vital resource for benchmarking Large Language Models (LLMs) in the creative writing domain.

Features

• Multi-model story generation with flexible provider choices.
• User-friendly web interface for easy story comparison and feedback submission.
• Automated embedding generation for seamless processing of story variations.
• Adaptable reward model training that incorporates weighted sources of feedback.
• Capability to export datasets formatted for DPO compatibility.
• Integration with Supabase to ensure robust data storage and management.

Participation and Contribution

• Star the Repository: Signify support by starring the repository to stay informed about the dataset release.
• Input Preferences: Visit the Story Preference Collection App to share story preferences and enrich the developing dataset.

Components Overview

1. Story Dataset Generator: Handles story prompt and variation generation using multiple AI models and facilitates evaluations integrated with Supabase.
2. Feedback Collection App: A Streamlit-based web application allowing users to evaluate and rate story variations, exporting feedback in DPO format.
3. Embedding Processor: Utilizes ModernBERT for embedding generation and efficiently manages storage through Supabase.
4. Reward Model Training: Focuses on learning preferences from feedback while supporting advanced configurations for tailored training methodologies.

Configuration Examples

Provider Configuration:
Customize AI provider settings:

PROVIDERS_CONFIG = {
    "prompt": {
        "provider": "github",
        "model": "openai/gpt-4o-mini"
    },
    "variations": [
        {
            "provider": "gemini",
            "model": "gemini/gemini-2.0-flash-exp"
        },
        {
            "provider": "azure",
            "model": "azure/gpt-4o-mini"
        }
    ]
}

Training Configuration:
Example parameters for training:

config = {
    'batch_size': 4,
    'num_epochs': 5,
    'learning_rate': 3e-5,
    'test_size': 0.2,
    'source_weights': {
        'model': 1.0,   # Weight from reward model feedback
        'llm': 0.5,     # Weight from LLM feedback
        'human': 2.0    # Enhanced weight for human feedback
    }
}

Contributing

Contributions are encouraged and welcomed. Submissions via Pull Requests are highly appreciated, helping to enhance the project's capabilities.