gemma-to-jev

Gemma-to-Jev is a Rust-based implementation that transforms Gemma (Gemma 3 4B) into a decision-making function modeled after Jev's System One. This project is designed to execute prompt-conditioned decisions with efficiency and precision, utilizing native Rust and leveraging the Candle framework on Apple Metal.

Key Features

• Efficient Decision-Making: Capable of warm decisions at a remarkable speed of approximately 47 ms per decision (21.2 decisions per second), making it suitable for high-performance applications.
• Stable Output: Ensures stability with a 7/7 option-order stability rating, guaranteeing reliable decision-making across varied runs.
• No Open-Ended Generation: The system simplifies the decision process, directly mapping prompt inputs to legal labels without any intermediate text generation or parsing.

How It Works

Instead of traditional token generation sequences, this implementation follows a streamlined process:

prompt -> legal-label logits -> softmax -> typed answer

The architecture efficiently directs inputs through a transformer, yielding output that directly informs decisions.

Usage

For seamless integration, the repository facilitates different modes of operation:

cargo run --release                      # Default conference demo
cargo run --release -- demo              # Demo with adjustable temperature
cargo run --release -- bench             # Performance benchmarking
cargo run --release -- serve             # Access local API compatible with TypeSafe

Decision Process

Utilizing a structured approach for question processing, every query is handled with clearly defined types and responses, ensuring precision in the decision context:

As an example, a standard decision request can be constructed and sent via a POST request to the local server:

curl http://127.0.0.1:8080/v1/systemone \
  -H 'content-type: application/json' \
  -d '{
    "state": "The build failed with: ld: library not found for -lssl",
    "questions": {
      "route": {
        "type": "choice",
        "instructions": "Choose the next owner.",
        "criteria": {
          "build": "Build or linker failure",
          "security": "Security incident",
          "remote_llm": "Needs deeper investigation"
        }
      }
    }
  }'

Performance Metrics

The implementation showcases impressive benchmark results:

• Shared-State KV Cache: 173 ms/decision
• Prefilled question sheet: 108 ms/decision
• Batched processing: 47 ms/decision

Comparison with Jev

While inspired by Jev's architecture, this implementation differs significantly in terms of output generation and processing efficiency. Notably, it does not utilize Jev's trained model but instead incorporates a rapid softmax-based decision mechanism tailored for responsive querying.

Future Directions

Future enhancements include calibrating probability outputs, fine-tuning with decision-specific datasets, and optimizing the execution performance further to meet increasing demands.

Explore the full scope of functionalities and outcomes on the project's official page.