embeddingGemma2 jeff — Efficiently porting jeffhub use…

Pitch

Description

Comments

About this project

This project provides ports of jeffhub.ai use cases to the EmbeddingGemma-2 model by Google. It features a runnable demo and benchmarks for latency and quality performance, allowing users to effortlessly see how embedding technology can enhance their applications. With a focus on efficiency and ease of integration, it caters to various use cases, ensuring high accuracy and performance.

Project Overview

The embeddingGemma2-jeff repository provides a seamless integration of use cases from jeffhub.ai with Google's EmbeddingGemma-2 embedding model. It features both a runnable demo and a comprehensive performance benchmarking system that evaluates latency and quality metrics.

Core Features

This project utilizes jeffhub's 0.8B "System 1" decider model, swapping the traditional decider for the EmbeddingGemma-2 embeddings. Each model adapter is designed to process input alongside a list of options, returning a probability for each option based on either cosine similarity with labeled exemplars for classification tasks or dense passage ranking for retrieval tasks.

Implemented Use Cases

jeffhub Adapter	Category	Ported Use Case	Quality Metric
`ground`	Retrieval	Dense passage retrieval/re-ranking	Recall@1, Recall@5, MRR
`support-intents`	Support	Nearest-exemplar intent classification	Accuracy, Macro-F1
`spam`	Safety	Legitimate/spam/phishing classification	Accuracy, Macro-F1
`triage`	Support	Ticket routing to a team	Accuracy, Macro-F1

Technical Specifications

The implementation environment is optimized for performance, featuring:

Node.js with @huggingface/transformers v4 (running on CPU without GPU).
The EmbeddingGemma-2 model operates in an int8 quantized format (approximately 314 MB in size).

EmbeddingGemma-2, comprising 740 million parameters (including a 270 million text backbone), effectively maps text into a unified 768-dimensional space using task-steering through specific instruction prefixes:

Purpose	Prefix
Retrieval Query	`task: search result
Document	`title: none
Classification	`task: classification

Example Output

Running the demo provides insightful results, with use cases demonstrating the process effectively:

jeffhub use case: ground  (Retrieval — pick the passage that answers)
Q: Which planet is known as the Red Planet?
  1. [0.8068] d1  Mars, known for its reddish appearance, is often referred to as ...
  2. [0.7056] d3  Jupiter is the largest planet in the solar system, with a ...
  3. [0.6650] d2  Venus is often called Earth's twin because of its similar size ...

jeffhub use case: spam
Input: Your account has been suspended, verify your password immediately at the link below.
  -> phishing  (confidence 99.7%)
     options: phishing=99.7% spam=0.2% legitimate=0.1%

Performance Metrics

A rigorous benchmarking process is implemented to measure:

Latency on CPU-only, single process, showcasing:
- p50 168 ms | p95 177 ms | mean 168 ms (n=30)
Throughput of 24 documents per second in batched processing.
Quality analysis provides perfect accuracy rates (100%) across all evaluated use cases due to specially curated datasets.

Limitations

While demonstrating impressive capabilities, certain limitations include:

Dependency on CPU for throughput; no GPU optimizations have been applied.
The dataset size is intentionally small, aimed at demonstrating functionality rather than serving as a definitive benchmark.