Benchmark-GPT-5-vs-Claude-4-Sonnet-on-200-Requests

Benchmarking GPT-5 vs. Claude 4 Sonnet: A Detailed Evaluation Study

This project presents a systematic evaluation of two leading large language models (LLMs): GPT-5 and Claude 4 Sonnet. The comparative study was conducted using 200 diverse prompts that encompass various domains such as reasoning, coding, analysis, knowledge, writing, and safety-critical scenarios, utilizing the Cubent VS Code Extension.

Key Findings

The evaluation highlights crucial insights into the performance of both models:

• Speed: Claude 4 Sonnet consistently outperforms GPT-5 with a median response time of 5.1 seconds compared to GPT-5's 6.4 seconds.
• Precision: Sonnet achieves slightly higher factual precision (93.2%) over GPT-5 (91.4%) and exhibits a lower hallucination rate of 6.8% versus GPT-5's 8.1%.
• Overall Quality: Although GPT-5 shows a higher overall task success rate (86%) compared to Sonnet (84%), it excels particularly in complex multi-step reasoning and code generation/debugging tasks.
• Safety & Refusals: Sonnet demonstrates better refusal correctness with 96%, while both models maintain robust safety compliance.
• Domain Performance: Sonnet leads in tasks such as summarization and short-form Q&A, while GPT-5 shows strengths in complex reasoning and data analysis tasks.

Repository Structure

The repository is organized as follows:

gpt5-vs-claude4-eval/
├── README.md                  # This file
├── LICENSE                    # MIT License
├── requirements.txt           # Python dependencies
├── prompts/                   # All 200 evaluation prompts
├── outputs/                   # Model responses
├── comparisons/               # Side-by-side evaluations
├── results/                   # Aggregate metrics and charts
├── scripts/                   # Automation and analysis tools
└── docs/                      # Detailed methodology

Evaluation Domains

The study covers various evaluation domains, including:

• Reasoning & Math - Tasks focused on logic and mathematical problem-solving.
• Coding & Debugging - Various programming challenges, including debugging and code review.
• Data Analysis - Statistical analysis and chart interpretation.
• Knowledge & Fact-Checking - Verification of factual accuracy and sources.
• Summarization & Editing - Improvements in text compression and style.
• Safety & Policy Edge Cases - Tests for harmful content and refusal scenarios.

Metrics Used

The evaluation uses several key performance metrics:

• Task Success (TS) - A measure of success rates for tasks.
• Factual Precision (FP) - Proportion of verifiable claims provided by the models.
• Reasoning Quality (RQ) - Assessed on a scale of 1-5 for logical structure.
• Helpfulness (H) - User-oriented rating of utility.
• Conciseness (Cnc) - Efficiency in communication.
• Hallucination Rate - Percentage of unsupported claims made by the models.
• Safety/Refusal Correctness - Accuracy in policy compliance.
• Latency - Response times measured using p50/p90/p95.

The findings from this study contribute valuable insights into the capabilities of GPT-5 and Claude 4 Sonnet, guiding further research and applications in the field of AI.