NiceTryGPT

NiceTryGPT ☕🤖

Overview
NiceTryGPT is a powerful tool designed to enhance Capture the Flag (CTF) challenges by minimizing the effectiveness of shortcuts often exploited by language learning models (LLMs). This innovative solution focuses on maintaining a balance where challenges remain engaging and solvable for human participants while also making them less susceptible to automated solutions.

Key Features

• Minimal-Diff Workflow: NiceTryGPT introduces a streamlined process for altering existing CTF challenges, specifically targeting and reducing the use of cheap LLM shortcuts with minimal changes.
• Human-Friendly Transformations: Fundamental to NiceTryGPT’s design philosophy is the commitment to preserve the integrity and intended difficulty of the original challenges, ensuring they remain approachable for human solvers.
• Increased Uncertainty, Not Complexity: By implementing slight modifications to existing challenges, the project fosters an environment of unpredictability, enhancing the challenge without significantly increasing the overall complexity.

How It Works
The transformation process of NiceTryGPT follows a straightforward flow:

UNDERSTAND  
    ↓  
SOLVE ORIGINAL  
    ↓  
FIND ONE CHEAP SHORTCUT  
    ↓  
MAKE 0–2 SMALL CHANGES  
    ↓  
SOLVE AGAIN  
    ↓  
REPORT

This approach allows the tool to identify shortcuts without compromising the educational objectives, exposure to necessary skills, or the nature of the original challenge.

Installation and Usage
Users can easily integrate NiceTryGPT into their CTF challenges by copying the nice-try-gpt directory to their project. This setup allows users to effortlessly leverage the skill by executing:

Use NiceTryGPT on this CTF. Solve it first, identify the cheapest LLM shortcut, make the smallest useful change, and verify the result end-to-end.

For plugin installations with Claude Code, the necessary configuration is straightforward and ensures a seamless integration process across various platforms.

Current Work and Roadmap
The current version, v0.2.0, has successfully demonstrated its capabilities across three vulnerability classes: IDOR, path traversal, and SQL injection. The roadmap outlines future enhancements, including expanded independent evaluation protocols.

Example Transformations

• mini-idor:
  • Before: Flag is revealed through adjacent order IDs.
  • After: Requires observation of foreign order IDs at runtime.
• mini-traversal:
  • Before: Direct access via a static path.
  • After: Requires dynamic observation during challenge execution.
• mini-sqli:
  • Before: Brightly exposed admin identity.
  • After: Requires connecting elements from two application surfaces to derive identity.

Each transformation aims to keep the initial vulnerability intact while adjusting the environment so that LLMs cannot trivially solve the challenges.

Community and Contribution
NiceTryGPT is open for community feedback and contributions, especially from those involved in designing CTF challenges. Input on transformation effectiveness and balancing human cost factors is highly encouraged.

Conclusion
NiceTryGPT offers a unique approach to refining CTF challenges, ensuring they are both engaging for human participants and resistant to shortcut exploitation by LLMs. This tool not only promotes better challenge design but also contributes to the ongoing dialogue in the cybersecurity training community.