proof-ate-the-project

Project Overview

proof-ate-the-project is an innovative experiment that explores the capabilities of AI in software development through a continuous verification process. Over the span of fifteen days, this project utilizes an AI that autonomously builds a software application, while being monitored and audited in real-time by a different AI model. The guiding principle of this endeavor is clear: prove everything.

Key Components

The project is built around two essential tools:

1. idea-to-ship: A set of specialized agent skills that enable the transition of a project from initial ideas through extensive research and planning, finally culminating in a polished version 1 of the software. This framework includes an autopilot feature which allows the process to run entirely unattended.
2. didrun: A groundbreaking tool that functions as a flight recorder, verifying every command that the AI executes. It captures what has been done, ensuring that no assumptions are made without solid proof. This tool is crucial in maintaining accountability, especially during lengthy, autonomous builds.

Experiment Insights

The project began with a simple yet ambitious premise: to run the AI-powered pipeline on Fable 5 and deliver a functioning application in a record time of just over 13.5 hours. Contrasting this success, the use of a later model, Codex 5.6, paired with the didrun tool, presented challenges that prolonged the timeline significantly beyond initial expectations. Proving the project required the AI to operate under strict adherence to the defined rules — no claims could be made without adequate recording of their execution.

The Findings

The experiment highlights some critical observations about the cost of maintaining verification over time. It revealed that the effort to provide proof of work tends to increase in correlation with the duration of the project, rather than the amount of product being built. Early milestones were reached swiftly, only to witness a slowdown as the overhead from verifying the work grew. Ultimately, the AI was faced with a significant overhead cost due to the rigorous maintenance of checks and balances that proved to be a project in its own right.

Current Status

As of now, the project is ongoing, with the public repository available here: Countershape project. The ledger of commits, along with supporting documentation, is accessible for review, providing insight into the project's evolution and the unique challenges faced by the AI system.

Conclusion

proof-ate-the-project stands as a pioneering exploration into how AI can autonomously manage software development while also rigorously verifying its own outputs. By establishing a rigorous framework for accountability, this project raises significant questions and paves the way for future innovations in AI-driven software engineering.