GitHub Repository to Single File

Efficiently consolidate a GitHub repository into a single, clean document with the github-repo-to-single-file TypeScript CLI tool. This utility fetches and merges all text-like files from a GitHub repository, outputting the results as either a .txt or .pdf file. Designed for seamless integration with Language Models (LLMs) and Retrieval-Augmented Generation (RAG) tasks, the tool allows for easy ingestion of an entire codebase as a single artifact, ensuring that models can seamlessly use the context from the repository.

Key Features

• Streamlined Output: Saves the generated document directly to the out/ directory, simplifying access to the final product.
• Customizable Fetching: Supports specific branches or tags during the fetch process, while defaulting to the repository's main branch if none is specified.
• Progress Streaming: Provides real-time updates on the download process, detailing current and total items to be processed.
• Smart Handling of Large Files: Automatically skips large files (over 5 MB) and binary blobs to ensure efficiency and speed.
• Error Recovery: Implements automatic retries for temporary network issues and respects GitHub rate limits, enhancing the robustness of the fetching process.
• Extensive Testing: Includes a comprehensive test suite using Jest to guarantee reliability and performance of the core functionalities.

Usage Examples

Fetch a repository:

npm run fetch -- https://github.com/owner/repository

Generate a PDF instead of text:

npm run fetch -- --pdf https://github.com/owner/repository

Specify a branch for fetching:

npm run fetch -- --branch release https://github.com/owner/repository

Project Structure

• main.ts: Entry point for the CLI.
• src/: Contains modular implementations, including the GitHub client and exporter logic.
• out/: Directory for generated output files.
• __tests__/: Jest test suites to validate core helpers and functionalities.

*By leveraging this tool, developers can quickly gather context from their repositories, facilitating enhanced model training and comprehension through the consolidation of repository data into a single file format.