OpenAI Embeddings Example

This repository provides a powerful solution for Retrieval Augmented Generation (RAG) and Hybrid Search using OpenAI's chatGPT capabilities. It leverages the Python OpenAI API Embeddings and integrates with a Pinecone vector database, orchestrated by LangChain.

Features

• System Prompting: Customize the behavior of language model text completions through intelligent system prompts.
• Template Creation: Maintain clear and reusable prompts with the DRY (Don't Repeat Yourself) principle by utilizing templates.
• LangChain Integration: Set up the project using LangChain, providing an alternative to vendor-specific LLM packages.
• PDF Loader Utility: A command-line tool that extracts text, vectorizes it, and loads it into a Pinecone database indexed to be compatible with OpenAI embeddings.
• Pinecone Vector Database: Learn how to create, manage, and query a Pinecone vector database tailored for embeddings.
• Retrieval Augmented Generation: Use hybrid search to find relevant documents in a vector database, enhancing OpenAI prompts by providing contextual data.

Usage Examples

The following command-line usage examples demonstrate the capabilities of this repository:

# Example 1 - Generic Assistant
python3 -m models.examples.prompt "you are a helpful assistant" "What analytics and accounting courses does Wharton offer?"

# Example 2 - Assistant with Improved System Prompting
python3 -m models.examples.prompt "You are a student advisor at University of Pennsylvania. You provide concise answers of 100 words or less." "What analytics and accounting courses does Wharton offer?"

# Example 5 - Retrieval Augmented Generation
python3 -m models.examples.load "/path/to/your/pdf/documents"
python3 -m models.examples.rag "What analytics and accounting courses does Wharton offer?"

Demonstration of RAG

For example, when prompted with "What analytics and accounting courses does Wharton offer?", the response generated by chatGPT may vary significantly with or without the use of embeddings from uploaded documents. By employing embeddings from a course's lecture notes, the results can reflect more accurate and relevant information.

Requirements

To utilize this project, ensure the following dependencies are installed:

• git
• make
• Python 3.12
• Access to the OpenAI API and Pinecone.

Additional Reading

For further information and tutorials, consider exploring resources such as:

• OpenAI Embeddings Guide
• Youtube - Vector Embeddings Tutorial – Code Your Own AI Assistant with GPT-4 API + LangChain + NLP
• Youtube - LangChain Explained in 13 Minutes | QuickStart Tutorial for Beginners
• OpenAI Embeddings
• What is a Vector Database?
• LangChain RAG
• LangChain Document Loaders
• LanchChain Caching