CocoIndex — Open source ETL framework to index fresh…

Pitch

Description

Comments

About this project

The world's first open source engine that supports both custom transformation logic and incremental updates specialized for data indexing. Allow you to build your RAG pipeline like LEGO.

We are thrilled to announce the open-source release of CocoIndex, the world's first engine that supports both custom transformation logic and incremental updates specialized for data indexing.

CocoIndex

CocoIndex is an ETL framework to prepare data for AI applications such as semantic search, retrieval-augmented generation (RAG). It offers a data-driven programming model that simplifies the creation and maintenance of data indexing pipelines, ensuring data freshness and consistency.

🔥 Key Features

Data Flow Programming: Build indexing pipelines by composing transformations like Lego blocks, with built-in state management and observability.
Support Custom Logic: Plug in your choice of chunking, embedding, and vector stores. Extend with custom transformations like deduplication and reconciliation.
Incremental Updates: Smart state management minimizes re-computation by tracking changes at the file level, with future support for chunk-level granularity.
Python SDK: Built with a RUST core 🦀 for performance, exposed through an intuitive Python binding 🐍 for ease of use.

Getting Started

An engaging onboarding experience is available through the Quick Start Guide and a quick start video tutorial.

Example Usage

The following code snippet illustrates how to define a simple indexing flow with CocoIndex:

@cocoindex.flow_def(name="TextEmbedding")
def text_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
    data_scope["documents"] = flow_builder.add_source(cocoindex.sources.LocalFile(path="markdown_files"))
    doc_embeddings = data_scope.add_collector()

    with data_scope["documents"].row() as doc:
        doc["chunks"] = doc["content"].transform(
            cocoindex.functions.SplitRecursively(),
            language="markdown", chunk_size=300, chunk_overlap=100)

        with doc["chunks"].row() as chunk:
            chunk["embedding"] = chunk["text"].transform(
                cocoindex.functions.SentenceTransformerEmbed(
                    model="sentence-transformers/all-MiniLM-L6-v2"))
            doc_embeddings.collect(filename=doc["filename"], location=chunk["location"],
                                   text=chunk["text"], embedding=chunk["embedding"])

    doc_embeddings.export(
        "doc_embeddings",
        cocoindex.storages.Postgres(),
        primary_key_fields=["filename", "location"],
        vector_index=[("embedding", cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)])

Continual enhancements and additional examples await, along with active community support available through the Discord community.

Documentation

For comprehensive insights and detailed instructions, refer to the CocoIndex Documentation which includes a dedicated Quickstart guide.

Contribution and Community Engagement

CocoIndex thrives on diverse contributions from the community. Interested parties can review the contributing guide for information on participation.
Engagement avenues include: