just bash wiki — A plugin for LLM maintained wikis with…

Pitch

Description

Comments

About this project

just-bash-wiki enhances the just-bash framework by providing an LLM-maintained wiki that allows for semantic search functionalities. It implements Karpathy's LLM Wiki pattern to create a structured knowledge base, making it easier to manage information effectively and enhance the productivity of users with seamless data querying.

just-bash-wiki is a versatile plugin for just-bash that brings the innovative LLM Wiki pattern to life, creating a persistent knowledge base powered by a language model (LLM) with semantic search capabilities.

Key Features

Structured Knowledge Base: The plugin allows for maintaining a well-organized database comprised of immutable source documents and LLM-generated wiki pages, ensuring that raw information remains untouched yet accessible.
Semantic Search: Integrated vector embeddings enable powerful semantic search across both sources and pages, enhancing discovery and retrieval.
Health Monitoring: The built-in linters and logs facilitate continuous oversight, identifying orphan pages, broken links, and missing sources efficiently.

Technology Stack

Built upon just-bash-data, just-bash-wiki merges data handling and vector operations, structured into three operational layers:

Layer	Storage Structure	Description
Sources	`db sources`	Immutable catalog of raw documents (articles, notes). Accessible for reading but not writable by the LLM.
Pages	`db pages` + `vec page_embeddings`	LLM-generated pages rich with information, interlinking, and embeddings for deep semantic search.
Log	`db log`	A comprehensive chronological log of all operations performed within the wiki system.

Core Operations

The core functionalities revolve around three main processes:

Ingesting Data: The ability to add sources, update wiki pages, and store embeddings.
Querying Data: Conducting semantic searches and synthesizing answers from relevant documents.
Linting: Conducting health checks on the wiki to ensure cleanliness and operational integrity.

Example Usage

Here’s a quick example demonstrating the ease of use in integrating just-bash-wiki:

import { Bash, InMemoryFs } from "just-bash";
import { createWikiPlugin } from "just-bash-wiki";

const bash = new Bash({
  fs: new InMemoryFs({}),
  customCommands: createWikiPlugin({ rootDir: "/wiki" }),
});

// Initialize wiki (creates collections + vector indexes)
await bash.exec(`wiki init --dim=1536`);
// Add a source
await bash.exec(`wiki source add '{"title":"AI Overview","type":"article","content":"..."}'`);
// Create a page
await bash.exec(`wiki page create '{"slug":"ai","title":"Artificial Intelligence","type":"concept","content":"# AI\\n...","tags":["ai"],"links_to":["ml"]}'`);
// Store embeddings for semantic search
await bash.exec(`wiki embed page ai '[0.1, 0.2, ...]'`);
// Search by vector similarity
await bash.exec(`wiki search '[0.1, 0.2, ...]' --k=5`);
// Health check
await bash.exec(`wiki lint`);

API Overview

The just-bash-wiki plugin exposes various commands for operations like collection management, source additions, page creations, embedding storage, and semantic searching, all aimed at making integration seamless and straightforward.

Insights and Statistics

Users have the ability to execute commands to check operational health and retrieve statistics about the wiki status, including the count of pages, sources, and recent activity, ensuring an informed and well-maintained knowledge base.

This plugin represents a significant step toward modern knowledge management, leveraging the capabilities of LLMs to maintain, search, and manage rich datasets using a concise command-line interface.