@pi-unipi/cocoindex

CocoIndex integration for Pi coding agent — AST-aware content indexing, semantic vector search, and incremental pipeline management.

Overview

Replaces the compactor's FTS5-based content indexing with CocoIndex, providing:

• AST-aware code chunking — language-aware splitting for code files
• Semantic vector search — find content by meaning, not just keywords
• Incremental indexing — only reprocesses changed files (delta-only)
• LanceDB storage — zero-config, local file-based vector database
• Shared embeddings — reuses memory package's OpenRouter API key and model

Prerequisites

1. Python 3.10+
2. CocoIndex CLI: pip install cocoindex 'cocoindex[lancedb]' (requires cocoindex >= 1.0)
3. LanceDB SDK (optional, for search): npm install @lancedb/lancedb
4. Embedding API key — configured via /unipi:memory-settings

Quick Start

# 1. Initialize the pipeline (once per project)
/unipi:cocoindex-init

# 2. Index the project
/unipi:cocoindex-update

# 3. Search indexed content
cocoindex_search({ query: "how does authentication work?" })

Architecture

Project files ──→ localfs.walk_dir (recursive)
                      │
                      ▼
              chunk_text (@coco.fn, memoized)
                      │
                      ▼
              LanceDB target (via ContextKey)
                      │
                      ▼
              Vector search → ranked results

Uses cocoindex v1.0+ App/fn/mount API with:

• @coco.lifespan for async environment setup (LanceDB connection)
• @coco.fn for memoized processing functions
• coco.mount() / coco.mount_target() for component management
• localfs.walk_dir for file enumeration
• lancedb.TableTarget for row-level target state management

Tools

Tool	Description
cocoindex_search	Search indexed content (semantic vector when available, LanceDB FTS when available, lexical fallback for text-only indexes)
cocoindex_status	Check indexing status, freshness, doc count

Commands

Command	Description
/unipi:cocoindex-update	Run incremental indexing
/unipi:cocoindex-status	Show pipeline status
/unipi:cocoindex-init	Scaffold default pipeline
/unipi:cocoindex-settings	View configuration

Configuration

• Pipeline: .unipi/cocoindex/main.py — auto-generated, fully customizable
• Data store: .unipi/cocoindex/.lancedb/
• Embeddings: ~/.unipi/memory/config.json (shared with memory package)
• Search fallback: Existing text-only LanceDB tables remain searchable through a lexical scan fallback when no vector column or FTS index exists

What Changed from FTS5

This package replaces compactor's content indexing subsystem:

Feature	Before (FTS5)	After (CocoIndex)
Chunking	Heading/paragraph	AST-aware recursive
Search	BM25 + trigram	Vector + full-text
Incremental	No (full re-index)	Yes (delta-only)
Storage	SQLite FTS5	LanceDB

Status

⚠️ Experimental — This is an experiment/cocoindex branch feature. Not yet merged to main.