chromadb-integration-skills

What this skill does

# ChromaDB Integration Skills

**Purpose**: This skill teaches agents how to integrate ChromaDB for semantic search, persistent storage, and pattern matching across ANY domain - research, code, trading, legal, documentation, and more.

**Critical Use Case**: When agents need to work with large datasets (1000+ items), perform semantic search, maintain persistent knowledge, or learn from historical patterns, ChromaDB eliminates token limits and enables powerful vector-based retrieval.

**Used By**: All agent types - researchers, developers, traders, legal analysts, documentation writers, QA testers, etc.

---

## When to Use ChromaDB Integration

Use ChromaDB when:
- **Large Datasets**: Working with 1000+ items (documents, code files, bugs, trades, contracts, etc.)
- **Semantic Search**: Finding items by meaning, not just keywords
- **Persistent Memory**: Knowledge needs to survive across sessions, days, months
- **Pattern Matching**: Identifying similar historical cases/patterns for decision-making
- **Cross-Session Learning**: Building institutional knowledge over time
- **Token Limits**: Data too large to fit in context window (100K+ tokens)
- **Aggregation**: Combining results from multiple queries/sources

---

## Core ChromaDB Concepts

### Collections
**Definition**: Named vector databases storing documents with embeddings and metadata

**Naming Strategy**:
- **Domain-based**: `{domain}_{purpose}_{identifier}`
- **Examples**:
  - Research: `research_prior_art_blockchain_2024`, `research_literature_ml_transformers`
  - Code: `codebase_api_endpoints`, `codebase_bug_patterns_auth`
  - Trading: `backtest_results_sma_strategy`, `market_conditions_spy_2024`
  - Legal: `case_law_patent_eligibility`, `contracts_saas_clauses`
  - Documentation: `api_docs_v2`, `architecture_decisions_2024`

### Documents
**Definition**: Text content to be searched semantically

**Best Practices**:
- **Chunk Size**: 200-500 words optimal (too small = context loss, too large = poor granularity)
- **Content Format**: Title + summary + key details (e.g., `"Patent US10123456 - Blockchain Authentication. Abstract: A method for..."`))
- **Deduplication**: Use unique IDs to prevent duplicate storage

### Metadata
**Definition**: Structured data for filtering, not semantic search

**Strategy**:
```javascript
{
  // Temporal filters
  "date": "2024-11-14",
  "year": 2024,
  "month": 11,

  // Categorical filters
  "type": "bug_report",
  "category": "authentication",
  "severity": "high",

  // Numeric filters
  "citations": 42,
  "price": 150.25,
  "performance_score": 0.87,

  // Source tracking
  "source": "github_issue",
  "author": "kim-asplund",
  "url": "https://..."
}
```

### Embeddings
**Definition**: Vector representations enabling semantic similarity

**How It Works**:
- ChromaDB automatically generates embeddings from document text
- Similar meanings → similar vectors → close in vector space
- Distance metrics (cosine, euclidean) measure similarity

---

## Universal ChromaDB Workflow

### Phase 1: Collection Design

```javascript
// Step 1: Design collection strategy based on agent type
const collectionStrategy = {
  research_agent: "One collection per research topic/question",
  code_agent: "Collections by codebase module/feature",
  trading_agent: "Collections by strategy/timeframe/symbol",
  legal_agent: "Collections by practice area/jurisdiction",
  documentation_agent: "Collections by project/version"
};

// Step 2: Create collection with descriptive metadata
mcp__chroma__create_collection({
  collection_name: "{domain}_{purpose}_{identifier}",
  embedding_function_name: "default",  // Uses sentence transformers
  metadata: {
    created_date: "2024-11-14",
    domain: "research|code|trading|legal|docs",
    purpose: "Descriptive purpose",
    total_items: 0,  // Will update
    last_updated: "2024-11-14"
  }
});
```

### Phase 2: Data Ingestion

```javascript
// Step 1: Batch data collection (minimize API calls)
const items = collectAllItems();  // From API, files, database, etc.

// Step 2: Transform to ChromaDB format
const documents = items.map(item => formatDocument(item));
const ids = items.map(item => item.id || generateUniqueId());
const metadatas = items.map(item => extractMetadata(item));

// Step 3: Batch insert (ChromaDB handles chunking automatically)
mcp__chroma__add_documents({
  collection_name: collectionName,
  documents: documents,
  ids: ids,
  metadatas: metadatas
});

// Step 4: Update collection metadata
mcp__chroma__modify_collection({
  collection_name: collectionName,
  new_metadata: {
    ...existingMetadata,
    total_items: items.length,
    last_updated: new Date().toISOString()
  }
});
```

### Phase 3: Semantic Search

```javascript
// Step 1: Formulate semantic query (natural language works!)
const query = "authentication failures in production environment";

// Step 2: Execute semantic search with filters
const results = mcp__chroma__query_documents({
  collection_name: collectionName,
  query_texts: [query],
  n_results: 20,
  where: {
    "$and": [
      { "environment": "production" },
      { "severity": { "$in": ["high", "critical"] } },
      { "date": { "$gte": "2024-01-01" } }
    ]
  },
  include: ["documents", "metadatas", "distances"]
});

// Step 3: Filter by semantic similarity (distance threshold)
const highlyRelevant = results.ids[0].filter((id, idx) =>
  results.distances[0][idx] < 0.3  // Adjust threshold based on use case
);

// Step 4: Retrieve full details if needed
const fullDetails = mcp__chroma__get_documents({
  collection_name: collectionName,
  ids: highlyRelevant,
  include: ["documents", "metadatas"]
});
```

### Phase 4: Pattern Matching

```javascript
// Cross-collection pattern detection
const allCollections = mcp__chroma__list_collections();
const relevantCollections = allCollections.filter(c =>
  c.startsWith(collectionPrefix)
);

const patterns = [];
for (const collection of relevantCollections) {
  const matches = mcp__chroma__query_documents({
    collection_name: collection,
    query_texts: [patternQuery],
    n_results: 10,
    where: { "outcome": "success" }  // Only successful cases
  });

  if (matches.ids[0].length > 0) {
    patterns.push({
      collection: collection,
      matches: matches,
      success_rate: calculateSuccessRate(matches)
    });
  }
}

// Identify best pattern
const bestPattern = patterns.sort((a, b) =>
  b.success_rate - a.success_rate
)[0];
```

---

## Use Case Templates

### Template 1: Research Agent - Literature Review

**Problem**: Store 1000+ research papers, find semantically similar work

```javascript
// Collection: research_literature_{topic}
const papers = fetchPapersFromAPI("machine learning transformers");

mcp__chroma__create_collection({
  collection_name: "research_literature_ml_transformers",
  metadata: { topic: "ML Transformers", papers_count: 0 }
});

// Store papers with rich metadata
papers.forEach(paper => {
  mcp__chroma__add_documents({
    collection_name: "research_literature_ml_transformers",
    documents: [`${paper.title}. ${paper.abstract}`],
    ids: [paper.doi || paper.id],
    metadatas: [{
      title: paper.title,
      authors: paper.authors.join(", "),
      year: paper.year,
      citations: paper.citation_count,
      venue: paper.venue,
      url: paper.url
    }]
  });
});

// Semantic search: "Find papers about attention mechanisms for vision"
const relevant = mcp__chroma__query_documents({
  collection_name: "research_literature_ml_transformers",
  query_texts: ["attention mechanisms computer vision"],
  n_results: 20,
  where: { "year": { "$gte": 2020 }, "citations": { "$gte": 50 } }
});
```

**Benefits**: No token limits, semantic discovery, citation filtering, persistent library

---

### Template 2: Code Agent - Bug Pattern Recognition

**Problem**: Store bug reports, identify similar issues, suggest solutions

```javascript
// Collection: codebase_bug_patterns_{module}
const bugs = fetchAllGitHubIssues("is:issue