gemini-embeddings

What this skill does

# Gemini Embeddings

Generate high-quality text embeddings for semantic search, similarity analysis, clustering, and RAG (Retrieval Augmented Generation) applications through executable scripts.

## When to Use This Skill

Use this skill when you need to:
- Find semantically similar documents or texts
- Build semantic search engines
- Implement RAG (Retrieval Augmented Generation)
- Cluster or group similar documents
- Calculate text similarity scores
- Power recommendation systems
- Enable semantic document retrieval
- Create vector databases for AI applications

## Available Scripts

### scripts/embed.js
**Purpose**: Generate embeddings and calculate similarity

**When to use**:
- Creating vector representations of text
- Comparing text similarity
- Building semantic search systems
- Implementing RAG pipelines
- Clustering documents

**Key parameters**:
| Parameter | Description | Example |
|-----------|-------------|---------|
| `texts` | Text(s) to embed (required) | `"Your text here"` |
| `--model`, `-m` | Embedding model | `gemini-embedding-001` |
| `--task`, `-t` | Task type | `SEMANTIC_SIMILARITY` |
| `--dim`, `-d` | Output dimensionality | `768`, `1536`, `3072` |
| `--similarity`, `-s` | Calculate pairwise similarity | Flag |
| `--json`, `-j` | Output as JSON | Flag |

**Output**: Embedding vectors or similarity scores

## Workflows

### Workflow 1: Single Text Embedding
```bash
node scripts/embed.js "What is the meaning of life?"
```
- Best for: Basic embedding generation
- Output: Vector with 3072 dimensions (default)
- Use when: Storing single document vectors

### Workflow 2: Semantic Search
```bash
# 1. Generate embedding for query
node scripts/embed.js "best practices for coding" --task RETRIEVAL_QUERY > query.json

# 2. Generate embeddings for documents (batch)
node scripts/embed.js "Coding best practices include version control" "Clean code is essential" --task RETRIEVAL_DOCUMENT > docs.json

# 3. Compare and find most similar (calculate similarity separately)
```
- Best for: Building search functionality
- Task types: `RETRIEVAL_QUERY`, `RETRIEVAL_DOCUMENT`
- Combines with: Similarity calculation for ranking

### Workflow 3: Text Similarity Comparison
```bash
node scripts/embed.js "What is the meaning of life?" "What is the purpose of existence?" "How do I bake a cake?" --similarity
```
- Best for: Comparing multiple texts, finding duplicates
- Output: Pairwise similarity scores (0-1)
- Use when: Need to rank text similarity

### Workflow 4: Dimensionality Reduction for Efficiency
```bash
node scripts/embed.js "Text to embed" --dim 768
```
- Best for: Faster storage and comparison
- Options: `768`, `1536`, or `3072` (default)
- Trade-off: Lower dimensions = less accuracy but faster

### Workflow 5: Document Clustering
```bash
# 1. Generate embeddings for multiple documents
node scripts/embed.js "Machine learning is AI" "Deep learning is a subset" "Neural networks power AI" --json > embeddings.jsonl

# 2. Process embeddings with clustering algorithm (your code)
# Use scikit-learn, KMeans, etc.
```
- Best for: Grouping similar documents, topic discovery
- Task type: `CLUSTERING`
- Combines with: Clustering libraries (scikit-learn)

### Workflow 6: RAG Implementation
```bash
# 1. Create document embeddings (one-time setup)
node scripts/embed.js "Document 1 content" "Document 2 content" --task RETRIEVAL_DOCUMENT --dim 1536

# 2. For each query, find similar documents
node scripts/embed.js "User query here" --task RETRIEVAL_QUERY

# 3. Use retrieved documents in prompt to LLM (gemini-text)
node skills/gemini-text/scripts/generate.js "Context: [retrieved docs]. Answer: [user query]"
```
- Best for: Building knowledge-based AI systems
- Combines with: gemini-text for generation with context

### Workflow 7: JSON Output for API Integration
```bash
node scripts/embed.js "Text to process" --json
```
- Best for: API responses, database storage
- Output: JSON array of embedding vectors
- Use when: Programmatic processing required

### Workflow 8: Batch Document Processing
```bash
# 1. Create JSONL with documents
echo '{"text": "Document 1"}' > docs.jsonl
echo '{"text": "Document 2"}' >> docs.jsonl

# 2. Process with script or custom code
python3 << 'EOF'
import json
from google import genai

client = genai.Client()

texts = []
with open("docs.jsonl") as f:
    for line in f:
        texts.append(json.loads(line)["text"])

response = client.models.embed_content(
    model="gemini-embedding-001",
    contents=texts,
    task_type="RETRIEVAL_DOCUMENT"
)

embeddings = [e.values for e in response.embeddings]
print(f"Generated {len(embeddings)} embeddings")
EOF
```
- Best for: Large document collections
- Combines with: Vector databases (Pinecone, Weaviate)

## Parameters Reference

### Task Types

| Task Type | Best For | When to Use |
|-----------|----------|-------------|
| `SEMANTIC_SIMILARITY` | Comparing text similarity | General comparison tasks |
| `RETRIEVAL_DOCUMENT` | Embedding documents | Storing documents for retrieval |
| `RETRIEVAL_QUERY` | Embedding search queries | Finding similar documents |
| `CLASSIFICATION` | Text classification | Categorizing text |
| `CLUSTERING` | Grouping similar texts | Document clustering |

### Dimensionality Options

| Dimensions | Use Case | Trade-off |
|------------|----------|-----------|
| 768 | High-volume, real-time | Lower accuracy, faster |
| 1536 | Balanced performance | Good accuracy/speed balance |
| 3072 | Highest accuracy | Slower, more storage |

### Similarity Scores

| Score | Interpretation |
|-------|---------------|
| 0.8 - 1.0 | Very similar (likely duplicates) |
| 0.6 - 0.8 | Highly related (same topic) |
| 0.4 - 0.6 | Moderately related |
| 0.2 - 0.4 | Weakly related |
| 0.0 - 0.2 | Unrelated |

## Output Interpretation

### Embedding Vector
- Format: List of float values (768, 1536, or 3072)
- Range: Typically -1.0 to 1.0
- Normalized for cosine similarity
- Can be stored in vector databases

### Similarity Output
```
Pairwise Similarity:
  'What is the meaning of life?...' <-> 'What is the purpose of existence?...': 0.8742
  'What is the meaning of life?...' <-> 'How do I bake a cake?...': 0.1234
```
- Higher scores = more similar
- Use threshold (e.g., 0.7) for matching

### JSON Output
```json
[[0.123, -0.456, 0.789, ...], [0.234, -0.567, 0.890, ...]]
```
- Array of embedding vectors
- One per input text
- Ready for database storage

## Common Issues

### "google-genai not installed"
```bash
npm install @google/genai@latest dotenv@latest
```

### "numpy not installed" (for similarity)
```bash
pip install numpy
```

### "Invalid task type"
- Use available tasks: SEMANTIC_SIMILARITY, RETRIEVAL_DOCUMENT, RETRIEVAL_QUERY, CLASSIFICATION, CLUSTERING
- Check spelling (case-sensitive)
- Use correct task for your use case

### "Invalid dimension"
- Options: 768, 1536, or 3072 only
- Check model supports requested dimension
- Default to 3072 if unsure

### "No similarity calculated"
- Need multiple texts for similarity comparison
- Use `--similarity` flag
- Check that at least 2 texts provided

### "Embedding size mismatch"
- All embeddings must have same dimensionality
- Use consistent `--dim` parameter
- Recompute if dimensions differ

## Best Practices

### Task Selection
- **SEMANTIC_SIMILARITY**: General text comparison
- **RETRIEVAL_DOCUMENT**: Storing documents for search
- **RETRIEVAL_QUERY**: Querying for similar documents
- **CLASSIFICATION**: Categorization tasks
- **CLUSTERING**: Grouping similar content

### Dimensionality Choice
- **768**: Real-time applications, high volume
- **1536**: Balanced choice for most use cases
- **3072**: Maximum accuracy, offline processing

### Performance Optimization
- Use lower dimensions for speed
- Batch multiple texts in one request
- Cache embeddings for repeated queries
- Precompute document embeddings for search

### Storage Tips
- Use vector databases (Pinecone, Weaviate, Chroma)
- Normalize vectors for consistent comparison
- Store