vector-databases

What this skill does

# Vector Databases

## When to Use This Skill

Use this skill when:

- Choosing between vector database options
- Designing semantic/similarity search systems
- Optimizing vector search performance
- Understanding ANN algorithm trade-offs
- Scaling vector search infrastructure
- Implementing hybrid search (vectors + filters)

**Keywords:** vector database, embeddings, vector search, similarity search, ANN, approximate nearest neighbor, HNSW, IVF, FAISS, Pinecone, Weaviate, Milvus, Qdrant, Chroma, pgvector, cosine similarity, semantic search

## Vector Database Comparison

### Managed Services

| Database | Strengths | Limitations | Best For |
| -------- | --------- | ----------- | -------- |
| **Pinecone** | Fully managed, easy scaling, enterprise | Vendor lock-in, cost at scale | Enterprise production |
| **Weaviate Cloud** | GraphQL, hybrid search, modules | Complexity | Knowledge graphs |
| **Zilliz Cloud** | Milvus-based, high performance | Learning curve | High-scale production |
| **MongoDB Atlas Vector** | Existing MongoDB users | Newer feature | MongoDB shops |
| **Elastic Vector** | Existing Elastic stack | Resource heavy | Search platforms |

### Self-Hosted Options

| Database | Strengths | Limitations | Best For |
| -------- | --------- | ----------- | -------- |
| **Milvus** | Feature-rich, scalable, GPU support | Operational complexity | Large-scale production |
| **Qdrant** | Rust performance, filtering, easy | Smaller ecosystem | Performance-focused |
| **Weaviate** | Modules, semantic, hybrid | Memory usage | Knowledge applications |
| **Chroma** | Simple, Python-native | Limited scale | Development, prototyping |
| **pgvector** | PostgreSQL extension | Performance limits | Postgres shops |
| **FAISS** | Library, not DB, fastest | No persistence, no filtering | Research, embedded |

### Selection Decision Tree

```text
Need managed, don't want operations?
├── Yes → Pinecone (simplest) or Weaviate Cloud
└── No (self-hosted)
    └── Already using PostgreSQL?
        ├── Yes, <1M vectors → pgvector
        └── No
            └── Need maximum performance at scale?
                ├── Yes → Milvus or Qdrant
                └── No
                    └── Prototyping/development?
                        ├── Yes → Chroma
                        └── No → Qdrant (balanced choice)
```

## ANN Algorithms

### Algorithm Overview

```text
Exact KNN:
• Search ALL vectors
• O(n) time complexity
• Perfect accuracy
• Impractical at scale

Approximate NN (ANN):
• Search SUBSET of vectors
• O(log n) to O(1) complexity
• Near-perfect accuracy
• Practical at any scale
```

### HNSW (Hierarchical Navigable Small World)

```text
Layer 3: ○───────────────────────○  (sparse, long connections)
          │                       │
Layer 2: ○───○───────○───────○───○  (medium density)
          │   │       │       │   │
Layer 1: ○─○─○─○─○─○─○─○─○─○─○─○─○  (denser)
          │││││││││││││││││││││││
Layer 0: ○○○○○○○○○○○○○○○○○○○○○○○○○  (all vectors)

Search: Start at top layer, greedily descend
• Fast: O(log n) search time
• High recall: >95% typically
• Memory: Extra graph storage
```

**HNSW Parameters:**

| Parameter | Description | Trade-off |
| --------- | ----------- | --------- |
| `M` | Connections per node | Memory vs. recall |
| `ef_construction` | Build-time search width | Build time vs. recall |
| `ef_search` | Query-time search width | Latency vs. recall |

### IVF (Inverted File Index)

```text
Clustering Phase:
┌─────────────────────────────────────────┐
│     Cluster vectors into K centroids    │
│                                         │
│    ●         ●         ●         ●     │
│   /│\       /│\       /│\       /│\    │
│  ○○○○○     ○○○○○     ○○○○○     ○○○○○   │
│ Cluster 1  Cluster 2 Cluster 3 Cluster 4│
└─────────────────────────────────────────┘

Search Phase:
1. Find nprobe nearest centroids
2. Search only those clusters
3. Much faster than exhaustive
```

**IVF Parameters:**

| Parameter | Description | Trade-off |
| --------- | ----------- | --------- |
| `nlist` | Number of clusters | Build time vs. search quality |
| `nprobe` | Clusters to search | Latency vs. recall |

### IVF-PQ (Product Quantization)

```text
Original Vector (128 dim):
[0.1, 0.2, ..., 0.9]  (128 × 4 bytes = 512 bytes)

PQ Compressed (8 subvectors, 8-bit codes):
[23, 45, 12, 89, 56, 34, 78, 90]  (8 bytes)

Memory reduction: 64x
Accuracy trade-off: ~5% recall drop
```

### Algorithm Comparison

| Algorithm | Search Speed | Memory | Build Time | Recall |
| --------- | ------------ | ------ | ---------- | ------ |
| **Flat/Brute** | Slow (O(n)) | Low | None | 100% |
| **IVF** | Fast | Low | Medium | 90-95% |
| **IVF-PQ** | Very fast | Very low | Medium | 85-92% |
| **HNSW** | Very fast | High | Slow | 95-99% |
| **HNSW+PQ** | Very fast | Medium | Slow | 90-95% |

### When to Use Which

```text
< 100K vectors:
└── Flat index (exact search is fast enough)

100K - 1M vectors:
└── HNSW (best recall/speed trade-off)

1M - 100M vectors:
├── Memory available → HNSW
└── Memory constrained → IVF-PQ or HNSW+PQ

> 100M vectors:
└── Sharded IVF-PQ or distributed HNSW
```

## Distance Metrics

### Common Metrics

| Metric | Formula | Range | Best For |
| ------ | ------- | ----- | -------- |
| **Cosine Similarity** | `A·B / (\|\|A\|\| \|\|B\|\|)` | [-1, 1] | Normalized embeddings |
| **Dot Product** | `A·B` | (-∞, ∞) | When magnitude matters |
| **Euclidean (L2)** | `√Σ(A-B)²` | [0, ∞) | Absolute distances |
| **Manhattan (L1)** | `Σ\|A-B\|` | [0, ∞) | High-dimensional sparse |

### Metric Selection

```text
Embeddings pre-normalized (unit vectors)?
├── Yes → Cosine = Dot Product (use Dot, faster)
└── No
    └── Magnitude meaningful?
        ├── Yes → Dot Product
        └── No → Cosine Similarity

Note: Most embedding models output normalized vectors
      → Dot product is usually the best choice
```

## Filtering and Hybrid Search

### Pre-filtering vs Post-filtering

```text
Pre-filtering (Filter → Search):
┌─────────────────────────────────────────┐
│ 1. Apply metadata filter               │
│    (category = "electronics")           │
│    Result: 10K of 1M vectors           │
│                                         │
│ 2. Vector search on 10K vectors        │
│    Much faster, guaranteed filter match │
└─────────────────────────────────────────┘

Post-filtering (Search → Filter):
┌─────────────────────────────────────────┐
│ 1. Vector search on 1M vectors         │
│    Return top-1000                      │
│                                         │
│ 2. Apply metadata filter               │
│    May return < K results!             │
└─────────────────────────────────────────┘
```

### Hybrid Search Architecture

```text
Query: "wireless headphones under $100"
           │
     ┌─────┴─────┐
     ▼           ▼
 ┌───────┐  ┌───────┐
 │Vector │  │Filter │
 │Search │  │ Build │
 │"wire- │  │price  │
 │less   │  │< 100  │
 │head-  │  │       │
 │phones"│  │       │
 └───────┘  └───────┘
     │           │
     └─────┬─────┘
           ▼
    ┌───────────┐
    │  Combine  │
    │  Results  │
    └───────────┘
```

### Metadata Index Design

| Metadata Type | Index Strategy | Query Example |
| ------------- | -------------- | ------------- |
| **Categorical** | Bitmap/hash index | category = "books" |
| **Numeric range** | B-tree | price BETWEEN 10 AND 50 |
| **Keyword search** | Inverted index | tags CONTAINS "sale" |
| **Geospatial** | R-tree/geohash | location NEAR (lat, lng) |

## Scaling Strategies

### Sharding Approaches

```text
Naive Sharding (by ID):
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Shard 1 │ │ Shard 2 │ │ Shard 3 │
│ IDs 0-N │ │IDs N-2N │ │IDs 2N-3N│
└─────────┘ └─────────┘ └─────────┘
Query → Search ALL shards → Merge results

Semantic Sharding (by cluster):
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Shard 1 │ │ Shard 2 │ │ Shard 3 │
│ Tech    │ │ Health  │ │ Finance │
│ docs    │ │ docs    │ │ docs    │
└─────────┘ └─────────┘ └─────────┘
Query → Route to relevant shar