ml-system-design

What this skill does

# ML System Design

This skill provides frameworks for designing production machine learning systems, from data pipelines to model serving.

## When to Use This Skill

**Keywords:** ML pipeline, machine learning system, feature store, model training, model serving, ML infrastructure, MLOps, A/B testing ML, feature engineering, model deployment

**Use this skill when:**

- Designing end-to-end ML systems for production
- Planning feature store architecture
- Designing model training pipelines
- Planning model serving infrastructure
- Preparing for ML system design interviews
- Evaluating ML platform tools and frameworks

## ML System Architecture Overview

### The ML System Lifecycle

```text
┌─────────────────────────────────────────────────────────────────────────┐
│                        ML SYSTEM LIFECYCLE                               │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌────────┐ │
│  │  Data    │──▶│ Feature  │──▶│  Model   │──▶│  Model   │──▶│ Monitor│ │
│  │ Ingestion│   │ Pipeline │   │ Training │   │ Serving  │   │ & Eval │ │
│  └──────────┘   └──────────┘   └──────────┘   └──────────┘   └────────┘ │
│       │              │              │              │              │      │
│       ▼              ▼              ▼              ▼              ▼      │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌────────┐ │
│  │  Data    │   │ Feature  │   │  Model   │   │ Inference│   │ Metrics│ │
│  │  Lake    │   │  Store   │   │ Registry │   │  Cache   │   │  Store │ │
│  └──────────┘   └──────────┘   └──────────┘   └──────────┘   └────────┘ │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘
```

### Key Components

| Component | Purpose | Examples |
| --------- | ------- | -------- |
| **Data Ingestion** | Collect raw data from sources | Kafka, Kinesis, Pub/Sub |
| **Feature Pipeline** | Transform raw data to features | Spark, Flink, dbt |
| **Feature Store** | Store and serve features | Feast, Tecton, Vertex AI |
| **Model Training** | Train and validate models | SageMaker, Vertex AI, Kubeflow |
| **Model Registry** | Version and track models | MLflow, Weights & Biases |
| **Model Serving** | Serve predictions | TensorFlow Serving, Triton, vLLM |
| **Monitoring** | Track model performance | Evidently, WhyLabs, Arize |

## Feature Store Architecture

### Why Feature Stores?

**Problems without a feature store:**

- Training-serving skew (features computed differently)
- Duplicate feature computation across teams
- No feature versioning or lineage
- Slow feature experimentation

### Feature Store Components

```text
┌─────────────────────────────────────────────────────────────────┐
│                      FEATURE STORE                               │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────────────┐       ┌─────────────────────┐          │
│  │   OFFLINE STORE     │       │   ONLINE STORE      │          │
│  │                     │       │                     │          │
│  │  - Historical data  │       │  - Low-latency      │          │
│  │  - Training queries │ ────▶ │  - Point lookups    │          │
│  │  - Batch features   │ sync  │  - Real-time serving│          │
│  │                     │       │                     │          │
│  │  (Data Warehouse)   │       │  (Redis, DynamoDB)  │          │
│  └─────────────────────┘       └─────────────────────┘          │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │                   FEATURE REGISTRY                          ││
│  │  - Feature definitions    - Version control                 ││
│  │  - Data lineage          - Access control                   ││
│  └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘
```

### Feature Types

| Type | Computation | Storage | Example |
| ---- | ----------- | ------- | ------- |
| **Batch** | Scheduled (hourly/daily) | Offline → Online | User purchase count (30 days) |
| **Streaming** | Real-time event processing | Direct to online | Items in cart (current) |
| **On-demand** | Request-time computation | Not stored | Distance to nearest store |

### Training-Serving Consistency

```text
TRAINING (Historical):
┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│ Historical   │───▶│ Point-in-Time│───▶│  Training    │
│ Events       │    │ Join         │    │  Dataset     │
└──────────────┘    └──────────────┘    └──────────────┘
                          │
                    Uses feature
                    definitions
                          │
SERVING (Real-time):      ▼
┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│ Online       │───▶│ Same Feature │───▶│  Prediction  │
│ Store        │    │ Definitions  │    │  Request     │
└──────────────┘    └──────────────┘    └──────────────┘
```

## Model Training Infrastructure

### Training Pipeline Components

```text
┌───────────────────────────────────────────────────────────────────────┐
│                     TRAINING PIPELINE                                  │
├───────────────────────────────────────────────────────────────────────┤
│                                                                        │
│  ┌────────────┐   ┌────────────┐   ┌────────────┐   ┌────────────┐   │
│  │   Data     │──▶│   Feature  │──▶│   Model    │──▶│  Model     │   │
│  │   Loader   │   │   Transform│   │   Train    │   │  Validate  │   │
│  └────────────┘   └────────────┘   └────────────┘   └────────────┘   │
│        │               │                │                 │           │
│        ▼               ▼                ▼                 ▼           │
│  ┌────────────┐   ┌────────────┐   ┌────────────┐   ┌────────────┐   │
│  │ Experiment │   │ Hyperparameter│ │  Checkpoint │  │   Model    │   │
│  │  Tracking  │   │    Tuning     │ │   Storage  │   │  Registry  │   │
│  └────────────┘   └────────────┘   └────────────┘   └────────────┘   │
│                                                                        │
└───────────────────────────────────────────────────────────────────────┘
```

### Training Infrastructure Patterns

| Pattern | Use Case | Tools |
| ------- | -------- | ----- |
| **Single-node** | Small datasets, quick experiments | Jupyter, local GPU |
| **Distributed data-parallel** | Large datasets, same model | Horovod, PyTorch DDP |
| **Model-parallel** | Large models that don't fit in memory | DeepSpeed, FSDP, Megatron |
| **Hyperparameter tuning** | Automated model optimization | Optuna, Ray Tune |

### Experiment Tracking

Track for reproducibility:

| What to Track | Why |
| ------------- | --- |
| **Hyperparameters** | Reproduce training runs |
| **Metrics** | Compare model performance |
| **Artifacts** | Model files, datasets |
| **Code version** | Git commit hash |
| **Environment** | Docker image, dependencies |
| **Data version** | Dataset hash or snapshot |

## Model Serving Architecture

### Serving Patterns

| Pattern | Latency | Throughput | Use Case |
| ------- | ------- | ---------- | -------- |
| **Online (REST/gRPC)** | Low (<100ms) | Medium | Real-time predictions |
| **Batch** | High (hours) | Very high | Bulk scoring |
| **Streaming** | Medium | High | Event-driven predictions |
| **Embedded** | Very low | Varies | Edge/mobile inference |

### Online Serving Architecture

```text
┌─────────────────────────────────────────────────────────────────────┐
│                     MODEL SERVING SYSTEM                             │
├──────────────────────────────────────────────────────────