ml-inference-optimization

What this skill does

# ML Inference Optimization

## When to Use This Skill

Use this skill when:

- Optimizing ML inference latency
- Reducing model size for deployment
- Implementing model compression techniques
- Designing inference caching strategies
- Deploying models at the edge
- Balancing accuracy vs. latency trade-offs

**Keywords:** inference optimization, latency, model compression, distillation, pruning, quantization, caching, edge ML, TensorRT, ONNX, model serving, batching, hardware acceleration

## Inference Optimization Overview

```text
┌─────────────────────────────────────────────────────────────────────┐
│                 Inference Optimization Stack                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │                    Model Level                                │  │
│  │  Distillation │ Pruning │ Quantization │ Architecture Search │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│                              ▼                                      │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │                   Compiler Level                              │  │
│  │  Graph optimization │ Operator fusion │ Memory planning       │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│                              ▼                                      │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │                  Runtime Level                                │  │
│  │  Batching │ Caching │ Async execution │ Multi-threading      │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                              │                                      │
│                              ▼                                      │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │                  Hardware Level                               │  │
│  │  GPU │ TPU │ NPU │ CPU SIMD │ Custom accelerators            │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

## Model Compression Techniques

### Technique Overview

| Technique | Size Reduction | Speed Improvement | Accuracy Impact |
| --------- | -------------- | ----------------- | --------------- |
| **Quantization** | 2-4x | 2-4x | Low (1-2%) |
| **Pruning** | 2-10x | 1-3x | Low-Medium |
| **Distillation** | 3-10x | 3-10x | Medium |
| **Low-rank factorization** | 2-5x | 1.5-3x | Low-Medium |
| **Weight sharing** | 10-100x | Variable | Medium-High |

### Knowledge Distillation

```text
┌─────────────────────────────────────────────────────────────────────┐
│                    Knowledge Distillation                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────────┐                                                   │
│  │ Teacher Model│ (Large, accurate, slow)                          │
│  │   GPT-4      │                                                   │
│  └──────────────┘                                                   │
│         │                                                           │
│         ▼ Soft labels (probability distributions)                   │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │                    Training Process                           │  │
│  │  Loss = α × CrossEntropy(student, hard_labels)               │  │
│  │       + (1-α) × KL_Div(student, teacher_soft_labels)         │  │
│  └──────────────────────────────────────────────────────────────┘  │
│         │                                                           │
│         ▼                                                           │
│  ┌──────────────┐                                                   │
│  │Student Model │ (Small, nearly as accurate, fast)                │
│  │  DistilBERT  │                                                   │
│  └──────────────┘                                                   │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

**Distillation Types:**

| Type | Description | Use Case |
| ---- | ----------- | -------- |
| **Response distillation** | Match teacher outputs | General compression |
| **Feature distillation** | Match intermediate layers | Better transfer |
| **Relation distillation** | Match sample relationships | Structured data |
| **Self-distillation** | Model teaches itself | Regularization |

### Pruning Strategies

```text
Unstructured Pruning (Weight-level):
Before: [0.1, 0.8, 0.2, 0.9, 0.05, 0.7]
After:  [0.0, 0.8, 0.0, 0.9, 0.0, 0.7]  (50% sparse)
• Flexible, high sparsity possible
• Needs sparse hardware/libraries

Structured Pruning (Channel/Layer-level):
Before: ┌───┬───┬───┬───┐
        │ C1│ C2│ C3│ C4│
        └───┴───┴───┴───┘
After:  ┌───┬───┬───┐
        │ C1│ C3│ C4│  (Removed C2 entirely)
        └───┴───┴───┘
• Works with standard hardware
• Lower compression ratio
```

**Pruning Decision Criteria:**

| Method | Description | Effectiveness |
| ------ | ----------- | ------------- |
| **Magnitude-based** | Remove smallest weights | Simple, effective |
| **Gradient-based** | Remove low-gradient weights | Better accuracy |
| **Second-order** | Use Hessian information | Best but expensive |
| **Lottery ticket** | Find winning subnetwork | Theoretical insight |

### Quantization (Detailed)

```text
Precision Hierarchy:

FP32 (32 bits): ████████████████████████████████
FP16 (16 bits): ████████████████
BF16 (16 bits): ████████████████  (different mantissa/exponent)
INT8 (8 bits):  ████████
INT4 (4 bits):  ████
Binary (1 bit): █

Memory and Compute Scale Proportionally
```

**Quantization Approaches:**

| Approach | When Applied | Quality | Effort |
| -------- | ------------ | ------- | ------ |
| **Dynamic quantization** | Runtime | Good | Low |
| **Static quantization** | Post-training with calibration | Better | Medium |
| **QAT** | During training | Best | High |

## Compiler-Level Optimization

### Graph Optimization

```text
Original Graph:
Input → Conv → BatchNorm → ReLU → Conv → BatchNorm → ReLU → Output

Optimized Graph (Operator Fusion):
Input → FusedConvBNReLU → FusedConvBNReLU → Output

Benefits:
• Fewer kernel launches
• Better memory locality
• Reduced memory bandwidth
```

### Common Optimizations

| Optimization | Description | Speedup |
| ------------ | ----------- | ------- |
| **Operator fusion** | Combine sequential ops | 1.2-2x |
| **Constant folding** | Pre-compute constants | 1.1-1.5x |
| **Dead code elimination** | Remove unused ops | Variable |
| **Layout optimization** | Optimize tensor memory layout | 1.1-1.3x |
| **Memory planning** | Optimize buffer allocation | 1.1-1.2x |

### Optimization Frameworks

| Framework | Vendor | Best For |
| --------- | ------ | -------- |
| **TensorRT** | NVIDIA | NVIDIA GPUs, lowest latency |
| **ONNX Runtime** | Microsoft | Cross-platform, broad support |
| **OpenVINO** | Intel | Intel CPUs/GPUs |
| **Core ML** | Apple | Apple devices |
| **TFLite** | Google | Mobile, embedded |
| **Apache TVM** | Open source | Custom hardware, research |

## Runtime Optimization

### Batching Strategies

```text
No Batching:
Request 1: [Process] → Response 1      10ms
Request 2: [Process] → Response 2      10ms
Request 3: [Process] → Response 3      10ms
Total: 30ms, GPU underutilized

Dynamic Batching