ml-mlops

What this skill does

# ML MLOps

## Overview

Use this skill for operational machine learning: experiment tracking, reproducibility, orchestration, registries, CI/CD, model deployment governance, monitoring, drift response, and retraining. MLOps turns notebooks and scripts into auditable, repeatable systems with clear ownership and rollback.

## MLOps Invariants

Every production ML workflow should answer:
- Which data, code, config, environment, and hardware produced this model?
- Which metrics, slices, and tests justified promotion?
- Where is the model artifact stored, and how can it be rolled back?
- What validates incoming data and serving features?
- What monitors quality, drift, latency, cost, safety, and fairness?
- Who owns incidents, retraining, approvals, and deprecation?

If any answer is missing, fix the lifecycle before adding more infrastructure.

## Experiment Tracking

Track parameters, metrics, artifacts, dataset versions, code revisions, environment, random seeds, run notes, and tags. MLflow is a strong default for open model registry workflows; Weights & Biases excels for rich experiment dashboards and sweeps; Neptune, Comet, and ClearML cover similar collaboration and artifact-management needs. Choose based on governance, hosting, integrations, and team workflow rather than dashboard aesthetics alone.

Use consistent naming for experiments and runs. Log both final metrics and learning curves. Store confusion matrices, calibration plots, feature importance, validation predictions, and representative errors. Avoid logging secrets, raw PII, or proprietary examples unless the tracking backend is approved for that data class.

## Model Registry and Versioning

A registry entry should include artifact URI, model signature or schema, input/output examples, preprocessing/tokenizer references, dependency environment, training dataset identifier, evaluation report, approval state, owner, changelog, and intended use.

### Standardized Model Registry Metadata Schema (JSON)
```json
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "ModelRegistryMetadata",
  "type": "object",
  "properties": {
    "model_name": { "type": "string" },
    "version": { "type": "string" },
    "artifact_uri": { "type": "string", "format": "uri" },
    "framework": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "version": { "type": "string" }
      },
      "required": ["name", "version"]
    },
    "signature": {
      "type": "object",
      "properties": {
        "inputs": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "type": { "type": "string" },
              "shape": { "type": "array", "items": { "type": "integer" } }
            },
            "required": ["name", "type"]
          }
        },
        "outputs": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "type": { "type": "string" },
              "shape": { "type": "array", "items": { "type": "integer" } }
            },
            "required": ["name", "type"]
          }
        }
      },
      "required": ["inputs", "outputs"]
    },
    "provenance": {
      "type": "object",
      "properties": {
        "git_commit": { "type": "string" },
        "dataset_snapshot_uri": { "type": "string" },
        "environment_lockfile_uri": { "type": "string" }
      },
      "required": ["git_commit", "dataset_snapshot_uri"]
    }
  },
  "required": ["model_name", "version", "artifact_uri", "framework", "signature", "provenance"]
}
```

Stage transitions should be controlled by automated quality gates and human approval where risk requires it. Use semantic model labels such as `candidate`, `staging`, `production`, and `archived` only if their meaning is enforced. Version datasets with tools such as DVC, lakehouse table versions, object-storage manifests, or feature-store materialization timestamps. The model version alone is not enough; the data and feature code are part of the model.

## Pipeline Orchestration

Choose orchestration by operational environment:

| Platform | Best fit |
|---|---|
| Kubeflow Pipelines | Kubernetes-native ML workflows, portable components, complex DAGs |
| SageMaker Pipelines | AWS-native training, processing, model registry, endpoints, Feature Store, Clarify, Model Monitor |
| Vertex AI Pipelines | GCP-native training, endpoints, Feature Store, Model Monitoring, AutoML integration |
| Azure ML Pipelines | Azure workspaces, managed compute, registries, managed endpoints, Responsible ML |
| Databricks Workflows | Lakehouse-centric feature engineering, MLflow registry, Spark, Delta, notebooks/jobs |
| Airflow/Prefect/Dagster | General data/ML orchestration with broad ecosystem integrations |

Keep pipelines modular: data validation, feature generation, training, evaluation, registration, deployment, and monitoring setup should be separate steps with explicit inputs and outputs. Make steps idempotent and cache-aware where safe. For Azure ML code asset registration in CI, ADF-to-Azure-ML version propagation, pointer blobs, or runtime validation of ADF WebActivity paths, load `ml-azureml-adf-automation`.

## CI/CD for ML

CI should run fast checks: code linting, unit tests for preprocessing, schema tests, deterministic small-data training smoke tests, model-loading tests, serialization tests, inference contract tests, and security scans. CD should deploy only artifacts that passed evaluation gates.

### Production GitHub Actions MLOps Workflow (`.github/workflows/mlops.yml`)
```yaml
name: MLOps CI/CD Workflow

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.10'
        cache: 'pip'
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install ruff pytest pytest-cov -r requirements.txt
    - name: Lint with Ruff
      run: ruff check src/
    - name: Run unit tests
      run: pytest tests/unit/ --cov=src --cov-report=xml
      
  model-smoke-test:
    needs: lint-and-test
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.10'
    - name: Run small-scale training smoke test
      run: |
        pip install -r requirements.txt
        python src/train.py --config configs/smoke_test.yaml
    - name: Verify model serialization & load contract
      run: python src/verify_contract.py --model_path outputs/model.pt

  promote-and-deploy:
    needs: model-smoke-test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Deploy model to Staging endpoint
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      run: |
        pip install awscli
        # Trigger staging endpoint update
        python scripts/deploy.py --stage staging
```

For model serving, use blue/green, canary, shadow, or A/B deployment. Canary small traffic first, monitor guardrails, then expand. Shadow mode is useful for risk-free comparison but cannot measure user-impact metrics unless decisions are acted on. Always keep rollback simple and tested.

## Data Validation

Validate data before training and serving. Great Expectations, TFDV, and Deequ can enforce schemas and distribution expectations. 

### Great Expectations Validation Suite Setup
```python
import great_expectations as ge
from great_expectations.core.expectation_suite import ExpectationSuite

def create_ingestion_validation_suite(suite_nam