ml-cloud-deployment

What this skill does

# ML Cloud Deployment

## Overview

Use this skill for deploying ML workloads to managed platforms, Kubernetes, serverless systems, GPU/TPU providers, and lakehouse environments. Start from workload requirements: training or inference, batch or online, latency SLO, throughput, model size, data gravity, compliance, region, hardware, team expertise, and budget.

## Platform Selection

| Requirement | Strong choices |
|---|---|
| AWS-native managed lifecycle | SageMaker Studio, Training, Processing, Pipelines, Model Registry, Endpoints, Feature Store, Clarify, Model Monitor |
| GCP-native managed lifecycle | Vertex AI Training, Pipelines, Endpoints, Feature Store, Model Monitoring, AutoML, Matching Engine, TPUs |
| Azure-native managed lifecycle | Azure ML workspaces, compute clusters, pipelines, registries, managed online/batch endpoints, AutoML, Responsible ML |
| Lakehouse/Spark-centric ML | Databricks on AWS/Azure/GCP with MLflow, Delta, Feature Store, Workflows |
| Kubernetes control | EKS/GKE/AKS with Kubeflow, KServe, Seldon, Ray, Triton, custom operators |
| Serverless or fast GPU apps | Modal, Replicate, Cloud Run with GPU where available, Lambda for small CPU inference |
| Flexible GPU rental | Lambda Labs, RunPod, self-managed cloud GPU VMs |
| Ray-native scale-out | Anyscale or Ray clusters on Kubernetes/cloud VMs |

Prefer managed services when governance, observability, and team velocity matter more than runtime customization. Prefer Kubernetes or VMs when custom networking, specialized runtimes, or cost/performance tuning dominate.

## AWS Patterns

SageMaker handles managed training jobs, feature stores, and real-time or serverless endpoints. The AWS CLI (`aws sagemaker` and `aws sagemaker-runtime`) is used to orchestrate these systems programmatically.

### AWS SageMaker Endpoint Variant Configuration (`endpoint_config.json`)
```json
{
  "EndpointConfigName": "production-llm-classifier-v1-cfg",
  "ProductionVariants": [
    {
      "VariantName": "AllTraffic",
      "ModelName": "llm-classifier-model-v1",
      "InitialInstanceCount": 2,
      "InstanceType": "ml.g5.2xlarge",
      "InitialVariantWeight": 1.0,
      "VolumeSizeInGB": 50,
      "ManagedInstanceScaling": {
        "MinInstanceCount": 2,
        "MaxInstanceCount": 10,
        "Status": "ENABLED"
      },
      "RoutingConfig": {
        "RoutingStrategy": "LEAST_OUTSTANDING_REQUESTS"
      }
    }
  ]
}
```

### AWS CLI SageMaker Lifecycle Management

```bash
# 1. Setup Domain and default IAM Role/VPC settings
aws sagemaker create-domain \
  --domain-name ProdMLDomain \
  --auth-mode IAM \
  --default-user-settings ExecutionRole=arn:aws:iam::123456789012:role/SageMakerExecutionRole \
  --subnet-ids subnet-1a2b3c,subnet-4d5e6f \
  --vpc-id vpc-0abc123 \
  --app-network-access-type PublicInternetOnly

# 2. Provision and start/stop an interactive notebook instance
aws sagemaker create-notebook-instance \
  --notebook-instance-name dev-notebook-instance \
  --instance-type ml.t3.medium \
  --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole

aws sagemaker start-notebook-instance --notebook-instance-name dev-notebook-instance
aws sagemaker stop-notebook-instance --notebook-instance-name dev-notebook-instance

# 3. Submit a managed GPU training job
aws sagemaker create-training-job \
  --training-job-name custom-pytorch-train-job \
  --algorithm-specification TrainingImage=763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.12.0-gpu-py38-cu113-ubuntu20.04,TrainingInputMode=File \
  --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \
  --input-data-config '[{"ChannelName": "training", "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": "s3://my-ml-bucket/train-data/", "S3DataDistributionType": "FullyReplicated"}}}]' \
  --output-data-config S3OutputPath=s3://my-ml-bucket/checkpoints/ \
  --resource-config InstanceType=ml.g5.2xlarge,InstanceCount=1,VolumeSizeInGB=50 \
  --stopping-condition MaxRuntimeInSeconds=86400

# 4. Register a model in the SageMaker Model Registry
aws sagemaker create-model-package-group \
  --model-package-group-name credit-scoring-group \
  --model-package-group-description "Model registry group for credit scoring neural nets"

aws sagemaker create-model \
  --model-name credit-scoring-v1 \
  --primary-container Image=763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:1.12.0-gpu-py38,ModelDataUrl=s3://my-ml-bucket/checkpoints/custom-pytorch-train-job/output/model.tar.gz \
  --execution-role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole

# 5. Create Endpoint Configuration and Deploy Endpoint
aws sagemaker create-endpoint-config \
  --endpoint-config-name credit-scoring-v1-cfg \
  --production-variants '[{"VariantName": "AllTraffic", "ModelName": "credit-scoring-v1", "InitialInstanceCount": 2, "InstanceType": "ml.m5.xlarge", "InitialVariantWeight": 1.0}]'

aws sagemaker create-endpoint \
  --endpoint-name credit-scoring-prod-endpoint \
  --endpoint-config-name credit-scoring-v1-cfg

# 6. Invoke/Test Endpoint via CLI
aws sagemaker-runtime invoke-endpoint \
  --endpoint-name credit-scoring-prod-endpoint \
  --content-type application/json \
  --body '{"features": [0.25, 1.4, 0.9]}' \
  response_output.json
```

## GCP Patterns

Vertex AI manages training pipelines, hyperparameter tuning sweeps, custom jobs, the model registry, and predictions via `gcloud ai`.

### GCP Vertex AI TPU CustomJob Specification (`tpu_job.yaml`)
```yaml
displayName: vertex-ai-tpu-training-job
studySpec:
  metrics:
    - metricId: val_loss
      goal: MINIMIZE
  parameters:
    - parameterId: learning_rate
      doubleValueSpec:
        minValue: 1e-5
        maxValue: 1e-3
      scaleType: UNIT_LOG_SCALE
trialJobSpec:
  workerPoolSpecs:
    - machineSpec:
        machineType: n1-standard-8
      replicaCount: 1
      containerSpec:
        imageUri: gcr.io/my-project/ml-trainer:latest
    - machineSpec:
        machineType: cloud-tpu-v4-podslice
        acceleratorType: TPU_V4
        acceleratorCount: 4
      replicaCount: 1
      containerSpec:
        imageUri: gcr.io/my-project/ml-tpu-trainer:latest
        args: [
          "--data_path", "gs://my-bucket/dataset-v1",
          "--epochs", "10"
        ]
```

### GCP Vertex AI CLI Operations

```bash
# 1. Create a user-managed Jupyter Workbench instance
gcloud notebooks instances create dev-workbench-instance \
  --location=us-central1-a \
  --vm-image-project=deeplearning-platform-release \
  --vm-image-family=common-gpu \
  --machine-type=n1-standard-4

# 2. Submit containerized custom training job
gcloud ai custom-jobs create \
  --region=us-central1 \
  --display-name=gpu-pytorch-training-run \
  --worker-pool-spec=replica-count=1,machine-type=n1-standard-8,accelerator-type=nvidia-tesla-t4,accelerator-count=1,container-image-uri=gcr.io/my-gcp-project/pytorch-train-custom:v1 \
  --args="--epochs=20,--data-path=gs://my-bucket/training-gold"

# 3. Upload model artifact to Model Registry
gcloud ai models upload \
  --region=us-central1 \
  --display-name=iris-classifier-v1 \
  --container-image-uri=us-docker.pkg.dev/vertex-ai/prediction/pytorch-cpu.1-11:latest \
  --artifact-uri=gs://my-bucket/models/iris-classifier/

# 4. Provision endpoint and deploy model with 100% traffic allocation
# Note: replace <ENDPOINT-ID> and <MODEL-ID> with their UUIDs
gcloud ai endpoints create \
  --region=us-central1 \
  --display-name=production-iris-endpoint

gcloud ai endpoints deploy-model <ENDPOINT-ID> \
  --region=us-central1 \
  --model=<MODEL-ID> \
  --display-name=iris-v1-blue-deployment \
  --machine-type=n1-standard-4 \
  --min-replica-count=2 \
  --max-replica-count=10 \
  --traffic-split=0=100

# 5. Predict via Vertex AI Endpoint CLI
gcloud ai endpoints predict <ENDPOINT-ID> \
  --region=us-central1 \
  --json-request=sample_payload.json
```

## Azure Patterns

Azure AI Foundry / Azure Machine Learning Workspace orchestrates the entire ML lifecycle. Work