validation-scripts

What this skill does

# ML Training Validation Scripts

**Purpose:** Production-ready validation and testing utilities for ML training workflows. Ensures data quality, model integrity, pipeline correctness, and dependency availability before and during training.

**Activation Triggers:**
- Validating training datasets before fine-tuning
- Checking model checkpoints and outputs
- Testing end-to-end training pipelines
- Verifying system dependencies and GPU availability
- Debugging training failures or data issues
- Ensuring data format compliance
- Validating model configurations
- Testing inference pipelines

**Key Resources:**
- `scripts/validate-data.sh` - Comprehensive dataset validation
- `scripts/validate-model.sh` - Model checkpoint and config validation
- `scripts/test-pipeline.sh` - End-to-end pipeline testing
- `scripts/check-dependencies.sh` - System and package dependency checking
- `templates/test-config.yaml` - Pipeline test configuration template
- `templates/validation-schema.json` - Data validation schema template
- `examples/data-validation-example.md` - Dataset validation workflow
- `examples/pipeline-testing-example.md` - Complete pipeline testing guide

## Quick Start

### Validate Training Data
```bash
bash scripts/validate-data.sh \
  --data-path ./data/train.jsonl \
  --format jsonl \
  --schema templates/validation-schema.json
```

### Validate Model Checkpoint
```bash
bash scripts/validate-model.sh \
  --model-path ./checkpoints/epoch-3 \
  --framework pytorch \
  --check-weights
```

### Test Complete Pipeline
```bash
bash scripts/test-pipeline.sh \
  --config templates/test-config.yaml \
  --data ./data/sample.jsonl \
  --verbose
```

### Check System Dependencies
```bash
bash scripts/check-dependencies.sh \
  --framework pytorch \
  --gpu-required \
  --min-vram 16
```

## Validation Scripts

### 1. Data Validation (validate-data.sh)

**Purpose:** Validate training datasets for format compliance, data quality, and schema conformance.

**Usage:**
```bash
bash scripts/validate-data.sh [OPTIONS]
```

**Options:**
- `--data-path PATH` - Path to dataset file or directory (required)
- `--format FORMAT` - Data format: jsonl, csv, parquet, arrow (default: jsonl)
- `--schema PATH` - Path to validation schema JSON file
- `--sample-size N` - Number of samples to validate (default: all)
- `--check-duplicates` - Check for duplicate entries
- `--check-null` - Check for null/missing values
- `--check-length` - Validate text length constraints
- `--check-tokens` - Validate tokenization compatibility
- `--tokenizer MODEL` - Tokenizer to use for token validation (default: gpt2)
- `--max-length N` - Maximum sequence length (default: 2048)
- `--output REPORT` - Output validation report path (default: validation-report.json)

**Validation Checks:**
- **Format Compliance**: Verify file format and structure
- **Schema Validation**: Check against JSON schema if provided
- **Data Types**: Validate field types (string, int, float, etc.)
- **Required Fields**: Ensure all required fields are present
- **Value Ranges**: Check numeric values within expected ranges
- **Text Quality**: Detect empty strings, excessive whitespace, encoding issues
- **Duplicates**: Identify duplicate entries (exact or near-duplicate)
- **Token Counts**: Verify sequences fit within model context length
- **Label Distribution**: Check class balance for classification tasks
- **Missing Values**: Detect and report null/missing values

**Example Output:**
```json
{
  "status": "PASS",
  "total_samples": 10000,
  "valid_samples": 9987,
  "invalid_samples": 13,
  "validation_errors": [
    {
      "sample_id": 42,
      "field": "text",
      "error": "Exceeds max token length: 2150 > 2048"
    },
    {
      "sample_id": 156,
      "field": "label",
      "error": "Invalid label value: 'unknwn' (typo)"
    }
  ],
  "statistics": {
    "avg_text_length": 487,
    "avg_token_count": 128,
    "label_distribution": {
      "positive": 4892,
      "negative": 4895,
      "neutral": 213
    }
  },
  "recommendations": [
    "Remove or fix 13 invalid samples before training",
    "Label distribution is imbalanced - consider class weighting"
  ]
}
```

**Exit Codes:**
- `0` - All validations passed
- `1` - Validation errors found
- `2` - Script error (invalid arguments, file not found)

### 2. Model Validation (validate-model.sh)

**Purpose:** Validate model checkpoints, configurations, and inference readiness.

**Usage:**
```bash
bash scripts/validate-model.sh [OPTIONS]
```

**Options:**
- `--model-path PATH` - Path to model checkpoint directory (required)
- `--framework FRAMEWORK` - Framework: pytorch, tensorflow, jax (default: pytorch)
- `--check-weights` - Verify model weights are loadable
- `--check-config` - Validate model configuration file
- `--check-tokenizer` - Verify tokenizer files are present
- `--check-inference` - Test inference with sample input
- `--sample-input TEXT` - Sample text for inference test
- `--expected-output TEXT` - Expected output for verification
- `--output REPORT` - Output validation report path

**Validation Checks:**
- **File Structure**: Verify required files are present
- **Config Validation**: Check model_config.json for correctness
- **Weight Integrity**: Load and verify model weights
- **Tokenizer Files**: Ensure tokenizer files exist and are loadable
- **Model Architecture**: Validate architecture matches config
- **Memory Requirements**: Estimate GPU/CPU memory needed
- **Inference Test**: Run sample inference if requested
- **Quantization**: Verify quantized models load correctly
- **LoRA/PEFT**: Validate adapter weights and configuration

**Example Output:**
```json
{
  "status": "PASS",
  "model_path": "./checkpoints/llama-7b-finetuned",
  "framework": "pytorch",
  "checks": {
    "file_structure": "PASS",
    "config": "PASS",
    "weights": "PASS",
    "tokenizer": "PASS",
    "inference": "PASS"
  },
  "model_info": {
    "architecture": "LlamaForCausalLM",
    "parameters": "7.2B",
    "precision": "float16",
    "lora_enabled": true,
    "lora_rank": 16
  },
  "memory_estimate": {
    "model_size_gb": 13.5,
    "inference_vram_gb": 16.2,
    "training_vram_gb": 24.8
  },
  "inference_test": {
    "input": "Hello, world!",
    "output": "Hello, world! How can I help you today?",
    "latency_ms": 142
  }
}
```

### 3. Pipeline Testing (test-pipeline.sh)

**Purpose:** Test end-to-end ML training and inference pipelines with sample data.

**Usage:**
```bash
bash scripts/test-pipeline.sh [OPTIONS]
```

**Options:**
- `--config PATH` - Pipeline test configuration file (required)
- `--data PATH` - Sample data for testing
- `--steps STEPS` - Comma-separated steps to test (default: all)
- `--verbose` - Enable detailed output
- `--output REPORT` - Output test report path
- `--fail-fast` - Stop on first failure
- `--cleanup` - Clean up temporary files after test

**Pipeline Steps:**
- **data_loading**: Test data loading and preprocessing
- **tokenization**: Verify tokenization pipeline
- **model_loading**: Load model and verify initialization
- **training_step**: Run single training step
- **validation_step**: Run single validation step
- **checkpoint_save**: Test checkpoint saving
- **checkpoint_load**: Test checkpoint loading
- **inference**: Test inference pipeline
- **metrics**: Verify metrics calculation

**Test Configuration (test-config.yaml):**
```yaml
pipeline:
  name: llama-7b-finetuning
  framework: pytorch

data:
  train_path: ./data/sample-train.jsonl
  val_path: ./data/sample-val.jsonl
  format: jsonl

model:
  base_model: meta-llama/Llama-2-7b-hf
  checkpoint_path: ./checkpoints/test
  load_in_8bit: false
  lora:
    enabled: true
    r: 16
    alpha: 32

training:
  batch_size: 1
  gradient_accumulation: 1
  learning_rate: 2e-4
  max_steps: 5

testing:
  sample_size: 10
  timeout_seconds: 300
  expected_loss_range: [0.5, 3.0]
```

**Example Output:**
```
Pipeline Test Report
====================

Pipeline: llama-7b-finetuning
Started: 2025-11-01 12:3