llm-fine-tuning-guide

What this skill does

# LLM Fine-Tuning Guide

Master the art of fine-tuning large language models to create specialized models optimized for your specific use cases, domains, and performance requirements.

## Overview

Fine-tuning adapts pre-trained LLMs to specific tasks, domains, or styles by training them on curated datasets. This improves accuracy, reduces hallucinations, and optimizes costs.

### When to Fine-Tune

- **Domain Specialization**: Legal documents, medical records, financial reports
- **Task-Specific Performance**: Better results on specific tasks than base model
- **Cost Optimization**: Smaller fine-tuned model replaces expensive large model
- **Style Adaptation**: Match specific writing styles or tones
- **Compliance Requirements**: Keep sensitive data within your infrastructure
- **Latency Requirements**: Smaller models deploy faster

### When NOT to Fine-Tune

- One-off queries (use prompting instead)
- Rapidly changing information (use RAG instead)
- Limited training data (< 100 examples typically insufficient)
- General knowledge questions (base model sufficient)

## Quick Start

**Full Fine-Tuning**:
```bash
python examples/full_fine_tuning.py
```

**LoRA (Recommended for most cases)**:
```bash
python examples/lora_fine_tuning.py
```

**QLoRA (Single GPU)**:
```bash
python examples/qlora_fine_tuning.py
```

**Data Preparation**:
```bash
python scripts/data_preparation.py
```

## Fine-Tuning Approaches

### 1. Full Fine-Tuning

Update all model parameters during training.

**Pros**:
- Maximum performance improvement
- Can completely rewrite model behavior
- Best for significant domain shifts

**Cons**:
- High computational cost
- Requires large dataset (1000+ examples)
- Risk of catastrophic forgetting
- Long training time

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer

model_id = "meta-llama/Llama-2-7b"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

training_args = TrainingArguments(
    output_dir="./fine-tuned-llama",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    weight_decay=0.01,
    logging_steps=10,
    save_steps=100,
    eval_strategy="steps",
    eval_steps=50,
    load_best_model_at_end=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

trainer.train()
```

### 2. Parameter-Efficient Fine-Tuning (PEFT)

Train only a small fraction of parameters.

#### LoRA (Low-Rank Adaptation)

Adds trainable low-rank matrices to existing weights.

**Pros**:
- 99% fewer trainable parameters
- Maintains base model knowledge
- Fast training (10-20x faster)
- Easy to switch between adapters

**Cons**:
- Slightly lower performance than full fine-tuning
- Requires base model at inference

```python
from peft import get_peft_model, LoraConfig, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "meta-llama/Llama-2-7b"
model = AutoModelForCausalLM.from_pretrained(base_model_id)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# Configure LoRA
lora_config = LoraConfig(
    r=8,  # Rank of low-rank matrices
    lora_alpha=16,  # Scaling factor
    target_modules=["q_proj", "v_proj"],  # Which layers to adapt
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

# Wrap model with LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,738,415,616 || trainable%: 0.06

# Train as normal
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)
trainer.train()

# Save only LoRA weights
model.save_pretrained("./llama-lora-adapter")
```

#### QLoRA (Quantized LoRA)

Combines LoRA with quantization for extreme efficiency.

```python
from peft import prepare_model_for_kbit_training, get_peft_model, LoraConfig
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# Quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_use_double_quant=True
)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b",
    quantization_config=bnb_config,
    device_map="auto"
)

# Prepare for training
model = prepare_model_for_kbit_training(model)

# Apply LoRA
lora_config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

model = get_peft_model(model, lora_config)

# Train on single GPU
trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="./qlora-output",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=4,
        learning_rate=5e-4,
        num_train_epochs=3,
    ),
    train_dataset=train_dataset,
)
trainer.train()
```

#### Prefix Tuning

Prepends trainable tokens to input.

```python
from peft import get_peft_model, PrefixTuningConfig

config = PrefixTuningConfig(
    num_virtual_tokens=20,
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, config)
# Only 20 * embedding_dim parameters trained
```

### 3. Instruction Fine-Tuning

Train model to follow instructions with examples.

```python
# Training data format
training_data = [
    {
        "instruction": "Translate to French",
        "input": "Hello, how are you?",
        "output": "Bonjour, comment allez-vous?"
    },
    {
        "instruction": "Summarize this text",
        "input": "Long document...",
        "output": "Summary..."
    }
]

# Template for training
template = """Below is an instruction that describes a task, paired with an input that provides further context.

### Instruction:
{instruction}

### Input:
{input}

### Response:
{output}"""

# Create formatted dataset
formatted_data = [
    template.format(**example) for example in training_data
]
```

### 4. Domain-Specific Fine-Tuning

Tailor models for specific industries or fields.

#### Legal Domain Example

```python
legal_training_data = [
    {
        "prompt": "What are the key clauses in an NDA?",
        "completion": """Key clauses typically include:
1. Definition of Confidential Information
2. Non-Disclosure Obligations
3. Permitted Disclosures
4. Term and Termination
5. Return of Information
6. Remedies"""
    },
    # ... more legal examples
]

# Train on legal domain
model = fine_tune_on_domain(
    base_model="gpt-3.5-turbo",
    training_data=legal_training_data,
    epochs=3,
    learning_rate=0.0002,
)
```

## Data Preparation

### 1. Dataset Quality

```python
class DatasetValidator:
    def validate_dataset(self, data):
        issues = {
            "empty_samples": 0,
            "duplicates": 0,
            "outliers": 0,
            "imbalance": {}
        }

        # Check for empty samples
        for sample in data:
            if not sample.get("text"):
                issues["empty_samples"] += 1

        # Check for duplicates
        texts = [s.get("text") for s in data]
        issues["duplicates"] = len(texts) - len(set(texts))

        # Check for length outliers
        lengths = [len(t.split()) for t in texts]
        mean_length = sum(lengths) / len(lengths)
        issues["outliers"] = sum(1 for l in lengths if l > mean_length * 3)

        return issues

# Validate before training
validator = DatasetValidator()
issues = validator.validate_dataset(training_data)
print(f"Dataset Issues: {issues}")
```

### 2. Data Augmentation

```python
from nlpaug.augmenter.word import SynonymAug, RandomWordAug
import nlpaug.flow as naf

# Create augmentation pipeline
text = "The quick brown fox jumps over the lazy dog"

# Synonym replacement
aug_syn = SynonymAug(aug_p=0.3)
augmented_syn = aug_syn.augment(text)

# Random word insert