xgboost-lightgbm

What this skill does

# XGBoost & LightGBM - Gradient Boosting for Tabular Data

XGBoost (eXtreme Gradient Boosting) and LightGBM (Light Gradient Boosting Machine) are the de facto standard libraries for machine learning on tabular/structured data. They consistently win Kaggle competitions and are widely used in industry for their speed, accuracy, and robustness.

## When to Use

- Classification or regression on tabular data (CSVs, databases, spreadsheets).
- Kaggle competitions or data science competitions on structured data.
- Feature importance analysis and feature selection.
- Handling missing values automatically (no need to impute).
- Working with imbalanced datasets (built-in class weighting).
- Need for fast training on large datasets (millions of rows).
- Hyperparameter tuning with cross-validation.
- Ranking tasks (learning-to-rank algorithms).
- When you need interpretable feature importances.
- Production ML systems requiring fast inference on tabular data.

## Reference Documentation

**XGBoost Official**: https://xgboost.readthedocs.io/  
**XGBoost GitHub**: https://github.com/dmlc/xgboost  
**LightGBM Official**: https://lightgbm.readthedocs.io/  
**LightGBM GitHub**: https://github.com/microsoft/LightGBM  
**Search patterns**: `xgboost.XGBClassifier`, `lightgbm.LGBMRegressor`, `xgboost.train`, `lightgbm.cv`

## Core Principles

### Gradient Boosting Trees
Both libraries build an ensemble of decision trees sequentially, where each new tree corrects errors from previous trees. This creates highly accurate models that capture complex non-linear patterns.

### Speed vs Accuracy Trade-offs
**XGBoost**: Slower but often slightly more accurate. Better for smaller datasets (<100k rows).  
**LightGBM**: Faster, especially on large datasets (millions of rows). Uses histogram-based learning.

### Regularization
Both include L1/L2 regularization (alpha, lambda parameters) to prevent overfitting. This is crucial when you have many features.

### Handling Categorical Features
LightGBM has native categorical feature support. XGBoost requires encoding (label encoding or one-hot).

## Quick Reference

### Installation

```bash
# Install both
pip install xgboost lightgbm

# For GPU support
pip install xgboost[gpu] lightgbm[gpu]
```

### Standard Imports

```python
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, mean_squared_error

# XGBoost
import xgboost as xgb
from xgboost import XGBClassifier, XGBRegressor

# LightGBM
import lightgbm as lgb
from lightgbm import LGBMClassifier, LGBMRegressor
```

### Basic Pattern - Classification with XGBoost

```python
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split

# 1. Prepare data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 2. Create and train model
model = XGBClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=6,
    random_state=42
)
model.fit(X_train, y_train)

# 3. Predict and evaluate
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")
```

### Basic Pattern - Regression with LightGBM

```python
from lightgbm import LGBMRegressor

# 1. Create model
model = LGBMRegressor(
    n_estimators=100,
    learning_rate=0.1,
    num_leaves=31,
    random_state=42
)

# 2. Train
model.fit(X_train, y_train)

# 3. Predict
y_pred = model.predict(X_test)
rmse = mean_squared_error(y_test, y_pred, squared=False)
print(f"RMSE: {rmse:.4f}")
```

## Critical Rules

### ✅ DO

- **Use Early Stopping** - Always use early stopping with a validation set to prevent overfitting and save training time.
- **Start with Defaults** - Both libraries have excellent default parameters. Start there before tuning.
- **Monitor Training** - Use `eval_set` parameter to track validation metrics during training.
- **Handle Imbalance** - For imbalanced classes, use `scale_pos_weight` (XGBoost) or `class_weight` (LightGBM).
- **Feature Engineering** - Create interaction features, polynomial features, aggregations - boosting excels with rich feature sets.
- **Use Native API for Advanced Control** - For complex tasks, use `xgb.train()` or `lgb.train()` instead of sklearn wrappers.
- **Save Models Properly** - Use `.save_model()` and `.load_model()` methods, not pickle (more robust).
- **Check Feature Importance** - Always examine feature importances to understand your model and detect data leakage.

### ❌ DON'T

- **Don't Forget to Normalize Target** - For regression, if target has wide range, consider log-transform or standardization.
- **Don't Ignore Tree Depth** - `max_depth` (XGBoost) or `num_leaves` (LightGBM) are critical. Too deep = overfit.
- **Don't Use Default Learning Rate for Large Datasets** - Reduce `learning_rate` to 0.01-0.05 for datasets >1M rows.
- **Don't Mix Up Parameters** - XGBoost uses `max_depth`, LightGBM uses `num_leaves`. They're different!
- **Don't One-Hot Encode for LightGBM** - Use categorical_feature parameter instead for better performance.
- **Don't Skip Cross-Validation** - Always CV before trusting a single train/test split.

## Anti-Patterns (NEVER)

```python
# ❌ BAD: Training without validation set or early stopping
model = XGBClassifier(n_estimators=1000)
model.fit(X_train, y_train)  # Will likely overfit

# ✅ GOOD: Use early stopping with validation
model = XGBClassifier(n_estimators=1000, early_stopping_rounds=10)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    verbose=False
)

# ❌ BAD: One-hot encoding categorical features for LightGBM
X_encoded = pd.get_dummies(X)  # Creates many sparse columns
model = LGBMClassifier()
model.fit(X_encoded, y)

# ✅ GOOD: Use categorical_feature parameter
model = LGBMClassifier()
model.fit(
    X, y,
    categorical_feature=['category_col1', 'category_col2']
)

# ❌ BAD: Ignoring class imbalance
model = XGBClassifier()
model.fit(X_train, y_train)  # Majority class dominates

# ✅ GOOD: Handle imbalance with scale_pos_weight
from sklearn.utils.class_weight import compute_sample_weight
scale_pos_weight = (y_train == 0).sum() / (y_train == 1).sum()
model = XGBClassifier(scale_pos_weight=scale_pos_weight)
model.fit(X_train, y_train)
```

## XGBoost Fundamentals

### Scikit-learn Style API

```python
from xgboost import XGBClassifier
import numpy as np

# Binary classification
model = XGBClassifier(
    n_estimators=100,        # Number of trees
    max_depth=6,             # Maximum tree depth
    learning_rate=0.1,       # Step size shrinkage (eta)
    subsample=0.8,           # Row sampling ratio
    colsample_bytree=0.8,    # Column sampling ratio
    random_state=42
)

# Train with early stopping
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    early_stopping_rounds=10,
    verbose=True
)

# Get best iteration
print(f"Best iteration: {model.best_iteration}")
print(f"Best score: {model.best_score}")
```

### Native XGBoost API (More Control)

```python
import xgboost as xgb

# 1. Create DMatrix (XGBoost's internal data structure)
dtrain = xgb.DMatrix(X_train, label=y_train)
dval = xgb.DMatrix(X_val, label=y_val)

# 2. Set parameters
params = {
    'objective': 'binary:logistic',  # or 'reg:squarederror' for regression
    'max_depth': 6,
    'eta': 0.1,                      # learning_rate
    'subsample': 0.8,
    'colsample_bytree': 0.8,
    'eval_metric': 'auc',
    'seed': 42
}

# 3. Train with cross-validation monitoring
evals = [(dtrain, 'train'), (dval, 'val')]
model = xgb.train(
    params,
    dtrain,
    num_boost_round=1000,
    evals=evals,
    early_stopping_rounds=10,
    verbose_eval=50
)

# 4. Predict
dtest = xgb.DMatrix(X_test)
y_pred_proba = model.predict(dtest)
y_pred = (y_pred_proba > 0.5).astype(int)
```

### Cross-Validation

```python
import xgboost as xgb

# Prepare data
dtrain = xgb.DMatrix(X_train, label=y_train)

# Parameters
params = {
    'objective': 'binary:logistic',
    'max_depth'