scikit-learn

What this skill does

# Scikit-learn: Machine Learning in Python

## Overview

Scikit-learn is Python's premier machine learning library, offering simple and efficient tools for predictive data analysis. Apply this skill for classification, regression, clustering, dimensionality reduction, model selection, preprocessing, and hyperparameter optimization.

## When to Use This Skill

This skill should be used when:
- Building classification models (spam detection, image recognition, medical diagnosis)
- Creating regression models (price prediction, forecasting, trend analysis)
- Performing clustering analysis (customer segmentation, pattern discovery)
- Reducing dimensionality (PCA, t-SNE for visualization)
- Preprocessing data (scaling, encoding, imputation)
- Evaluating model performance (cross-validation, metrics)
- Tuning hyperparameters (grid search, random search)
- Creating machine learning pipelines
- Detecting anomalies or outliers
- Implementing ensemble methods

## Core Machine Learning Workflow

### Standard ML Pipeline

Follow this general workflow for supervised learning tasks:

1. **Data Preparation**
   - Load and explore data
   - Split into train/test sets
   - Handle missing values
   - Encode categorical features
   - Scale/normalize features

2. **Model Selection**
   - Start with baseline model
   - Try more complex models
   - Use domain knowledge to guide selection

3. **Model Training**
   - Fit model on training data
   - Use pipelines to prevent data leakage
   - Apply cross-validation

4. **Model Evaluation**
   - Evaluate on test set
   - Use appropriate metrics
   - Analyze errors

5. **Model Optimization**
   - Tune hyperparameters
   - Feature engineering
   - Ensemble methods

6. **Deployment**
   - Save model using joblib
   - Create prediction pipeline
   - Monitor performance

### Classification Quick Start

```python
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
from sklearn.pipeline import Pipeline

# Create pipeline (prevents data leakage)
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
])

# Split data (use stratify for imbalanced classes)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Train
pipeline.fit(X_train, y_train)

# Evaluate
y_pred = pipeline.predict(X_test)
print(classification_report(y_test, y_pred))

# Cross-validation for robust evaluation
from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipeline, X_train, y_train, cv=5)
print(f"CV Accuracy: {scores.mean():.3f} (+/- {scores.std():.3f})")
```

### Regression Quick Start

```python
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.pipeline import Pipeline

# Create pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('regressor', RandomForestRegressor(n_estimators=100, random_state=42))
])

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train
pipeline.fit(X_train, y_train)

# Evaluate
y_pred = pipeline.predict(X_test)
rmse = mean_squared_error(y_test, y_pred, squared=False)
r2 = r2_score(y_test, y_pred)
print(f"RMSE: {rmse:.3f}, R²: {r2:.3f}")
```

## Algorithm Selection Guide

### Classification Algorithms

**Start with baseline**: LogisticRegression
- Fast, interpretable, works well for linearly separable data
- Good for high-dimensional data (text classification)

**General-purpose**: RandomForestClassifier
- Handles non-linear relationships
- Robust to outliers
- Provides feature importance
- Good default choice

**Best performance**: HistGradientBoostingClassifier
- State-of-the-art for tabular data
- Fast on large datasets (>10K samples)
- Often wins Kaggle competitions

**Special cases**:
- **Small datasets (<1K)**: SVC with RBF kernel
- **Very large datasets (>100K)**: SGDClassifier or LinearSVC
- **Interpretability critical**: LogisticRegression or DecisionTreeClassifier
- **Probabilistic predictions**: GaussianNB or calibrated models
- **Text classification**: LogisticRegression with TfidfVectorizer

### Regression Algorithms

**Start with baseline**: LinearRegression or Ridge
- Fast, interpretable
- Works well when relationships are linear

**General-purpose**: RandomForestRegressor
- Handles non-linear relationships
- Robust to outliers
- Good default choice

**Best performance**: HistGradientBoostingRegressor
- State-of-the-art for tabular data
- Fast on large datasets

**Special cases**:
- **Regularization needed**: Ridge (L2) or Lasso (L1 + feature selection)
- **Very large datasets**: SGDRegressor
- **Outliers present**: HuberRegressor or RANSAC

### Clustering Algorithms

**Known number of clusters**: KMeans
- Fast and scalable
- Assumes spherical clusters

**Unknown number of clusters**: DBSCAN or HDBSCAN
- Handles arbitrary shapes
- Automatic outlier detection

**Hierarchical relationships**: AgglomerativeClustering
- Creates hierarchy of clusters
- Good for visualization (dendrograms)

**Soft clustering (probabilities)**: GaussianMixture
- Provides cluster probabilities
- Handles elliptical clusters

### Dimensionality Reduction

**Preprocessing/feature extraction**: PCA
- Fast and efficient
- Linear transformation
- ALWAYS standardize first

**Visualization only**: t-SNE or UMAP
- Preserves local structure
- Non-linear
- DO NOT use for preprocessing

**Sparse data (text)**: TruncatedSVD
- Works with sparse matrices
- Latent Semantic Analysis

**Non-negative data**: NMF
- Interpretable components
- Topic modeling

## Working with Different Data Types

### Numeric Features

**Continuous features**:
1. Check distribution
2. Handle outliers (remove, clip, or use RobustScaler)
3. Scale using StandardScaler (most algorithms) or MinMaxScaler (neural networks)

**Count data**:
1. Consider log transformation or sqrt
2. Scale after transformation

**Skewed data**:
1. Use PowerTransformer (Yeo-Johnson or Box-Cox)
2. Or QuantileTransformer for stronger normalization

### Categorical Features

**Low cardinality (<10 categories)**:
```python
from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(drop='first', sparse_output=True)
```

**High cardinality (>10 categories)**:
```python
from sklearn.preprocessing import TargetEncoder
encoder = TargetEncoder()
# Uses target statistics, prevents leakage with cross-fitting
```

**Ordinal relationships**:
```python
from sklearn.preprocessing import OrdinalEncoder
encoder = OrdinalEncoder(categories=[['small', 'medium', 'large']])
```

### Text Data

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

text_pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(max_features=1000, stop_words='english')),
    ('classifier', MultinomialNB())
])

text_pipeline.fit(X_train_text, y_train)
```

### Mixed Data Types

```python
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

# Define feature types
numeric_features = ['age', 'income', 'credit_score']
categorical_features = ['country', 'occupation']

# Separate preprocessing pipelines
numeric_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('onehot', OneHotEncoder(handle_unknown='ignore', sparse_output=True))
])

# Combine with ColumnTransformer
preprocessor = ColumnTransform