bio-machine-learning-model-validation

What this skill does

## Version Compatibility

Reference examples tested with: numpy 1.26+, scikit-learn 1.4+ (note 1.6/1.8 API changes below).

Before using code patterns, verify installed versions match. If versions differ:
- Python: `pip show <package>` then `help(module.function)` to check signatures

scikit-learn drift to watch: `CalibratedClassifierCV(cv='prefit')` was deprecated in 1.6 and removed in 1.8 (it now raises; wrap a fitted model in `sklearn.frozen.FrozenEstimator` instead); `ensemble` default became `'auto'` in 1.6; `method='temperature'` was added in 1.8. If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

# Model Validation for Biomedical and Omics Data

**"Validate my omics classifier honestly"** -> Keep every data-dependent step inside the resampling loop, never use the same data to both choose and grade, and report calibration and net benefit, not just AUC.
- Nested CV: `GridSearchCV` (inner) wrapped by `cross_val_score` (outer)
- Group/structured: `StratifiedGroupKFold`, `TimeSeriesSplit`
- Calibration: `calibration_curve`, `brier_score_loss`, `CalibratedClassifierCV`

## The Single Most Important Modern Insight -- A Reported Number Is an Honest Estimate Only If Nothing Leaked and Nothing Was Graded on What Was Chosen

A reported performance number is a claim about a data-generating process that will never recur. Almost every inflated result in ML-for-biology traces to one of two root causes: information from the test distribution leaked into model construction, or the same data was used to both choose and grade a decision. A clean train/test split is necessary but nowhere near sufficient -- the leakage has usually already contaminated the test set (a scaler fit on all data, a duplicate patient, ComBat run across the split). Leakage causes a reproducibility crisis across ML-based science (Kapoor 2023), and the bias is largest exactly when the true signal is weakest -- the omics regime.

A second, equally load-bearing insight: discrimination (AUC/C) and calibration (do predicted probabilities match observed frequencies) are *orthogonal*. AUC is invariant to any monotone transform of the score, so it is blind to calibration. For any decision that uses the probability itself, calibration -- not AUC -- is the property that matters, and it is the one routinely ignored (Van Calster 2019, "the Achilles heel").

## Leakage Taxonomy

| Leakage type | How it happens in omics | Symptom | Prevention |
|--------------|-------------------------|---------|------------|
| Preprocessing (most common, most missed) | z-scoring, quantile/library normalization, ComBat/SVA, PCA, kNN/MICE imputation, VST fit on the *full* dataset before splitting | Test performance suspiciously close to train; collapses on external data | Fit every transform inside the CV fold via a `Pipeline` |
| Feature selection (severe special case) | top-k DE genes / highest-variance / univariate filter chosen on all samples, then CV only the classifier | Near-perfect CV from pure noise; unstable selected set | Selection lives in the CV fold (Ambroise 2002) |
| Target / label | a feature is a proxy for or downstream of the outcome (post-diagnosis labs, treatment-derived fields, a collection-site that tracks case/control) | One feature dominates implausibly; fails when removed | Audit temporal/causal admissibility; exclude post-outcome variables |
| Group / patient / replicate | same patient, tumor, organoid, or technical replicate in train and test; `KFold` scatters them | Inflated metrics that vanish under leave-one-group-out | Split by the highest independent unit (`GroupKFold`/`StratifiedGroupKFold`) |
| Batch | batch correlated with outcome and not respected in the split, or ComBat across the train/test boundary | Model discriminates batches not biology; external batch destroys it | Block the split by batch; never run unsupervised correction across the split |
| Temporal | random-splitting time-ordered data; future-period statistics standardize the past | Backtest beats prospective deployment | Time-based split (`TimeSeriesSplit`); never shuffle first |
| Duplicate / homolog | near-identical samples, augmented copies, public-dataset overlap, homologous sequences across the split | Memorization passes as generalization | Deduplicate / cluster-then-split before CV |
| Test-reuse / threshold | repeatedly peeking to pick features, thresholds, "best epoch"; choosing the classification threshold on the test set | Irreproducible SOTA; fragile config | One locked test set; all tuning + thresholds inside nested CV |

## Decision Tree by Scenario

| Scenario / generalization question | Recommended scheme | Why |
|------------------------------------|--------------------|-----|
| "A new sample like training" (and any tuning occurs) | Nested CV: inner `GridSearchCV`, outer `cross_val_score`, Pipeline inside | Tuning and grading on the same CV is optimistic (Cawley-Talbot 2010) |
| "A new patient" (repeated measures) | `GroupKFold`/`StratifiedGroupKFold` by patient/donor | The unit of independence is not the row |
| "A new hospital/site" (transportability) | Leave-one-site-out (internal-external CV) | Approximates external validation |
| "Next year" (time-ordered) | `TimeSeriesSplit` forward-chaining | Random folds leak the future |
| Small n (dozens), need a stable estimate | `RepeatedStratifiedKFold` (5x10) with an interval | A single CV is one high-variance draw |
| Probabilities will drive a decision | Add calibration + decision-curve net benefit | AUC is blind to calibration and utility |
| Final evidence for a clinical model | External/temporal validation + TRIPOD+AI report | Internal CV cannot detect a whole-dataset confound |
| Choosing the features themselves | -> machine-learning/biomarker-discovery | Selection is its own discipline (run it inside the fold) |
| Confirmatory trial inference (HR, p-value) | -> clinical-biostatistics/trial-reporting | Estimand is a treatment effect, not a prediction |

## Nested Cross-Validation

**Goal:** Estimate the performance of the whole procedure (tuning + fit) without optimistic bias.

**Approach:** The inner loop does all tuning, feature selection, and threshold choice; the outer loop grades the winning configuration once on a fold it never touched. The reported number is the aggregate over outer folds; it answers "if I run this pipeline on new data, what do I get?"

```python
from sklearn.model_selection import cross_val_score, StratifiedKFold, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([('scaler', StandardScaler()),
                 ('select', SelectKBest(f_classif)),         # re-fit per inner fold -> no leakage
                 ('clf', LogisticRegression(max_iter=5000))])
grid = {'select__k': [10, 50, 200], 'clf__C': [0.01, 0.1, 1]}

inner = StratifiedKFold(5, shuffle=True, random_state=0)
outer = StratifiedKFold(5, shuffle=True, random_state=1)
search = GridSearchCV(pipe, grid, cv=inner, scoring='roc_auc')
scores = cross_val_score(search, X, y, cv=outer, scoring='roc_auc')   # unbiased estimate
print(f'Nested AUC: {scores.mean():.3f} +/- {scores.std():.3f}')
```

Nested CV is needed whenever model selection happens -- even informal "tried three options, kept the best." Flat CV with tuning is a known reviewer red flag (Varma-Simon 2006).

## Group-Aware, Structured, and Small-Sample CV

**Goal:** Match the CV scheme to the real unit of independence and get a variance-aware estimate.

**Approach:** Pass a grouping vector so no group spans folds; for tiny n, repeat stratified k-fold and report the spread, not a bare number. Standard `KFold` assumes i.i.d. rows, which biomedical data almost never satisfy.

```python
from sklearn.model_selection import StratifiedGro