bio-machine-learning-survival-analysis

What this skill does

## Version Compatibility

Reference examples tested with: scikit-survival 0.22+, lifelines 0.30+, numpy 1.26+, pandas 2.2+ (pycox 0.3+ for deep models).

Before using code patterns, verify installed versions match. If versions differ:
- Python: `pip show <package>` then `help(module.function)` to check signatures

scikit-survival requires the target `y` to be a structured array with a boolean event field and a float time field; its IPCW metrics need the *training* `y` first. lifelines `concordance_index` expects higher-score = longer-survival (negate the partial hazard). If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

# Predictive Survival Modeling

**"Build a validated risk model from time-to-event data"** -> Fit a penalized Cox or ensemble survival model, then evaluate with censoring-robust discrimination AND calibration, not the C-index alone.
- Penalized Cox / RSF / boosting: `sksurv.linear_model.CoxnetSurvivalAnalysis`, `sksurv.ensemble.RandomSurvivalForest`
- Evaluation: `concordance_index_ipcw`, `cumulative_dynamic_auc`, `integrated_brier_score`
- Deep survival (large n): `pycox` DeepSurv / DeepHit

## The Single Most Important Modern Insight -- C-Index Only Is the Cardinal Sin

The C-index is necessary but radically insufficient, for three reasons most papers miss. It is **censoring-distribution-dependent**: Harrell's C is biased upward under heavy censoring and gives different values on cohorts that differ only in follow-up -- report Uno's IPCW C with an explicit truncation tau instead (Uno 2011). It is **invariant to any monotone transform of the risk score**, so a model can have an excellent C and be wildly miscalibrated and clinically harmful -- C measures ranking, not the correctness of the predicted probabilities. And it is **insensitive**: adding a genuinely useful marker barely moves it, which is exactly why reclassification metrics (NRI/IDI) were invented. Decision-grade evaluation is Uno's C(tau) + time-dependent AUC(t) + integrated Brier vs a Kaplan-Meier baseline + calibration curves, all on honestly held-out or external data.

## ML-vs-Confirmatory Scope Boundary

Two survival cultures, two skills; mixing them is the most common authoring error. Resolve every overlap by one question: *is the goal to estimate and test a treatment effect in a (pre-specified) study, or to build and validate a risk-prediction model?*

| This skill (machine-learning) -- prediction | clinical-biostatistics/survival-analysis -- inference |
|---------------------------------------------|--------------------------------------------------------|
| Estimand is an individualized risk (survival curve, risk score, CIF) | Estimand is a treatment effect (hazard ratio with CI, p-value) |
| Penalized Cox, RSF, boosting, DeepSurv/DeepHit | Kaplan-Meier, log-rank, classical low-dimensional Cox |
| p>>n omics signatures; feature selection inside the resampling loop | Pre-specified analysis plan, FWER control, regulated trial |
| Judged by out-of-sample prediction (Uno's C, IBS, calibration) | Judged by validity of the HR and PH diagnostics (`cox.zph`) |
| PH is an assumption to relax (RSF/DeepHit do not assume it) | PH is a hypothesis whose violation invalidates the reported HR |

PH concepts, censoring definitions, and the Cox partial likelihood are foundational to both -- stated in clinical-biostatistics and referenced here. This skill's value begins at "I want a validated predictor."

## Model Taxonomy

| Model | Assumes PH? | Handles p>>n? | Competing risks? | Best when |
|-------|-------------|----------------|-------------------|-----------|
| Penalized Cox (elastic-net, coxnet) | Yes | Yes -- the omics workhorse; elastic-net handles correlated genes | Cause-specific by recoding | Sparse, interpretable, reproducible risk score; the default |
| Random Survival Forest | No | Yes (tune mtry/nodesize) | Yes (per-cause CIF) | Nonlinear/interaction effects, non-PH, moderate n |
| Gradient-boosted survival | Componentwise: yes (sparse); tree base: no | Yes (componentwise selects) | Via cause-specific | Boosting accuracy + sparsity, or relaxing PH with trees |
| Survival SVM | No (optimizes concordance) | Yes (kernel) | No | Pure ranking goals; gives a score, not a survival function |
| DeepSurv (pycox CoxPH) | Yes (NN replaces the linear predictor) | Needs large n | No | Large n, nonlinear main effects, PH plausible |
| DeepHit | No (discrete-time PMF) | Needs large n | Yes -- purpose-built | Large n + competing risks + non-PH |
| Cox-Time | No (time-dependent NN) | Needs large n | Discrete-hazard extensions | Large n, non-PH, flexible survival function |

Load-bearing empirical fact: on typical clinical-omics n, **penalized Cox and RSF are very hard to beat**; deep survival models usually only pull ahead at large n and/or with genuine non-PH or competing-risk structure. Start with elastic-net Cox and RSF baselines; escalate to deep models only if they demonstrably beat those on a held-out set.

## Decision Tree by Scenario

| Scenario | Recommended approach | Why |
|----------|---------------------|-----|
| Prognostic omics signature, p>>n | Elastic-net Cox (coxnet), selection inside nested CV | Sparse + correlated-gene grouping; the workhorse |
| Nonlinear/interaction effects, non-PH suspected | Random survival forest | Assumes no PH; captures interactions |
| Large n, suspected nonlinear main effects | DeepSurv, benchmarked against coxnet/RSF | Deep only earns its keep at large n |
| Competing events (death from other causes) | Fine-Gray (CIF) or DeepHit; report cause-specific too | 1-KM overestimates incidence; sHR is not the rate effect |
| Dynamic prediction with updating biomarkers | Landmarking | Internal time-varying covariates cannot be plugged into the future |
| Evaluating any survival model | Uno's C(tau) + AUC(t) + IBS-vs-KM + calibration | C-index alone is insufficient |
| KM curve, log-rank, or a trial hazard ratio | -> clinical-biostatistics/survival-analysis | Confirmatory inference, not prediction |
| Selecting the prognostic genes | -> machine-learning/biomarker-discovery (same irreproducibility) | Selection is its own discipline, inside the loop |

## Fitting Predictive Survival Models

**Goal:** Fit a penalized Cox and an RSF baseline with the correct target format.

**Approach:** Build the structured `y` (boolean event, float time), fit coxnet with an elastic-net mix and a baseline model for survival functions, and an RSF for nonlinear structure.

```python
from sksurv.util import Surv
from sksurv.linear_model import CoxnetSurvivalAnalysis
from sksurv.ensemble import RandomSurvivalForest

# event MUST be bool, time float. Field order in from_arrays is (event, time).
y = Surv.from_arrays(event=df['status'].astype(bool), time=df['time'].astype(float))

# Elastic-net Cox: l1_ratio in (0,1]; fit_baseline_model=True enables predict_survival_function.
coxnet = CoxnetSurvivalAnalysis(l1_ratio=0.9, alpha_min_ratio=0.01, fit_baseline_model=True)
coxnet.fit(X_train, y_train)

rsf = RandomSurvivalForest(n_estimators=500, min_samples_leaf=15, max_features='sqrt', n_jobs=-1)
rsf.fit(X_train, y_train)
risk = coxnet.predict(X_test)              # a risk score (higher = higher risk), NOT a probability
```

## Prediction-Grade Evaluation

**Goal:** Report discrimination AND calibration on out-of-sample data, not the C-index alone.

**Approach:** Use Uno's IPCW C (truncated at tau), time-dependent AUC over a horizon grid, integrated Brier vs the KM baseline, and a calibration check at clinical horizons. IPCW metrics take the *training* `y` first to estimate the censoring distribution.

```python
import numpy as np
from sksurv.metrics import concordance_index_ipcw, cumulative_dynamic_auc, integrated_brier_score

c_uno = concordance_index_ipcw(y_train, y_test, risk, tau=t_horizon)[0]      # censoring-robust

times = np.percentile(y_test['time'][y_test[