bio-experimental-design-batch-design

What this skill does

## Version Compatibility

Reference examples tested with: designit 0.5+, OSAT 1.50+ (Bioconductor), sva 3.50+, RUVSeq 1.36+, limma 3.58+, edgeR 4.0+.

Before using code patterns, verify installed versions match. If versions differ:
- R: `packageVersion('<pkg>')` then `?function_name` to verify parameters

If code throws an error, introspect the installed package and adapt to the actual API. Notes: OSAT uses `optimal.shuffle()` on a setup object (there is no bare `osat()` function); designit is R6 (`BatchContainer$new()`, `optimize_design()`, `*_score_generator()`) and its signatures drift between releases; `sva::ComBat_seq()` is for integer counts while `ComBat()` expects log-normalized values. Confirm each against the installed vignette before relying on it.

# Batch Design

**"Design my experiment so batch effects don't ruin it"** -> Assign samples to batches/lanes/plates so the biological variable is balanced against (orthogonal to) every technical nuisance factor, making batch estimable rather than confounded — because no post-hoc correction recovers a design where batch and condition are aliased.
- R: `designit::optimize_design()`, `OSAT::optimal.shuffle()` — constrained assignment at design time
- R: `sva::sva()`/`num.sv()` — detect hidden batches; `sva::ComBat_seq()`, `RUVSeq::RUVg()` — DOWNSTREAM correction (executed in differential-expression/batch-correction)

## The Single Most Important Modern Insight -- No Post-Hoc Method Recovers a Confounded Design

When a technical factor is perfectly aliased with the biological factor (all treated in batch 1, all controls in batch 2), batch and condition occupy the same column space and are mathematically non-identifiable; ComBat, SVA, and RUV cannot separate them, and "removing the batch effect" removes the biology with it. Worse, on *partially* confounded or merely **unbalanced** designs, mean-centering batches can *manufacture* false positives and inflate downstream confidence — Nygaard, Rødland & Hovig 2016 *Biostatistics* 17:29 showed a pipeline that returned >1000 spurious DE probes where the honest analysis (batch kept *in the model*) found 11. The operative rules: (1) balance the biological variable across batches **at design time** because correction is not a rescue (Leek 2010 *Nat Rev Genet* 11:733); (2) for inference, keep batch *in the model* so its degrees of freedom are charged honestly — reserve a batch-"cleaned" matrix for visualization and clustering only. The experimental unit, not the measurement, still defines replication (see experimental-design/randomization-blocking).

## Confounding vs Blocking vs Nuisance -- the Causal-Graph View

Batch is the genomics face of confounding, and not all metadata should be "adjusted for". A **confounder** is a common cause of treatment and outcome (adjust for it); a **mediator** lies on the causal path (adjusting removes signal); a **collider** is a common effect (adjusting *induces* spurious association). "Adjust for everything measured" is therefore wrong in general — conditioning on a collider opens a backdoor path. In a properly *randomized* design the biological variable has no confounders by construction, so batch is handled by **balanced assignment + a block/covariate term**, not by scrubbing every measured variable.

## Algorithmic Taxonomy -- Design Strategies for Technical Variation

| Strategy | What it does | When to use | Fails when |
|----------|--------------|-------------|------------|
| Balanced (orthogonal) assignment | every condition appears equally in every batch | always achievable when batches hold >=1 of each condition | not all conditions fit per batch |
| Block randomization across batches | randomize condition within each batch | batch = a block; conditions fit per batch | batch variance is genuinely zero (rare) |
| Incomplete block + batch in model | conditions split across smaller batches, batch term retained | plate/chip smaller than #conditions | unbalanced split inflates artifacts (Nygaard 2016) |
| Reference / bridge sample per batch | shared anchor measured in every batch | cross-batch normalization (TMT proteomics, large cohorts) | anchor not representative |
| Multiplexing + demultiplexing | pool biological units in one lane, split by barcode/genotype | breaking the donor<->lane confound (scRNA-seq) | insufficient SNPs/hashes to assign |
| Run-order randomization | randomize processing/injection order | position/time gradients (LC-MS, plate edge) | order set by convenience |

## Decision Tree by Scenario

| Scenario | Recommended design | Why |
|----------|--------------------|-----|
| 24 samples, 3 batches, 2 conditions | balanced: 4 of each condition per batch | batch orthogonal to condition; estimable |
| Conditions outnumber batch capacity | incomplete block; keep batch in the DE model | preserves estimability; no scrubbing |
| Large cohort across many runs | include a shared reference sample per batch | enables cross-batch normalization |
| scRNA-seq, several donors, few lanes | pool donors per lane, demultiplex (demuxlet / hashing) | removes donor<->lane confound (Kang 2018) |
| Hidden/unknown technical structure suspected | estimate surrogate variables (SVA), include in model | captures unmodeled variation (Leek & Storey 2007) |
| Design already confounds batch with condition | redesign; no correction will rescue it | non-identifiable (Nygaard 2016; Leek 2010) |
| General randomization / blocking / unit choice | -> experimental-design/randomization-blocking | foundational design structure |
| Running ComBat-seq / RUVSeq / SVA on real data | -> differential-expression/batch-correction | execution lives there; this skill decides |

## Confounded vs Balanced -- the Canonical Contrast

**Goal:** Make batch effects correctable by keeping the biological variable orthogonal to batch.

**Approach:** Never place all of one condition in one batch. Distribute conditions (and known covariates such as sex) equally across batches so a linear model can estimate batch and condition separately.

```r
# BAD (confounded): batch is aliased with condition -> non-identifiable
#   batch 1: treat, treat, treat, treat       batch 2: ctrl, ctrl, ctrl, ctrl
# GOOD (balanced): batch is orthogonal to condition -> batch effect estimable, removable
#   batch 1: 2 treat + 2 ctrl                 batch 2: 2 treat + 2 ctrl
```

## Constrained Sample-to-Batch Assignment

**Goal:** Allocate samples to batches/lanes/plates to minimize correlation between batch and the biological variables of interest.

**Approach:** Use a block-randomization-with-optimization tool that scores assignments by how evenly the biological factors spread across batches and returns a near-optimal layout.

```r
library(designit)                              # verify API against installed vignette
samples <- data.frame(id = sprintf('S%02d', 1:24),
                      condition = rep(c('ctrl', 'treat'), each = 12),
                      sex = rep(c('M', 'F'), 12))
bc <- BatchContainer$new(dimensions = list(batch = 3, position = 8))
bc <- assign_in_order(bc, samples = samples)
bc <- optimize_design(
  bc,
  scoring = osat_score_generator(batch_vars = 'batch',
                                 feature_vars = c('condition', 'sex')))   # balance both factors
assignment <- bc$get_samples()                # R6 method on the container (no standalone get_samples())

# OSAT alternative (Bioconductor): build a setup object, then optimal.shuffle() -- NOT a bare osat().
```

## Downstream Correction -- Choose by Design, Execute Elsewhere

Correction method selection is a design decision; the execution lives in differential-expression/batch-correction (and single-cell/batch-integration for scRNA-seq). Prefer keeping batch in the analysis model over producing a "cleaned" matrix for inference.

| Method | When it applies | Assumption / caveat | Owner of execution |
|--------|-----------------|---------------------|--------------------|
| Batch as a model covariate | batch known,