bio-imaging-mass-cytometry-data-preprocessing

What this skill does

## Version Compatibility

Reference examples tested with: steinbock 0.16+, readimc 0.7+, numpy 1.26+, CATALYST 1.28+ (R/Bioconductor), cytomapper 1.16+ (R)

Before using code patterns, verify installed versions match. If versions differ:
- Python: `pip show <package>` then `help(module.function)` to check signatures
- R: `packageVersion('<pkg>')` then `?function_name` to verify parameters
- CLI: `<tool> --version` then `<tool> --help` to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed
package and adapt the example to match the actual API rather than retrying.

Notes specific to this skill: IMC pixels are integer ion COUNTS, not fluorescence intensities. `CATALYST::compCytof` defaults to `method='nnls'` (non-negativity preserved); pass `cofactor=1` explicitly for IMC single-cell means (the compCytof default is `NULL`/5, the suspension value). The spillover-matrix and SCE channel names must both be `(metal)(mass)Di` (e.g. `Sm152Di`) or compensation silently no-ops. steinbock's hot-pixel filter is `steinbock preprocess imc images --hpf 50` (a signed 8-neighbor difference, not a median filter).

# IMC Data Preprocessing

**"Preprocess my imaging mass cytometry data"** -> Ingest raw acquisitions, suppress acquisition noise, compensate channel spillover, and variance-stabilize counts so each cell's measured intensity reflects real antigen abundance.
- Python/CLI: `readimc.MCDFile`, `steinbock preprocess imc images --hpf 50`
- R: `CATALYST::compCytof`, `cytomapper::compImage` for spillover compensation

## The Single Most Important Modern Insight -- IMC pixels are ion counts, and non-negativity is physics, not a preference

Every IMC/MIBI pixel is an integer number of detected metal ions from one ~1 um laser shot, drawn from a low-count, zero-inflated, near-Poisson regime where most pixels read 0-2 counts and the limit of detection is ~6 counts. Four consequences govern every preprocessing decision, and importing fluorescence-microscopy habits violates all four. (1) There is no continuous Gaussian background to subtract -- the floor is a count floor, and a true-negative pixel still reads 1-2 counts by Poisson chance. (2) Negative values are physically meaningless, so flow-style spillover compensation (exact matrix inverse) is wrong because it manufactures negatives; non-negative least squares (NNLS) is mandatory (Chevrier 2018 *Cell Syst* 6:612). (3) Per-pixel "expression" is mostly shot noise -- signal emerges only after segmentation sums a cell's pixels, so pixel maps are for localization and QC, never quantification. (4) Spillover is SPATIAL: a bright cell bleeding into a neighboring mass channel contaminates adjacent pixels, fabricating co-localization and false marker positivity at cell borders -- which means uncompensated spillover manufactures the exact cell-cell interactions IMC exists to measure. The corollary that trips up suspension-CyTOF veterans: the arcsinh cofactor is NOT 5 (cofactor 1 is the modern IMC default, Hunter 2024 *Cytometry A* 105:36), and there are no in-stream calibration beads in ablated tissue.

## Preprocessing Pipeline Order (load-bearing)

```
read .mcd (readimc) -> panel filter+sort (keep column) -> hot-pixel removal (DIMR or --hpf 50)
  -> [optional] DeepSNiF on low-SNR channels only -> spillover compensation (NNLS)
  -> segmentation (on compensated nuclear/membrane channels) -> per-cell aggregation
  -> arcsinh(cofactor 1) -> z-score / cohort-anchored normalization
```

Order is not cosmetic: denoise operates on RAW counts (the Poisson noise model is defined there), compensate BEFORE segmentation when spatial fidelity matters (corrupted membrane channels yield wrong boundaries that no later step recovers), and transform/normalize LAST.

## Denoising Taxonomy

| Method | Targets | Risk | When to use | Fails when |
|--------|---------|------|-------------|------------|
| steinbock `--hpf 50` | hot pixels | low | default fast hot-pixel pass | absolute-count threshold over-clips bright channels, under-cleans dim ones |
| IMC-Denoise DIMR (Lu 2023) | hot pixels | low | self-calibrating hot-pixel removal | misclassifies large multi-pixel hot-pixel clusters as signal |
| IMC-Denoise DeepSNiF (Lu 2023) | shot noise | HIGH | only channels with mean positive intensity < ~7 | over-smooths sparse/punctate markers and sub-1-2 um structure; biases extreme-low-count regions |
| 3x3 median filter | (do not use) | severe | never | returns 0 for isolated real single-positive pixels -- erases sparse biology |

## Decision Tree by Scenario

| Scenario | Recommended | Why |
|----------|-------------|-----|
| Single-stain QC shows >~2% off-target spillover at relevant masses | Compensate (NNLS) before phenotyping | leak corrupts type calls and (spatially) neighborhood stats |
| Spatial neighborhood / interaction analysis is the endpoint | Pixel-level compensation (`compImage`) before segmentation | spillover is spatial and fakes interactions; cell-mean compensation comes too late |
| Means-only phenotyping, segmentation channels uncontaminated | Cell-level `compCytof` on SCE means | cheaper, less per-pixel Poisson noise |
| Channel driven above ~5,000 dual counts | Re-titrate at acquisition; do not compensate | linearity (and thus the matrix) breaks above saturation |
| Panel pre-designed to avoid bright/dim mass adjacencies, QC near-clean | Compensation ~ identity; skipping is defensible | avoids NNLS noise on near-zero pixels |
| Channel mean positive intensity < ~7, cannot phenotype on it | DIMR + DeepSNiF | shot noise dominates; denoising is the only way to use it |
| Channel clean, or punctate, or the one segmentation runs on | DIMR / `--hpf` only; skip DeepSNiF | DeepSNiF over-smooths real isolated signal and blurs boundaries |

## Read Raw Acquisitions

**Goal:** Ingest the multi-ROI MCD (the canonical source) and keep the metal-to-target panel mapping intact.

**Approach:** Open the MCD with readimc, iterate slides and acquisitions, and carry both `channel_names` (metals) and `channel_labels` (targets) -- conflating them is the most common ingestion bug. Prefer MCD over TXT (one ROI per file, and absent on Hyperion XTi).

```python
from readimc import MCDFile

with MCDFile('slide.mcd') as f:
    slide = f.slides[0]
    for acq in slide.acquisitions:
        img = f.read_acquisition(acq)              # (channels, y, x) float32 ion counts
        metals = acq.channel_names                 # e.g. 'Sm152' -- the mass channel
        targets = acq.channel_labels               # e.g. 'CD3'  -- the antibody target
        print(acq.id, img.shape, dict(zip(metals, targets)))
```

## Extract and Hot-Pixel Filter with steinbock

**Goal:** Build per-channel TIFF stacks, filtered to the analysis panel and de-spiked.

**Approach:** The panel CSV is both a filter and a sort key -- only `keep==1` rows are written and their row order defines channel order in the stack, so it must be pinned as a versioned artifact. The `--hpf` filter compares each pixel to its 8 neighbors with a signed difference and replaces spikes with the neighbor maximum (a conservative, valley-preserving operation), not a median.

```bash
# generate the panel template (edit the keep column before extracting)
steinbock preprocess imc panel

# extract TIFFs (keep-filtered, panel-ordered) with hot-pixel removal; 50 is a count
# difference, not a universal constant -- raise it for high-dynamic-range markers
steinbock preprocess imc images --hpf 50
```

## Spillover Compensation (NNLS)

**Goal:** Remove channel crosstalk (oxide M+16, abundance-sensitivity M+-1, isotopic impurity) without introducing negative counts.

**Approach:** Estimate the spillover matrix from single-stain controls per positive EVENT then take the median (population-summary estimation overcompensates because IMC's zero background biases ratios), then apply NNLS. Compensate pixels (`compImage`) before segmentation for spatial work, or cell means (`compCytof`) aft