anndata

What this skill does

# AnnData

## Overview

AnnData (Annotated Data) is Python's standard for storing and manipulating annotated data matrices, particularly in single-cell genomics. Work with AnnData objects for data creation, manipulation, file I/O, concatenation, and memory-efficient workflows.

## Core Capabilities

### 1. Creating and Structuring AnnData Objects

Create AnnData objects from various data sources and organize multi-dimensional annotations.

**Basic creation:**
```python
import anndata as ad
import numpy as np
from scipy.sparse import csr_matrix

# From dense or sparse arrays
counts = np.random.poisson(1, size=(100, 2000))
adata = ad.AnnData(counts)

# With sparse matrix (memory-efficient)
counts = csr_matrix(np.random.poisson(1, size=(100, 2000)), dtype=np.float32)
adata = ad.AnnData(counts)
```

**With metadata:**
```python
import pandas as pd

obs_meta = pd.DataFrame({
    'cell_type': pd.Categorical(['B', 'T', 'Monocyte'] * 33 + ['B']),
    'batch': ['batch1'] * 50 + ['batch2'] * 50
})
var_meta = pd.DataFrame({
    'gene_name': [f'Gene_{i}' for i in range(2000)],
    'highly_variable': np.random.choice([True, False], 2000)
})

adata = ad.AnnData(counts, obs=obs_meta, var=var_meta)
```

**Understanding the structure:**
- **X**: Primary data matrix (observations × variables)
- **obs**: Row (observation) annotations as DataFrame
- **var**: Column (variable) annotations as DataFrame
- **obsm**: Multi-dimensional observation annotations (e.g., PCA, UMAP coordinates)
- **varm**: Multi-dimensional variable annotations (e.g., gene loadings)
- **layers**: Alternative data matrices with same dimensions as X
- **uns**: Unstructured metadata dictionary
- **obsp/varp**: Pairwise relationship matrices (graphs)

### 2. Adding Annotations and Layers

Organize different data representations and metadata within a single object.

**Cell-level metadata (obs):**
```python
adata.obs['n_genes'] = (adata.X > 0).sum(axis=1)
adata.obs['total_counts'] = adata.X.sum(axis=1)
adata.obs['condition'] = pd.Categorical(['control', 'treated'] * 50)
```

**Gene-level metadata (var):**
```python
adata.var['highly_variable'] = gene_variance > threshold
adata.var['chromosome'] = pd.Categorical(['chr1', 'chr2', ...])
```

**Embeddings (obsm/varm):**
```python
# Dimensionality reduction results
adata.obsm['X_pca'] = pca_coordinates  # Shape: (n_obs, n_components)
adata.obsm['X_umap'] = umap_coordinates  # Shape: (n_obs, 2)
adata.obsm['X_tsne'] = tsne_coordinates

# Gene loadings
adata.varm['PCs'] = principal_components  # Shape: (n_vars, n_components)
```

**Alternative data representations (layers):**
```python
# Store multiple versions
adata.layers['counts'] = raw_counts
adata.layers['log1p'] = np.log1p(adata.X)
adata.layers['scaled'] = (adata.X - mean) / std
```

**Unstructured metadata (uns):**
```python
# Analysis parameters
adata.uns['preprocessing'] = {
    'normalization': 'TPM',
    'min_genes': 200,
    'date': '2024-01-15'
}

# Results
adata.uns['pca'] = {'variance_ratio': variance_explained}
```

### 3. Subsetting and Views

Efficiently subset data while managing memory through views and copies.

**Subsetting operations:**
```python
# By observation/variable names
subset = adata[['Cell_1', 'Cell_10'], ['Gene_5', 'Gene_1900']]

# By boolean masks
b_cells = adata[adata.obs.cell_type == 'B']
high_quality = adata[adata.obs.n_genes > 200]

# By position
first_cells = adata[:100, :]
top_genes = adata[:, :500]

# Combined conditions
filtered = adata[
    (adata.obs.batch == 'batch1') & (adata.obs.n_genes > 200),
    adata.var.highly_variable
]
```

**Understanding views:**
- Subsetting returns **views** by default (memory-efficient, shares data with original)
- Modifying a view affects the original object
- Check with `adata.is_view`
- Convert to independent copy with `.copy()`

```python
# View (memory-efficient)
subset = adata[adata.obs.condition == 'treated']
print(subset.is_view)  # True

# Independent copy
subset_copy = adata[adata.obs.condition == 'treated'].copy()
print(subset_copy.is_view)  # False
```

### 4. File I/O and Backed Mode

Read and write data efficiently, with options for memory-limited environments.

**Writing data:**
```python
# Standard format with compression
adata.write('results.h5ad', compression='gzip')

# Alternative formats
adata.write_zarr('results.zarr')  # For cloud storage
adata.write_loom('results.loom')  # For compatibility
adata.write_csvs('results/')      # As CSV files
```

**Reading data:**
```python
# Load into memory
adata = ad.read_h5ad('results.h5ad')

# Backed mode (disk-backed, memory-efficient)
adata = ad.read_h5ad('large_file.h5ad', backed='r')
print(adata.isbacked)  # True
print(adata.filename)  # Path to file

# Close file connection when done
adata.file.close()
```

**Reading from other formats:**
```python
# 10X format
adata = ad.read_mtx('matrix.mtx')

# CSV
adata = ad.read_csv('data.csv')

# Loom
adata = ad.read_loom('data.loom')
```

**Working with backed mode:**
```python
# Read in backed mode for large files
adata = ad.read_h5ad('large_dataset.h5ad', backed='r')

# Process in chunks
for chunk in adata.chunk_X(chunk_size=1000):
    result = process_chunk(chunk)

# Load to memory if needed
adata_memory = adata.to_memory()
```

### 5. Concatenating Multiple Datasets

Combine multiple AnnData objects with control over how data is merged.

**Basic concatenation:**
```python
# Concatenate observations (most common)
combined = ad.concat([adata1, adata2, adata3], axis=0)

# Concatenate variables (rare)
combined = ad.concat([adata1, adata2], axis=1)
```

**Join strategies:**
```python
# Inner join: only shared variables (no missing data)
combined = ad.concat([adata1, adata2], join='inner')

# Outer join: all variables (fills missing with 0)
combined = ad.concat([adata1, adata2], join='outer')
```

**Tracking data sources:**
```python
# Add source labels
combined = ad.concat(
    [adata1, adata2, adata3],
    label='dataset',
    keys=['exp1', 'exp2', 'exp3']
)
# Creates combined.obs['dataset'] with values 'exp1', 'exp2', 'exp3'

# Make duplicate indices unique
combined = ad.concat(
    [adata1, adata2],
    keys=['batch1', 'batch2'],
    index_unique='-'
)
# Cell names become: Cell_0-batch1, Cell_0-batch2, etc.
```

**Merge strategies for metadata:**
```python
# merge=None: exclude variable annotations (default)
combined = ad.concat([adata1, adata2], merge=None)

# merge='same': keep only identical annotations
combined = ad.concat([adata1, adata2], merge='same')

# merge='first': use first occurrence
combined = ad.concat([adata1, adata2], merge='first')

# merge='unique': keep annotations with single value
combined = ad.concat([adata1, adata2], merge='unique')
```

**Complete example:**
```python
# Load batches
batch1 = ad.read_h5ad('batch1.h5ad')
batch2 = ad.read_h5ad('batch2.h5ad')
batch3 = ad.read_h5ad('batch3.h5ad')

# Concatenate with full tracking
combined = ad.concat(
    [batch1, batch2, batch3],
    axis=0,
    join='outer',              # Keep all genes
    merge='first',             # Use first batch's annotations
    label='batch_id',          # Track source
    keys=['b1', 'b2', 'b3'],  # Custom labels
    index_unique='-'           # Make cell names unique
)
```

### 6. Data Conversion and Extraction

Convert between AnnData and other formats for interoperability.

**To DataFrame:**
```python
# Convert X to DataFrame
df = adata.to_df()

# Convert specific layer
df = adata.to_df(layer='log1p')
```

**Extract vectors:**
```python
# Get 1D arrays from data or annotations
gene_expression = adata.obs_vector('Gene_100')
cell_metadata = adata.obs_vector('n_genes')
```

**Transpose:**
```python
# Swap observations and variables
transposed = adata.T
```

### 7. Memory Optimization

Strategies for working with large datasets efficiently.

**Use sparse matrices:**
```python
from scipy.sparse import csr_matrix

# Check sparsity
density = (adata.X != 0).sum() / adata.X.size
if density < 0.3:  # Less than 30% non-zero
    adata.X = c