bio-biomart-queries

What this skill does

## Version Compatibility

Reference examples tested with: pybiomart 0.9+, R biomaRt 2.58+ (Bioconductor); Ensembl BioMart (release 110+)

Before using code patterns, verify installed versions match. If versions differ:
- Python: `pip show pybiomart`
- R: `packageVersion('biomaRt')`

The BioMart XML query format is stable across Ensembl releases; the underlying mart names and attribute IDs can change between Ensembl releases. For published work, pin the Ensembl release via `useEnsembl(version=110)`.

# BioMart Queries

**"Bulk-convert IDs / pull coordinate tables / extract ortholog wide tables"** -> BioMart is the right answer for any Ensembl-rooted query producing >5,000 rows. It is a separate service from the Ensembl REST API, with separate rate behavior and a different query model (XML-based, batch-oriented). For one-off lookups (<100 records), Ensembl REST is more convenient; for bulk anything, BioMart wins.

The single most important fact: **BioMart returns a flat table from a single query**. There is no per-record loop, no rate-limit cascade, no async polling. One XML query in; one TSV out.

- Python: `pybiomart` (https://github.com/jrderuiter/pybiomart) is the lightest client
- R: `biomaRt` Bioconductor (Durinck et al. 2009 *Nat Protoc* 4:1184) is the canonical client
- CLI: `curl` against the XML endpoint works but is rarely used directly
- Web: `https://www.ensembl.org/biomart/martview` for interactive query design

## Installation

```bash
pip install pybiomart pandas
# R:
# BiocManager::install('biomaRt')
```

## BioMart hierarchy

| Level | Examples |
|---|---|
| Mart | `ENSEMBL_MART_ENSEMBL` (genes), `ENSEMBL_MART_SNP` (variants), `ENSEMBL_MART_MOUSE` (mouse-specific) |
| Dataset | `hsapiens_gene_ensembl`, `mmusculus_gene_ensembl`, etc. (per species) |
| Attribute | Fields to return: `ensembl_gene_id`, `external_gene_name`, `chromosome_name`, etc. |
| Filter | Constraints on the query: `chromosome_name = 17`, `biotype = protein_coding`, etc. |

A query is: pick a mart, pick a dataset, list attributes to return, list filters to constrain. BioMart returns a single TSV.

Discovery:
```python
from pybiomart import Server
server = Server(host='http://www.ensembl.org')
print(server.marts)                                          # list marts
mart = server['ENSEMBL_MART_ENSEMBL']
print(mart.datasets)                                         # list datasets (species)
ds = mart['hsapiens_gene_ensembl']
print(ds.attributes)                                         # list attributes
print(ds.filters)                                            # list filters
```

## Decision matrix: BioMart vs Ensembl REST

| Question | BioMart | Ensembl REST |
|---|---|---|
| Bulk ID mapping (>5000 IDs) | yes (1 query) | rate-limited cascade |
| Single-gene lookup | overkill | yes |
| Coordinate tables for thousands of genes | yes | rate-limited |
| Ortholog wide-table across species | yes (multi-species mart) | per-gene loop |
| VEP variant annotation | no | yes (or local VEP) |
| Sequence retrieval | partial | yes |
| Real-time | no (batch) | yes (per-record) |
| Reproducibility (version pin) | `useEnsembl(version=110)` | archive URL `e110.rest.ensembl.org` |

For >5K rows, BioMart is the right tool. For real-time per-record lookups, REST.

## Common attribute selectors

| Attribute | Returns |
|---|---|
| `ensembl_gene_id` | Stable Ensembl Gene ID |
| `ensembl_gene_id_version` | With `.N` version suffix |
| `external_gene_name` | HGNC symbol (or species-equivalent) |
| `hgnc_id`, `hgnc_symbol` | HGNC permanent ID and symbol |
| `entrezgene_id` | NCBI Gene ID |
| `refseq_mrna`, `refseq_peptide` | RefSeq accessions |
| `uniprotswissprot`, `uniprotsptrembl` | UniProt accessions |
| `chromosome_name`, `start_position`, `end_position`, `strand` | Gene coordinates |
| `transcript_count`, `exon_count` | Counts |
| `biotype` | protein_coding, lncRNA, miRNA, etc. |
| `description` | Free-text gene description |
| `go_id`, `name_1006`, `namespace_1003` | GO term ID, name, namespace |

## Common filter selectors

| Filter | Constraint |
|---|---|
| `ensembl_gene_id` | List of Gene IDs |
| `external_gene_name` | List of symbols |
| `entrezgene_id` | List of NCBI Gene IDs |
| `chromosome_name` | One or more chromosomes |
| `start` / `end` | Coordinate range |
| `biotype` | One or more biotypes |
| `with_<source>` | Boolean: has cross-ref to `<source>` (e.g. `with_hpa` = has Human Protein Atlas) |

## Code patterns

### Bulk ID mapping: Ensembl Gene -> HGNC + RefSeq + UniProt

**Goal:** Convert 5,000 Ensembl Gene IDs to HGNC symbols, RefSeq mRNA accessions, and UniProt accessions in one query.

**Approach:** pybiomart query with three attributes; ID list as a filter; returns one TSV.

**Reference (pybiomart 0.9+, Ensembl release 110+):**
```python
from pybiomart import Server
import pandas as pd

server = Server(host='http://www.ensembl.org')
mart = server['ENSEMBL_MART_ENSEMBL']
ds = mart['hsapiens_gene_ensembl']

ensembl_ids = ['ENSG00000139618', 'ENSG00000141510', 'ENSG00000171862']  # ...up to 5K+

df = ds.query(
    attributes=['ensembl_gene_id', 'external_gene_name', 'hgnc_id',
                'refseq_mrna', 'uniprotswissprot'],
    filters={'ensembl_gene_id': ensembl_ids},
)
print(df.head())
# One row per (gene, cross-ref) pair; genes with multiple RefSeq mRNAs get multiple rows.
```

### Pull gene coordinate table for a chromosome

```python
df = ds.query(
    attributes=['ensembl_gene_id', 'external_gene_name', 'chromosome_name',
                'start_position', 'end_position', 'strand', 'biotype'],
    filters={'chromosome_name': '17', 'biotype': 'protein_coding'},
)
print(f'{len(df)} protein-coding genes on chr17')
```

### Bulk ortholog wide-table (human <-> mouse <-> zebrafish)

**Goal:** One TSV with human Ensembl ID, mouse ortholog Ensembl ID, zebrafish ortholog Ensembl ID per row.

**Approach:** Ortholog attributes from the human mart query both species' orthologs.

```python
df = ds.query(
    attributes=['ensembl_gene_id', 'external_gene_name',
                'mmusculus_homolog_ensembl_gene', 'mmusculus_homolog_orthology_type',
                'drerio_homolog_ensembl_gene', 'drerio_homolog_orthology_type'],
    filters={'chromosome_name': '17'},
)
# pybiomart columns use the mart display names, which can vary across releases.
# Resolve column names defensively rather than hardcoding strings:
mouse_type_col = next(c for c in df.columns if 'Mouse' in c and 'type' in c)
zebra_type_col = next(c for c in df.columns if 'Zebrafish' in c and 'type' in c)
df_one2one = df[(df[mouse_type_col] == 'ortholog_one2one') &
                (df[zebra_type_col] == 'ortholog_one2one')]
print(f'{len(df_one2one)} 1:1 orthologs across all three species on chr17')
```

### GO term annotation for a gene set

```python
df = ds.query(
    attributes=['ensembl_gene_id', 'external_gene_name',
                'go_id', 'name_1006', 'namespace_1003'],
    filters={'external_gene_name': ['TP53', 'BRCA1', 'MYC', 'EGFR']},
)
# Long format: one row per (gene, GO term) pair
```

### Version-pinned query (R biomaRt)

```r
# Reference: Bioconductor biomaRt 2.58+ | Verify API if version differs
library(biomaRt)

# Pin to release 110 for reproducibility
ensembl <- useEnsembl(biomart='genes', dataset='hsapiens_gene_ensembl', version=110)

# Or via host URL (for older or specific assemblies)
# ensembl <- useMart('ENSEMBL_MART_ENSEMBL',
#                     dataset='hsapiens_gene_ensembl',
#                     host='https://nov2020.archive.ensembl.org')

df <- getBM(
    attributes = c('ensembl_gene_id', 'external_gene_name', 'entrezgene_id',
                   'uniprotswissprot', 'refseq_mrna'),
    filters = 'ensembl_gene_id',
    values = c('ENSG00000139618', 'ENSG00000141510'),
    mart = ensembl
)
head(df)
```

### Discover attributes / filters programmatically

```python
# What attributes are available?
attrs = ds.attributes
ortho_attrs = [a for a in attrs if 'homolog' in a]
print(f'{len(orth