bio-genome-annotation-annotation-transfer

What this skill does

## Version Compatibility

Reference examples tested with: Liftoff 1.6.3+, LiftoffTools 0.4+, miniprot 0.13+, CrossMap 0.7+, UCSC liftOver (current), BioPython 1.83+, gffutils 0.12+.

Before using code patterns, verify installed versions match. If versions differ:
- CLI: `<tool> --version` then `<tool> --help` to confirm flags
- Python: `pip show <package>` then `help(module.function)` to check signatures

The **chain file must match the exact assembly pair** (build and patch); the **source and target build must be recorded** with every coordinate (a coordinate without a build is unusable). If code throws an error, introspect the installed tool and adapt rather than retrying.

# Annotation Transfer

**"Transfer annotations from a reference to my new assembly"** -> Map gene models from a well-annotated reference onto a target, by coordinate liftover (same-species, fast) or by re-aligning the actual gene sequence (cross-assembly/species, structure-aware), then validate against the target.
- CLI: `liftoff -g ref.gff3 -o out.gff3 -u unmapped.txt target.fa reference.fa` (note: target before reference), `liftOver in.bed map.chain out.bed unmapped` (intervals)

## The Single Most Important Modern Insight -- A Lift Is Geometry, Not Biology

Coordinate liftover and feature projection answer different questions, and choosing the wrong one is the dominant failure mode:

- **Coordinate liftover** (liftOver/CrossMap, on pre-computed chains) answers *"where does this interval sit in the other assembly's coordinate system?"* It moves numbers; it never re-examines the sequence.
- **Feature projection** (Liftoff/miniprot/TOGA) answers *"where is this gene, with its exon-intron structure intact, and is it still a functional gene?"* It re-aligns the biological sequence.

Three load-bearing consequences:

1. **A successful lift is not biological confirmation.** liftOver can place a gene at perfectly valid target coordinates that land in a pseudogenized, frameshifted, or collapsed-duplication region - the coordinate is right and the gene is dead, and the tool has no vocabulary to flag it. The output GFF inherits the reference's `gene`/`CDS` feature types verbatim. The "it mapped, ship it" culture is how a lifted GFF acquires the social status of a validated annotation while no one ever re-derived a model from sequence. Treat every lifted annotation as a hypothesis until target evidence (intact ORF, identity distribution, BUSCO, RNA-seq) has touched it.
2. **Transfer is reference-biased: it can only reproduce what the reference annotated.** Lineage-specific genes, target-specific expansions, novel isoforms, and orphan genes are *structurally invisible* - the new assembly's actual novelty (the reason it is interesting) is exactly what transfer cannot see. Always pair transfer with de novo + evidence; TOGA can call gene *loss* but is constitutionally incapable of calling gene *gain*.
3. **Read and classify the unmapped file - it is the most information-rich output.** liftOver writes failures to a side file, exits 0, and prints a clean shorter GFF with no visual tell; entire gene families can vanish silently. "Deleted in new" vs "Split in new" vs "Duplicated in new" are biologically distinct diagnoses (chain gap / rearrangement breakpoint / segmental duplication or gene family).

## Two Paradigms

| Paradigm | Tools | Operates on | Right for |
|----------|-------|-------------|-----------|
| A. Coordinate liftover | UCSC liftOver, CrossMap, segment_liftover, paftools | pre-computed chains; intervals (BED/GFF/VCF/BAM) | same-species version updates (hg19<->hg38, mm10<->mm39); variant/peak/CNV harmonization |
| B. Feature/sequence projection | Liftoff (nt), miniprot (protein), GeMoMa, TOGA, CAT, LiftOn | re-aligned gene sequence | cross-assembly/species; full gene models; polyploid/duplicated; no reliable chain |

## Decision Tree by Divergence

| Divergence | Recommended | Why |
|------------|-------------|-----|
| Same species, transfer intervals | liftOver / CrossMap | chain is dense; geometry suffices for variants/peaks |
| Same species, transfer gene models | Liftoff (`-chroms`) | structure-aware; per-interval liftOver fragments transcripts |
| Same genus (a few % divergence) | Liftoff + miniprot rescue for the divergent tail | nucleotide alignment robust; protein for the rest |
| Same family/order (tens-hundreds My) | TOGA or GeMoMa (multi-reference) | nucleotide saturates; orthology + gene-loss reasoning |
| Beyond family / lineage-specific content / heavy rearrangement | -> eukaryotic-gene-prediction (de novo) + transfer as evidence | reference too far; only de novo sees target-specific biology |
| Pan-genome / multi-haplotype | `vg annotate` onto the graph | avoids single-reference bias (tooling still maturing) |

Cross-species *coordinate* liftover is a methodological error (synteny fragments into thousands of short chains; most genes drop silently) - it is the wrong paradigm, not a tuning problem.

## Liftoff (Same / Close Species, Nucleotide)

```bash
liftoff -g reference.gff3 -o lifted.gff3 -u unmapped.txt -p 16 \
    -chroms chrom_map.txt -polish target.fasta reference.fasta
```

Positional args are **target first, then reference** (commonly swapped - a silent error). Liftoff extracts each gene's exon sequence, aligns with minimap2, and chooses the placement maximizing identity while preserving exon-intron structure. Key flags: `-a` (alignment coverage, default 0.5), `-s` (sequence identity, default 0.5), `-copies`/`-sc` (search for extra gene copies - a per-family decision, not a default), `-polish` (re-align to restore intact start/stop/splice, writes `*_polished.gff3`), `-exclude_partial`, `-chroms` (ordered chromosome mapping; reduces false cross-chromosome placements). LiftoffTools QCs the result (variants, synteny, copy-number changes). Same-species version updates should lift ≥99% - a 97% rate is a four-alarm signal of the wrong chain or coordinate-convention mismatch, not "pretty good."

## miniprot (Cross-Species, Protein)

```bash
miniprot -t 16 -d target.mpi target.fasta            # optional index
miniprot -Iut 16 --gff target.mpi proteins.faa > out.gff
```

Protein conserves far deeper than nucleotide (synonymous sites saturate), so miniprot works across species where Liftoff's nucleotide alignment fails. `-I` auto-sets max intron from genome length; `--gff` emits GFF3. **Frameshift and in-frame-stop tags in the output are the signal that the "gene" is pseudogenized in the target, not a clean ortholog** - inspect them; do not treat a miniprot hit as a functional gene by default. For a polished multi-reference, intron-aware annotation use GeMoMa (which reasons about intron-position conservation); for the DNA+protein hybrid use LiftOn.

## TOGA (Distant Species, Orthology + Gene Loss)

TOGA consumes a genome-alignment chain + reference BED12 and uses ML on chain features (including intronic/intergenic flanks - orthologs share flanking context, paralogs/retrocopies do not) to classify orthology (one2one ... one2zero) and **gene-loss/intactness** (intact / partially intact / lost / missing). It exists *precisely because* across deep time an inactivated gene still aligns - a coordinate lift reports the corpse as "present." Use TOGA for whole-clade ortholog projection; it does not discover target-specific novel genes (the reference-bias caveat of all of Paradigm B).

## Validating a Transfer with Python

**Goal:** Quantify transfer quality and, critically, check that lifted CDS are biologically intact, not just placed.

**Approach:** Compare gene counts for a transfer rate, then translate each lifted CDS from the target and check for a valid start, a single terminal stop, and correct length - coordinate success is not intactness.

```python
import gffutils
from Bio import SeqIO

def orf_integrity(lifted_gff, target_fasta):
    genome = SeqIO.to_dict(SeqIO.parse(target_fasta, 'fasta'))
    db = gffutils.create_db(lifted_gff, ':memory:', merge_strategy='me