vgp-pipeline

What this skill does

# VGP Assembly Pipeline Skill

## Overview
The Vertebrate Genome Project (VGP) assembly pipeline consists of Galaxy workflows for producing high-quality, phased, chromosome-level genome assemblies. This skill covers workflow selection, execution patterns, and quality control checkpoints.

**Supporting files** (detailed reference material):
- [RESOURCE_ANALYSIS.md](RESOURCE_ANALYSIS.md) - Workflow canonical names, official/non-official filtering, metric availability, tool-level resource optimization
- [DATA_INTEGRATION.md](DATA_INTEGRATION.md) - ToLID patterns, GenomeArk S3 integration, NCBI accession recovery, Meryl k-mer management, species-metrics merging
- [QUALITY_VALIDATION.md](QUALITY_VALIDATION.md) - Curation impact analysis, GenomeScope data validation, assembly size interpretation, communication patterns

## Pre-flight Checks Before Testing

VGP user accounts have a 250 GB quota. If `quota_percent` is at 100%, all new upload jobs land in `paused` state immediately and planemo dies with a cryptic message:

```
Exception: Upload job [...] failed with state [paused]
```

The error gives no hint that quota is the cause. Check before launching any planemo test:

```python
mcp__Galaxy__connect(url=$GXYVGP, api_key=$TESTKEY)
mcp__Galaxy__get_user()  # Inspect quota_percent in response
```

If quota is full, free up space (delete old test histories) and re-check before retrying.

## Trajectories (by frequency of use)

### Trajectory A: HiFi + Hi-C (Most Common)
- **Inputs**: HiFi Reads, Hi-C Reads
- **Path**: WF1 -> WF4 -> [WF6] -> WF8 -> WF9 -> PreCuration
- **Output**: HiC Phased assembly (hap1/hap2)
- **WF6**: Optional (can skip directly to WF8)

### Trajectory B: HiFi + Trio
- **Inputs**: HiFi Reads, Hi-C Reads, Parental Reads
- **Path**: WF2 -> WF5 -> [WF6] -> WF8 -> WF9 -> PreCuration
- **Output**: Trio Phased assembly (maternal/paternal)
- **WF6**: Optional (can skip directly to WF8)

### Trajectory C: HiFi Only (Least Common)
- **Inputs**: HiFi Reads only
- **Path**: WF1 -> WF3 -> WF6 -> WF9 -> PreCuration
- **Output**: Pseudohaplotype assembly (primary/alternate)
- **WF6**: **Required** (no Hi-C scaffolding step)
- **Note**: Skips WF8 entirely

## Workflow Selection by Data Availability

### Non-trio workflows (HiFi reads only)
- **VGP1 (WF1)**: K-mer profiling with HiFi reads alone
- **VGP3 (WF3)**: HiFi-only assembly with HiFiasm

### Trio workflows (HiFi + Parental Illumina)
- **VGP2 (WF2)**: Trio k-mer profiling (HiFi child + Illumina parents)
- **VGP5 (WF5)**: Trio-phased assembly with HiFiasm

### Universal scaffolding workflows
- **RagTag scaffolding**: Used for both trio and non-trio assemblies
- Requires reference genome specification

### Methods language pattern
When documenting workflow selection in publications:
```
"For species with available parental data (trio datasets), we employed
VGP2 -> VGP5 workflows. For species without parental data (non-trio datasets),
we performed VGP1 -> VGP3 workflows."
```

## Workflow Descriptions

| Workflow | Name | Description |
|----------|------|-------------|
| WF0 | Mitochondrial Assembly | MitoHiFi assembly (runs in parallel, may fail if no mito reads) |
| WF1 | K-mer Profiling | Genome size, heterozygosity estimation (HiFi) |
| WF2 | Trio K-mer Profiling | K-mer profiling with parental data |
| WF3 | Hifiasm | HiFi-only assembly |
| WF4 | Hifiasm + HiC | HiC-phased assembly |
| WF5 | Hifiasm Trio | Trio-phased assembly |
| WF6 | Purge Duplicates | Remove haplotypic duplications |
| ~~WF7~~ | ~~Bionano~~ | **Deprecated - no longer used** |
| WF8 | Hi-C Scaffolding | YAHS chromosome scaffolding |
| WF9 | Decontamination | Remove contaminants |
| PreCuration | Pretext Snapshot | Prepare files for manual curation |
| PostCuration | Post Curation | Apply curation, split haplotypes, rename/reorient, QC |

### Post-Curation Processing

Takes curated AGP + combined haplotype FASTA from pre-curation, applies curation changes, splits by haplotype, assigns chromosome names, renames/reorients hap2 to match hap1, generates QC.

**Key inputs:**
- Curated AGP file (PretextView AGP v2.1 — 11 columns: 9 standard + "Painted" + haplotype label)
- Combined haplotype FASTA (both haplotypes in one file)
- Hi-C reads (list:paired), PacBio reads (list)
- Parameters: species name, assembly name, telomere pattern, compleasm DB

**Haplotype labels in AGP:** `Hap_1`, `Hap_2`, `Z`, `W`, `Unloc`, `Haplotig`

**Rename/Reorient logic:** Chromosomes assigned by size independently per haplotype, then mashmap (seqLength=50000) aligns hap2 to hap1 to detect mismatches → renames + inversions.

**Key parameter:** mashmap `seqLength: 50000` — sequences must be ≥50kb for rename/reorient to be exercised.

**Compleasm parameter gotcha:** The workflow input "Database for Compleasm Genes" maps to compleasm's `lineage_dataset` parameter — pass a lineage like `vertebrata_odb10`, NOT the busco database version (`v5`). The `busco_database: v5` is hardcoded in the tool_state.

**Chromosome naming:** `vgp_chromosome_assignment` produces `SUPER_N` names (SUPER_1, SUPER_2, SUPER_Z, SUPER_W), not `chr1`/`chr2`.

**Known issue:** `vgp_chromosome_assignment` v1.1 crashes when AGP contains multi-component scaffolds with `Unloc` pieces (e.g., Scaffold_1 = main + gap + unloc). Workaround: make Unloc scaffolds standalone single-component entries in the AGP.

**Tag inheritance pitfall:** The curation subworkflow sets `#hap1` (via `name:hap1` in .ga) on the Hap1 AGP at the "convert to agp" step. This propagates through chromosome assignment → mashmap → sak generation → reoriented hap2, causing Curated Hap2 to inherit `#hap1`. Fix: add `RemoveTagDatasetAction` for `name:hap1` at key junction points (sak_file output, Curated Hap2 output).

## IWC Workflow Versions (as of March 2026)

| Workflow | IWC Repo | Latest Version | Dockstore ID |
|----------|----------|----------------|--------------|
| WF1 | kmer-profiling-hifi-VGP1 | v0.6 | github.com/iwc-workflows/kmer-profiling-hifi-VGP1/main |
| WF4 | Assembly-Hifi-HiC-phasing-VGP4 | v0.5 | github.com/iwc-workflows/Assembly-Hifi-HiC-phasing-VGP4/main |
| WF8 | Scaffolding-HiC-VGP8 | v3.3 | github.com/iwc-workflows/Scaffolding-HiC-VGP8/main |

### Recent Breaking Changes

**BUSCO -> Compleasm (WF4 v0.5, WF8 v3.3)**:
- Compleasm (`0.2.5+galaxy0`) replaced BUSCO for gene completeness assessment
- Uses miniprot for protein-to-genome alignment (faster than BUSCO's BLAST approach)
- Same output categories: Complete (Single-copy + Duplicated), Fragmented, Missing
- Input parameters still named "Database for Busco Lineage" and "Lineage" (backward compat)

**Hi-C reads format change (WF4 v0.5, WF8 v3.3)**:
- Changed from separate forward/reverse datasets to **list:paired collection**
- Users must build a list:paired collection before running these workflows

**New required inputs across all workflows**:
- Species Name (text) -- used for workflow reports
- Assembly Name (text) -- used for workflow reports

**WF4 additional new inputs**: Trim Hi-C reads? (boolean), Name for Haplotype 1/2 (defaults: Hap1/Hap2), Bits for bloom filter (default: 37)
**WF8 additional new inputs**: Haplotype (restricted: Haplotype 1/2, Maternal/Paternal, Primary/Alternate), Trim Hi-C Data? (boolean), Minimum Mapping Quality (default: 10)

### Verifying IWC Versions

Check latest versions via Dockstore API:
```
https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fiwc-workflows%2F{REPO}%2Fmain/versions
```

Check workflow inputs by fetching the .ga file from GitHub:
```
https://raw.githubusercontent.com/iwc-workflows/{REPO}/main/{WORKFLOW_NAME}.ga
```

## Haplotype Execution Patterns

### Run Once (Both Haplotypes Together)
- WF1, WF2 (K-mer profiling)
- WF3, WF4, WF5 (Assembly)
- WF6 (Purge Duplicates) - *depends on trajectory*
- PreCuration

### Run Twice (x2 per Haplotype)
- WF8 (Hi-C Scaffolding)
- WF9 (Decontamination)

## WF6 (Purge Duplicates) Decision Logic

```
if trajectory == "C" (HiFi only):
    WF6 is