busco-phylogeny

What this skill does

# BUSCO-based Phylogenomics Workflow Generator

This skill provides phylogenomics expertise for generating comprehensive, scheduler-aware workflows for phylogenetic inference from genome assemblies using single-copy orthologs.

## Purpose

This skill helps users generate phylogenies from genome assemblies by:
1. Handling mixed input (local files and NCBI accessions)
2. Creating scheduler-specific scripts (SLURM, PBS, cloud, local)
3. Setting up complete workflows from raw genomes to final trees
4. Providing quality control and recommendations
5. Supporting flexible software management (bioconda, Docker, custom)

## Available Resources

The skill provides access to these bundled resources:

### Scripts (`scripts/`)
- **`query_ncbi_assemblies.py`** - Query NCBI for available genome assemblies by taxon name (new!)
- **`download_ncbi_genomes.py`** - Download genomes from NCBI using BioProjects or Assembly accessions
- **`rename_genomes.py`** - Rename genome files with meaningful sample names (important!)
- **`generate_qc_report.sh`** - Generate quality control reports from compleasm results
- **`extract_orthologs.sh`** - Extract and reorganize single-copy orthologs
- **`run_aliscore.sh`** - Wrapper for Aliscore to identify randomly similar sequences (RSS)
- **`run_alicut.sh`** - Wrapper for ALICUT to remove RSS positions from alignments
- **`run_aliscore_alicut_batch.sh`** - Batch process all alignments through Aliscore + ALICUT
- **`convert_fasconcat_to_partition.py`** - Convert FASconCAT output to IQ-TREE partition format
- **`predownloaded_aliscore_alicut/`** - Pre-tested Aliscore and ALICUT Perl scripts

### Templates (`templates/`)
- **`slurm/`** - SLURM job scheduler templates
- **`pbs/`** - PBS/Torque job scheduler templates
- **`local/`** - Local machine templates (with GNU parallel)
- **`README.md`** - Complete template documentation

### References (`references/`)
- **`REFERENCE.md`** - Detailed technical reference including:
  - Sample naming best practices
  - BUSCO lineage datasets (complete list)
  - Resource recommendations (memory, CPUs, walltime)
  - Detailed step-by-step implementation guides
  - Quality control guidelines
  - Aliscore/ALICUT detailed guide
  - Tool citations and download links
  - Software installation guide
  - Common issues and troubleshooting

## Workflow Overview

The complete phylogenomics pipeline follows this sequence:

**Input Preparation** → **Ortholog Identification** → **Quality Control** → **Ortholog Extraction** → **Alignment** → **Trimming** → **Concatenation** → **Phylogenetic Inference**

## Initial User Questions

When a user requests phylogeny generation, gather the following information systematically:

### Step 1: Detect Computing Environment

Before asking questions, attempt to detect the local computing environment:

```bash
# Check for job schedulers
command -v sbatch >/dev/null 2>&1  # SLURM
command -v qsub >/dev/null 2>&1    # PBS/Torque
command -v parallel >/dev/null 2>&1  # GNU parallel
```

Report findings to the user, then confirm: **"I detected [X] on this machine. Will you be running the scripts here or on a different system?"**

### Required Information

Ask these questions to gather essential workflow parameters:

1. **Computing Environment**
   - Where will these scripts run? (SLURM cluster, PBS/Torque cluster, Cloud computing, Local machine)

2. **Input Data**
   - Local genome files, NCBI accessions, or both?
   - If NCBI: Do you already have Assembly accessions (GCA_*/GCF_*) or BioProject accessions (PRJNA*/PRJEB*/PRJDA*)?
   - If user doesn't have accessions: Offer to help find assemblies using `query_ncbi_assemblies.py` (see "STEP 0A: Query NCBI for Assemblies" below)
   - If local files: What are the file paths?

3. **Taxonomic Scope & Dataset Details**
   - What taxonomic group? (determines BUSCO lineage dataset)
   - How many taxa/genomes will be analyzed?
   - What is the approximate phylogenetic breadth? (species-level, genus-level, family-level, order-level, etc.)
   - See `references/REFERENCE.md` for complete lineage list

4. **Environment Management**
   - Use unified conda environment (default, recommended), or separate environments per tool?

5. **Resource Constraints**
   - How many CPU cores/threads to use in total? (Ask user to specify, do not auto-detect)
   - Available memory (RAM) per node/machine?
   - Maximum walltime for jobs?
   - See `references/REFERENCE.md` for resource recommendations

6. **Parallelization Strategy**

   Ask the user how they want to handle parallel processing:

   - **For job schedulers (SLURM/PBS)**:
     - Use array jobs for parallel steps? (Recommended: Yes)
     - Which steps to parallelize? (Steps 2, 5, 6, 8C recommended)

   - **For local machines**:
     - Use GNU parallel for parallel steps? (requires `parallel` installed)
     - How many concurrent jobs?

   - **For all systems**:
     - Optimize for maximum throughput or simplicity?

7. **Scheduler-Specific Configuration** (if using SLURM or PBS)
   - Account/Username for compute time charges
   - Partition/Queue to submit jobs to
   - Email notifications? (address and when: START, END, FAIL, ALL)
   - Job dependencies? (Recommended: Yes for linear workflow)
   - Output log directory? (Default: `logs/`)

8. **Alignment Trimming Preference**
   - Aliscore/ALICUT (traditional, thorough), trimAl (fast), BMGE (entropy-based), or ClipKit (modern)?

9. **Substitution Model Selection** (for IQ-TREE phylogenetic inference)

   **Context needed**: Taxonomic breadth, number of taxa, evolutionary rates

   **Action**: Fetch IQ-TREE model documentation and suggest appropriate amino acid substitution models based on dataset characteristics.

   Use the substitution model recommendation system (see "Substitution Model Recommendation" section below).

10. **Educational Goals**
   - Are you learning bioinformatics and would you like comprehensive explanations of each workflow step?
   - If yes: After completing each major workflow stage, offer to explain what the step accomplishes, why certain choices were made, and what best practices are being followed.
   - Store this preference to use throughout the workflow.

---

## Recommended Directory Structure

Organize analyses with dedicated folders for each pipeline step:

```
project_name/
├── logs/                          # All log files
├── 00_genomes/                    # Input genome assemblies
├── 01_busco_results/              # BUSCO/compleasm outputs
├── 02_qc/                         # Quality control reports
├── 03_extracted_orthologs/        # Extracted single-copy orthologs
├── 04_alignments/                 # Multiple sequence alignments
├── 05_trimmed/                    # Trimmed alignments
├── 06_concatenation/              # Supermatrix and partition files
├── 07_partition_search/           # Partition model selection
├── 08_concatenated_tree/          # Concatenated ML tree
├── 09_gene_trees/                 # Individual gene trees
├── 10_species_tree/               # ASTRAL species tree
└── scripts/                       # All analysis scripts
```

**Benefits**: Easy debugging, clear workflow progression, reproducibility, prevents root directory clutter.

---

## Template System

This skill uses a template-based system to reduce token usage and improve maintainability. Script templates are stored in the `templates/` directory and organized by computing environment.

### How to Use Templates

When generating scripts for users:

1. **Read the appropriate template** for their computing environment:
   ```
   Read("templates/slurm/02_compleasm_first.job")
   ```

2. **Replace placeholders** with user-specific values:
   - `TOTAL_THREADS` → e.g., `64`
   - `THREADS_PER_JOB` → e.g., `16`
   - `NUM_GENOMES` → e.g., `20`
   - `NUM_LOCI` → e.g., `2795`
   - `LINEAGE` → e.g., `insecta_odb10`
   - `MODEL_SET` → e.g., `LG,WAG,JTT,Q.pfam`

3. **Present the customized script** to the user with setup instructions

### Available Templates