devtu-optimize-skills

What this skill does

# Optimizing ToolUniverse Skills

Best practices for high-quality research skills with evidence grading and source attribution.

## Tool Quality Standards

1. **Error messages must be actionable** — tell the user what went wrong AND what to do
2. **Schema must match API reality** — run `python3 -m tooluniverse.cli run <Tool> '<json>'` to verify
3. **Coverage transparency** — state what data is NOT included
4. **Input validation before API calls** — don't silently send invalid values
5. **Cross-tool routing** — name the correct tool when query is out-of-scope
6. **No silent parameter dropping** — if a parameter is ignored, say so

## Core Principles (13 Patterns)

Full details: [references/optimization-patterns.md](references/optimization-patterns.md)

| # | Pattern | Key Idea |
|---|---------|----------|
| 1 | Tool Interface Verification | `get_tool_info()` before first call; maintain corrections table |
| 2 | Foundation Data Layer | Query aggregator (Open Targets, PubChem) FIRST |
| 3 | Versioned Identifiers | Capture both `ENSG00000123456` and `.12` version |
| 4 | Disambiguation First | Resolve IDs, detect collisions, build negative filters |
| 5 | Report-Only Output | Narrative in report; methodology in appendix only if asked |
| 6 | Evidence Grading | T1 (mechanistic) → T2 (functional) → T3 (association) → T4 (mention) |
| 7 | Quantified Completeness | Numeric minimums per section (>=20 PPIs, top 10 tissues) |
| 8 | Mandatory Checklist | All sections exist, even if "Limited evidence" |
| 9 | Aggregated Data Gaps | Single section consolidating all missing data |
| 10 | Query Strategy | High-precision seeds → citation expansion → collision-filtered broad |
| 11 | Tool Failure Handling | Primary → Fallback 1 → Fallback 2 → document unavailable |
| 12 | Scalable Output | Narrative report + JSON/CSV bibliography |
| 13 | Synthesis Sections | Biological model + testable hypotheses, not just paper lists |

## Optimized Skill Workflow

```
Phase -1: Tool Verification (check params)
Phase  0: Foundation Data (aggregator query)
Phase  1: Disambiguation (IDs, collisions, baseline)
Phase  2: Specialized Queries (fill gaps)
Phase  3: Report Synthesis (evidence-graded narrative)
```

## Testing Standards

Full details: [references/testing-standards.md](references/testing-standards.md)

**Critical rule**: NEVER write skill docs without testing all tool calls first.

- 30+ tests per skill, 100% pass rate
- All tests use real data (no placeholders)
- Phase + integration + edge case tests
- SOAP tools (IMGT, SAbDab, TheraSAbDab) need `operation` parameter
- Distinguish transient errors (retry) from real bugs (fix)
- API docs are often wrong — always verify with actual calls

## Pattern 14: Reasoning Frameworks Over Tool Catalogs (CRITICAL)

Skills that just list tools ("call A, then B, then C") score 3-5/10 in usefulness tests. Skills that explain HOW to interpret and combine data score 7-9/10. Every skill MUST include:

### 14a. Interpretation Tables
Map raw API data to biological/clinical meaning. Don't just retrieve — explain.

| Bad (tool catalog) | Good (reasoning framework) |
|---|---|
| "Get GO terms from MGnify" | GO terms → interpretation table: butyrate genes = barrier integrity, LPS genes = inflammation |
| "Get DepMap dependency scores" | Score < -0.5 = essential, but pan-essential = bad drug target (toxicity); selective = good target |
| "Get FAERS counts" | PRR > 5 = strong signal, but signal ≠ causation (channeling bias, notoriety bias) |

### 14b. Synthesis Phases
Every multi-phase skill needs a final phase that answers "so what?" — not just collecting data:
- "What changed and why does it matter?"
- "Is this cause or consequence?"
- "What's the actionable recommendation?"

### 14c. Honest Limitations
If a tool API can't deliver what the skill promises, say so explicitly. Don't describe aspirational capabilities. Example: "DepMap_get_gene_dependencies returns gene metadata only, NOT per-cell-line CRISPR scores."

## Pattern 15: Computational Procedures When Tools Can't Help

Some scientific analyses require computation, not just API queries. When no tool exists for a capability, embed a Python code procedure directly in the skill using packages available in ToolUniverse (pandas, scipy, numpy, statsmodels, biopython, networkx).

### When to use computational procedures:
| Gap | Procedure | Packages |
|-----|-----------|----------|
| API doesn't return needed data (e.g., DepMap scores) | Download CSV + pandas analysis | pandas |
| Statistical testing (differential abundance, enrichment) | scipy.stats + FDR correction | scipy, statsmodels |
| Sequence analysis (alignment, conservation) | Biopython SeqIO + pairwise alignment | biopython |
| Chemical similarity (analog search, fingerprints) | RDKit fingerprints + Tanimoto | rdkit (visualization extra) |
| Network analysis (hub genes, clustering) | NetworkX graph metrics | networkx |
| Scoring algorithms (ACMG classification, viability scores) | Custom Python functions | built-in |
| Dose feasibility (Cmax vs IC50 comparison) | Numerical comparison + PK data | pandas, numpy |

### Template for computational procedures in skills:
```markdown
**Computational procedure: [Name]**
[When to use this: explain the gap it fills]

\`\`\`python
# [What this computes]
# Requires: [packages] (included in ToolUniverse dependencies)
import pandas as pd
from scipy.stats import mannwhitneyu

# Input: [describe expected input format]
# Output: [describe output]
# [Full working code with example data]
\`\`\`

[Interpretation guidance for the output]
```

### Key rules for computational procedures:
1. **Only use packages in ToolUniverse dependencies** (pyproject.toml): pandas, scipy, numpy, networkx, requests, biopython (optional extra)
2. **Include example data** so the procedure is immediately testable
3. **Explain the output** — a code block without interpretation is useless
4. **Note when external data download is needed** (e.g., DepMap CSV from depmap.org)

### Pattern 15b: Download-and-Process for Datasets Without REST APIs

Many critical scientific datasets have NO REST API but provide bulk download files. Skills should include concrete download-and-process instructions when this is the only path to essential data.

**Template for download-and-process procedures:**

```markdown
**Step 1: Download data files**
- URL: [exact download page URL]
- Files needed: [filename] (~[size]) — [what it contains]
- Registration: [required/not required]
- Update frequency: [quarterly/annually/etc.]

**Step 2: Process with Python**
[Working code with pandas/scipy that loads the CSV and produces the analysis]

**Step 3: Interpret results**
[Table mapping output values to biological/clinical meaning]

**When files are not available**: [Fallback strategy using API tools]
```

**Known download-only datasets that skills reference:**

| Dataset | Download URL | Files | Used By |
|---------|-------------|-------|---------|
| **DepMap CRISPR** | depmap.org/portal/download/all/ | CRISPRGeneEffect.csv (~300MB), Model.csv (~2MB) | functional-genomics, cell-line-profiling |
| **TCGA clinical** | portal.gdc.cancer.gov | Clinical + mutation TSVs | cancer-genomics-tcga |
| **GTEx expression** | gtexportal.org/home/downloads | GTEx_Analysis_v8_Annotations.csv | expression-data-retrieval |
| **ClinGen gene-disease** | clinicalgenome.org/docs/ | gene_curation_list.tsv | variant-interpretation |
| **gnomAD constraint** | gnomad.broadinstitute.org/downloads | constraint metrics TSV | functional-genomics |

**Critical rule**: Always include a fallback for when the download is unavailable (user may not have registration, file may be too large, etc.). The fallback should use available API tools even if they provide less complete data.

## Common Anti-Patterns

| Anti-Pattern | Fix |
|-------------|-----|
| "Search Log" reports | Keep methodology internal; report findings only |
| Missing disambiguation | Add collision detection; build negative filters