audit-agents-skills

What this skill does

# Audit Agents/Skills/Commands (Advanced Skill)

Comprehensive quality audit system for Claude Code agents, skills, and commands. Provides quantitative scoring, comparative analysis, and production readiness grading based on industry best practices.

## Purpose

**Problem**: Manual validation of agents/skills is error-prone and inconsistent. According to the LangChain Agent Report 2026, 29.5% of organizations deploy agents without systematic evaluation, leading to "agent bugs" as the top challenge (18% of teams).

**Solution**: Automated quality scoring across 16 weighted criteria with production readiness thresholds (80% = Grade B minimum for production deployment).

**Key Features**:
- Quantitative scoring (32 points for agents/skills, 20 for commands)
- Weighted criteria (Identity 3x, Prompt 2x, Validation 1x, Design 2x)
- Production readiness grading (A-F scale with 80% threshold)
- Comparative analysis vs reference templates
- JSON/Markdown dual output for programmatic integration
- Fix suggestions for failing criteria

---

## Modes

| Mode | Usage | Output |
|------|-------|--------|
| **Quick Audit** | Top-5 critical criteria only | Fast pass/fail (3-5 min for 20 files) |
| **Full Audit** | All 16 criteria per file | Detailed scores + recommendations (10-15 min) |
| **Comparative** | Full + benchmark vs templates | Analysis + gap identification (15-20 min) |

**Default**: Full Audit (recommended for first run)

---

## Methodology

### Why These Criteria?

The 16-criteria framework is derived from:
1. **Claude Code Best Practices** (Ultimate Guide line 4921: Agent Validation Checklist)
2. **Industry Data** (LangChain Agent Report 2026: evaluation gaps)
3. **Production Failures** (Community feedback on hardcoded paths, missing error handling)
4. **Composition Patterns** (Skills should reference other skills, agents should be modular)

### Scoring Philosophy

**Weight Rationale**:
- **Identity (3x)**: If users can't find/invoke the agent, quality is irrelevant (discoverability > quality)
- **Prompt (2x)**: Determines reliability and accuracy of outputs
- **Validation (1x)**: Improves robustness but is secondary to core functionality
- **Design (2x)**: Impacts long-term maintainability and scalability

**Grade Standards**:
- **A (90-100%)**: Production-ready, minimal risk
- **B (80-89%)**: Good, meets production threshold
- **C (70-79%)**: Needs improvement before production
- **D (60-69%)**: Significant gaps, not production-ready
- **F (<60%)**: Critical issues, requires major refactoring

**Industry Alignment**: The 80% threshold aligns with software engineering best practices for production deployment (e.g., code coverage >80%, security scan pass rates).

---

## Workflow

### Phase 1: Discovery

1. **Scan directories**:
   ```
   .claude/agents/
   .claude/skills/
   .claude/commands/
   examples/agents/      (if exists)
   examples/skills/      (if exists)
   examples/commands/    (if exists)
   ```

2. **Classify files** by type (agent/skill/command)

3. **Load reference templates** (for Comparative mode):
   ```
   guide/examples/agents/     (benchmark files)
   guide/examples/skills/     (benchmark files)
   guide/examples/commands/   (benchmark files)
   ```

### Phase 2: Scoring Engine

Load scoring criteria from `scoring/criteria.yaml`:

```yaml
agents:
  max_points: 32
  categories:
    identity:
      weight: 3
      criteria:
        - id: A1.1
          name: "Clear name"
          points: 3
          detection: "frontmatter.name exists and is descriptive"
        # ... (16 total criteria)
```

For each file:
1. Parse frontmatter (YAML)
2. Extract content sections
3. Run detection patterns (regex, keyword search)
4. Calculate score: `(points / max_points) × 100`
5. Assign grade (A-F)

### Phase 3: Comparative Analysis (Comparative Mode Only)

For each project file:
1. Find closest matching template (by description similarity)
2. Compare scores per criterion
3. Identify gaps: `template_score - project_score`
4. Flag significant gaps (>10 points difference)

**Example**:
```
Project file: .claude/agents/debugging-specialist.md (Score: 78%, Grade C)
Closest template: examples/agents/debugging-specialist.md (Score: 94%, Grade A)

Gaps:
- Anti-hallucination measures: -2 points (template has, project missing)
- Edge cases documented: -1 point (template has 5 examples, project has 1)
- Integration documented: -1 point (template references 3 skills, project none)

Total gap: 16 points (explains C vs A difference)
```

### Phase 4: Report Generation

**Markdown Report** (`audit-report.md`):
- Summary table (overall + by type)
- Individual scores with top issues
- Detailed breakdown per file (collapsible)
- Prioritized recommendations

**JSON Output** (`audit-report.json`):
```json
{
  "metadata": {
    "project_path": "/path/to/project",
    "audit_date": "2026-02-07",
    "mode": "full",
    "version": "1.0.0"
  },
  "summary": {
    "overall_score": 82.5,
    "overall_grade": "B",
    "total_files": 15,
    "production_ready_count": 10,
    "production_ready_percentage": 66.7
  },
  "by_type": {
    "agents": { "count": 5, "avg_score": 85.2, "grade": "B" },
    "skills": { "count": 8, "avg_score": 78.9, "grade": "C" },
    "commands": { "count": 2, "avg_score": 92.0, "grade": "A" }
  },
  "files": [
    {
      "path": ".claude/agents/debugging-specialist.md",
      "type": "agent",
      "score": 78.1,
      "grade": "C",
      "points_obtained": 25,
      "points_max": 32,
      "failed_criteria": [
        {
          "id": "A2.4",
          "name": "Anti-hallucination measures",
          "points_lost": 2,
          "recommendation": "Add section on source verification"
        }
      ]
    }
  ],
  "top_issues": [
    {
      "issue": "Missing error handling",
      "affected_files": 8,
      "impact": "Runtime failures unhandled",
      "priority": "high"
    }
  ]
}
```

### Phase 5: Fix Suggestions (Optional)

For each failing criterion, generate **actionable fix**:

```markdown
### File: .claude/agents/debugging-specialist.md
**Issue**: Missing anti-hallucination measures (2 points lost)

**Fix**:
Add this section after "Methodology":

## Source Verification

- Always cite sources for technical claims
- Use phrases: "According to [documentation]...", "Based on [tool output]..."
- If uncertain, state: "I don't have verified information on..."
- Never invent: statistics, version numbers, API signatures, stack traces

**Detection**: Grep for keywords: "verify", "cite", "source", "evidence"
```

---

## Scoring Criteria

See `scoring/criteria.yaml` for complete definitions. Summary:

### Agents (32 points max)

| Category | Weight | Criteria Count | Max Points |
|----------|--------|----------------|------------|
| Identity | 3x | 4 | 12 |
| Prompt Quality | 2x | 4 | 8 |
| Validation | 1x | 4 | 4 |
| Design | 2x | 4 | 8 |

**Key Criteria**:
- Clear name (3 pts): Not generic like "agent1"
- Description with triggers (3 pts): Contains "when"/"use"
- Role defined (2 pts): "You are..." statement
- 3+ examples (1 pt): Usage scenarios documented
- Single responsibility (2 pts): Focused, not "general purpose"

### Skills (32 points max)

| Category | Weight | Criteria Count | Max Points |
|----------|--------|----------------|------------|
| Structure | 3x | 4 | 12 |
| Content | 2x | 4 | 8 |
| Technical | 1x | 4 | 4 |
| Design | 2x | 4 | 8 |

**Key Criteria**:
- Valid SKILL.md (3 pts): Proper naming
- Name valid (3 pts): Lowercase, 1-64 chars, no spaces
- Methodology described (2 pts): Workflow section exists
- No hardcoded paths (1 pt): No `/Users/`, `/home/`
- Clear triggers (2 pts): "When to use" section

### Commands (20 points max)

| Category | Weight | Criteria Count | Max Points |
|----------|--------|----------------|------------|
| Structure | 3x | 4 | 12 |
| Quality | 2x | 4 | 8 |

**Key Criteria**:
- Valid frontmatter (3 pts): name + description
- Argument hint (3 pts): If uses `$ARGUMENTS`
- Step-by-step workflow (3 pts):