dev-ai-coding-metrics

What this skill does

# AI Coding Agent Metrics for Engineering Teams

Measure what matters when adopting AI coding tools. This skill provides metrics frameworks, measurement methodology, ROI models, and reporting templates for engineering managers, VPs of Engineering, and CTOs evaluating or scaling AI coding agents.

## When to Use This Skill

- Evaluating AI coding tool ROI before or after purchase
- Building a metrics program for AI-assisted development
- Reporting AI tool impact to leadership or board
- Designing controlled experiments to measure AI effectiveness
- Comparing productivity across AI-equipped and traditional teams
- Tracking adoption health and identifying stall patterns
- Assessing quality impact of AI-generated code
- Running developer experience surveys for AI tools

## Quick Reference

| Task | Reference | Asset |
|------|-----------|-------|
| **Track tool adoption** | adoption-metrics.md | adoption-survey-template.md |
| **Measure productivity** | productivity-metrics.md | metric-dashboard-template.md |
| **Monitor code quality** | quality-metrics.md | metric-dashboard-template.md |
| **Calculate ROI** | roi-framework.md | roi-calculator-template.md |
| **Assess developer experience** | developer-experience-metrics.md | adoption-survey-template.md |
| **Design experiments** | benchmarking-methodology.md | experiment-design-template.md |
| **Report to executives** | roi-framework.md | executive-report-template.md |
| **Measure AI coding impact** | this skill | — |
| **Context engineering for AI** | dev-context-engineering | — |
| **Per-task agent ROI** | ai-agents | — |
| **Observability for systems** | qa-observability | — |

---

## Core Metrics Taxonomy

Five measurement categories. Start with Adoption (you can't optimize what people aren't using), then layer in the others.

### 1. Adoption Metrics

Track whether and how developers use AI tools.

| Metric | Formula | Target (Mature) | Source |
|--------|---------|-----------------|--------|
| License Utilization | active_users / licensed_seats | >85% | License admin |
| DAU/WAU Ratio | daily_active / weekly_active | >0.6 | Tool telemetry |
| Feature Breadth | features_used / features_available | >0.5 | Tool telemetry |
| Acceptance Rate | suggestions_accepted / suggestions_shown | 25-35% | Copilot API / tool logs |
| Organic Usage Ratio | voluntary_sessions / total_sessions | >0.8 | Survey + telemetry |

**Deep dive**: references/adoption-metrics.md — 8 additional metrics, adoption curve phases, tool-specific tracking, stall patterns.

### 2. Velocity Metrics

Measure speed and throughput changes.

| Metric | Formula | Expected AI Impact | Source |
|--------|---------|-------------------|--------|
| Deploy Frequency | deploys / time_period | +15-30% | CI/CD pipeline |
| Lead Time for Changes | commit_to_production | -20-40% | Git + CI/CD |
| Cycle Time | ticket_start_to_deploy | -15-35% | Project management + Git |
| PR Throughput | merged_PRs / developer / week | +20-40% | Git platform |
| Time to First Commit | onboard_date_to_first_commit | -30-50% | Git + HR data |

**Deep dive**: references/productivity-metrics.md — DORA adaptations, SPACE framework, cycle time decomposition, confounding variables.

### 3. Quality Metrics

Track whether AI helps or hurts code quality.

| Metric | Formula | Watch Direction | Source |
|--------|---------|----------------|--------|
| Bug Density | bugs / KLOC | Should decrease | Issue tracker |
| Defect Escape Rate | prod_bugs / total_bugs | Should decrease | Issue tracker |
| Rework Rate | followup_PRs / total_PRs | Watch for increase | Git platform |
| Test Coverage | covered_lines / total_lines | Should increase | CI coverage |
| Vulnerability Rate | new_vulns / sprint | Watch for increase | SAST tools |

**Critical warning**: Early studies show mixed quality results. AI can increase velocity while *also* increasing bug density if guardrails are missing. Monitor both.

**Deep dive**: references/quality-metrics.md — complexity tracking, security metrics, technical debt, quality guardrails.

### 4. Economic Metrics

Calculate costs, benefits, and ROI.

| Metric | Formula | Benchmark | Source |
|--------|---------|-----------|--------|
| Cost per Seat | (license + infra + training) / developers | $20-50/dev/month | Finance |
| Hours Saved/Dev/Week | measured_or_estimated_time_savings | 2-8 hrs (varies widely) | Survey + telemetry |
| ROI | (net_benefits - costs) / costs × 100 | 100-300% yr1 (vendor data) | Calculated |
| Payback Period | total_investment / monthly_net_benefit | 2-6 months | Calculated |
| Break-Even Adoption | cost / (max_benefit × developers) | 25-40% of team | Calculated |

**Caveat**: Most published ROI figures come from tool vendors. Independent studies show lower but still positive returns. Always triangulate.

**Deep dive**: references/roi-framework.md — cost model, value model, formulas, executive reporting, benchmarks with caveats.

### 5. Experience Metrics

Measure developer satisfaction and cognitive impact.

| Metric | Formula | Target | Source |
|--------|---------|--------|--------|
| AI Tool Satisfaction | survey_score (1-5 Likert) | >3.8/5.0 | Quarterly survey |
| Tool NPS | promoters% - detractors% | >30 | Quarterly survey |
| Cognitive Load | NASA-TLX adaptation (1-7) | <4.0/7.0 | Post-task survey |
| Give-Up Rate | started_AI_finished_manual / total | <20% | Telemetry |
| Trust Calibration | appropriate_review_rate | >80% | Code review data |

**Deep dive**: references/developer-experience-metrics.md — survey design, cognitive load measurement, friction indicators, trust metrics.

---

## Measurement Maturity Model

Where is your organization in measuring AI coding impact?

| Level | Name | Characteristics | Key Action |
|-------|------|----------------|------------|
| **L0** | No Measurement | No tracking beyond license count | Install basic telemetry, run first survey |
| **L1** | Basic Tracking | License utilization + adoption rate tracked | Add DORA metrics baseline, first ROI estimate |
| **L2** | Structured Program | DORA + adoption + quality metrics active, quarterly survey | Design controlled experiment, build dashboard |
| **L3** | Evidence-Based | Controlled experiments, statistical rigor, executive reporting | Cross-team benchmarking, predictive models |
| **L4** | Optimized | Continuous measurement, automated dashboards, data-driven tool selection | Industry benchmarking, publish findings |

### L0 → L1 Quick Start (2 hours)

1. Pull license utilization from admin console
2. Run the adoption survey (assets/adoption-survey-template.md)
3. Calculate basic ROI estimate (assets/roi-calculator-template.md)
4. Present 1-page summary to leadership (assets/executive-report-template.md)

### L1 → L2 (2-4 weeks)

1. Establish DORA metric baselines (references/productivity-metrics.md)
2. Set up quality tracking (references/quality-metrics.md)
3. Build three-tier dashboard (assets/metric-dashboard-template.md)
4. Schedule quarterly developer experience surveys

### L2 → L3 (1-3 months)

1. Design first controlled experiment (assets/experiment-design-template.md)
2. Apply statistical rigor (references/benchmarking-methodology.md)
3. Create executive reporting cadence (assets/executive-report-template.md)
4. Cross-reference with dev-context-engineering maturity model for context quality impact

### L3 → L4 (ongoing)

1. Automate data collection and dashboards
2. Build predictive models (adoption → productivity correlation)
3. Benchmark against industry data
4. Contribute findings to community (conference talks, blog posts)

---

## Metric Selection Decision Tree

Not every org needs every metric. Start from what you're trying to prove.

```text
WHAT ARE YOU TRYING TO PROVE?
  │
  ├─ "Should we buy AI coding tools?"
  │   └─ START: roi-framework.md → roi-calculator-template.md
  │       Metrics: cost per seat, estimated hours saved, break-even adoption rate
  │
  ├─ "Are developers actually using the tools?"
  │   └─ START: a