evaluation-rubrics

What this skill does

# Evaluation Rubrics

## Table of Contents
- [Workflow](#workflow)
- [Common Patterns](#common-patterns)
- [Guardrails](#guardrails)
- [Quick Reference](#quick-reference)

## Example

**Scenario**: Evaluating technical blog posts (1-5 scale)

| Criterion | 1 (Poor) | 3 (Adequate) | 5 (Excellent) |
|-----------|----------|--------------|---------------|
| **Technical Accuracy** | Multiple factual errors, misleading | Mostly correct, minor inaccuracies | Fully accurate, technically rigorous |
| **Clarity** | Confusing, jargon-heavy, poor structure | Clear to experts, some structure | Accessible to target audience, well-organized |
| **Practical Value** | No actionable guidance, theoretical only | Some examples, limited applicability | Concrete examples, immediately applicable |
| **Originality** | Rehashes common knowledge, no new insight | Some fresh perspective, builds on existing | Novel approach, advances understanding |

**Scoring**: Post A [4, 5, 3, 2] = 3.5 avg. Post B [5, 4, 5, 4] = 4.5 avg.
**Feedback for Post A**: "Strong clarity (5) and good accuracy (4), but needs more practical examples (3) and offers less original insight (2)."

## Workflow

Copy this checklist and track your progress:

```
Rubric Development Progress:
- [ ] Step 1: Define purpose and scope
- [ ] Step 2: Identify evaluation criteria
- [ ] Step 3: Design the scale
- [ ] Step 4: Write performance descriptors
- [ ] Step 5: Test and calibrate
- [ ] Step 6: Use and iterate
```

**Step 1: Define purpose and scope**

Clarify what you're evaluating, who evaluates, who uses results, what decisions depend on scores. See [resources/template.md](resources/template.md#purpose-definition-template) for scoping questions.

**Step 2: Identify evaluation criteria**

Brainstorm quality dimensions, prioritize most important/observable, balance coverage vs. simplicity (4-8 criteria typical). See [resources/template.md](resources/template.md#criteria-identification-template) for brainstorming framework.

**Step 3: Design the scale**

Choose number of levels (1-5, 1-4, 1-10), scale type (numeric, qualitative), anchors (what does each level mean?). See [resources/methodology.md](resources/methodology.md#scale-design-principles) for scale selection guidance.

**Step 4: Write performance descriptors**

For each criterion × level, write observable description of what that performance looks like. See [resources/template.md](resources/template.md#descriptor-writing-template) for writing guidelines.

**Step 5: Test and calibrate**

Have multiple reviewers score sample work, compare scores, discuss discrepancies, refine rubric. See [resources/methodology.md](resources/methodology.md#calibration-techniques) for inter-rater reliability testing.

**Step 6: Use and iterate**

Apply rubric, collect feedback from evaluators and evaluatees, revise criteria/descriptors as needed. Validate using [resources/evaluators/rubric_evaluation_rubrics.json](resources/evaluators/rubric_evaluation_rubrics.json). **Minimum standard**: Average score ≥ 3.5.

## Common Patterns

**Pattern 1: Analytic Rubric (Most Common)**
- **Structure**: Multiple criteria (rows), multiple levels (columns), descriptor for each cell
- **Use case**: Detailed feedback needed, want to see performance across dimensions, diagnostic assessment
- **Pros**: Specific feedback, identifies strengths/weaknesses by criterion, high reliability
- **Cons**: Time-consuming to create and use, can feel reductive
- **Example**: Code review rubric (Correctness, Efficiency, Readability, Maintainability × 1-5 scale)

**Pattern 2: Holistic Rubric**
- **Structure**: Single overall score, descriptors integrate multiple criteria
- **Use case**: Quick overall judgment, summative assessment, criteria hard to separate
- **Pros**: Fast, intuitive, captures gestalt quality
- **Cons**: Less actionable feedback, lower reliability, can't diagnose specific weaknesses
- **Example**: Essay holistic scoring (1=poor essay, 3=adequate essay, 5=excellent essay with detailed descriptors)

**Pattern 3: Single-Point Rubric**
- **Structure**: Criteria listed with only "meets standard" descriptor, space to note above/below
- **Use case**: Growth mindset feedback, encourage self-assessment, less punitive feel
- **Pros**: Emphasizes improvement not deficit, simpler to create, encourages dialogue
- **Cons**: Less precision, requires written feedback to supplement
- **Example**: Design critique (list criteria like "Visual hierarchy", "Accessibility", note "+Clear focal point, -Poor contrast")

**Pattern 4: Checklist (Binary)**
- **Structure**: List of yes/no items, must-haves for acceptance
- **Use case**: Compliance checks, minimum quality gates, pass/fail decisions
- **Pros**: Very clear, objective, easy to use
- **Cons**: No gradations, misses quality beyond basics, can feel rigid
- **Example**: Pull request checklist (Tests pass? Code linted? Documentation updated? Security review?)

**Pattern 5: Standards-Based Rubric**
- **Structure**: Criteria tied to learning objectives/competencies, levels = degree of mastery
- **Use case**: Educational assessment, skill certification, training evaluation, criterion-referenced
- **Pros**: Aligned to standards, shows progress toward mastery, diagnostic
- **Cons**: Requires clear standards, can be complex to design
- **Example**: Data science skills (Proficiency in: Data cleaning, Modeling, Visualization, Communication × Novice/Competent/Expert)

## Guardrails

1. **Criteria should be observable and measurable**: Not "good attitude" (subjective), but "arrives on time, volunteers for tasks, helps teammates" (observable). Test: Can two independent reviewers score this criterion consistently?

2. **Descriptors should distinguish levels clearly**: Each level needs concrete differences from adjacent levels. Avoid "5=very good, 4=good, 3=okay". Better: "5=zero bugs, meets all requirements, 4=1-2 minor bugs, meets 90% requirements."

3. **Use appropriate scale granularity**: 1-3 is too coarse, 1-10 is too fine. Sweet spot: 1-4 (forced choice, no middle) or 1-5 (allows neutral middle). Match granularity to actual observable differences.

4. **Balance comprehensiveness with simplicity**: Aim for 4-8 criteria covering essential quality dimensions. If >10 criteria, consider grouping or prioritizing.

5. **Calibrate for inter-rater reliability**: Have multiple reviewers score same work, measure agreement (Kappa, ICC). If <70% agreement, refine descriptors.

6. **Provide examples at each level**: Include concrete examples of work at each level (anchor papers, reference designs, code samples) to calibrate reviewers.

7. **Share rubric before evaluation**: If evaluatees see the rubric only after being scored, it is grading not guidance. Share upfront so people know expectations and can self-assess.

8. **Weight criteria appropriately**: If "Security" matters more than "Code style", weight it (Security x3, Style x1). Or use thresholds (score >=4 on Security to pass, regardless of other scores).

**Common pitfalls:**

- ❌ **Subjective language**: "Shows effort", "creative", "professional" - not observable without concrete descriptors
- ❌ **Overlapping criteria**: "Clarity" and "Organization" often conflated - define boundaries clearly
- ❌ **Hidden expectations**: Rubric doesn't mention X, but evaluators penalize for missing X - document all criteria
- ❌ **Central tendency bias**: Reviewers avoid extremes (always score 3/5) - use even-number scales (1-4) to force choice
- ❌ **Halo effect**: High score on one criterion biases other scores up - score each criterion independently before looking at others
- ❌ **Rubric drift**: Descriptors erode over time, reviewers interpret differently - periodic re-calibration required

## Quick Reference

**Key resources:**

- **[resources/template.md](resources/template.md)**: Purpose definition, criteria brainstorming, scale selection, descriptor templates, rubric formats
- **[resources/methodology.md](resources/methodology.md)**: Sc