apply-rubric

What this skill does

# EAROS Rubric Evaluator

You are applying an EAROS rubric to an architecture artifact using the three-pass agent evaluation pattern. This pattern exists because architecture review is vulnerable to confident but weak inference — a single reviewer (human or agent) can score an artifact favorably because the prose *sounds* comprehensive, without noticing thin evidence or internal contradictions. The three-pass model catches this.

## The three passes

| Pass | Agent role | Purpose | Why it's separate |
|------|-----------|---------|-------------------|
| **1. Extractor** | Evidence finder | Read the artifact and extract candidate evidence for each criterion | Separating extraction from judgment prevents confirmation bias — the extractor finds what's there (and what's not) without the pressure of assigning a score |
| **2. Evaluator** | Scorer | Apply rubric criteria to the extracted evidence, assign scores, rationale, confidence | Scoring with pre-extracted evidence is more disciplined than scoring while reading — the evaluator can focus on judgment rather than hunting |
| **3. Challenger** | Adversarial reviewer | Challenge the evaluation for unsupported claims, over-scoring, missed gaps, rubric misuse | The challenger catches the evaluator's blind spots — disagreements surface the ambiguous cases that need human attention |

Read `references/agent-prompts.md` for the full prompt templates for each agent. Read `references/evaluation-schema.md` for the exact output format.

## How to interact with the user

This skill requires input from the user before evaluation can begin. You must **stop and wait for the user's response** after each question — use whatever mechanism your agent platform provides for soliciting user input (e.g., a question tool, a prompt, a form). Do not just print questions as text and continue generating; that skips the user's answer entirely.

## How to run this skill

### Step 0: Gather inputs

You need two things from the user. Stop and ask for each one, waiting for a response before proceeding.

**First, stop and ask the user for the rubric:**

> Which EAROS rubric should I apply? Please provide the file path to the rubric YAML file (profile or overlay).
>
> If you're not sure, tell me the artifact type (e.g., solution architecture, ADR, capability map) and I'll check for a matching rubric in the repository.

If the user mentions an artifact type but no rubric, check the `tmp/profiles/` and `tmp/overlays/` directories for a matching EAROS rubric. If none exists, suggest they create one first using the `earos-rubric` skill.

**Then, stop and ask the user for the artifact:**

> Which architecture artifact should I evaluate? Please provide the file path(s). This could be a markdown file, a PDF, a Word document, or a collection of files.

**Then, stop and ask for metadata:**

> A few more details for the evaluation record:
> - **Artifact ID and title** — a short identifier and human-readable name
> - **Artifact owner** — who is responsible for this artifact
> - **Additional overlays** — should I apply any cross-cutting overlays (security, data, regulatory) in addition to the profile?
>
> (If you're unsure about any of these, just say so and I'll use reasonable defaults.)

### Step 1: Load and compose the rubric

Read the rubric YAML file. If it has `inherits: [[email protected]]`, also load the core meta-rubric from `tmp/profiles/core-meta-rubric.v1.yaml`. Compose the full criterion set by merging:

1. All criteria from the core meta-rubric
2. All criteria from the artifact profile
3. All criteria from any attached overlays

Build a complete criterion list with all fields: id, question, required_evidence, scoring_guide, gate configuration, anti_patterns.

### Step 2: Assess artifact scope and plan agent strategy

Read the artifact to estimate its scope. Consider:

- **Number of sections/chapters** — more sections means more evidence to extract
- **Number of diagrams or views** — each needs separate analysis
- **Number of criteria to evaluate** — core (9-10) + profile-specific + overlay criteria
- **Artifact length** — longer artifacts benefit from parallel extraction

Based on this assessment, decide the agent strategy:

**Standard strategy** (artifact < ~50 pages, < 15 total criteria):
- 1 extractor agent
- 1 evaluator agent
- 1 challenger agent

**Parallel strategy** (artifact is comprehensive — many sections, many criteria, or > ~50 pages):
Split the criteria into groups and run multiple evaluator agents in parallel. Each evaluator handles a subset of dimensions. This reduces the risk of evaluator fatigue (where later criteria get less attention) and speeds up the evaluation.

Partition criteria by dimension for the parallel split — keep all criteria within the same dimension together so the evaluator can assess coherence within the dimension.

Tell the user which strategy you're using and why.

### Step 3: Run Pass 1 — Extractor

Spawn an extractor agent with the following task. The extractor must NOT score — it only extracts evidence.

Use the Agent tool to spawn a subagent with the prompt from `references/agent-prompts.md` Section "Extractor prompt". Provide:
- The full artifact content (read all artifact files)
- The complete criterion list with required_evidence fields
- Instructions to produce a structured evidence map

The extractor returns a JSON/YAML evidence map:
```yaml
evidence_map:
  - criterion_id: STK-01
    evidence_found:
      - location: "Section 1.2 Audience"
        excerpt: "This document is intended for the architecture board and engineering leads."
        evidence_class: observed
      - location: "Section 1.3 Purpose"
        excerpt: "The purpose is to gain approval for the proposed integration approach."
        evidence_class: observed
    evidence_gaps:
      - "No stakeholder-concern mapping found"
      - "Decision context not explicitly stated"
    evidence_sufficiency: partial
```

Wait for the extractor to complete before proceeding.

### Step 4: Run Pass 2 — Evaluator(s)

Spawn one or more evaluator agents using the prompt from `references/agent-prompts.md` Section "Evaluator prompt". Provide each evaluator with:
- The rubric criteria assigned to it (all criteria if single evaluator, a dimension-group if parallel)
- The evidence map from the extractor (filtered to relevant criteria)
- The scoring guidance from the rubric

**If using parallel strategy:** Spawn all evaluator agents simultaneously using the Agent tool. Each evaluator handles a distinct set of dimensions, so there are no dependencies between them.

Each evaluator returns criterion results:
```yaml
criterion_results:
  - criterion_id: STK-01
    score: 3
    judgment_type: observed
    confidence: high
    evidence_sufficiency: sufficient
    evidence_refs:
      - location: "Section 1.2 Audience"
        excerpt: "Architecture board and engineering leads are named."
    rationale: >
      Stakeholders are explicitly named and the decision purpose is clear.
      Minor gap: concerns are not systematically mapped to views.
    missing_information:
      - "Concern-to-view matrix"
    recommended_actions:
      - "Add stakeholder-concern-view table"
```

Collect results from all evaluators. If parallel, merge the criterion_results arrays.

### Step 5: Run Pass 3 — Challenger

Spawn a challenger agent using the prompt from `references/agent-prompts.md` Section "Challenger prompt". Provide:
- The original artifact content
- The evidence map from the extractor
- The complete evaluation results from the evaluator(s)
- The rubric criteria and scoring guidance

The challenger reviews each criterion result and produces challenges:
```yaml
challenges:
  - criterion_id: STK-01
    original_score: 3
    challenge_type: potential_over_score
    argument: >
      The evaluator scored 3 based on named stakeholders, but the
      scoring guide requires "explicit and mostly complete" for a 3.
      No concern mapping exists, which is a signif