verify-rfc

What this skill does

# RFC Verification

You are verifying a design document (RFC, spec, design doc) against the actual implementation. Identify completeness, gaps, and discrepancies.

## Input

**RFC:** `$ARGUMENTS`

If the input is empty or literal "$ARGUMENTS", show brief usage with 2-3 examples, then stop. Otherwise continue.

---

## Core Principles

- **Thorough but efficient**: Don't over-analyze trivial items
- **Quote evidence**: Show specific code when claiming implementation exists
- **Be skeptical**: Just finding related code doesn't mean the requirement is met
- **Honesty over completeness**: Better to say "couldn't verify" than claim incorrectly

---

## Phase 1: Fetch the Design Document

**Goal**: Get the full RFC content

Input: $ARGUMENTS

**Actions**:
1. Get the document:
   - If given a URL: `read_document "[URL]"`
   - If given a topic: `search "[topic] RFC OR design doc"` then `read_document` on the best match

2. Extract verifiable requirements:
   - Functional requirements
   - Technical specifications
   - Integration points
   - Non-functional requirements (performance, security)

3. Create a checklist of requirements to verify

---

## Phase 2: Search the Codebase

**Goal**: Find implementations for each requirement

**Actions**:
1. For each major requirement area, search the local codebase:
   - Use `Glob` to find relevant files by name patterns
   - Use `Grep` to find specific implementations
   - Use `Read` to examine implementation details

2. Also search Glean for related internal code:
   ```
   code_search "[requirement] implementation"
   ```

3. Document what you find for each requirement

---

## Phase 3: Vet Each Finding (CRITICAL)

**Goal**: Rigorously verify claims - BE SKEPTICAL

For each requirement × implementation pair, evaluate:

**Match Quality Test**
- Does the code actually implement this requirement?
- ✅ IMPLEMENTED: Clear, direct implementation with evidence
- ⚠️ PARTIAL: Some aspects present, others missing
- ❓ UNCLEAR: Code exists but doesn't clearly fulfill requirement
- ❌ NOT FOUND: No evidence of implementation

**Evidence Strength Test**
- How confident are you in this assessment?
- ✅ HIGH: Can point to specific code that clearly implements this
- ⚠️ MEDIUM: Code appears related but not definitively implementing
- ❌ LOW: Inferring based on file names or tangential code

**Specification Fidelity Test**
- Does implementation match what the spec says?
- ✅ MATCHES: Implementation follows spec
- ⚠️ DIFFERS: Implementation exists but deviates from spec
- ❌ CONTRADICTS: Implementation does opposite of spec

**Common False Positives - BE CAREFUL**:
- Code that USES an API doesn't mean the API is IMPLEMENTED
- Test code doesn't count as implementation
- Commented-out code doesn't count
- TODO comments don't count
- Dead code or deprecated code doesn't count

**For each "Implemented" claim**:
- Require file:line reference
- Verify the code actually does what the spec requires
- Note any deviations

---

## Phase 4: Generate Verification Report

**Goal**: Present honest, evidence-based verification results

**Actions**:
Present the verification report:

```markdown
# RFC Verification Report

## Document
- **Title**: [RFC title]
- **URL**: [link]
- **Last Updated**: [date if available]

## Verification Confidence
| Confidence Level | Requirements |
|------------------|--------------|
| High (clear evidence) | [X] |
| Medium (likely but not certain) | [Y] |
| Low (couldn't verify definitively) | [Z] |

## Summary
| Status | Count | % |
|--------|-------|---|
| Implemented | X | X% |
| Partially Implemented | Y | Y% |
| Not Found | Z | Z% |
| Differs from Spec | W | W% |

**Overall**: [X of Y requirements verified with high confidence]

## Detailed Findings

### Implemented (High Confidence)
| Requirement | Evidence | Confidence |
|-------------|----------|------------|
| [Requirement] | [file:line] - [what it does] | High |

### Partially Implemented
| Requirement | Present | Missing | Evidence |
|-------------|---------|---------|----------|
| [Requirement] | [what exists] | [what's missing] | [file:line] |

### Not Found / Could Not Verify
| Requirement | What I Searched | Confidence |
|-------------|-----------------|------------|
| [Requirement] | [where we looked] | Low - may exist but couldn't locate |

### Differs from Spec
| Requirement | Spec Says | Implementation Does | Evidence |
|-------------|-----------|---------------------|----------|
| [Requirement] | [X] | [Y] | [file:line] |

## Limitations & Caveats

**What I couldn't verify:**
- [Requirement] - Would need [access/expertise] to verify
- [Requirement] - Code exists but unclear if it fulfills requirement

**Potential false negatives:**
- Implementation may exist under different name
- May be in private/restricted repo

## Recommendations
1. [Priority action to close gap]
2. [Verification that needs human review]

## Notes
- [Observations about spec being outdated]
- [Ambiguities in the original spec]
```

---

## If Verification Is Inconclusive

Be honest about limitations:

```markdown
# RFC Verification Report

## Verification Incomplete

I was able to verify [X] of [Y] requirements with high confidence.

**Well-verified areas:**
- [List of requirements with clear evidence]

**Could not verify:**
- [List with reasons]

**Why verification was limited:**
- [Reasons: access, complexity, ambiguity in spec]

**Recommended next steps:**
1. Have [team/person] review this report
2. Manually verify [specific requirements]
3. Update spec if requirements have changed
```

---

## Troubleshooting

### RFC Not Found
If the document can't be found:
- Ask for the exact URL or document title
- Search for alternative names
- Check if user has access to the document

### Implementation Not Found
If code can't be found:
- The feature may not be implemented yet - note this clearly
- Check if the RFC was superseded or abandoned
- Search for related terms

### Ambiguous Requirements
If the RFC has unclear requirements:
- Note which requirements are ambiguous
- Document assumptions made during verification
- Suggest RFC author clarify the spec