skill-comply

What this skill does

# Skill Comply

Test whether an instruction artifact changes agent behavior in realistic
conditions. The goal is not to prove that the file is well-written; it is to
measure whether an agent follows it when the user prompt is supportive, neutral,
or competing.

## Contract

Inputs:

- Path to a `SKILL.md`, rule file, command, agent definition, or instruction doc
- Optional scenario list, trace logs, target harness, and pass threshold

Outputs:

- Expected behavior spec
- Scenario prompts by strictness level
- Compliance scores and failure categories
- Action timeline with matched, missing, reordered, or extra steps

Creates/Modifies:

- May create a local report file when requested
- Does not change the target skill or rule unless asked to fix findings

External Side Effects:

- May run local agent commands or tests when explicitly available
- No network calls or production writes by default

Confirmation Required:

- Before running costly model trials, modifying the target artifact, or writing reports outside the repo

Delegates To:

- `skill-validator` for frontmatter/spec checks
- `skill-auditor` for portfolio-level duplicate and quality audits
- `evaluation` for broader agent eval design

## When to Use

- A new skill, rule, command, or agent definition was added.
- An instruction was rewritten and behavior needs regression testing.
- The agent keeps ignoring a known rule.
- A maintainer wants a pass/fail signal before publishing an instruction pack.
- A rule works only when the user explicitly repeats it, but not in normal tasks.

## Core Idea

Compliance has three layers:

1. **Spec extraction**: convert the instruction artifact into observable steps.
2. **Scenario pressure**: test supportive, neutral, and competing prompts.
3. **Timeline matching**: compare observed actions to the expected sequence.

Measure behavior, not prose quality.

## Process

### 1. Derive the Expected Behavior

Read the target artifact and extract:

- trigger conditions
- required actions
- forbidden actions
- ordering constraints
- confirmation gates
- expected output shape
- side effects that must or must not happen

Represent the spec as a short table:

| Step | Required Behavior | Evidence Type | Required Order |
| --- | --- | --- | --- |
| 1 | Inspect existing examples before editing | search/read action | before write |
| 2 | Preserve unrelated user changes | diff/status check | before final |

### 2. Build Scenarios

Create three prompts:

- **Supportive**: explicitly asks for the workflow.
- **Neutral**: asks for the task but does not mention the instruction.
- **Competing**: adds pressure that often causes skipping, such as urgency,
  ambiguity, or a tempting shortcut.

For destructive or side-effecting skills, use dry-run scenarios against fixture
files or synthetic repos. Do not create real external side effects for a
compliance test.

### 3. Execute or Simulate

Use the strongest safe evidence available:

- actual tool/action trace from a sandbox run
- local command logs
- saved transcript
- reviewer reconstruction from a completed session
- dry-run plan when execution is not safe

Do not count a step as compliant because the final answer claims it happened.
Require an action trace, command output, file diff, or visible artifact.

### 4. Grade the Timeline

Classify each expected behavior:

| Result | Meaning |
| --- | --- |
| Matched | Required behavior happened with adequate evidence |
| Missing | Required behavior did not happen |
| Reordered | Behavior happened, but too late to matter |
| Weak | Behavior happened superficially or without enough evidence |
| Violated | A forbidden action happened |

Compute a simple score:

```text
score = matched / required
critical violation = automatic fail
```

Use judgment for ordering and gates. A late safety check after writing code
does not satisfy a pre-edit investigation requirement.

## Failure Categories

- **Trigger failure**: the skill or rule did not activate in neutral wording.
- **Ordering failure**: the right action happened after the risky action.
- **Evidence failure**: the agent claimed compliance without observable proof.
- **Scope failure**: the agent followed the rule for one file but not the full surface.
- **Conflict failure**: another instruction overrode the rule.
- **Overhead failure**: the rule caused excessive work for low-risk tasks.

## Output Format

```markdown
Compliance result: 2/3 scenarios passed. Neutral prompt failed because the agent edited before checking existing patterns.

| Scenario | Score | Verdict | Main Failure |
| --- | --- | --- | --- |
| Supportive | 5/5 | Pass | - |
| Neutral | 3/5 | Fail | Missing pre-edit example search |
| Competing | 4/5 | Pass | Weak final evidence |

## Timeline Findings

- Missing: [step] [evidence]
- Reordered: [step] [why order matters]

## Recommended Fix

1. Tighten the trigger phrase in `description`.
2. Move the precondition into the Contract.
3. Add a short output checklist.
```

## Promotion Guidance

When a step repeatedly fails:

- Make the trigger clearer if the skill is not selected.
- Move mandatory behavior into the Contract if it is buried in prose.
- Split the skill if it has multiple unrelated workflows.
- Convert fragile mandatory behavior into a hook, script, or command only when
  the current platform can actually enforce it.
- Add examples only when the failure is ambiguity, not lack of enforcement.

## Anti-Patterns

- Testing only prompts that name the skill.
- Treating final-answer claims as proof.
- Running live destructive actions to test a safety rule.
- Using a single scenario to certify a broad workflow.
- Fixing compliance by adding long repetitive wording instead of clarifying the
  trigger, contract, or gate.