eve-verification-plans

What this skill does

# Eve Verification Plans

Author **agentic verification plans** — markdown documents that fully specify the steps to verify an Eve-compatible app works correctly AND conforms to Eve platform conventions. Plans are actionable by humans or agents.

## When to Use

- Building verification for a new Eve-compatible app
- Auditing an existing app for Eve platform conformance
- After significant feature work that needs structured validation
- Before handoff — proving the app works the Eve way, end to end
- Onboarding a new team to understand what "correctly built on Eve" looks like

## Quick Start

```bash
# 1. Create the verification directory in your project
mkdir -p e2e-verification/00-smoke

# 2. Copy the smoke template
cp templates/00-smoke-test-plan.md e2e-verification/00-smoke/

# 3. Customize for your app (endpoints, services, agents)
# 4. Run it
```

## The Six Verification Dimensions

Every Eve app has up to six dimensions to verify. Cover all that apply.

| Dimension | Tool | When | Conformance Check |
|-----------|------|------|--------------------|
| **Platform conformance** | Eve CLI + manifest inspection | Always | CLI parity, manifest conventions, secrets model |
| **Service layer** | Eve CLI + REST API | Always | Every endpoint reachable via CLI; no kubectl needed |
| **Input / ingestion** | Repo fixtures + upload commands | When app accepts files | Deterministic fixtures, real parsing flows |
| **Data layer** | Eve CLI + DB migrations | When app has DB | Migrations via Eve pipeline, not manual SQL |
| **UI / visual** | agent-browser or Playwright | When frontend exists | SSO login, dark/light mode, agent-testable |
| **Agent behavior** | `eve job follow` + optimization | When app has agents | Efficient completion, no blind alleys |

## Verification Plan Format

Each scenario is a self-contained markdown document:

```markdown
# Scenario NN: <Name>

**Time:** ~Nm
**Environment:** staging | local | both
**Parallel Safe:** Yes/No
**Requires:** LLM | Browser | None

<one-paragraph description>

## Prerequisites

- What must be true before running
- Required secrets, auth, prior scenarios

## Fixtures

- File paths used by this scenario
- Provenance or generation command for each
- Why these files are representative

## Setup

\```bash
# Environment detection + auth
# Project/org setup
# Fixture validation or generation
\```

## Phases

### Phase 1: <Name>

\```bash
# Commands to execute
\```

**Expected:**
- Bullet list of assertions
- Each assertion is pass/fail verifiable

### Phase 2: ...

## Success Criteria

- [ ] Checkboxes for every pass/fail assertion
- [ ] Grouped by phase

## Debugging

| Symptom | Diagnostic | Fix |
|---------|-----------|-----|
| ... | ... | ... |

## Cleanup

\```bash
# Teardown commands
\```
```

### Format Rules

- **Environment-aware**: Every plan starts with environment detection — `EVE_API_URL` determines cloud vs local
- **Self-contained**: No assumed state beyond documented prerequisites
- **Fixture-explicit**: Every uploaded/imported artifact is checked in or generated from documented commands
- **Phased**: Break into phases that can run independently (parallel where safe)
- **Assertion-driven**: Every step has explicit `Expected:` with pass/fail criteria
- **Debuggable**: Troubleshooting section with symptom → diagnostic → fix

## Platform Conformance Verification

Before testing functionality, verify the app is built the Eve way. Every verification suite starts with this checklist:

- [ ] `.eve/manifest.yaml` exists and passes `eve project sync --dry-run`
- [ ] Manifest follows current conventions (`name` preferred over legacy `project`)
- [ ] All services have health endpoints reachable via Eve ingress
- [ ] CLI can interact with every API endpoint (no "UI-only" functionality)
- [ ] Secrets managed via `eve secrets`, not hardcoded or env-file-based
- [ ] DB migrations run as pipeline steps, not manual scripts
- [ ] Agents (if any) defined in `agents.yaml` with harness profiles
- [ ] Pipelines (if any) defined in manifest and runnable via `eve pipeline run`
- [ ] Frontend (if any) authenticates via Eve SSO, not custom auth
- [ ] Upload/import flows (if any) have deterministic fixtures checked in or generated locally

See `references/eve-conformance-checks.md` for the full checklist with rationale.

## Service Layer Verification

Test every API surface CLI-first:

```bash
TOKEN=$(eve auth token --raw)

# Health check (always first)
curl -sf "${APP_SCHEME}://api.${APP_DOMAIN}/health" | jq '.'

# App API via auth token
curl -sf -H "Authorization: Bearer $TOKEN" \
  "${APP_SCHEME}://api.${APP_DOMAIN}/endpoint" | jq '.field'
```

**CLI parity assertion**: For every `curl` call in a test plan, ask: "Can this also be done via a CLI command?" If not, file an issue — don't accept it.

Pattern:
1. **Eve CLI** commands where they exist (deploy, env, job, secrets)
2. **App CLI** if the app follows `eve-app-cli` patterns
3. **REST API** via `curl` with auth tokens for custom endpoints
4. **Auth**: Mint tokens via `eve auth mint` or `eve auth token --raw`

## Input / Ingestion Verification

When the app accepts uploads, imports, or document bundles, verification must include a fixture plan.

### Fixture Selection Order

1. Reuse existing repo fixtures if they match accepted file types and are deterministic
2. Manufacture synthetic fixtures locally with committed scripts
3. Source small public-domain fixtures only when local manufacture would reduce realism

### Fixture Matrix

- **Minimal valid** — smallest acceptable file that exercises the happy path
- **Typical real-world** — representative document/media/import file
- **Boundary / invalid** — wrong type, malformed structure, or size edge
- **Cross-format** — if the app accepts multiple types (PDF + Markdown + CSV), verify each

### What to Check

- File accepted through the real app surface (CLI, REST endpoint, or browser upload)
- MIME/type detection and metadata are correct
- Storage/persistence path is correct
- Downstream processing produces expected results
- Error handling is explicit for rejected or malformed fixtures

**Rule**: If a plan says "upload a sample PDF", it must include an actual file path or generator step. "Find a PDF online" is not acceptable.

See `references/fixture-patterns.md` for detailed guidance.

## UI Verification

When the app has a frontend, verify visual quality and interaction flows.

### SSO Token Injection

```bash
# Mint an SSO token via CLI
SSO_TOKEN=$(eve auth mint --email [email protected] --org $ORG_ID --format sso-jwt)

# Use agent-browser with the token
agent-browser --session verify open "${APP_URL}/auth/callback?token=${SSO_TOKEN}"
agent-browser --session verify wait --url "**/dashboard"
agent-browser --session verify screenshot ./e2e-verification/artifacts/dashboard.png
```

### What to Check

- Pages render without console errors
- Dark mode and light mode both work (screenshot both)
- Key user flows complete (login → dashboard → action → result)
- Responsive layout at standard breakpoints
- Forms submit correctly and validation fires

See `references/ui-verification-patterns.md` for tool choice guidance and patterns.

## Agent Verification

When the app includes Eve agents, verification extends to behavior quality.

1. Create a job that exercises the agent's primary workflow
2. Follow execution: `eve job follow <job-id>`
3. Check receipt: `eve job receipt <job-id>` (tokens, cost, duration)
4. Apply `eve-agent-optimisation` diagnostic workflow
5. Record baseline metrics in the test plan for regression detection

### What to Check

- Agent completes its task (correct output)
- Token usage within acceptable bounds
- No unnecessary tool calls or blind alleys
- Error cases handled gracefully (bad input, missing secrets)
- Multi-agent coordination works (jobs complete in dependency order)

See `references/agent-verification-patterns.md` for integration with optimization.

## Deploy Cycle Patterns

Verification often rev