shannon-execution-verifier

What this skill does

# Shannon Execution Verifier - Comprehensive Build Validation

## Purpose

Verify that Shannon Framework successfully built production-ready applications by inspecting physical outputs, testing runtime functionality, and validating cross-domain integration. This skill transforms Shannon testing from "does it output correct text?" to "does it build working applications?"

**Core Innovation**: Three-layer verification (Flow + Artifacts + Functionality) proves Shannon builds production applications, not just analyzes specifications.

---

## When to Use

Use this skill when:
- Shannon completed building an application via /shannon:wave
- Need to verify build quality and completeness
- Validating Shannon's cross-domain integration
- Testing NO MOCKS compliance in generated tests
- Preparing for production deployment
- Debugging why Shannon build might not work
- Meta-testing Shannon Framework itself (v5.0 validation)

Do NOT use when:
- Shannon only analyzed (didn't build)
- Partial builds (incomplete waves)
- Still in development (not ready to verify)

---

## The Three-Layer Verification Model

### Layer 1: Flow Verification (Execution Trace Analysis)

**Purpose**: Verify Shannon executed correct logic paths

**Method**: Analyze SDK message stream for tool_call sequences

**Verification Points**:
1. ✅ Expected skills invoked? (spec-analysis, wave-orchestration, etc.)
2. ✅ Skills chained correctly? (spec-analysis → mcp-discovery → phase-planning)
3. ✅ Agents spawned in parallel? (multiple Task calls in single message)
4. ✅ Correct agents for domains? (Frontend domain → FRONTEND agent)
5. ✅ MCP tools used appropriately? (puppeteer for web, xc-mcp for iOS)
6. ✅ Results saved to Serena? (write_memory calls present)
7. ✅ SITREP protocol if needed? (complexity >= 0.70)

**Example**:
```python
from inspection_lib.trace_analyzer import analyze_execution_trace

# Capture Shannon execution
trace = []
async for msg in query("/shannon:wave 1", options):
    trace.append(msg)

# Verify flow
flow_result = analyze_execution_trace(trace, expected_flow='sh_wave_flow.yaml')

# Report
if flow_result.passed:
    print("✅ Flow Verification: PASSED")
    print(f"   - Skills invoked: {flow_result.skills_used}")
    print(f"   - Agents spawned: {flow_result.agents_spawned}")
    print(f"   - Parallel execution: {flow_result.parallel_detected}")
else:
    print("❌ Flow Verification: FAILED")
    print(f"   - Missing: {flow_result.missing_steps}")
```

---

### Layer 2: Artifact Verification (Physical Output Inspection)

**Purpose**: Verify Shannon created expected physical outputs

**Method**: Inspect file system, Serena memories, git state

**Verification Points**:
1. ✅ Files created? (React components, API files, schemas, tests)
2. ✅ File structure correct? (src/, tests/, config files in right places)
3. ✅ Serena memories exist? (wave_N_complete, agent_results checkpoints)
4. ✅ Memory structure valid? (JSON with expected fields)
5. ✅ Git commits made? (code committed during execution)
6. ✅ Dependencies configured? (package.json, requirements.txt populated)

**Example**:
```python
from inspection_lib.file_inspector import verify_file_structure
from inspection_lib.memory_inspector import verify_serena_artifacts

# Verify files created
file_result = verify_file_structure(
    expected_structure='scenarios/prd_creator.yaml'
)

print(f"Frontend files: {file_result.frontend_files_found}/{file_result.frontend_files_expected}")
print(f"Backend files: {file_result.backend_files_found}/{file_result.backend_files_expected}")

# Verify Serena memories
memory_result = verify_serena_artifacts(wave_number=1)

print(f"Wave checkpoint: {'✅' if memory_result.checkpoint_exists else '❌'}")
print(f"Agent results: {len(memory_result.agent_results)}")
```

---

### Layer 3: Functional Verification (Runtime Testing)

**Purpose**: Verify built application actually works in production-like environment

**Method**: Start services, test with real tools (curl, Playwright, psql, xc-mcp)

**Verification Points**:
1. ✅ Servers start? (npm run dev, uvicorn, docker)
2. ✅ Endpoints accessible? (HTTP 200 responses)
3. ✅ Frontend renders? (Playwright can load and interact)
4. ✅ Backend processes requests? (curl POST/GET/PUT/DELETE work)
5. ✅ Database persists data? (psql queries return data)
6. ✅ Validation enforced? (Invalid input → proper error)
7. ✅ Error handling present? (404, 500 pages exist)
8. ✅ Integration works? (Frontend → Backend → Database loop)
9. ✅ NO MOCKS compliant? (Real browser, real HTTP, real database)
10. ✅ Cross-platform? (Mobile/desktop viewports, multiple browsers)

**Example**:
```python
from inspection_lib.runtime_inspector import start_and_test_services
from domain_verifiers.frontend_verifier import verify_frontend
from domain_verifiers.backend_verifier import verify_backend
from domain_verifiers.database_verifier import verify_database

# Start all services
services = start_and_test_services()

# Verify frontend
frontend_result = verify_frontend(
    url='http://localhost:5173',
    browser='chromium',
    viewports=['mobile', 'desktop']
)

# Verify backend (curl tests, NO pytest)
backend_result = verify_backend(
    base_url='http://localhost:8000',
    test_mode='curl'  # Real HTTP, not TestClient
)

# Verify database
database_result = verify_database(
    connection='postgresql://localhost:5432/prd_creator',
    expected_tables=['prds', 'users']
)

# Verify integration
integration_result = verify_end_to_end_flow(
    frontend_url='http://localhost:5173',
    backend_url='http://localhost:8000',
    database_connection=database_result.connection
)

# Report
print(f"✅ Frontend: {frontend_result.passed}")
print(f"✅ Backend: {backend_result.passed}")
print(f"✅ Database: {database_result.passed}")
print(f"✅ Integration: {integration_result.passed}")
```

---

## Domain-Specific Verification

### Frontend Verification (frontend_verifier.py)

**For**: React, Vue, Angular, Svelte applications

**Checks**:
1. **Build Verification**:
   - npm install succeeds
   - npm run build succeeds
   - Build outputs dist/ or build/ directory

2. **Development Server**:
   - npm run dev starts without errors
   - Server listens on configured port
   - Health endpoint returns 200

3. **Playwright Functional Testing**:
   - Can navigate to application
   - Page renders without JavaScript errors
   - Forms are interactive
   - Buttons trigger actions
   - Data displays correctly

4. **NO MOCKS Compliance**:
   - Scan test files for: jest.mock, vi.mock, cy.stub
   - Verify Playwright used for browser testing
   - Verify real API calls (not mocked fetch)

5. **Responsive Design**:
   - Mobile viewport (375x667): UI adapts
   - Desktop viewport (1920x1080): Full layout
   - Tablet viewport (768x1024): Intermediate layout

6. **Cross-Browser**:
   - Chrome: Renders and functions
   - Firefox: Renders and functions
   - Safari: Renders and functions

---

### Backend Verification (backend_verifier.py)

**For**: Express, FastAPI, Django, Flask APIs

**Checks**:
1. **Server Start**:
   - Server process starts (uvicorn, node, python)
   - Listens on configured port
   - No startup errors

2. **Health Check**:
   - /health or / endpoint returns 200
   - OpenAPI docs accessible (/docs for FastAPI)

3. **Functional API Testing (curl, NO pytest TestClient)**:
   ```bash
   # Create
   curl -X POST http://localhost:8000/api/resource \
     -H "Content-Type: application/json" \
     -d '{"field": "value"}'
   # Expect: 201 Created

   # Read
   curl http://localhost:8000/api/resource/1
   # Expect: 200 OK, returns JSON

   # Update
   curl -X PUT http://localhost:8000/api/resource/1 \
     -d '{"field": "updated"}'
   # Expect: 200 OK

   # Delete
   curl -X DELETE http://localhost:8000/api/resource/1
   # Expect: 204 No Content
   ```

4. **Validation Enforcement**:
   ```bash
   # Invalid input
   curl -X POST http://localhost:8000/api/resource \
     -d '{"invalid": "data"}'
   # Expect: 400 or 422 with error me