reflect-appworld-failure

What this skill does

# Reflect on AppWorld Failure

Analyze failed AppWorld tasks to extract specific, actionable learnings that can be added to the playbook.

## Purpose

When an AppWorld task fails, the Reflector calls this Skill with error details and failed code. You analyze the failure semantically and generate a high-quality bullet with:
1. Specific title describing the pattern
2. Detailed content with working code examples
3. Relevant tags for retrieval
4. Appropriate confidence level

## Input Format

The input will be a text description with sections:

```
# Task
<task instruction>

## Apps
<comma-separated list of apps used>

## Error Type
<error_type: api_misuse, logic_error, timeout, etc.>

## Error Messages
<list of error messages from execution>

## Failed Code Snippet
<relevant code that failed>

## Missing Patterns (from heuristics)
<list of patterns the old system identified>

## Suggested Fixes (from heuristics)
<list of fix suggestions>
```

## Your Analysis Process

1. **Identify Root Cause**: What was the fundamental mistake?
   - Wrong API method name?
   - Missing authentication?
   - Incorrect data structure access?
   - Logic error?

2. **Extract Pattern**: What general pattern does this represent?
   - Is this specific to one app or applies to multiple?
   - Is this about API order (login first)?
   - Is this about method naming conventions?
   - Is this about data validation?

3. **Generate Concrete Example**: Create working code that demonstrates the CORRECT pattern

4. **Write Actionable Bullet**: Make it specific enough that the Generator can apply it

## Output Format

Return a JSON object with this structure:

```json
{
  "bullet": {
    "id": "bullet-YYYY-MM-DD-HHMMSS",
    "title": "<Specific pattern title>",
    "content": "<Detailed explanation with working code example>",
    "tags": ["app.<app_name>", "<error_category>", "<pattern_type>"],
    "evidence": [
      {
        "type": "execution",
        "ref": "<task_id>",
        "note": "<brief note about failure>"
      }
    ],
    "confidence": "high|medium|low",
    "scope": "app|global"
  }
}
```

## Bullet Quality Guidelines

### GOOD Bullets (Specific and Actionable)

**Title**: "Spotify: Use show_playlist_songs() not get_tracks()"
**Content**: "Spotify API uses show_playlist_songs(access_token, playlist_id) to retrieve tracks. The method get_tracks() does not exist. Example: `songs = apis.spotify.show_playlist_songs(access_token=token, playlist_id=playlist['id'])`"
**Tags**: ["app.spotify", "api_misuse", "method_names", "playlists"]

**Title**: "Venmo: Call login() before search_transactions()"
**Content**: "Venmo API requires authentication token for all operations. Always call venmo.login() first to get access_token, then pass it to other methods. Example: `response = apis.venmo.login(username='user', password='pass'); token = response['access_token']; results = apis.venmo.search_transactions(access_token=token, query={'friend': 'Alice'})`"
**Tags**: ["app.venmo", "authentication", "api_order", "search"]

### BAD Bullets (Too Generic)

**Title**: "Verify venmo API logic and requirements"
**Content**: "When implementing venmo operations: Check task logic and requirements; Missing login() call for venmo"
**Tags**: ["logic", "debugging", "api", "app.venmo"]

**Why Bad**: No concrete code example, vague guidance, doesn't teach the specific pattern

## Example Analysis

### Input:
```
# Task
What is the title of the most-liked song in my Spotify playlists

## Apps
spotify

## Error Type
api_misuse

## Error Messages
AttributeError: 'Spotify' object has no attribute 'get_tracks'

## Failed Code Snippet
songs = spotify.get_tracks(playlist_id=pid)

## Missing Patterns
- Use correct Spotify API methods

## Suggested Fixes
- Check Spotify API documentation for available methods
```

### Your Analysis:

1. **Root Cause**: Code used non-existent method `get_tracks()` instead of correct `show_playlist_songs()`

2. **Pattern**: Spotify uses `show_*` naming convention for retrieval methods

3. **Scope**: App-specific (Spotify)

### Output:
```json
{
  "bullet": {
    "id": "bullet-2025-10-27-123456",
    "title": "Spotify: Use show_playlist_songs() to get tracks from playlist",
    "content": "To retrieve songs from a Spotify playlist, use show_playlist_songs(access_token, playlist_id). Don't use get_tracks() - it doesn't exist. Example: `token = apis.spotify.login()['access_token']; playlists = apis.spotify.show_playlist_library(access_token=token); songs = apis.spotify.show_playlist_songs(access_token=token, playlist_id=playlists[0]['id']); most_liked = max(songs, key=lambda s: s['likes'])`",
    "tags": ["app.spotify", "api_misuse", "method_names", "playlists", "retrieval"],
    "evidence": [
      {
        "type": "execution",
        "ref": "spotify_task_001",
        "note": "AttributeError: 'Spotify' object has no attribute 'get_tracks'"
      }
    ],
    "confidence": "high",
    "scope": "app"
  }
}
```

## Common AppWorld Patterns to Look For

### Authentication Order
- Most apps require login() first to get access_token
- Token must be passed to subsequent API calls

### Method Naming Conventions
- Spotify: `show_*` for retrieval (show_playlist_songs, show_album_library)
- Venmo: `show_friends`, `send_payment`, `search_transactions`
- Gmail: `fetch_emails`, `send_email`
- Contacts: `show_contacts`, `add_contact`
- Calendar: `show_events`, `create_event`

### Data Structure Access
- API responses may have nested structures
- Always check if keys exist before accessing
- Use `.get()` with defaults for safety

### Aggregation Patterns
- To find "most-liked song in playlists": Get all playlists → Get songs from each → Find max by likes
- To find "most expensive transaction": Get all transactions → Find max by amount

### Task Completion
- ALWAYS call `apis.supervisor.complete_task()` at the end
- This signals successful completion to test framework

## Important Rules

1. **Be Specific**: Include actual method names, parameter names, and code examples
2. **Be Actionable**: The Generator should know exactly what to do after reading your bullet
3. **Include Working Code**: Show a complete example that demonstrates the correct pattern
4. **Tag Appropriately**: Use `app.<app_name>` for app-specific bullets, plus semantic tags
5. **Set Confidence**: "high" for clear patterns, "medium" for uncertain, "low" for speculative
6. **Return ONLY JSON**: No explanations, no markdown formatting outside the JSON

## Response Format

Return the JSON object as plain text. Make sure it's valid JSON that can be parsed directly.