implement-retry-logic

What this skill does

Works with async/await operations in Python.
# Implement Retry Logic

## Purpose

Add retry logic with exponential backoff and jitter to async operations, following project patterns for resilience, configuration injection, and ServiceResult wrapping.

## When to Use

Use this skill when:
- **Implementing external service calls** - API calls that may timeout or fail
- **Database operations** - Database connections that may drop
- **Network requests** - Any network operation subject to transient failures
- **Operations that may fail transiently** - Temporary errors that can be retried

**Trigger phrases:**
- "Add retry logic for X"
- "Implement exponential backoff for Y"
- "Make Z resilient to failures"
- "Handle transient errors in X"

## Table of Contents

### Core Sections

- [Purpose](#purpose)
  - Core capability of the skill
- [Quick Start](#quick-start)
  - Pattern detection and basic implementation
- [Instructions](#instructions)
  - [Step 1: Identify Retry Requirements](#step-1-identify-retry-requirements) - When to use retry logic
  - [Step 2: Add Configuration](#step-2-add-configuration) - Configuration injection patterns
  - [Step 3: Implement Retry Logic](#step-3-implement-retry-logic) - Core retry implementation
  - [Step 4: Add Backoff Helper](#step-4-add-backoff-helper) - Exponential backoff with jitter
  - [Step 5: Classify Errors](#step-5-classify-errors) - Retriable vs permanent errors
  - [Step 6: Add Tests](#step-6-add-tests) - Test coverage for retry behavior

### Examples & Reference

- [Examples](#examples)
  - [Example 1: Database Operation Retry](#example-1-database-operation-retry) - Neo4j database operations with retry
  - [Example 2: External API Retry](#example-2-external-api-retry) - HTTP API calls with rate limiting
- [Requirements](#requirements)
  - Dependencies and project patterns
- [See Also](#see-also)
  - Supporting resources and reference implementations

### Supporting Resources

- [references/reference.md](./references/reference.md) - Advanced patterns and troubleshooting
- [templates/retry-template.py](./templates/retry-template.py) - Copy-paste template

### Utility Scripts
- [Add Retry Logic](./scripts/add_retry_logic.py) - Auto-add retry logic to async service methods
- [Analyze Retryable Operations](./scripts/analyze_retryable_operations.py) - Analyze codebase to find operations that need retry logic
- [Validate Retry Patterns](./scripts/validate_retry_patterns.py) - Validate retry logic implementations against best practices

## Quick Start

**Pattern Detection**: Look for external API calls, database operations, or network requests without retry logic.

**Basic Implementation**: Add retry loop with exponential backoff + jitter:

```python
# Configuration in settings
max_retries: int = 3
retry_delay: float = 1.0

# Implementation
for attempt in range(max_retries):
    try:
        result = await external_operation()
        return ServiceResult.ok(result)
    except RetriableError as e:
        if attempt < max_retries - 1:
            delay = min(retry_delay * (2 ** attempt), 30.0)
            jitter = delay * 0.2 * (2 * (time.time() % 1) - 1)
            await asyncio.sleep(max(0.1, delay + jitter))
        else:
            return ServiceResult.fail(f"Failed after {max_retries} retries: {e}")
```

## Instructions

### Step 1: Identify Retry Requirements

**Check if retry is needed:**
- [ ] External service call (API, database, network)
- [ ] Operation may fail transiently (timeouts, rate limits)
- [ ] Operation is idempotent (safe to retry)
- [ ] Failures should not crash the system

**Anti-patterns to avoid:**
- ❌ Retrying non-idempotent operations (creates duplicates)
- ❌ Retrying permanent errors (syntax errors, bad input)
- ❌ No backoff delay (hammers failing service)
- ❌ Infinite retries (never gives up)

### Step 2: Add Configuration

**Add retry settings to config/settings.py:**

```python
@dataclass
class ServiceSettings:
    """Configuration for [service name]."""

    # Existing fields...

    # Retry configuration
    max_retries: int = 3  # Maximum retry attempts
    retry_delay: float = 1.0  # Base delay in seconds

    @classmethod
    def from_env(cls) -> "ServiceSettings":
        return cls(
            # Existing fields...
            max_retries=int(os.getenv("SERVICE_MAX_RETRIES", "3")),
            retry_delay=float(os.getenv("SERVICE_RETRY_DELAY", "1.0")),
        )
```

**Configuration Rules:**
1. Always inject via Settings (never hardcode)
2. Provide environment variable overrides
3. Use sensible defaults (max_retries=3, retry_delay=1.0)
4. Document units (seconds, milliseconds)

### Step 3: Implement Retry Logic

**Use project pattern with exponential backoff + jitter:**

```python
async def _call_with_retry(self, operation_name: str) -> ServiceResult[T]:
    """Call external service with retry logic.

    Args:
        operation_name: Name for logging

    Returns:
        ServiceResult with operation result or error
    """
    last_error: str = ""

    for attempt in range(self.settings.max_retries):
        try:
            # Perform operation
            result = await self._perform_operation()
            return ServiceResult.ok(result)

        except aiohttp.ClientConnectionError as e:
            # Connection errors are retriable
            last_error = f"Connection error: {e}"
            if attempt < self.settings.max_retries - 1:
                delay = self._calculate_backoff_delay(attempt)
                logger.warning(
                    f"{last_error}. Retrying in {delay:.1f}s "
                    f"(attempt {attempt + 1}/{self.settings.max_retries})"
                )
                await asyncio.sleep(delay)

        except TimeoutError as e:
            # Timeouts are retriable
            last_error = f"Request timed out: {e}"
            if attempt < self.settings.max_retries - 1:
                delay = self._calculate_backoff_delay(attempt)
                logger.warning(
                    f"{last_error}. Retrying in {delay:.1f}s "
                    f"(attempt {attempt + 1}/{self.settings.max_retries})"
                )
                await asyncio.sleep(delay)

        except ValueError as e:
            # Validation errors are NOT retriable (permanent)
            logger.error(f"Validation error (non-retriable): {e}")
            return ServiceResult.fail(f"Validation error: {e}")

        except Exception as e:
            # Unexpected errors - fail fast
            logger.error(f"Unexpected error in {operation_name}: {e}")
            return ServiceResult.fail(f"Unexpected error: {e}")

    # All retries exhausted
    return ServiceResult.fail(
        f"Failed after {self.settings.max_retries} retries: {last_error}"
    )
```

**Key Components:**
1. **Retry Loop**: `for attempt in range(max_retries)`
2. **Error Classification**: Retriable vs permanent errors
3. **Backoff Calculation**: Exponential with jitter
4. **Logging**: Warning on retry, error on failure
5. **ServiceResult**: Always return ServiceResult, never raise

### Step 4: Add Backoff Helper

**Helper method for exponential backoff with jitter:**

```python
def _calculate_backoff_delay(self, attempt: int) -> float:
    """Calculate exponential backoff with jitter.

    Args:
        attempt: Current attempt number (0-based)

    Returns:
        Delay in seconds with jitter
    """
    base_delay = self.settings.retry_delay

    # Exponential backoff: base * 2^attempt, capped at 30s
    delay = min(base_delay * (2 ** attempt), 30.0)

    # Add jitter to avoid thundering herd (±20%)
    jitter = delay * 0.2 * (2 * (time.time() % 1) - 1)

    return max(0.1, delay + jitter)
```

**Jitter prevents thundering herd:**
- Multiple clients don't retry at exact same time
- Uses time.time() fractional seconds for randomness
- ±20% variance is industry standard

### Step 5: Classify Errors

**Determine which exceptions are retriable:**

```python
def _is_retriable_error(self, error: Exc