python

What this skill does

# Python Backend Development Skill

## File Organization

This skill uses a split structure for HIGH-RISK requirements:
- **SKILL.md**: Core principles, patterns, and essential security (this file)
- **references/security-examples.md**: Complete CVE details and OWASP implementations
- **references/advanced-patterns.md**: Advanced Python patterns and optimization
- **references/threat-model.md**: Attack scenarios and STRIDE analysis

## Validation Gates

| Gate | Status | Notes |
|------|--------|-------|
| 0.1 Domain Expertise | PASSED | Type safety, async, security, testing |
| 0.2 Vulnerability Research | PASSED | 5+ CVEs documented (2025-11-20) |
| 0.5 Hallucination Check | PASSED | Examples tested on Python 3.11+ |
| 0.11 File Organization | Split | HIGH-RISK, ~450 lines + references |

---

## 1. Overview

**Risk Level**: HIGH

**Justification**: Python backend services handle authentication, database access, file operations, and external API communication. Vulnerabilities in input validation, deserialization, command execution, and cryptography can lead to data breaches and system compromise.

You are an expert Python backend developer specializing in secure, maintainable, and performant services.

### Core Expertise Areas
- Type annotations and runtime validation
- Async programming with asyncio
- Security: input validation, cryptography, secrets management
- Testing: pytest, property-based testing, security testing
- Database access with SQLAlchemy/asyncpg
- API development with FastAPI/Starlette

---

## 2. Core Responsibilities

### Fundamental Principles

1. **TDD First**: Write tests before implementation, design API through test cases
2. **Performance Aware**: Use async, generators, efficient data structures by default
3. **Type Safety**: Use type hints everywhere, validate at runtime boundaries
4. **Defense in Depth**: Multiple validation layers, fail securely
5. **Secure Defaults**: Use safe libraries, reject unsafe operations
6. **Explicit over Implicit**: Clear error handling, explicit dependencies
7. **Testability**: Design for testing, write security tests

### Decision Framework

| Situation | Approach |
|-----------|----------|
| User input | Validate with Pydantic, sanitize output |
| Database queries | Use ORM or parameterized queries, never format strings |
| File operations | Validate paths, use pathlib, check containment |
| Subprocess | Use list args, never shell=True with user input |
| Secrets | Load from environment or secret manager |
| Cryptography | Use cryptography library, never roll your own |

---

## 2.1 Implementation Workflow (TDD)

### Step 1: Write Failing Test First

```python
import pytest
from my_service import UserService, UserNotFoundError

class TestUserService:
    @pytest.mark.asyncio
    async def test_get_user_returns_user_when_exists(self, db_session):
        service = UserService(db_session)
        user_id = await service.create_user("alice", "[email protected]")
        user = await service.get_user(user_id)
        assert user.username == "alice"

    @pytest.mark.asyncio
    async def test_get_user_raises_when_not_found(self, db_session):
        service = UserService(db_session)
        with pytest.raises(UserNotFoundError):
            await service.get_user(99999)

    @pytest.mark.asyncio
    async def test_create_user_validates_email(self, db_session):
        service = UserService(db_session)
        with pytest.raises(ValueError, match="Invalid email"):
            await service.create_user("bob", "not-an-email")
```

### Step 2: Implement Minimum to Pass

```python
class UserNotFoundError(Exception): pass

class UserService:
    def __init__(self, db: AsyncSession):
        self.db = db

    async def get_user(self, user_id: int) -> User:
        user = await self.db.get(User, user_id)
        if not user:
            raise UserNotFoundError(f"User {user_id} not found")
        return user

    async def create_user(self, username: str, email: str) -> int:
        if "@" not in email:
            raise ValueError("Invalid email format")
        # ... minimal implementation to pass tests
```

### Step 3: Refactor if Needed

- Extract common patterns, add type hints, ensure errors don't leak internals

### Step 4: Run Full Verification

```bash
pytest --cov=src           # All tests pass
mypy src/ --strict         # Type check passes
bandit -r src/ -ll         # Security scan passes
pip-audit && safety check  # Dependencies clean
```

---

## 2.2 Performance Patterns

### Pattern 1: Async I/O with asyncio.gather

```python
# BAD: Sequential requests (slow)
for url in urls:
    response = await client.get(url)  # Waits for each one

# GOOD: Concurrent requests with gather
tasks = [client.get(url) for url in urls]
responses = await asyncio.gather(*tasks)  # All at once
```

### Pattern 2: Generators for Large Data Processing

```python
# BAD: Load all into memory
return [process(line) for line in f.readlines()]  # OOM risk

# GOOD: Generator yields one at a time
def process_large_file(filepath: str) -> Iterator[dict]:
    with open(filepath) as f:
        for line in f:
            yield process(line)  # Memory efficient
```

### Pattern 3: Efficient Data Structures

```python
# BAD: List for membership testing - O(n)
required in user_perms_list  # Slow for large lists

# GOOD: Set for membership testing - O(1)
required in user_perms_set  # Fast lookup

# BAD: Repeated string concatenation
result = ""; for f in fields: result += f + ", "  # Creates new string each time

# GOOD: Join for string building
", ".join(fields)  # Single allocation
```

### Pattern 4: Connection Pooling

```python
# BAD: New connection per request
engine = create_async_engine(DATABASE_URL)  # Connection overhead each time

# GOOD: Reuse pooled connections
engine = create_async_engine(DATABASE_URL, pool_size=20, max_overflow=10)
async_session = sessionmaker(engine, class_=AsyncSession)

async def get_user(user_id: int):
    async with async_session() as session:  # Reuses pooled connection
        return await session.get(User, user_id)
```

### Pattern 5: Batch Database Operations

```python
# BAD: Individual inserts (N round trips)
for user in users:
    db.add(User(**user)); await db.commit()  # N commits = slow

# GOOD: Batch insert (1 round trip)
stmt = insert(User).values(users)
await db.execute(stmt); await db.commit()  # Single commit

# GOOD: Chunked for very large datasets
for i in range(0, len(users), 1000):
    await db.execute(insert(User).values(users[i:i+1000]))
await db.commit()
```

---

## 3. Technical Foundation

### Version Recommendations

| Category | Version | Notes |
|----------|---------|-------|
| **LTS/Recommended** | Python 3.11+ | Performance improvements, better errors |
| **Minimum** | Python 3.9 | Security support until Oct 2025 |
| **Avoid** | Python 3.8- | EOL, no security patches |

### Security Dependencies

```toml
# pyproject.toml
[project]
dependencies = [
    "pydantic>=2.0", "email-validator>=2.0",      # Validation
    "cryptography>=41.0", "argon2-cffi>=21.0",    # Cryptography
    "PyJWT>=2.8", "sqlalchemy>=2.0", "asyncpg>=0.28",
    "httpx>=0.25", "bandit>=1.7",
]

[project.optional-dependencies]
dev = ["pytest>=7.0", "pytest-asyncio>=0.21", "hypothesis>=6.0", "safety>=2.0", "pip-audit>=2.0"]
```

---

## 4. Implementation Patterns

### Pattern 1: Type-Safe Input Validation

```python
from pydantic import BaseModel, Field, field_validator, EmailStr
from typing import Annotated
import re

class UserCreate(BaseModel):
    """Validated user creation request."""
    username: Annotated[str, Field(min_length=3, max_length=50)]
    email: EmailStr
    password: Annotated[str, Field(min_length=12)]

    @field_validator('username')
    @classmethod
    def validate_username(cls, v: str) -> str:
        if not re.match(r'^[a-zA-Z0-9_-]+$', v):
            raise ValueError('Username must be alphanumeric')
        return v

    @field_validator('password')
    @classmethod