schemathesis

What this skill does

# Schemathesis

## Overview

Automatically generate and run API tests from OpenAPI and GraphQL schemas. Schemathesis finds bugs by generating thousands of test cases — boundary values, invalid types, malformed payloads, deep nesting — that developers never think to write manually.

## Instructions

### Installation

```bash
pip install schemathesis

# Or with all extras
pip install schemathesis[all]
```

### Quick Start

```bash
# Test a live API using its OpenAPI schema
st run https://api.example.com/openapi.json

# Test from a local schema file
st run ./openapi.yaml --base-url http://localhost:8080

# Test a GraphQL API
st run https://api.example.com/graphql
```

### How It Works

Schemathesis reads your API schema (OpenAPI 2.0/3.0/3.1 or GraphQL) and:

1. **Generates test cases** — valid and invalid inputs based on parameter types, constraints, and formats
2. **Sends requests** — fires thousands of combinations at your API
3. **Checks for failures** — 500 errors, schema violations, response timeouts, crashes
4. **Shrinks failures** — reduces failing test cases to the minimal reproducible example
5. **Reports results** — shows exactly which input caused which failure

```
Schema → Generator → Request → Response → Checker → Report
         ├── Valid values        ├── Status code OK?
         ├── Boundary values     ├── Response matches schema?
         ├── Invalid types       ├── No 500 errors?
         ├── Null/empty          ├── Response time OK?
         ├── Overflow values     └── No crashes?
         └── Unicode/special chars
```

### CLI Options

```bash
# Basic testing
st run https://api.example.com/openapi.json

# Authentication
st run URL --auth user:password                    # Basic auth
st run URL --header "Authorization: Bearer TOKEN"  # Bearer token
st run URL --header "X-API-Key: KEY"               # API key

# Target specific endpoints
st run URL --include-path "/api/users"             # Only test /api/users
st run URL --include-method POST                   # Only test POST endpoints
st run URL --exclude-path "/api/admin"             # Skip admin endpoints

# Control test volume
st run URL --hypothesis-max-examples=500    # Max test cases per endpoint
st run URL --hypothesis-deadline=5000       # Max ms per test case
st run URL --workers 4                      # Parallel workers

# Output
st run URL --report                     # Generate HTML report
st run URL --cassette-path=cassette.yaml  # Save all requests/responses
st run URL --junit-xml=results.xml      # JUnit format for CI
```

### Test Strategies

### Negative testing

Schemathesis automatically generates inputs that violate schema constraints:

```
If schema says: { "type": "integer", "minimum": 1, "maximum": 100 }
Schemathesis tries: 0, -1, -2147483648, 101, 999999999, null, "string", 1.5, []

If schema says: { "type": "string", "format": "email" }
Schemathesis tries: "", "not-an-email", "a@b", null, 12345, very-long-string...

If schema says: { "type": "array", "maxItems": 10 }
Schemathesis tries: [], [1000 items], null, "not-array", nested arrays...
```

### Stateful testing (link-based)

Schemathesis can chain API calls using OpenAPI links:

```bash
# Enable stateful testing — creates resources, then tests operations on them
st run URL --stateful=links

# Example flow:
# 1. POST /users → creates user, gets ID
# 2. GET /users/{id} → uses the created ID
# 3. PUT /users/{id} → updates with fuzzed data
# 4. DELETE /users/{id} → cleanup
```

This catches bugs that only appear with real resource IDs, not random values.

### Custom checks

```python
# custom_checks.py
# Add custom validation logic to Schemathesis test runs

import schemathesis

@schemathesis.check
def no_sensitive_data_in_errors(response, case):
    """Ensure error responses don't leak sensitive information.
    
    Checks that 4xx/5xx responses don't contain stack traces,
    database queries, or internal paths.
    """
    if response.status_code >= 400:
        body = response.text.lower()
        sensitive_patterns = [
            "traceback",
            "stack trace",
            "sql",
            "select * from",
            "/usr/local/",
            "/home/",
            "password",
            "secret",
            "internal server",
        ]
        for pattern in sensitive_patterns:
            assert pattern not in body, (
                f"Sensitive data '{pattern}' found in error response"
            )

@schemathesis.check
def response_time_acceptable(response, case):
    """Ensure no endpoint takes longer than 5 seconds.
    
    Slow responses might indicate injection vulnerabilities
    (time-based SQL injection) or denial-of-service potential.
    """
    assert response.elapsed.total_seconds() < 5.0, (
        f"Response took {response.elapsed.total_seconds():.1f}s "
        f"(limit: 5s) — possible DoS vector"
    )
```

```bash
# Run with custom checks
st run URL --checks all --hypothesis-max-examples=200
```

### Python API

```python
# test_api.py
# Use Schemathesis in pytest for CI integration

import schemathesis

# Load schema
schema = schemathesis.from_url("https://api.example.com/openapi.json")

# Or from file
schema = schemathesis.from_path("./openapi.yaml", base_url="http://localhost:8080")

# Generate test cases for all endpoints
@schema.parametrize()
def test_api(case):
    """Property-based test generated from OpenAPI schema.
    
    Schemathesis generates hundreds of test cases per endpoint,
    testing boundary values, invalid types, and edge cases.
    """
    response = case.call()
    case.validate_response(response)  # Check response matches schema

# Target specific endpoint
@schema.parametrize(endpoint="/api/users", method="POST")
def test_create_user(case):
    """Test user creation with generated inputs."""
    response = case.call()
    case.validate_response(response)
    assert response.status_code != 500, f"Server error with input: {case.body}"
```

### CI Integration

```yaml
# .github/workflows/api-test.yml
name: API Schema Testing
on: [push, pull_request]

jobs:
  schemathesis:
    runs-on: ubuntu-latest
    services:
      api:
        image: your-api:latest
        ports:
          - 8080:8080
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Schemathesis
        uses: schemathesis/action@v1
        with:
          schema: http://localhost:8080/openapi.json
          args: >-
            --checks all
            --stateful=links
            --hypothesis-max-examples=200
            --junit-xml=results.xml
      
      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: api-test-results
          path: results.xml
```

### Security-Focused Testing

For penetration testing, configure Schemathesis to look for security issues:

```bash
# Test for injection vulnerabilities
# Schemathesis will try SQL injection, XSS, and command injection payloads
# in string parameters
st run URL \
  --checks all \
  --hypothesis-max-examples=1000 \
  --header "Authorization: Bearer TOKEN" \
  --stateful=links \
  --report

# Common findings:
# - 500 errors on special characters → potential injection
# - Slow responses on certain inputs → time-based injection
# - Different error messages → information disclosure
# - Bypassed validation → missing server-side checks
# - Schema violations in responses → data leakage
```

## Examples

### Fuzz a REST API to find crashes

```prompt
Our REST API has an OpenAPI 3.0 spec at /api/docs/openapi.json. Run Schemathesis against all endpoints with 500 test cases per endpoint. Focus on finding 500 errors, schema violations, and slow responses (>3 seconds). Use stateful testing to chain CRUD operations. Generate an HTML report showing all findings with reproducible curl commands.
```

### Add API fuzzing to CI pipeline

```prompt
Set up Schemathesis in our GitHub Actions CI to run on every PR. The API starts in Docker (docker-compose up), schema is at lo