test-quality-inspector

What this skill does

# Example Test Inspection Report

## Scenario: User Registration Feature

### Engineer's Test Suite

```python
# test_user_registration.py

def test_user_creation():
    """Test user creation"""
    user = create_user("[email protected]", "password123")
    assert user

def test_login():
    """Test login"""
    user = create_user("[email protected]", "password123")
    result = login("[email protected]", "password123")
    assert result

def test_duplicate_email():
    """Test duplicate email"""
    create_user("[email protected]", "password123")
    user2 = create_user("[email protected]", "password456")
    assert user2
```

---

## QA Inspection Report

### Test 1: test_user_creation()

**Claimed Intent:** Test user creation
**Actually Tests:** Object instantiation only

#### Issues Found

🔴 **CRITICAL: Worthless Assertion**
```python
assert user  # Line 5
```
- Would pass with corrupt data
- Would pass with wrong email
- Would pass with invalid password hash
- Would pass if database save failed

**Mental Debug:** User object with garbage data would pass this test.

🟠 **HIGH: Missing Validation Tests**
- No test for invalid email format
- No test for weak password
- No test for duplicate email (separate test exists but broken)
- No test for SQL injection in email

🟠 **HIGH: No Persistence Verification**
- Test doesn't verify user is saved to database
- Test doesn't verify user can be retrieved
- Transaction might be rolled back

🟡 **MEDIUM: Vague Test Name**
- "test_user_creation" doesn't describe what should happen
- Better: "test_user_creation_with_valid_data_persists_to_database"

#### Suggested Improvements

```python
def test_user_creation_with_valid_data_persists_to_database():
    """Creating a user with valid email and password should:
    1. Create user object with correct attributes
    2. Save to database
    3. Hash password (not store plaintext)
    4. Set default role and active status
    """
    email = "[email protected]"
    password = "SecurePass123!"

    user = create_user(email, password)

    # Verify user object
    assert user.id is not None, "User should have an ID after creation"
    assert user.email == email, "Email should match input"
    assert user.is_active is True, "New users should be active by default"
    assert user.role == "user", "New users should have 'user' role"
    assert user.created_at is not None, "Created timestamp should be set"

    # Verify password is hashed, not plaintext
    assert user.password_hash != password, "Password should be hashed"
    assert len(user.password_hash) > 50, "Password hash should be substantial"

    # Verify persistence
    retrieved_user = User.get_by_email(email)
    assert retrieved_user is not None, "User should be retrievable from database"
    assert retrieved_user.id == user.id, "Retrieved user should match created user"

def test_user_creation_with_invalid_email_format_raises_validation_error():
    """Creating a user with malformed email should raise ValidationError"""
    invalid_emails = [
        "not-an-email",
        "@example.com",
        "test@",
        "test [email protected]",
        "",
    ]

    for invalid_email in invalid_emails:
        with pytest.raises(ValidationError) as exc:
            create_user(invalid_email, "password123")

        assert "email" in str(exc.value).lower()
        assert "invalid" in str(exc.value).lower()

def test_user_creation_with_weak_password_raises_validation_error():
    """Creating a user with weak password should raise ValidationError"""
    weak_passwords = [
        "123",          # Too short
        "password",     # No numbers
        "12345678",     # No letters
        "",             # Empty
    ]

    for weak_password in weak_passwords:
        with pytest.raises(ValidationError) as exc:
            create_user("[email protected]", weak_password)

        assert "password" in str(exc.value).lower()
```

**Risk Level:** 🔴 CRITICAL
**Action:** ❌ BLOCK - Core functionality not tested
**Estimated Fix Time:** 30 minutes

---

### Test 2: test_login()

**Claimed Intent:** Test login
**Actually Tests:** Function call completes

#### Issues Found

🔴 **CRITICAL: Worthless Assertion**
```python
assert result  # Line 11
```
- Passes with any truthy value
- Doesn't verify session/token
- Doesn't verify user authentication state

🔴 **CRITICAL: Missing Negative Tests**
- No test for wrong password
- No test for non-existent user
- No test for locked account
- No test for expired credentials

🟠 **HIGH: No Session Verification**
- Doesn't verify authentication token
- Doesn't verify session expiry
- Doesn't verify user context in session

🟡 **MEDIUM: Test Depends on Previous Test**
- Creates user in this test
- Should use fixture or setup
- Tests should be independent

#### Suggested Improvements

```python
@pytest.fixture
def registered_user():
    """Fixture providing a registered user for login tests"""
    user = create_user("[email protected]", "SecurePass123!")
    yield user
    # Cleanup if needed
    User.delete(user.id)

def test_login_with_valid_credentials_returns_authenticated_session(registered_user):
    """Logging in with correct email and password should:
    1. Return authentication token/session
    2. Set authenticated state
    3. Include user context
    4. Set appropriate expiry
    """
    session = login(registered_user.email, "SecurePass123!")

    assert session is not None, "Login should return session"
    assert session.is_authenticated is True, "Session should be authenticated"
    assert session.user_id == registered_user.id, "Session should contain user ID"
    assert session.token is not None, "Session should have authentication token"
    assert session.expires_at > datetime.now(), "Session should have future expiry"
    assert (session.expires_at - datetime.now()).seconds >= 3600, "Session should last at least 1 hour"

def test_login_with_wrong_password_raises_authentication_error(registered_user):
    """Logging in with incorrect password should raise AuthenticationError"""
    with pytest.raises(AuthenticationError) as exc:
        login(registered_user.email, "WrongPassword")

    assert "Invalid credentials" in str(exc.value)
    assert "password" in str(exc.value).lower()

def test_login_with_nonexistent_email_raises_authentication_error():
    """Logging in with non-existent email should raise AuthenticationError"""
    with pytest.raises(AuthenticationError) as exc:
        login("[email protected]", "password")

    assert "Invalid credentials" in str(exc.value)
    # Note: Don't reveal if email exists (security)

def test_login_with_locked_account_raises_account_locked_error(registered_user):
    """Logging in to locked account should raise AccountLockedError"""
    lock_account(registered_user.id)

    with pytest.raises(AccountLockedError) as exc:
        login(registered_user.email, "SecurePass123!")

    assert registered_user.email in str(exc.value)

def test_login_with_empty_password_raises_validation_error(registered_user):
    """Logging in with empty password should raise ValidationError"""
    with pytest.raises(ValidationError) as exc:
        login(registered_user.email, "")

    assert "password" in str(exc.value).lower()
    assert "required" in str(exc.value).lower()
```

**Risk Level:** 🔴 CRITICAL
**Action:** ❌ BLOCK - Authentication not actually tested
**Estimated Fix Time:** 45 minutes

---

### Test 3: test_duplicate_email()

**Claimed Intent:** Test duplicate email handling
**Actually Tests:** Second user creation succeeds (WRONG!)

#### Issues Found

🔴 **CRITICAL: Test is Backwards**
```python
user2 = create_user("[email protected]", "password456")
assert user2  # Line 17
```
- This test expects duplicate creation to SUCCEED
- It should expect it to FAIL with an error
- Test passes when it should fail
- **This is testing the opposite of what's needed**

🔴 **CRITICAL: False Confidence**
- Production bug: duplicate emails are allowed
- Te