redis-state-management

What this skill does

# Redis State Management

A comprehensive skill for mastering Redis state management patterns in distributed systems. This skill covers caching strategies, session management, pub/sub messaging, distributed locks, data structures, and production-ready patterns using redis-py.

## When to Use This Skill

Use this skill when:

- Implementing high-performance caching layers for web applications
- Managing user sessions in distributed environments
- Building real-time messaging and event distribution systems
- Coordinating distributed processes with locks and synchronization
- Storing and querying structured data with Redis data structures
- Optimizing application performance with Redis
- Scaling applications horizontally with shared state
- Implementing rate limiting, counters, and analytics
- Building microservices with Redis as a communication layer
- Managing temporary data with automatic expiration (TTL)
- Implementing leaderboards, queues, and real-time features

## Core Concepts

### Redis Fundamentals

**Redis** (Remote Dictionary Server) is an in-memory data structure store used as:
- **Database**: Persistent key-value storage
- **Cache**: High-speed data layer
- **Message Broker**: Pub/sub and stream messaging
- **Session Store**: Distributed session management

**Key Characteristics:**
- In-memory storage (microsecond latency)
- Optional persistence (RDB snapshots, AOF logs)
- Rich data structures beyond key-value
- Atomic operations on complex data types
- Built-in replication and clustering
- Pub/sub messaging support
- Lua scripting for complex operations
- Pipelining for batch operations

### Redis Data Structures

Redis provides multiple data types for different use cases:

1. **Strings**: Simple key-value pairs, binary safe
   - Use for: Cache values, counters, flags, JSON objects
   - Max size: 512 MB
   - Commands: SET, GET, INCR, APPEND

2. **Hashes**: Field-value maps (objects)
   - Use for: User profiles, configuration objects, small entities
   - Efficient for storing objects with multiple fields
   - Commands: HSET, HGET, HMGET, HINCRBY

3. **Lists**: Ordered collections (linked lists)
   - Use for: Queues, activity feeds, recent items
   - Operations at head/tail are O(1)
   - Commands: LPUSH, RPUSH, LPOP, RPOP, LRANGE

4. **Sets**: Unordered unique collections
   - Use for: Tags, unique visitors, relationships
   - Set operations: union, intersection, difference
   - Commands: SADD, SMEMBERS, SISMEMBER, SINTER

5. **Sorted Sets**: Ordered sets with scores
   - Use for: Leaderboards, time-series, priority queues
   - Range queries by score or rank
   - Commands: ZADD, ZRANGE, ZRANGEBYSCORE, ZRANK

6. **Streams**: Append-only logs with consumer groups
   - Use for: Event sourcing, activity logs, message queues
   - Built-in consumer group support
   - Commands: XADD, XREAD, XREADGROUP

### Connection Management

**Connection Pools:**
Redis connections are expensive to create. Always use connection pools:

```python
import redis

# Connection pool (recommended)
pool = redis.ConnectionPool(host='localhost', port=6379, db=0, max_connections=10)
r = redis.Redis(connection_pool=pool)

# Direct connection (avoid in production)
r = redis.Redis(host='localhost', port=6379, db=0)
```

**Best Practices:**
- Use connection pools for all applications
- Set appropriate max_connections based on workload
- Enable decode_responses=True for string data
- Configure socket_timeout and socket_keepalive
- Handle connection errors with retries

### Data Persistence

Redis offers two persistence mechanisms:

**RDB (Redis Database)**: Point-in-time snapshots
- Compact binary format
- Fast restart times
- Lower disk I/O
- Potential data loss between snapshots

**AOF (Append-Only File)**: Log of write operations
- Better durability (fsync policies)
- Larger files, slower restarts
- Can be automatically rewritten/compacted
- Minimal data loss potential

**Hybrid Approach**: RDB + AOF for best of both worlds

### RESP 3 Protocol

Redis Serialization Protocol version 3 offers:
- Client-side caching support
- Better data type support
- Push notifications
- Performance improvements

```python
import redis
from redis.cache import CacheConfig

# Enable RESP3 with client-side caching
r = redis.Redis(host='localhost', port=6379, protocol=3,
                cache_config=CacheConfig())
```

## Caching Strategies

### Cache-Aside (Lazy Loading)

**Pattern**: Application checks cache first, loads from database on miss

```python
import redis
import json
from typing import Optional, Dict, Any

r = redis.Redis(decode_responses=True)

def get_user(user_id: int) -> Optional[Dict[str, Any]]:
    """Cache-aside pattern for user data."""
    cache_key = f"user:{user_id}"

    # Try cache first
    cached_data = r.get(cache_key)
    if cached_data:
        return json.loads(cached_data)

    # Cache miss - load from database
    user_data = database.get_user(user_id)  # Your DB query
    if user_data:
        # Store in cache with 1 hour TTL
        r.setex(cache_key, 3600, json.dumps(user_data))

    return user_data
```

**Advantages:**
- Only requested data is cached (efficient memory usage)
- Cache failures don't break the application
- Simple to implement

**Disadvantages:**
- Cache miss penalty (latency spike)
- Thundering herd on popular items
- Stale data until cache expiration

### Write-Through Cache

**Pattern**: Write to cache and database simultaneously

```python
def update_user(user_id: int, user_data: Dict[str, Any]) -> bool:
    """Write-through pattern for user updates."""
    cache_key = f"user:{user_id}"

    # Write to database first
    success = database.update_user(user_id, user_data)

    if success:
        # Update cache immediately
        r.setex(cache_key, 3600, json.dumps(user_data))

    return success
```

**Advantages:**
- Cache always consistent with database
- No read penalty for recently written data

**Disadvantages:**
- Write latency increases
- Unused data may be cached
- Extra cache write overhead

### Write-Behind (Write-Back) Cache

**Pattern**: Write to cache immediately, sync to database asynchronously

```python
import redis
import json
from queue import Queue
from threading import Thread

r = redis.Redis(decode_responses=True)
write_queue = Queue()

def async_writer():
    """Background worker to sync cache to database."""
    while True:
        user_id, user_data = write_queue.get()
        try:
            database.update_user(user_id, user_data)
        except Exception as e:
            # Log error, potentially retry
            print(f"Failed to write user {user_id}: {e}")
        finally:
            write_queue.task_done()

# Start background writer
Thread(target=async_writer, daemon=True).start()

def update_user_fast(user_id: int, user_data: Dict[str, Any]):
    """Write-behind pattern for fast writes."""
    cache_key = f"user:{user_id}"

    # Write to cache immediately (fast)
    r.setex(cache_key, 3600, json.dumps(user_data))

    # Queue database write (async)
    write_queue.put((user_id, user_data))
```

**Advantages:**
- Minimal write latency
- Can batch database writes
- Handles write spikes

**Disadvantages:**
- Risk of data loss if cache fails
- Complex error handling
- Consistency challenges

### Cache Invalidation Strategies

**Time-based Expiration (TTL):**

```python
# Set key with expiration
r.setex("session:abc123", 1800, session_data)  # 30 minutes

# Or set TTL on existing key
r.expire("user:profile:123", 3600)  # 1 hour

# Check remaining TTL
ttl = r.ttl("user:profile:123")
```

**Event-based Invalidation:**

```python
def update_product(product_id: int, product_data: dict):
    """Invalidate cache on update."""
    # Update database
    database.update_product(product_id, product_data)

    # Invalidate related caches
    r.delete(f"product:{product_id}")
    r.delete(f"product_list:category:{product_data['category']}")
    r.delete("products:featured")
```

**Pattern-based Invalidation:**

```pyt