data-lake-architect

What this skill does

# Data Lake Architect Skill

You are an expert data lake architect specializing in modern lakehouse patterns using Rust, Parquet, Iceberg, and cloud storage. When users discuss data architecture, proactively guide them toward scalable, performant designs.

## When to Activate

Activate this skill when you notice:
- Discussion about organizing data in cloud storage
- Questions about partitioning strategies
- Planning data lake or lakehouse architecture
- Schema design for analytical workloads
- Data modeling decisions (normalization vs denormalization)
- Storage layout or directory structure questions
- Mentions of data retention, archival, or lifecycle policies

## Architectural Principles

### 1. Storage Layer Organization

**Three-Tier Architecture** (Recommended):

```
data-lake/
├── raw/              # Landing zone (immutable source data)
│   ├── events/
│   │   └── date=2024-01-01/
│   │       └── hour=12/
│   │           └── batch-*.json.gz
│   └── transactions/
├── processed/        # Cleaned and validated data
│   ├── events/
│   │   └── year=2024/month=01/day=01/
│   │       └── part-*.parquet
│   └── transactions/
└── curated/          # Business-ready aggregates
    ├── daily_metrics/
    └── user_summaries/
```

**When to Suggest**:
- User is organizing a new data lake
- Data has multiple processing stages
- Need to separate concerns (ingestion, processing, serving)

**Guidance**:
```
I recommend a three-tier architecture for your data lake:

1. RAW (Bronze): Immutable source data, any format
   - Keep original data for reprocessing
   - Use compression (gzip/snappy)
   - Organize by ingestion date

2. PROCESSED (Silver): Cleaned, validated, Parquet format
   - Columnar format for analytics
   - Partitioned by business dimensions
   - Schema enforced

3. CURATED (Gold): Business-ready aggregates
   - Optimized for specific use cases
   - Pre-joined and pre-aggregated
   - Highest performance

Benefits: Separation of concerns, reprocessability, clear data lineage.
```

### 2. Partitioning Strategies

#### Time-Based Partitioning (Most Common)

**Hive-Style**:
```
events/
├── year=2024/
│   ├── month=01/
│   │   ├── day=01/
│   │   │   ├── part-00000.parquet
│   │   │   └── part-00001.parquet
│   │   └── day=02/
│   └── month=02/
```

**When to Use**:
- Time-series data (events, logs, metrics)
- Queries filter by date ranges
- Retention policies by date
- Need to delete old data efficiently

**Guidance**:
```
For time-series data, use Hive-style date partitioning:

data/events/year=2024/month=01/day=15/part-*.parquet

Benefits:
- Partition pruning for date-range queries
- Easy retention (delete old partitions)
- Standard across tools (Spark, Hive, Trino)
- Predictable performance

Granularity guide:
- Hour: High-frequency data (>1GB/hour)
- Day: Most use cases (10GB-1TB/day)
- Month: Low-frequency data (<10GB/day)
```

#### Multi-Dimensional Partitioning

**Pattern**:
```
events/
├── event_type=click/
│   └── date=2024-01-01/
├── event_type=view/
│   └── date=2024-01-01/
└── event_type=purchase/
    └── date=2024-01-01/
```

**When to Use**:
- Queries filter on specific dimensions consistently
- Multiple independent filter dimensions
- Dimension has low-to-medium cardinality (<1000 values)

**When NOT to Use**:
- High-cardinality dimensions (user_id, session_id)
- Dimensions queried inconsistently
- Too many partition columns (>4 typically)

**Guidance**:
```
Be careful with multi-dimensional partitioning. It can cause:
- Partition explosion (millions of small directories)
- Small file problem (many <10MB files)
- Poor compression

Alternative: Use Iceberg's hidden partitioning:
- Partition on derived values (year, month from timestamp)
- Users query on timestamp, not partition columns
- Can evolve partitioning without rewriting data
```

#### Hash Partitioning

**Pattern**:
```
users/
├── hash_bucket=00/
├── hash_bucket=01/
...
└── hash_bucket=ff/
```

**When to Use**:
- No natural partition dimension
- Need consistent file sizes
- Parallel processing requirements
- High-cardinality distribution

**Guidance**:
```
For data without natural partitions (like user profiles):

// Hash partition user_id into 256 buckets
let bucket = hash(user_id) % 256;
let path = format!("users/hash_bucket={:02x}/", bucket);

Benefits:
- Even data distribution
- Predictable file sizes
- Good for full scans with parallelism
```

### 3. File Sizing Strategy

**Target Sizes**:
- Individual files: **100MB - 1GB** (compressed)
- Row groups: **100MB - 1GB** (uncompressed)
- Total partition: **1GB - 100GB**

**When to Suggest**:
- User has many small files (<10MB)
- User has very large files (>2GB)
- Performance issues with queries

**Guidance**:
```
Your files are too small (<10MB). This causes:
- Too many S3 requests (slow + expensive)
- Excessive metadata overhead
- Poor compression ratios

Target 100MB-1GB per file:

// Batch writes
let mut buffer = Vec::new();
for record in records {
    buffer.push(record);
    if estimated_size(&buffer) > 500 * 1024 * 1024 {
        write_parquet_file(&buffer).await?;
        buffer.clear();
    }
}

Or implement periodic compaction to merge small files.
```

### 4. Schema Design Patterns

#### Wide Table vs. Normalized

**Wide Table** (Denormalized):
```rust
// events table with everything
struct Event {
    event_id: String,
    timestamp: i64,
    user_id: String,
    user_name: String,        // Denormalized
    user_email: String,       // Denormalized
    user_country: String,     // Denormalized
    event_type: String,
    event_properties: String,
}
```

**Normalized**:
```rust
// Separate tables
struct Event {
    event_id: String,
    timestamp: i64,
    user_id: String,  // Foreign key
    event_type: String,
}

struct User {
    user_id: String,
    name: String,
    email: String,
    country: String,
}
```

**Guidance**:
```
For analytical workloads, denormalization often wins:

Pros of wide tables:
- No joins needed (faster queries)
- Simpler query logic
- Better for columnar format

Cons:
- Data duplication
- Harder to update dimension data
- Larger storage

Recommendation:
- Use wide tables for immutable event data
- Use normalized for slowly changing dimensions
- Pre-join fact tables with dimensions in curated layer
```

#### Nested Structures

**Flat Schema**:
```rust
struct Event {
    event_id: String,
    prop_1: Option<String>,
    prop_2: Option<String>,
    prop_3: Option<String>,
    // Rigid, hard to evolve
}
```

**Nested Schema** (Better):
```rust
struct Event {
    event_id: String,
    properties: HashMap<String, String>,  // Flexible
}

// Or with strongly-typed structs
struct Event {
    event_id: String,
    metadata: Metadata,
    metrics: Vec<Metric>,
}
```

**Guidance**:
```
Parquet supports nested structures well. Use them for:
- Variable/evolving properties
- Lists of related items
- Hierarchical data

But avoid over-nesting (>3 levels) as it complicates queries.
```

### 5. Table Format Selection

#### Raw Parquet vs. Iceberg

**Use Raw Parquet when**:
- Append-only workload
- Schema is stable
- Single writer
- Simple use case
- Cost-sensitive (fewer metadata files)

**Use Iceberg when**:
- Schema evolves frequently
- Need ACID transactions
- Multiple concurrent writers
- Updates/deletes required
- Time travel needed
- Partition evolution needed

**Guidance**:
```
Based on your requirements, I recommend Iceberg:

You mentioned:
- Schema might change (✓ schema evolution)
- Multiple services writing (✓ ACID transactions)
- Need to correct historical data (✓ updates)

Iceberg provides:
- Safe concurrent writes
- Schema evolution without rewriting
- Partition evolution
- Time travel for debugging
- Snapshot isolation

Trade-off: More metadata files and complexity
Benefit: Much better operational characteristics
```

### 6. Retention and Lifecycle

**Pattern**:
```
data/events/
├── hot/           # Last 7 days (frequent access)
│   └── year=2024/month=01/day=08/
├── warm/          # 8-90 da