data-lakehouse

What this skill does

# data-lakehouse

## Purpose
This skill enables the design and implementation of data lakehouse architectures, combining data lakes with warehouse features for scalable big data storage and analytics. Use it to manage petabyte-scale data with ACID transactions, schema evolution, and optimized query performance on platforms like Delta Lake or Iceberg.

## When to Use
- When handling large-scale data ingestion from sources like S3 or Kafka, requiring both raw storage and structured querying.
- For analytics workloads needing real-time updates, such as ETL pipelines in e-commerce or IoT data processing.
- If you're integrating with Spark or Presto for SQL analytics on unstructured data.

## Key Capabilities
- **Architecture Design**: Generate blueprints for lakehouse setups, including partitioning strategies and metadata management (e.g., using Iceberg for table formats).
- **Data Ingestion**: Support for batch and streaming ingestion with tools like Apache Spark, handling formats like Parquet or ORC.
- **Query Optimization**: Implement caching and indexing for faster queries, such as creating Delta Lake tables with Z-order clustering.
- **Scalability**: Auto-scale storage and compute resources via cloud APIs, e.g., AWS Glue for ETL jobs.
- **Security**: Enforce row-level access controls using policies like AWS Lake Formation grants.

## Usage Patterns
- **Pattern 1**: For new lakehouse setup, invoke the skill to generate a configuration file, then use it to initialize storage. Example: Create a Delta Lake table from CSV data.
- **Pattern 2**: In analytics workflows, use the skill to optimize queries by adding indexes, then execute via Spark SQL.
- **Pattern 3**: For maintenance, periodically run checks for schema evolution and merge operations on existing tables.

## Common Commands/API
Use the OpenClaw CLI or API for this skill. Authentication requires setting `$DATA_LAKEHOUSE_API_KEY` as an environment variable.

- **CLI Command**: Initialize a lakehouse project:
  ```
  openclaw skill data-lakehouse init --project my-lakehouse --storage s3://my-bucket --engine delta
  ```
  This creates a basic configuration file with S3 bucket and Delta Lake engine.

- **API Endpoint**: Create a table via POST request:
  ```
  curl -H "Authorization: Bearer $DATA_LAKEHOUSE_API_KEY" \
       -d '{"table_name": "sales_data", "format": "parquet", "partition_by": ["date"]}' \
       https://api.openclaw.ai/data-lakehouse/tables
  ```
  Response includes table metadata for immediate use.

- **Code Snippet**: In Python, integrate with Spark:
  ```
  from pyspark.sql import SparkSession
  spark = SparkSession.builder.appName("lakehouse").getOrCreate()
  df = spark.read.format("delta").load("s3://my-bucket/sales_data")
  df.write.format("delta").mode("append").save("s3://my-bucket/sales_data")
  ```
  This appends data to a Delta table; ensure Spark is configured with AWS credentials.

- **Config Format**: Use JSON for lakehouse configs, e.g.:
  ```
  {
    "storage": "s3://my-bucket",
    "engine": "iceberg",
    "auth": {"key": "$DATA_LAKEHOUSE_API_KEY"}
  }
  ```
  Load this via CLI: `openclaw skill data-lakehouse apply --config path/to/config.json`.

## Integration Notes
- **With Other Skills**: Link to "big-data" skills by passing outputs, e.g., pipe data from a Kafka ingestion skill into this one using Spark streaming.
- **External Tools**: Integrate with AWS S3 by setting bucket policies; use `--aws-region us-west-2` in CLI commands. For Spark, ensure dependencies like `spark.delta` are in your environment.
- **API Integration**: When calling from other services, handle retries for rate limits; example: Use the same `$DATA_LAKEHOUSE_API_KEY` in chained API calls.
- **Dependency Management**: Always specify versions, e.g., require Spark 3.0+ for Delta Lake compatibility.

## Error Handling
- **Common Errors**: Handle authentication failures by checking if `$DATA_LAKEHOUSE_API_KEY` is set; use `os.environ.get('DATA_LAKEHOUSE_API_KEY')` in scripts.
- **Prescriptive Steps**: For storage access errors (e.g., S3 permissions), add try-except blocks:
  ```
  try:
      spark.read.format("delta").load("s3://my-bucket/data")
  except Exception as e:
      print(f"Error: {e}. Check bucket permissions and retry.")
  ```
  Retry transient errors like network issues with exponential backoff in API calls.
- **Debugging**: Use CLI flag `--verbose` for detailed logs, e.g., `openclaw skill data-lakehouse init --verbose`. Validate configs with `openclaw skill data-lakehouse validate --file config.json`.

## Concrete Usage Examples
- **Example 1**: Building a sales analytics lakehouse:
  First, initialize: `openclaw skill data-lakehouse init --project sales-ana --storage s3://sales-data`. Then, ingest data: Use the API to create a table, and append via Spark as shown above. Finally, query with: `spark.sql("SELECT * FROM sales_data WHERE date > '2023-01-01'")`.
  
- **Example 2**: Optimizing an existing lakehouse for IoT data:
  Run: `openclaw skill data-lakehouse optimize --table iot_metrics --add-index date`. This adds an index; verify with a query snippet: `df = spark.read.format("iceberg").load("s3://iot-bucket/metrics").filter(df.date > current_date())`. Monitor performance post-optimization.

## Graph Relationships
- Related to: "big-data" (shares tags for data processing pipelines), "data-engineering" (cluster affiliation for ETL workflows).
- Connected via: Tags like "data-lakehouse" for cross-skill queries, and "data-engineering" cluster for sequential task flows.