lumen-dashboards

What this skill does

# Lumen Dashboards Skill

## Overview

Lumen is a declarative framework for creating data dashboards through YAML specifications. Build interactive data exploration dashboards without writing code - just configuration.

### What is Lumen?

Lumen provides a declarative approach to building data dashboards:

- **No-code dashboards**: Define everything in YAML
- **Data pipelines**: Sources → Transforms → Views
- **Interactive exploration**: Built-in filters and cross-filtering
- **Component library**: Reusable sources, transforms, views
- **Live updates**: Auto-reload and real-time data

### Lumen vs Panel vs Lumen AI

| Feature | Lumen Dashboards | Panel | Lumen AI |
|---------|------------------|-------|----------|
| **Approach** | Declarative YAML | Imperative Python | Conversational |
| **Code Required** | No | Yes | No |
| **Use Case** | Fixed dashboards | Custom apps | Ad-hoc exploration |
| **Flexibility** | Medium | High | High |
| **Development Speed** | Very fast | Medium | Very fast |

**Use Lumen when**:
- Building standard data exploration dashboards
- Working with non-programmers
- Want rapid prototyping with configuration
- Need reproducible dashboard specifications

**Use Panel when**:
- Need fine-grained control over components
- Building custom application logic
- Creating novel interactions

**Use Lumen AI when**:
- Users need ad-hoc exploration
- Questions vary unpredictably
- Enabling self-service analytics

## Quick Start

### Installation

```bash
pip install lumen
```

### Your First Dashboard

**File: `dashboard.yaml`**

```yaml
sources:
  data:
    type: file
    tables:
      penguins: https://datasets.holoviz.org/penguins/v1/penguins.csv

pipelines:
  main:
    source: data
    table: penguins

    filters:
      - type: widget
        field: species

layouts:
  - title: Penguin Explorer
    views:
      - type: hvplot
        pipeline: main
        kind: scatter
        x: bill_length_mm
        y: bill_depth_mm
        by: species
        title: Bill Dimensions
```

**Launch:**
```bash
lumen serve dashboard.yaml --show
```

## Core Concepts

### 1. Sources

Data sources provide tables for your dashboard.

**Supported sources**:
- **File**: CSV, Parquet, Excel, JSON
- **Database**: PostgreSQL, DuckDB, SQLite
- **REST API**: JSON endpoints
- **Intake**: Data catalogs

**Quick example**:
```yaml
sources:
  mydata:
    type: file
    tables:
      sales: ./data/sales.csv
```

**See**: [Data Sources Reference](../../resources/lumen-dashboards/sources.md) for comprehensive source configuration.

### 2. Pipelines

Pipelines define data flows: Source → Filters → Transforms → Views

**Basic pipeline**:
```yaml
pipelines:
  sales_pipeline:
    source: mydata
    table: sales

    filters:
      - type: widget
        field: region

    transforms:
      - type: aggregate
        by: ['category']
        aggregate:
          total_sales: {revenue: sum}
```

**Components**:
- **Filters**: Interactive widgets for user input
- **Transforms**: Data manipulation (filter, aggregate, sort, SQL)
- **Views**: Visualizations and tables

### 3. Filters

Add interactive controls:

```yaml
filters:
  # Dropdown select
  - type: widget
    field: category

  # Multi-select
  - type: widget
    field: region
    multiple: true

  # Date range
  - type: widget
    field: date
    widget: date_range_slider

  # Numeric slider
  - type: param
    parameter: min_revenue
    widget_type: FloatSlider
    start: 0
    end: 100000
```

### 4. Transforms

Process data in pipelines:

**Common transforms**:
- `columns`: Select specific columns
- `query`: Filter rows with pandas query
- `aggregate`: Group and aggregate
- `sort`: Sort data
- `sql`: Custom SQL queries

**Example**:
```yaml
transforms:
  - type: columns
    columns: ['date', 'region', 'revenue']

  - type: query
    query: "revenue > 1000"

  - type: aggregate
    by: ['region']
    aggregate:
      total: {revenue: sum}
      avg: {revenue: mean}
```

**See**: [Data Transforms Reference](../../resources/lumen-dashboards/transforms.md) for all transform types.

### 5. Views

Visualize data:

**View types**:
- `hvplot`: Interactive plots (line, scatter, bar, etc.)
- `table`: Data tables
- `indicator`: KPI metrics
- `vega`: Vega-Lite specifications
- `altair`: Altair charts
- `plotly`: Plotly charts

**Example**:
```yaml
views:
  - type: hvplot
    pipeline: main
    kind: line
    x: date
    y: revenue
    by: category

  - type: indicator
    pipeline: main
    field: total_revenue
    title: Total Sales
    format: '${value:,.0f}'
```

**See**: [Views Reference](../../resources/lumen-dashboards/views.md) for all view types and options.

### 6. Layouts

Arrange views on the page:

```yaml
layouts:
  - title: Overview
    layout: [[0, 1, 2], [3], [4, 5]]  # Grid positions
    views:
      - type: indicator
        # View 0 config...

      - type: indicator
        # View 1 config...

      - type: hvplot
        # View 2 config...
```

**Layout types**:
- **Grid**: `[[0, 1], [2, 3]]`
- **Tabs**: Multiple layouts become tabs
- **Responsive**: Adapts to screen size

**See**: [Layouts Reference](../../resources/lumen-dashboards/layouts.md) for advanced layout patterns.

## Common Patterns

### Pattern 1: KPI Dashboard

```yaml
sources:
  metrics:
    type: file
    tables:
      data: ./metrics.csv

pipelines:
  kpis:
    source: metrics
    table: data
    transforms:
      - type: aggregate
        aggregate:
          total_revenue: {revenue: sum}
          total_orders: {orders: sum}
          avg_order_value: {revenue: mean}

layouts:
  - title: KPIs
    layout: [[0, 1, 2]]
    views:
      - type: indicator
        pipeline: kpis
        field: total_revenue
        format: '${value:,.0f}'

      - type: indicator
        pipeline: kpis
        field: total_orders
        format: '{value:,.0f}'

      - type: indicator
        pipeline: kpis
        field: avg_order_value
        format: '${value:.2f}'
```

### Pattern 2: Filtered Exploration

```yaml
pipelines:
  explorer:
    source: mydata
    table: sales

    filters:
      - type: widget
        field: region
        label: Region

      - type: widget
        field: category
        label: Category
        multiple: true

      - type: widget
        field: date
        widget: date_range_slider

    views:
      - type: hvplot
        kind: scatter
        x: price
        y: quantity
        by: category

      - type: table
        page_size: 20
```

### Pattern 3: Multi-Source Dashboard

```yaml
sources:
  sales_db:
    type: postgres
    connection_string: postgresql://localhost/sales
    tables: [orders, customers]

  inventory_file:
    type: file
    tables:
      stock: ./inventory.csv

pipelines:
  sales_pipeline:
    source: sales_db
    table: orders

  inventory_pipeline:
    source: inventory_file
    table: stock
```

### Pattern 4: Cross-Filtering

```yaml
pipelines:
  main:
    source: data
    table: sales

    filters:
      - type: widget
        field: region

layouts:
  - title: Analysis
    views:
      # Clicking bar filters other views
      - type: hvplot
        pipeline: main
        kind: bar
        x: category
        y: revenue
        selection_group: category_filter

      # Responds to selection above
      - type: hvplot
        pipeline: main
        kind: scatter
        x: price
        y: quantity
        selection_group: category_filter
```

### Pattern 5: SQL Transform

```yaml
transforms:
  - type: sql
    query: |
      SELECT
        region,
        category,
        SUM(revenue) as total_revenue,
        COUNT(*) as order_count,
        AVG(revenue) as avg_order_value
      FROM table
      WHERE date >= '2024-01-01'
      GROUP BY region, category
      HAVING total_revenue > 10000
      ORDER BY total_revenue DESC
```

## Python API

While Lumen is designed for YAML, you can also use Python:

```python
from lumen.sources import FileSource
from lumen.pipeline import Pipeline
from lumen.v