dagu-workflows

What this skill does

# Dagu Workflow Authoring

This skill activates when creating or modifying Dagu workflow definitions, configuring workflow steps, scheduling, or composing complex workflows.

## When to Use This Skill

Activate when:
- Writing Dagu workflow YAML files
- Configuring workflow steps and executors
- Setting up workflow scheduling with cron
- Defining step dependencies and data flow
- Implementing error handling and retries
- Composing hierarchical workflows
- Using environment variables and parameters

## Basic Workflow Structure

### Minimal Workflow

```yaml
# hello.yaml
steps:
  - name: hello
    command: echo "Hello from Dagu!"
```

### Complete Workflow Structure

```yaml
name: my_workflow
description: Description of what this workflow does

# Schedule (optional)
schedule: "0 2 * * *"  # Cron format: daily at 2 AM

# Environment variables
env:
  - KEY: value
  - DB_HOST: localhost

# Parameters
params: ENVIRONMENT=production

# Email notifications (optional)
mailOn:
  failure: true
  success: false

smtp:
  host: smtp.example.com
  port: 587

errorMail:
  from: [email protected]
  to: [email protected]

# Workflow steps
steps:
  - name: step1
    command: echo "First step"

  - name: step2
    command: echo "Second step"
    depends:
      - step1
```

## Steps

### Basic Step

```yaml
steps:
  - name: greet
    command: echo "Hello, World!"
```

### Step with Script

```yaml
steps:
  - name: process
    command: |
      echo "Starting processing..."
      ./scripts/process.sh
      echo "Done!"
```

### Step with Working Directory

```yaml
steps:
  - name: build
    dir: /path/to/project
    command: make build
```

### Step with Environment Variables

```yaml
steps:
  - name: deploy
    env:
      - ENVIRONMENT: production
      - API_KEY: $API_KEY  # From global env
    command: ./deploy.sh
```

## Executors

### Command Executor (Default)

```yaml
steps:
  - name: shell_command
    command: ./script.sh
```

### Docker Executor

```yaml
steps:
  - name: run_in_container
    executor:
      type: docker
      config:
        image: alpine:latest
    command: echo "Running in Docker"

  - name: with_volumes
    executor:
      type: docker
      config:
        image: node:18
        volumes:
          - /host/path:/container/path
        env:
          - NODE_ENV=production
    command: npm run build
```

### SSH Executor

```yaml
steps:
  - name: remote_execution
    executor:
      type: ssh
      config:
        user: deploy
        host: server.example.com
        key: /path/to/ssh/key
    command: ./remote_script.sh
```

### HTTP Executor

```yaml
steps:
  - name: api_call
    executor:
      type: http
      config:
        method: POST
        url: https://api.example.com/webhook
        headers:
          Content-Type: application/json
          Authorization: Bearer $API_TOKEN
        body: |
          {
            "event": "workflow_complete",
            "timestamp": "{{.timestamp}}"
          }
```

### Mail Executor

```yaml
steps:
  - name: send_notification
    executor:
      type: mail
      config:
        to: [email protected]
        from: [email protected]
        subject: Workflow Complete
        message: |
          The workflow has completed successfully.
          Time: {{.timestamp}}
```

### JQ Executor

```yaml
steps:
  - name: transform_json
    executor:
      type: jq
      config:
        query: '.users[] | select(.active == true) | .email'
    command: cat users.json
```

## Step Dependencies

### Simple Dependencies

```yaml
steps:
  - name: download
    command: wget https://example.com/data.zip

  - name: extract
    depends:
      - download
    command: unzip data.zip

  - name: process
    depends:
      - extract
    command: ./process.sh
```

### Multiple Dependencies

```yaml
steps:
  - name: fetch_data
    command: ./fetch.sh

  - name: fetch_config
    command: ./fetch_config.sh

  - name: process
    depends:
      - fetch_data
      - fetch_config
    command: ./process.sh
```

### Parallel Execution

```yaml
# These run in parallel (no dependencies)
steps:
  - name: task1
    command: ./task1.sh

  - name: task2
    command: ./task2.sh

  - name: task3
    command: ./task3.sh

  # This waits for all above to complete
  - name: finalize
    depends:
      - task1
      - task2
      - task3
    command: ./finalize.sh
```

## Conditional Execution

### Preconditions

```yaml
steps:
  - name: deploy_production
    preconditions:
      - condition: "`echo $ENVIRONMENT`"
        expected: "production"
    command: ./deploy.sh
```

### Continue On Failure

```yaml
steps:
  - name: optional_step
    continueOn:
      failure: true
    command: ./might_fail.sh

  - name: cleanup
    depends:
      - optional_step
    command: ./cleanup.sh  # Runs even if optional_step fails
```

## Error Handling and Retries

### Retry Configuration

```yaml
steps:
  - name: flaky_api_call
    command: curl https://api.example.com/data
    retryPolicy:
      limit: 3
      intervalSec: 10
```

### Exponential Backoff

```yaml
steps:
  - name: with_backoff
    command: ./external_api.sh
    retryPolicy:
      limit: 5
      intervalSec: 5
      exponentialBackoff: true  # 5s, 10s, 20s, 40s, 80s
```

### Signal on Stop

```yaml
steps:
  - name: graceful_shutdown
    command: ./long_running_process.sh
    signalOnStop: SIGTERM  # Send SIGTERM instead of SIGKILL
```

## Data Flow

### Output Variables

```yaml
steps:
  - name: generate_id
    command: echo "ID_$(date +%s)"
    output: PROCESS_ID

  - name: use_id
    depends:
      - generate_id
    command: echo "Processing with ID: $PROCESS_ID"
```

### Script Output

```yaml
steps:
  - name: get_config
    script: |
      #!/bin/bash
      export DB_HOST="localhost"
      export DB_PORT="5432"
    output: DB_CONFIG

  - name: connect
    depends:
      - get_config
    command: ./connect.sh $DB_HOST $DB_PORT
```

## Scheduling

### Cron Schedule

```yaml
# Daily at 2 AM
schedule: "0 2 * * *"

# Every Monday at 9 AM
schedule: "0 9 * * 1"

# Every 15 minutes
schedule: "*/15 * * * *"

# First day of month at midnight
schedule: "0 0 1 * *"
```

### Start/Stop Times

```yaml
# Only run during business hours
schedule:
  start: "2024-01-01"
  end: "2024-12-31"
  cron: "0 9-17 * * 1-5"  # Mon-Fri, 9 AM to 5 PM
```

## Environment Variables

### Global Environment

```yaml
env:
  - ENVIRONMENT: production
  - LOG_LEVEL: info
  - API_URL: https://api.example.com

steps:
  - name: use_env
    command: echo "Environment: $ENVIRONMENT"
```

### Step-Level Environment

```yaml
steps:
  - name: with_custom_env
    env:
      - CUSTOM_VAR: value
      - OVERRIDE: step_value
    command: ./script.sh
```

### Environment from File

```yaml
env:
  - .env  # Load from .env file

steps:
  - name: use_env_file
    command: echo "DB_HOST: $DB_HOST"
```

## Parameters

### Defining Parameters

```yaml
params: ENVIRONMENT=development VERSION=1.0.0

steps:
  - name: deploy
    command: ./deploy.sh $ENVIRONMENT $VERSION
```

### Using Parameters

```bash
# Run with default parameters
dagu start workflow.yaml

# Override parameters
dagu start workflow.yaml ENVIRONMENT=production VERSION=2.0.0
```

## Sub-Workflows

### Calling Sub-Workflows

```yaml
# main.yaml
steps:
  - name: run_sub_workflow
    run: sub_workflow.yaml
    params: PARAM=value

  - name: another_sub
    run: workflows/another.yaml
```

### Hierarchical Workflows

```yaml
# orchestrator.yaml
steps:
  - name: data_ingestion
    run: workflows/ingest.yaml

  - name: data_processing
    depends:
      - data_ingestion
    run: workflows/process.yaml

  - name: data_export
    depends:
      - data_processing
    run: workflows/export.yaml
```

## Handlers

### Cleanup Handler

```yaml
handlerOn:
  exit:
    - name: cleanup
      command: ./cleanup.sh

steps:
  - name: main_task
    command: ./task.sh
```

### Error Handler

```yaml
handlerOn:
  failure:
    - name: send_alert
      executor:
        type: mail
        config:
          t