new-project

What this skill does

# Build a New AI Agent Project with Olakai

This skill guides you through creating a new AI agent that is fully integrated with Olakai for analytics, KPI tracking, and governance.

## Prerequisites

Before starting, ensure:
1. Olakai CLI installed: `npm install -g olakai-cli`
2. CLI authenticated: `olakai login`
3. API key for SDK (generated per-agent via CLI — see Step 2.2)

## Why Custom KPIs Are Essential

Olakai's core value is **tracking business-specific KPIs for your AI agents**. Without KPIs, you're tracking events without gaining actionable insights.

**What you can measure with KPIs:**
- Business outcomes (items processed, success rates, revenue impact)
- Operational data (step counts, retry rates, execution time)
- Quality indicators (error rates, user satisfaction signals)

**Without KPIs configured:**
- No dashboard KPIs beyond basic token counts
- No aggregated performance views
- No alerting thresholds
- No ROI calculations

> **Every agent should have 2-4 KPIs that answer: "How do I know this agent is performing well?"**

> **KPIs created here belong to this specific agent only.** If you later create additional agents, each one needs its own KPI definitions — KPIs cannot be shared or reused across agents.

## Understanding the customData to KPI Pipeline

Before diving into implementation, understand how data flows through Olakai:

```
SDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula → kpiData
```

### How It Works

1. **customData** (SDK): Raw JSON you send with each event
2. **CustomDataConfig** (Platform): Schema defining which fields are processed
3. **Context Variables**: CustomDataConfig fields become available for formulas
4. **KPI Formula**: Expression that computes a value (e.g., `SuccessRate * 100`)
5. **kpiData** (Response): Computed KPI values returned with each event

### Critical Rules

| Rule | Consequence |
|------|-------------|
| Only CustomDataConfig fields become variables | Unregistered customData fields are NOT usable in KPIs |
| Formula evaluation is case-insensitive | `stepCount`, `STEPCOUNT`, `StepCount` all work in formulas |
| NUMBER configs need numeric values | Don't send `"5"` (string), send `5` (number) |
| KPIs are unique per agent | Each KPI belongs to exactly one agent — create separately for each |

### Built-in Context Variables (Always Available)

| Variable | Type | Description |
|----------|------|-------------|
| `Prompt` | string | The prompt text sent to the LLM |
| `Response` | string | The LLM response text |
| `Documents count` | number | Number of attached documents |
| `PII detected` | boolean | Whether PII was detected |
| `PHI detected` | boolean | Whether PHI was detected |
| `CODE detected` | boolean | Whether code was detected |
| `SECRET detected` | boolean | Whether secrets were detected |

## Step 1: Design the Agent Architecture

### 1.1 Determine Agent Type

**Agentic AI** (Multi-step autonomous workflows):
- Research agents, document processors, data pipelines
- Track as SINGLE events aggregating all internal LLM calls
- Focus on workflow-level KPIs (total tokens, total time, success/failure)

**Assistive AI** (Interactive chatbots/copilots):
- Customer support agents, coding assistants, Q&A systems
- Track EACH interaction as separate events
- Focus on conversation-level KPIs (per-message tokens, response quality)

### 1.2 Design Your KPI Schema (CRITICAL)

**Design your KPIs BEFORE writing any SDK code.** This ensures only meaningful data is sent and tracked.

#### Step A: Identify Business Questions

What do stakeholders need to know about this agent?
- "How many items does it process per run?"
- "What's the success/failure rate?"
- "How efficient is each execution?"

#### Step B: Map Questions to Data Fields

| Business Question | Field Name | Type | KPI Formula | Aggregation |
|-------------------|------------|------|-------------|-------------|
| Throughput | ItemsProcessed | NUMBER | `ItemsProcessed` | SUM |
| Reliability | SuccessRate | NUMBER | `SuccessRate * 100` | AVERAGE |
| Error count | SuccessRate | NUMBER | `IF(SuccessRate < 1, 1, 0)` | SUM |
| Correlation | ExecutionId | STRING | (for filtering only) | - |

#### Step C: Plan Your customData Structure

```typescript
// ONLY include fields you'll register as CustomDataConfigs
customData: {
  // Business KPIs
  ItemsProcessed: number,  // Count of items handled
  SuccessRate: number,     // 0-1 success ratio

  // Performance KPIs
  StepCount: number,       // Number of workflow steps

  // Identification (for filtering, not KPIs)
  ExecutionId: string,     // Correlation ID
}
```

> **IMPORTANT**: Only include fields you will register as CustomDataConfigs. Unregistered fields are stored but **cannot be used in KPIs**.

### What NOT to Include in customData

The Olakai platform automatically tracks these — do NOT duplicate them:

| Already Tracked | Where | Don't Send As customData |
|-----------------|-------|--------------------------|
| Session ID | Main payload | `sessionId` |
| Agent ID | API key association | `agentId` |
| User email | `userEmail` parameter | `email`, `userEmail` |
| Timestamp | Event metadata | `timestamp`, `createdAt` |
| Request time | `requestTime` parameter | `duration`, `latency` |
| Token count | `tokens` parameter | `tokenCount` |
| Model | Auto-detected | `model`, `modelName` |
| Provider | Client config | `provider` |

**customData is ONLY for:**
1. **KPI variables** — Fields you'll use in formula calculations
2. **Tagging/filtering** — Fields you'll filter by in queries

## Step 2: Configure Olakai Platform

### 2.1 Create a Workflow (Required)

> **Every agent MUST belong to a workflow**, even if it's the only agent in that workflow.

```bash
olakai workflows create --name "Your Workflow Name" --json
# Output: { "id": "wfl_xxx...", "name": "Your Workflow Name" }
```

### 2.2 Create the Agent in Olakai

```bash
olakai agents create \
  --name "Your Agent Name" \
  --description "What this agent does" \
  --workflow WORKFLOW_ID \
  --with-api-key \
  --json

# Returns agent details including apiKey:
# {
#   "id": "cmkbteqn501kyjy4yu6p6xrrx",
#   "name": "Your Agent Name",
#   "workflowId": "wfl_xxx...",
#   "apiKey": "sk_agent_xxxxx..."   <-- Use this in your SDK
# }
```

**Agent-Workflow Hierarchy:**
```
Workflow: "Customer Support Pipeline"
├── Agent: "Ticket Classifier"
├── Agent: "Response Generator"
└── Agent: "Quality Checker"

Workflow: "Document Processing"
└── Agent: "Document Summarizer"  ← single-agent workflows are valid
```

### 2.3 Create Custom Data Configurations (BEFORE Writing SDK Code)

> **This step MUST be completed before Step 3 (SDK Integration).** Only fields registered here can be used in KPI formulas.

> **ONLY create configs for data you'll use in KPIs or for filtering.** Don't create configs for data already tracked automatically.

```bash
# For numeric fields (can be used in KPI calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ItemsProcessed" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "SuccessRate" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "StepCount" --type NUMBER

# For string fields (for filtering/grouping, not calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ExecutionId" --type STRING

# Verify all configs are created
olakai custom-data list --agent-id YOUR_AGENT_ID
```

### 2.4 Create KPI Definitions

#### Quick Start with Templates

Instead of writing formulas from scratch, use predefined classifier templates:

```bash
# List available templates
olakai kpis templates

# Create a classifier KPI from a template
olakai kpis create --name "User Satisfaction" \
  --calculator-id classifier --template-id sentiment_scorer \
  --scope CHAT --agent-id $AGENT_ID

# Create a time-saved estimator
olakai kpis create --name "Time Saved" \
  --calculator-id classifier --template-id time_saved_estimator \
  --scope CHAT --agent-i