genai-services

What this skill does

# OCI Generative AI Services - Expert Knowledge

## 🏗️ Use OCI Landing Zone Terraform Modules

**Don't reinvent the wheel.** Use [oracle-terraform-modules/landing-zone](https://github.com/orgs/oci-landing-zones/repositories) for GenAI infrastructure.

**Landing Zone solves:**
- ❌ Bad Practice #1: Generic compartments (Landing Zone creates AI/ML workload compartments)
- ❌ Bad Practice #4: Poor segmentation (Landing Zone isolates GenAI endpoints in private subnets)
- ❌ Bad Practice #10: No monitoring (Landing Zone configures GenAI usage alarms)

**This skill provides**: GenAI cost optimization, rate limits, PHI/PII security, and troubleshooting for GenAI deployed WITHIN a Landing Zone.

---

## ⚠️ OCI CLI/API Knowledge Gap

**You don't know OCI CLI commands or OCI API structure.**

Your training data has limited and outdated knowledge of:
- OCI CLI syntax and parameters (updates monthly)
- OCI GenAI API endpoints and request/response formats
- GenAI service CLI operations (`oci generative-ai`)
- Available models, token limits, and pricing (changes frequently)
- Latest GenAI features (Agents, RAG) and API changes

**When OCI operations are needed:**
1. Use exact CLI commands from this skill's references
2. Do NOT guess OCI CLI syntax or parameters
3. Do NOT assume model availability or pricing
4. Load reference files for detailed GenAI API documentation

**What you DO know:**
- General LLM concepts and prompting patterns
- Token estimation and context management
- API integration patterns

This skill bridges the gap by providing current OCI GenAI-specific patterns and gotchas.

---

You are an OCI GenAI expert. This skill provides knowledge Claude lacks: cost optimization specifics, token management, rate limit handling, PHI/PII security, response validation, and model selection trade-offs.

## NEVER Do This

❌ **NEVER send PHI/PII identifiers to GenAI APIs (HIPAA/GDPR violation)**
```python
# WRONG - patient identifiers sent to external service
prompt = f"Transcribe note for patient {patient_name}, MRN {mrn}, SSN {ssn}: {note}"

# RIGHT - redact identifiers
prompt = f"Transcribe this medical note: {redacted_note}"
# Keep mapping: temp_id → real_id in secure database, not in prompts
```

**Why critical**: GenAI service logs may retain data, violates healthcare regulations

❌ **NEVER trust GenAI output without validation (hallucination risk)**
```python
# WRONG - use response directly in critical systems
diagnosis = genai_response.text
db.execute("UPDATE patients SET diagnosis = ?", diagnosis)

# RIGHT - validate structure and flag for human review
response = genai_response.text
if validate_medical_format(response):
    db.execute("UPDATE patients SET ai_suggested_diagnosis = ?, status = 'PENDING_REVIEW'", response)
```

**Hallucination rate**: 5-15% for factual queries, higher for medical/legal domains

❌ **NEVER ignore token limits**
- **command-r-plus**: 128k context window (input + output)
- **command-r**: 4k context (much cheaper but limited)
- **Exceeding limit**: Request truncated silently or fails with 400 error

❌ **NEVER call GenAI without rate limit handling**
```python
# WRONG - no retry logic, fails on rate limit
response = genai_client.chat(request)

# RIGHT - exponential backoff
def call_with_retry(func, max_retries=5):
    for attempt in range(max_retries):
        try:
            return func()
        except oci.exceptions.ServiceError as e:
            if e.status == 429 and attempt < max_retries - 1:
                wait = (2 ** attempt) + random.uniform(0, 1)
                logger.warning(f"Rate limited, retry in {wait:.2f}s")
                time.sleep(wait)
            else:
                raise
```

❌ **NEVER cache responses without consent (data privacy)**
- Caching saves costs BUT may violate privacy policies
- Get explicit user consent before caching medical/personal data
- Cache anonymized data only

❌ **NEVER use GenAI for deterministic tasks**
- Wrong: "Extract invoice total from OCR text" (use regex/structured parsing)
- Wrong: "Validate email format" (use validation library)
- Right: "Summarize patient history", "Generate report narrative" (creative tasks)

## Model Selection: Cost vs Performance

| Model | Context | Cost (per 1M tokens) | Best For | Avoid For |
|-------|---------|---------------------|----------|-----------|
| **command-r-plus** | 128k | ~$15 input, $75 output | Complex reasoning, long documents | Simple tasks (expensive) |
| **command-r** | 4k | ~$1.50 input, $7.50 output | Chat, short prompts, high volume | Long documents, RAG |
| **embed-english-v3** | N/A | ~$0.10 per 1M | Semantic search, clustering | Text generation |
| **llama-2-70b** | 4k | ~$2 input, $10 output | Open weights, cost-effective | Production (limited support) |

**Cost optimization strategy:**
1. **Use embeddings for search first** (1000x cheaper than generation)
2. **Cache responses** for repeated queries (with consent)
3. **Use command-r for simple tasks**, command-r-plus only when needed
4. **Truncate input** intelligently (keep relevant context only)

## Cost Calculation Examples

**Scenario: Medical transcription service**
- Average note: 500 tokens input, 300 tokens output = 800 tokens total
- 1000 notes/day = 800k tokens/day = 24M tokens/month

**Without optimization:**
```
Model: command-r-plus
Input:  12M × ($15/1M) = $180/month
Output: 12M × ($75/1M) = $900/month
Total: $1,080/month
```

**With optimization (30% cache hit, use command-r for simple notes):**
```
70% unique notes = 16.8M tokens
60% simple (command-r): 10M × $1.50 input + $7.50 output = $90
40% complex (command-r-plus): 6.72M × $15 input + $75 output = $605
Total: $695/month (36% savings)
```

## Token Management

### Token Limits by Model

| Model | Max Context | Max Output | Notes |
|-------|-------------|------------|-------|
| command-r-plus | 128k (input+output) | Varies | ~4 chars per token (rough) |
| command-r | 4k | 2k | Good for chat |
| embed-english-v3 | 512 | N/A | Embeddings only |

### Truncation Strategy

```python
def truncate_for_model(text: str, model: str = "command-r-plus", max_output: int = 2000):
    """Truncate input to fit token budget"""

    # Rough estimate: 1 token ≈ 4 characters
    if model == "command-r-plus":
        max_input_tokens = 128000 - max_output
    elif model == "command-r":
        max_input_tokens = 4000 - max_output
    else:
        max_input_tokens = 2000

    max_chars = max_input_tokens * 4

    if len(text) <= max_chars:
        return text

    # Keep most recent content (chronological data like logs, notes)
    logger.warning(f"Input exceeds {max_input_tokens} tokens, truncating")
    return "...[earlier content truncated]...\n" + text[-max_chars:]
```

### Prompt Optimization

**Inefficient** (wastes tokens):
```
Please carefully analyze the following medical record and provide a comprehensive
summary including all diagnoses, medications, allergies, and treatment plans. Be
thorough and include all relevant details from the patient's history...

[5000 word medical record]
```

**Optimized** (40% token reduction):
```
Summarize: diagnoses, meds, allergies, treatment plan.

[5000 word medical record]
```

**Token savings**: ~50 tokens on prompt × 1000 requests/day = 50k tokens/day saved = **$2.25/day** ($68/month)

## Rate Limits

### OCI GenAI Service Limits (per compartment):

| Model | Requests/Minute | Requests/Day | Tokens/Request |
|-------|-----------------|--------------|----------------|
| command-r-plus | 20 | 1000 | 128k |
| command-r | 60 | 3000 | 4k |
| Embeddings | 100 | 10000 | 512 |

### Error Handling

```python
import time
import random
from oci.exceptions import ServiceError

def generate_with_backoff(genai_client, request, max_retries=5):
    """Call GenAI with exponential backoff on rate limits"""

    for attempt in range(max_retries):
        try:
            response = genai_client.chat(request)
            return response.data.chat_response.text

        except ServiceError as e