future-agi-platform

What this skill does

# Future AGI Platform

> Skill by [ara.so](https://ara.so) — Daily 2026 Skills collection.

Future AGI is an open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. It provides tracing (OpenTelemetry-native), 50+ evaluation metrics, multi-turn simulations, guardrails/protect, an OpenAI-compatible gateway, and prompt optimization — all in one self-hostable platform with a closed feedback loop.

---

## Installation

### Python SDK

```bash
pip install ai-evaluation
# For instrumentation/tracing:
pip install fi-instrumentation
# Framework-specific instrumentors:
pip install traceai-openai
pip install traceai-langchain
pip install traceai-llamaindex
pip install traceai-crewai
```

### TypeScript/Node SDK

```bash
npm install @traceai/fi-core
npm install @traceai/openai
```

### Self-Host via Docker Compose

```bash
git clone https://github.com/future-agi/future-agi.git
cd future-agi
cp futureagi/.env.example futureagi/.env
# Edit .env with your API keys and config
docker compose up -d
# Access at http://localhost:3031
```

### Self-Host via Kubernetes

```bash
# Plain manifests available in deploy/
kubectl apply -f deploy/

# Helm chart (in progress)
helm repo add futureagi https://charts.futureagi.com
helm install fagi futureagi/future-agi
```

---

## Configuration

### Environment Variables

```bash
# .env for self-hosted deployment
FI_API_KEY=your_api_key_here           # Future AGI API key
FI_BASE_URL=http://localhost:3031       # Self-hosted URL (or https://api.futureagi.com for cloud)

# For Cloud usage
FI_API_KEY=$FI_API_KEY                 # From app.futureagi.com
FI_BASE_URL=https://api.futureagi.com

# Database (self-host)
POSTGRES_URL=$POSTGRES_URL
CLICKHOUSE_URL=$CLICKHOUSE_URL
REDIS_URL=$REDIS_URL
RABBITMQ_URL=$RABBITMQ_URL
```

### SDK Configuration in Code

```python
import os
from fi_instrumentation import register

# Register project — reads FI_API_KEY and FI_BASE_URL from env
tracer_provider = register(
    project_name="my-agent",
    project_type="AGENT",  # or "LLM", "PIPELINE"
    # Explicit config (override env vars):
    # fi_api_key=os.environ["FI_API_KEY"],
    # fi_base_url=os.environ["FI_BASE_URL"],
)
```

---

## Core Feature 1: Tracing / Observability

### Python — OpenAI Instrumentation

```python
from fi_instrumentation import register
from traceai_openai import OpenAIInstrumentor
from openai import OpenAI

# Register once at app startup
register(project_name="my-agent")
OpenAIInstrumentor().instrument()

client = OpenAI()  # api_key from OPENAI_API_KEY env var

# All subsequent calls are automatically traced
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(response.choices[0].message.content)
```

### Python — LangChain Instrumentation

```python
from fi_instrumentation import register
from traceai_langchain import LangChainInstrumentor
from langchain_openai import ChatOpenAI
from langchain.schema import HumanMessage

register(project_name="langchain-agent")
LangChainInstrumentor().instrument()

llm = ChatOpenAI(model="gpt-4o")
response = llm.invoke([HumanMessage(content="Explain quantum computing")])
print(response.content)
```

### Python — LlamaIndex Instrumentation

```python
from fi_instrumentation import register
from traceai_llamaindex import LlamaIndexInstrumentor
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

register(project_name="llamaindex-rag")
LlamaIndexInstrumentor().instrument()

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)
```

### Python — Manual Span Creation

```python
from fi_instrumentation import register
from opentelemetry import trace

register(project_name="custom-agent")
tracer = trace.get_tracer(__name__)

def process_user_query(query: str) -> str:
    with tracer.start_as_current_span("process_query") as span:
        span.set_attribute("query", query)
        span.set_attribute("model", "gpt-4o")
        
        # Your LLM call here
        result = call_llm(query)
        
        span.set_attribute("response_length", len(result))
        return result
```

### TypeScript — OpenAI Instrumentation

```typescript
import { register } from "@traceai/fi-core";
import { OpenAIInstrumentation } from "@traceai/openai";
import OpenAI from "openai";

// Register at app startup
register({
  projectName: "my-ts-agent",
  // fiApiKey: process.env.FI_API_KEY,  // auto-read from env
  // fiBaseUrl: process.env.FI_BASE_URL,
});
new OpenAIInstrumentation().instrument();

const client = new OpenAI(); // OPENAI_API_KEY from env

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```

---

## Core Feature 2: Evaluations

### Basic Evaluation

```python
from fi.evals import evaluate
from fi.evals.metrics import Hallucination, Groundedness, ResponseRelevance

# Single evaluation
result = evaluate(
    metrics=[Hallucination()],
    query="What is the capital of France?",
    response="The capital of France is Berlin.",
    context="France is a country in Western Europe. Its capital city is Paris.",
)
print(result)  # {"hallucination": {"score": 1.0, "label": "hallucinated"}}
```

### Multiple Metrics at Once

```python
from fi.evals import evaluate
from fi.evals.metrics import (
    Hallucination,
    Groundedness,
    ResponseRelevance,
    ToneCheck,
    PIICheck,
    ToolCallAccuracy,
)

result = evaluate(
    metrics=[
        Hallucination(),
        Groundedness(),
        ResponseRelevance(),
        ToneCheck(expected_tone="professional"),
        PIICheck(),
    ],
    query="Explain the benefits of exercise.",
    response="Exercise reduces the risk of heart disease and improves mental health.",
    context="Regular physical activity has numerous health benefits including cardiovascular health improvement.",
)

for metric_name, metric_result in result.items():
    print(f"{metric_name}: {metric_result['score']} — {metric_result.get('label', '')}")
```

### Batch Evaluation on a Dataset

```python
from fi.evals import batch_evaluate
from fi.evals.metrics import Hallucination, Groundedness

dataset = [
    {
        "query": "What year was Python created?",
        "response": "Python was created in 1991.",
        "context": "Python is a programming language created by Guido van Rossum. It was first released in 1991.",
    },
    {
        "query": "Who wrote Hamlet?",
        "response": "Hamlet was written by Charles Dickens.",
        "context": "Hamlet is a tragedy written by William Shakespeare, believed to have been written around 1600.",
    },
]

results = batch_evaluate(
    metrics=[Hallucination(), Groundedness()],
    data=dataset,
    project_name="batch-eval-demo",
)

for i, result in enumerate(results):
    print(f"Item {i}: {result}")
```

### Custom Rubric / LLM-as-Judge

```python
from fi.evals import evaluate
from fi.evals.metrics import CustomRubric

result = evaluate(
    metrics=[
        CustomRubric(
            criteria="Does the response correctly answer the question without making up facts?",
            rubric={
                1: "Response is completely correct and factual",
                0: "Response contains fabricated or incorrect information",
            },
        )
    ],
    query="What is 2 + 2?",
    response="2 + 2 equals 4.",
)
print(result)
```

### Evaluation with Tool Calls

```python
from fi.evals import evaluate
from fi.evals.metrics import ToolCallAccuracy

result = evaluate(
    metrics=[ToolCallAccuracy()],
    query="What's the weather in New York?",
    response="The weather in New York is 72°F and sunny.",
    expected_tool_calls=[
        {"name": "get_weather"