app-observability

What this skill does

# Grafana Cloud Application Observability Skill

## Overview

Grafana Cloud provides three tightly related application monitoring products:

1. **Application Observability (APM)** - RED metrics from OTel traces, service inventory, service maps
2. **Frontend Observability** - RUM/Faro SDK for browser apps, session replay, web vitals
3. **AI Observability** - LLM/model monitoring via OpenLIT + OTel, token/cost/latency metrics

All three integrate with Grafana Tempo (traces), Loki (logs), and Pyroscope (profiles) for full-stack correlation.

---

## Application Observability (APM)

### What It Is

Application Observability is a pre-built APM experience in Grafana Cloud built on top of OpenTelemetry. It generates RED (Rate, Error, Duration) metrics from distributed traces via span metrics, then surfaces them in:

- **Service Inventory** - table of all services with RED metrics at a glance
- **Service Overview** - per-service RED metrics, top operations, error breakdown
- **Service Map** - node graph of service dependencies with flow visualization
- **Operations view** - per-endpoint RED metrics with p50/p95/p99 latency

### How Metrics Are Generated

Application Observability does NOT rely on traditional Prometheus scraping. Metrics come from **span metrics** - aggregations computed from OTel trace data:

- Source: OTel traces sent to Grafana Tempo or Grafana Alloy
- Generation method: Tempo's metrics-generator OR the `spanmetrics` connector in Alloy/OTel Collector
- Result: Prometheus-compatible metrics stored in Grafana Mimir

Key generated metric names:
- Via Tempo metrics-generator: `traces_spanmetrics_calls_total`, `traces_spanmetrics_duration_seconds`
- Via OTel Collector spanmetrics connector: `traces_span_metrics_calls_total`, `traces_span_metrics_duration_seconds`

### Required OTel Resource Attributes

These attributes MUST be present on all spans for Application Observability to work:

| Attribute | Grafana Label | Purpose |
|---|---|---|
| `service.name` | `service_name` / part of `job` | Identifies the service |
| `service.namespace` | part of `job` label | Groups services; `job = namespace/service.name` |
| `deployment.environment` | `deployment_environment` | Env filter (prod/dev/staging) |

The `job` label is constructed as:
- `service.namespace/service.name` when namespace is set
- `service.name` alone when no namespace

Additional recommended attributes:
- `service.version` - shown in service overview
- `k8s.cluster.name` - for K8s environments
- `k8s.namespace.name` - Kubernetes namespace
- `cloud.region` - for multi-region setups

### Setting Environment Variables for OTel SDK

```bash
export OTEL_SERVICE_NAME="my-api"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=myteam,deployment.environment=production,service.version=1.2.3"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
```

### Grafana Alloy Configuration (River syntax)

Alloy acts as a local OTel Collector and forwards data to Grafana Cloud:

```river
// Receive traces, metrics, logs from instrumented apps
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }
  output {
    metrics = [otelcol.processor.resourcedetection.default.input]
    logs    = [otelcol.processor.resourcedetection.default.input]
    traces  = [otelcol.processor.resourcedetection.default.input]
  }
}

// Auto-detect host/cloud metadata
otelcol.processor.resourcedetection "default" {
  detectors = ["env", "system", "gcp", "aws", "azure"]
  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

// Batch for efficiency
otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.otlphttp.grafana_cloud.input]
    logs    = [otelcol.exporter.otlphttp.grafana_cloud.input]
    traces  = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

// Auth
otelcol.auth.basic "grafana_cloud" {
  username = env("GRAFANA_CLOUD_INSTANCE_ID")
  password = env("GRAFANA_CLOUD_API_KEY")
}

// Export to Grafana Cloud OTLP endpoint
otelcol.exporter.otlphttp "grafana_cloud" {
  client {
    endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")
    auth     = otelcol.auth.basic.grafana_cloud.handler
  }
}
```

Required environment variables for Alloy:
```bash
GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-<region>.grafana.net/otlp
GRAFANA_CLOUD_INSTANCE_ID=<your-instance-id>
GRAFANA_CLOUD_API_KEY=<your-api-key>
```

### Service Map

The Service Map uses Tempo's **metrics-generator** to produce service graph metrics:
- Node graph shows services as nodes, HTTP/gRPC calls as edges
- Edge thickness indicates request rate; color indicates error rate
- Clicking a node navigates to Service Overview
- Requires `span.kind` (CLIENT/SERVER) on spans for directional edges

Enable in Tempo (managed by Grafana Cloud automatically):
- `service-graphs` metrics generator enabled by default in Grafana Cloud Tempo
- Uses `traces_service_graph_request_total`, `traces_service_graph_request_failed_total` metrics

### Integration with Traces, Logs, Profiles

Application Observability provides one-click correlation:
- **Traces**: Click any metric spike to open exemplar traces in Grafana Tempo
- **Logs**: Service logs shown in Service Overview; correlated via `service.name` label
- **Profiles**: "Go to profiles" button in Service Overview when Pyroscope is configured
- **Frontend**: Link from Application Observability to Frontend Observability for the same service

---

## Frontend Observability (Faro)

### What It Is

Grafana Faro is an open-source JavaScript/TypeScript SDK for **Real User Monitoring (RUM)**. It instruments browser applications to capture:

- **Web vitals**: Core Web Vitals (LCP, CLS, INP) and additional performance metrics
- **Errors**: Unhandled exceptions, rejected promises with stack traces
- **Sessions**: User journeys, page views, navigation timing
- **Logs**: Custom log messages from frontend code
- **Traces**: Distributed traces via OpenTelemetry-JS (correlates with backend spans)
- **Session replay**: Rrweb-based DOM recording for reproducing user issues

Data flows: Faro SDK -> Grafana Alloy (faro receiver) OR Grafana Cloud OTLP endpoint -> Loki (logs) + Tempo (traces) + Mimir (metrics)

### Faro SDK Packages

```
@grafana/faro-core          # Core SDK - signals, transports, API
@grafana/faro-web-sdk       # Web instrumentations + transports
@grafana/faro-web-tracing   # OpenTelemetry-JS distributed tracing
@grafana/faro-react         # React-specific integrations (error boundary, router)
```

### Basic JavaScript Setup (npm)

```bash
npm install @grafana/faro-web-sdk
# or
yarn add @grafana/faro-web-sdk
```

```javascript
import {
  initializeFaro,
  getWebInstrumentations,
} from '@grafana/faro-web-sdk';

const faro = initializeFaro({
  url: 'https://faro-collector-prod-<region>.grafana.net/collect/<app-key>',
  app: {
    name: 'my-frontend-app',
    version: '1.0.0',
    environment: 'production',
  },
  instrumentations: [
    ...getWebInstrumentations({
      captureConsole: true,
    }),
  ],
});

// Manual API usage
faro.api.pushLog(['User clicked checkout button']);
faro.api.pushError(new Error('Payment failed'));
faro.api.pushEvent('button_click', { button: 'checkout' });
```

### CDN Setup (no bundler)

```html
<script src="https://unpkg.com/@grafana/faro-web-sdk@latest/dist/library/faro-web-sdk.iife.js"></script>
<script>
  const { initializeFaro, getWebInstrumentations } = GrafanaFaroWebSdk;

  initializeFaro({
    url: 'https://faro-collector-prod-<region>.grafana.net/collect/<app-key>',
    app: { name: 'my-app', version: '1.0.0' },
    instrumentations: [...getWebInstrumentations()],
  });
</script>
```

### React Setup with Tracing

```bash
npm install @grafana/faro-react @grafana/faro-web-tracing
```

```javascript
import { initializeFaro, getWebInstrumentati