fiftyone-model-evaluation

What this skill does

# Evaluate Model Predictions in FiftyOne

## Key Directives

**ALWAYS follow these rules:**

### 1. Check if dataset exists and has required fields
```python
list_datasets()
set_context(dataset_name="my-dataset")
dataset_summary(name="my-dataset")
```
Verify the dataset has both **prediction** and **ground truth** fields of compatible types.

### 2. Install evaluation plugin if not available
```python
list_plugins()
# If @voxel51/evaluation not listed:
download_plugin(url_or_repo="voxel51/fiftyone-plugins", plugin_names=["@voxel51/evaluation"])
enable_plugin(plugin_name="@voxel51/evaluation")
```

### 3. Ask user for evaluation parameters
Always confirm with the user:
- Prediction field name
- Ground truth field name
- Evaluation key (unique identifier for this evaluation)
- Evaluation method (coco, open-images, simple, top-k, binary)
- Whether to compute mAP (for detection tasks)

### 4. Launch App for evaluation operators
```python
launch_app(dataset_name="my-dataset")
```

### 5. Close app when done
```python
close_app()
```

## Workflow

### Step 1: Verify Dataset and Fields

```python
list_datasets()
set_context(dataset_name="my-dataset")
dataset_summary(name="my-dataset")
```

Review:
- Sample count
- Available label fields and their types
- Identify prediction field (model outputs)
- Identify ground truth field (annotations)

**Label Types and Compatible Evaluations:**

| Label Type | Evaluation Method | Supported Methods |
|------------|-------------------|-------------------|
| `Detections` | `evaluate_detections()` | coco, open-images |
| `Polylines` | `evaluate_detections()` | coco, open-images |
| `Keypoints` | `evaluate_detections()` | coco, open-images |
| `TemporalDetections` | `evaluate_detections()` | activitynet |
| `Classification` | `evaluate_classifications()` | simple, top-k, binary |
| `Segmentation` | `evaluate_segmentations()` | simple |
| `Regression` | `evaluate_regressions()` | simple |

### Step 2: Ensure Evaluation Plugin is Installed

```python
list_plugins()
```

If `@voxel51/evaluation` is not in the list:
```python
download_plugin(
    url_or_repo="voxel51/fiftyone-plugins",
    plugin_names=["@voxel51/evaluation"]
)
enable_plugin(plugin_name="@voxel51/evaluation")
```

### Step 3: Launch App

```python
launch_app(dataset_name="my-dataset")
```

### Step 4: Run Evaluation

Ask user for:
- Prediction field (`pred_field`)
- Ground truth field (`gt_field`)
- Evaluation key (`eval_key`) - must be unique identifier
- Evaluation method

```python
execute_operator(
    operator_uri="@voxel51/evaluation/evaluate_model",
    params={
        "pred_field": "predictions",
        "gt_field": "ground_truth",
        "eval_key": "eval",
        "method": "coco",
        "iou": 0.5,
        "compute_mAP": true
    }
)
```

### Step 5: View Results

After evaluation, the dataset will have new fields:
- `{eval_key}_tp` - True positive count per sample
- `{eval_key}_fp` - False positive count per sample
- `{eval_key}_fn` - False negative count per sample

**View only samples with false positives:**
```python
set_view(filters={"eval_fp": {"$gt": 0}})
```

**Use the Model Evaluation Panel in the App** to interactively explore:
- Summary metrics (mAP, precision, recall)
- Confusion matrices
- Per-class performance
- Scenario analysis

### Step 6: View Evaluation Patches (TP/FP/FN)

To examine individual true positives, false positives, and false negatives, guide users to the Python SDK:

```python
import fiftyone as fo

dataset = fo.load_dataset("my-dataset")

# Convert to evaluation patches view
eval_patches = dataset.to_evaluation_patches("eval")

# Count by type
print(eval_patches.count_values("type"))
# Output: {'fn': 246, 'fp': 4131, 'tp': 986}

# View only false positives
fp_view = eval_patches.match(F("type") == "fp")
session = fo.launch_app(view=fp_view)
```

### Step 7: Clean Up

```python
close_app()
```

## Evaluation Types

### Detection Evaluation

For `Detections`, `Polylines`, `Keypoints` labels.

**COCO-style (default):**
```python
execute_operator(
    operator_uri="@voxel51/evaluation/evaluate_model",
    params={
        "pred_field": "predictions",
        "gt_field": "ground_truth",
        "eval_key": "eval_coco",
        "method": "coco",
        "iou": 0.5,
        "classwise": true,
        "compute_mAP": true
    }
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `iou` | float | 0.5 | IoU threshold for matching |
| `classwise` | bool | true | Only match objects with same class |
| `compute_mAP` | bool | false | Compute mAP, mAR, and PR curves |
| `use_masks` | bool | false | Use instance masks for IoU (if available) |
| `iscrowd` | string | null | Attribute name for crowd annotations |
| `iou_threshs` | string | null | Comma-separated IoU thresholds for mAP |
| `max_preds` | int | null | Max predictions per sample for mAP |

**Open Images-style:**
```python
execute_operator(
    operator_uri="@voxel51/evaluation/evaluate_model",
    params={
        "pred_field": "predictions",
        "gt_field": "ground_truth",
        "eval_key": "eval_oi",
        "method": "open-images",
        "iou": 0.5
    }
)
```

Supports additional parameters:
- `pos_label_field`: Classifications specifying which classes should be evaluated
- `neg_label_field`: Classifications specifying which classes should NOT be evaluated

**ActivityNet-style (temporal):**

For `TemporalDetections` in video datasets:
```python
execute_operator(
    operator_uri="@voxel51/evaluation/evaluate_model",
    params={
        "pred_field": "predictions",
        "gt_field": "ground_truth",
        "eval_key": "eval_temporal",
        "method": "activitynet",
        "compute_mAP": true
    }
)
```

### Classification Evaluation

For `Classification` labels.

**Simple (default):**
```python
execute_operator(
    operator_uri="@voxel51/evaluation/evaluate_model",
    params={
        "pred_field": "predictions",
        "gt_field": "ground_truth",
        "eval_key": "eval_cls",
        "method": "simple"
    }
)
```

Per-sample field `{eval_key}` stores boolean indicating if prediction was correct.

**Top-k:**

Requires predictions with `logits` field:
```python
execute_operator(
    operator_uri="@voxel51/evaluation/evaluate_model",
    params={
        "pred_field": "predictions",
        "gt_field": "ground_truth",
        "eval_key": "eval_topk",
        "method": "top-k",
        "k": 5
    }
)
```

**Binary:**

For binary classifiers:
```python
execute_operator(
    operator_uri="@voxel51/evaluation/evaluate_model",
    params={
        "pred_field": "predictions",
        "gt_field": "ground_truth",
        "eval_key": "eval_binary",
        "method": "binary"
    }
)
```

Per-sample field `{eval_key}` stores: "tp", "fp", "tn", or "fn".

### Segmentation Evaluation

For `Segmentation` labels.

```python
execute_operator(
    operator_uri="@voxel51/evaluation/evaluate_model",
    params={
        "pred_field": "predictions",
        "gt_field": "ground_truth",
        "eval_key": "eval_seg",
        "method": "simple",
        "bandwidth": 5  # Optional: evaluate only boundary pixels
    }
)
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `bandwidth` | int | null | Pixels along contours to evaluate (null = entire mask) |
| `average` | string | "micro" | Averaging strategy: micro, macro, weighted, samples |

Per-sample fields:
- `{eval_key}_accuracy`
- `{eval_key}_precision`
- `{eval_key}_recall`

### Regression Evaluation

For `Regression` labels.

```python
execute_operator(
    operator_uri="@voxel51/evaluation/evaluate_model",
    params={
        "pred_field": "predictions",
        "gt_field": "ground_truth",
        "eval_key": "eval_reg",
        "method": "simple",
        "metric": "squared_error"  # or "absolute_error"
    }
)
```

Per-sample field `{eval_key}` stores the error value.

Metrics available:
- Mean Squared Error (MSE)
- Root