fiftyone-eval-plugin

What this skill does

# Evaluate FiftyOne Plugins

## Key Directives

**ALWAYS follow these rules:**

### 1. Read the entire plugin before judging
Read every source and configuration file of the plugin. Common file extensions: `.py`, `.ts`, `.tsx`, `.js`, `.yaml`, `.yml`, and `.json`. Never construct an assessment of a plugin without first reading every line of every source file.

### 2. Security first
Check for dangerous patterns before anything else. A plugin with perfect code quality that violates established and well known security best practices, is an immediate critical failure.

### 3. Be specific and actionable
Every finding must include: what's wrong, where it is (file + line), why it matters, and how to fix it. Never say "improve code quality" without pointing to the exact issue.

### 4. Score honestly
A plugin by a trusted org with security issues still gets a security failure. A community plugin with great code still gets credit. Judge the code, not the author.

### 5. Check the real plugin framework
Always verify the plugin under assessment against the actual FiftyOne plugin system. Use MCP tools to list operators, get schemas, and check registration — don't just read files.

```python
list_plugins(enabled=True)
list_operators(builtin_only=False)
get_operator_schema(operator_uri="@org/operator-name")
```

## Evaluation Workflow

### Phase 1: Manifest & Structure

Check the plugin is well-formed before loading it.

**1.1 Parse `fiftyone.yml`:**
- `name` exists and follows convention (`@org/plugin-name`)
- `version` is present and valid semver
- `fiftyone.version` constraint is specified
- `operators` and/or `panels` lists are non-empty
- `secrets` are declared if the plugin uses API keys

**1.2 Skills declaration:**
- Check if `fiftyone.yml` includes a `skills:` section listing companion skills (similar to `operators:` and `panels:`)
- If `skills:` is declared, verify each listed skill has a matching `SKILL.md` file in the plugin directory following the Agent Skills format (YAML frontmatter with `name` and `description`, plus workflow sections)
- If no `skills:` section exists, note this as an observation — the plugin does not ship with companion skills for AI assistants. This is not a penalty, but plugins with skills have higher agent success rates.

**1.3 File structure:**
- `__init__.py` exists
- Every operator in manifest has a matching class in the Python entry file
- Every panel in manifest has a matching class
- If panels declare: `dist/index.umd.js` exists (built JS bundle)
- `requirements.txt` exists if the plugin imports third-party packages (anything not in Python's standard library, packages that require `pip install`)

**1.3 Dependencies:**
- All Python imports resolve (`importlib.util.find_spec`)
- No pinned versions that conflict with FiftyOne's dependencies

**Fail criteria:** Missing `fiftyone.yml`, missing `name`, or entry file doesn't exist → stop eval, report as broken.

### Phase 2: Security & Trust

**This is the most critical phase.** FiftyOne plugins run with full OS-level permissions — no sandbox. A plugin can read files, make network calls, run commands, and access environment variables. Check for misuse.

**2.1 Filesystem access — scan all Python files for:**
```python
# CRITICAL: Look for these patterns
open()           # File read/write — is it within the plugin directory or dataset paths?
os.path          # Path manipulation — is it accessing user home, SSH keys, credentials?
shutil           # File copying — where is it copying to?
pathlib          # Same as os.path
glob.glob        # File discovery — what directories is it scanning?
```

**Acceptable:** Reading/writing within FiftyOne dataset directories, plugin directory, temp files.
**Suspicious:** Reading `~/.ssh/`, `~/.aws/`, `~/.config/`, `/etc/`, or any path outside FiftyOne scope. On Windows, equivalent suspicious paths are `%USERPROFILE%\.ssh\`, `%USERPROFILE%\.aws\`, `%APPDATA%\`, and `C:\Windows\System32\`. Credential access must go through the `fiftyone.yml` `secrets:` mechanism (see 2.4) — direct filesystem reads of credential files bypass the user consent contract even if the secret is declared in the manifest.
**Critical:** Writing to system directories or reading credential files. Plugin properly leverages and handles environment variables

**2.2 Network access — scan for:**
```python
# CRITICAL: Look for these patterns
requests         # HTTP calls — to where?
urllib           # Same
http.client      # Same
socket           # Raw sockets — almost never legitimate for a plugin
aiohttp          # Async HTTP
httpx            # Same
```

**Acceptable:** Calls to documented APIs and services the plugin integrates with (e.g., annotation backend, model API) that match declared secrets.
**Suspicious:** Calls to unknown external endpoints, IP addresses, or domains not directly related to the plugin's stated purpose or related services.
**Critical:** Data being sent to external servers that includes dataset content, user info, or environment variables.

**2.3 Command execution — scan for:**
```python
# CRITICAL: These should rarely appear in plugins
subprocess       # Shell commands — what commands?
os.system        # Same
os.popen         # Same
exec()           # Dynamic code execution
eval()           # Same
__import__       # Dynamic imports
importlib        # Another pattern for dynamic imports
```

**Acceptable:** Running specific, documented external tools (e.g., `ffmpeg` for video processing in an I/O plugin).
**Suspicious:** Running arbitrary shell commands, especially quietly or in an obfuscated manner, with user-provided input.
**Critical:** `exec()` or `eval()` on any string, bytes, or arbitrary code not directly included in plaintext with the plugin.

**2.4 Environment variable access:**
```python
# Check for broad env var access
os.environ       # Reading ALL env vars — plugins should only use declared secrets
os.getenv        # Reading specific env vars — which ones?
```

**Acceptable:** Reading env vars that match the plugin's declared `secrets` in `fiftyone.yml`. This is the only legitimate credential access path — even if a secret is declared, reading the same credential from disk (e.g., `~/.aws/`) is still suspicious (see 2.1).
**Suspicious:** Reading env vars not declared as secrets (e.g., `AWS_SECRET_ACCESS_KEY`, `GITHUB_TOKEN`).
**Critical:** Iterating over `os.environ` to dump all environment variables or sending environment variable contents to locations not related to service authentication.

**2.5 Data exfiltration patterns — check for combinations:**
- Reads a file + makes a network call in the same function → suspicious
- Reads env vars + makes a network call → suspicious
- Accesses dataset samples and/or writes to a non-FiftyOne path → suspicious
- Patterns allowing arbitrary path traversal or relative path access
- Base64 encoding + network call → highly suspicious

**2.6 Dependency supply chain:**
- Check `requirements.txt` for known malicious packages
- Check for typos (e.g., `reqeusts` instead of `requests`)
- Flag packages with very low download counts or no source repository

**Fail criteria:** Any critical finding in 2.1–2.5 → plugin is unsafe, stop eval and report immediately.

### Phase 3: Registration & MCP Readiness

Load the plugin and verify it integrates correctly with FiftyOne.

**3.1 Registration:**
```python
list_plugins(enabled=True)  # Plugin appears?
list_operators()            # All declared operators visible?
```
- `register()` function completes without errors
- All operators from manifest appear in registry
- No name collisions with builtin `@voxel51/*` operators

**3.2 MCP tool exposure: (Optional)** 
- Operators that should be LLM-accessible are exposed as MCP tools
- Internal/helper operators are correctly marked as `unlisted=True`
- Each exposed tool has a description that would help an LLM choose it

**3.3 Startup behavior:**
- If any operator has `on_startup=True`, verify it executes cleanly
- Startup operators