fiftyone-generate-data-lens-connector

What this skill does

# Generate Data Lens Connector

Generate a fully functional `DataLensOperator` plugin from a user-provided
database
schema. The generated connector lets users browse, preview, and import samples
from
their external data source directly through the FiftyOne App's Data Lens panel.

## Enterprise Notice

Data Lens is a **FiftyOne Enterprise** feature. Before proceeding, inform
the user:

> **Note:** Data Lens is an enterprise-only feature available in
> [FiftyOne Enterprise](https://docs.voxel51.com/enterprise/index.html). The
> connector generated by this skill requires a FiftyOne Enterprise deployment
> to run. If you're using the open-source version of FiftyOne, this connector
> will not work in your environment.
>
> Would you like to proceed?

Wait for the user to confirm before continuing. If they ask about alternatives
for OSS, suggest they look into custom operators for similar data-import
workflows, or the standard
[dataset import](https://docs.voxel51.com/user_guide/dataset_creation/index.html)
utilities.

## Key Directives

**ALWAYS follow these rules:**

### 1. Schema first, code second

Never generate connector code until you fully understand the source schema. Ask
clarifying questions about anything ambiguous — column semantics, coordinate
systems,
filepath conventions, image dimensions.

### 2. Propose the field mapping before generating

Present a clear table mapping source columns to FiftyOne field types. Get user
approval before writing any code. This is the most important step — a connector
that maps fields wrong is worse than no connector.

### 3. Use parameterized queries

Never interpolate user input directly into SQL strings. Use the database
driver's
parameterized query mechanism (e.g., `%s` for psycopg, `@param` for BigQuery,
`:param` for Databricks).

### 4. Respect the batching contract

Always yield `DataLensSearchResponse` objects in batches of
`request.batch_size`.
Buffer results and yield when the buffer is full, plus a final yield for
remaining
samples.

### 5. Keep it minimal

Generate only what's needed. Don't add vector search, text search, or advanced
features unless the user's schema and requirements call for them. A simple
metadata
filter connector is the right default.

## Workflow

### Phase 1: Understand the Schema

Gather the information needed to generate the connector:

1. **Get the schema.** Accept any of these formats:
    - DDL (`CREATE TABLE` statements)
    - Column list with types (JSON, markdown table, plain text)
    - A request to introspect a live database (guide the user to export the
      schema)

2. **Identify key columns.** Ask about any that aren't obvious:
    - **Filepath column** — which column contains the media path? Is it
      absolute,
      relative (needs a prefix), or a cloud URI?
    - **Label columns** — which columns contain annotations? What format?
      (JSON blobs, foreign key joins, flat columns)
    - **Metadata columns** — which columns should become filterable properties?
    - **Coordinate system** — if bounding boxes exist: pixel-absolute or
      normalized?
      What are the image dimensions (fixed or per-sample)?

3. **Identify the database type.** This determines:
    - Which Python driver to use
    - Query syntax (parameterization style, JSON functions, etc.)
    - Connection string format and required secrets

4. **Get sample data** (if available). Even 2-3 example rows dramatically
   improve
   the quality of the field mapping. Ask for them.

### Phase 2: Propose Field Mapping

Present a mapping table for user approval:

```
| Source Column     | FiftyOne Field        | Type                | Notes                          |
|-------------------|-----------------------|---------------------|--------------------------------|
| filepath          | filepath              | str                 | Prefix: gs://bucket/path/      |
| weather           | weather               | Classification      | Filterable enum                |
| bbox_x1/y1/x2/y2 | detections            | Detections          | Pixel coords, normalize by WxH |
| category          | detections[].label    | str                 | Detection label                |
| ...               | ...                   | ...                 | ...                            |
```

Include:

- **Filepath construction** — how the full path is built from the column value
- **Coordinate normalization** — formula if bounding boxes are in pixel
  coordinates
- **Label hierarchy** — how nested/joined label data maps to FiftyOne label
  types
- **Filters** — which columns become `resolve_input` enum/text fields, with
  known values if available

**Wait for user approval before proceeding.**

### Phase 3: Generate Connector

Generate a complete plugin directory with these files:

| File               | Purpose                                           |
|--------------------|---------------------------------------------------|
| `__init__.py`      | Operator class + handler class + sample transform |
| `fiftyone.yml`     | Plugin manifest with operator name and secrets    |
| `requirements.txt` | Python driver dependency                          |

Use the patterns from [CONNECTOR-TEMPLATE.md](CONNECTOR-TEMPLATE.md) as your
structural guide. The generated code should follow these architectural layers:

**Layer 1 — Operator class** (`DataLensOperator` subclass):

- `config` property with `execute_as_generator=True`, `unlisted=True`
- `resolve_input()` defining the UI filter form
- `handle_lens_search_request()` delegating to the handler

**Layer 2 — Handler class** (connection + query logic):

- Context manager for connection lifecycle
- `iter_batches()` implementing the batching loop
- `_generate_query()` building parameterized SQL from `search_params`
- `_transform_sample()` mapping raw rows to `fo.Sample(...).to_dict()`

**Layer 3 — Data models** (optional, for complex schemas):

- `@dataclass` for query parameters (validated from `search_params`)
- `@dataclass` for query result rows (typed column access)

See [FIELD-MAPPING-GUIDE.md](FIELD-MAPPING-GUIDE.md) for the rules on mapping
database types to FiftyOne field types, especially for spatial data (bounding
boxes,
keypoints, segmentation masks).

### Phase 4: Validate

After generating the connector:

1. **Syntax check** — the generated code must parse without errors:
   ```bash
   python -c "import ast; ast.parse(open('__init__.py').read()); print('OK')"
   ```

2. **Sample construction check** — if sample data was provided, construct
   `fo.Sample` objects from it and verify they serialize correctly:
   ```python
   import fiftyone as fo
   sample = fo.Sample(
       filepath="...",
       # ... mapped fields
   )
   sample.to_dict()  # Must not raise
   ```

3. **Walk through the generated code** with the user:
    - Confirm the query logic matches their schema
    - Confirm the transform logic produces correct FiftyOne samples
    - Confirm the `resolve_input` filters are right

4. **Installation guidance:**
   ```bash
   # Copy plugin to FiftyOne plugins directory
   PLUGINS_DIR=$(python -c "import fiftyone as fo; print(fo.config.plugins_dir)")
   cp -r ./my-connector "$PLUGINS_DIR/"

   # Set required secrets
   export MY_SECRET_KEY="..."

   # Register in Data Lens UI: Data sources > Add > plugin-name/operator_name
   ```

### Phase 5: Iterate

The first generation likely needs refinement. Common adjustments:

- Adding/removing filter fields
- Fixing coordinate normalization
- Adjusting the filepath prefix
- Handling NULL/missing values in optional columns
- Adding conditional WHERE clauses for "all" filter values

## Database-Specific Patterns

### PostgreSQL

```python
# Driver: psycopg[binary]
# Parameterization: %s positional
# Connection: ctx.secret("POSTGRES_CONNECTION_STRING")
# JSON: JSON_AGG, JSON_BUILD_OBJECT
# Streaming: cursor.stream(query, params, size=batch_size)
import psycopg
from psycopg.rows import dict_row
```

### BigQuery

```python
# Driver: go