ai-photos

What this skill does

# ai-photos

ai-photos turns one or more local photo sources into a searchable AI photo album for OpenClaw.

Supported formats:
- macOS: `jpg`, `jpeg`, `png`, `webp`, `heic`
- Linux: `jpg`, `jpeg`, `png`, `webp`
- Linux `heic`: best-effort only; do not promise captioning or preview support

When talking to users:
- try to match the user's language
- explain the outcome simply: choose local folders now, then use OpenClaw to search and organize them
- stay focused on the current ai-photos request
- keep user-facing replies short and product-level: progress, readiness, and what the user can do next
- keep implementation details internal unless the user asks or troubleshooting requires them
- once indexing is complete and the backend is confirmed ready, say the album is ready and invite the user to try a search
- when the user asks what ai-photos can do, or when handing off a ready album, briefly describe the product in user terms:
  - natural-language search across captions, scene labels, and tags
  - date-based browsing and filtering
  - a local web gallery for thumbnail browsing and large-photo viewing
  - photo detail view with caption, scene, tags, capture time, device, location, orientation, and file info when available
  - opening the original local file from the web UI
  - manual sync now, optional automatic indexing later
- when introducing the web UI, describe it as a local searchable gallery rather than an API or server unless implementation details are needed
- keep these capability descriptions short, concrete, and user-facing; do not drift into backend details

Suggested user-facing capability summary:

- "You can search your photos in plain language, filter by date, and browse everything in a local gallery."
- "The web UI shows thumbnails, opens large previews, and lets you inspect captions, tags, time, device, location, and other file details when available."
- "You can also open the original local file directly, and later either sync changes manually or turn on automatic indexing."

## Required outcome

This task is not complete until all of the following are true:

1. at least one photo source is chosen and readable for a new album
2. image analysis is verified to work in the current OpenClaw runtime
3. the album backend is created or reconnected and writable
4. the first import succeeds, or an existing album is verified reachable
5. the user explicitly approved automatic indexing or explicitly declined it
6. if automatic indexing was approved, OpenClaw heartbeat is configured without breaking existing heartbeat tasks, the ai-photos block is present in `HEARTBEAT.md`, and one verification heartbeat has run
7. the user has been told the album is ready and has been invited to try a search
8. the user has been sent the final handoff

## Internal terms

Use these terms for agent reasoning, troubleshooting, or recovery only.
Do not introduce them to the user unless needed.

- `photo sources`: one or more local paths scanned into the same album
- `album backend`: where the searchable photo index is stored
- `album profile`: saved reconnect information, stored automatically under `~/.openclaw/ai-photos/albums/default.json`
- `caption input JSONL`: the manifest file that still needs vision captions and import

If the user asks what to save for later, explain that OpenClaw saves the reconnect information automatically at `~/.openclaw/ai-photos/albums/default.json`, and that they only need to keep that file if they want a manual backup.

## Caption schema

Each captioned JSONL line should contain the original manifest fields plus vision-model output.

Required base fields:
- `file_path`
- `filename`
- `sha256`
- `mime_type`
- `size_bytes`
- `width`
- `height`
- `taken_at`
- `exif`

Vision fields:
- `caption`: one short factual sentence
- `tags`: array of 5-12 short tags
- `scene`: short scene label
- `objects`: array of the main visible objects
- `text_in_image`: visible text or `null`

Optional fields:
- `metadata`: free-form JSON object
- `search_text`: concatenated retrieval text; if omitted, the importer builds it

Example:

```json
{
  "file_path": "/photos/2026/03/cat.jpg",
  "filename": "cat.jpg",
  "sha256": "abc123",
  "mime_type": "image/jpeg",
  "size_bytes": 231231,
  "width": 3024,
  "height": 4032,
  "taken_at": "2026-03-12T09:12:00+00:00",
  "exif": {"Make": "Apple", "Model": "iPhone 15 Pro"},
  "caption": "A white cat resting on a gray sofa near a sunlit window.",
  "tags": ["cat", "sofa", "indoor", "sunlight", "pet"],
  "scene": "living room",
  "objects": ["cat", "sofa", "window"],
  "text_in_image": null,
  "metadata": {"source": "demo"}
}
```

## CLI runtime

This skill does not depend on a local Python environment or a checked-out Go source tree.
It uses the latest published `ai-photos` CLI release from:

- repository: `https://github.com/zoubingwu/openclaw-ai-photos`
- install dir: `~/.openclaw/ai-photos/bin`
- binary path: `~/.openclaw/ai-photos/bin/ai-photos`

At the start of every ai-photos task, run the bootstrap flow exactly once and reuse the resulting binary path for the rest of the task.

### Bootstrap flow

Run this shell block and capture its stdout as `AI_PHOTOS_BIN`:

```bash
ensure_ai_photos() {
  AI_PHOTOS_REPO="zoubingwu/openclaw-ai-photos"
  AI_PHOTOS_BIN_DIR="$HOME/.openclaw/ai-photos/bin"
  AI_PHOTOS_BIN="$AI_PHOTOS_BIN_DIR/ai-photos"

  mkdir -p "$AI_PHOTOS_BIN_DIR"

  os="$(uname -s | tr '[:upper:]' '[:lower:]')"
  case "$os" in
    darwin) goos="darwin" ;;
    linux) goos="linux" ;;
    *)
      echo "unsupported platform: $os" >&2
      return 1
      ;;
  esac

  arch="$(uname -m)"
  case "$arch" in
    x86_64|amd64) goarch="amd64" ;;
    arm64|aarch64) goarch="arm64" ;;
    *)
      echo "unsupported architecture: $arch" >&2
      return 1
      ;;
  esac

  archive_name="ai-photos_${goos}_${goarch}.tar.gz"
  archive_url="https://github.com/${AI_PHOTOS_REPO}/releases/latest/download/${archive_name}"
  tmp_dir="$(mktemp -d)"
  had_existing_binary=0
  if [ -x "$AI_PHOTOS_BIN" ]; then
    had_existing_binary=1
  fi

  if curl -fL "${archive_url}" -o "$tmp_dir/${archive_name}" \
    && tar -xzf "$tmp_dir/${archive_name}" -C "$tmp_dir" \
    && install -m 0755 "$tmp_dir/ai-photos" "$AI_PHOTOS_BIN"; then
    rm -rf "$tmp_dir"
    printf '%s\n' "$AI_PHOTOS_BIN"
    return 0
  fi

  rm -rf "$tmp_dir"
  if [ "$had_existing_binary" -eq 1 ]; then
    printf '%s\n' "$AI_PHOTOS_BIN"
    return 0
  fi

  echo "could not download ai-photos release archive" >&2
  return 1
}

AI_PHOTOS_BIN="$(ensure_ai_photos)"
```

Rules:
- always run the bootstrap flow before using the CLI
- the bootstrap flow downloads the latest stable release asset from `releases/latest/download/...` and does not call `api.github.com`
- if the latest asset download or unpack step fails, continue with the cached binary when one already exists
- if the latest asset download fails and no cached binary exists, setup is blocked
- do not tell the user to clone the repository or build the CLI locally unless troubleshooting requires it
- if you need command details, use `"$AI_PHOTOS_BIN" help` or `"$AI_PHOTOS_BIN" help <subcommand>`

## Onboarding

### Step 0 - Choose mode

User-facing:

- Ask whether the user wants to create a new photo album, reconnect an existing one, or search an already configured album.
- If they want to reconnect, explain that you will try the saved connection first and only ask for more details if needed.

`[AGENT]` Branching:

- `1`: continue to Step 1
- `2`: continue to Step 3 and Step 4
- `3`: go directly to Search flow
- if the user wants search but no configured album exists, tell them setup is required first

### Step 1 - Ask for photo folders

User-facing:

- Ask for one or more local folder paths that contain photos.

`[AGENT]`

Do not continue until the user has provided at least one photo source.

### Step 2 - Run preflight

User-facing:

- Tell the user you will quickly verify that the folders are readable and that