dockkit

What this skill does

# DockKit

Framework for integrating with motorized camera stands and gimbals that
physically track subjects by rotating the iPhone. DockKit handles motor
control, subject detection, and framing so camera apps get 360-degree pan
and 90-degree tilt tracking with no additional code. Apps can override
system tracking to supply custom observations, control motors directly,
or adjust framing. iOS 17+, Swift 6.3.

## Contents

- [Setup](#setup)
- [Discovering Accessories](#discovering-accessories)
- [System Tracking](#system-tracking)
- [Custom Tracking](#custom-tracking)
- [Framing and Region of Interest](#framing-and-region-of-interest)
- [Motor Control](#motor-control)
- [Animations](#animations)
- [Tracking State and Subject Selection](#tracking-state-and-subject-selection)
- [Accessory Events](#accessory-events)
- [Battery Monitoring](#battery-monitoring)
- [Common Mistakes](#common-mistakes)
- [Review Checklist](#review-checklist)
- [References](#references)

## Setup

Import DockKit:

```swift
import DockKit
```

DockKit requires a physical DockKit-compatible accessory and a real device.
The Simulator cannot connect to dock hardware.

DockKit itself requires no special entitlements or DockKit-specific
Info.plist keys. Camera apps that use device cameras still need normal
camera privacy handling, including `NSCameraUsageDescription`. The framework
communicates with paired accessories automatically through the DockKit
system daemon.

The app must use AVFoundation camera APIs. DockKit hooks into the camera
pipeline to analyze frames for system tracking.

## Discovering Accessories

Use `DockAccessoryManager.shared` to observe dock connections:

```swift
import DockKit

func observeAccessories() async throws {
    for await stateChange in try DockAccessoryManager.shared.accessoryStateChanges {
        switch stateChange.state {
        case .docked:
            guard let accessory = stateChange.accessory else { continue }
            // Accessory is connected and ready
            configureAccessory(accessory)
        case .undocked:
            // iPhone removed from dock
            handleUndocked()
        @unknown default:
            break
        }
    }
}
```

`accessoryStateChanges` emits `DockAccessory.StateChange` values with `state`,
`accessory`, and `trackingButtonEnabled`. Use `accessory.identifier` for the
name, category, and UUID; hardware details are available via `firmwareVersion`
and `hardwareModel`.

## System Tracking

System tracking is DockKit's default mode. When enabled, the system
analyzes camera frames through built-in ML inference, detects faces and
bodies, and drives the motors to keep subjects in frame. Any app using
AVFoundation camera APIs benefits automatically.

### Enable or Disable

```swift
// Enable system tracking (default)
try await DockAccessoryManager.shared.setSystemTrackingEnabled(true)

// Disable system tracking for custom control
try await DockAccessoryManager.shared.setSystemTrackingEnabled(false)
```

System tracking state does not persist across app termination, reboots,
or background/foreground transitions. Set it explicitly whenever the app
needs a specific value.

### Tap to Select Subject

Allow users to select a specific subject by tapping:

```swift
// Select the subject at a unit point in video-frame coordinates
try await accessory.selectSubject(at: CGPoint(x: 0.5, y: 0.5))

// Select specific subjects by identifier
try await accessory.selectSubjects([subjectUUID])

// Clear selection (return to automatic selection)
try await accessory.selectSubjects([])
```

## Custom Tracking

Disable system tracking and provide your own observations when using
custom ML models or the Vision framework.

### Providing Observations

Construct `DockAccessory.Observation` values from your inference output
and pass them to the accessory at 10-30 fps:

```swift
import DockKit
import AVFoundation

func processFrame(
    _ sampleBuffer: CMSampleBuffer,
    accessory: DockAccessory,
    activeDevice: AVCaptureDevice
) async throws {
    let cameraInfo = DockAccessory.CameraInformation(
        captureDevice: activeDevice.deviceType,
        cameraPosition: activeDevice.position,
        orientation: .corrected,
        cameraIntrinsics: frameIntrinsics(from: sampleBuffer),
        referenceDimensions: frameDimensions(from: sampleBuffer)
    )

    let detection = try await detector.detect(sampleBuffer)
    let observationType: DockAccessory.Observation.ObservationType = switch detection.kind {
    case .face: .humanFace
    case .body: .humanBody
    case .object: .object
    }

    let observation = DockAccessory.Observation(
        identifier: detection.id,
        type: observationType,
        rect: detection.rect,       // normalized, lower-left origin
        faceYawAngle: detection.faceYawAngle
    )

    try await accessory.track([observation], cameraInformation: cameraInfo)
}
```

### Observation Types

When reviewing custom tracking, explicitly choose among the only supported
`ObservationType` cases: `.humanFace`, `.humanBody`, and `.object`.
Do not answer with only `.humanFace` when body or object detections are possible.

The `rect` uses normalized coordinates with a lower-left origin (same
coordinate system as Vision framework -- no conversion needed).

### Camera Information

`DockAccessory.CameraInformation` describes the active camera; do not hardcode
placeholder device, intrinsics, or frame-size values. Set orientation to
`.corrected` when coordinates are already relative to the bottom-left corner.
In review answers, reject opaque optional `cameraInfo` placeholders and show
construction from the active `AVCaptureDevice` plus the current `CMSampleBuffer`.

Track variants also accept `[AVMetadataObject]` instead of observations.
Use the `image: CVPixelBuffer` overloads when DockKit should combine
observations or metadata with the captured image buffer; the image argument
is required in those overloads.

## Framing and Region of Interest

### Framing Modes

Control how the system frames tracked subjects:

```swift
try await accessory.setFramingMode(.automatic) // documented default
try await accessory.setFramingMode(.center)    // explicit opt-in
```

| Mode | Behavior |
|---|---|
| `.automatic` | Documented default; system decides optimal framing |
| `.center` | Explicit opt-in mode to keep subject centered |
| `.left` | Frame subject in left third |
| `.right` | Frame subject in right third |

Default system behavior often centers the primary subject, but `.center` is
never the default-like mode; `.automatic` is. Use `.left` or `.right` when
graphic overlays occupy part of the frame.

### Region of Interest

Constrain tracking to a specific area of the video frame:

```swift
// Normalized coordinates, origin at upper-left
let squareRegion = CGRect(x: 0.25, y: 0.0, width: 0.5, height: 1.0)
try await accessory.setRegionOfInterest(squareRegion)
```

Use region of interest when cropping to a non-standard aspect ratio
(e.g., square video for conferencing) so subjects stay within the
visible area.

## Motor Control

Disable system tracking before controlling motors directly.

### Angular Velocity

Set continuous rotation speed in radians per second:

```swift
import Spatial

// Pan right at 0.2 rad/s, tilt down at 0.1 rad/s
let velocity = Vector3D(x: 0.1, y: 0.2, z: 0.0)
try await accessory.setAngularVelocity(velocity)

// Stop all motion
try await accessory.setAngularVelocity(Vector3D())
```

Axes:
- `x` -- pitch (tilt). Positive tilts down on iOS.
- `y` -- yaw (pan). Positive pans right.
- `z` -- roll (if supported by hardware).

### Set Orientation

Move to a specific position over a duration:

```swift
let target = Vector3D(x: 0.0, y: 0.5, z: 0.0)  // Yaw 0.5 rad
let progress = try accessory.setOrientation(
    target,
    duration: .seconds(2),
    relative: false
)
```

Also accepts `Rotation3D` for quaternion-based orientation. Set
`relative: true` to move relative to the current position. The returne