realitykit

What this skill does

# RealityKit

Build AR experiences on iOS using RealityKit for rendering and ARKit for world
tracking. Covers `RealityView`, entity management, raycasting, scene
understanding, and gesture-based interactions. Targets Swift 6.3 / iOS 26+.

## Contents

- [Setup](#setup)
- [RealityView Basics](#realityview-basics)
- [Loading and Creating Entities](#loading-and-creating-entities)
- [Anchoring and Placement](#anchoring-and-placement)
- [Raycasting](#raycasting)
- [Gestures and Interaction](#gestures-and-interaction)
- [Scene Understanding](#scene-understanding)
- [Common Mistakes](#common-mistakes)
- [Review Checklist](#review-checklist)
- [References](#references)

## Setup

### Project Configuration

1. Add `NSCameraUsageDescription` to Info.plist
2. On iOS, `RealityViewCameraContent` displays an AR camera view by default (iOS 18+, macOS 15+); use `.virtual` camera mode for explicit non-AR fallback
3. No entitlement is required for basic AR. If AR is core to the app, add the `arkit` required-device capability; otherwise gate AR UI with `isSupported`.

### Device Requirements

AR features require devices with an A9 chip or later. Always check
`ARWorldTrackingConfiguration.isSupported` before presenting AR UI.

```swift
import ARKit

guard ARWorldTrackingConfiguration.isSupported else {
    showUnsupportedDeviceMessage()
    return
}
```

### Key Types

| Type | Platform | Role |
|---|---|---|
| `RealityView` | iOS 18+, visionOS 1+ | SwiftUI view that hosts RealityKit content |
| `RealityViewCameraContent` | iOS 18+, macOS 15+ | Content displayed through an AR camera view on iOS, non-AR on macOS |
| `Entity` | All | Base class for all scene objects |
| `ModelEntity` | All | Entity with a visible 3D model |
| `AnchorEntity` | All | Tethers entities to a real-world anchor |

## RealityView Basics

`RealityView` is the SwiftUI entry point for RealityKit.
`RealityViewCameraContent` is the iOS/macOS content type. On iOS, it uses an AR
camera view by default and can use `content.camera = .virtual` for non-AR mode
when requested or when AR/camera access is unavailable.

```swift
import ARKit
import SwiftUI
import RealityKit

struct ARExperienceView: View {
    var body: some View {
        RealityView { (content: RealityViewCameraContent) in
            if !ARWorldTrackingConfiguration.isSupported {
                content.camera = .virtual
            }

            let sphere = ModelEntity(
                mesh: .generateSphere(radius: 0.05),
                materials: [SimpleMaterial(
                    color: .blue,
                    isMetallic: true
                )]
            )
            sphere.position = [0, 0, -0.5]  // 50cm in front of camera
            content.add(sphere)
        }
    }
}
```

### Make and Update Pattern

Use the `update` closure to respond to SwiftUI state changes:

```swift
struct PlacementView: View {
    @State private var modelColor: UIColor = .red

    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(
                    color: .red,
                    isMetallic: false
                )]
            )
            box.name = "colorBox"
            box.position = [0, 0, -0.5]
            content.add(box)
        } update: { content in
            if let box = content.entities.first(
                where: { $0.name == "colorBox" }
            ) as? ModelEntity {
                box.model?.materials = [SimpleMaterial(
                    color: modelColor,
                    isMetallic: false
                )]
            }
        }

        Button("Change Color") {
            modelColor = modelColor == .red ? .green : .red
        }
    }
}
```

## Loading and Creating Entities

### Loading from USDZ Files

Load 3D models asynchronously to avoid blocking the main thread:

```swift
RealityView { content in
    if let robot = try? await ModelEntity(named: "robot") {
        robot.position = [0, -0.2, -0.8]
        robot.scale = [0.01, 0.01, 0.01]
        content.add(robot)
    }
}
```

### Adding Components

Entities use an ECS (Entity Component System) architecture. Add components
to give entities behavior:

```swift
let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)

// Make it respond to physics
box.components.set(PhysicsBodyComponent(
    massProperties: .default,
    material: .default,
    mode: .dynamic
))

// Add collision shape for interaction
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))

// Enable input targeting for gestures
box.components.set(InputTargetComponent())
```

## Anchoring and Placement

### AnchorEntity

Use `AnchorEntity` to anchor content to detected surfaces or world positions:

```swift
RealityView { content in
    // Anchor to a horizontal surface
    let floorAnchor = AnchorEntity(.plane(
        .horizontal,
        classification: .floor,
        minimumBounds: [0.2, 0.2]
    ))

    let model = ModelEntity(
        mesh: .generateBox(size: 0.1),
        materials: [SimpleMaterial(color: .orange, isMetallic: false)]
    )
    floorAnchor.addChild(model)
    content.add(floorAnchor)
}
```

### Anchor Targets

| Target | Description |
|---|---|
| `.plane(.horizontal, ...)` | Horizontal surfaces (floors, tables) |
| `.plane(.vertical, ...)` | Vertical surfaces (walls) |
| `.plane(.any, ...)` | Any detected plane |
| `.world(transform:)` | Fixed world-space position |

## Raycasting

Keep RealityKit scene queries separate from ARKit real-world raycasts:

- `RealityViewCameraContent.ray(through:in:to:)` returns a camera ray in
  RealityKit coordinate spaces. It projects a screen point into the virtual
  scene; it is not proof of a detected physical surface.
- `RealityViewCameraContent.hitTest(point:in:query:mask:)` hits virtual
  entities made hittable by `CollisionComponent` shapes. Use those shapes for
  entity picking and targeted gestures, not ARKit plane detection.
- Use `AnchorEntity(.plane(...))` for simple placement on detected planes.
- Use ARKit `ARRaycastQuery` plus `ARSession.raycast(_:)` when the task needs
  a one-shot intersection with real-world surfaces, then anchor with
  `AnchorEntity(raycastResult:)`.

```swift
let results = session.raycast(query)
if let result = results.first {
    let anchor = AnchorEntity(raycastResult: result)
    anchor.addChild(model)
    content.add(anchor)
}
```

Do not treat entity hit tests as substitutes for ARKit surface raycasts.

## Gestures and Interaction

For gesture-based entity interaction, add `CollisionComponent` for the hittable
shape and `InputTargetComponent` for input targeting. Use
`AccessibilityComponent` for entity labels/actions. Hand detailed SwiftUI gesture
composition and VoiceOver/Switch Control policy to sibling skills.

### Drag Gesture on Entities

```swift
struct DraggableARView: View {
    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .blue, isMetallic: true)]
            )
            box.position = [0, 0, -0.5]
            box.components.set(CollisionComponent(
                shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
            ))
            box.components.set(InputTargetComponent())
            box.name = "draggable"
            content.add(box)
        }
        .gesture(
            DragGesture()
                .targetedToAnyEntity()
                .onChanged { value in
                    let entity = value.entity
                    guard let parent = entity.parent else { return }
                    entity.position = value.convert(
                        value.location3D,
                        from: .local,
                        to: parent
                    )
                }
        )
    }
}
```

For se