on-device-ai

What this skill does

# React Native ExecuTorch

Software Mansion's production patterns for on-device AI in React Native and Expo using [React Native ExecuTorch](https://github.com/software-mansion/react-native-executorch).

Targets the current published API (v0.10.x). Load at most one reference file per question. For hook signatures, model constants, or config options not covered here, webfetch the matching page from [docs.swmansion.com/react-native-executorch](https://docs.swmansion.com/react-native-executorch/docs/).

## Decision Tree

```
What does the feature need?
│
├── Generate / chat with text?
│   └── useLLM                                          → see llm.md
│       ├── Plain chat → standard useLLM
│       ├── Image + text input → useLLM with a VLM model (LFM2_VL_*)
│       ├── Tool / function calling → configure with toolsConfig
│       └── Structured JSON output → getStructuredOutputPrompt
│
├── Understand or transform images?
│   ├── What is in this image? → useClassification      → see vision.md
│   ├── Where are the objects? → useObjectDetection     → see vision.md
│   ├── Per-pixel class → useSemanticSegmentation       → see vision.md
│   ├── Per-instance segmentation → useInstanceSegmentation → see vision.md
│   ├── Human pose keypoints → usePoseEstimation        → see vision.md
│   ├── Read text from image → useOCR / useVerticalOCR  → see vision.md
│   ├── Apply artistic style → useStyleTransfer         → see vision.md
│   ├── Generate image from prompt → useTextToImage     → see vision.md
│   └── Embed image as vector → useImageEmbeddings      → see vision.md
│
├── Speech / audio?
│   ├── Transcribe speech → useSpeechToText             → see speech.md
│   ├── Synthesize speech → useTextToSpeech             → see speech.md
│   └── Detect speech segments → useVAD                 → see speech.md
│
├── Text utilities?
│   ├── Embed text as vector → useTextEmbeddings        → see vision.md
│   ├── Count or inspect tokens → useTokenizer          → see setup.md
│   └── Redact PII from text → usePrivacyFilter         → see setup.md
│
├── Full RAG pipeline (retrieval + generation + vector store)?
│   └── react-native-rag (sibling library)              → see setup.md
│
└── Custom `.pte` model not covered by a dedicated hook?
    └── useExecutorchModule                             → see setup.md
```

## Critical Rules

- **Call `initExecutorch()` at app entry, before any other API.** The library does not bundle a network/file layer — you must register a resource-fetcher adapter (`ExpoResourceFetcher` for Expo, `BareResourceFetcher` for bare RN). Any hook called before initialization throws `ResourceFetcherAdapterNotInitialized`.

- **Check `isReady` before calling `forward` / `generate` / `transcribe`.** All hooks load asynchronously. Inference before the model is ready throws `ModuleNotLoaded`.

- **Interrupt LLM generation before unmounting.** Unmounting while `isGenerating` is `true` crashes. Call `llm.interrupt()` and wait for `isGenerating === false` before navigating away.

- **Use quantized model variants on mobile.** Full-precision variants exceed device memory on most phones. Every supported model ships a `_QUANTIZED` variant — prefer it unless you've measured otherwise.

- **Audio for speech-to-text and VAD must be 16 kHz mono.** Mismatched sample rates produce silently garbled transcriptions. Decode with `new AudioContext({ sampleRate: 16000 })`.

- **Audio from text-to-speech is 24 kHz.** Create the playback context with `new AudioContext({ sampleRate: 24000 })`.

- **The New Architecture (Fabric) is required.** Old architecture is unsupported. Expo Go is unsupported — use a custom dev build (`npx expo prebuild`). iOS release builds need a real device (the simulator lacks the Metal APIs ExecuTorch relies on).

## Minimal Setup

```tsx
// App.tsx (Expo)
import { initExecutorch } from 'react-native-executorch';
import { ExpoResourceFetcher } from 'react-native-executorch-expo-resource-fetcher';

initExecutorch({ resourceFetcher: ExpoResourceFetcher });
```

```tsx
// App.tsx (bare React Native)
import { initExecutorch } from 'react-native-executorch';
import { BareResourceFetcher } from 'react-native-executorch-bare-resource-fetcher';

initExecutorch({ resourceFetcher: BareResourceFetcher });
```

Full setup, Metro config for bundled `.pte` files, custom adapters, model-loading strategies, and error handling: see [setup.md](setup.md).

## Hook Quick Reference

| Hook | Purpose | Reference |
|---|---|---|
| `useLLM` | Text generation, chat, tool calling, VLM | [llm.md](llm.md) |
| `useClassification` | Image categorisation | [vision.md](vision.md) |
| `useObjectDetection` | Bounding-box detection (YOLO26, RF-DETR, SSDLite) | [vision.md](vision.md) |
| `useSemanticSegmentation` | Per-pixel class segmentation | [vision.md](vision.md) |
| `useInstanceSegmentation` | Per-instance segmentation | [vision.md](vision.md) |
| `usePoseEstimation` | COCO 17-keypoint human pose | [vision.md](vision.md) |
| `useStyleTransfer` | Artistic image filters | [vision.md](vision.md) |
| `useTextToImage` | Stable Diffusion image generation | [vision.md](vision.md) |
| `useImageEmbeddings` | CLIP image embeddings | [vision.md](vision.md) |
| `useOCR` | Horizontal text OCR | [vision.md](vision.md) |
| `useVerticalOCR` | Vertical text OCR (experimental, CJK) | [vision.md](vision.md) |
| `useTextEmbeddings` | Sentence embeddings for similarity / RAG | [vision.md](vision.md) |
| `useSpeechToText` | Whisper transcription (batch + streaming) | [speech.md](speech.md) |
| `useTextToSpeech` | Kokoro TTS (batch + streaming, phoneme input) | [speech.md](speech.md) |
| `useVAD` | FSMN voice activity detection | [speech.md](speech.md) |
| `useTokenizer` | HuggingFace-compatible tokenization | [setup.md](setup.md) |
| `usePrivacyFilter` | On-device PII / privacy redaction | [setup.md](setup.md) |
| `useExecutorchModule` | Custom `.pte` model inference | [setup.md](setup.md) |

Every hook also has a non-React `Module` counterpart (e.g. `LLMModule.fromModelName(...)`, `ClassificationModule.fromModelName(...)`) for use outside React components.

## Common Pitfalls

| Symptom | Likely cause | Fix |
|---|---|---|
| `ResourceFetcherAdapterNotInitialized` | `initExecutorch` not called | Call it at app entry with an adapter |
| `ModuleNotLoaded` | Inference before model finished loading | Gate calls on `isReady` |
| `MemoryAllocationFailed` on launch | Model too large for device | Switch to `_QUANTIZED` variant or smaller parameter count |
| App crashes on screen navigation | Unmount during active generation | `llm.interrupt()` and await `isGenerating === false` |
| Whisper produces garbled text | Wrong sample rate | Decode audio at 16 kHz mono |
| TTS output sounds chipmunked | Playback context at wrong rate | Create `AudioContext({ sampleRate: 24000 })` |
| Build fails on iOS simulator (release) | Simulator lacks Metal APIs | Build release on real device |

Full error code list and recovery patterns: [setup.md](setup.md).

## References

| File | When to read |
|---|---|
| [llm.md](llm.md) | `useLLM` functional + managed modes, tool calling, structured output (JSON Schema / Zod), interrupting, vision-language models, generation config |
| [vision.md](vision.md) | Image classification, object detection, semantic + instance segmentation, pose estimation, OCR (horizontal + vertical), style transfer, text-to-image, image + text embeddings |
| [speech.md](speech.md) | Speech-to-text (Whisper batch + streaming with timestamps), text-to-speech (Kokoro batch + streaming, phoneme input, voice catalogue), voice activity detection, audio sample-rate requirements |
| [setup.md](setup.md) | `initExecutorch`, Expo / bare resource-fetcher adapters, model loading strategies, Metro config, error codes and recovery, `useExecutorchModule` for custom `.pte` models, `useTokenizer`, `usePrivacyFilter`, full model catalogue |

## External Resources

- Official docs: https://docs.swmansion.com/react-native-executorc