type4me-macos-voice-input

What this skill does

# Type4Me macOS Voice Input

> Skill by [ara.so](https://ara.so) — Daily 2026 Skills collection.

Type4Me is a macOS voice input tool that captures audio via global hotkey, transcribes it using local (SherpaOnnx/Paraformer/Zipformer) or cloud (Volcengine/Deepgram) ASR engines, optionally post-processes text via LLM, and injects the result into any app. All credentials and history are stored locally — no telemetry, no cloud sync.

## Architecture Overview

```
Type4Me/
├── ASR/                    # ASR engine abstraction
│   ├── ASRProvider.swift          # Provider enum + protocols
│   ├── ASRProviderRegistry.swift  # Plugin registry
│   ├── Providers/                 # Per-vendor config files
│   ├── SherpaASRClient.swift      # Local streaming ASR
│   ├── SherpaOfflineASRClient.swift
│   ├── VolcASRClient.swift        # Volcengine streaming ASR
│   └── DeepgramASRClient.swift    # Deepgram streaming ASR
├── Bridge/                 # SherpaOnnx C API Swift bridge
├── Audio/                  # Audio capture
├── Session/                # Core state machine: record→ASR→inject
├── Input/                  # Global hotkey management
├── Services/               # Credentials, hotwords, model manager
├── Protocol/               # Volcengine WebSocket codec
└── UI/                     # SwiftUI (FloatingBar + Settings)
```

## Installation

### Prerequisites

```bash
# Xcode Command Line Tools
xcode-select --install

# CMake (for local ASR engine)
brew install cmake
```

### Build & Deploy from Source

```bash
git clone https://github.com/joewongjc/type4me.git
cd type4me

# Step 1: Compile SherpaOnnx local engine (~5 min, one-time)
bash scripts/build-sherpa.sh

# Step 2: Build, bundle, sign, install to /Applications, and launch
bash scripts/deploy.sh
```

### Download Pre-built App

Download `Type4Me-v1.2.3.dmg` from releases (cloud ASR only, no local engine):
```
https://github.com/joewongjc/type4me/releases/tag/v1.2.3
```

If macOS blocks the app:
```bash
xattr -d com.apple.quarantine /Applications/Type4Me.app
```

### Download Local ASR Models

```bash
mkdir -p ~/Library/Application\ Support/Type4Me/Models

# Option A: Lightweight ~20MB
tar xjf ~/Downloads/sherpa-onnx-streaming-zipformer-small-ctc-zh-int8-2025-04-01.tar.bz2 \
    -C ~/Library/Application\ Support/Type4Me/Models/

# Option B: Balanced ~236MB (recommended)
tar xjf ~/Downloads/sherpa-onnx-streaming-zipformer-ctc-multi-zh-hans-2023-12-13.tar.bz2 \
    -C ~/Library/Application\ Support/Type4Me/Models/

# Option C: Bilingual Chinese+English ~1GB
tar xjf ~/Downloads/sherpa-onnx-streaming-paraformer-bilingual-zh-en.tar.bz2 \
    -C ~/Library/Application\ Support/Type4Me/Models/
```

Expected structure for Paraformer model:
```
~/Library/Application Support/Type4Me/Models/
└── sherpa-onnx-streaming-paraformer-bilingual-zh-en/
    ├── encoder.int8.onnx
    ├── decoder.int8.onnx
    └── tokens.txt
```

## Key Protocols

### SpeechRecognizer Protocol

Every ASR client must implement this protocol:

```swift
protocol SpeechRecognizer: AnyObject {
    /// Start a new recognition session
    func startRecognition() async throws
    
    /// Feed raw PCM audio data
    func appendAudio(_ buffer: AVAudioPCMBuffer) async
    
    /// Stop and get final result
    func stopRecognition() async throws -> String
    
    /// Cancel without result
    func cancelRecognition() async
    
    /// Streaming partial results (optional)
    var partialResultHandler: ((String) -> Void)? { get set }
}
```

### ASRProviderConfig Protocol

Each vendor's credential definition:

```swift
protocol ASRProviderConfig {
    /// Unique identifier string
    static var providerID: String { get }
    
    /// Display name in Settings UI
    static var displayName: String { get }
    
    /// Credential fields shown in Settings
    static var credentialFields: [CredentialField] { get }
    
    /// Validate credentials before use
    static func validate(_ credentials: [String: String]) -> Bool
    
    /// Create the recognizer instance
    static func createClient(
        credentials: [String: String],
        config: RecognitionConfig
    ) throws -> SpeechRecognizer
}
```

## Adding a New ASR Provider

### Step 1: Create Provider Config

Create `Type4Me/ASR/Providers/OpenAIWhisperProvider.swift`:

```swift
import Foundation

struct OpenAIWhisperProvider: ASRProviderConfig {
    static let providerID = "openai_whisper"
    static let displayName = "OpenAI Whisper"
    
    static let credentialFields: [CredentialField] = [
        CredentialField(
            key: "api_key",
            label: "API Key",
            placeholder: "sk-...",
            isSecret: true
        ),
        CredentialField(
            key: "model",
            label: "Model",
            placeholder: "whisper-1",
            isSecret: false
        )
    ]
    
    static func validate(_ credentials: [String: String]) -> Bool {
        guard let apiKey = credentials["api_key"], !apiKey.isEmpty else {
            return false
        }
        return apiKey.hasPrefix("sk-")
    }
    
    static func createClient(
        credentials: [String: String],
        config: RecognitionConfig
    ) throws -> SpeechRecognizer {
        guard let apiKey = credentials["api_key"] else {
            throw ASRError.missingCredential("api_key")
        }
        let model = credentials["model"] ?? "whisper-1"
        return OpenAIWhisperASRClient(apiKey: apiKey, model: model, config: config)
    }
}
```

### Step 2: Implement the ASR Client

Create `Type4Me/ASR/OpenAIWhisperASRClient.swift`:

```swift
import Foundation
import AVFoundation

final class OpenAIWhisperASRClient: SpeechRecognizer {
    var partialResultHandler: ((String) -> Void)?
    
    private let apiKey: String
    private let model: String
    private let config: RecognitionConfig
    private var audioData: Data = Data()
    
    init(apiKey: String, model: String, config: RecognitionConfig) {
        self.apiKey = apiKey
        self.model = model
        self.config = config
    }
    
    func startRecognition() async throws {
        audioData = Data()
    }
    
    func appendAudio(_ buffer: AVAudioPCMBuffer) async {
        // Convert PCM buffer to raw bytes and accumulate
        guard let channelData = buffer.floatChannelData?[0] else { return }
        let frameCount = Int(buffer.frameLength)
        let bytes = UnsafeBufferPointer(start: channelData, count: frameCount)
        // Convert Float32 PCM to Int16 for Whisper API
        let int16Samples = bytes.map { sample -> Int16 in
            return Int16(max(-32768, min(32767, Int(sample * 32767))))
        }
        int16Samples.withUnsafeBytes { ptr in
            audioData.append(contentsOf: ptr)
        }
    }
    
    func stopRecognition() async throws -> String {
        // Build multipart form request to Whisper API
        var request = URLRequest(url: URL(string: "https://api.openai.com/v1/audio/transcriptions")!)
        request.httpMethod = "POST"
        request.setValue("Bearer \(apiKey)", forHTTPHeaderField: "Authorization")
        
        let boundary = UUID().uuidString
        request.setValue("multipart/form-data; boundary=\(boundary)", 
                        forHTTPHeaderField: "Content-Type")
        
        var body = Data()
        // Append audio file part
        body.append("--\(boundary)\r\n".data(using: .utf8)!)
        body.append("Content-Disposition: form-data; name=\"file\"; filename=\"audio.raw\"\r\n".data(using: .utf8)!)
        body.append("Content-Type: audio/raw\r\n\r\n".data(using: .utf8)!)
        body.append(audioData)
        body.append("\r\n".data(using: .utf8)!)
        // Append model part
        body.append("--\(boundary)\r\n".data(using: .utf8)!)
        body.append("Content-Disposition: form-data; name=\"model\"\r\n\r\n".data(using: .utf8)!)
        body.append("\(model)\r\n".data(using: .utf8)!)
        body.append("--\(boundary)--\r\n".data(using: .utf8)!)
        
        re