lsfg-android-frame-generation

What this skill does

# LSFG-Android Frame Generation Skill

> Skill by [ara.so](https://ara.so) — Daily 2026 Skills collection.

LSFG-Android ports the [`lsfg-vk`](https://github.com/PancakeTAS/lsfg-vk) Vulkan frame-generation pipeline to Android. Because Android 12+ blocks external code injection into non-debuggable processes, the app captures the screen via `MediaProjection`, runs LSFG interpolation on-GPU using `AHardwareBuffer` sharing, and composites generated frames into a `SYSTEM_ALERT_WINDOW` or `TYPE_ACCESSIBILITY_OVERLAY` sitting above the target game.

---

## Repository Layout

```
LSFG-Android/          # Android Studio project (Kotlin + Compose + JNI/C++)
lsfg-vk-android/       # Submodule: lsfg-vk 1.0.0 + Android patches (#ifdef __ANDROID__)
```

Both folders **must** be siblings on disk — the CMake path `../../../../../lsfg-vk-android` from inside the JNI sources is hard-coded.

---

## Prerequisites

| Tool | Version |
|------|---------|
| Android Studio | Ladybug or newer |
| NDK | 27.0.12077973 |
| CMake | 3.22.1 |
| JDK | 17 |
| C++ standard | C++20 |
| minSdk | 29 (Android 10) |
| targetSdk | 35 (Android 15) |
| ABIs | `arm64-v8a` (production), `x86_64` (emulator) |

---

## Build & Install

```sh
# Clone with submodule
git clone --recurse-submodules https://github.com/FrankBarretta/LSFG-Android.git
cd LSFG-Android/LSFG-Android

# Debug build
./gradlew :app:assembleDebug

# Release build
./gradlew :app:assembleRelease

# Install directly to connected device
adb install app/build/outputs/apk/debug/app-debug.apk
```

The native `lsfg-vk-android` library is compiled automatically by CMake as part of the Gradle build — no separate `.so` to ship.

---

## First-Run Setup

1. **Grant overlay permission** (`SYSTEM_ALERT_WINDOW`) via Settings → Apps → Special App Access.
2. **Enable `LsfgAccessibilityService`** (for touch-passthrough on strict OEMs).
3. **Load shaders**: pick your legitimately purchased `Lossless.dll` via the Storage Access Framework picker. Shaders are extracted to private storage; the DLL copy is deleted immediately after.
4. **Select target apps**: the overlay arms automatically when a target app comes to the foreground.

---

## Key Configuration Parameters

All parameters are exposed in the live settings drawer. Most trigger a native context re-init; bypass/pacing/Shizuku timing have hot-apply paths.

| Parameter | Range | Notes |
|-----------|-------|-------|
| Multiplier | 2×–8× | Frame generation ratio |
| Flow scale | 0.25–1.0 | Optical-flow resolution |
| Performance mode | bool | Reduces accuracy for speed |
| HDR mode | bool | HDR-aware interpolation |
| Anti-artifacts | bool | Artifact suppression pass |
| Bypass | bool | Hot-apply, no session drop |
| VSync alignment | bool + slack | Aligns output to display vsync |
| Pacing preset | enum | Controls frame pacing strategy |
| Target FPS cap | int | Output FPS ceiling |
| Queue depth | int | Vulkan swapchain queue depth |
| EMA jitter smoothing | float | Exponential moving average factor |

---

## Architecture Overview

```
MediaProjection capture
       │
       ▼
AHardwareBuffer (shared GPU memory)
       │
       ▼
lsfg-vk framegen (Vulkan, arm64-v8a)
  ├── LSFG_3_1 / LSFG_3_1P model
  ├── Optical flow pass
  └── Frame synthesis pass
       │
       ▼
Vulkan swapchain output  ──OR──  CPU-blit fallback
       │
       ▼
SYSTEM_ALERT_WINDOW / TYPE_ACCESSIBILITY_OVERLAY
(composited over target game)
```

---

## JNI / Native Integration Pattern

### Declaring the native interface (Kotlin)

```kotlin
// In your FrameGenBridge.kt or equivalent
object FrameGenBridge {
    init {
        System.loadLibrary("lsfg_android")
    }

    external fun nativeInit(
        surface: Surface,
        width: Int,
        height: Int,
        multiplier: Int,
        flowScale: Float,
        performanceMode: Boolean,
        hdrMode: Boolean
    ): Long  // returns native context handle

    external fun nativeApplyBypass(handle: Long, bypass: Boolean)
    external fun nativeApplyPacing(handle: Long, presetIndex: Int)
    external fun nativeSetTargetFps(handle: Long, targetFps: Int)
    external fun nativeDestroy(handle: Long)
}
```

### Calling from a ViewModel

```kotlin
class FrameGenViewModel : ViewModel() {
    private var nativeHandle: Long = 0L

    fun startSession(surface: Surface, config: FrameGenConfig) {
        nativeHandle = FrameGenBridge.nativeInit(
            surface       = surface,
            width         = config.width,
            height        = config.height,
            multiplier    = config.multiplier,
            flowScale     = config.flowScale,
            performanceMode = config.performanceMode,
            hdrMode       = config.hdrMode
        )
    }

    fun applyBypass(enabled: Boolean) {
        if (nativeHandle != 0L) FrameGenBridge.nativeApplyBypass(nativeHandle, enabled)
    }

    override fun onCleared() {
        if (nativeHandle != 0L) {
            FrameGenBridge.nativeDestroy(nativeHandle)
            nativeHandle = 0L
        }
    }
}
```

---

## MediaProjection Capture Setup

```kotlin
class CaptureService : Service() {
    private lateinit var mediaProjection: MediaProjection
    private lateinit var imageReader: ImageReader

    fun startCapture(
        resultCode: Int,
        data: Intent,
        width: Int,
        height: Int,
        density: Int
    ) {
        val mpManager = getSystemService(MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
        mediaProjection = mpManager.getMediaProjection(resultCode, data)

        imageReader = ImageReader.newInstance(width, height, PixelFormat.RGBA_8888, 2)
        val surface = imageReader.surface

        mediaProjection.createVirtualDisplay(
            "LSFG-Capture",
            width, height, density,
            DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR,
            surface, null, null
        )

        imageReader.setOnImageAvailableListener({ reader ->
            reader.acquireLatestImage()?.use { image ->
                // Pass AHardwareBuffer to native context
                val hardwareBuffer = image.hardwareBuffer
                if (hardwareBuffer != null) {
                    nativeSubmitFrame(nativeHandle, hardwareBuffer)
                    hardwareBuffer.close()
                }
            }
        }, Handler(Looper.getMainLooper()))
    }
}
```

---

## Overlay Window Setup

```kotlin
class OverlayManager(private val context: Context) {
    private val windowManager = context.getSystemService(WINDOW_SERVICE) as WindowManager

    fun createOverlayView(useAccessibility: Boolean): WindowManager.LayoutParams {
        val type = if (useAccessibility)
            WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY
        else
            WindowManager.LayoutParams.TYPE_APPLICATION_OVERLAY

        return WindowManager.LayoutParams(
            WindowManager.LayoutParams.MATCH_PARENT,
            WindowManager.LayoutParams.MATCH_PARENT,
            type,
            WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or
                WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL or
                WindowManager.LayoutParams.FLAG_LAYOUT_IN_SCREEN or
                WindowManager.LayoutParams.FLAG_HARDWARE_ACCELERATED,
            PixelFormat.TRANSLUCENT
        ).apply {
            gravity = Gravity.TOP or Gravity.START
        }
    }
}
```

---

## Accessibility Service for Touch Passthrough

```kotlin
// LsfgAccessibilityService.kt
class LsfgAccessibilityService : AccessibilityService() {
    override fun onAccessibilityEvent(event: AccessibilityEvent?) {}
    override fun onInterrupt() {}

    override fun onServiceConnected() {
        serviceInfo = serviceInfo.apply {
            flags = flags or AccessibilityServiceInfo.FLAG_REQUEST_TOUCH_EXPLORATION_MODE
        }
        // Notify the overlay that the privileged window type is available
        LocalBroadcastManager.getInstance(this)
            .sendBroadcast(Intent("LSFG_ACCESSIBILITY_CONNECTED