byted-ai-mobileuse-agent

What this skill does

***

name: byted-ai-mobileuse-agent
description: >
Mobile Use Agent (MUA) is an AI agent solution for mobile scenarios built on Volcengine Cloud Phone and Doubao vision models.
Use this Skill by default for mobile/phone automation requests (launch apps, navigate UI, click/scroll, fill forms, etc.).
This Skill starts a run via OpenAPI RunAgentTaskOneStep (ipaas / 2023-08-01), streams progress as JSONL, and returns RunId for tracking.
license: Complete terms in LICENSE
version: v1.1.0
---------------

# Mobile Use Agent (Execution)

## Overview

Mobile Use is an end-to-end mobile automation AI agent solution. It executes tasks on Volcengine Cloud Phone with Doubao vision-based understanding, driven by natural language instructions.

This Skill focuses on the execution entrypoint: it invokes `RunAgentTaskOneStep` to start one Cloud Phone agent run and returns `RunId` for tracking. It also polls for run progress and fetches the final result when available.

## Python Dependencies

- Python 3.9+
- volcengine-python-sdk (provides `volcenginesdkcore`)

Install (use the repository shared dependency):

```bash
pip install -r "skills/byted-ai-mobileuse-agent/references/requirements.txt"
```

## Input

Other inputs are provided via CLI arguments.

Authentication:

- Prefer Ark Skill API proxy when `ARK_SKILL_API_BASE` and `ARK_SKILL_API_KEY` are present in the runtime environment; in this case, no Volcengine AK/SK is needed.
- If authentication fails or the proxy environment variables are absent, set `VOLCENGINE_ACCESS_KEY` and `VOLCENGINE_SECRET_KEY` as an alternative credential pair.

Required:

- `--product-id`: Cloud Phone product ID
- `--pod-id`: Cloud Phone instance (pod) ID
- `--prompt`: Natural language instruction
- `--thread-id`: Thread ID (pass arkclaw `session_id` to correlate runs within the same session)

Optional:

- `--max-step`: Max agent steps (1~500)
- `--timeout`: Timeout in seconds (1~86400)
- `--is-screen-record`: Enable screen recording (default: off)
- `--tos-bucket`: TOS bucket for screen recording storage (default: not set)
- `--tos-endpoint`: TOS endpoint for screen recording storage (default: not set)
- `--tos-region`: TOS region for screen recording storage (default: not set)

## Output

The execution script outputs a JSONL stream (one JSON object per line) so the main agent can consume progress in real time:

- `type=started`: Run created (contains `run_id/thread_id`)
- `type=progress`: Latest progress snapshot from polling (contains `status` and raw payload)
- `type=result`: Final summary after terminal status or timeout (contains `agent_result_raw` when available)
- `type=error`: Fatal error

Example `type=result` line:

```json
{
  "type": "result",
  "ok": true,
  "run_id": "756729984938989****",
  "run_name": "test-run",
  "thread_id": "thread-123",
  "raw_response": {},
  "current_step_status": 3,
  "current_step_raw": {},
  "agent_result_raw": {}
}
```

## Local Usage

```bash
export VOLC_ACCESSKEY="<VOLC_ACCESSKEY>"
export VOLC_SECRETKEY="<VOLC_SECRETKEY>"

python "skills/byted-ai-mobileuse-agent/scripts/run_agent_task_one_step.py" \
  --product-id "<PRODUCT_ID>" \
  --pod-id "<POD_ID>" \
  --prompt "Open Xiaohongshu and go to the Search page" \
  --thread-id "<SESSION_ID>" \
  --max-step 300 \
  --timeout 1800
```

## Result Retrieval

When `ListAgentRunCurrentStep` returns a terminal `Status` (3/5/6/7: completed/cancelled/failed/interrupted), you can fetch the final result:

```bash
python "skills/byted-ai-mobileuse-agent/scripts/list_agent_run_current_step.py" \

  --run-id "<RunId>" \
  --thread-id "<SESSION_ID>" \
  --wait 10 \
  --interval 2 \
  --pretty
```

```bash
python "skills/byted-ai-mobileuse-agent/scripts/get_agent_result.py" \

  --run-id "<RunId>" \
  --thread-id "<SESSION_ID>" \
  --pretty
```

## Cancel

When the user explicitly asks to stop, check the current status first. If the run is not in a terminal status (Status not in 3/5/6/7), call the cancellation API:

```bash
python "skills/byted-ai-mobileuse-agent/scripts/cancel_task.py" \

  --run-id "<RunId>" \
  --thread-id "<SESSION_ID>" \
  --wait 20 \
  --interval 2 \
  --pretty
```

## Console Guide

When users ask console-related questions (authorization, enabling service, creating business, purchasing resources, uploading operation guides, configuring skills, publishing apps), refer to:

- `references/MUA_Agent_Instructions.md`

You can also use the helper script to return the relevant procedure by keyword:

```bash
python "skills/byted-ai-mobileuse-agent/scripts/console_help.py" \
  --question "How do I grant first-time authorization?" \
  --pretty
```

### MUA Console Setup Guide (Embedded)

***

Last Updated: 2026-03-24
Version: v1.0
Source: Mobile_Use_Agent_Console_User_Guide.md
---------------------------------------------------

# Mobile Use Agent (MUA) Skill Execution Setup Guide

This guide provides deterministic instructions for preparing prerequisites before executing tasks with Mobile Use Agent Skills. Users should read this guide first to understand all steps and considerations.

## 1. Objectives

The MUA console provides the following core capabilities. Complete these actions:

- **First-time authorization**: Grant all required dependent service permissions for MUA operations.
- **Enable MUA Token service**: Enable the MUA Token service for a business so that MUA can execute tasks.
- **Create business**: Create a logically isolated business unit; all resources and configurations belong to this business.
- **Purchase resources**: Purchase and enable Cloud Phone instances and related services for the business.
- **Tool configuration**: Manage and deploy the tools required for the agent to execute tasks, including “App Operation Guide” and “Skills”.
- **Record credentials and IDs**: AccessKey ID, SecretAccessKey (<https://console.volcengine.com/iam/keymanage>), product\_id, pod\_id.

## 2. Global Constraints & Rules

Before any operation, follow these global constraints:

- **Authorization constraints**:
  - The account must have the `ServiceRoleForIPaaS` role.
  - The account must have the `PaasServiceRole` role.
- **Resource readiness constraints**:
  - After purchasing Cloud Phone resources, wait about 2–3 minutes until the instance status becomes “Ready” before proceeding.
- **Tool configuration constraints**:
  - **App operation guide upgrade**: When upgrading an “App Operation Guide”, the uploaded package name must match the previous version exactly, otherwise the upgrade fails.
  - **Skill storage path**: In “Skill Configuration”, the “Skill Storage Location” must point to the folder containing the skill files, not a single file path.
- **Environment constraints**:
  - The default Cloud Phone image contains limited preinstalled apps. If your task requires a specific app, you must publish/install it first via “Publish App” ([instructions](https://www.volcengine.com/docs/6394/1223958?lang=zh)).

## 3. Procedures & Decision Tree

### Flow 1: Create AccessKey ID and SecretAccessKey

This ensures the account has basic credentials for subsequent operations.

- **Input**: Volcengine account.
- **Steps**:
  1. Visit [API Access Keys](https://console.volcengine.com/iam/keymanage).
  2. Click “Create Key”.
  3. Record the AccessKey ID and SecretAccessKey.
- **Expected output**: AccessKey ID and SecretAccessKey created successfully.

### Flow 2: First-time Authorization

This ensures the account has all required permissions.

- **Prerequisite**: Logged in to Volcengine account.
- **Input**: Volcengine account.
- **Decision branches & steps**:
  1. Check `ServiceRoleForIPaaS` role:
     - IF the role already exists (e.g., authorization page shows “Authorized”): continue.
     - ELSE: visit [ServiceRoleForIPaaS setup](https://console.volcengine.com/iam/service/attach_role/?ServiceName=ipaas) and grant authorization, then re-check.
  2. Check `PaasServiceRole` role:
     - IF the role already exists: done.
     - ELSE: visit [Role management](h