zms-troubleshoot-agent-deployment

What this skill does

# ZMS: Troubleshoot Agent Deployment

## Keywords

agent deployment, agent enrollment, agent disconnected, provisioning key, nonce, TOTP, agent version, agent connectivity, agent group, agent health, microsegmentation agent, agent upgrade, agent troubleshoot

## Overview

Troubleshoot Zscaler Microsegmentation (ZMS) agent deployment, enrollment, and connectivity issues. This skill systematically investigates agent fleet health, identifies disconnected or outdated agents, verifies agent group configuration, checks provisioning key availability, and retrieves enrollment credentials. It covers the full agent lifecycle from initial provisioning through ongoing health monitoring.

The ZMS API is a GraphQL endpoint on OneAPI. The full API supports both Query (read) and Mutation (write) operations for agents, agent groups, and nonces:

- **Query operations** (available via MCP): `agents`, `agentGroups`, `AgentConnectionStatusStatisticsConnection`, `agentVersionStatistics`, `nonces`, `nonce`
- **Mutation operations** (not yet in MCP, available via API/portal): `agentUpdate`, `agentDelete`, `agentGroupCreate/Update/Delete`, `agentGroupUpgradeStatusReset`, `nonceCreate/Update/Delete`, `nonceWithAgentGroupCreate`

**Use this skill when:** An administrator reports agents not connecting, enrollment failures, version mismatches, or needs to set up provisioning for new agent deployments.

**Important:**

- All ZMS tools require `ZSCALER_CUSTOMER_ID` to be set as an environment variable.
- All current MCP tools are **read-only** (Query operations). Write operations (create/update/delete agents, groups, nonces) must be performed through the Zscaler admin portal or the ZMS API directly.
- GraphQL errors may return HTTP 200 with errors in the response body — always check for the `errors` field.

---

## Workflow

Follow this 6-step process to troubleshoot agent deployment.

### Step 1: Identify the Problem

Gather from the administrator:

**Required:**

- What is the symptom? (agent not connecting, enrollment failure, version mismatch, missing agent)
- Which agents or hosts? (hostname, IP, agent group)
- When did the issue start?

**Helpful:**

- Is this a new deployment or an existing agent?
- Cloud provider (AWS, Azure, GCP, on-premises)?
- OS type and version?
- Any recent infrastructure changes?
- Error messages from the agent installer?

---

### Step 2: Check Overall Fleet Health

**Get connection status statistics:**

```text
zms_get_agent_connection_status_statistics()
```text

This provides the fleet-wide overview:

- Total agent count
- Connected vs disconnected counts and percentages
- Per-type and per-status breakdown

**Assess fleet health:**

- **> 95% connected**: Healthy fleet -- issue is likely isolated
- **90-95% connected**: Some connectivity issues -- check network
- **< 90% connected**: Widespread issue -- check infrastructure

**Get version statistics:**

```text
zms_get_agent_version_statistics()
```text

Check for:

- Number of distinct versions in use (should be minimal)
- Agents on deprecated or unsupported versions
- Upgrade rollout progress

---

### Step 3: Find the Specific Agent

**Search for the problematic agent:**

```text
zms_list_agents(search="<hostname_or_ip>", page_size=20)
```text

**If agent is found**, check:

- **connectionStatus**: Is the agent connected, disconnected, or in another state?
- **version**: Is the agent running the latest version?
- **os**: Is the OS supported?
- **ipAddresses**: Are the IP addresses correct for the expected network?
- **agentGroup**: Is the agent in the correct group?
- **lastSeen**: When was the agent last active?

**If agent is NOT found:**

- The agent was never enrolled, or enrollment failed
- The agent may be registered under a different hostname/IP
- Try broader search or list all agents:

```text
zms_list_agents(page=1, page_size=100)
```

---

### Step 4: Verify Agent Group Configuration

**List agent groups:**

```text
zms_list_agent_groups(page=1, page_size=50)
```text

For each group, verify:

- **Type**: Cloud provider type or on-premises
- **Agent count**: Expected number of agents
- **Policy status**: Whether policies are applied to the group
- **Auto-upgrade settings**: Whether agents auto-upgrade
- **Upgrade schedule**: Scheduled upgrade windows
- **Tamper protection**: Whether tamper protection is enabled

**Sort by name for easier navigation:**

```text
zms_list_agent_groups(sort="name", sort_dir="ASC", page_size=50)
```text

**Get TOTP secrets for enrollment (if enrollment is the issue):**

```text
zms_get_agent_group_totp_secrets(eyez_id="<agent_group_eyez_id>")
```text

The TOTP secret is required for agent enrollment. This returns:

- **TOTP secret**: The secret key for generating one-time passwords
- **QR code**: For scanning with an authenticator app
- **Generation timestamp**: When the secret was created

---

### Step 5: Check Provisioning Keys (Nonces)

**List available provisioning keys:**

```text
zms_list_nonces(page=1, page_size=50)
```text

Provisioning keys (nonces) are one-time keys used to register new agents. For each key, check:

- **name**: Descriptive key name
- **value**: The actual provisioning key string
- **maxUsage**: Maximum number of times the key can be used
- **usageCount**: How many times the key has been used
- **agentGroup**: Which agent group the key enrolls agents into
- **productType**: The product type the key is for
- **created/modified**: Key timestamps

**Common provisioning key issues:**

- **Key exhausted**: `usageCount >= maxUsage` -- create a new key (via `nonceCreate` mutation in the API/portal)
- **Wrong agent group**: Key enrolls into the wrong group -- verify with `zms_get_nonce()`
- **Expired key**: Key was created long ago and may have been deactivated
- **Need key + new group**: The API supports `nonceWithAgentGroupCreate` to create both a provisioning key and a new agent group simultaneously (via portal/API only)

**Search for a specific key:**

```text
zms_list_nonces(search="<key_name>")
```text

**Get specific key details:**

```text
zms_get_nonce(eyez_id="<nonce_eyez_id>")
```text

---

### Step 6: Present Diagnosis and Resolution

#### Diagnosis Template

```text
ZMS Agent Deployment Troubleshooting Report
=============================================
Date: <current_date>
Reported by: <administrator>

## Issue Summary

- **Symptom:** <Agent not connecting / Enrollment failing / Version mismatch>
- **Affected:** <hostname(s) / agent group / all agents>
- **Duration:** <Since when>
- **Severity:** <Critical / High / Medium / Low>
- **Status:** <Root cause identified / Investigating>

---

## Fleet Health Overview

| Metric              | Value       | Status     |
|--------------------|------------|-----------|
| Total Agents        | 245        | --         |
| Connected           | 238 (97%)  | Healthy    |
| Disconnected        | 7 (3%)     | 3 expected |
| Agent Versions      | 3 in use   | Review     |
| Latest Version      | v4.2.1     | Target     |

---

## Investigation Results

### Agent Status
- **Hostname:** web-srv-03
- **Connection Status:** DISCONNECTED
- **Last Seen:** 3 days ago
- **Version:** v4.1.8 (OUTDATED -- latest is v4.2.1)
- **OS:** Ubuntu 22.04
- **Agent Group:** Production Web Servers
- **IP Address:** 10.0.1.45

### Agent Group Configuration
- **Group:** Production Web Servers
- **Auto-Upgrade:** Disabled
- **Tamper Protection:** Enabled
- **Policy Status:** Active
- **Agent Count:** 12

### Provisioning Key Status
- **Available Keys:** 3
- **"Prod Web Key":** 45/50 used (90% -- nearing limit)
- **"Staging Key":** 12/100 used (12% -- available)

---

## Root Cause

<Describe the identified root cause>

Examples:
- Agent disconnected due to network configuration change on the host
- Enrollment failing because provisioning key has reached max usage
- Agent running outdated version that is incompatible with current policies
- Agent group auto-upgrade is disabled, preventing version updates

---