zpa-troubleshoot-app-connector

What this skill does

# ZPA: Troubleshoot App Connector

## Keywords

app connector, connector down, connector not enrolling, enrollment failure, connector upgrade, connector high cpu, connector memory, connector troubleshoot, connector status, connector group, provisioning key, broker connection, public service edge, connector health

## Overview

Troubleshoot ZPA App Connector issues by combining MCP API inspection (connector groups, provisioning keys, server groups) with operational runbook knowledge. This skill covers the four major App Connector failure categories: enrollment failures, upgrade issues, Public Service Edge connectivity, and resource utilization.

**Use this skill when:** An administrator reports App Connector problems -- connector not appearing, enrollment failures, upgrade failures, high resource usage, or application unreachability traced to a connector issue.

---

## Workflow

### Step 1: Gather Issue Details

Collect from the administrator:

**Required:**

- Connector name or the App Connector Group it belongs to
- Symptom: not enrolling, showing disconnected, upgrade failed, high CPU/memory, applications unreachable

**Helpful:**

- When did the issue start?
- Was anything changed recently (provisioning key, network, firewall)?
- Is it one connector or multiple connectors in the group?
- Cloud name (e.g., prod.zpath.net, zpatwo.net)

---

### Step 1.5: Config-Only Pre-Flight Checks (baseline alignment)

Before concluding a connector is broken, verify the deployment matches ZPA Baseline Recommendations v1.0 §App Connector Recommendations. These are **configuration checks only** — ZPA does not expose live connector telemetry (CPU, memory, throughput, bandwidth utilization), so we cannot prove a connector is *currently* overloaded; we can only verify the deployment shape.

| Pre-flight check | How to verify | Doc reference |
|---|---|---|
| AC group is in the **same DC / VPC** as the apps it serves | Compare AC group `latitude` / `longitude` / `location` to the application server location | Page 11 — "directly adjacent to application workloads" |
| AC group has **N+1 connectors** | Count connectors in `zpa_list_app_connectors` filtered by `app_connector_group_id`. Doc formula: `N = ceil(Total_Mbps / 500); deploy N+1`. | Page 11 |
| **SIPA / sensitive / LSS connectors are isolated** in their own AC groups | Inspect AC group naming and bound server groups — they should not share AC groups with general traffic | Page 11 |
| AC group has unrestricted outbound to ZPA cloud + redundant DNS | Out-of-band check (network team) — not visible via API | Page 13 |

**Important — what we cannot check via API:** per-connector CPU/memory/throughput, app probe results, bandwidth utilization, session counts. If the symptom is "connector is slow" and the config checks above pass, the next step is operational telemetry from outside ZPA (hypervisor / cloud-provider metrics, syslog from the connector VM). Flag this gap to the user so they don't expect the API to return runtime metrics.

---

### Step 2: Inspect Connector and Connector Group Status via API

**List individual app connectors** to get their runtime status directly:

```text
zpa_list_app_connectors(search="<connector_name>")
```text

Get detailed info for a specific connector:

```text
zpa_get_app_connector(connector_id="<connector_id>")
```text

**Check for:**

- `runtime_status` -- `ZPN_STATUS_AUTHENTICATED` (healthy), `ZPN_STATUS_DISCONNECTED`, `ZPN_STATUS_NOT_ENROLLED`
- `current_version` -- software version running
- `expected_version` -- version it should be running (upgrade needed if different)
- `last_broker_connect_time` -- when it last connected to ZPA cloud
- `last_broker_disconnect_time` -- when it last disconnected
- `control_channel_status` -- control connection state
- `connector_group_id` -- which group this connector belongs to
- `private_ip` / `public_ip` -- network addresses
- `platform` -- OS/hypervisor platform
- `enrollment_cert` -- certificate details and expiry

**Then inspect the connector group** for group-level settings:

```text
zpa_list_app_connector_groups(search="<connector_group_name>")
zpa_get_app_connector_group(group_id="<group_id>")
```text

**Check group settings:**

- `enabled` -- is the group enabled?
- `upgrade_day` and `upgrade_time_in_secs` -- maintenance window
- `version_profile` -- version track (Default, Previous Default, New Release)

**Connector status values:**

| Status | Meaning |
|--------|---------|
| `ZPN_STATUS_AUTHENTICATED` | Healthy, control connection established |
| `ZPN_STATUS_DISCONNECTED` | Lost connection to ZPA cloud |
| `ZPN_STATUS_NOT_ENROLLED` | Never enrolled or enrollment failed |
| `ZPN_STATUS_PENDING` | Enrollment in progress |

---

### Step 3: Check Provisioning Keys

Provisioning key issues are the #1 cause of enrollment failures.

```text
zpa_list_provisioning_keys(key_type="connector")
```text

For a specific key:

```text
zpa_get_provisioning_key(key_id="<key_id>", key_type="connector")
```text

**Check for:**

- `max_usage` vs current enrollment count -- if equal, no new enrollments can use this key
- `enabled` -- is the key active?
- The `component_id` must match the target App Connector Group
- Key must not be expired

**If max usage is reached:**

```text
The provisioning key has reached its maximum enrollment count.

Resolution: Increase the max_usage value or create a new provisioning key
for this connector group.
```text

```text
zpa_update_provisioning_key(
  key_id="<key_id>",
  key_type="connector",
  max_usage=<current + needed>
)
```text

Or create a new key:

```text
zpa_create_provisioning_key(
  name="<name>",
  key_type="connector",
  max_usage=10,
  component_id="<connector_group_id>",
  enrollment_cert_id="<cert_id>"
)
```text

---

### Step 4: Check Server Groups and Application Segments

If applications are unreachable through the connector, verify the infrastructure chain.

```text
zpa_list_server_groups()
```text

Find server groups that reference the affected connector group:

```text
zpa_get_server_group(group_id="<server_group_id>")
```text

**Check for:**

- `app_connector_groups` -- does it reference the affected connector group?
- `enabled` -- is the server group active?
- `servers` -- are application servers configured?

Then check which application segments use this server group:

```text
zpa_list_application_segments()
```text

**If applications are unreachable:**

- Verify the domain names in the application segment match what users access
- Verify ports are correct
- Confirm the server group → connector group → connector chain is intact
- Check that connectors can reach the internal application servers (network-level)

---

### Step 5: Diagnose by Symptom Category

#### 5A: Enrollment Failure

If the connector shows `ZPN_STATUS_NOT_ENROLLED`:

1. **Provisioning key issues** (most common):
   - Key max_usage reached (Step 3 above)
   - Key copied incorrectly -- administrator should re-copy the exact key
   - Key associated with wrong connector group

2. **DNS resolution failure:**
   - Connector must resolve `co2br.<cloudname>.net`
   - Check DNS servers in `/etc/resolv.conf` on the connector
   - Common clouds: `prod.zpath.net`, `zpatwo.net`

3. **Network connectivity:**
   - Connector needs outbound TCP 443 to ZPA Public Service Edges
   - Check firewall rules per `config.zscaler.com/<cloudname>/zpa`
   - SSL interception on the path will break enrollment (connector logs show "self signed certificate in certificate chain")

4. **Certificate/time issues:**
   - If connector clock is skewed, certificate validation fails
   - Logs show: "certificate is not yet valid" or similar
   - Resolution: sync NTP on the connector host

5. **VM migration issues:**
   - If connector was cloned/migrated (changed MAC/hardware ID), enrollment breaks
   - Logs show: "Cannot decrypt data from instance_id.crypt"
   - Resolution: wipe and re-provision the connector

**Wipe and re-provision procedure:**

```text
1. Sto