arista-device-health

What this skill does

# Arista EOS Device Health Check

Structured triage procedure for assessing Arista EOS device health. Produces a
prioritized findings report with severity classifications and recommended actions.

EOS runs on a Linux kernel with individual processes (agents) managing each
protocol and subsystem. This architecture means Linux-native tools (`bash top`,
`bash df`, `bash dmesg`) are valid supplements to EOS show commands. Agent
health is a first-class concern — a crashed or stuck agent can silently affect
a subsystem even when aggregate device metrics look normal.

The procedure covers core device health first, then extends to MLAG and
VXLAN/EVPN for data center deployments.

## When to Use

- Device reported as slow, dropping traffic, or unresponsive
- Scheduled health audit of Arista switches (leaf, spine, or border)
- Post-change verification after upgrades, patches, or configuration changes
- Capacity planning data collection for CPU, memory, and link utilization
- Incident response when an Arista switch is suspected as the fault domain
- MLAG inconsistency or split-brain detection in a data center pair
- VXLAN/EVPN overlay health assessment — VTEP reachability, BGP EVPN peering
- Agent crash or restart detected in syslog

## Prerequisites

- SSH or console access to the device (privilege 1 minimum)
- EOS 4.28 or later (commands validated against EOS 4.30+)
- Network reachability to management interface confirmed
- Awareness of the device's normal baseline (CPU, memory, traffic patterns)
- For MLAG configurations: access to both MLAG peers for cross-validation
- For VXLAN/EVPN: knowledge of expected VTEP count and BGP EVPN peer list

## Procedure

Follow this sequence. Core device health (Steps 1–5) applies to all deployments.
Steps 6–7 are data center extensions for MLAG and VXLAN/EVPN — skip if not
configured on this device.

### Step 1: Establish Baseline Context

```
show version
show uptime
show clock
show inventory
```

Record: hostname, EOS version, uptime, hardware model, total memory, current time.
Short uptime indicates a recent reload or crash — check `show reload cause` for
the trigger. Note the hardware platform — Arista's 7050X, 7280R, 7500R, and
720X series have different memory and CPU profiles.

### Step 2: CPU and Memory Assessment

EOS reports CPU and memory through both EOS commands and Linux-native tools.

```
show processes top once
show version | include Free memory
```

**Linux-native alternative** (provides more granularity):
```
bash timeout 5 top -bn1 | head 20
bash free -m
```

`show processes top once` shows per-process CPU and memory similar to Linux `top`.
EOS CPU is typically consumed by agents — each protocol has its own process.

Key agents to watch:
- **Stp** — Spanning Tree: high CPU indicates topology instability
- **Rib** — Routing Information Base: route churn or large table operations
- **Ebra** — EOS Bridge Agent: L2 forwarding, MAC learning storms
- **IpRib** — IP RIB agent: routing table processing
- **Bgp** — BGP agent: peer negotiations, route processing
- **Fap-sobek** / **Sand** — ASIC agents: forwarding plane, hardware programming

EOS memory: `show version` includes total and free memory. Free memory below
20% of total is warning. Linux `free -m` provides buffer/cache detail — EOS
uses Linux page cache aggressively, so "used" memory includes reclaimable cache.

### Step 3: Agent and Daemon Health

EOS runs each subsystem as an independent agent. A crashed or stuck agent
affects its subsystem silently — device-level metrics may look normal while
one protocol is completely inoperative.

```
show agent
show agent [name] logs | tail 20
show logging last 50 | include AGENT|agent|crash|restart
```

`show agent` lists all EOS agents with their PID, state, and uptime.

Healthy state: all agents show `running` with uptimes matching device uptime.
Red flags:
- Agent with shorter uptime than device → it crashed and restarted
- Agent in `crashed` or `not running` state → subsystem is down
- Multiple agent restarts in logs → recurring instability

If an agent shows recent restart, check its logs: `show agent [name] logs`.
Common restart causes: memory exhaustion, uncaught exception, watchdog timeout.

### Step 4: Interface Health

```
show interfaces status
show interfaces counters errors
show interfaces counters discards
```

For interfaces with errors:
```
show interfaces [name]
show interfaces [name] transceiver
```

Error classification:
- CRC / FCS errors → Layer 1 (cabling, optics, SFP seating)
- Input errors without CRC → buffer overruns, MTU mismatch
- Alignment errors → duplex mismatch or L1 issue
- Output discards → egress congestion; check QoS or link capacity
- Late collisions → duplex mismatch (should not occur on modern links)

Optics DOM: `show interfaces transceiver` provides Tx/Rx power, temperature,
voltage. Low Rx power (below -10 dBm for most 10G SFP+) indicates fiber or
optic degradation.

### Step 5: Environment and Platform

```
show environment all
show environment cooling
show environment power
show environment temperature
show reload cause
show logging last 30 | include ERR|WARN|traceback|panic
```

Check: all temperature sensors within limits, all power supplies OK, all
fans operational. `show environment all` is a comprehensive single command.

`show reload cause` — if the device recently reloaded, this explains why.
Expected causes: user-initiated, software upgrade. Unexpected causes:
kernel panic, watchdog timeout, power loss.

Review syslog for tracebacks, panics, or agent crashes. EOS logs are also
accessible via `bash cat /var/log/messages | tail 50`.

### Step 6: MLAG Health (Data Center Extension)

Skip this step if MLAG is not configured. MLAG state is often the single most
important health indicator in an Arista data center deployment — a degraded
MLAG pair can cause traffic black-holes or asymmetric forwarding.

```
show mlag
show mlag detail
show mlag interfaces
show mlag config-sanity
```

**MLAG state assessment:**

| Field | Healthy Value | Problem Indicator |
|-------|--------------|-------------------|
| state | active | disabled, inactive |
| negotiation status | connected | connecting, disconnected |
| peer-link status | up | down, errdisabled |
| local-interface status | up | down |
| config-sanity | consistent | inconsistent |

`show mlag detail` provides:
- **Peer address** and **peer-link** interface status
- **Heartbeat** interval and status — lost heartbeats indicate control plane
  reachability issues between peers
- **Reload delay** timers — critical for preventing traffic loss during boot

`show mlag config-sanity` checks for configuration mismatches between peers.
Inconsistencies can cause traffic loops or black-holes. Address any `inconsistent`
result before continuing other triage.

`show mlag interfaces` — verify all MLAG interfaces show `active-full` on both
sides. `active-partial` means one side is down — reduced redundancy.
`disabled` means the MLAG interface is administratively or operationally down.

### Step 7: VXLAN/EVPN Health (Data Center Extension)

Skip this step if VXLAN/EVPN is not configured. This assesses overlay fabric
health for VXLAN deployments using BGP EVPN as the control plane.

```
show vxlan vtep
show vxlan address-table
show interfaces vxlan 1
show bgp evpn summary
```

**VXLAN health checks:**
- `show vxlan vtep` — verify expected VTEP count. Missing VTEPs indicate
  underlay reachability or BGP EVPN peering issues.
- `show interfaces vxlan 1` — VTEP source interface must be up. Flood list
  should match expected remote VTEPs (for flood-and-learn) or be empty
  (for BGP EVPN with ingress replication).
- `show vxlan address-table` — MAC learning across the overlay. Stale or
  missing entries indicate control plane issues.

**BGP EVPN peering:**
- `show bgp evpn summary` — all EVPN peers must be in Established state.
  Peers in Active/Connect state have transport or configuration issues.
- Compare EVPN route count to baseline — si