infra-manage-ssh-services

What this skill does

Works with SSH commands, Docker remote management, and infrastructure health checks.
# Infrastructure SSH Service Management

## Quick Start

**Discover available infrastructure:**
```bash
# List all hosts and their status
ping -c 1 -W 1 infra.local && echo "✅ infra.local (primary)" || echo "❌ infra.local"
ping -c 1 -W 1 192.168.68.135 && echo "✅ deus (development)" || echo "❌ deus"
ping -c 1 -W 1 homeassistant.local && echo "✅ homeassistant.local" || echo "❌ homeassistant.local"
```

**Check primary infrastructure services:**
```bash
# View all running Docker services on infra.local
ssh infra "docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"

# Quick MongoDB health check (MongoDB 4.4 uses 'mongo' not 'mongosh')
ssh infra "docker exec local-infra-mongodb-1 mongo off --quiet --eval 'db.runCommand({ping: 1})'" 2>/dev/null
```

**Before using remote MongoDB (NomNom project):**
```bash
# Verify MongoDB is accessible
nc -z infra.local 27017 && echo "✅ MongoDB port open" || echo "❌ MongoDB unreachable"
```

## Connection Reference

**To connect to infra.local, you have three equivalent options:**

```bash
# Option 1: Use the connect function (recommended)
connect infra

# Option 2: Use the SSH alias from ~/.ssh/config
ssh infra

# Option 3: Use the full hostname
ssh [email protected]
```

**All three commands do the same thing:**
- Connect to `infra.local`
- Authenticate as user `dawiddutoit`
- Use SSH key `~/.ssh/id_ed25519`

**First-time setup (if SSH key not yet copied):**
```bash
connect infra --setup
# This copies your SSH public key to infra.local for passwordless authentication
```

**For other hosts:**
```bash
connect deus        # or: ssh deus        # or: ssh [email protected]
connect ha          # or: ssh ha          # or: ssh [email protected]
connect motor       # or: ssh motor       # or: ssh [email protected]
connect armitage    # or: ssh [email protected]
```

**Running commands on infra.local (without interactive shell):**
```bash
# Execute single command
ssh infra "docker ps"

# Execute multiple commands
ssh infra "cd ~/projects/local-infra && docker compose ps"

# Chain commands
ssh infra "docker ps -f name=mongodb && docker logs --tail 10 local-infra-mongodb-1"
```

## Table of Contents

1. [When to Use This Skill](#when-to-use-this-skill)
2. [What This Skill Does](#what-this-skill-does)
3. [Instructions](#instructions)
   - 3.1 [Discovery Phase - Find Available Hosts and Services](#31-discovery-phase)
   - 3.2 [Health Check Phase - Verify Connectivity](#32-health-check-phase)
   - 3.3 [Execution Phase - Manage Services](#33-execution-phase)
4. [Supporting Files](#supporting-files)
5. [Common Workflows](#common-workflows)
6. [Expected Outcomes](#expected-outcomes)
7. [Integration Points](#integration-points)
8. [Expected Benefits](#expected-benefits)
9. [Requirements](#requirements)
10. [Red Flags to Avoid](#red-flags-to-avoid)

## When to Use This Skill

### Explicit Triggers (User Requests)
- "Check infrastructure status"
- "Connect to infra/deus/ha"
- "View Docker services on infra"
- "Test MongoDB connectivity"
- "What services are running on infra.local?"
- "Troubleshoot remote MongoDB connection"
- "Check Langfuse status"
- "View OTLP collector logs"

### Implicit Triggers (Contextual Needs)
- Before using remote MongoDB in NomNom project
- When remote service connection fails (MongoDB, Neo4j, Langfuse)
- Before starting development session that uses remote resources
- When planning to use OpenTelemetry/Langfuse observability
- When investigating service availability for integration work

### Debugging/Troubleshooting Triggers
- Connection refused errors to infra.local services
- MongoDB ServerSelectionTimeoutError
- SSH authentication failures
- Docker container not responding
- Service appears running but not accessible
- Neo4j or Infinity in restart loop

## What This Skill Does

This skill provides systematic workflows for:

1. **Service Discovery** - Identify available hosts (5 total) and running services (16+ on infra.local)
2. **Connectivity Testing** - Verify network reachability, port availability, SSH access
3. **Docker Management** - View, restart, and monitor remote Docker containers
4. **Health Verification** - Check service health status and logs
5. **Troubleshooting** - Diagnose connection issues and service failures
6. **Infrastructure Integration** - Ensure remote resources (MongoDB, Langfuse, OTLP) are ready for use

## Instructions

### 3.1 Discovery Phase

**Step 1: Identify Target Host**

Use the `connect` function to determine which host you need:

```bash
# View available hosts
connect
# Output: Hosts: infra, armitage, deus, ha, motor
```

**Infrastructure Inventory:**

| Host | Connection | Status | Primary Services |
|------|------------|--------|------------------|
| **infra.local** | `connect infra` | ✅ Online | MongoDB, Langfuse, OTLP, Jaeger, Neo4j, Infinity, PostgreSQL, Redis, MinIO, Mosquitto, Caddy |
| **deus** | `connect deus` | ✅ Online | None detected (development machine) |
| **homeassistant.local** | `connect ha` | ✅ Online | Home Assistant (port 8123) |
| **pi4-motor.local** | `connect motor` | ❌ Offline | Motor control (Raspberry Pi 4) |
| **armitage.local** | `connect armitage` | ❌ Offline | Neo4j, Infinity Embeddings (WSL2 PC) |

**Step 2: Test Host Reachability**

```bash
# Quick network ping test
ping -c 1 -W 1 infra.local

# Test specific port availability
nc -z infra.local 27017  # MongoDB
nc -z infra.local 3000   # Langfuse
nc -z infra.local 4317   # OTLP Collector
nc -z infra.local 7687   # Neo4j (if not in restart loop)
```

**Step 3: Discover Running Services**

```bash
# View all Docker containers on infra.local
ssh infra "docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"

# Count running services
ssh infra "docker ps --format '{{.Names}}' | wc -l"

# Check specific service
ssh infra "docker ps -f name=mongodb"
```

### 3.2 Health Check Phase

**Step 1: Verify SSH Connectivity**

```bash
# Test basic SSH connection
ssh infra "echo 'Connection OK'"

# If SSH fails, check SSH agent
ssh-add -l

# Copy SSH key if needed (first-time setup)
connect infra --setup
```

**Step 2: Check Service Health**

```bash
# MongoDB health check
ssh infra "docker inspect --format='{{.State.Health.Status}}' local-infra-mongodb-1"
ssh infra "docker exec local-infra-mongodb-1 mongo off --quiet --eval 'db.runCommand({ping: 1})'"

# Langfuse health check (HTTP)
curl -s -o /dev/null -w "%{http_code}" http://infra.local:3000

# OTLP Collector health check
ssh infra "docker inspect --format='{{.State.Status}}' local-infra-otel-collector-1"

# View container logs for errors
ssh infra "docker logs --tail 50 local-infra-mongodb-1"
```

**Step 3: Verify Application-Level Connectivity**

For **MongoDB** (NomNom project):
```bash
# Test from application environment
cd ~/projects/play/nomnom
python -c "from motor.motor_asyncio import AsyncIOMotorClient; import asyncio; asyncio.run(AsyncIOMotorClient('mongodb://infra.local:27017').admin.command('ping'))" && echo "✅ MongoDB reachable"
```

For **Langfuse**:
```bash
# Check web UI accessibility
curl -I http://infra.local:3000 | grep "HTTP"
```

### 3.3 Execution Phase

**Service Management Commands:**

```bash
# Restart single service
ssh infra "cd ~/projects/local-infra && docker compose restart mongodb"

# Restart all services
ssh infra "cd ~/projects/local-infra && docker compose restart"

# Stop service
ssh infra "cd ~/projects/local-infra && docker compose stop mongodb"

# Start service
ssh infra "cd ~/projects/local-infra && docker compose up -d mongodb"

# View Docker Compose configuration
ssh infra "cd ~/projects/local-infra && docker compose config"
```

**Monitoring Commands:**

```bash
# Follow logs in real-time
ssh infra "docker logs -f local-infra-mongodb-1"

# View last 100 lines
ssh infra "docker logs --tail 100 local-infra-langfuse-web-1"

# View logs for all services
ssh infra "cd ~/proje