debug:docker

What this skill does

# Docker Debugging Guide

This guide provides a systematic approach to debugging Docker containers, images, networks, and volumes. Follow the four-phase methodology for efficient problem resolution.

## The Four Phases of Docker Debugging

### Phase 1: Quick Assessment (30 seconds)
Get immediate context about the issue.

```bash
# Check container status
docker ps -a

# View recent logs (last 50 lines)
docker logs --tail 50 <container>

# Check container exit code
docker inspect <container> --format='{{.State.ExitCode}}'

# Quick health check
docker inspect <container> --format='{{.State.Health.Status}}'
```

### Phase 2: Log Analysis (2-5 minutes)
Deep dive into container logs and events.

```bash
# Follow logs in real-time
docker logs -f <container>

# View logs with timestamps
docker logs --timestamps <container>

# View logs since specific time
docker logs --since 30m <container>

# Check Docker daemon events
docker events --since 1h

# View system-wide logs
journalctl -u docker.service --since "1 hour ago"
```

### Phase 3: Interactive Investigation (5-15 minutes)
Get hands-on access to the container environment.

```bash
# Open shell in running container
docker exec -it <container> /bin/sh
# or
docker exec -it <container> /bin/bash

# Run commands without shell
docker exec <container> cat /etc/hosts
docker exec <container> env

# Use docker debug for enhanced debugging (Docker Desktop 4.27+)
docker debug <container>

# Inspect container configuration
docker inspect <container>

# Check network configuration
docker network inspect <network>
```

### Phase 4: Deep Analysis (15+ minutes)
Comprehensive investigation for complex issues.

```bash
# Monitor resource usage
docker stats <container>

# Check disk usage
docker system df -v

# Inspect image layers
docker history <image>

# Export container filesystem for analysis
docker export <container> -o container.tar

# View detailed container info
docker inspect <container> | jq '.'
```

---

## Common Error Patterns and Solutions

### Exit Codes

| Exit Code | Meaning | Common Causes | Solution |
|-----------|---------|---------------|----------|
| **0** | Success | Normal termination | No action needed |
| **1** | General error | Application error, missing file | Check logs, verify files exist |
| **126** | Permission problem | Cannot execute command | Check file permissions, add execute bit |
| **127** | Command not found | Missing binary or PATH issue | Verify command exists in image |
| **137** | SIGKILL (OOM) | Out of memory | Increase memory limit, optimize app |
| **139** | SIGSEGV | Segmentation fault | Debug application code |
| **143** | SIGTERM | Graceful shutdown | Normal behavior during stop |
| **255** | Exit status out of range | Various | Check application error handling |

### OOM Killed Containers (Exit Code 137)

```bash
# Check if container was OOM killed
docker inspect <container> --format='{{.State.OOMKilled}}'

# View memory limits
docker inspect <container> --format='{{.HostConfig.Memory}}'

# Run with increased memory
docker run -m 2g --memory-swap 4g <image>

# Monitor memory in real-time
docker stats --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"
```

### Image Build Failures

```bash
# Build with verbose output
docker build --progress=plain -t <image> .

# Build without cache
docker build --no-cache -t <image> .

# Build specific stage for debugging
docker build --target <stage-name> -t <image> .

# Inspect build cache
docker builder prune --dry-run

# Debug failed layer by running from previous successful layer
docker run -it <last-successful-layer-id> /bin/sh
```

**Common Build Issues:**
- `COPY failed: file not found` - Check paths are relative to build context
- `RUN failed` - Check command syntax, ensure dependencies are installed
- `Permission denied` - Add `chmod` or run as appropriate user

### Networking Issues

```bash
# List networks
docker network ls

# Inspect network
docker network inspect <network>

# Check container network settings
docker inspect <container> --format='{{json .NetworkSettings.Networks}}'

# Test connectivity between containers
docker exec <container1> ping <container2>

# Test DNS resolution
docker exec <container> nslookup <hostname>

# Check exposed ports
docker port <container>

# Debug with network tools
docker run --rm --network=<network> nicolaka/netshoot ping <target>
```

**Common Network Issues:**
- Containers on different networks cannot communicate
- Port already in use: `lsof -i :<port>` to find conflicting process
- DNS resolution fails: Check Docker DNS settings

### Volume Mount Problems

```bash
# List volumes
docker volume ls

# Inspect volume
docker volume inspect <volume>

# Check mount points
docker inspect <container> --format='{{json .Mounts}}'

# Verify host path exists
ls -la /path/to/host/directory

# Check permissions inside container
docker exec <container> ls -la /mount/path

# Test with simple container
docker run --rm -v /host/path:/container/path alpine ls -la /container/path
```

**Common Volume Issues:**
- `Permission denied` - Check UID/GID mapping, use `:z` or `:Z` for SELinux
- Path not found - Ensure host path exists before mounting
- Windows paths - Use forward slashes or escaped backslashes

### Permission Denied Errors

```bash
# Add user to docker group (Linux)
sudo usermod -aG docker $USER
newgrp docker

# Check Docker socket permissions
ls -la /var/run/docker.sock

# Run container as specific user
docker run --user $(id -u):$(id -g) <image>

# Fix file permissions in Dockerfile
RUN chown -R appuser:appuser /app
USER appuser
```

### Container Exits Immediately

```bash
# Check what command is running
docker inspect <container> --format='{{.Config.Cmd}}'
docker inspect <container> --format='{{.Config.Entrypoint}}'

# Keep container running for debugging
docker run -d <image> tail -f /dev/null
# or
docker run -d <image> sleep infinity

# Override entrypoint for debugging
docker run -it --entrypoint /bin/sh <image>

# Check if process is foreground
# (Docker needs a foreground process to keep running)
```

### Docker Desktop Won't Start

**Windows:**
- Enable Hyper-V: `Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All`
- Enable WSL2: `wsl --install`
- Check virtualization in BIOS (VT-x/AMD-V)
- Temporarily disable antivirus

**Mac:**
- Check available disk space
- Reset Docker Desktop: Delete `~/Library/Group\ Containers/group.com.docker/`
- Reinstall Docker Desktop

**Linux:**
- Check Docker daemon: `sudo systemctl status docker`
- View daemon logs: `journalctl -u docker.service`

---

## Debugging Tools Reference

### Essential Commands

```bash
# Container lifecycle
docker ps -a                          # List all containers
docker logs <container>               # View logs
docker logs -f --tail 100 <container> # Follow last 100 lines
docker exec -it <container> sh        # Interactive shell
docker inspect <container>            # Full container details
docker top <container>                # Running processes

# Images
docker images                         # List images
docker history <image>                # Show image layers
docker inspect <image>                # Image details

# System
docker system df                      # Disk usage
docker system events                  # Real-time events
docker system info                    # System-wide info
docker system prune                   # Clean up unused resources

# Network
docker network ls                     # List networks
docker network inspect <network>      # Network details

# Volumes
docker volume ls                      # List volumes
docker volume inspect <volume>        # Volume details
```

### Advanced Debugging

```bash
# Docker debug (Docker Desktop 4.27+)
docker debug <container>              # Enhanced shell with tools

# Process inspection
docker exec <container> ps aux        # List processes
docker exec <container> top           # Interactive process viewer

# Network debugging
docker ex