debugging-techniques

What this skill does

# Debugging Techniques

## Purpose

Provides systematic debugging workflows for local, remote, container, and production environments across Python, Go, Rust, and Node.js. Covers interactive debuggers, container debugging with ephemeral containers, and production-safe techniques using correlation IDs and distributed tracing.

## When to Use This Skill

Trigger this skill for:
- Setting breakpoints in Python, Go, Rust, or Node.js code
- Debugging running containers or Kubernetes pods
- Setting up remote debugging connections
- Safely debugging production issues
- Inspecting goroutines, threads, or async tasks
- Analyzing core dumps or stack traces
- Choosing the right debugging tool for a scenario

## Quick Reference by Language

### Python Debugging

**Built-in: pdb**
```python
# Python 3.7+
def buggy_function(x, y):
    breakpoint()  # Stops execution here
    return x / y

# Older Python
import pdb
pdb.set_trace()
```

**Essential pdb commands:**
- `list` (l) - Show code around current line
- `next` (n) - Execute current line, step over functions
- `step` (s) - Execute current line, step into functions
- `continue` (c) - Continue until next breakpoint
- `print var` (p) - Print variable value
- `where` (w) - Show stack trace
- `quit` (q) - Exit debugger

**Enhanced tools:**
- `ipdb` - Enhanced pdb with tab completion, syntax highlighting (`pip install ipdb`)
- `pudb` - Terminal GUI debugger (`pip install pudb`)
- `debugpy` - VS Code integration (included in Python extension)

**Debugging tests:**
```bash
pytest --pdb  # Drop into debugger on test failure
```

For detailed Python debugging patterns, see `references/python-debugging.md`.

### Go Debugging

**Delve - Official Go debugger**

**Installation:**
```bash
go install github.com/go-delve/delve/cmd/dlv@latest
```

**Basic usage:**
```bash
dlv debug main.go              # Debug main package
dlv test github.com/me/pkg     # Debug test suite
dlv attach <pid>               # Attach to running process
dlv debug -- --config prod.yaml  # Pass arguments
```

**Essential commands:**
- `break main.main` (b) - Set breakpoint at function
- `break file.go:10` (b) - Set breakpoint at line
- `continue` (c) - Continue execution
- `next` (n) - Step over
- `step` (s) - Step into
- `print x` (p) - Print variable
- `goroutine` (gr) - Show current goroutine
- `goroutines` (grs) - List all goroutines
- `goroutines -t` - Show goroutine stacktraces
- `stack` (bt) - Show stack trace

**Goroutine debugging:**
```bash
(dlv) goroutines                 # List all goroutines
(dlv) goroutines -t              # Show stacktraces
(dlv) goroutines -with user      # Filter user goroutines
(dlv) goroutine 5                # Switch to goroutine 5
```

For detailed Go debugging patterns, see `references/go-debugging.md`.

### Rust Debugging

**LLDB - Default Rust debugger**

**Compilation:**
```bash
cargo build  # Debug build includes symbols by default
```

**Usage:**
```bash
rust-lldb target/debug/myapp   # LLDB wrapper for Rust
rust-gdb target/debug/myapp    # GDB wrapper (alternative)
```

**Essential LLDB commands:**
- `breakpoint set -f main.rs -l 10` - Set breakpoint at line
- `breakpoint set -n main` - Set breakpoint at function
- `run` (r) - Start program
- `continue` (c) - Continue execution
- `next` (n) - Step over
- `step` (s) - Step into
- `print variable` (p) - Print variable
- `frame variable` (fr v) - Show local variables
- `backtrace` (bt) - Show stack trace
- `thread list` - List all threads

**VS Code integration:**
- Install CodeLLDB extension (`vadimcn.vscode-lldb`)
- Configure `launch.json` for Rust projects

For detailed Rust debugging patterns, see `references/rust-debugging.md`.

### Node.js Debugging

**Built-in: node --inspect**

**Basic usage:**
```bash
node --inspect-brk app.js       # Start and pause immediately
node --inspect app.js           # Start and run
node --inspect=0.0.0.0:9229 app.js  # Specify host/port
```

**Chrome DevTools:**
1. Open `chrome://inspect`
2. Click "Open dedicated DevTools for Node"
3. Set breakpoints, inspect variables

**VS Code integration:**
Configure `launch.json`:
```json
{
  "type": "node",
  "request": "launch",
  "name": "Launch Program",
  "program": "${workspaceFolder}/app.js"
}
```

**Docker debugging:**
```dockerfile
EXPOSE 9229
CMD ["node", "--inspect=0.0.0.0:9229", "app.js"]
```

For detailed Node.js debugging patterns, see `references/nodejs-debugging.md`.

## Container & Kubernetes Debugging

### kubectl debug with Ephemeral Containers

**When to use:**
- Container has crashed (kubectl exec won't work)
- Using distroless/minimal image (no shell, no tools)
- Need debugging tools without rebuilding image
- Debugging network issues

**Basic usage:**
```bash
# Add ephemeral debugging container
kubectl debug -it <pod-name> --image=nicolaka/netshoot

# Share process namespace (see other container processes)
kubectl debug -it <pod-name> --image=busybox --share-processes

# Target specific container
kubectl debug -it <pod-name> --image=busybox --target=app
```

**Recommended debugging images:**
- `nicolaka/netshoot` (~380MB) - Network debugging (curl, dig, tcpdump, netstat)
- `busybox` (~1MB) - Minimal shell and utilities
- `alpine` (~5MB) - Lightweight with package manager
- `ubuntu` (~70MB) - Full environment

**Node debugging:**
```bash
kubectl debug node/<node-name> -it --image=ubuntu
```

**Docker container debugging:**
```bash
docker exec -it <container-id> sh

# If no shell available
docker run -it --pid=container:<container-id> \
           --net=container:<container-id> \
           busybox sh
```

For detailed container debugging patterns, see `references/container-debugging.md`.

## Production Debugging

### Production Debugging Principles

**Golden rules:**
1. **Minimal performance impact** - Profile overhead, limit scope
2. **No blocking operations** - Use non-breaking techniques
3. **Security-aware** - Avoid logging secrets, PII
4. **Reversible** - Can roll back quickly (feature flags, Git)
5. **Observable** - Structured logging, correlation IDs, tracing

### Safe Production Techniques

**1. Structured Logging**
```python
import logging
import json

logger = logging.getLogger(__name__)
logger.info(json.dumps({
    "event": "user_login_failed",
    "user_id": user_id,
    "error": str(e),
    "correlation_id": request_id
}))
```

**2. Correlation IDs (Request Tracing)**
```go
func handleRequest(w http.ResponseWriter, r *http.Request) {
    correlationID := r.Header.Get("X-Correlation-ID")
    if correlationID == "" {
        correlationID = generateUUID()
    }
    ctx := context.WithValue(r.Context(), "correlationID", correlationID)
    log.Printf("[%s] Processing request", correlationID)
}
```

**3. Distributed Tracing (OpenTelemetry)**
```python
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def process_order(order_id):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        span.add_event("Order validated")
```

**4. Error Tracking Platforms**
- Sentry - Exception tracking with context
- New Relic - APM with error tracking
- Datadog - Logs, metrics, traces
- Rollbar - Error monitoring

**Production debugging workflow:**
1. **Detect** - Error tracking alert, log spike, metric anomaly
2. **Locate** - Find correlation ID, search logs, view distributed trace
3. **Reproduce** - Try to reproduce in staging with production data (sanitized)
4. **Fix** - Create feature flag, deploy to canary first
5. **Verify** - Check error rates, review logs, monitor traces

For detailed production debugging patterns, see `references/production-debugging.md`.

## Decision Framework

### Which Debugger for Which Language?

| Language | Primary Tool | Installation | Best For |
|----------|-------------|--------------|----------|
| **Python** | pdb | Built-in | Simple scripts, server environments |
| | ipdb | `pip install ipdb` | Enhanced UX, IPython users |
| | debugpy | VS Code extensio