caddy-https-troubleshoot

What this skill does

# Troubleshoot HTTPS Skill

Systematic diagnosis and resolution of HTTPS/SSL certificate issues in the Caddy reverse proxy with Cloudflare DNS-01 challenge.

## Quick Start

Run the diagnostic script to identify issues:

```bash
/home/dawiddutoit/projects/network/.claude/skills/troubleshoot-https/scripts/diagnose-https.sh
```

Or follow the step-by-step diagnosis in [Instructions](#3-instructions).

## Table of Contents

1. [When to Use This Skill](#1-when-to-use-this-skill)
2. [What This Skill Does](#2-what-this-skill-does)
3. [Instructions](#3-instructions)
   - 3.1 Check API Key Configuration
   - 3.2 Verify Cloudflare DNS Plugin
   - 3.3 Analyze Caddy Logs
   - 3.4 Validate Caddyfile Syntax
   - 3.5 Test Certificate Validity
   - 3.6 Diagnose Specific Error
   - 3.7 Apply Fix
4. [Supporting Files](#4-supporting-files)
5. [Expected Outcomes](#5-expected-outcomes)
6. [Common Error Reference](#6-common-error-reference)
7. [Requirements](#7-requirements)
8. [Red Flags to Avoid](#8-red-flags-to-avoid)

## When to Use This Skill

**Explicit Triggers:**
- "HTTPS not working"
- "SSL certificate error"
- "Certificate not obtained"
- "Caddy keeps restarting"
- "Fix HTTPS certificates"
- "Invalid format for Authorization header"

**Implicit Triggers:**
- Browser shows "Your connection is not private"
- Services accessible via HTTP but not HTTPS
- Caddy container in restart loop
- New domain not getting certificate

**Debugging Triggers:**
- "Why is my certificate expired?"
- "Why can't Caddy get a certificate?"
- "Cloudflare DNS challenge failing"

## What This Skill Does

1. **Checks API Key** - Verifies CLOUDFLARE_API_KEY is set and passed to container
2. **Verifies Plugin** - Confirms Cloudflare DNS plugin is compiled into Caddy
3. **Analyzes Logs** - Searches for certificate errors in Caddy logs
4. **Validates Config** - Tests Caddyfile syntax
5. **Tests Certificates** - Checks certificate validity for each domain
6. **Identifies Error** - Matches symptoms to known issues
7. **Provides Fix** - Gives specific commands to resolve the issue

## Instructions

### 3.1 Check API Key Configuration

**Step 1: Verify API key is set in .env**

```bash
grep "CLOUDFLARE_API_KEY" /home/dawiddutoit/projects/network/.env | head -1
```

Expected: `CLOUDFLARE_API_KEY="Xv5MOdOT...` (starts with alphanumeric, not `v4:`)

**Step 2: Verify API key is passed to container**

```bash
docker exec caddy env | grep CLOUDFLARE_API_KEY
```

Expected: Shows the API key value (not empty)

**If empty or missing:**
```bash
# Recreate container to pick up .env changes
docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml up -d --force-recreate caddy
```

**Key distinction:**
- API Token format: `Xv5MOdOT...` (alphanumeric string) - CORRECT
- Global API Key format: `v4:...` or long hex string - WRONG

### 3.2 Verify Cloudflare DNS Plugin

```bash
docker exec caddy caddy list-modules | grep cloudflare
```

Expected: `dns.providers.cloudflare`

**If not present:** Plugin not compiled into Caddy. Rebuild:

```bash
cd /home/dawiddutoit/projects/network && \
docker compose build --no-cache caddy && \
docker compose up -d --force-recreate caddy
```

### 3.3 Analyze Caddy Logs

**Check for certificate-related messages:**

```bash
docker logs caddy 2>&1 | grep -i -E "certificate|error|failed|invalid" | tail -30
```

**Look for success messages:**

```bash
docker logs caddy 2>&1 | grep "certificate obtained successfully"
```

Expected for each domain: `certificate obtained successfully {"identifier": "domain.temet.ai"}`

### 3.4 Validate Caddyfile Syntax

```bash
docker exec caddy caddy validate --config /etc/caddy/Caddyfile
```

Expected: `Valid configuration` (no errors)

**If syntax error:**
1. Read the Caddyfile to identify the error
2. Fix the syntax issue
3. Reload Caddy

### 3.5 Test Certificate Validity

**Test a single domain:**

```bash
echo | openssl s_client -servername pihole.temet.ai -connect pihole.temet.ai:443 2>/dev/null | openssl x509 -noout -dates -issuer
```

Expected output:
```
notBefore=<date>
notAfter=<date>
issuer=C = US, O = Let's Encrypt, CN = R3
```

**Test all domains:**

```bash
for domain in pihole jaeger langfuse sprinkler ha; do
  echo "=== $domain.temet.ai ==="
  echo | openssl s_client -servername $domain.temet.ai -connect $domain.temet.ai:443 2>/dev/null | \
    openssl x509 -noout -dates 2>&1 || echo "FAILED to get certificate"
  echo
done
```

### 3.6 Diagnose Specific Error

Match error message to diagnosis:

| Error Message | Diagnosis | Go to Fix |
|---------------|-----------|-----------|
| `Invalid format for Authorization header` | Using Global API Key instead of API Token | Fix A |
| `missing API token` | Environment variable not set | Fix B |
| `unknown directive 'dns'` | Cloudflare plugin not compiled | Fix C |
| `certificate obtain error` | Rate limit or DNS propagation | Fix D |
| `403 Forbidden` from Cloudflare | API token lacks permissions | Fix E |
| Container restart loop | Caddyfile syntax error | Fix F |

### 3.7 Apply Fix

**Fix A: Wrong API Key Type**

The error "Invalid format for Authorization header" means you're using the Global API Key instead of an API Token.

1. Create new API Token:
   - Go to: https://dash.cloudflare.com/profile/api-tokens
   - Click "Create Token"
   - Select template: "Edit zone DNS"
   - Zone Resources: Include -> Specific zone -> temet.ai
   - Click "Continue to summary" -> "Create Token"

2. Update .env:
   ```bash
   # Edit .env and replace CLOUDFLARE_API_KEY with the new token
   nano /home/dawiddutoit/projects/network/.env
   ```

3. Recreate Caddy:
   ```bash
   docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml up -d --force-recreate caddy
   ```

4. Verify:
   ```bash
   docker logs caddy 2>&1 | grep "certificate obtained successfully"
   ```

**Fix B: Environment Variable Not Set**

1. Verify .env has the key:
   ```bash
   grep CLOUDFLARE_API_KEY /home/dawiddutoit/projects/network/.env
   ```

2. Verify docker-compose.yml passes it:
   ```bash
   grep -A5 "caddy:" /home/dawiddutoit/projects/network/docker-compose.yml | grep CLOUDFLARE
   ```

3. Recreate container:
   ```bash
   docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml up -d --force-recreate caddy
   ```

**Fix C: Cloudflare Plugin Not Compiled**

```bash
cd /home/dawiddutoit/projects/network && \
docker compose build --no-cache caddy && \
docker compose up -d --force-recreate caddy
```

Build takes approximately 5 minutes on Raspberry Pi.

**Fix D: Rate Limit or DNS Propagation**

1. Wait 5 minutes for DNS propagation
2. Restart Caddy:
   ```bash
   docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml restart caddy
   ```

3. If still failing, check Let's Encrypt rate limits:
   - Limit: 50 certificates per domain per week
   - Check: https://crt.sh/?q=temet.ai

**Fix E: API Token Missing Permissions**

1. Go to: https://dash.cloudflare.com/profile/api-tokens
2. Find your token and click "Edit"
3. Verify permissions:
   - Zone -> DNS -> Edit (required)
   - Zone Resources -> Include -> Specific zone -> temet.ai
4. If missing, create new token with correct permissions

**Fix F: Caddyfile Syntax Error**

1. Check error in logs:
   ```bash
   docker logs caddy 2>&1 | tail -50
   ```

2. Validate Caddyfile:
   ```bash
   docker exec caddy caddy validate --config /etc/caddy/Caddyfile
   ```

3. Common syntax issues:
   - Missing closing braces `}`
   - Invalid directive names
   - Wrong environment variable syntax

4. Fix the Caddyfile and reload:
   ```bash
   docker exec caddy caddy reload --config /etc/caddy/Caddyfile
   ```

## Supporting Files

| File | Purpose |
|------|---------|
| `references/reference.md` | Complete error reference, API token creation guide, rate limit details |
| `scripts/diagnose-https.sh` | Automated diagnostic script |

## Expected Outcomes

**Success:**
- All diagnostic checks pass
- `certificate obtained successfully` for each domain