troubleshoot-replication

What this skill does

# Troubleshoot AEM 6.5 LTS Replication

This skill provides systematic troubleshooting guidance for Adobe Experience Manager 6.5 LTS replication issues. Use this to diagnose and resolve problems with content distribution, agent configuration, and replication workflows.

## When to Use This Skill

Use this skill when experiencing:
- Replication queues blocked or stuck
- Content not appearing on Publish instances
- Replication agent showing red/error status
- "Replication triggered, but no agent found" errors
- Timeout errors during replication
- Authentication failures (401 Unauthorized)
- Connection refused errors
- Slow or delayed replication
- Dispatcher cache not invalidating
- Reverse replication failures
- Missing or incorrect replication status

## Prerequisites

- AEM 6.5 LTS Author and Publish instances
- Administrator access to AEM environments
- Access to replication agent configuration
- Access to log files (error.log, replication.log)
- Understanding of your replication topology

## Diagnostic Workflow

Follow this systematic approach to identify and resolve replication issues:

```
1. Verify Symptoms
   ↓
2. Check Agent Status
   ↓
3. Review Replication Queue
   ↓
4. Test Connectivity
   ↓
5. Examine Logs
   ↓
6. Verify Configuration
   ↓
7. Apply Fix
   ↓
8. Validate Resolution
```

## Common Issues and Solutions

### Issue 1: Replication Queue Blocked

**Symptoms:**
- Red status indicator on replication agent
- Queue shows items waiting
- First item in queue failed
- Subsequent items cannot process (FIFO blocking)

**Diagnosis:**

1. **Check agent status:**
   ```
   Navigate to: Tools → Deployment → Replication → Agents on author
   Look for: Red indicator next to agent name
   ```

2. **View queue details:**
   ```
   Click agent name
   Review queue entries
   Check error message on failed item
   ```

**Root Causes:**
- Network connectivity lost to Publish instance
- Publish instance down or unavailable
- Authentication credentials expired or incorrect
- Insufficient permissions on target content
- Disk space full on Publish
- Large package timeout
- SSL/TLS certificate issues

**Solutions:**

**Solution A: Retry Failed Item**
```
Steps:
1. Open blocked replication agent
2. Click "Force Retry" button
3. Monitor queue to see if item processes
4. If successful, remaining items will process automatically
```

**Solution B: Clear Failed Item**
```
Steps:
1. Open blocked replication agent
2. Select failed item in queue
3. Click "Clear" to remove it
4. Remaining items will process
5. Manually re-replicate cleared content if needed
```

**Solution C: Restart Replication Components**
```
Navigate to: /system/console/bundles
Search for: "replication"

Restart these bundles:
- com.day.cq.cq-replication
- com.day.cq.cq-replication-audit
- com.day.cq.wcm.cq-wcm-replication

Steps:
1. Find bundle
2. Click "Stop"
3. Wait for status: Resolved
4. Click "Start"
5. Verify status: Active
```

**Solution D: Restart Event Processing**
```
OSGi Console: /system/console/bundles
Restart: Apache Sling Event Support (org.apache.sling.event)

This clears event queue backlogs
```

### Issue 2: Connection Refused

**Symptoms:**
- Error: "Connection refused"
- Test connection fails
- Replication queue blocked with connectivity errors

**Diagnosis:**

1. **Verify Publish instance is running:**
   ```bash
   # Check if Publish is accessible
   curl -I http://publish-host:4503/system/console
   
   # Or browse to:
   http://publish-host:4503/system/console
   ```

2. **Test network connectivity:**
   ```bash
   # From Author server
   telnet publish-host 4503
   
   # Or
   nc -zv publish-host 4503
   
   # Or
   ping publish-host
   ```

3. **Check replication agent URI:**
   ```
   Navigate to: Agent → Edit → Transport tab
   Verify: URI matches Publish host and port
   Expected: http://publish-host:4503/bin/receive?sling:authRequestLogin=1
   ```

**Root Causes:**
- Publish instance not running
- Firewall blocking connection
- Incorrect hostname or port in agent configuration
- Network routing issues
- DNS resolution failures

**Solutions:**

**Solution A: Start Publish Instance**
```bash
cd /path/to/publish/crx-quickstart
./bin/start
```

**Solution B: Fix Network/Firewall**
```
1. Verify firewall rules allow Author → Publish on port 4503
2. Check network ACLs and security groups (cloud environments)
3. Verify no proxy blocking connection
4. Test from Author server command line
```

**Solution C: Correct Agent URI**
```
Steps:
1. Edit replication agent
2. Transport tab
3. Update URI to correct host/port:
   http://correct-publish-host:4503/bin/receive?sling:authRequestLogin=1
4. Save
5. Test Connection
```

### Issue 3: 401 Unauthorized

**Symptoms:**
- Error: "401 Unauthorized"
- Authentication failures in logs
- Test connection fails with credential error

**Diagnosis:**

1. **Check agent credentials:**
   ```
   Agent → Edit → Transport tab
   Verify: User and Password fields
   ```

2. **Verify user exists on Publish:**
   ```
   Publish instance: http://publish:4503/crx/explorer
   Navigate to: /home/users
   Search for: replication service user
   ```

3. **Check user permissions:**
   ```
   On Publish instance:
   User → Permissions
   Required: Read, Write, Replicate privileges
   ```

**Root Causes:**
- Incorrect username or password
- User doesn't exist on target instance
- User password changed
- User disabled or locked
- Insufficient permissions

**Solutions:**

**Solution A: Update Credentials**
```
Steps:
1. Edit replication agent
2. Transport tab
3. Enter correct username
4. Enter correct password
5. Save
6. Test Connection
```

**Solution B: Create/Enable User on Publish**
```
On Publish instance:
1. Navigate to: Security → Users
2. Create user: replication-service
3. Set password (match Agent configuration)
4. Save

Grant permissions:
1. Navigate to: Security → Permissions
2. Select user: replication-service
3. Add entries:
   - Path: /content
   - Privileges: jcr:read, crx:replicate, jcr:write
4. Save
```

**Solution C: Reset Password**
```
On Publish instance:
1. Navigate to: Security → Users
2. Find user in agent configuration
3. Click "Set Password"
4. Enter new password
5. Save

On Author:
1. Update replication agent with new password
2. Save
3. Test Connection
```

### Issue 4: SSL/TLS Certificate Errors

**Symptoms:**
- SSL handshake failed
- Certificate validation errors
- HTTPS connection failures

**Diagnosis:**

1. **Check agent URI protocol:**
   ```
   Agent → Transport tab
   URI: https://... or http://...
   ```

2. **Review error logs:**
   ```
   error.log contains:
   - javax.net.ssl.SSLHandshakeException
   - PKIX path building failed
   - Certificate validation failed
   ```

**Solutions:**

**Solution A: Enable Relaxed SSL (Development Only)**
```
WARNING: Only for development/testing environments

Steps:
1. Edit replication agent
2. Transport tab
3. SSL section:
   ✓ Relaxed SSL (allow self-signed certificates)
   ✓ Allow expired (allow expired certificates)
4. Save
5. Test Connection
```

**Solution B: Import Certificates (Production)**
```
On Author instance:

1. Export certificate from Publish:
   openssl s_client -connect publish:4503 -showcerts > publish-cert.pem

2. Import into Java keystore:
   cd $JAVA_HOME/jre/lib/security
   keytool -import -alias publish-aem -file publish-cert.pem \
     -keystore cacerts -storepass changeit

3. Restart AEM Author

4. Test replication agent connection
```

**Solution C: Use HTTP (Not Recommended for Production)**
```
If SSL is not required:
1. Edit agent
2. Transport tab
3. Change URI from https:// to http://
4. Save
5. Test Connection
```

### Issue 5: Content Not Appearing on Publish

**Symptoms:**
- Replication succeeds (green status)
- Queue processes successfully
- Content still doesn't appear on Publish
- Old content served

**Diagnosis:**

1. **Check content directly on Publish:**
   ```
   Bypass Dispatcher:
   http://publish:4503/content/mysite/en/page.html