managing-database-replication

What this skill does

# Database Replication Manager

## Overview

Configure and manage database replication topologies for PostgreSQL (streaming replication, logical replication), MySQL (source-replica, group replication), and MongoDB (replica sets). This skill covers primary-replica setup, read scaling through replica routing, failover automation, replication lag monitoring, and conflict resolution for multi-primary configurations.

## Prerequisites

- Superuser or replication-role credentials on primary and replica servers
- Network connectivity between all replication nodes (verify with `pg_isready` or `mysqladmin ping`)
- `psql`, `mysql`, or `mongosh` CLI tools installed on all nodes
- Matching major database versions across all replication nodes
- Sufficient disk space on replicas (equal to or greater than primary)
- SSH access to replica servers for initial base backup transfer

## Instructions

1. Choose the replication topology based on requirements:
   - **Single primary + read replicas**: Best for read-heavy workloads. All writes go to primary; reads distributed across replicas.
   - **Multi-primary (active-active)**: Best for geographic distribution. Requires conflict resolution. Use PostgreSQL logical replication or MySQL Group Replication.
   - **Cascading replication**: Replica A replicates from primary, Replica B replicates from Replica A. Reduces primary load for many replicas.

2. For PostgreSQL streaming replication, configure the primary:
   - Set `wal_level = replica`, `max_wal_senders = 10`, `max_replication_slots = 10`
   - Create replication user: `CREATE ROLE replicator WITH REPLICATION LOGIN PASSWORD 'secure_password'`
   - Add replication entry to `pg_hba.conf`: `host replication replicator replica_ip/32 scram-sha-256`
   - Reload configuration: `SELECT pg_reload_conf()`

3. Initialize the replica with a base backup: `pg_basebackup -h primary_host -U replicator -D /var/lib/postgresql/data -Fp -Xs -P -R`. The `-R` flag creates `standby.signal` and configures `primary_conninfo` automatically.

4. For MySQL source-replica replication, configure the source:
   - Set `server-id = 1`, `log_bin = mysql-bin`, `binlog_format = ROW`
   - Create replication user: `CREATE USER 'replicator'@'replica_ip' IDENTIFIED BY 'secure_password'; GRANT REPLICATION SLAVE ON *.* TO 'replicator'@'replica_ip'`
   - Record binary log position: `SHOW MASTER STATUS`

5. Configure the MySQL replica: `CHANGE REPLICATION SOURCE TO SOURCE_HOST='primary_host', SOURCE_USER='replicator', SOURCE_PASSWORD='...', SOURCE_LOG_FILE='mysql-bin.000001', SOURCE_LOG_POS=154; START REPLICA`.

6. For MongoDB replica sets: initialize with `rs.initiate({_id: "rs0", members: [{_id: 0, host: "node1:27017"}, {_id: 1, host: "node2:27017"}, {_id: 2, host: "node3:27017"}]})`. MongoDB handles leader election and failover automatically.

7. Monitor replication lag continuously:
   - PostgreSQL: `SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, (sent_lsn - replay_lsn) AS lag_bytes FROM pg_stat_replication`
   - MySQL: `SHOW REPLICA STATUS\G` (check `Seconds_Behind_Source`)
   - MongoDB: `rs.printReplicationInfo()` and `rs.printSecondaryReplicationInfo()`

8. Configure application-level read routing: direct write queries to the primary connection and read queries to a load-balanced replica pool. Use connection poolers (PgBouncer, ProxySQL) or application middleware for automatic routing.

9. Set up automated failover using Patroni (PostgreSQL), MySQL InnoDB Cluster + MySQL Router, or MongoDB's built-in election mechanism. Test failover by stopping the primary and verifying the replica promotes automatically within the target RTO.

10. Configure alerting for replication lag exceeding 10 seconds, replication slot inactive for more than 1 hour, and replica connection drops. Stale replication slots in PostgreSQL cause WAL accumulation and can fill the disk.

## Output

- **Replication configuration files** for primary and replica nodes
- **Base backup and initialization scripts** for setting up new replicas
- **Replication monitoring queries** with lag measurement and health checks
- **Failover runbook** with manual and automated promotion procedures
- **Read routing configuration** for application or connection pooler

## Error Handling

| Error | Cause | Solution |
|-------|-------|---------|
| Replication lag increasing steadily | Replica cannot keep up with primary write volume | Check replica I/O and CPU; increase `max_parallel_workers` on replica; consider upgrading replica hardware; reduce write-heavy batch operations on primary |
| `replication slot is inactive` warning | Replica disconnected or paused, WAL accumulating on primary | Reconnect replica; if permanently removed, drop the slot with `SELECT pg_drop_replication_slot('slot_name')` to prevent disk fill |
| `could not connect to primary` on replica | Network partition, primary down, or authentication failure | Verify network connectivity; check `pg_hba.conf` entries; confirm replication user credentials; check primary process status |
| Replica has diverged from primary | Split-brain after failed failover or manual writes to replica | Re-initialize replica from fresh base backup; for PostgreSQL, use `pg_rewind` if timeline divergence is small |
| Conflict in logical replication | Same row modified on both publisher and subscriber | Configure conflict resolution policy; use `UPDATE` conflict handler; design schema to avoid cross-node writes to same rows |

## Examples

**Setting up PostgreSQL read replicas for a web application**: A primary database handles 2,000 writes/second but read traffic is 10x higher. Two streaming replicas are added with PgBouncer routing read queries to replicas in round-robin. Result: primary CPU drops from 90% to 40%, read latency improves by 60%.

**Automated failover with Patroni**: A 3-node PostgreSQL cluster managed by Patroni with etcd for consensus. When the primary fails, Patroni promotes the most up-to-date replica within 10 seconds. Application reconnects automatically through the Patroni-managed VIP or DNS endpoint.

**Cross-region logical replication for compliance**: EU customer data must stay in EU region. Logical replication publishes only non-PII tables to the US region replica. EU application reads locally; US analytics queries use the replicated subset. Publication filter: `CREATE PUBLICATION us_analytics FOR TABLE orders, products, categories`.

## Resources

- PostgreSQL streaming replication: https://www.postgresql.org/docs/current/warm-standby.html
- PostgreSQL logical replication: https://www.postgresql.org/docs/current/logical-replication.html
- MySQL replication: https://dev.mysql.com/doc/refman/8.0/en/replication.html
- Patroni (PostgreSQL HA): https://patroni.readthedocs.io/
- MongoDB replica sets: https://www.mongodb.com/docs/manual/replication/