13.07.2026 Problem:
A high-traffic Patroni-managed PostgreSQL cluster experienced an automatic failover: the former primary (node A) lost leadership, and a secondary (node B) was successfully promoted. However, after the failover, node A failed to automatically rejoin the cluster as a replica.
The symptoms unfolded in three stages. First, direct streaming replication was immediately rejected by the new primary due to a timeline divergence, as node A contained local WAL records from the old timeline that were not replicated prior to the crash. Second, Patroni correctly triggered pg_rewind to synchronize the diverged data. The utility completed successfully (exit code 0), rolling back changes to the last common checkpoint. Finally, despite a successful rewind, the PostgreSQL process on node A failed to start and entered a restart loop. Logs revealed a fatal error: the requested WAL segment had already been removed on the new primary.
Operational context: the cluster used Patroni for etcd leader election, WAL retention (wal_keep_size) was set to minimal baseline values (128 MB), and no WAL archiving was configured. As a result, restoring node A required a full data rebuild from scratch using pg_basebackup.
Process:
Step 1: Determine the cause of leadership loss
Log analysis showed that the PostgreSQL server on node A did not crash. However, due to a peak load spike, Patroni’s internal health-check query (get_postgresql_status) exceeded the time limit and was canceled by the database with a statement_timeout error. Because of this, Patroni could not renew the leader lock in etcd before the TTL (Time To Live) expired. Leadership transitioned to node B, and node A correctly initiated a self-demotion to avoid a split-brain scenario.
Step 2: Analyze timeline divergence
After node B became the new primary (advancing to a new timeline), node A attempted to connect as a replica. However, it held local WAL records from the old timeline that were absent on the new primary. This is expected behavior when replication lag exists during a crash, which blocks direct streaming and necessitates running pg_rewind.
Step 3: Identify the long-running pg_rewind trap
The pg_rewind utility operates by comparing data blocks from the last common checkpoint. Due to intensive cluster writes and infrequent checkpoints, the volume of scanned data was enormous. The pg_rewind process took over 20 minutes to complete.
Step 4: Uncover the WAL loss mechanics
While node A was offline performing the resource-intensive pg_rewind, the new primary (node B) continued to serve heavy production traffic. Since no active replication slot retained the WAL files and wal_keep_size was set to a minimal value, node B routinely recycled old WAL files to free up disk space. By the time pg_rewind finished, the WAL segments critically required for PostgreSQL to start on node A were already gone.
Solution:
To prevent similar incidents in the future, a set of architectural and configuration changes was implemented:
1) Statement timeout isolation: the statement_timeout parameter was disabled (set to 0) exclusively for the Patroni superuser. This ensures internal health checks are not interrupted even during brief database load spikes, preventing spurious failovers.
2) Adaptive WAL retention: wal_keep_size was recalculated based on the actual WAL generation rate under load and increased to 64 GB to guarantee a safe offline window of at least 2 hours.
3) Replication slot protection: max_slot_wal_keep_size was introduced and set to 128 GB. This allows the primary to retain WAL for a temporarily disconnected replica while protecting the primary’s disk from total exhaustion during prolonged outages.
4) pg_rewind acceleration: checkpoint_timeout was reduced to 15 minutes. More frequent checkpoints narrow the window pg_rewind must scan, drastically cutting its execution time.
5) WAL archive implementation: robust WAL archiving was configured using an external tool like pgBackRest alongside restore_command, ensuring replicas can fetch missing segments from the archive if deleted locally by the primary.
Conclusion:
Engineering analysis demonstrated that a successful failover is only half the battle in ensuring high availability. Automatic replica rejoin after a crash is vulnerable to a race condition between pg_rewind execution time and aggressive log rotation on the active primary.
The implemented suite of measures—timeout tuning, mathematically calculated wal_keep_size, slot protection, and WAL archiving—allowed the cluster to successfully pass subsequent stress tests. During simulated failures, the former primary successfully recovered and automatically rejoined the cluster without requiring manual intervention or resource-intensive pg_basebackup rebuilds.