16.09.2026 Problem:
A Patroni-managed PostgreSQL cluster showed recurring replication interruptions: the replica repeatedly reported a large streaming lag (initially ~565 MB, later observed ~5.5 GB) and the cluster’s physical replication slot on the leader appeared inactive (active = false, wal_status = reserved). Patroni logs on the primary had not recorded recent cluster events and appeared to stop growing; the patroni.yml included log level set to WARNING and an urllib3: DEBUG logger. Queries showed pg_stat_replication on the leader returned no walsender rows and pg_stat_wal_receiver on the replica returned no walreceiver rows while the replica still had primary_slot_name configured. The customer could not find an obvious reason in Patroni logs and asked to investigate persistent inactivity of the internal replication slot and recurring lag.
Process:
Step 1: Verify reported replication state and disk pressure
Reviewed patronictl output and the customer’s df output. Observed the replica lag increase from a few hundred megabytes to multiple gigabytes between checks. Verified safe_wal_size was unset (max_slot_wal_keep_size effectively unlimited), which explained the potential for WAL accumulation on the primary but the cluster had free space at the time. This established replication was broken repeatedly and WAL retention could become a risk if the slot stayed inactive.
Step 2: Confirm replication endpoints were not connected
Ran the standard queries the customer had provided: SELECT from pg_stat_replication on the leader returned no rows; SELECT from pg_stat_wal_receiver on the replica returned no rows while SHOW primary_slot_name on the replica matched the leader’s slot name. This showed configuration was correct but the replica was not connected to the leader—the inactive slot was a symptom of a missing walreceiver, not a corrupted slot.
Step 3: Inspect leader logs for connection failures
Reviewed leader PostgreSQL logs provided by the customer. Found a network-level connection reset message from the leader’s walsender during a streaming attempt. That indicated at least one replication session was reset by the network, so network resets were a plausible trigger, but further evidence was needed because the replica had not reconnected since the event.
Step 4: Inspect replica process state and replica logs
Examined the replica’s PostgreSQL logs and process table. The replica log contained a FATAL 57P01 entry indicating the walreceiver process was terminated by an administrator command. ps showed no active walreceiver process and pg_stat_wal_receiver was empty. The replica remained in recovery and had stopped replay at a specific LSN while already having received later WAL segments. This showed the walreceiver had been purposefully killed on the replica and was not being restarted automatically.
Step 5: Review Patroni and PostgreSQL configuration for clues
Checked patroni.yml and PostgreSQL parameters supplied by the customer. Patroni was set to WARNING logging with urllib3: DEBUG, which suppressed informative Patroni events and produced noisy HTTP logs; tcp_keepalives_* were set to 0 (OS defaults, long detection times); archive_mode was enabled but there was no restore_command on the replica, so archived WAL files could not be used to catch up. These settings explained why events were not visible in Patroni logs, why TCP death detection would be slow, and why the replica had only streaming as its recovery path.
Step 6: Decide remediation path and gather final verification data
Because wal_status on the leader remained reserved and the needed WAL was still available, a full rebuild was unnecessary. Collected a final set of checks on the replica (pg_is_in_recovery, connection info, process stack, recent replica logs, and socket state) to confirm the walreceiver absence and to rule out resource exhaustion. The evidence supported a corrective restart of the replica walreceiver combined with configuration changes to avoid recurrence.
Solution:
Applied a targeted set of changes on both nodes and restarted the replica via Patroni so streaming reattached to the existing physical slot: (1) set TCP keepalives to 60/10/6 in PostgreSQL to detect dead TCP sessions faster; (2) changed Patroni log level to INFO and removed urllib3: DEBUG so cluster events and reconnection attempts are recorded; (3) configured a bounded max_slot_wal_keep_size (example: 50 GB) to prevent unbounded WAL accumulation on the primary; (4) added a restore_command on the replica so archived WAL files are usable as a fallback stream; (5) restarted PostgreSQL on the replica using patronictl to re-establish the walreceiver. The replica resumed streaming using the existing slot and caught up without requiring basebackup because the leader had retained the required WAL (wal_status = reserved).
Conclusion:
Streaming was restored without a full rebuild, replication lag returned to normal as the walreceiver caught up, Patroni now logs reconnection events at INFO level, and bounded WAL retention removes the risk of the primary’s WAL filesystem filling. The keepalive and restore_command changes reduce recurrence risk and improve recovery options when streaming interruptions occur.