25.09.2026 Problem:
A two-site PostgreSQL 13.4 deployment managed by Patroni experienced replication breakage at the DR site after any failover inside the primary site. The DR standby leader’s primary_conninfo was hardcoded to the former primary node’s address, so when the primary role moved to the other primary-site node the standby kept attempting to connect to the old address and required a manual rebuild to resume streaming. The environment used Patroni 2.1.3 on both sites, an external etcd-based DCS shared by both clusters and apparently located in the primary-site network, and a physical replication slot named patroni_c that existed only on the then-current primary. Additional configuration observations included a bootstrap pg_hba section (which may not reflect the running pg_hba.conf) and use of a password file under /tmp on DR nodes.
Process:
Step 1: Review client-provided runtime state
Reviewed patronictl list and local PostgreSQL status outputs provided by the client for primary-site and DR nodes. The client confirmed Patroni version 2.1.3 on both sites. The DR standby leader’s SHOW primary_conninfo contained a single host entry pointing to the current primary node. pg_stat_replication on the current primary showed streaming to an on-site replica and to a separate DR walreceiver. This established the immediate cause: the standby would keep trying the same single host after primary-site leader moves.
Step 2: Review configuration in the DCS and bootstrap sections
Reviewed patronictl show-config output and patroni.yml files shared by the client from both clusters, including the bootstrap sections. Found that standby_cluster.host on the DR side contained one primary address only, and the primary cluster had no dynamic slots section declared in the DCS. Noted that wal_keep_segments remained set on the primary cluster (a parameter removed in PostgreSQL 13) and no wal_keep_size was present. This showed a gap where connection routing and WAL retention could both break replication after promotion.
Step 3: Validate replication slot and access assumptions
Reviewed pg_replication_slots output from the active primary. Given the absence of a slots section in the DCS, concluded that patroni_c existed only on that node and had likely been created manually. Checked the pg_hba intent in the bootstrap entries and identified that the running pg_hba.conf on the other primary node may not contain the required replicator entries for DR addresses, since bootstrap settings are applied only at cluster initialization. Also noted that DR nodes relied on /tmp/pgpass for replication credentials. These findings pointed to two secondary failure modes: a missing slot on the promoted leader and possible authentication failures.
Step 4: Select and justify the remediation approach
With Patroni 2.1.3 in place, confirmed that the standby_cluster multiple-host feature is supported: it lets Patroni include multiple primary addresses in primary_conninfo with target_session_attrs=read-write. Assessed the client’s original VIP/VRRP design (keepalived) as viable, but it would add infrastructure and depend on network support for VRRP and gratuitous ARP. Recommended the in-place Patroni configuration change, combined with fixes for the slot and WAL retention, as the primary remediation, with keepalived documented as a fallback. The client agreed to proceed with the multi-host approach first.
Step 5: Define the implementation plan and validation procedure
Prepared an ordered change plan to be applied via patronictl edit-config: (a) declare patroni_c as a permanent physical slot in the primary cluster configuration, so Patroni recreates it on whichever node becomes leader; (b) add postgresql.parameters.wal_keep_size (4GB) on the primary cluster and remove the ineffective wal_keep_segments, to protect WAL retention during the window between promotion and slot creation on Patroni 2.1.3; (c) verify that the running pg_hba.conf on both primary-site nodes allows the replicator user from the DR addresses, and correct it where missing; (d) update the DR standby_cluster.host to list both primary-site node addresses, so Patroni generates a multi-host primary_conninfo with target_session_attrs=read-write. The plan also recommended relocating the replication password file from /tmp to a persistent path. For validation, the procedure specified checking SHOW primary_conninfo on the DR standby leader and pg_replication_slots on the primary, then running a controlled switchover inside the primary site. The expected result is that the DR walreceiver reconnects to the new leader, follows the new timeline and resumes streaming without a rebuild. A hard-failure test (stopping Patroni on the leader) was suggested if the change window allowed.
Solution:
The recommended fix relies on Patroni and PostgreSQL configuration only, with no additional infrastructure. It has three parts: a permanent physical replication slot declared in the primary cluster configuration, a wal_keep_size value that retains WAL across leader promotion, and a DR standby_cluster.host listing both primary-site nodes, so libpq can fail over client-side to whichever node is writable. On PostgreSQL 13, primary_conninfo changes can be applied with a reload, so the change is expected to require only a walreceiver restart on the DR side, not a full PostgreSQL restart. A floating VIP with keepalived (or vip-manager, which follows the Patroni leader key in the DCS) was documented as a fully workable alternative, for example if applications also need a single stable address. Upgrading Patroni to 3.2.0 or later was advised as a future improvement: those versions pre-create permanent slots on all leader-eligible nodes and close the WAL retention window entirely.
Conclusion:
The proposed design addresses both the connection-routing cause and the secondary failure modes (missing slot on the promoted leader, insufficient WAL retention, authentication gaps). With it in place, clean switchovers and normal failovers inside the primary site should be followed by the DR cluster automatically, without manual rebuilds. One edge case remains outside the scope of any routing solution. With asynchronous replication, if a crash failover occurs while the DR standby leader has already received WAL beyond the point where the new leader diverged, that node will need pg_rewind or reinitialization. This is expected PostgreSQL behavior, not a failure of the design. The review also surfaced two broader recommendations: reassess DCS placement, since a shared etcd ensemble located with the primary site would leave the DR cluster without a DCS if the whole primary site were lost, and rotate credentials that had been shared in plain text.