Problem:

A production Patroni cluster (Patroni 3.3.2, PostgreSQL 15.17 on VMs) exhibited very slow replication between nodes, producing significant replication lag and impacting services. Replication throughput on the standby was measured in single MB/s ranges, nightly replay lag reached hours, and a recent switchover required a full base backup because the replica was unable to catch up in time. OpenZFS datasets were configured with sync=always and primarycache=metadata across the cluster; these settings had been applied previously based on an earlier RCA.

Process:

Step 1: Verify reported symptoms and collect configuration

PostgreSQL and Patroni versions, node roles, and filesystem properties were reviewed alongside the client’s notes about a switchover. ZFS dataset properties (sync and primarycache) and shared_buffers sizing were confirmed. This established that synchronous ZFS writes (sync=always) and metadata-only primarycache were in effect and framed which layers could be causing durable-write path stalls versus caching inefficiencies.

Step 2: Inspect performance metrics and transfer rates during recovery

Observed replication throughput and checkpoint timings were examined from supplied logs and live measurements. Checkpoint flushes on the primary had long wall times and standby WAL replay rates stayed under 20 MB/s; changing the receiver node’s ZFS sync to standard produced an immediate throughput jump (from ~50 MB/s to ~673 MB/s, later observed at ~2.31 TB/hour). This showed the storage synchronous write policy was directly throttling replication bandwidth.

Step 3: Review socket and network indications

Socket buffer and TCP window behavior were assessed in kernel and PostgreSQL receiver metrics; the receiver’s socket buffer appeared nearly full during slow replay periods. This supported a hypothesis that slow durable commits on the receiver were back-pressuring the sender and limiting throughput—consistent with synchronous filesystem behavior.

Step 4: Cross-check earlier RCA and open-source bug context

Documentation from the previous RCA and public OpenZFS issue reports were re-evaluated. The prior recommendation to use ZFS sync=always was contrasted with current observations showing no additional durability benefit for PostgreSQL beyond fsync, while causing severe performance penalties. The historical context clarified why sync=always had been chosen earlier, and why it was not protecting against the specific WAL corruption concerns without accompanying archive/restore safeguards.

Step 5: Assess cache policy impact on PostgreSQL buffer usage

primarycache=metadata configuration was analyzed relative to PostgreSQL’s reliance on the OS page cache. Disabling ZFS data caching was found to reduce effective read performance and force more I/O pressure into ARC sizing decisions. Shared_buffers and effective_cache_size values were compared to ARC sizing to determine practical memory balance implications and to decide whether to revert primarycache.

Step 6: Validate log coverage for the switchover and request missing records

Uploaded log archives were inspected and found to contain earlier dates that did not cover the reported switchover window. Missing Patroni history, etcd logs, and the relevant PostgreSQL log rotation files were requested to confirm whether the switchover was operator-initiated, triggered by timeout, or followed crash recovery. This step ensured the root cause was isolated to storage configuration before final remediation.

Step 7: Implement targeted change and observe immediate effect

After confirming the storage and network signs, sync was changed to standard on the receiver node during recovery; replication throughput and base-backup completion were observed to increase dramatically. This operational validation led to an agreed plan to apply sync=standard across remaining nodes in a controlled switchover sequence and to revisit primarycache and ARC sizing during the next maintenance window—transitioning investigation into the implemented fix.

Solution:

OpenZFS sync was set to standard on the recovering (receiver) node and rolled to other nodes after the controlled switchover; primarycache was recommended to be returned to all with a capped ARC (zfs_arc_max ≈ 25–30% of RAM) and pg_wal placed on a separate filesystem when possible. PostgreSQL continued to enforce durability through its own fsync behavior; changing ZFS sync to standard removed redundant synchronous persistence at the filesystem layer that serialized writes and throttled WAL shipping. The result was unblock of the WAL sender/receiver pipeline while preserving PostgreSQL-level durability.

Conclusion:

Replication throughput improved from tens of MB/s to several hundred MB/s (and sustained multi‑TB/hour rates during catch-up), allowing base backup completion and reducing replay lag from hours to minutes. The cluster returned to stable primary/standby operation with lower risk of long re-clones; follow-up actions include applying the change cluster-wide in a staged manner, reverting or tuning primarycache with ARC caps, and configuring archive/restore mechanisms to avoid full re-clones in future recoveries.