Problem:
A client running a production Patroni PostgreSQL cluster on OpenZFS reported a critical incident. Replication between nodes had become extremely slow, a switchover had failed, and the cluster was running on a single node with no working standby, so a failure of that node would have caused total data loss. The goals were to restore replication, explain the root cause of the outage, and determine whether the existing ZFS settings should be kept or rolled back.
Process:
Step 1: Emergency troubleshooting
The experts found that the receiving node’s network buffers were nearly full because the disk could not acknowledge writes fast enough. The cause was the ZFS sync=always setting, which forced every write to be synchronous. Switching the receiving node to sync=standard immediately raised replication throughput more than tenfold, and the full re-sync of the standby completed shortly afterwards. A safe order was then defined for applying the same change to the remaining node after a switchover.
Step 2: Review of the ZFS settings
We confirmed that sync=standard is safe because PostgreSQL enforces its own durability through fsync on commits and checkpoints. Historical logs showed the cost of sync=always: slow checkpoints, low WAL replay speed, and standby lag of several hours every night. We also recommended re-enabling data caching in ZFS with a capped ARC size, since PostgreSQL relies on the OS cache.
Step 3: Root cause analysis of the failed switchover
Database, Patroni, etcd, and system logs covering the incident window were analyzed. They showed that the slow I/O made every checkpoint and recovery take 30–50 minutes. The switchover was interrupted on the standby, which forced a long crash recovery, and the standby was promoted without the last few seconds of the old primary’s transactions. The old primary then required a full re-initialization, whereas pg_rewind would have allowed a much faster recovery. We also found that the application connected directly to a node rather than to the current leader, and that a heavy unindexed query was adding load on the new primary.
Step 4: Why the earlier configuration did not help
The sync=always setting had been applied as a protection against a known OpenZFS WAL-corruption issue, but it does not address that issue’s mechanism. The measures that actually protect against it are WAL archive recovery and moving WAL off ZFS.
Step 5: Advise on further storage tuning
For further tuning, we recommended a smaller recordsize for the data dataset to reduce write amplification, the default recordsize for WAL, and storing extended attributes in metadata. We also described a rollout with no application downtime: apply the changes on the standby, re-clone it, switch over, then repeat on the other node.
Solution:
Replication was restored by switching ZFS to sync=standard, and a complete root cause analysis of the outage was provided. The recommended corrective actions included:
- keeping sync=standard on all clusters;
- disabling automatic data-directory removal and using pg_rewind before any reinit;
- hardening the Patroni service settings;
- a strict switchover procedure without manual intervention;
- WAL archive recovery for standbys instead of full re-clones;
- routing application traffic through the leader endpoint;
- adding an index for the heavy query;
- a planned storage tuning rollout.
Conclusion:
The incident was caused by a filesystem setting that throttled all database I/O, combined with an interrupted switchover. Fixing the setting restored fast replication and a working standby. The root cause analysis and corrective actions provide a safer switchover procedure, faster standby recovery, and a clear path to tune storage across the production environment.