29.10.2026 Problem:
A client running a production Patroni PostgreSQL cluster on OpenZFS reported a critical incident. Replication between nodes had become extremely slow, a switchover had failed, and the cluster was running on a single node with no working standby, so a failure of that node would have caused total data loss. The client asked us to restore replication, explain the root cause of the outage, and confirm whether the ZFS settings they had previously applied across all their clusters should be kept or rolled back.
Process:
Step 1: Emergency troubleshooting session
During a live call, the experts found that the receiving node’s network buffers were nearly full because the disk could not acknowledge writes fast enough. The cause was the ZFS sync=always setting, which forced every write to be synchronous. Switching the receiving node to sync=standard immediately raised replication throughput more than tenfold, and the multi-terabyte re-sync of the standby completed within the session. We then agreed a safe order for applying the same change to the remaining node after a switchover.
Step 2: Review of the ZFS settings
We confirmed that sync=standard is safe because PostgreSQL enforces its own durability through fsync on commits and checkpoints. Historical logs showed the cost of sync=always: slow checkpoints, low WAL replay speed, and standby lag of several hours every night. We also recommended re-enabling data caching in ZFS with a capped ARC size, since PostgreSQL relies on the OS cache.
Step 3: Root cause analysis of the failed switchover
The first log set did not cover the incident window, so we requested the correct database, Patroni, etcd, and system logs. Analysis showed that the slow I/O made every checkpoint and recovery take 30–50 minutes. During the switchover, the Patroni service on the standby was stopped and a database process was killed manually. This forced a long crash recovery, and the standby was promoted without the last few seconds of the old primary’s transactions. On the old primary, the data volume was not mounted after a reboot, and a reinit command then deleted data that could have been recovered quickly with pg_rewind. We also found that the application connected directly to a node rather than to the current leader, and that a heavy unindexed query was adding load on the new primary.
Step 4: Explain why the earlier configuration did not help
At the client’s request, we explained that sync=always had been applied as a protection against a known OpenZFS WAL-corruption issue, but it does not address that issue’s mechanism. Meanwhile, the measures that actually protect against it, such as WAL archive recovery and moving WAL off ZFS, had not been implemented.
Step 5: Advise on further storage tuning
The client asked about additional ZFS properties. We recommended a smaller recordsize for the data dataset to reduce write amplification, the default recordsize for WAL, and storing extended attributes in metadata. We also described a rollout with no application downtime: apply the changes on the standby, re-clone it, switch over, then repeat on the other node.
Solution:
Replication was restored by switching ZFS to sync=standard, and the client received a complete root cause analysis of the outage. The recommended corrective actions included:
- keeping sync=standard on all clusters;
- disabling automatic data-directory removal and using pg_rewind before any reinit;
- hardening the Patroni service settings;
- a strict switchover procedure with no manual stops or kills during the switchover;
- WAL archive recovery for standbys instead of full re-clones;
- routing application traffic through the leader endpoint;
- adding an index for the heavy query;
- a planned storage tuning rollout.
Conclusion:
The incident was caused by a filesystem setting that throttled all database I/O, combined with manual actions during a slow switchover. Fixing the setting restored fast replication and a working standby. The root cause analysis and corrective actions give the client a safer switchover procedure, faster standby recovery, and a clear path to tune storage across all their production clusters.