Database

Problem: Production PostgreSQL clusters (PostgreSQL community edition 15.17) are running under Patroni (v3.3.2) with HAProxy (2.6.21) in front. All client connections go through HAProxy using virtual IPs and virtual ports; on the PostgreSQL side the reported client address is the HAProxy VIP rather than the originating physical client IP. The operations team needed a reliable […]

Problem: A production PostgreSQL cluster (community PostgreSQL 15.8, managed by Patroni v2.1.4) experienced an unplanned automatic switchover. Symptoms reported included an unexpected promotion of the standby after the primary stopped responding to health checks. Monitoring and OS traces showed memory usage climbing to ~95% on the primary node, PostgreSQL failing to respond to Patroni probe […]

Problem The database architecture reached its limit when a critical node stopped responding. A standby node experienced a PostgreSQL Patroni replica failure, repeatedly logging “incorrect resource manager data checksum” errors. The system stopped working because corrupted Write-Ahead Log segments completely broke the replication stream. A dangerous shortcut would involve running continuous base backups to force […]

Problem: A customer running Patroni-managed PostgreSQL v15.17 (Patroni 3.3.2) asked whether Commvault backups can be taken from the DR site’s Standby Leader. The DR cluster is a replicating standby of Production. The request asked specifically whether WAL file generation can be started on the DR Standby Leader and whether Commvault’s option to delete WALs after […]

Problem: A Patroni-managed PostgreSQL cluster (PostgreSQL 15.17, Patroni 3.3.2) running asynchronous replication intermittently failed to rejoin an ex-master after a physical host reboot performed as part of high‑availability testing. Test pattern: hard shutdown of the former primary for ~5 minutes, then restart. Symptom observed on restart: the node failed to rejoin with the error “requested […]

Problem: Seven identical Patroni-managed PostgreSQL 15.15 clusters (Patroni 3.3.2) showed divergent behaviour for the same SELECT query: clusters labelled Cust 1–5 returned results quickly while Cust 6–7 returned the same query much slower. Dataset sizes, indexes, constraints and foreign keys were reported as equivalent across clusters. One temporary mitigation (set enable_mergejoin = false) had been […]

Problem: The data mount was approaching full capacity. Investigation showed that the keyspace-level backups directories inside the Cassandra data directories were consuming the majority of space (≈150 GB). No active snapshots were present on the cluster when the issue was reported. The cluster configuration had incremental_backups: true in cassandra.yaml because the customer wanted to retain […]

Problem: A Patroni-managed PostgreSQL cluster (PostgreSQL 15.8, Patroni 2.1.4) was reporting sustained high memory consumption on a host with ~275 GB of physical RAM and only ~8% free. The environment had HugePages enabled at the OS level. Patroni configuration showed shared_buffers=68GB, work_mem=20MB, max_parallel_workers_per_gather=4 and max_connections=2000. Observed behavior included consistently ~90% system memory utilization across a […]

Problem: An application UPDATE statement against a partitioned PostgreSQL table (UPDATE tab1 SET sys_update_date = $1, agg_status = $2, output_filename = $3, merge_type = $4 WHERE period_key = $5 and record_id = $6) experienced consistent slowdowns in a nightly early‑morning window. The customer reported that partition file sizes showed normal small footprints for partitions p1–p12, […]