Case Studies

Problem: Production PostgreSQL clusters (PostgreSQL community edition 15.17) are running under Patroni (v3.3.2) with HAProxy (2.6.21) in front. All client connections go through HAProxy using virtual IPs and virtual ports; on the PostgreSQL side the reported client address is the HAProxy VIP rather than the originating physical client IP. The operations team needed a reliable […]

Problem: A production PostgreSQL cluster (community PostgreSQL 15.8, managed by Patroni v2.1.4) experienced an unplanned automatic switchover. Symptoms reported included an unexpected promotion of the standby after the primary stopped responding to health checks. Monitoring and OS traces showed memory usage climbing to ~95% on the primary node, PostgreSQL failing to respond to Patroni probe […]

Problem: A security scan of a deployed Apache Spark binary distribution reported two vulnerabilities located in Hive artifacts bundled with Spark: CVE-2024-29869 (org.apache.hive:hive-exec, fixed in Hive 4.0.1) and CVE-2024-23953 (org.apache.hive:hive-llap-common, fixed in Hive 4.0.1). The scan showed the vulnerable artifacts present in the shipped Spark distribution (the Spark 3.5.x line shipped a Hive 2.3.x fork […]

Problem: A high-traffic Patroni-managed PostgreSQL cluster experienced an automatic failover: the former primary (node A) lost leadership, and a secondary (node B) was successfully promoted. However, after the failover, node A failed to automatically rejoin the cluster as a replica. The symptoms unfolded in three stages. First, direct streaming replication was immediately rejected by the […]

Problem: A customer running Patroni-managed PostgreSQL (PostgreSQL 15.17, Patroni v3.3) had production databases ranging from >2-3 TB up to a ~52 TB physical replica. Rebuilding the empty DR site from production relied on manual pg_basebackup executions and manual post-backup recovery steps, which was too slow, operationally intensive, and prone to human error for such large volumes. The […]

Problem: An OLTP system using Patroni-managed PostgreSQL 15 experienced global slowdowns under load: CPU climbed to ~80%, large numbers of concurrent sessions, and application timeouts. The client reported bursts of traffic (hundreds of workers in parallel) routed through HAProxy to the leader, and provided PostgreSQL and Patroni logs, auto_explain output, and query samples. Key database […]

Problem: A production Patroni-managed PostgreSQL 15 cluster experienced periodic heavy queries that threatened availability. An example slow job ran for ~84 seconds and performed a full scan of a 1.8 TB partitioned table (arbor.CDR_DATA) that uses daily partitions starting in early April. Most clients connect through generic application users rather than distinct personal accounts. The […]

Problem: A 5-node Apache Cassandra 2.2.5 cluster (two data centers) reported severe per-node disk imbalance: each node was configured with five data_file_directories (e.g. /cassandra/data1/data … /cassandra/data5/data) but some mount points on individual nodes were near full (examples showed mounts at 93% and 95% used). On one node a particular keyspace (jessi) had large sstable directories […]

Problem: A production Patroni-managed PostgreSQL cluster (PostgreSQL v15.17, Patroni 3.3.2) experienced a primary process abort with SIGABRT during normal operation. Server logs reported that a server process was terminated by signal 6 (Aborted) and that the failed process was executing a COMMIT when the postmaster began terminating other server processes. Subsequent messages showed PostgreSQL could […]