Case Studies

Problem: A production PostgreSQL cluster (community PostgreSQL 15.8, managed by Patroni v2.1.4) experienced an unplanned automatic switchover. Symptoms reported included an unexpected promotion of the standby after the primary stopped responding to health checks. Monitoring and OS traces showed memory usage climbing to ~95% on the primary node, PostgreSQL failing to respond to Patroni probe […]

Problem: A security scan of a deployed Apache Spark binary distribution reported two vulnerabilities located in Hive artifacts bundled with Spark: CVE-2024-29869 (org.apache.hive:hive-exec, fixed in Hive 4.0.1) and CVE-2024-23953 (org.apache.hive:hive-llap-common, fixed in Hive 4.0.1). The scan showed the vulnerable artifacts present in the shipped Spark distribution (the Spark 3.5.x line shipped a Hive 2.3.x fork […]

Problem: A production alert arrived with the subject indicating “client replied sms” and a Severity 3 priority: inbound reply messages were not being recorded by the downstream processing pipeline. The affected pipeline is an Apache Spark Structured Streaming job that consumes reply events from a message broker and uses a persistent checkpoint directory to track […]

Problem: An automated notification opened a Severity 2 case for Apache Spark with minimal details from the reporter. The only information attached to the alert was the product (Apache Spark) and the high-severity flag; no application logs or job identifiers were provided. Initial observable behavior from follow-up checks showed repeat Spark application failures on the […]

Problem: A high-traffic Patroni-managed PostgreSQL cluster experienced an automatic failover: the former primary (node A) lost leadership, and a secondary (node B) was successfully promoted. However, after the failover, node A failed to automatically rejoin the cluster as a replica. The symptoms unfolded in three stages. First, direct streaming replication was immediately rejected by the […]

Problem: A customer running Patroni-managed PostgreSQL (PostgreSQL 15.17, Patroni v3.3) had production databases ranging from >2-3 TB up to a ~52 TB physical replica. Rebuilding the empty DR site from production relied on manual pg_basebackup executions and manual post-backup recovery steps, which was too slow, operationally intensive, and prone to human error for such large volumes. The […]

Problem: An OLTP system using Patroni-managed PostgreSQL 15 experienced global slowdowns under load: CPU climbed to ~80%, large numbers of concurrent sessions, and application timeouts. The client reported bursts of traffic (hundreds of workers in parallel) routed through HAProxy to the leader, and provided PostgreSQL and Patroni logs, auto_explain output, and query samples. Key database […]

Problem: A production Patroni-managed PostgreSQL 15 cluster experienced periodic heavy queries that threatened availability. An example slow job ran for ~84 seconds and performed a full scan of a 1.8 TB partitioned table (arbor.CDR_DATA) that uses daily partitions starting in early April. Most clients connect through generic application users rather than distinct personal accounts. The […]

Problem: A 5-node Apache Cassandra 2.2.5 cluster (two data centers) reported severe per-node disk imbalance: each node was configured with five data_file_directories (e.g. /cassandra/data1/data … /cassandra/data5/data) but some mount points on individual nodes were near full (examples showed mounts at 93% and 95% used). On one node a particular keyspace (jessi) had large sstable directories […]