Case Studies

Problem: A security scan of a deployed Apache Spark binary distribution reported two vulnerabilities located in Hive artifacts bundled with Spark: CVE-2024-29869 (org.apache.hive:hive-exec, fixed in Hive 4.0.1) and CVE-2024-23953 (org.apache.hive:hive-llap-common, fixed in Hive 4.0.1). The scan showed the vulnerable artifacts present in the shipped Spark distribution (the Spark 3.5.x line shipped a Hive 2.3.x fork […]

Problem: A high-traffic Patroni-managed PostgreSQL cluster experienced an automatic failover: the former primary (node A) lost leadership, and a secondary (node B) was successfully promoted. However, after the failover, node A failed to automatically rejoin the cluster as a replica. The symptoms unfolded in three stages. First, direct streaming replication was immediately rejected by the […]

Problem: A customer running Patroni-managed PostgreSQL (PostgreSQL 15.17, Patroni v3.3) had production databases ranging from >2-3 TB up to a ~52 TB physical replica. Rebuilding the empty DR site from production relied on manual pg_basebackup executions and manual post-backup recovery steps, which was too slow, operationally intensive, and prone to human error for such large volumes. The […]

Problem: An OLTP system using Patroni-managed PostgreSQL 15 experienced global slowdowns under load: CPU climbed to ~80%, large numbers of concurrent sessions, and application timeouts. The client reported bursts of traffic (hundreds of workers in parallel) routed through HAProxy to the leader, and provided PostgreSQL and Patroni logs, auto_explain output, and query samples. Key database […]

Problem: A production Patroni-managed PostgreSQL 15 cluster experienced periodic heavy queries that threatened availability. An example slow job ran for ~84 seconds and performed a full scan of a 1.8 TB partitioned table (arbor.CDR_DATA) that uses daily partitions starting in early April. Most clients connect through generic application users rather than distinct personal accounts. The […]

Problem: A 5-node Apache Cassandra 2.2.5 cluster (two data centers) reported severe per-node disk imbalance: each node was configured with five data_file_directories (e.g. /cassandra/data1/data … /cassandra/data5/data) but some mount points on individual nodes were near full (examples showed mounts at 93% and 95% used). On one node a particular keyspace (jessi) had large sstable directories […]

Problem: A production Patroni-managed PostgreSQL cluster (PostgreSQL v15.17, Patroni 3.3.2) experienced a primary process abort with SIGABRT during normal operation. Server logs reported that a server process was terminated by signal 6 (Aborted) and that the failed process was executing a COMMIT when the postmaster began terminating other server processes. Subsequent messages showed PostgreSQL could […]

Problem The database architecture reached its limit when a critical node stopped responding. A standby node experienced a PostgreSQL Patroni replica failure, repeatedly logging “incorrect resource manager data checksum” errors. The system stopped working because corrupted Write-Ahead Log segments completely broke the replication stream. A dangerous shortcut would involve running continuous base backups to force […]

Problem: The client needed to modernize their Zabbix v7.4.8 setup by migrating a Low-Level Discovery rule away from a static, local JSON file. They required best practices and an architectural recommendation for dynamically triggering and feeding the discovery rule from an external Windows server, while ensuring reliable and efficient updates. Process: Step 1: Architecture Review […]