Case Studies

Problem: A production Patroni cluster (Patroni 3.3.2, PostgreSQL 15.17 on VMs) exhibited very slow replication between nodes, producing significant replication lag and impacting services. Replication throughput on the standby was measured in single MB/s ranges, nightly replay lag reached hours, and a recent switchover required a full base backup because the replica was unable to […]

Problem: A client running a production Patroni PostgreSQL cluster on OpenZFS reported a critical incident. Replication between nodes had become extremely slow, a switchover had failed, and the cluster was running on a single node with no working standby, so a failure of that node would have caused total data loss. The client asked us […]

Problem: A client running multiple Patroni-managed PostgreSQL high-availability clusters in production added PgBouncer on the database VMs. During load tests, they saw no CPU or RAM savings compared with direct application connections. They asked for the correct PgBouncer configuration approach, whether the pooler should run locally or on separate servers, and how it should fit […]

Problem The database went completely offline after the Kubernetes worker nodes restarted at the same time. A three-node MariaDB 10.5 Galera cluster, deployed with the Bitnami mariadb-galera Helm chart, did not come back. The first pod, mariadb-galera-0, entered CrashLoopBackOff and logged It may not be safe to bootstrap the cluster from this node. It was […]

Problem The database stopped working after a routine host upgrade. Once the Proxmox VE hosts moved to version 9.2.3 with the new default kernel 7.0.6-2-pve, a Debian virtual machine running MariaDB 10.11 began receiving hard I/O errors (EIO) on its virtual disk. The guest kernel logged write failures such as I/O error, dev sdb, sector […]

Problem: A client planned a rolling patch upgrade of a 5-node Apache Cassandra 4.1.x cluster running on cloud VMs, combined with a minor RHEL 9.x OS update. They submitted their runbook for review and asked whether the OS update affected JVM and Python support, whether there were known compatibility issues, and whether the plan carried […]

Problem: Clients reported write queries failing with timeout errors against an Apache Cassandra DC despite all production nodes appearing UP. Application and database logs showed request failures with “Unexpected exception during request” and quorum-level write timeouts. System logs contained FailureDetector warnings about not marking nodes down due to large local pauses and MessagingService warnings about […]

Problem: A production PostgreSQL 15.17 cluster managed by Patroni 3.3.2, with PgBouncer 1.25.1 in front of it, showed intermittent application failures during night operations. The application logged ODBC connection errors with SQLSTATE 08P01: “server login has been failing, cached error: connect failed (server_login_retry)”. The customer suspected PostgreSQL connection exhaustion. Metrics showed otherwise: max_connections was 4,000 […]

Problem: An on-prem Apache Cassandra 2.2.5 cluster had two of five nodes in its DR datacenter reporting DOWN in nodetool status, while the primary datacenter remained UP. The cluster had recently gone through a storage migration in which data was moved to larger disks by taking nodes offline and copying files. The failing nodes logged […]