Problem: A production Patroni cluster (Patroni 3.3.2, PostgreSQL 15.17 on VMs) exhibited very slow replication between nodes, producing significant replication lag and impacting services. Replication throughput on the standby was measured in single MB/s ranges, nightly replay lag reached hours, and a recent switchover required a full base backup because the replica was unable to […]
Knowledge Base Case Studies Data Management and Analytics Database 29 Oct 2026 Resolving severe Patroni PostgreSQL replication slowdown and switchover failure caused by OpenZFS settingsProblem: A client running a production Patroni PostgreSQL cluster on OpenZFS reported a critical incident. Replication between nodes had become extremely slow, a switchover had failed, and the cluster was running on a single node with no working standby, so a failure of that node would have caused total data loss. The client asked us […]
Knowledge Base Case Studies Data Management and Analytics Database 27 Oct 2026 Optimizing PgBouncer placement and configuration for Patroni PostgreSQL clustersProblem: A client running multiple Patroni-managed PostgreSQL high-availability clusters in production added PgBouncer on the database VMs. During load tests, they saw no CPU or RAM savings compared with direct application connections. They asked for the correct PgBouncer configuration approach, whether the pooler should run locally or on separate servers, and how it should fit […]
Knowledge Base Case Studies Data Management and Analytics Database 26 Oct 2026 Recovering a MariaDB Galera Cluster on Kubernetes Stuck in CrashLoopBackOffProblem The database went completely offline after the Kubernetes worker nodes restarted at the same time. A three-node MariaDB 10.5 Galera cluster, deployed with the Bitnami mariadb-galera Helm chart, did not come back. The first pod, mariadb-galera-0, entered CrashLoopBackOff and logged It may not be safe to bootstrap the cluster from this node. It was […]
Database 12 Oct 2026 Reducing JVM heap to stop Cassandra write timeouts caused by long GC pausesProblem: Clients reported write queries failing with timeout errors against an Apache Cassandra DC despite all production nodes appearing UP. Application and database logs showed request failures with “Unexpected exception during request” and quorum-level write timeouts. System logs contained FailureDetector warnings about not marking nodes down due to large local pauses and MessagingService warnings about […]
Knowledge Base Case Studies Data Management and Analytics Database 5 Oct 2026 Intermittent PgBouncer login rejections caused by IPv6 localhost resolutionProblem: A production PostgreSQL 15.17 cluster managed by Patroni 3.3.2, with PgBouncer 1.25.1 in front of it, showed intermittent application failures during night operations. The application logged ODBC connection errors with SQLSTATE 08P01: “server login has been failing, cached error: connect failed (server_login_retry)”. The customer suspected PostgreSQL connection exhaustion. Metrics showed otherwise: max_connections was 4,000 […]
Knowledge Base Data Management and Analytics Database Case Studies Data Management and Analytics Database 18 Sep 2026 PostgreSQL WAL archive failures caused by incomplete copies and full filesystemsProblem: A production PostgreSQL 15 instance experienced a Severity‑1 outage after the filesystem containing WAL files reached 100% usage. The customer observed failed Commvault log-backup jobs and that WAL segments were not being archived: SHOW archive_command returned an external script invocation (/path/to/archive_wal.sh %p %f). The shipped copy script used a “test ! -f … && […]
Knowledge Base Case Studies Data Management and Analytics Database 16 Sep 2026 PostgreSQL replication slot repeatedly went inactive; replica walreceiver stopped and did not reconnectProblem: A Patroni-managed PostgreSQL cluster showed recurring replication interruptions: the replica repeatedly reported a large streaming lag (initially ~565 MB, later observed ~5.5 GB) and the cluster’s physical replication slot on the leader appeared inactive (active = false, wal_status = reserved). Patroni logs on the primary had not recorded recent cluster events and appeared to […]
Knowledge Base Case Studies Data Management and Analytics Database 9 Sep 2026 PostgreSQL SSL handshake failures causing intermittent connection errors in a Patroni clusterProblem: A production PostgreSQL 15.8 deployment running inside a Patroni-managed HA cluster (Patroni 2.1.4) reported transient application failures where processes could not connect and surfaced the message “PostgreSQL connection error, Cannot connect to FX DB”. Database logs showed repeated messages indicating SSL accept-stage failures with the text “could not accept SSL connection: EOF detected”. The […]
Knowledge Base Case Studies Data Management and Analytics Database