Data Management and Analytics

Problem: A client planned a rolling patch upgrade of a 5-node Apache Cassandra 4.1.x cluster running on cloud VMs, combined with a minor RHEL 9.x OS update. They submitted their runbook for review and asked whether the OS update affected JVM and Python support, whether there were known compatibility issues, and whether the plan carried […]

Problem: A production PostgreSQL 15.17 cluster managed by Patroni 3.3.2, with PgBouncer 1.25.1 in front of it, showed intermittent application failures during night operations. The application logged ODBC connection errors with SQLSTATE 08P01: “server login has been failing, cached error: connect failed (server_login_retry)”. The customer suspected PostgreSQL connection exhaustion. Metrics showed otherwise: max_connections was 4,000 […]

Problem: An on-prem Apache Cassandra 2.2.5 cluster had two of five nodes in its DR datacenter reporting DOWN in nodetool status, while the primary datacenter remained UP. The cluster had recently gone through a storage migration in which data was moved to larger disks by taking nodes offline and copying files. The failing nodes logged […]

Problem: A two-site PostgreSQL 13.4 deployment managed by Patroni experienced replication breakage at the DR site after any failover inside the primary site. The DR standby leader’s primary_conninfo was hardcoded to the former primary node’s address, so when the primary role moved to the other primary-site node the standby kept attempting to connect to the […]

Problem: A production Patroni-managed PostgreSQL 15.17 cluster (Patroni 3.3.2) reported persistent high memory consumption on database VMs, frequently exceeding 80% of allocated RAM and once reaching ~95% on a previous 300GB node. The deployment uses max_connections=4000 (cannot be reduced by the application), huge_pages configured as on with approximately 60,000 pages reserved, and vm.overcommit_memory=2 set per […]

Problem: An OpenSearch 2.9 cluster (3 nodes) generated a critical alert: one node reported swap utilization at ~80%, with the OpenSearch process identified as the primary swap consumer. The node’s runtime Java command line showed -Xms10g -Xmx10g and -XX:MaxDirectMemorySize=5368709120 (5GB). The environment reported ~23 GB total RAM. The customer asked how to reduce swap use […]

Problem: Production Patroni-managed PostgreSQL 15.17 cluster (Patroni 3.3.2) experienced intermittent performance degradation that caused application delays and connection resets. The environment uses a connection pooler (PGBouncer) fronting ~1,600 read-only connections while the primary accepted ~1,600 direct read-write connections. The team reported long query latencies, mass connection timeouts, and large temporary file creation observed in database […]

Problem: A production Patroni-managed PostgreSQL (community 15.17, Patroni 3.3.2) cluster experienced recurrent nightly performance degradation. Symptoms included large numbers of LWLock wait events and high query latency during night operations reported by the application and visible in the database analytical report. Patroni and PostgreSQL logs were provided for analysis. Operational context: PgBouncer was present as […]

Problem: A production Patroni-managed PostgreSQL 15 cluster experienced periodic heavy queries that threatened availability. An example slow job ran for ~84 seconds and performed a full scan of a 1.8 TB partitioned table (arbor.CDR_DATA) that uses daily partitions starting in early April. Most clients connect through generic application users rather than distinct personal accounts. The […]