Database

Problem: Applications using Redisson MapCache began throwing RedisResponseTimeoutException errors (client timeout = 3000 ms) during eviction activity. Errors were raised while executing EVALSHA with context pointing to org.redisson.eviction.MapCacheEvictionTask; application threads were reporting “Unable to evict elements” outputs correlated with redisson-timer-* threads. Environment details: a Redis Cluster (client traffic on shard port 7000) served multiple integration […]

Problem: The customer reported degraded performance in a PostgreSQL query joining multiple views and tables, including V_CDD_PRF_PARTY_PARTY_R_DRF. Although the outer query applied a highly selective filter (C.FIRST_PARTY_KEY = 'CDD000248470'), the PostgreSQL optimizer did not push this predicate into the view. As a result, the view was fully evaluated, leading to increased CPU usage, higher I/O, […]

Problem: The client experienced an unexpected failover in a production PostgreSQL Patroni cluster running PostgreSQL Community version 15.8 and Patroni version 2.1.4 configured in asynchronous replication mode. Node A unexpectedly went offline and Patroni automatically promoted Node B to the primary node. The client needed to determine: The reason for the failover. Whether any transactions […]

Problem: After restoring a PostgreSQL 15.8 database from Commvault, the client noticed multiple errors during startup. Autovacuum began trying to remove temporary tables, but instead reported issues such as: “Dropping orphan temp table” “pg_attribute catalog is missing attributes” Indexes and constraints appearing only partially restored About 30 temp tables were affected, and several indexes or […]

Problem: The client operates a 6-node multi-DC replication setup for Cassandra, consisting of 3 nodes in the PROD datacenter and 3 nodes in the DR datacenter. They are planning to perform DDL changes, including altering tables to adjust the default_time_to_live parameter and dropping and recreating a table with a new definition. However, they are unsure […]

Problem: The client encountered a critical issue while starting one of their production pods during a Cassandra migration. Although the PostgreSQL migration completed successfully, the Cassandra migration failed with a com.datastax.oss.driver.api.core.AllNodesFailedException, indicating that the driver could not connect to any Cassandra nodes. This blocked the production deployment. Process: Step 1: Initial Analysis The logs revealed […]

Problem: A client managing a multi-datacenter Apache Cassandra setup observed an inconsistency in rack configuration between two datacenters. In the primary datacenter (`dc1`), all three nodes were configured with the same rack identifier (`rack1`), whereas in the secondary datacenter (`drdc`), each node was correctly configured with unique rack values (`rack1`, `rack2`, `rack3`). This raised concerns […]

Problem: The client encountered a critical issue in their Apache Cassandra setup: the Disaster Recovery (DR) cluster was consuming significantly more filesystem (FS) space than the primary cluster, despite having the same data replication strategy. As a result, one of the DR nodes hit 100% disk usage, causing the replica to go down. The client […]

Problem: The client wanted to integrate Apache Cassandra metrics into Prometheus for monitoring and alerting. While using JMX Exporter with Node Exporter, they noticed that too many unwanted metrics were being fetched. This resulted in performance issues such as delayed response times from the curl command and unnecessary load on Prometheus. Additionally, key metrics like […]