Database

Problem: The client was planning an upgrade of the MySQL database server from Red Hat Enterprise Linux (RHEL) 7.x to RHEL 8.x. The MySQL version currently running on the server was 14.14 (Distrib 5.7.34). The client needed to understand the compatibility of this MySQL version with RHEL 8.x and sought guidance on any additional steps […]

Problem: The client reported a recurring issue with the application, which consistently threw an error: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend. He was uncertain about the root cause of this error and sought assistance to diagnose and resolve the issue. The client’s initial request was to understand what should be checked […]

Problem: The client, deploying a production Cassandra cluster on OpenShift, encountered persistent communication issues between two datacenters, each hosting three Cassandra nodes. Despite initial configuration and verification, nodes in Datacenter 1 (“RCKL”) and Datacenter 2 (“CLSP”) were unable to establish consistent communication. SSL handshake failures and inconsistent OpenSSL connectivity tests exacerbated the problem, indicating potential […]

Problem: The client had a 6-node multi-DC setup for Cassandra (3 nodes in PROD – East US2 and 3 nodes in DR – West US2) and needed to alter a keyspace. The keyspace was initially defined as follows: CREATE KEYSPACE bulk_api_management WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true; The client wanted […]

Problem: A client was experiencing data inconsistencies with the Lucene index on Cassandra and suspected improper shutdown procedures as the root cause. The kill -9 command was used to shut down Cassandra, which led to concerns that data was not being written to disk properly. The client sought guidance on the best way to shut […]

Problem: The client managed a 5-node Cassandra cluster across two data centers (DC1 and DR1), each containing 5 nodes. The data_file_directories were distributed across multiple mount points. On one node, the mount point /cassandra/data2 was nearly full due to a large table in the “jesi” keyspace, specifically the “service_monitoring_payload” table. This resulted in significant storage […]

Problem: The application fails to establish an SSL connection to the database, displaying the error “could not accept SSL connection: Success” in the database logs. Despite this, manual `psql` connections using SSL work fine. Process: Step 1 – Verify Connection String: Confirm that the application’s connection string includes the necessary SSL parameters. Example: postgresql://user:password@hostname:port/dbname?sslmode=require Step […]

Problem: The client experienced connection issues on the PostgreSQL database server, with an abnormally high number of connections reaching around 20,000 at a given time. The client asked for assistance from the expert team in identifying the possible reasons behind this issue. Based on the client’s analysis, there were multiple wait events on HAProxy, and […]

Problem: Restoring a Cassandra 3.11 snapshot to a 4.0.6 cluster using the nodetool refresh command results in an empty table, indicating a potential compatibility issue. This affects the DR environment, which needs to accurately replicate the PROD environment’s data. Solution: Step 1. Verify Snapshot Content: Use nodetool listsnapshots and a test environment to ensure the […]